Transform and CNN fusion-based small focus segmentation and feature learning method

Abstract

The invention relates to a small focus segmentation and feature learning method based on Transform and CNN fusion. The method comprises the following steps: preprocessing an input three-dimensional image to obtain original input features, and inputting the original input features into a segmentation network model of an encoder-bottleneck-decoder structure; after passing through an encoder based on channel-Transform fusion, obtaining fusion features through a bottleneck used for expanding a receptive field; wherein the encoder comprises a channel convolution module for local feature capture and a local attention mechanism module for global modeling; the preprocessed input features pass through a channel convolution module and a local attention mechanism module in sequence, and then pass through a bottleneck containing cavity convolution to carry out feature fusion; and inputting the fused features into a decoder based on a jump connection mechanism to gradually restore the low-resolution features to the same size as the input image, and generating a final segmentation result or classification result. According to the method, efficient feature extraction and fine prediction of three-dimensional images such as medical images can be realized.

Description

Technical Field

[0001] The present invention relates to a small lesion segmentation and feature learning method based on Transformer and CNN fusion, and belongs to the fields of artificial intelligence, image segmentation and medical image processing. Background Art

[0002] Medical images are important tools for clinical disease screening, diagnosis, treatment guidance and evaluation. Reliable medical image segmentation and classification can help doctors interpret and analyze medical images quickly and accurately. However, analyzing these images is time-consuming and arduous. Even experienced doctors are prone to misdiagnosis due to visual fatigue and impatience. Therefore, the development of computer-aided diagnosis (CAD) systems is of great significance to help doctors automatically and accurately analyze medical images. With the rapid development of artificial intelligence, the analysis and processing technology of medical images has gradually become one of the most changing and fastest-growing fields in medical information. However, many traditional medical image processing methods often require a large number of medical image training samples, but some medical image samples are small and unevenly distributed, which will lead to poor training and learning effects.

[0003] Among them, most diseases only present tiny lesions in the initial stage. These lesions may be insignificant or even difficult to detect, but it is these tiny lesions that often carry important information about the future development of the disease. If these tiny changes can be accurately captured and identified, more treatment options can be provided to patients in early diagnosis, and they greatly affect the treatment effect and the patient's prognosis. For example, cancer may only have tiny lesions in the early stages. If these lesions are discovered early through medical imaging technology and treatment measures are taken in the early stage, the mortality rate of patients can be greatly reduced. However, due to the great diversity of shapes, positions, and sizes of tiny lesions, the edges of tiny lesions are generally blurred in CT and MRI images, which increases the difficulty of segmentation. In addition, there are class imbalance and instance imbalance problems for lesion segmentation, that is, the number of backgrounds will be greater than the number of foregrounds, larger lesions will dominate smaller lesions, and the details of some larger lesions may also be ignored. In addition, image segmentation currently usually focuses on the segmentation of specific diseases such as stroke lesions or lung tumors, and the model can often only be trained for a single type of disease, resulting in the lack of generalization ability of the model on other disease types. How to solve the above problems is the main difficulty in lesion segmentation at present.

[0004] In summary, most existing medical image segmentation networks, especially models for lesion segmentation, face multiple problems. For example, due to the complexity and high dissimilarity of lesions, they cannot solve the problem of class imbalance, resulting in the inability to determine the edges and shapes of lesions, thereby confusing the segmentation of lesions and non-lesions. For another example, most networks use a fully supervised approach, which relies heavily on the dataset and weakens the generalization ability of the network. For another example, most CNN (Convolutional Neural Networks)-based segmentation networks improve segmentation accuracy by increasing the depth of convolution, which not only impairs the learning rate of the network, but also risks losing surface feature information as the layers deepen. Summary of the invention

[0005] The present invention provides a small lesion segmentation and feature learning method based on Transformer and CNN fusion, aiming to solve at least one of the technical problems existing in the prior art.

[0006] The technical solution of the present invention relates to a small lesion segmentation and feature learning method based on Transformer and CNN fusion. The method according to the present invention comprises the following steps:

[0007] S100, performing data preprocessing on the input three-dimensional image to obtain original input features, and inputting the original features into a segmentation network model using an encoder-bottleneck-decoder structure;

[0008] S200, the original input features pass through an encoder based on channel-Transformer fusion, and then pass through a bottleneck for expanding the receptive field to obtain fusion features; wherein the encoder includes a channel convolution module for local feature capture and a local attention mechanism module for global modeling; the preprocessed input features pass through the channel convolution module and the local attention mechanism module successively, and then pass through a bottleneck including a dilated convolution for feature fusion;

[0009] S300, inputting the obtained fusion features into a decoder based on a skip connection mechanism to gradually restore the low-resolution features to the same size as the input image and generate a final segmentation result or classification result.

[0010] Further, the encoder includes a Transformer layer, and the CTE module in each Transformer layer includes the channel convolution module and the local attention mechanism module;

[0011] Wherein, the step S200 includes:

[0012] S210, dividing the original input features into non-overlapping small patches through a patch embedding module, and generating feature embedding for each patch to form an initial feature representation;

[0013] S220, the obtained initial features are passed through the channel convolution module and the local attention mechanism module in the first Transformer layer to perform pixel-level addition;

[0014] S230, passing through the second and third Transformer layers successively, so as to continue to extract deep features and perform downsampling through the subsequent channel convolution module, and to capture detail information through the subsequent partial attention mechanism module;

[0015] S240, by introducing the bottleneck of the dilated convolution, the receptive field is expanded by setting different expansion rates to capture long-distance feature dependencies.

[0016] Further, in step S200, the channel convolution module is configured as follows:

[0017] The first convolutional layer used to transform the input feature map from in_channels to out_channels;

[0018] A channel mixing module where both the input channel and the output channel are the width of the input feature map;

[0019] Batch Normalization layers for normalizing feature maps to accelerate model convergence and stabilize training;

[0020] Final convolutional layer with out_channels*2 input channels and out_channels output channels.

[0021] Further, in step S200, the local attention block of the local attention mechanism module includes the following processing operations:

[0022] Input features are operated through layer normalization layers for accelerated training and stability;

[0023] Through the local window self-attention operation, the feature map is divided into multiple non-overlapping small blocks (windows), and then the self-attention is calculated independently in each window. Through the calculation of local self-attention, in each window, each input position calculates the self-attention relationship with all other positions, and a new representation is obtained based on the weighted average of the obtained relationship;

[0024] According to the multi-head mechanism of multi-head self-attention operation, each head learns different feature representations in different subspaces to perform self-attention calculations of the local attention block within each window;

[0025] The features obtained from the self-attention operation are operated by a fully connected multi-layer perceptron to further enhance the nonlinear representation ability of the model, wherein the multi-layer perceptron operation passes through two fully connected layers, the first fully connected layer expands the feature dimension, and the second fully connected layer reduces the dimension back to the original size;

[0026] Further, in step S200,

[0027] Each local window self-attention operation, multi-head self-attention operation, and multi-layer perceptron operation is connected with a residual connection to add the input and output to facilitate the flow of information;

[0028] Each local window self-attention operation, multi-head self-attention operation, and multi-layer perceptron operation is followed by a layer normalization operation to speed up training and improve the stability of the model.

[0029] Further, in step S300, the decoder includes three transposed convolution modules; the decoding process based on the jump connection mechanism includes: inputting the fused features output by the encoder into the decoder to restore the image resolution layer by layer; enlarging the length, width and height of the image layer by layer through the transposed convolution operation, and gradually halving the number of channels to restore it to a resolution close to that of the input image for layer-by-layer restoration; and restoring the output image to the resolution of the original input image through the last layer of convolution of the decoder to obtain the final restoration.

[0030] Furthermore, in the segmentation network model training, the forward propagation step includes:

[0031] D10, initial convolution operation to complete preliminary feature extraction;

[0032] D20, batch normalization to alleviate the problem of internal covariate shift;

[0033] D30, channel mixing operation, after exchanging the channel dimension and width dimension of the normalized feature map, mix the channel information through a 3x3x3 convolution layer, and transpose the mixed feature map back to the original dimension;

[0034] D40, pixel-level feature fusion, to add the channel-mixed feature map to the normalized feature map;

[0035] D50, batch normalization again;

[0036] D60, semantic mixed operations;

[0037] D70, feature cascade and final fusion to concatenate the semantic features semantic_features and the initial normalized feature map bn_features along the channel dimension, reduce the number of channels back to out_channels through the conv_last convolution layer, and finally output the processed feature map.

[0038] Furthermore, the loss function of the segmentation network model training is a fusion of the Dice loss function and the Focal loss function; wherein, by mixing the Dice loss function L dice And the Focal loss function L focal The loss function L obtained total It is expressed as follows:

[0039]

[0040] L total =L dice +λL focal

[0041] Where P ti is the model’s predicted probability for category i; α i is the weight parameter of category i, which is set dynamically according to the frequency of the category, that is, γ is a factor used in Focal Loss to adjust easy-to-classify samples; λ is a trade-off parameter between Dice Loss and Focal Loss.

[0042] The technical solution of the present invention also relates to a computer-readable storage medium on which program instructions are stored, and the above-mentioned method is implemented when the program instructions are executed by a processor.

[0043] The technical solution of the present invention also relates to a small lesion segmentation and feature learning system based on deep learning, wherein the system comprises a computer device which contains the above-mentioned computer-readable storage medium.

[0044] The beneficial effects of the present invention are as follows:

[0045] The present invention adopts a method based on deep learning and a U-shaped structure encoder as the basic framework, taking into account the fusion of global and local features, the training stability of the model and the accuracy of the results. It is an efficient 3D image processing solution that can realize efficient feature extraction and fine prediction of three-dimensional images such as medical images. The present invention can realize the fusion of global and local information, the encoder extracts local detail features, the hole convolution expands the receptive field to capture global dependencies, and the decoder integrates the two to achieve fine-grained prediction. The present invention can realize multi-resolution information flow, retain high-resolution spatial information through jump connections, and enable the decoder to supplement low-level details when reconstructing features. The present invention adopts an adaptive loss function, and by combining Dice and Focal losses, it effectively copes with the problem of sample category imbalance and improves the model's adaptability to complex tasks. The present invention adopts a modular design of encoder-bottleneck-decoder, which is convenient for model optimization and expansion, and can be applied to a variety of three-dimensional image analysis tasks, such as medical image segmentation, target detection, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the structure of the segmentation network model according to the method of the present invention.

[0047] Figure 2 Schematic diagram of the structure of the CET module according to the method of the present invention. DETAILED DESCRIPTION

[0048] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0049] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms of "," "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art. The terms used in this specification are intended to describe specific embodiments only and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0050] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0051] Reference Figure 1 to Figure 2 In some embodiments, the small lesion segmentation and feature learning method based on Transformer and CNN fusion according to the present invention comprises at least the following steps:

[0052] S100, performing data preprocessing on the input three-dimensional image to obtain original input features, and inputting the original features into a segmentation network model using an encoder-bottleneck-decoder structure;

[0053] S200, the original input features pass through the encoder based on channel-Transformer fusion, and then pass through the bottleneck for expanding the receptive field to obtain fusion features; wherein the encoder includes a channel convolution module for local feature capture and a local attention mechanism module for global modeling; the preprocessed input features pass through the channel convolution module and the local attention mechanism module successively, and then pass through the bottleneck containing the dilated convolution for feature fusion;

[0054] S300, inputting the obtained fusion features into a decoder based on a skip connection mechanism to gradually restore the low-resolution features to the same size as the input image and generate a final segmentation result or classification result.

[0055] The present invention adopts a method based on deep learning and a U-shaped structure encoder as the basic framework, taking into account the fusion of global and local features, the training stability of the model and the accuracy of the results. It is an efficient 3D image processing solution that can realize efficient feature extraction and fine prediction of three-dimensional images such as medical images. The present invention can realize the fusion of global and local information, the encoder extracts local detail features, the hole convolution expands the receptive field to capture global dependencies, and the decoder integrates the two to achieve fine-grained prediction. The present invention can realize multi-resolution information flow, retain high-resolution spatial information through jump connections, and enable the decoder to supplement low-level details when reconstructing features. The present invention adopts an adaptive loss function, and by combining Dice and Focal losses, it effectively copes with the problem of sample category imbalance and improves the model's adaptability to complex tasks. The present invention adopts a modular design of encoder-bottleneck-decoder, which is convenient for model optimization and expansion, and can be applied to a variety of three-dimensional image analysis tasks, such as medical image segmentation, target detection, etc.

[0056] The segmentation network model F-transformer of the present invention adopts a channel-Transformer fusion encoder, and adds the UNETR encoder with the output features of the channel convolution at the same level. The encoder of the present invention does not simply splice the two features together, but adopts a feature addition method. Through the addition operation, the fusion of the Transformer encoder and the channel convolution output features is realized, so that the model can make full use of the information of the two features. According to the disadvantages of the Transformer model relative to CNN in extracting local information, that is, the Transformer performs well in extracting long-distance dependent information, but is not as powerful as CNN in capturing local information, and according to the characteristics that the Transformer is not sensitive enough to position information and causes information loss, the improved encoder design adopted in the present invention can make up for the above-mentioned limitations of the Transformer model. It jointly utilizes the hierarchical output of the UNETR encoder and the channel convolution output of the corresponding level, and maximizes the advantages of the Transformer model in extracting long-distance dependencies through a fusion strategy. At the same time, the correlation information between channels is further captured with the help of channel convolution. Especially for 3D image data, channel convolution can more comprehensively capture the relevant information between channels, thereby improving the model's ability to learn complex image structures and features.

[0057] The network structure adopted in the present invention is a fusion network based on local attention mechanism and channel convolution, which is specially designed for efficient feature extraction and image segmentation tasks. Through multi-level encoders, fusion modules and decoders, it can realize multi-resolution feature capture, fusion and reconstruction of input data.

[0058] In some embodiments, in the input preprocessing of the network model of the present invention, the input image is three-dimensional data with a size of B×C×H×W×D, where B represents the batch size; C represents the number of channels, and the default is C=1 (grayscale image); H, W and D represent the height, width and depth (i.e., three-dimensional size) of the image. The present invention performs a data normalization operation, which adjusts the pixel values ​​of the input image to a standardized value suitable for the model value range through data normalization to ensure efficient training of the model.

[0059] In one application embodiment, the input 3D image is standardized and preprocessed to meet the input requirements of the model. Specifically, the original 3D image is cropped or interpolated to a uniform size of 256×256×256 to ensure the scale consistency of the data and facilitate the subsequent batch training and reasoning of the model.

[0060] In some embodiments, the preprocessed three-dimensional image is used as input to an encoder with a U-shaped structure, and a channel-Transformer fusion-based encoder is used to extract deep features. Among them, the encoder used in the present invention combines the global modeling ability of the Transformer and the local feature capture ability of the channel convolution, and achieves efficient fusion of the two by adding features. The encoder combined with convolution and Transformer can give full play to the advantages of the Transformer in long-distance dependent information, while making up for its shortcomings in capturing local information.

[0061] Specifically, the encoder of the present invention extracts local features through channel convolution operations, so that the information between channels is enhanced, so that the correlation between channels can be effectively captured when processing multi-channel data (such as three-dimensional images). In addition, the encoder of the present invention uses Transformer to capture long-distance dependencies, and uses the advantages of Transformer in extracting information in a global range, especially long-distance feature relationships, and uses its flexible attention mechanism to dynamically focus on the salient areas of the image.

[0062] It should be noted that Transformer (Transformer model architecture) is a deep learning model architecture used for natural language processing (NLP) and other sequence-to-sequence tasks. The core feature of the Transformer model is the use of a self-attention mechanism, which enables the model to consider all positions in the input sequence at the same time. However, compared with convolutional neural networks (CNNs), Transformer has the limitation of insufficient local information extraction. Specifically, it has disadvantages in capturing details and local texture features. For example, due to the lack of explicit position encoding, it is insensitive to position information, which can easily lead to partial loss of spatial information.

[0063] In an application embodiment, the encoder of the network model of the present invention consists of two parts, namely, a channel convolution module and a local attention mechanism module, and each layer realizes feature extraction and step-by-step downsampling. Specifically, in the channel convolution module, each layer contains operations: convolution operation to extract features; batch normalization (BN) to accelerate training and reduce overfitting; channel mixing and semantic mixing to integrate different channel and semantic features. Further, in the local attention mechanism module, the input image is first divided into non-overlapping small patches by the patch embedding module, and feature embedding is generated for each patch to form an initial feature representation. The local attention mechanism captures the local features of the input image and strengthens the expression of relevant information. Specifically, in the first layer, the image passes through the feature embedding layer and after passing through the first channel convolution module, it is processed by maximum pooling and local attention mechanism. The output results of the two modules are added at the pixel level; in the 2nd to 3rd layers, the subsequent convolution modules continue to extract deep features and perform downsampling, while the local attention mechanism captures detail information; in the last layer output, the obtained features are processed by the hole convolution module and passed to the bottleneck layer for fusion. Further, see Figure 1 In the Transformer layer, three Transformer layers are used, and each layer performs multi-head self-attention mechanism and multi-layer perceptron (MLP) operation. It captures the local and global feature information of the image through self-attention mechanism and window partitioning.

[0064] In some embodiments, see Figure 1 and Figure 2 The CTE module of the model of the present invention mainly consists of channel convolution and local attention mechanism. Among them, for channel convolution, the channel module of the present invention is a deep learning module for feature fusion and processing, which enhances the feature expression ability through channel mixing and semantic mixing operations.

[0065] Specifically, the module initialization part of the channel convolution includes input parameters, where in_channels is the number of channels of the input feature; out_channels is the number of channels of the output feature; height and width are the height and width of the input feature map, respectively, which are used for the operation of the channel mixing module.

[0066] Furthermore, the module initialization content of the channel convolution includes the first convolution layer (conv), the channel mixing module (channel_mix), the batch normalization layer (bn), the semantic mixing module (semantic_mix) and the final convolution layer (conv_last). Among them, the first convolution layer is a standard two-dimensional convolution layer, which is used to convert the input feature map from in_channels to out_channels, with a convolution kernel size of 3x3x3, unit padding (padding=1), and no bias (bias=False) is used. The channel mixing module contains a two-dimensional convolution layer, the input channel and the output channel are both the width width of the input feature map, the convolution kernel size is 3x3x3, unit padding (padding=1), and its two-dimensional convolution layer is followed by a GELU activation function to introduce nonlinearity. The batch normalization layer (bn) is used to normalize the feature map processed by the convolution layer, in order to accelerate the convergence of the model and stabilize the training. The semantic mixing module contains a 2D convolutional layer with out_channels as input and output channels, a kernel size of 3x3x3, and unit padding (padding=1), followed by a GELU activation function. The final convolutional layer is a 2D convolutional layer with out_channels*2 input channels (doubled after cascading), out_channels as output channels, a kernel size of 3x3x3, and unit padding.

[0067] The CTE module adopted by the model of the present invention enhances the channel dimension information of the feature map through channel mixing exchange dimension and convolution operation, and improves the expression ability between channels. And through semantic mixing, higher-order semantic information is further extracted through convolution operation after pixel-level fusion. And through the fusion mechanism, channel mixing and semantic mixing are combined, and deep fusion of features is achieved through cascade operation. And multiple normalization is performed, that is, batch normalization is used multiple times in convolution, fusion and mixing steps to ensure the stability of feature distribution.

[0068] The present invention adopts a local attention mechanism module, in which a local attention block can be divided into the following stages: layer normalization, window-based self-attention, multi-head self-attention, shifted window, MLP layer, and residual connection.

[0069] Among them, for the input layer and layer normalization, the present invention receives a tensor through the input layer, the local attention mechanism block is usually a feature map output by Patch Embedding or the previous Transformer block, where the input feature usually has a shape of [B, out_channels, H, W, D]. And, the input feature is layer normalized through the layer normalization layer (LayerNorm), which helps to accelerate training and stability.

[0070] Among them, for local self-attention (Window-based Self Attention), it applies the local window self-attention mechanism in each Transformer block through local window self-attention, that is, the feature map is divided into multiple non-overlapping small blocks (windows), and then the self-attention is calculated independently in each window. And, through the calculation of local self-attention, in each window, each input position calculates the self-attention relationship with all other positions, and a new representation is obtained by weighted average based on these relationships, which helps to capture local spatial information, wherein the number of channels x2 per local attention block is halved in length, width and height. Among them, the self-attention calculation method in each window is consistent with the standard Transformer, including the following steps: Query, Key, Value calculation, that is, for each position in the input feature map, the corresponding query (Query), key (Key) and value (Value) representation are calculated. The self-attention calculation of the present invention calculates the attention weight of each position for each window to obtain a new representation.

[0071] Among them, for multi-head self-attention, the self-attention calculation of its local attention block in each window is implemented through a multi-head mechanism, in which each head learns different feature representations in different subspaces, thereby enhancing the expressiveness of the model. The calculation steps of multi-head self-attention include: for each window, first map the input Query, Key, and Value vectors to the subspaces of multiple heads, then perform self-attention calculations on each head, and finally concatenate the outputs of all heads to obtain the final output.

[0072] Among them, for the MLP layer (multi-layer perceptron), after the self-attention operation, the features will be processed by a fully connected MLP layer, where this layer is usually composed of two fully connected layers (each followed by a ReLU activation function). Specifically, the first fully connected layer expands the feature dimension, and the second fully connected layer reduces the dimension back to the original size. The MLP layer of the present invention can further enhance the nonlinear representation ability of the model and provide richer feature mapping.

[0073] Among them, for residual connection and layer normalization, in the residual connection, there is a residual connection after each operation (self-attention and MLP layer), which adds the input and output to achieve the flow of information and prevent the gradient disappearance problem. In layer normalization, there is a layer normalization operation after each submodule (attention and MLP), which helps to speed up training and improve the stability of the model.

[0074] In some embodiments, the present invention introduces dilated convolution at the bottleneck of the U-shaped structure encoder, and expands the receptive field by setting different dilation rates, thereby capturing long-distance feature dependencies without increasing the computational cost. The present invention improves the bottleneck layer to enhance the perception of global information and enable the model to effectively integrate remote contextual relationships. Specifically, the output features of the bottleneck layer will contain rich global semantic information, providing a more comprehensive feature expression for subsequent decoding.

[0075] Specifically, the bottleneck layer fusion of the encoder of the present invention can generate a fused feature representation containing rich global dependencies and local details by expanding the receptive field and combining local and global features. Among them, the bottleneck input of the encoder is the fused feature from the channel convolution and the local attention mechanism, which generates the fused feature out_fused as the input of the decoder part. It should be noted that for the extended receptive field enhancement unit of the present invention, since the data becomes smaller in size after multiple downsamplings in the bottleneck stage, the use of hole convolution can cover the receptive field range more widely and perform global feature extraction. In this case, the hole convolution can more effectively capture the association between distant pixels, provide more comprehensive contextual information, and help the model better understand the overall data characteristics. This larger range of receptive fields can enhance the model's global understanding and representation capabilities for images or data, and provide a richer and more accurate information basis for subsequent tasks.

[0076] In some embodiments, the decoder of the present invention is composed of three transposed convolution modules, which are used to restore the resolution of the feature map layer by layer. A skip connection is also introduced in the decoding process to better fuse the high-level features of the encoder. Among them, in the skip connection, each skip connection fuses the features of the corresponding layer of the encoder with the features of the current layer of the decoder, thereby retaining more detailed information.

[0077] Specifically, in the decoding process of the present invention, the fused feature out_fused is used as the input of the decoder to restore the image resolution layer by layer; during the layer-by-layer restoration, the decoder enlarges the length, width and height of the image layer by layer through the transposed convolution operation, and the number of channels is gradually halved to restore it to a resolution close to that of the input image; the final restoration of the decoder is at the last layer, and the output image of the decoder is restored to the resolution of the original input image.

[0078] In some embodiments, the present invention adopts a skip connection mechanism to extract features from each layer of the encoder (especially the high-resolution layer), and splices them with the corresponding layer of the decoder by direct transmission, thereby preventing the loss of deep features, retaining high-resolution detail information, and accelerating gradient transmission, thereby alleviating the gradient vanishing problem of deep networks.

[0079] In an application embodiment, the decoder of the present invention uses a step-by-step upsampling method to restore the spatial resolution of the feature map. Specifically, the input source of the decoder includes the information that the input of the current decoder is spliced ​​by the output of the previous decoder and the jump connection features of the corresponding layer (splicing along the channel dimension), and the high-resolution information extracted and integrated by passing the spliced ​​feature map to the convolution layer of the decoder. Furthermore, the decoder of the present invention uses 3D transposed convolution for sampling to gradually restore the low-resolution features to the same size as the input image.

[0080] In one embodiment, the last layer of the decoder of the present invention uses 1×1×1 convolution to complete the final feature mapping, thereby mapping the decoded features to the number of channels required for the output through a dimensionality reduction operation, and restoring the features to the original feature size to generate the final prediction result. Specifically, the final prediction of the model is: the last layer (prediction layer) maps the decoded features to the number of categories through a 1×1 convolution to generate the final segmentation result or classification result.

[0081] In some embodiments, in the model training of the present invention, the specific steps of forward propagation (forward) include: first, initial convolution operation (self.conv); second, batch normalization (self.bn); third, channel mixing operation (self.channel_mix); fourth, pixel-level feature fusion; fifth, batch normalization again; sixth, semantic mixing operation (self.semantic_mix); seventh, feature cascading and final fusion.

[0082] Specifically, the initial convolution operation passes the input feature map through the conv convolution layer to complete the preliminary feature extraction, and the output shape is [B, out_channels, H, W, D]. Batch normalization performs batch normalization on the convolutional feature map to alleviate the internal covariate shift problem and obtain the normalized feature map bn_features. The channel mixing operation first converts the shape of the normalized feature map bn_features from [B, out_channels, H, W, D] to [B, W, H, out_channels, D], that is, exchanges the channel dimension and the width dimension, and then inputs it into the channel mixing module, mixes the channel information through a 3x3x3 convolution layer, activates the nonlinear relationship, and finally transposes the mixed feature map back to the original dimension [B, out_channels, H, W, D]. Pixel-level feature fusion adds the channel-mixed feature map to the normalized feature map bn_features to achieve pixel-level fusion and obtain fused_features. Batch normalization is performed again on the pixel-level fused feature map fused_features to obtain bn_features1. The semantic mixing operation inputs the normalized feature map bn_features1 into the semantic mixing module, extracts semantic features through a 3x3x3 convolution layer, and then activates through GELU. The result of semantic mixing is semantic_features. Feature concatenation and final fusion concatenate the semantic features semantic_features and the initial normalized feature map bn_features along the channel dimension (the shape becomes [B, out_channels*2, H, W, D]), and the number of channels is reduced back to out_channels through the conv_last convolution layer, and the processed feature map is finally output.

[0083] In some embodiments, the Dice loss function and the Focal loss function are used in combination in the loss function of the model training of the present invention to adapt to the segmentation task of unbalanced sample distribution. Specifically, the Dice loss function is used to evaluate the degree of overlap between the predicted result and the true label to suit the segmentation of uneven areas. The Focal loss function is used to introduce a weight mechanism to reduce the impact of easy-to-divide samples on the loss and enhance the model's attention to difficult samples (such as small targets or edge areas). The present invention can effectively correct the problem of imbalance in the ratio of positive and negative samples and improve the overall robustness of the model by combining the two.

[0084] It should be noted that when only the Dice loss function is used, the model may perform poorly in the face of class imbalance. The Dice loss function treats each pixel equally, and in some cases, the imbalance between pixel categories may cause the model to pay more attention to categories with a larger number and ignore categories with a smaller number. This imbalance may affect the model's learning of important areas, especially for tasks such as medical image segmentation, where accurate identification of minority categories (such as lesion areas) is crucial. In order to solve the problem of class imbalance, the present invention introduces the Focal Loss function. Among them, FocalLoss assigns higher weights to difficult samples (usually minority categories) during training, so that the model is more focused on learning these samples, thereby improving the performance of the model in the face of unbalanced class distribution. While Dice Loss is used to measure the similarity between the predicted segmentation result and the true label, Focal Loss is a weighted loss function that adds different weights to the loss of the sample according to the difficulty of the sample.

[0085] In an application embodiment, by mixing the Dice loss function L dice And the Focal loss function L focal , we get a loss function L total .in,

[0086]

[0087] L total =L dice +λL focal

[0088] Where P ti is the model’s predicted probability for category i; α i is the weight parameter of category i, which is set dynamically according to the frequency of the category, that is, γ is a factor used in Focal Loss to adjust easy-to-classify samples; λ is a trade-off parameter between Dice Loss and Focal Loss.

[0089] The present invention uses this mixed loss function L total , in medical image segmentation tasks, it can better balance the sensitivity to small-sized lesions and positive and negative samples, and improve the stability and performance of the model.

[0090] It is understandable that, in the segmentation of stroke lesions in MRI three-dimensional images, due to the sample imbalance problem of the data, in order to improve the accuracy and efficiency of the segmentation of small lesions, the data must be preprocessed, which includes but is not limited to image enhancement, denoising, standardization, and possible image registration and alignment to ensure that the model obtains high-quality, consistent data when input, and a suitable loss function is also used to optimize this problem. After obtaining the segmentation results, post-processing the results can further improve the segmentation accuracy. In order to ensure the generalization and robustness of the model, the model must also be strictly cross-validated and evaluated to ensure performance. Other methods usually slice 3D images into 2D images for segmentation, which easily loses the feature connection in space and limits the capture of stereoscopic information in the image. Therefore, the present invention uses complete 3D images for training and segmentation.

[0091] Furthermore, when segmenting lesions, how to detect and locate tiny lesions is a key issue. First, in order to improve the accuracy of segmentation, deep learning models can be designed to have higher perception capabilities by optimizing neural network architecture and feature extraction methods to accurately capture and characterize the tiny features of small lesions. The accuracy of the model is crucial for early diagnosis and treatment plan formulation of patients. Therefore, accuracy needs to be used as a key indicator of research to ensure that small lesions are completely and accurately located and segmented. Secondly, enhance sensitivity to specific lesions. There are various types of small lesions, including tumors, stroke lesions, etc. For different types of lesions, it is necessary to develop generalized feature extraction and segmentation methods. Enable the model to effectively distinguish and accurately locate different types of small lesions. Therefore, it is necessary to deeply study the characteristics of each lesion, combined with deep learning technology, so that the model can identify and segment specific types of tiny lesions in a more targeted and generalized manner, and achieve better segmentation. Finally, transform the research results into practical clinical application value to provide doctors with reliable auxiliary diagnostic tools. Through deep learning technology, the present invention adopts an efficient and reliable segmentation model and integrates it into clinical medical practice, which helps to improve the diagnostic accuracy and efficiency of doctors and help medical teams better formulate personalized treatment plans.

[0092] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0093] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0094] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, an RSM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0095] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0096] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A small lesion segmentation and feature learning method based on Transformer and CNN fusion, characterized in that: The method comprises the following steps: S100, performing data preprocessing on the input three-dimensional image to obtain original input features, and inputting the original features into a segmentation network model using an encoder-bottleneck-decoder structure; S200, the original input features pass through an encoder based on channel-Transformer fusion, and then pass through a bottleneck for expanding the receptive field to obtain fusion features; wherein the encoder includes a channel convolution module for local feature capture and a local attention mechanism module for global modeling; the preprocessed input features pass through the channel convolution module and the local attention mechanism module successively, and then pass through a bottleneck including a dilated convolution for feature fusion; S300, inputting the obtained fusion features into a decoder based on a skip connection mechanism to gradually restore the low-resolution features to the same size as the input image and generate a final segmentation result or classification result.

2. The method according to claim 1, characterized in that: The encoder includes a Transformer layer, and the CTE module in each Transformer layer includes the channel convolution module and the local attention mechanism module; wherein the step S200 includes: S210, dividing the original input features into non-overlapping small patches through a patch embedding module, and generating feature embedding for each patch to form an initial feature representation; S220, the obtained initial features are passed through the channel convolution module and the local attention mechanism module in the first Transformer layer to perform pixel-level addition; S230, passing through the second and third Transformer layers successively, so as to continue to extract deep features and perform downsampling through the subsequent channel convolution module, and to capture detail information through the subsequent partial attention mechanism module; S240, by introducing the bottleneck of the dilated convolution, the receptive field is expanded by setting different expansion rates to capture long-distance feature dependencies.

3. The method according to claim 1, characterized in that In step S200, the channel convolution module is configured as follows: The first convolutional layer used to transform the input feature map from in_channels to out_channels; A channel mixing module where both the input channel and the output channel are the width of the input feature map; Batch Normalization layers for normalizing feature maps to accelerate model convergence and stabilize training; Final convolutional layer with out_channels*2 input channels and out_channels output channels.

4. The method according to claim 1, characterized in that: In step S200, the local attention block of the local attention mechanism module includes the following processing operations: Input features are operated through layer normalization layers for accelerated training and stability; Through the local window self-attention operation, the feature map is divided into multiple non-overlapping small blocks (windows), and then the self-attention is calculated independently in each window. Through the calculation of local self-attention, in each window, each input position calculates the self-attention relationship with all other positions, and a new representation is obtained based on the weighted average of the obtained relationship; According to the multi-head mechanism of multi-head self-attention operation, each head learns different feature representations in different subspaces to perform self-attention calculations of the local attention block within each window; The features obtained from the self-attention operation are operated through a fully connected multi-layer perceptron to further enhance the nonlinear representation ability of the model, wherein the multi-layer perceptron operation passes through two fully connected layers, the first fully connected layer expands the feature dimension, and the second fully connected layer reduces the dimension back to the original size.

5. The method according to claim 4, characterized in that In the step S200, Each local window self-attention operation, multi-head self-attention operation, and multi-layer perceptron operation is connected with a residual connection to add the input and output to facilitate the flow of information; Each local window self-attention operation, multi-head self-attention operation, and multi-layer perceptron operation is followed by a layer normalization operation to speed up training and improve the stability of the model.

6. The method according to claim 5, characterized in that In step S300, the decoder includes three transposed convolution modules; The decoding process based on the jump connection mechanism includes: inputting the fused features output by the encoder into the decoder to restore the image resolution layer by layer; enlarging the length, width and height of the image layer by layer through the transposed convolution operation, while gradually halving the number of channels to restore it to a resolution close to that of the input image for layer-by-layer restoration; through the last layer of convolution of the decoder, the output image is restored to the resolution of the original input image to obtain the final restoration.

7. The method according to claim 1, characterized in that In the segmentation network model training, the forward propagation step includes: D10, initial convolution operation to complete preliminary feature extraction; D20, batch normalization to alleviate the problem of internal covariate shift; D30, channel mixing operation, after exchanging the channel dimension and width dimension of the normalized feature map, mix the channel information through a 3x3x3 convolution layer, and transpose the mixed feature map back to the original dimension; D40, pixel-level feature fusion, to add the channel-mixed feature map to the normalized feature map; D50, batch normalization again; D60, semantic mixed operations; D70, feature cascade and final fusion to concatenate the semantic features semantic_features and the initial normalized feature map bn_features along the channel dimension, reduce the number of channels back to out_channels through the conv_last convolution layer, and finally output the processed feature map.

8. The method according to claim 7, characterized in that The loss function of the segmentation network model training is a fusion of the Dice loss function and the Focal loss function; wherein, by mixing the Dice loss function L dice And the Focal loss function L focal The loss function L obtained total It is expressed as follows: THE total =L dice +λL focal Where P ti is the model’s predicted probability for category i; α i is the weight parameter of category i, which is set dynamically according to the frequency of the category, that is, γ is a factor used in Focal Loss to adjust easy-to-classify samples; λ is a trade-off parameter between Dice Loss and Focal Loss.

9. A computer-readable storage medium having program instructions stored thereon, wherein the program instructions are executed by a processor to implement the method according to any one of claims 1 to 8.

10. A small lesion segmentation and feature learning system based on deep learning, characterized in that: include: A computer device comprising a computer readable storage medium according to claim 9.

Images