Medical image enhancement method and device based on deep features, equipment and medium

Abstract

The invention discloses a medical image enhancement method and device based on deep features, equipment and a medium, and relates to the technical field of image processing. The method comprises steps S1 to S6. The method comprises the following steps: S1, acquiring a low-resolution medical image; s2, inputting the low-resolution medical image as an initial convolutional layer of a shallow feature extraction module, and generating initial shallow features; and S3, inputting the preliminary shallow layer features into a first Mamba module to obtain enhanced shallow layer features. And S4, inputting the enhanced shallow features into a deep feature extraction module, and generating deep feature representation in the deep feature extraction module through processing of a residual group repeating # imgabs0 # times. And S5, performing jump connection on the deep feature representation and the enhanced shallow feature output by the initial convolutional layer after passing through one convolutional layer. And S6, enabling the feature representation after jump connection to pass through a convolutional layer and a second Mama module, then inputting the feature representation into an image reconstruction module, carrying out up-sampling through pixel rearrangement operation, and carrying out processing by using the convolutional layer to generate a high-resolution medical image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for medical image enhancement based on deep features. Background Art

[0002] Medical imaging technologies, such as CT (computed tomography), MRI (magnetic resonance imaging) and ultrasound, are limited by factors such as equipment resolution, imaging speed and radiation dose, and the images obtained are often low in resolution and high in noise. This not only affects doctors' observation and analysis of lesions, but also limits the application of medical images in precision medicine.

[0003] Although traditional image enhancement methods, such as interpolation, filtering, and contrast enhancement, can improve image quality to a certain extent, they often have problems such as noise amplification and blurred details, and are unable to meet the high image quality requirements of clinical diagnosis. Summary of the invention

[0004] The present invention provides a medical image enhancement method, apparatus, device and medium based on deep features to improve at least one of the above technical problems.

[0005] In a first aspect, the present invention provides a medical image enhancement method based on deep features, which comprises steps S1 to S6.

[0006] S1. Obtain low-resolution medical images .

[0007] S2, the low-resolution medical image The input is used as the initial convolution layer of the shallow feature extraction module to generate preliminary shallow features. , the size is H×W×C, where H is the height, W is the width, and C is the number of channels.

[0008] S3, the preliminary shallow features Input to the first Mamba module for enhancement processing to obtain enhanced shallow features .

[0009] S4 enhances the shallow features Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation .

[0010] S5. The deep feature representation After passing through a convolution layer, the enhanced shallow features are output from the initial convolution layer. Make a jump connection.

[0011] S6, the feature representation after the jump connection, passes through a convolutional layer, and the second Mamba module, and then is input into the image reconstruction module, upsampled by pixel rearrangement operation, and further processed by the convolutional layer to generate a high-resolution medical image .

[0012] In an optional embodiment, the network structure of the first Mamba module and the second Mamba module includes a convolution layer, a block embedding layer, a state space model layer, a two-dimensional reconstruction layer, an addition layer, a normalization layer and a GELU activation function layer connected in sequence; wherein the addition layer is used to add the input of the Mamba module and the output of the two-dimensional reconstruction layer; In an optional implementation, step S3 specifically includes steps S301 to S306.

[0013] S301: Enhancing shallow features Through the normalization layer , get the first standardized feature .

[0014] S302: The first standardized feature After the adaptive spatial self-attention module, the self-attention mechanism is applied in the spatial dimension to perform local attention calculation by dividing the blocks to obtain the first feature map Then, an enhanced second feature map is obtained through the convolution layer. .

[0015] S303: The second feature map With the enhanced shallow features Perform residual connection to obtain the first connection feature .in, .

[0016] S304: The first connection feature Through the normalization layer , obtain the second standardized feature .

[0017] S305: The second standardized feature Input spatial gated feedforward network, output third feature map containing finer spatial information .

[0018] S306: The third feature map Connect with the first feature Perform residual connection to obtain the second connection feature .in, .

[0019] In an optional implementation, step S3 also includes steps S307 to S312.

[0020] S307: The second connection feature Through the normalization layer , and obtain the third standardized feature .

[0021] S308: The third standardized feature After the adaptive channel self-attention module, the self-attention operation is performed on the channel dimension to obtain the fourth feature map Then, an enhanced fifth feature map is obtained through the convolution layer. .

[0022] S309: the fifth feature map With the second connection feature Perform residual connection to obtain the third connection feature .in, .

[0023] S310: The third connection feature Through the normalization layer , get the fourth standardized feature .

[0024] S311, the fourth standardized feature Input to the spatial gated feedforward network, outputting the sixth feature map containing more refined spatial information .

[0025] S312: the sixth characteristic map With the third connection feature Perform residual connection to obtain the deep feature representation .in, .

[0026] In an optional implementation, step S302 specifically includes S30201 to S30212.

[0027] S30201. The first standardized feature The first query matrix is ​​generated through a linear projection operation , first bond matrix and the first value matrix . The linear projection operation here refers to A 1×1 convolution operation is applied, and each convolution kernel is of size C×C, so that the output dimension is the same as the input dimension, with a size of H×W×C.

[0028] S30202, will , , Divide into non-overlapping windows, each containing pixels, the features in the window are rearranged as The attention weights are then normalized using the SoftMax function to generate the spatial attention distribution.

[0029] S30203, the spatial attention distribution and the first value matrix Multiply them together to get the spatial self-attention output , and reshape it into a H×W×C shape.

[0030] S30204, outputting the spatial self-attention of shape H×W×C A 1×1 convolution operation is performed to reduce the number of channels from C to , and get the seventh feature map .in, is the first compression ratio or the first dimensionality reduction factor.

[0031] S30205, the seventh characteristic map Apply the GELU activation function to activate and get the first activation feature map , Represents the GELU activation function.

[0032] S30206: Activate the first feature map Perform a 1×1 convolution operation to increase the number of channels from Restore to H × W × 1. Then, use the Sigmoid function to limit the output to the range of [0, 1] to obtain the spatial attention map Space1-Map.

[0033] S30207, for each channel's first value matrix Perform deep convolution operation to generate the eighth feature map .

[0034] S30208, the eighth characteristic map Perform global average pooling to obtain the first global representation of each channel , The dimensions are 1×1×C.

[0035] S30209, Yes Perform a 1×1 convolution to reduce the number of channels from C to , and get the ninth feature map .in, is the second compression ratio or the second dimensionality reduction factor.

[0036] S30210, the ninth characteristic map Perform GELU activation to obtain the second activation feature map .

[0037] S30211, activating the second feature map Perform a 1×1 convolution to restore the number of channels to the original 1×1×C and obtain the tenth feature map . Then use the Sigmoid function to limit the output to the range of [0,1] to obtain the channel attention map channel1-Map.

[0038] S30212, the spatial attention map Space1-Map and the eighth feature map , and output the channel attention map channel1-Map and spatial self-attention Perform element-by-element multiplication to obtain features of shape H×W×C, and then add them element-by-element to obtain the first feature map .

[0039] In an optional implementation, step S308 specifically includes S30801 to S30814.

[0040] S30801, the third standardized feature The second query matrix is ​​generated through a linear projection operation , the second key matrix and the second value matrix . The linear projection operation here refers to A 1×1 convolution operation is applied, and each convolution kernel is of size C×C, so that the output dimension is the same as the input dimension, with a size of H×W×C.

[0041] S30802, split the channel into Each head performs self-attention calculation independently to obtain the dimension of each head .

[0042] S30803. According to the dimension , calculate the second query matrix and the second key matrix Dot product in the channel dimension . =softmax( ), Represents the second query matrix and the second key matrix The matrix multiplication of , the dimension of the result is C×C, Indicates transpose.

[0043] S30804, using the dot product Weighted second value matrix , and get the weighted output .

[0044] S30805: weighted output Perform multi-head aggregation and concatenate the outputs of all heads in the channel dimension. Then use a linear transformation to map it back to the original channel dimension C to obtain the channel self-attention output , For splicing, For the Output of the head, is a linear transformation matrix used to map the concatenated output to the C-dimensional space.

[0045] S30806: Output the self-attention of the channel Perform global average pooling to obtain the second global representation of each channel , the size becomes 1×1×C.

[0046] S30807, yes Perform a 1×1 convolution to reduce the number of channels from C to , and get the eleventh feature map .in, is the third compression ratio or the third dimensionality reduction factor.

[0047] S30808: the eleventh characteristic graph Apply the GELU activation function to activate and get the third activation feature map .

[0048] S30809: Activate the third feature map Perform a 1×1 convolution to restore the number of channels to the original 1×1×C and obtain the twelfth feature map Then, the Sigmoid function is used to limit the output to the range of [0,1] to obtain the channel attention map channel2-Map.

[0049] S30810, the second value matrix Perform channel-by-channel convolution to obtain the thirteenth feature map .

[0050] S30811, the thirteenth characteristic map Perform a 1×1 convolution operation to reduce the number of channels from C to , and get the fourteenth feature map .in, is the fourth compression ratio or the fourth dimensionality reduction factor.

[0051] S30812, the fourteenth characteristic graph Apply the GELU activation function to activate and get the fourth activation feature map , Represents the GELU activation function.

[0052] S30813, activating the fourth feature map Perform a 1×1 convolution operation to increase the number of channels from Restore to H × W × 1. Then, use the Sigmoid function to limit the output to the range of [0, 1] to obtain the spatial attention map Space2-Map.

[0053] S30814, the channel attention map channel2-Map and the thirteenth feature map , and output the spatial attention map Space2-Map and the channel self-attention Multiply each element by element to get features of shape H×W×C, then add each element by element to get the fourth feature map .

[0054] In an optional embodiment, during the training phase, the model outputs high-resolution medical images and real images The difference between them is calculated by the following loss function Calculate by minimizing To optimize network parameters.

[0055]

[0056] In the formula, represents a real high-resolution image, To reconstruct the image, is the number of pixels in the image.

[0057] In a second aspect, the present invention provides a medical image enhancement device based on deep features, which includes an image acquisition module, a first feature extraction module, a second feature extraction module, a third feature extraction module, a connection module and a reconstruction module.

[0058] Image acquisition module, used to acquire low-resolution medical images .

[0059] The first feature extraction module is used to extract the low-resolution medical image The input is used as the initial convolution layer of the shallow feature extraction module to generate preliminary shallow features. , the size is H×W×C, where H is the height, W is the width, and C is the number of channels.

[0060] The second feature extraction module is used to extract the preliminary shallow features Input to the first Mamba module for enhancement processing to obtain enhanced shallow features .

[0061] The third feature extraction module is used to enhance the shallow feature Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation .

[0062] Connection module for the deep feature representation After passing through a convolution layer, the enhanced shallow features are output from the initial convolution layer. Make a jump connection.

[0063] The reconstruction module is used for feature representation after skip connection, and then passes through a convolution layer and the second Mamba module, and then input into the image reconstruction module, which is upsampled by pixel rearrangement operation and further processed by convolution layer to generate high-resolution medical images. .

[0064] In a third aspect, the present invention provides a medical image enhancement device based on deep features, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a medical image enhancement method based on deep features as described in any paragraph of the first aspect.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a medical image enhancement method based on deep features as described in any paragraph of the first aspect.

[0066] By adopting the above technical solution, the present invention can achieve the following technical effects: The medical image enhancement method based on deep features of the present invention adopts a Transformer module after enhancing shallow features in a Mamba module, and fully extracts spatial features and channel information in medical images by alternately using spatial and channel self-attention, thereby effectively improving the detail performance of the image.

[0067] In addition, the adaptive interaction module realizes the fusion of global and local information, generates more delicate enhanced images, and improves the clarity and contrast of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the specific implementation methods of the present invention. It should be understood that the following drawings only show certain specific implementation methods of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 It is a flowchart of the medical image enhancement method based on deep features.

[0070] Figure 2 This is a network structure diagram of the medical image enhancement method based on deep features.

[0071] Figure 3 It is the network structure diagram of the deep feature extraction module.

[0072] Figure 4 It is the network structure diagram of the adaptive spatial self-attention module.

[0073] Figure 5 It is the network structure diagram of the adaptive channel self-attention module.

[0074] Figure 6 This is the image comparison before and after processing (the left side is before processing, and the right side is after processing).

[0075] Figure 7 This is the network structure diagram of the Mamba module. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Example 1, please refer to Figures 1 to 6 The first embodiment of the present invention provides a medical image enhancement method based on deep features, which can be performed by a medical image enhancement device based on deep features (hereinafter referred to as: medical image enhancement device). In particular, it is performed by one or more processors in the medical image enhancement device to implement steps S1 to S6.

[0078] S1. Obtain low-resolution medical images .like Figure 6 as shown on the left.

[0079] It is understandable that the medical image enhancement device may be an electronic device with computing capabilities, such as a portable notebook computer, a desktop computer, a server, a smart phone or a tablet computer.

[0080] S2, the low-resolution medical image The input is used as the initial convolution layer of the shallow feature extraction module to generate preliminary shallow features. ; Wherein, the preliminary shallow features The size is H×W×C. H is the height, W is the width, and C is the number of channels.

[0081] Using low-resolution medical images as the initial convolutional layer of the shallow feature extraction module can effectively extract the basic feature information of the image. First, low-resolution medical images usually contain important anatomical structures and lesion areas, but due to resolution limitations, the details may be blurred. When input into the convolutional layer, the convolution kernel can scan the entire image and extract local features through the local receptive field.

[0082] For example, for a 256×256 low-resolution CT image, 64 3×3 convolution kernels can be used for feature extraction, with the stride set to 1 and the padding set to "same". After the convolution layer, a 256×256×64 feature map can be obtained, that is, 64 feature channels are extracted, each channel represents a local feature pattern. The size and number of convolution kernels, as well as the choice of stride and padding are adjusted by technicians in this field according to specific tasks and data sets.

[0083] In summary, inputting low-resolution medical images into the initial convolutional layer of the shallow feature extraction module can effectively extract local features of the image, laying the foundation for subsequent deep feature extraction and image reconstruction. Reasonable parameter settings and hierarchical feature extraction strategies can help fully explore the diagnostic information contained in low-resolution images and provide strong support for the final high-resolution image reconstruction.

[0084] S3, the preliminary shallow features Input to the first Mamba module for enhancement processing to obtain enhanced shallow features .

[0085] Specifically, the preliminary shallow features are input into the first Mamba module for enhancement processing to obtain enhanced shallow features. The core of the Mamba module lies in its unique network structure design, which can effectively capture the dynamic information in the features and enhance them.

[0086] like Figure 7 As shown in Figure 1, the network structure of the Mamba module includes a convolutional layer, a block embedding layer, a state space model layer, a two-dimensional reconstruction layer, an addition layer, a normalization layer, and a GELU activation function layer connected in sequence. The addition layer adds the input features of the Mamba module to the output features of the two-dimensional reconstruction layer, thereby retaining the original information of the input features and enhancing the expressiveness of the features.

[0087] This design can not only make full use of the information in the input features, but also capture the dynamic changes of the features through the state space model layer, thereby improving the detail performance and overall quality of the features, and providing a higher quality feature foundation for subsequent deep feature extraction and image reconstruction. Through the enhancement processing of the Mamba module, the noise and fuzzy information in the initial shallow features are reduced, and the detail information is enhanced, so as to better serve the subsequent medical image enhancement tasks.

[0088] S4, enhancing the shallow features Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation .

[0089] Shallow features usually refer to feature representations obtained after processing with a small number of convolutional layers or pooling layers. These features contain low-level semantic information of the input data, such as edges, textures, etc. In order to further extract higher-level semantic features, the shallow features need to be input into the deep feature extraction module for processing. The deep feature extraction module is usually composed of multiple residual groups, each of which contains multiple convolutional layers and normalization layers. By repeatedly stacking residual groups, more abstract and advanced feature representations can be gradually extracted.

[0090] In the present invention, the deep feature extraction module is repeatedly stacked times, each containing a stack The residual group of times, and is a hyperparameter that can be adjusted based on the complexity of the task and the size of the dataset.

[0091] S5. The deep feature representation After passing through a convolution layer, the enhanced shallow features are output from the initial convolution layer. Make a jump connection.

[0092] Skip connections are used to fuse enhanced shallow features with deep features. Specifically, the deep feature representation is skipped with the enhanced shallow features output by the first mamba module after passing through a convolutional layer. This is done to alleviate the gradient vanishing problem during deep neural network training, and also to enable the model to better utilize feature information at different levels.

[0093] For example, suppose we have a deep neural network with 5 convolutional layers, where the first layer outputs shallow features and the fifth layer outputs deep features. If the output of the fifth layer is directly used as the final feature representation, the model may not be able to fully utilize the low-level features extracted by the first layer. With skip connections, we can concatenate or add the output of the first layer with the output of the fifth layer, so that the final feature representation contains both low-level and high-level features.

[0094] This fusion method not only helps to improve the representation ability of the model, but also accelerates the convergence speed of the model. In the task of medical image super-resolution reconstruction, jump connection technology also plays an important role. By fusing deep features with enhanced shallow features, the model can better capture the multi-scale information in the image, thereby generating clearer and more detailed high-resolution images.

[0095] S6, the feature representation after the jump connection, passes through a convolutional layer, and the second Mamba module, and then is input into the image reconstruction module, upsampled by pixel rearrangement operation, and further processed by the convolutional layer to generate a high-resolution medical image Among them, the network structure of the second Mamba module is the same as that of the first Mamba module.

[0096] Specifically, the feature representation after the jump connection is first processed by a convolution layer and then input into the image reconstruction module. In the reconstruction module, we usually use pixel shuffle operation to upsample and convert the low-resolution feature map into a high-resolution feature map. Then, a series of convolution layers are used to further refine and enhance the upsampled feature map, and finally generate a high-quality medical image.

[0097] As a practical example, suppose we want to reconstruct a 256x256 low-resolution medical image into a 1024x1024 high-resolution image. First, we use an initial convolutional layer and the first mamba module to convert the input image into an enhanced shallow feature map with 64 channels. Then, the enhanced shallow feature map is input into a deep feature extraction module stacked by multiple residual groups to extract a deep feature representation of 512 channels. Next, we use a 1x1 convolutional layer to reduce the number of channels of the deep feature representation to 64, and perform a jump connection with the enhanced shallow feature map. The feature representation after the jump connection is then passed through a 3x3 convolutional layer to obtain the final 64-channel feature map. Finally, the feature map is input into the image reconstruction module, which is upsampled to a 1024x1024 high-resolution image through two pixel rearrangement operations and post-processed using two 3x3 convolutional layers to obtain the final reconstruction result.

[0098] From the above examples, we can see that the jump connection technology is widely used in medical image super-resolution reconstruction. It can not only effectively fuse feature information at different levels, but also improve the representation ability and convergence speed of the model. At the same time, the clever combination of pixel rearrangement operation and convolution layer also enables us to efficiently complete the conversion process from low resolution to high resolution. The combination of these technologies ultimately enables us to reconstruct clear, detailed, high-resolution images from poor quality medical images, providing more accurate and reliable imaging data for medical diagnosis and treatment.

[0099] Based on the above embodiments, in an optional embodiment of the present invention, as Figure 3 As shown, step S3 specifically includes steps S301 to S306.

[0100] S301: Enhancing shallow features Through the normalization layer , get the first standardized feature .

[0101] S302: The first standardized feature After the adaptive spatial self-attention module, the self-attention mechanism is applied in the spatial dimension to perform local attention calculation by dividing the blocks to obtain the first feature map Then, an enhanced second feature map is obtained through the convolution layer. .

[0102] S303: The second feature map With the enhanced shallow features Perform residual connection to obtain the first connection feature .in, .

[0103] S304: The first connection feature Through the normalization layer , obtain the second standardized feature .

[0104] S305: The second standardized feature Input spatial gated feedforward network, output third feature map containing finer spatial information .

[0105] S306: The third feature map Connect with the first feature Perform residual connection to obtain the second connection feature .in, .

[0106] Specifically, the enhanced shallow features are first normalized through the normalization layer to obtain the first normalized features. The normalization layer can accelerate the convergence speed of the model and improve the training stability. Common normalization methods include batch normalization and instance normalization.

[0107] Next, the first normalized feature is processed by the adaptive spatial self-attention module. The self-attention mechanism can model the long-range dependencies between features and capture global contextual information. Applying self-attention in the spatial dimension can highlight salient areas and suppress background noise. In specific implementation, the feature map can be divided into multiple local areas, the attention weight is calculated in each area, and then the enhanced feature map is obtained by weighted aggregation. For example, the feature map can be divided into 8x8 blocks, the attention weight is calculated in each block, and then the weighted feature map is fused through the convolution layer to obtain the second feature map.

[0108] In order to avoid the network degradation problem, the second feature map is residually connected with the enhanced shallow feature to obtain the first connection feature. The residual connection allows the gradient to be directly transferred to the shallow layer, alleviating the gradient vanishing problem. At the same time, the residual connection can also make it easier for the network to learn the identity mapping and accelerate the convergence speed. For example, the second feature map can be directly added to the enhanced shallow feature to obtain the first connection feature.

[0109] Then, the first connection feature is normalized again through the normalization layer to obtain the second normalized feature. This step can further stabilize the feature distribution and improve the generalization ability of the model.

[0110] Next, the second standardized feature is input into the spatial gated feedforward network to obtain a third feature map containing finer spatial information. The spatial gated feedforward network can model nonlinear interactions between features and improve feature representation capabilities. In specific implementation, two convolutional layers and a gating unit can be used to form a feedforward network. The first convolutional layer is used for feature conversion, the second convolutional layer is used for feature fusion, and the gating unit is used to control the flow of information. For example, a 3x3 convolution kernel can be used to convolve the second standardized feature to obtain an intermediate feature map. Then, a 1x1 convolution kernel is used to reduce the dimension of the intermediate feature map, and the gating coefficient is obtained through the Sigmoid activation function. Finally, the intermediate feature map is multiplied by the gating coefficient and added to the second standardized feature to obtain the third feature map.

[0111] Finally, the third feature map is residually connected with the first connection feature to obtain the second connection feature. This step can further fuse feature information of different scales and improve the robustness of feature representation.

[0112] By repeating the above steps N times, the final deep feature representation can be obtained. This deep feature extraction module can effectively extract high-level semantic information of the input data and achieve excellent performance in tasks such as image classification and object detection. At the same time, by introducing the self-attention mechanism and gating unit, the network's expression and generalization capabilities can be enhanced, making the model more adaptable to different data distributions and task requirements.

[0113] Based on the above embodiments, in an optional embodiment of the present invention, as Figure 3 As shown, step S3 also includes steps S307 to S312.

[0114] S307: The second connection feature Through the normalization layer , and obtain the third standardized feature .

[0115] S308: The third standardized feature After the adaptive channel self-attention module, the self-attention operation is performed on the channel dimension to obtain the fourth feature map Then, an enhanced fifth feature map is obtained through the convolution layer. .

[0116] S309: the fifth feature map With the second connection feature Perform residual connection to obtain the third connection feature .in, .

[0117] S310: The third connection feature Through the normalization layer , get the fourth standardized feature .

[0118] S311, the fourth standardized feature Input to the spatial gated feedforward network, outputting the sixth feature map containing more refined spatial information .

[0119] S312: the sixth characteristic map With the third connection feature Perform residual connection to obtain the deep feature representation .in, .

[0120] In the deep feature extraction module, more abstract and high-level feature representations can be gradually extracted through multiple residual group processing. The residual group fuses the enhanced shallow features with the transformed features through jump connections, so that the network can better learn the essential characteristics of the input data. The normalization layer can standardize the features to make their distribution more uniform, which is conducive to subsequent feature extraction and fusion. Through normalization, the convergence speed of the network can be accelerated and the generalization ability of the model can be improved.

[0121] The adaptive channel self-attention module can automatically learn the importance weights between different channels by performing self-attention operations on the channel dimension, thereby adaptively weighted fusion of features. Through the stacking and progression of multiple residual groups, higher-level and abstract feature representations can be gradually extracted to capture the essential characteristics and inherent laws of the input data. Through the layer-by-layer abstraction and fusion of features, a comprehensive and effective feature representation can be obtained, providing more powerful support for subsequent tasks.

[0122] Based on the above embodiments, in an optional embodiment of the present invention, as Figure 4 As shown, step S302 specifically includes S30201 to S30212.

[0123] S30201. The first standardized feature The first query matrix is ​​generated through a linear projection operation , first bond matrix and the first value matrix . That is: query (Query), key (Key) and value (Value).

[0124] The linear projection operation here refers to A 1×1 convolution operation is applied, and each convolution kernel is of size C×C, so that the output dimension is the same as the input dimension, with a size of H×W×C. H is the height, W is the width, and C is the number of channels.

[0125] To generate the first query matrix, the first key matrix, and the first value matrix, we apply a convolution operation to the feature map. After the convolution operation, we get three new feature maps with the same size. These are the first query matrix, the first key matrix, and the first value matrix. The purpose of this step is to map the original features into three different feature spaces in preparation for subsequent attention calculations. In this way, the model can learn the relationship between features from different perspectives.

[0126] S30202, will , , Divide into non-overlapping windows, each containing pixels, the features in the window are rearranged as The attention weights are then normalized using the SoftMax function to generate the spatial attention distribution.

[0127] The first query matrix, the first key matrix, and the first value matrix are divided into non-overlapping windows. For each window, we rearrange the features. For example, for a window in the first query matrix, we calculate its dot product with the corresponding window in the first key matrix to obtain an attention weight matrix. This matrix represents the correlation between each pixel in the window and other pixels. Then, the SoftMax function is applied to this attention weight matrix, which is normalized to obtain a probability distribution representing the attention weight of each pixel to other pixels in the window. This process enables the model to focus on important local information in the window.

[0128] S30203, the spatial attention distribution and the first value matrix Multiply them together to get the spatial self-attention output , and reshape it to the size of H×W×C.

[0129] Perform matrix multiplication of the spatial attention distribution obtained in the previous step with the corresponding window in the first value matrix. Get an output matrix. This output matrix is ​​the weighted sum of the features in the window, and the weight is determined by the attention distribution. Then, reshape this output matrix and concatenate the outputs of all windows to get a spatial self-attention output. The purpose of this step is to integrate local attention information into the global feature map.

[0130] S30204, outputting the spatial self-attention of shape H×W×C A 1×1 convolution operation is performed to reduce the number of channels from C to , and get the seventh feature map .in, is the first compression ratio or the first dimensionality reduction factor.

[0131] A 1×1 convolution operation is performed on the spatial self-attention output to compress the number of channels. After the convolution operation, we get the seventh feature map. The purpose of this step is to reduce the dimension of the feature map and reduce the amount of calculation while retaining important feature information.

[0132] S30205, the seventh characteristic map Apply the GELU activation function to activate and get the first activation feature map . , Represents the GELU activation function.

[0133] Apply the GELU activation function to the seventh feature map. The GELU activation function is a nonlinear activation function that can enhance the expressiveness of the model. After activation, the first activated feature map is obtained.

[0134] S30206: Activate the first feature map Perform a 1×1 convolution operation to increase the number of channels from Restore to H × W × 1. Then, use the Sigmoid function to limit the output to the range of [0, 1] to obtain the spatial attention map Space1-Map.

[0135] This spatial attention map represents the importance of each spatial location.

[0136] S30207, for each channel's first value matrix Perform deep convolution operation to generate the eighth feature map The purpose of this step is to extract the local feature information of each channel.

[0137] S30208, the eighth characteristic map Perform global average pooling to obtain the first global representation of each channel , The size of is 1×1×C. The purpose of this step is to extract the global feature information of each channel.

[0138] S30209, Yes Perform a 1×1 convolution to reduce the number of channels from C to , and get the ninth feature map .in, is the second compression ratio or the second dimensionality reduction factor. The purpose of this step is to reduce the dimension of the feature map, reduce the amount of calculation, and retain important feature information.

[0139] S30210, the ninth characteristic map Perform GELU activation to obtain the second activation feature map The purpose of this step is to enhance the expressiveness of the model.

[0140] S30211, activating the second feature map Perform a 1×1 convolution to restore the number of channels to the original 1×1×C and obtain the tenth feature map Then the Sigmoid function is used to limit the output to the range of [0,1] to obtain the channel attention map channel1-Map. This channel attention map indicates the importance of each channel.

[0141] S30212, the spatial attention map Space1-Map and the eighth feature map , and output the channel attention map channel1-Map and spatial self-attention Perform element-by-element multiplication to obtain features of shape H×W×C, and then add them element-by-element to obtain the first feature map The purpose of this step is to fuse the spatial attention and channel attention information into the original feature map to enhance the feature representation ability of the model.

[0142] Based on the above embodiments, in an optional embodiment of the present invention, as Figure 5 As shown, step S308 specifically includes S30801 to S30814. Performing self-attention operation on the channel dimension can effectively capture the dependency and interaction information between channels.

[0143] S30801, the third standardized feature The second query matrix is ​​generated through a linear projection operation , the second key matrix and the second value matrix .

[0144] The linear projection operation here refers to A 1×1 convolution operation is applied, and each convolution kernel is of size C×C, so that the output dimension is the same as the input dimension, with a size of H×W×C. H is the height, W is the width, and C is the number of channels.

[0145] S30802, split the channel into Each head performs self-attention calculation independently to obtain the dimension of each head .

[0146] S30803. According to the dimension , calculate the second query matrix and the second key matrix Dot product in the channel dimension (ie: channel attention weight matrix).

[0147] =softmax( ).

[0148] In the formula, Represents the second query matrix and the second key matrix Matrix multiplication of (i.e., attention weights), the resulting dimension is C×C, Indicates transpose.

[0149] S30804, using the dot product Weighted second value matrix , and get the weighted output . .

[0150] S30805: weighted output Perform multi-head aggregation and concatenate the outputs of all heads in the channel dimension. Then use a linear transformation to map it back to the original channel dimension C to obtain the channel self-attention output .

[0151] .

[0152] In the formula, For splicing, For the Output of the head, is a linear transformation matrix used to map the concatenated output to the C-dimensional space.

[0153] S30806: Output the self-attention of the channel Perform global average pooling to obtain the second global representation of each channel , the size becomes 1×1×C. Global average pooling can effectively capture the global information at the channel level.

[0154] S30807, yes Perform a 1×1 convolution to reduce the number of channels from C to , and get the eleventh feature map .in, is the third compression ratio or the third dimensionality reduction factor. Dimensionality reduction can reduce the amount of calculation and parameters, while extracting a more compact feature representation.

[0155] S30808: the eleventh characteristic graph Apply the GELU activation function to activate and get the third activation feature map The GELU activation function is applied to the feature map after dimensionality reduction for nonlinear transformation, which introduces nonlinear characteristics and improves the expressiveness of the model. The GELU activation function can smoothly approximate the ReLU function and has better gradient fluidity.

[0156] S30809: Activate the third feature map Perform a 1×1 convolution to restore the number of channels to the original 1×1×C and obtain the twelfth feature map Then, the Sigmoid function is used to limit the output to the range of [0,1] to obtain the channel attention map channel2-Map. The channel attention map can adaptively adjust the importance of each channel, emphasize important channels, and suppress unimportant channels.

[0157] S30810, the second value matrix Perform channel-by-channel convolution to obtain the thirteenth feature map Channel-by-channel convolution can process each channel independently and extract channel-specific information.

[0158] S30811, the thirteenth characteristic map Perform a 1×1 convolution operation to reduce the number of channels from C to , and get the fourteenth feature map .in, is the fourth compression ratio or the fourth dimensionality reduction factor. This step can further compress the dimension of the feature and reduce the computational overhead.

[0159] S30812, the fourteenth characteristic graph Apply the GELU activation function to activate and get the fourth activation feature map . , Represents the GELU activation function. The GELU activation function is applied to the feature map after dimensionality reduction to perform nonlinear transformation and enhance the expressiveness of the features.

[0160] S30813, activating the fourth feature map Perform a 1×1 convolution operation to increase the number of channels from Restore to H×W×1. Then, use the Sigmoid function to limit the output to the range of [0,1] to obtain the spatial attention map Space2-Map. The spatial attention map can adaptively adjust the importance of each spatial position, focus on important areas, and ignore unimportant areas.

[0161] S30814, the channel attention map channel2-Map and the thirteenth feature map , and output the spatial attention map Space2-Map and the channel self-attention Multiply each element by element to get features of shape H×W×C, then add each element by element to get the fourth feature map .

[0162] The combination of channel attention and spatial attention can consider the importance of both channel and spatial dimensions at the same time, achieving more refined and effective feature enhancement. Through the adaptive channel self-attention module, the importance of channel and spatial dimensions can be adaptively adjusted to capture the dependencies and interaction information between channels while focusing on important spatial regions.

[0163] This adaptive attention mechanism can effectively enhance feature representation and improve the expressiveness and generalization ability of the model. In practical applications, hyperparameters such as compression ratio and number of heads can be adjusted according to specific tasks and data characteristics to obtain the best performance.

[0164] Figure 6 Figure 2 shows images before and after processing using a deep feature-based medical image enhancement method. Figure 6 It can be seen that this method can enhance medical images well.

[0165] Based on the above embodiment, in an optional embodiment of the present invention, during the training phase, the high-resolution medical images output by the model and real images The difference between them is calculated by the following loss function Calculate by minimizing To optimize network parameters.

[0166]

[0167] In the formula, represents a real high-resolution image, To reconstruct the image, is the number of pixels in the image.

[0168] Embodiment 2: The present invention provides a medical image enhancement device based on deep features, which includes an image acquisition module, a first feature extraction module, a second feature extraction module, a third feature extraction module, a connection module and a reconstruction module.

[0169] Image acquisition module, used to acquire low-resolution medical images .

[0170] The first feature extraction module is used to extract the low-resolution medical image The input is used as the initial convolution layer of the shallow feature extraction module to generate preliminary shallow features. , the size is H×W×C, where H is the height, W is the width, and C is the number of channels.

[0171] The second feature extraction module is used to extract the preliminary shallow features Input to the first Mamba module for enhancement processing to obtain enhanced shallow features .

[0172] The third feature extraction module is used to enhance the shallow feature Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation .

[0173] Connection module for the deep feature representation After passing through a convolution layer, the enhanced shallow features are output from the initial convolution layer. Make a jump connection.

[0174] The reconstruction module is used for feature representation after skip connection, and then passes through a convolution layer and the second Mamba module, and then input into the image reconstruction module, which is upsampled by pixel rearrangement operation and further processed by convolution layer to generate high-resolution medical images. .

[0175] Embodiment 3: The present invention provides a medical image enhancement device based on deep features, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a medical image enhancement method based on deep features as described in any paragraph of Embodiment 1.

[0176] Embodiment 4: The present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a medical image enhancement method based on deep features as described in any paragraph of Embodiment 1.

[0177] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0179] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0180] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0181] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0182] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0183] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A medical image enhancement method based on deep features, characterized in that: Include: Acquiring low-resolution medical images ; The low-resolution medical image The input is used as the initial convolution layer of the shallow feature extraction module to generate preliminary shallow features. , the size is H×W×C; where H is the height, W is the width, and C is the number of channels; The preliminary shallow features Input to the first Mamba module for enhancement processing to obtain enhanced shallow features ; The enhanced shallow features Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation ; The deep feature representation After passing through a convolution layer, the enhanced shallow features are output from the initial convolution layer. Make a jump connection; The feature representation after the jump connection passes through a convolutional layer and the second Mamba module, and then is input into the image reconstruction module, upsampled by pixel rearrangement operation, and further processed by the convolutional layer to generate a high-resolution medical image. ; The network structure of the first Mamba module and the second Mamba module includes a convolution layer, a block embedding layer, a state space model layer, a two-dimensional reconstruction layer, an addition layer, a normalization layer and a GELU activation function layer connected in sequence; wherein the addition layer is used to add the input of the Mamba module and the output of the two-dimensional reconstruction layer.

2. The medical image enhancement method based on deep features according to claim 1, characterized in that: The enhanced shallow features Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation , specifically including: The enhanced shallow features Through the normalization layer , get the first standardized feature ; The first standardized feature After the adaptive spatial self-attention module, the self-attention mechanism is applied in the spatial dimension to perform local attention calculation by dividing the blocks to obtain the first feature map ; Then, an enhanced second feature map is obtained through the convolution layer ; The second feature map With the enhanced shallow features Perform residual connection to obtain the first connection feature ;in, ; The first connection feature Through the normalization layer , obtain the second standardized feature ; The second normalized feature Input spatial gated feedforward network, output third feature map containing finer spatial information ; The third feature map Connect with the first feature Perform residual connection to obtain the second connection feature ;in, .

3. The method for medical image enhancement based on deep features according to claim 2, characterized in that: The enhanced shallow features Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation , also includes: The second connection feature Through the normalization layer , and obtain the third standardized feature ; The third standardized feature After the adaptive channel self-attention module, the self-attention operation is performed on the channel dimension to obtain the fourth feature map ; Then, an enhanced fifth feature map is obtained through the convolution layer ; The fifth feature map With the second connection feature Perform residual connection to obtain the third connection feature ;in, ; The third connection feature Through the normalization layer , get the fourth standardized feature ; The fourth normalized feature Input to the spatial gated feedforward network, outputting the sixth feature map containing more refined spatial information ; The sixth characteristic map With the third connection feature Perform residual connection to obtain the deep feature representation ;in, .

4. The medical image enhancement method based on deep features according to claim 2, characterized in that: The first standardized feature After the adaptive spatial self-attention module, the self-attention mechanism is applied in the spatial dimension to perform local attention calculation by dividing the blocks to obtain the first feature map , specifically including: The first standardized feature The first query matrix is ​​generated through a linear projection operation , first bond matrix and the first value matrix ; Wherein, the linear projection operation here refers to Apply a 1×1 convolution operation, and each convolution kernel is of size C×C, so that the output dimension is the same as the input dimension, with a size of H×W×C; Will , , Divide into non-overlapping windows, each containing pixels, the features in the window are rearranged as format, and calculate the attention weight; then normalize the attention weight through the SoftMax function to generate the spatial attention distribution; The spatial attention distribution and the first value matrix Multiply them together to get the spatial self-attention output , and reshape it to the size of H×W×C; The spatial self-attention output of shape H×W×C A 1×1 convolution operation is performed to reduce the number of channels from C to , and get the seventh feature map ;in, is the first compression ratio or the first dimensionality reduction factor; For the seventh characteristic diagram Apply the GELU activation function to activate and get the first activation feature map ; , represents the GELU activation function; For the first activation feature map Perform a 1×1 convolution operation to increase the number of channels from Restore to H×W×1; then, use the Sigmoid function to limit the output to the range of [0,1] to obtain the spatial attention map Space1-Map; The first value matrix for each channel Perform deep convolution operation to generate the eighth feature map ; For the eighth characteristic diagram Perform global average pooling to obtain the first global representation of each channel , The size is 1×1×C; right Perform a 1×1 convolution to reduce the number of channels from C to , and get the ninth feature map ;in, is the second compression ratio or the second dimensionality reduction factor; For the ninth characteristic diagram Perform GELU activation to obtain the second activation feature map ; For the second activation feature map Perform a 1×1 convolution to restore the number of channels to the original 1×1×C and obtain the tenth feature map ; Then use the Sigmoid function to limit the output to the range of [0,1] to obtain the channel attention map channel1-Map; The spatial attention map Space1-Map and the eighth feature map , and output the channel attention map channel1-Map and spatial self-attention Perform element-by-element multiplication to obtain features of shape H×W×C, and then add them element-by-element to obtain the first feature map .

5. The method for medical image enhancement based on deep features according to claim 3, characterized in that: The third standardized feature After the adaptive channel self-attention module, the self-attention operation is performed on the channel dimension to obtain the fourth feature map , specifically including: The third standardized feature The second query matrix is ​​generated through a linear projection operation , the second key matrix and the second value matrix ; Wherein, the linear projection operation here refers to Apply a 1×1 convolution operation, and each convolution kernel is of size C×C, so that the output dimension is the same as the input dimension, with a size of H×W×C; Split the channel into Each head performs self-attention calculation independently to obtain the dimension of each head ; ; According to the dimensions , calculate the second query matrix and the second key matrix Dot product in the channel dimension ; =softmax( ), Represents the second query matrix and the second key matrix The matrix multiplication of , the dimension of the result is C×C, represents transpose; Using the dot product Weighted second value matrix , and get the weighted output ; ; The weighted output Perform multi-head aggregation and concatenate the outputs of all heads in the channel dimension; then map them back to the original channel dimension C through a linear transformation to obtain the channel self-attention output ; , For splicing, For the Output of the head, is a linear transformation matrix used to map the concatenated output to a C-dimensional space; Self-attention output for the channel Perform global average pooling to obtain the second global representation of each channel , the size becomes 1×1×C; right Perform a 1×1 convolution to reduce the number of channels from C to , and get the eleventh feature map ;in, is the third compression ratio or the third dimensionality reduction factor; For the eleventh characteristic graph Apply the GELU activation function to activate and get the third activation feature map ; For the third activation feature map Perform a 1×1 convolution to restore the number of channels to the original 1×1×C and obtain the twelfth feature map ; Then, use the Sigmoid function to limit the output to the range of [0,1] to obtain the channel attention map channel2-Map; For the second value matrix Perform channel-by-channel convolution to obtain the thirteenth feature map ; For the thirteenth characteristic diagram Perform a 1×1 convolution operation to reduce the number of channels from C to , and get the fourteenth feature map ;in, is the fourth compression ratio or the fourth dimensionality reduction factor; For the fourteenth characteristic diagram Apply the GELU activation function to activate and get the fourth activation feature map ; , represents the GELU activation function; For the fourth activation feature map Perform a 1×1 convolution operation to increase the number of channels from Restore to H×W×1; then, use the Sigmoid function to limit the output to the range of [0,1] to obtain the spatial attention map Space2-Map; The channel attention map channel2-Map and the thirteenth feature map , and output the spatial attention map Space2-Map and the channel self-attention Multiply each element by element to get features of shape H×W×C, then add each element by element to get the fourth feature map .

6. The medical image enhancement method based on deep features according to claim 1, characterized in that: During the training phase, the model outputs high-resolution medical images. and real images The difference between them is calculated by the following loss function Calculate by minimizing To optimize network parameters; In the formula, represents a real high-resolution image, To reconstruct the image, is the number of pixels in the image.

7. A medical image enhancement device based on deep features, characterized in that: Include: Image acquisition module, used to acquire low-resolution medical images ; The first feature extraction module is used to extract the low-resolution medical image The input is used as the initial convolution layer of the shallow feature extraction module to generate preliminary shallow features. , the size is H×W×C; where H is the height, W is the width, and C is the number of channels; The second feature extraction module is used to extract the preliminary shallow features Input to the first Mamba module for enhancement processing to obtain enhanced shallow features ; The third feature extraction module is used to enhance the shallow feature Input to the stack The deep feature extraction module is repeated in the deep feature extraction module. The residual group is processed to generate deep feature representation ; Connection module for the deep feature representation After passing through a convolution layer, the enhanced shallow features are output from the initial convolution layer. Make a jump connection; The reconstruction module is used for feature representation after skip connection, and then passes through a convolution layer and the second Mamba module, and then input into the image reconstruction module, which is upsampled by pixel rearrangement operation and further processed by convolution layer to generate high-resolution medical images. ; The network structure of the first Mamba module and the second Mamba module includes a convolution layer, a block embedding layer, a state space model layer, a two-dimensional reconstruction layer, an addition layer, a normalization layer and a GELU activation function layer connected in sequence; wherein the addition layer is used to add the input of the Mamba module and the output of the two-dimensional reconstruction layer.

8. A medical image enhancement device based on deep features, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a medical image enhancement method based on deep features as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a medical image enhancement method based on deep features as described in any one of claims 1 to 6.

Images