Image processing method and device based on deep dynamic self-adjustment
By using a multi-level image segmentation network with lightweight deformable residual module and multi-scale enhanced self-attention mechanism module in medical image segmentation, combined with the semantic mitigation module, the lack of feature channel association neglect and semantic difference mitigation in the prior art is solved. The problem of improving the accuracy and performance of medical image segmentation.
Patent Information
- Application Number
- CN202411345394.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-26
AI Technical Summary
The existing medical image segmentation methods have shortcomings in extracting context information and mitigating semantic differences, resulting in poor segmentation performance, neglected association between feature channels and feature space, convolution operation deviation and codec feature set incompatibility.
Using a multi-level image segmentation network based on lightweight deformable residual module and multi-scale enhanced self-attention mechanism module, the edge profile is dynamically fitted, the channel space features are extracted, and the multi-level image feature is used to extract and upsample. Semantic mitigation modules with different depths perform semantic connections, and adaptively adjust the semantic differences on jump connections.
It improves the accuracy and segmentation performance of medical image segmentation, avoids calculation consumption and feature information redundancy, and achieves more accurate lesion segmentation results.
Smart Images

Figure CN118864503B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device based on depth dynamic self-adjustment. Background Art
[0002] With the continuous advancement of medical technology and the increasing public health awareness, people have increasingly stringent demands for accurate analysis of medical images, especially for the processing of complex medical images such as CT (computed tomography) images and magnetic resonance imaging images. In order to obtain more accurate segmentation results, the combination of medical image segmentation technology and deep learning has become an inevitable trend in the development of the industry. This fusion not only improves the accuracy of segmentation, but also provides a solid and reliable foundation for subsequent disease assessment and medical advice. Although algorithms designed based on convolutional neural networks (CNN) and vision transformers (ViT) have dominated the field of medical image segmentation and have shown significant performance and great potential, these methods are still insufficient in effectively extracting contextual information and alleviating semantic differences.
[0003] Therefore, traditional segmentation methods ignore the correlation between feature channels and feature spaces, which limits the model's ability to express complex features. Secondly, the inherent bias in convolution operations cannot fully and effectively extract contextual information. Finally, due to the incompatible feature sets between codecs, some jump connections are affected by the semantic differences between codecs, which in turn has a negative effect on segmentation performance, making it difficult to achieve accurate segmentation tasks. Summary of the invention
[0004] The purpose of the present invention is to provide an image processing method and device based on depth dynamic self-adjustment, which solves the technical problems existing in the prior art.
[0005] The present invention is achieved through the following technical solutions:
[0006] In one aspect, the present invention provides an image processing method based on deep dynamic self-adjustment, comprising:
[0007] Acquire a target processed medical image; wherein the target processed medical image is used to represent an image of a lesion site containing a tumor;
[0008] A multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module is used to extract multi-level image features of the target medical image to dynamically fit edge contours and extract channel space features, thereby obtaining down-sampled processed images of multiple sizes corresponding to the number of layers of the multi-level image segmentation network;
[0009] The downsampled image of the last size is upsampled in turn through a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module to obtain upsampled images of multiple sizes; wherein the multi-level upsampling network has the same number of layers as the multi-level image segmentation network;
[0010] In the multiple upsampling processes, the downsampled image and the upsampled image are semantically connected through multiple semantic relief modules with different depths, and the semantic difference on the skip connection of different layers is adaptively adjusted to use the upsampled image of the previous size to generate the input image of the upsampling process of the next size;
[0011] Among them, the down-sampled image of the u-th size corresponds to the up-sampled image of the v-th size, and when v =1,2,…,J-1, u =I-1,I-2,…,1; I represents the total number of layers of the multi-level image segmentation network, J represents the total number of layers of the multi-level upsampling network with an upper limit, I=J;
[0012] Convolution is used to replace the multi-scale enhanced self-attention mechanism module in the last size upsampling process to convert the upsampled image output by the penultimate size upsampling process into an output processed image to obtain the target image segmentation processing result.
[0013] In a possible implementation, the lightweight deformable residual module includes: a second image input layer, a sixth convolutional layer, a first maximum pooling layer, a seventh convolutional layer, a deformable convolutional layer, a fully connected layer, a bias layer, an eighth convolutional layer and a second image output layer connected in sequence; the output of the sixth convolutional layer is residually connected to the output of the eighth convolutional layer, and the output of the seventh convolutional layer is residually connected to the output of the bias layer; wherein the second image input layer is used to receive the second processed image of the input lightweight deformable residual module, and the second image output layer is used to output the second processed image processed by the lightweight deformable residual module.
[0014] In a possible implementation, the multi-scale enhanced self-attention mechanism module includes:
[0015] A first image input layer, a channel attention layer, a segmentation layer, a multi-scale global feature extraction submodule, a third convolution layer, a first addition layer and a first image output layer connected in sequence, and the first image input layer is also connected to the multi-scale global feature extraction submodule and the first addition layer respectively; wherein the first image input layer is used to receive a first processed image input to the multi-scale enhanced self-attention mechanism module, the segmentation layer is used to divide the first processed image output by the channel attention layer into four groups along the channel dimension, the first addition layer is used to add the feature map received by the first image input layer to the first processed image output by the third convolution layer, and the first image output layer is used to output the first processed image processed by the multi-scale enhanced self-attention mechanism module;
[0016] The multi-scale global feature extraction submodule includes a first depth convolution layer, a second depth convolution layer, a third depth convolution layer, a fourth depth convolution layer, a first dot product layer, a second dot product layer, a third dot product layer, a fourth dot product layer, a fifth dot product layer, a sixth dot product layer, a first average pooling layer, a first convolution layer, a bilinear interpolation layer, a first splicing layer and a second convolution layer;
[0017] The four outputs of the segmentation layer are respectively connected to the first depth convolution layer, the second depth convolution layer, the third depth convolution layer and the fourth depth convolution layer; wherein the convolution kernels of the first depth convolution layer, the second depth convolution layer, the third depth convolution layer and the fourth depth convolution layer are different;
[0018] The output of the first depth convolution layer and the output of the second depth convolution layer are both connected to the first dot multiplication layer, the output of the first depth convolution layer and the output of the third depth convolution layer are both connected to the second dot multiplication layer, the output of the first depth convolution layer and the output of the fourth depth convolution layer are both connected to the third dot multiplication layer, the output of the second depth convolution layer and the output of the third depth convolution layer are both connected to the fourth dot multiplication layer, the output of the second depth convolution layer and the output of the fourth depth convolution layer are both connected to the fifth multiplication layer, and the output of the third depth convolution layer and the output of the fourth depth convolution layer are both connected to the sixth dot multiplication layer; wherein the first dot multiplication layer to the sixth multiplication layer are all used to perform dot multiplication on the input features;
[0019] The outputs of the first to sixth multiplication layers are all connected to the first splicing layer, the first image input layer is also respectively connected to the first average pooling layer and the second convolution layer, the output of the first average pooling layer is connected to the first splicing layer through the first convolution layer and the bilinear interpolation layer in sequence, and the output of the second convolution layer is connected to the first splicing layer; wherein the bilinear interpolation layer is used to adjust the first processed image output by the first convolution layer to a preset size through an interpolation operation, and the first splicing layer is used to splice the first processed images output by the six dot product layers, the first processed image output by the bilinear interpolation layer, and the first processed image output by the second convolution layer along the channel dimension.
[0020] In a possible implementation, the multi-level image segmentation network based on the lightweight deformable residual module and the multi-scale enhanced self-attention mechanism module includes:
[0021] Constructing a first lightweight deformable residual module and a first multi-scale enhanced self-attention mechanism module that are sequentially connected to form a first-level feature extraction process;
[0022] Constructing a second lightweight deformable residual module and a second multi-scale enhanced self-attention mechanism module connected sequentially to form a second-level feature extraction process;
[0023] Constructing a third lightweight deformable residual module and a third multi-scale enhanced self-attention mechanism module connected sequentially to form a third-level feature extraction process;
[0024] Constructing a fourth lightweight deformable residual module and a fourth multi-scale enhanced self-attention mechanism module connected sequentially to form a fourth-level feature extraction process;
[0025] Constructing a fifth lightweight deformable residual module and a fifth multi-scale enhanced self-attention mechanism module connected sequentially to form a fifth-level feature extraction process;
[0026] The structures corresponding to the first-level feature extraction process, the second-level feature extraction process, the third-level feature extraction process, the fourth-level feature extraction process and the fifth-level feature extraction process are sequentially spliced to obtain a multi-level image segmentation network.
[0027] In a possible implementation, a multi-level image feature extraction is performed on a target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, including:
[0028] In the feature extraction process at each level, the first intermediate processed image is obtained by expanding the receptive field and dynamically fitting the edge contour through a lightweight deformable residual module;
[0029] The spatial features and channel features of the first intermediate processed image are extracted through the multi-scale enhanced self-attention mechanism module in the multi-level image segmentation network, and the down-sampled processed image corresponding to each level feature extraction process is obtained, and the down-sampled processed image corresponding to each level feature extraction process is used as the input feature map of the next level feature extraction process, thereby realizing multi-level image feature extraction;
[0030] Among them, the downsampled image output by the last level feature extraction process is used as the input of the multi-level upsampling network.
[0031] In a possible implementation, a multi-level upsampling network based on a multi-scale enhanced self-attention mechanism module includes:
[0032] Constructing a first upsampling module and a sixth multi-scale enhanced self-attention mechanism module connected sequentially to form a first-level upsampling process;
[0033] Constructing a second upsampling module and a seventh multi-scale enhanced self-attention mechanism module connected sequentially to form a second-level upsampling process;
[0034] Constructing a third upsampling module and an eighth multi-scale enhanced self-attention mechanism module connected sequentially to form a third-level upsampling process;
[0035] Constructing the fourth upsampling module and the ninth multi-scale enhanced self-attention mechanism module connected sequentially to form the fourth level upsampling process;
[0036] Constructing a fifth upsampling module and a tenth convolutional layer connected in sequence to form a fifth level upsampling process;
[0037] The structures corresponding to the first level upsampling process, the second level upsampling process, the third level upsampling process, the fourth level upsampling process and the fifth level upsampling process are sequentially spliced to obtain a multi-level upsampling network;
[0038] Among them, the tenth convolutional layer replaces the convolution of the multi-scale enhanced self-attention mechanism module in the last size upsampling process.
[0039] In a possible implementation, the first upsampling module to the fifth upsampling module each include a third input layer, a ninth convolutional layer, a batch normalization layer, a linear interpolation layer, a Relu activation function layer and a third output layer connected in sequence; wherein the third input layer is used to receive a third processed image input to the upsampling module, and the third output layer is used to output the third processed image processed by the upsampling module.
[0040] In a possible implementation, a multi-level upsampling network based on a multi-scale enhanced self-attention mechanism module is used to sequentially perform multi-level upsampling on the downsampled processed image of the last size to obtain multiple upsampled processed images of different sizes, including:
[0041] In the upsampling process at each level, the feature map size is restored through the upsampling module to obtain the second intermediate processed image;
[0042] The spatial features and channel features of the second intermediate processing image are extracted through the multi-scale enhanced self-attention mechanism module in the multi-level upsampling network to obtain the up-sampled processing image corresponding to each level upsampling process, and the up-sampled processing image corresponding to each level upsampling process is jump-connected with the corresponding down-sampled processing image as the input feature map of the next level upsampling process, thereby realizing multi-level upsampling and obtaining multiple up-sampled processing images of different sizes;
[0043] Among them, the upsampled processed image output by the last level upsampling process is used as the target image segmentation processing result.
[0044] In a possible implementation, semantically connecting the downsampled image and the upsampled image through a plurality of semantic mitigation modules with different depths includes:
[0045] For the v The upsampled image of size and its corresponding downsampled image are processed with a depth of v The semantic relief module is skipped.
[0046] in, v =1,2,…,J-1, where J represents the upper limit of the total number of layers in the multi-level upsampling network.
[0047] On the other hand, the present invention provides an image processing device based on deep dynamic self-adjustment, comprising: a target image acquisition module, an encoder, a decoder, a semantic connection module, and a target image segmentation processing result acquisition module;
[0048] The target image acquisition module is used to acquire a target processed medical image; wherein the target processed medical image is used to characterize an image of a lesion site containing a tumor;
[0049] The first sampling unit is used to perform multi-level image feature extraction on the target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, so as to dynamically fit edge contours and extract channel space features, and obtain down-sampled processed images of multiple sizes corresponding to the number of layers of the multi-level image segmentation network;
[0050] The second sampling unit is used to sequentially perform multi-level upsampling on the down-sampled processed image of the last size through a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module to obtain a plurality of up-sampled processed images of different sizes; wherein the multi-level upsampling network has the same number of layers as the multi-level image segmentation network;
[0051] The semantic connection module is used to semantically connect the downsampled processed image and the upsampled processed image through multiple semantic relief modules with different depths during multiple upsampling processes, and adaptively adjust the semantic difference on the skip connections of different layers to generate the input processed image of the next size upsampling process using the upsampled processed image of the previous size;
[0052] Among them, the down-sampled image of the u-th size corresponds to the up-sampled image of the v-th size, and when v =1,2,…,J-1, u =I-1,I-2,…,1; I represents the total number of layers of the multi-level image segmentation network, J represents the total number of layers of the multi-level upsampling network with an upper limit, I=J;
[0053] The target image segmentation processing result acquisition module is used to replace the multi-scale enhanced self-attention mechanism module in the last size upsampling process with convolution to convert the upsampled processing image output by the penultimate size upsampling process into an output processing image to obtain the target image segmentation processing result.
[0054] The present invention provides an image processing method and device based on deep dynamic self-adjustment. First, a multi-level image feature extraction is performed on the target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module. Then, a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module is used to sequentially perform multi-level upsampling on the downsampled processed image of the last size. This method can dynamically capture shallow contour information while avoiding the problems of excessive computational consumption and difficulty in training. It avoids huge computational overhead and effectively alleviates the redundancy of feature information. Finally, a plurality of semantic mitigation modules with different depths are used to semantically connect the downsampled processed image with the upsampled processed image, so as to alleviate the semantic difference in a simple and efficient manner and improve the network segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0056] Figure 1 A flowchart of an image processing method based on depth dynamic self-adjustment provided by an embodiment of the present invention;
[0057] Figure 2A structural relationship diagram of a multi-level image segmentation network, a multi-level upsampling network, and a semantic mitigation module provided by an embodiment of the present invention;
[0058] Figure 3 A schematic diagram of the structure of a lightweight deformable residual module provided by an embodiment of the present invention;
[0059] Figure 4 A schematic diagram of the structure of a multi-scale enhanced self-attention mechanism module provided in an embodiment of the present invention;
[0060] Figure 5 A schematic diagram of the structure of an upsampling module provided in an embodiment of the present invention;
[0061] Figure 6 A schematic diagram of test comparison results provided by an embodiment of the present invention;
[0062] Figure 7 A schematic diagram of the structure of a semantic mitigation module provided by an embodiment of the present invention;
[0063] Figure 8 A schematic diagram of the structure of an image processing device based on depth dynamic self-adjustment provided by an embodiment of the present invention.
[0064] Marks and corresponding parts names in the attached drawings:
[0065] LDR1-first lightweight deformable residual module, LDR2-second lightweight deformable residual module, LDR3-third lightweight deformable residual module, LDR4-fourth lightweight deformable residual module, LDR5-fifth lightweight deformable residual module, MSM1-first multi-scale enhanced self-attention mechanism module, MSM2-second multi-scale enhanced self-attention mechanism module, MSM3-third multi-scale enhanced self-attention mechanism module, MSM4-fourth multi-scale enhanced self-attention mechanism module, MSM5-fifth multi-scale enhanced self-attention mechanism module, MSM6-first Six multi-scale enhanced self-attention mechanism modules, MSM7-seventh multi-scale enhanced self-attention mechanism module, MSM8-eighth multi-scale enhanced self-attention mechanism module, MSM9-ninth multi-scale enhanced self-attention mechanism module, UP1-first upsampling module, UP2-second upsampling module, UP3-third upsampling module, UP4-fourth upsampling module, UP5-fifth upsampling module, 801-target image acquisition module, 802-encoder, 803-decoder, 804-semantic connection module, 805-target image segmentation processing result acquisition module. DETAILED DESCRIPTION
[0066] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0067] like Figure 1 As shown, an embodiment of the present invention provides an image processing method based on depth dynamic self-adjustment, comprising:
[0068] S101, obtaining a target processed medical image; wherein the target processed medical image is used to represent an image of a lesion site containing a tumor;
[0069] For example, the target medical image to be processed may be a CT image containing a liver tumor, and the tumor portion can be extracted by performing image segmentation on the CT image.
[0070] S102. Perform multi-level image feature extraction on the target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, so as to dynamically fit edge contours and extract channel space features, and obtain down-sampled processed images of multiple sizes corresponding to the number of layers of the multi-level image segmentation network.
[0071] The lightweight deformable residual module (LDR) can effectively expand the receptive field, dynamically fit edge contours, and capture shallow contour information, while avoiding the problems of excessive computational consumption and difficulty in training.
[0072] The multi-scale enhanced self-attention mechanism (MSM) first splits the information channel and then performs point multiplication to extract spatial and channel features. The multi-scale enhanced self-attention mechanism learns global and local contexts with limited computational complexity, avoiding huge computational overhead and improving the performance of network segmentation. After the above two modules, the feature information is transmitted to the multi-level upsampling network through jump connections.
[0073] S103, performing multi-level upsampling on the downsampled processed image of the last size in turn through a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module to obtain a plurality of upsampled processed images of different sizes; wherein the multi-level upsampling network has the same number of layers as the multi-level image segmentation network;
[0074] The multi-level upsampling network is composed of repeated upsampling modules and multi-scale enhanced self-attention mechanism modules, which are connected with jump connections before the second upsampling, which helps to alleviate semantic differences and promote the accuracy of network segmentation.
[0075] S104, in the multiple upsampling processes, semantically connecting the downsampled processed image with the upsampled processed image through multiple semantic relief modules with different depths, and adaptively adjusting the semantic difference on the skip connections of different layers, so as to use the upsampled processed image of the previous size to generate the input processed image of the upsampling process of the next size;
[0076] Among them, the down-sampled image of the u-th size corresponds to the up-sampled image of the v-th size, and when v =1,2,…,J-1, u =I-1,I-2,…,1; I represents the total number of layers of the multi-level image segmentation network, J represents the total number of layers of the multi-level upsampling network with an upper limit, I=J;
[0077] The number of convolutional layers of the semantic mitigation module (SMM) varies at different depths. This layout is designed to reduce the computational cost while alleviating the semantic gap. The semantic mitigation module uses its own convolutional blocks to alleviate the semantic gap between the multi-level image segmentation network and the multi-level upsampling network information. At the same time, the module's own jump connections also alleviate the semantic gap between internal information.
[0078] In order for those skilled in the art to further understand the technical solution described in the embodiment of the present invention, the jump connection is described below by way of example.
[0079] Assuming that the total number of layers of the multi-level image segmentation network and the total number of layers of the multi-level upsampling network are both set to 5 layers, then the up-sampled processed image output by the first-level upsampling process should be jump-connected to the down-sampled processed image of the fourth size, the up-sampled processed image output by the second-level upsampling process should be jump-connected to the down-sampled processed image of the third size, and so on, until the up-sampled processed image output by the fourth-level upsampling process should be jump-connected to the down-sampled processed image of the first size.
[0080] S105. Use convolution to replace the multi-scale enhanced self-attention mechanism module in the last size upsampling process to convert the upsampled image output by the penultimate size upsampling process into an output processed image to obtain a target image segmentation processing result.
[0081] like Figure 2As shown, in order to further understand the embodiments of the present invention, the structural relationship between the multi-level image segmentation network, the multi-level upsampling network and the semantic mitigation module is first introduced.
[0082] In a possible implementation, the multi-level image segmentation network based on the lightweight deformable residual module and the multi-scale enhanced self-attention mechanism module includes:
[0083] Constructing a first lightweight deformable residual module LDR1 and a first multi-scale enhanced self-attention mechanism module MSM1 that are sequentially connected to form a first-level feature extraction process;
[0084] Constructing a second lightweight deformable residual module LDR2 and a second multi-scale enhanced self-attention mechanism module MSM2 connected sequentially to form a second-level feature extraction process;
[0085] Constructing a third lightweight deformable residual module LDR3 and a third multi-scale enhanced self-attention mechanism module MSM3 connected sequentially to form a third-level feature extraction process;
[0086] Constructing a fourth lightweight deformable residual module LDR4 and a fourth multi-scale enhanced self-attention mechanism module MSM4 that are sequentially connected to form a fourth-level feature extraction process;
[0087] Constructing a fifth lightweight deformable residual module LDR5 and a fifth multi-scale enhanced self-attention mechanism module MSM5 connected sequentially to form a fifth-level feature extraction process;
[0088] The structures corresponding to the first-level feature extraction process, the second-level feature extraction process, the third-level feature extraction process, the fourth-level feature extraction process and the fifth-level feature extraction process are sequentially spliced to obtain a multi-level image segmentation network.
[0089] In a possible implementation, a multi-level upsampling network based on a multi-scale enhanced self-attention mechanism module includes:
[0090] Constructing a first upsampling module UP1 and a sixth multi-scale enhanced self-attention mechanism module MSM6 connected sequentially to form a first-level upsampling process;
[0091] Constructing a second upsampling module UP2 and a seventh multi-scale enhanced self-attention mechanism module MSM7 connected sequentially to form a second-level upsampling process;
[0092] Constructing a third upsampling module UP3 and an eighth multi-scale enhanced self-attention mechanism module MSM8 connected sequentially to form a third-level upsampling process;
[0093] Constructing a fourth upsampling module UP4 and a ninth multi-scale enhanced self-attention mechanism module MSM9 connected sequentially to form a fourth-level upsampling process;
[0094] Constructing a fifth upsampling module UP5 and a tenth convolutional layer connected in sequence to form a fifth level upsampling process;
[0095] The structures corresponding to the first level upsampling process, the second level upsampling process, the third level upsampling process, the fourth level upsampling process and the fifth level upsampling process are sequentially spliced to obtain a multi-level upsampling network;
[0096] Among them, the tenth convolutional layer replaces the convolution of the multi-scale enhanced self-attention mechanism module in the last size upsampling process.
[0097] In a possible implementation, the lightweight deformable residual module includes: a second image input layer, a sixth convolutional layer, a first maximum pooling layer, a seventh convolutional layer, a deformable convolutional layer, a fully connected layer, a bias layer, an eighth convolutional layer and a second image output layer connected in sequence; the output of the sixth convolutional layer is residually connected to the output of the eighth convolutional layer, and the output of the seventh convolutional layer is residually connected to the output of the bias layer; wherein the second image input layer is used to receive the second processed image of the input lightweight deformable residual module, and the second image output layer is used to output the second processed image processed by the lightweight deformable residual module.
[0098] The bias layer is set to increase the flexibility and expressiveness of the model, so that the model can better adapt to and fit the data, and allow the network to adjust parameters more flexibly.
[0099] Due to the inherent defects of Double Conv (double convolution structure), feature information cannot be effectively extracted. The emergence of Deformable Convolutional (deformable convolution) can extract richer feature information. Therefore, the embodiment of the present invention proposes a lightweight deformable residual module.
[0100] The lightweight deformable residual module solves the computational complexity and computational instability of deformable convolutions. It can replace ordinary counterparts in deep CNNs and can also be trained end-to-end using standard back-propagation. To prevent the gradient update problem in back-propagation, two residual connections are used in this module to provide double protection.
[0101] like Figure 3As shown in the figure, the working principle of the lightweight deformable residual module may include: first receiving the feature map from the second image input layer, the feature map passes through the 3×3 sixth convolution layer, and then passes through the 1×1 seventh convolution layer. Since the deformable convolution layer needs to learn redundant parameter matrices, it will bring more computational overhead. Therefore, in order to avoid the above problems, the 1×1 seventh convolution layer is used to compress the channel, and then pass through the deformable convolution layer. After extracting feature information through the deformable convolution layer, in order to restore the number of channels, it will pass through the 1×1 eighth convolution layer, and the final output will be residually connected with the image after the 3×3 sixth convolution layer. In addition, the output after the 1×1 seventh convolution layer will also be residually connected with the output after the deformable convolution block. Since this module can avoid excessive computational overhead while dynamically obtaining contour information, the addition of the lightweight deformable residual module has a better improvement in the overall network segmentation effect.
[0102] During back propagation, if the 1×1 seventh convolutional layer has a gradient update problem, information can be transferred through the outer residual connection to avoid gradient explosion or vanishing problems. During back propagation, if the variable convolution cannot effectively transfer information, information can be transferred through the inner residual connection. Although the variable convolution process is missing, the 3×3 sixth convolutional layer can also effectively extract feature information.
[0103] Among them, the variable convolution formula is as follows:
[0104]
[0105] in, Indicated in i Line j The pooling result calculated in the bin of the column usually represents the value of a certain position in the RoI pooling output processing image. Indicates getting the pixel value on the input feature map at a specific location. represents that for each position on the output processed image y, p represents an arbitrary position, and , Indicates gauge offset, Enumerate the positions in R, Indicates that for i Line j The bin position offset of the column, Represents the number of pixels in the feature map, bin ( i , j ) represents an area where pooling operation is performed in a specific area,
[0106] The overall process formula of this module is as follows:
[0107]
[0108] Among them, Y represents the output, It means that after a conventional convolution operation with a convolution kernel size of 1×1, the output channel is C, k represents the convolution kernel size, C represents the number of original input channels, and c represents the number of channels. It means that the channel becomes C / 4 after passing through the lightweight deformable residual module. It means that after the conventional convolution operation with a convolution kernel size of 1×1, the output channel is C / 4. It means that e undergoes a conventional convolution operation with a convolution kernel size of 1×1, and the output channel is C, and e represents the input image.
[0109] In a possible implementation, the multi-scale enhanced self-attention mechanism module includes:
[0110] A first image input layer, a channel attention layer, a segmentation layer, a multi-scale global feature extraction submodule, a third convolution layer, a first addition layer and a first image output layer connected in sequence, and the first image input layer is also connected to the multi-scale global feature extraction submodule and the first addition layer respectively; wherein the first image input layer is used to receive a first processed image input to the multi-scale enhanced self-attention mechanism module, the segmentation layer is used to divide the first processed image output by the channel attention layer into four groups along the channel dimension, the first addition layer is used to add the feature map received by the first image input layer to the first processed image output by the third convolution layer, and the first image output layer is used to output the first processed image processed by the multi-scale enhanced self-attention mechanism module;
[0111] The multi-scale global feature extraction submodule includes a first depth convolution layer, a second depth convolution layer, a third depth convolution layer, a fourth depth convolution layer, a first dot product layer, a second dot product layer, a third dot product layer, a fourth dot product layer, a fifth dot product layer, a sixth dot product layer, a first average pooling layer, a first convolution layer, a bilinear interpolation layer, a first splicing layer and a second convolution layer;
[0112] The four outputs of the segmentation layer are respectively connected to the first depth convolution layer, the second depth convolution layer, the third depth convolution layer and the fourth depth convolution layer; wherein the convolution kernels of the first depth convolution layer, the second depth convolution layer, the third depth convolution layer and the fourth depth convolution layer are different;
[0113] The output of the first deep convolution layer and the output of the second deep convolution layer are both connected to the first dot multiplication layer, the output of the first deep convolution layer and the output of the third deep convolution layer are both connected to the second dot multiplication layer, the output of the first deep convolution layer and the output of the fourth deep convolution layer are both connected to the third dot multiplication layer, the output of the second deep convolution layer and the output of the third deep convolution layer are both connected to the fourth dot multiplication layer, the output of the second deep convolution layer and the output of the fourth deep convolution layer are both connected to the fifth multiplication layer, and the output of the third deep convolution layer and the output of the fourth deep convolution layer are both connected to the sixth dot multiplication layer; wherein, the first dot multiplication layer to the sixth multiplication layer are all used to perform dot multiplication on the input features; the dot multiplication refers to multiplying the corresponding data points in the two feature maps to reconstruct a feature map.
[0114] The outputs of the first to sixth multiplication layers are all connected to the first splicing layer, the first image input layer is also respectively connected to the first average pooling layer and the second convolution layer, the output of the first average pooling layer is connected to the first splicing layer through the first convolution layer and the bilinear interpolation layer in sequence, and the output of the second convolution layer is connected to the first splicing layer; wherein the bilinear interpolation layer is used to adjust the first processed image output by the first convolution layer to a preset size through an interpolation operation, and the first splicing layer is used to splice the first processed images output by the six dot product layers, the first processed image output by the bilinear interpolation layer, and the first processed image output by the second convolution layer along the channel dimension.
[0115] In order for those skilled in the art to further understand the technical solutions described in the embodiments of the present invention, Figure 4 The working principle of the multi-scale enhanced self-attention mechanism module is explained.
[0116] like Figure 4 As shown in the figure, before channel segmentation, the channel information is first modeled through the channel attention layer to avoid unimportant channel information affecting the subsequent spatial information extraction. The feature map is then divided into four groups along the channel dimension, and each group of channels undergoes deep convolutions of different scales (i.e., the first deep convolution layer with a convolution kernel of 3, the second deep convolution layer with a convolution kernel of 5, the third deep convolution layer with a convolution kernel of 7, and the fourth deep convolution layer with a convolution kernel of 9), so that spatial information of different scales can be extracted. These four groups of features are dot-multiplied in pairs and then spliced to achieve interaction between features of different scales. Subsequent experimental results show that one-quarter of the channels can effectively obtain all features. At the same time, average pooling is used to retain more background information, avoid weakening edge information in the later stage, and do a good job in the early stage of obtaining global contextual relationships.
[0117] Since the multi-scale enhanced self-attention mechanism module does not change the number of channels in the feature map, it is placed after each change in the number of channels. First, the feature map passes through the channel attention layer, analyzes the channels and assigns corresponding weights. Then the feature map channels are cut and evenly divided into four parts, defined as x1, x2, x3, and x4.
[0118] Among them, the dot product of x1 and x2 is X1, the dot product of x1 and x3 is X2, the dot product of x1 and x4 is X3, the dot product of x2 and x3 is X4, the dot product of x2 and x4 is X5, and the dot product of x3 and x4 is X6, thus obtaining six new feature maps: X1, X2, X3, X4, X5, and X6.
[0119] At the same time, the input feature map passes through the first average pooling layer and the first 1×1 convolution, and finally passes through the functional.interpolate.fi (bilinear interpolation layer) to interpolate the feature map and adjust it to a specific size. , the feature map that only passes through the second 1×1 convolutional layer will be obtained Then the feature map y1, feature map y2, and feature map output by the first point multiplication layer , the feature map output by the second multiplication layer , the feature map output by the third multiplication layer , the feature map output by the fourth multiplication layer , the feature map output by the fifth multiplication layer , the feature map output by the sixth multiplication layer Splicing along the channel dimension yields :
[0120]
[0121] Among them, concat means splitting channels. It means that after the conventional convolution operation with a convolution kernel size of 1×1, Indicates average pooling of the input image. It represents a regular convolution operation with a convolution kernel size of 1×1 on the input image, where H represents height and W represents width.
[0122] In order to meet the requirements of subsequent operations, it is necessary to perform a 1×1 convolution again to restore the feature map to the channel size of the input, so that it is As the network gets deeper, the correlation between the gradients propagated back will become worse and worse, so a residual connection is added between the input and output of the multi-scale enhanced self-attention mechanism module. This branch allows the network to directly pass the gradient to the earlier layers of the network through this shortcut during the back propagation process, thereby effectively alleviating the problem of gradually weakening correlation. The final output is The original input feature map Add it to z2.
[0123] In a possible implementation, a multi-level image feature extraction is performed on a target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, including:
[0124] In the feature extraction process at each level, the first intermediate processed image is obtained by expanding the receptive field and dynamically fitting the edge contour through a lightweight deformable residual module;
[0125] The spatial features and channel features of the first intermediate processed image are extracted through the multi-scale enhanced self-attention mechanism module in the multi-level image segmentation network, and the down-sampled processed image corresponding to each level feature extraction process is obtained, and the down-sampled processed image corresponding to each level feature extraction process is used as the input feature map of the next level feature extraction process, thereby realizing multi-level image feature extraction;
[0126] Among them, the downsampled image output by the last level feature extraction process is used as the input of the multi-level upsampling network.
[0127] In a possible implementation, the first upsampling module to the fifth upsampling module each include a third input layer, a ninth convolutional layer, a batch normalization layer, a linear interpolation layer, a Relu activation function layer and a third output layer connected in sequence; wherein the third input layer is used to receive a third processed image input to the upsampling module, and the third output layer is used to output the third processed image processed by the upsampling module.
[0128] Due to the difference in network depth, there will be semantic deviation. The present invention only combines ordinary convolution and upsampling operations to achieve effective fusion of context feature information, while being able to achieve more specific and effective semantic difference mitigation according to different depths. Figure 5As shown in the figure, in the overall framework, the number of convolution layers in each upsampling module decreases from top to bottom. The top upsampling module has four convolution layers, while the bottom upsampling module has only one 3×3 convolution block. Therefore, the overall upsampling modules with 4, 3, 2, and 1 layers (respectively denoted as SMM4, SMM3, SMM2, and SMM1) are used. This method can adaptively adjust the semantic difference on the jump connection of different layers, which will be better than the presentation result of using the same convolution layer upsampling module, which will be confirmed in subsequent experimental data.
[0129] In a possible implementation, a multi-level upsampling network based on a multi-scale enhanced self-attention mechanism module is used to sequentially perform multi-level upsampling on the downsampled processed image of the last size to obtain multiple upsampled processed images of different sizes, including:
[0130] In the upsampling process at each level, the feature map size is restored through the upsampling module to obtain the second intermediate processed image;
[0131] The spatial features and channel features of the second intermediate processing image are extracted through the multi-scale enhanced self-attention mechanism module in the multi-level upsampling network to obtain the up-sampled processing image corresponding to each level upsampling process, and the up-sampled processing image corresponding to each level upsampling process is jump-connected with the corresponding down-sampled processing image as the input feature map of the next level upsampling process, thereby realizing multi-level upsampling and obtaining multiple up-sampled processing images of different sizes;
[0132] Among them, the upsampled processed image output by the last level upsampling process is used as the target image segmentation processing result.
[0133] In a possible implementation, semantically connecting the downsampled image and the upsampled image through a plurality of semantic mitigation modules with different depths includes:
[0134] For the v The upsampled image of size and its corresponding downsampled image are processed with a depth of v The semantic relief module is skipped.
[0135] in, v =1,2,…,J-1, where J represents the upper limit of the total number of layers in the multi-level upsampling network.
[0136] Experiments were conducted on the CVC-ClinicDB dataset (a database of frames extracted from colonoscopy videos). From Table 1, it can be seen that compared with ordinary U-Net (82.05%), under the guidance of various attention mechanisms, such as Attention-Unet (84.74%) and PAttention-Unet (85.93%), the segmentation accuracy is significantly higher than that of U-Net. This illustrates the important role of attention perception mechanisms in medical image segmentation. In recent years, transformer-based frameworks, namely UCTransNet (87.56%) and TransAttention-Unet (85.97%), have also achieved good results, thanks to the fact that these methods apply self-attention technology to enhance the network's ability to model long-range dependencies. In contrast, the DDS-UNet (i.e. Figure 2 All indicators of the proposed algorithm are higher than those of all other algorithms, which can be attributed to the superiority of the proposed algorithm in self-attention, multi-scale information extraction and mitigation of semantic gap. Figure 6 It can be clearly seen that the results of the method of the present invention are more accurate than those of other methods, the contours are smoother, and there is no incompleteness or misjudgment of pathological areas.
[0137] Table 1: Performance comparison of DDS-UNet and other baseline methods on the CVC-ClinicDB dataset
[0138]
[0139] Among them, UNet represents U-type network, UNet++ represents nested UNet, Attention-UNet represents intention-gated UNet, ResUNet represents weighted residual U-type network, UTNet represents U-type hybrid transformer network, UCTransNet represents U-type channel transformer network, PAttUNet represents partial attention U-type network, DCSAU-Net represents a deeper and more compact attention-based UNet, TransAttUnet represents a multi-level attention-guided UNet with transformer, and DDS-Unet represents a deep dynamic self-adjusting UNet for medical image segmentation. Dice represents the similarity coefficient used to evaluate the segmentation result and the true label.
[0140] Experiments were conducted on the Kvasir-SEG dataset (gastrointestinal polyp dataset), and the corresponding results are shown in Table 2. Compared with the CVC-ClinicDB dataset, although the Kvasir-SEG dataset contains more colon polyp images, it also contains more images that are more difficult to segment, which leads to a decline in the performance of all networks on this dataset. However, the DDS-UNet proposed in the present invention is always better than other algorithms, and produces the highest IoU score of 78.1% and DICE score of 87.37%, which once again proves the superiority of the DDS-UNet method of the present invention. Compared with the algorithms based on the transformer structure in recent years, the algorithm of the present invention is also more advantageous, namely TransAttention-Unet (76.92%) and UCTransNet (77.19%). This once again verifies that the algorithm of the present invention is more advantageous in alleviating semantic differences and self-attention, and also proves the powerful ability of DDS-UNet in medical image segmentation. From Figure 6 It can be seen from the qualitative result diagram that when processing the polyp segmentation diagram, the DDS-UNet segmentation result of the present invention is more complete, and the edges are not blurred.
[0141] Table 2: Performance comparison of DDS-UNet and other baseline methods on the Kvasir-SEG dataset
[0142]
[0143] The segmentation effect of DDS-UNet on the skin lesion dataset was evaluated. The experimental results of all algorithms are shown in Table 3. Based on the experimental results in Table 3, it can be seen that compared with other algorithms, the DDS-UNet of the present invention achieves better segmentation evaluation results on the skin lesion dataset, and produces the highest IoU score of 83.89% and Dice score of 91.01%, which is 3.07% higher than the IoU of Unet and 1.92% higher than the Dice. This demonstrates the superiority of DDS-UNet. The IoU of DCSAUNet is 83.61%, ranking second. Although the gap is not large, the performance of the comparison algorithm on other datasets is relatively poor, and the segmentation performance is not stable. And from Figure 6 It can also be seen that the algorithm of the present invention is more accurate in segmenting skin diseases and has clearer edge processing. This is also due to the self-attention, multi-scale information extraction and jump connection of DDS-UNet to alleviate semantic differences, which can help learn global context information and distinguish features to distinguish skin lesion areas from surrounding structures.
[0144] Table 3: Performance comparison of DDS-UNet and other baseline methods on the isic2018 dataset
[0145]
[0146] Finally, a comparative experiment on the Dataset B dataset was conducted to demonstrate the effectiveness of the technical solution of the present invention on the breast ultrasound dataset, and the experimental results are shown in Table 4. According to these experimental results, the present invention can be observed that through the self-attention mechanism, the multi-scale feature extraction module, and the alleviated semantic difference jump connection, the DDS-UNet of the present invention achieves better performance than the comparison algorithm and provides a new state-of-the-art technology for breast segmentation. In particular, DDS-UNet (72.9%) is far superior to Attention-Unet (67.69%), ResUnet (66.64%), TransAttention-Unet (67.83%), and UCTransNet (66.28%), which proves the powerful ability of DDS-UNet in medical image segmentation. In addition, Figure 6 It shows that the DDS-UNet of the present invention can better distinguish the breast itself and the surrounding tissues, thereby achieving excellent gland segmentation performance.
[0147] Table 4: Performance comparison of DDS-UNet and other baseline methods on Breast Ultrasound Dataset B
[0148]
[0149] The overall process of the technical solution of the present invention is described below in the form of examples, which may include:
[0150] In the first level feature extraction process:
[0151] A1. Input a target processed medical image with a height × width × channel (H × W × C) of 384 × 384 × 3, and perform LDR1 operation on the target processed medical image;
[0152] The operation of LDR1 may include: performing a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and outputting a first feature map of 384×384×64; performing a maximum pooling operation with a convolution kernel of 2×2 and a step of 2 on the first feature map, and outputting a second feature map of 192×192×64; performing a conventional convolution operation with a convolution kernel of 1×1 and a step of 1 on the second feature map, and outputting a third feature map of 192×192×16; the third feature map does not change the number of channels and the size of the image after the deformable volume, that is, the output fourth feature map is 192×192×16; finally, the fourth feature map is subjected to a convolution kernel operation of 1×1, and the fifth feature map is output as 192×192×64;
[0153] A2. Multi-scale global feature extraction.
[0154] (1) First, the SE (channel) attention mechanism is performed on the fifth feature map, and the output sixth feature map is 192×192×64; the sixth feature map is divided into four equal parts along the channel dimension. The four parts are defined as the first feature submap x1, the second feature submap x2, the third feature submap x3, and the fourth feature submap x4. The four feature maps are all 192×192×16; the first feature submap x1 is subjected to a normal convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output seventh feature map is 192×192×16; the second feature submap x2 is subjected to a normal convolution operation with a convolution kernel of 5×5, a step of 1, and a padding of 2, and the output eighth feature map is 192×192×1 6; The third feature sub-graph x3 is subjected to a normal convolution operation with a convolution kernel of 7×7, a step of 1, and a padding of 3, and the output of the ninth feature graph is 192×192×16; The fourth feature sub-graph x4 is subjected to a normal convolution operation with a convolution kernel of 9×9, a step of 1, and a padding of 4, and the output of the tenth feature graph is 192×192×16; The seventh feature graph, the eighth feature graph, the ninth feature graph, and the tenth feature graph are dot-multiplied in pairs to obtain 6 new feature graphs, which are defined as the first dot-multiplied feature graph X1, the second dot-multiplied feature graph X2, the third dot-multiplied feature graph X3, the fourth dot-multiplied feature graph X4, the fifth dot-multiplied feature graph X5, and the sixth dot-multiplied feature graph X6, respectively. These 6 output images are all 192×192×16;
[0155] (2) Perform average pooling on the fifth feature map, and output the eleventh feature map as 1×1×64; then perform a conventional convolution operation on the eleventh feature map with a convolution kernel of 1×1 and a step of 1, and output the twelfth feature map as 1×1×64; perform an interpolation operation on the twelfth feature map, and output the thirteenth feature map as 192×192×64;
[0156] (3) Perform a conventional convolution operation on the fifth feature map with a convolution kernel of 1×1 and a step of 1, and output the fourteenth feature map to be 192×192×32;
[0157] The first dot product feature map X1, the second dot product feature map X2, the third dot product feature map X3, the fourth dot product feature map X4, the fifth dot product feature map X5, the sixth dot product feature map X6, the thirteenth feature map, and the fourteenth feature map outputted from steps (1), (2), and (3) are concatenated to output a fifteenth feature map of 192×192×192; a conventional convolution operation with a convolution kernel of 1×1 and a step of 1 is then performed on the fifteenth feature map to output a sixteenth feature map of 192×192×64; the sixteenth feature map is then added to the fifth feature map to obtain a final first down-sampled image of 192×192×64;
[0158] A3, performing SMM4 operation on the first down-sampled image to obtain a fourth skip connection feature map (denoted as s4);
[0159] like Figure 7 As shown, the operation of SMM4 includes:
[0160] The first down-sampled image is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the first target feature map is 96×96×128 (denoted as a1); the first target feature map is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the second target feature map is 48×48×256 (denoted as a2); the second target feature map is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the third target feature map is 24×24×512 (denoted as a3); the third target feature map is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the fourth target feature map is 12×12×1024; the fourth target feature map is upsampled , the fifth target feature map is output as 24×24×512; the fifth target feature map is added to the third target feature map a3, and the sixth target feature map is output as 24×24×512; the sixth target feature map is upsampled, and the seventh target feature map is output as 48×48×256; the seventh target feature map is added to the second target feature map a2, and the eighth target feature map is output as 48×48×256; the eighth target feature map is upsampled, and the ninth target feature map is output as 96×96×128; the ninth target feature map is added to the first target feature map a1, and the tenth target feature map is output as 96×96×128; the tenth target feature map is upsampled, and the fourth jump connection feature map is output as 192×192×64.
[0161] It is worth noting that the subsequent operations of SMM3, SMM2 and SMM1 are similar, but the depth is different.
[0162] In the second level feature extraction process:
[0163] A4, performing LDR2 operation on the first down-sampled image, and outputting a seventeenth feature map of 96×96×128; wherein the operation of LDR2 is similar to the operation of LDR1, and will not be described in detail here;
[0164] A5. Repeat the multi-scale global feature extraction operation for the seventeenth feature map, and output the second down-sampled processed image to 96×96×128;
[0165] A6. Perform SMM3 operation on the second down-sampled image to obtain a third skip connection feature map (denoted as s3);
[0166] Among them, the operation of SMM3 includes:
[0167] The second down-sampled image is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the eleventh target feature map is 48×48×256 (denoted as b1); the eleventh target feature map is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the twelfth target feature map is 24×24×512 (denoted as b2); the twelfth target feature map is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output of the thirteenth target feature map is 12×12×1024; The image is upsampled, and the fourteenth target feature map is output as 24×24×512; the fourteenth target feature map is added to the twelfth target feature map b2, and the fifteenth target feature map is output as 24×24×512; the fifteenth target feature map is upsampled, and the sixteenth target feature map is output as 48×48×256; the sixteenth target feature map is added to the eleventh target feature map b1, and the seventeenth target feature map is output as 48×48×256; the seventeenth target feature map is upsampled, and the third jump connection feature map is output as 96×96×128.
[0168] In the third level feature extraction process:
[0169] A7, performing LDR3 operation on the second down-sampled image, and outputting an eighteenth feature map of 48×48×256; wherein the operation of LDR3 is similar to the operation of LDR1, and will not be described in detail here;
[0170] A8. Repeat the multi-scale global feature extraction operation for the eighteenth feature map, and output the third down-sampled processed image to 48×48×256;
[0171] A9. Perform SMM2 operation on the third down-sampled image to obtain a second skip connection feature map (denoted as s2).
[0172] Among them, the operation of SMM2 includes:
[0173] A conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1 is performed on the third down-sampled image, and the output of the eighteenth target feature map is 24×24×512 (denoted as c1); a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1 is performed on the eighteenth target feature map, and the output of the nineteenth target feature map is 12×12×1024; the nineteenth target feature map is upsampled, and the output of the twentieth target feature map is 24×24×512; the twentieth target feature map is added to the eighteenth target feature map c1, and the output of the twenty-first target feature map is 24×24×512; the twenty-first target feature map is upsampled, and the output of the second skip connection feature map is 48×48×256.
[0174] In the fourth level feature extraction process:
[0175] A10, performing an LDR4 operation on the third down-sampled image, and outputting a nineteenth feature map of 24×24×512; wherein the operation of LDR4 is similar to the operation of LDR1, and will not be described in detail here;
[0176] A11. Repeat the multi-scale global feature extraction operation for the nineteenth feature map, and output a fourth down-sampled processed image of 24×24×512;
[0177] A12, performing SMM1 operation on the fourth down-sampled image to obtain a first skip connection feature map (denoted as s1);
[0178] The operations of SMM1 include:
[0179] The fourth down-sampled image is subjected to a conventional convolution operation with a convolution kernel of 3×3, a step of 1, and a padding of 1, and the output first skip connection feature map is 24×24×512. In the fifth level feature extraction process:
[0180] A13, performing LDR4 operation on the fourth down-sampled image, and outputting a twentieth feature image of 12×12×1024;
[0181] A14, for the twentieth feature map, repeat the multi-scale global feature extraction operation, and output the fifth down-sampled processed image to 12×12×1024;
[0182] During upsampling at the first level:
[0183] A15. Perform a conventional convolution operation on the fifth downsampled image with a convolution kernel of 3×3, a step of 1, and a padding of 1, and output a twenty-first feature map of 12×12×512; perform batch normalization and Interpolate on the twenty-first feature map, and output a twenty-second feature map of 24×24×512; perform a ReLu activation function on the twenty-second feature map, and output a twenty-third feature map of 24×24×512;
[0184] A16. Repeat the multi-scale global feature extraction operation for the twenty-third feature map, and output the twenty-fourth feature map as 24×24×512.
[0185] A17, adding the twenty-fourth feature map to the first skip connection feature map (i.e., s1) subjected to the SMM1 operation, and outputting a first up-sampled processed image of 24×24×512;
[0186] During the upsampling process at the second level:
[0187] A18. Perform a conventional convolution operation on the first upsampled image with a convolution kernel of 3×3, a step of 1, and a padding of 1, and output the twenty-fifth feature map of 24×24×256; perform batch normalization and Interpolate on the twenty-fifth feature map, and output the twenty-sixth feature map of 48×48×256; perform a ReLu activation function on the twenty-sixth feature map, and output the twenty-seventh feature map of 48×48×256;
[0188] A19. Repeat the multi-scale global feature extraction operation for the twenty-seventh feature map, and output the twenty-eighth feature map as 48×48×256.
[0189] A20, adding the twenty-eighth feature map to the second skip connection feature map (i.e., s2) subjected to the SMM2 operation, and outputting a second up-sampled processed image of 48×48×256;
[0190] During the sampling process at the third level:
[0191] A21. Perform a conventional convolution operation on the second upsampled image with a convolution kernel of 3×3, a step of 1, and a padding of 1, and output the twenty-ninth feature map of 48×48×128; perform batch normalization and interpolation on the twenty-ninth feature map, and output the thirtieth feature map of 96×96×128; perform a ReLu activation function on the thirtieth feature map, and output the thirty-first feature map of 96×96×128;
[0192] A22. Repeat the multi-scale global feature extraction operation for the thirty-first feature map, and output the thirty-second feature map as 96×96×128.
[0193] A23, adding the thirty-second feature map to the third skip connection feature map (i.e., s3) subjected to the SMM3 operation, and outputting a third up-sampled processed image of 96×96×128;
[0194] During the upsampling process at the fourth level:
[0195] A24. Perform a conventional convolution operation on the third upsampled image with a convolution kernel of 3×3, a step of 1, and a padding of 1, and output a thirty-third feature map of 96×96×64; perform batch normalization and Interpolate on the thirty-third feature map, and output a thirty-fourth feature map of 192×192×64; perform a ReLu activation function on the thirty-fourth feature map, and output a thirty-fifth feature map of 192×192×64;
[0196] A25. Repeat the multi-scale global feature extraction operation for the thirty-fifth feature map, and output the thirty-sixth feature map as 192×192×64.
[0197] A26, adding the thirty-sixth feature map to the fourth skip connection feature map (i.e., s4) subjected to the SMM4 operation, and outputting a fourth up-sampled processed image of 192×192×64;
[0198] During the sampling process at the fifth level:
[0199] A27. Perform a conventional convolution operation on the fourth up-sampled image with a convolution kernel of 3×3, a step of 1, and a padding of 1, and output the thirty-sixth feature map of 192×192×32; perform batch normalization and Interpolate on the thirty-sixth feature map, and output the thirty-seventh feature map of 384×384×32; perform a ReLu activation function on the thirty-seventh feature map, and output the thirty-eighth feature map of 384×384×32;
[0200] A28. Perform a conventional convolution operation on the thirty-eighth feature map with a convolution kernel of 1×1 and a step of 1, and output the target image segmentation processing result to be 384×384×3.
[0201] like Figure 8 As shown, the embodiment of the present invention provides an image processing device based on deep dynamic self-adjustment, including: a target image acquisition module 801, an encoder 802, a decoder 803, a semantic connection module 804 and a target image segmentation processing result acquisition module 805;
[0202] The target image acquisition module 801 is used to acquire a target processed medical image; wherein the target processed medical image is used to represent an image of a lesion site containing a tumor;
[0203] The encoder 802 is used to perform multi-level image feature extraction on the target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, so as to dynamically fit edge contours and extract channel space features, and obtain down-sampled processed images of multiple sizes corresponding to the number of layers of the multi-level image segmentation network;
[0204] The decoder 803 is used to sequentially perform multi-level upsampling on the downsampled processed image of the last size through a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module to obtain a plurality of upsampled processed images of different sizes; wherein the multi-level upsampling network has the same number of layers as the multi-level image segmentation network;
[0205] The semantic connection module 804 is used to semantically connect the downsampled processed image and the upsampled processed image through multiple semantic relief modules with different depths during multiple upsampling processes, and adaptively adjust the semantic difference on the skip connection of different layers to use the upsampled processed image of the previous size to generate the input processed image of the upsampling process of the next size;
[0206] Among them, the down-sampled image of the u-th size corresponds to the up-sampled image of the v-th size, and when v =1,2,…,J-1, u =I-1,I-2,…,1; I represents the total number of layers of the multi-level image segmentation network, J represents the total number of layers of the multi-level upsampling network with an upper limit, I=J;
[0207] The target image segmentation processing result acquisition module 805 is used to replace the multi-scale enhanced self-attention mechanism module in the last size upsampling process with convolution to convert the upsampled processing image output by the penultimate size upsampling process into an output processing image to obtain the target image segmentation processing result.
[0208] The image processing device based on depth dynamic self-adjustment described in the embodiment of the present invention can execute the above-mentioned method and technical solution, and its principles and beneficial effects are similar, which will not be repeated here.
[0209] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.
[0210] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0211] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0213] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0214] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An image processing method based on deep dynamic self-adjustment, characterized in that: include: Acquiring a target processed medical image, wherein the target processed medical image is used to represent an image of a lesion site containing a tumor; A multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module is used to extract multi-level image features of the target medical image to dynamically fit edge contours and extract channel space features, thereby obtaining down-sampled processed images of multiple sizes corresponding to the number of layers of the multi-level image segmentation network; The multi-level image segmentation network based on the lightweight deformable residual module and the multi-scale enhanced self-attention mechanism module includes: constructing a first lightweight deformable residual module and a first multi-scale enhanced self-attention mechanism module connected in sequence to form a first-level feature extraction process; constructing a second lightweight deformable residual module and a second multi-scale enhanced self-attention mechanism module connected in sequence to form a second-level feature extraction process; constructing a third lightweight deformable residual module and a third multi-scale enhanced self-attention mechanism module connected in sequence to form a third-level feature extraction process; constructing a fourth lightweight deformable residual module and a fourth multi-scale enhanced self-attention mechanism module connected in sequence to form a fourth-level feature extraction process; constructing a fifth lightweight deformable residual module and a fifth multi-scale enhanced self-attention mechanism module connected in sequence to form a fifth-level feature extraction process; sequentially splicing the structures corresponding to the first-level feature extraction process, the second-level feature extraction process, the third-level feature extraction process, the fourth-level feature extraction process, and the fifth-level feature extraction process to obtain a multi-level image segmentation network; The lightweight deformable residual module includes: a second image input layer, a sixth convolution layer, a first maximum pooling layer, a seventh convolution layer, a deformable convolution layer, a fully connected layer, a bias layer, an eighth convolution layer and a second image output layer connected in sequence; the output of the sixth convolution layer is residually connected to the output of the eighth convolution layer, and the output of the seventh convolution layer is residually connected to the output of the bias layer; the second image input layer is used to receive the second processed image of the lightweight deformable residual module, and the second image output layer is used to output the second processed image processed by the lightweight deformable residual module; The downsampled image of the last size is upsampled in turn through a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module to obtain upsampled images of multiple sizes; wherein the multi-level upsampling network has the same number of layers as the multi-level image segmentation network; A multi-level upsampling network based on a multi-scale enhanced self-attention mechanism module includes: constructing a first upsampling module and a sixth multi-scale enhanced self-attention mechanism module connected in sequence to form a first-level upsampling process; constructing a second upsampling module and a seventh multi-scale enhanced self-attention mechanism module connected in sequence to form a second-level upsampling process; constructing a third upsampling module and an eighth multi-scale enhanced self-attention mechanism module connected in sequence to form a third-level upsampling process; constructing a fourth upsampling module and a ninth multi-scale enhanced self-attention mechanism module connected in sequence to form a fourth-level upsampling process; constructing a fifth upsampling module and a tenth convolutional layer connected in sequence to form a fifth-level upsampling process; sequentially splicing the structures corresponding to the first-level upsampling process, the second-level upsampling process, the third-level upsampling process, the fourth-level upsampling process and the fifth-level upsampling process to obtain a multi-level upsampling network; wherein the tenth convolutional layer replaces the convolution of the multi-scale enhanced self-attention mechanism module in the last size upsampling process; The multi-scale enhanced self-attention mechanism module includes: a first image input layer, a channel attention layer, a segmentation layer, a multi-scale global feature extraction submodule, a third convolution layer, a first addition layer and a first image output layer connected in sequence, and the first image input layer is also connected to the multi-scale global feature extraction submodule and the first addition layer respectively; wherein the first image input layer is used to receive the first processed image input to the multi-scale enhanced self-attention mechanism module, the segmentation layer is used to divide the first processed image output by the channel attention layer into four groups along the channel dimension, the first addition layer is used to add the feature map received by the first image input layer to the first processed image output by the third convolution layer, and the first image output layer is used to output the first processed image processed by the multi-scale enhanced self-attention mechanism module; The multi-scale global feature extraction submodule includes a first depth convolution layer, a second depth convolution layer, a third depth convolution layer, a fourth depth convolution layer, a first dot product layer, a second dot product layer, a third dot product layer, a fourth dot product layer, a fifth dot product layer, a sixth dot product layer, a first average pooling layer, a first convolution layer, a bilinear interpolation layer, a first splicing layer and a second convolution layer; The four outputs of the segmentation layer are respectively connected to the first depth convolution layer, the second depth convolution layer, the third depth convolution layer and the fourth depth convolution layer; wherein the convolution kernels of the first depth convolution layer, the second depth convolution layer, the third depth convolution layer and the fourth depth convolution layer are different; The output of the first depth convolution layer and the output of the second depth convolution layer are both connected to the first dot product layer, the output of the first depth convolution layer and the output of the third depth convolution layer are both connected to the second dot product layer, the output of the first depth convolution layer and the output of the fourth depth convolution layer are both connected to the third dot product layer, the output of the second depth convolution layer and the output of the third depth convolution layer are both connected to the fourth dot product layer, the output of the second depth convolution layer and the output of the fourth depth convolution layer are both connected to the fifth dot product layer, and the output of the third depth convolution layer and the output of the fourth depth convolution layer are both connected to the sixth dot product layer; wherein the first dot product layer to the sixth dot product layer are all used to perform dot product on the input features; The outputs of the first to sixth dot product layers are all connected to the first splicing layer, the first image input layer is also respectively connected to the first average pooling layer and the second convolution layer, the output of the first average pooling layer is sequentially connected to the first splicing layer through the first convolution layer and the bilinear interpolation layer, and the output of the second convolution layer is connected to the first splicing layer; wherein the bilinear interpolation layer is used to adjust the first processed image output by the first convolution layer to a preset size through an interpolation operation, and the first splicing layer is used to splice the first processed images output by the six dot product layers, the first processed image output by the bilinear interpolation layer, and the first processed image output by the second convolution layer along the channel dimension; In the multiple upsampling processes, the downsampled image and the upsampled image are semantically connected through multiple semantic relief modules with different depths, and the semantic difference on the skip connection of different layers is adaptively adjusted to use the upsampled image of the previous size to generate the input image of the upsampling process of the next size; Among them, the down-sampled image of the u-th size corresponds to the up-sampled image of the v-th size, and when v =1,2,…,J-1, u =I-1,I-2,…,1; I represents the total number of layers of the multi-level image segmentation network, J represents the total number of layers of the multi-level upsampling network with an upper limit, I=J; Convolution is used to replace the multi-scale enhanced self-attention mechanism module in the last size upsampling process to convert the upsampled image output by the penultimate size upsampling process into an output processed image to obtain the target image segmentation processing result.
2. The image processing method based on depth dynamic self-adjustment according to claim 1, characterized in that: The multi-level image feature extraction of the target processed medical image is performed through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, including: In the feature extraction process at each level, the first intermediate processed image is obtained by expanding the receptive field and dynamically fitting the edge contour through a lightweight deformable residual module; The spatial features and channel features of the first intermediate processed image are extracted through the multi-scale enhanced self-attention mechanism module in the multi-level image segmentation network, and the down-sampled processed image corresponding to each level feature extraction process is obtained, and the down-sampled processed image corresponding to each level feature extraction process is used as the input feature map of the next level feature extraction process, thereby realizing multi-level image feature extraction; Among them, the downsampled image output by the last level feature extraction process is used as the input of the multi-level upsampling network.
3. The image processing method based on depth dynamic self-adjustment according to claim 1, characterized in that: The upsampling module includes a third input layer, a ninth convolutional layer, a batch normalization layer, a linear interpolation layer, a Relu activation function layer and a third output layer connected in sequence; wherein the third input layer is used to receive the third processed image input to the upsampling module, and the third output layer is used to output the third processed image processed by the upsampling module.
4. The image processing method based on depth dynamic self-adjustment according to claim 3, characterized in that: The multi-level upsampling network composed of multi-scale enhanced self-attention mechanism modules is used to perform multi-level upsampling on the downsampled image of the last size in turn, and multiple upsampled images of different sizes are obtained, including: In the upsampling process at each level, the feature map size is restored through the upsampling module to obtain the second intermediate processed image; The spatial features and channel features of the second intermediate processing image are extracted through the multi-scale enhanced self-attention mechanism module in the multi-level upsampling network to obtain the up-sampled processing image corresponding to each level upsampling process, and the up-sampled processing image corresponding to each level upsampling process is jump-connected with the corresponding down-sampled processing image as the input feature map of the next level upsampling process, thereby realizing multi-level upsampling and obtaining multiple up-sampled processing images of different sizes; Among them, the upsampled processed image output by the last level upsampling process is used as the target image segmentation processing result.
5. The image processing method based on depth dynamic self-adjustment according to claim 4, characterized in that: The downsampled image and the upsampled image are semantically connected through multiple semantic mitigation modules with different depths, including: For the v The upsampled image of size and its corresponding downsampled image are processed with a depth of v The semantic relief module is skipped. in, v =1,2,…,J-1, where J represents the upper limit of the total number of layers in the multi-level upsampling network.
6. An image processing device based on dynamic self-adjustment of depth, applied to the image processing method based on dynamic self-adjustment of depth according to any one of claims 1 to 5, characterized in that: include: A target image acquisition module, a first sampling unit, a second sampling unit, a semantic connection module, and a target image segmentation processing result acquisition module; The target image acquisition module is used to acquire a target processed medical image; wherein the target processed medical image is used to characterize an image of a lesion site containing a tumor; The first sampling unit is used to perform multi-level image feature extraction on the target processed medical image through a multi-level image segmentation network composed of a lightweight deformable residual module and a multi-scale enhanced self-attention mechanism module, so as to dynamically fit edge contours and extract channel space features, and obtain down-sampled processed images of multiple sizes corresponding to the number of layers of the multi-level image segmentation network; The second sampling unit is used to sequentially perform multi-level upsampling on the down-sampled processed image of the last size through a multi-level upsampling network composed of a multi-scale enhanced self-attention mechanism module to obtain a plurality of up-sampled processed images of different sizes; wherein the multi-level upsampling network has the same number of layers as the multi-level image segmentation network; The semantic connection module is used to semantically connect the downsampled processed image and the upsampled processed image through multiple semantic relief modules with different depths during multiple upsampling processes, and adaptively adjust the semantic difference on the skip connections of different layers to generate the input processed image of the next size upsampling process using the upsampled processed image of the previous size; Among them, the down-sampled image of the u-th size corresponds to the up-sampled image of the v-th size, and when v =1,2,…,J-1, u =I-1,I-2,…,1; I represents the total number of layers of the multi-level image segmentation network, J represents the total number of layers of the multi-level upsampling network with an upper limit, I=J; The target image segmentation processing result acquisition module is used to replace the multi-scale enhanced self-attention mechanism module in the last size upsampling process with convolution to convert the upsampled processing image output by the penultimate size upsampling process into an output processing image to obtain the target image segmentation processing result.
Citation Information
Patent Citations
Dual-time remote sensing image semantic change detection method based on twin residual network
CN118429819A
Medical image segmentation method based on u-net
US20220309674A1
