Medical image small lesion segmentation method, storage medium and electronic device

Abstract

The application provides a medical image small lesion segmentation method, a storage medium and an electronic device, and the method comprises the following steps: constructing an image segmentation network model; inputting a medical image to be processed into an encoding module, extracting global features of the medical image at different scales to obtain feature maps of multiple different scales; inputting the feature maps of multiple different scales into a skip connection module, calculating the position relationship between different subblocks by using a position correlation network, constructing a position weight matrix to strengthen the attention degree of the organ region on the medical image; extracting deep features according to the feature map output by the last layer of the encoding module, inputting the extracted deep features into a decoding module, and performing scale restoration on the feature maps of various scales through upsampling, and fusing the feature maps of different scale restorations and the feature matrix output by the position correlation network to output a final feature map. Through the above scheme, the accuracy of lesion segmentation can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation, and in particular to a medical image small lesion segmentation method, a storage medium and an electronic device. BACKGROUND

[0002] The purpose of medical image segmentation is to clearly and effectively segment the organ and possible lesion area in the medical image to assist diagnosis, improve diagnosis efficiency and accuracy. It usually plays a key role in computer-aided diagnosis and intelligent medicine.

[0003] Small lesions are very common in the medical field, such as early tumor tissue. Because the tumor is small in the early stage, and has low differentiation degree with medical imaging noise, it is difficult to effectively segment it out. However, the successful segmentation of small lesion areas is of great significance to medical image segmentation.

[0004] At present, the segmentation method based on convolutional neural network has achieved great success in the field of medical image segmentation due to its excellent feature expression ability. However, due to the inherent locality of convolution operation, it usually shows deficiency when modeling long-term dependencies explicitly. Therefore, segmentation loss usually occurs when segmenting small targets. Thus, medical image segmentation of small lesions is still one of the most challenging tasks in computer vision. SUMMARY

[0005] Therefore, it is necessary to provide a technical scheme for medical image small lesion segmentation, so that small lesions in medical images can be segmented out faster, while reducing the false segmentation caused by imaging noise in non-organ regions.

[0006] To achieve the above purpose, in a first aspect, the present application provides a medical image small lesion segmentation method, the method comprising the following steps:

[0007] S1: constructing an image segmentation network model; the image segmentation network model comprises an encoding module, a decoding module and a skip connection module;

[0008] S2: inputting the medical image to be processed into the encoding module to extract the global features of the medical image at different scales to obtain a plurality of feature maps of different scales; the medical image is segmented into a plurality of sub-blocks;

[0009] S3: inputting the plurality of feature maps of different scales into the skip connection module, calculating the position relationship between different sub-blocks by using a position correlation network, and constructing a position weight matrix to enhance the attention degree of the organ region on the medical image;

[0010] S4: extracting deep layer features according to the feature map output by the last layer of the encoding module, inputting the extracted deep layer features into a decoding module, performing feature reconstruction on the deep layer features by convolution operation of the decoding module, performing scale restoration on feature maps of various scales by upsampling, fusing the feature maps of different scale restoration and the feature matrix output by the position correlation network, and outputting a final feature map.

[0011] Further, step S2 comprises:

[0012] S21: the encoding module divides the medical image to be processed into a plurality of non-overlapping SxSx3 sub-blocks, a total of sub-blocks, and maps to an arbitrary dimension C through a linear embedding layer to obtain feature maps of different scales.

[0013] S22: extracting global features through a plurality of layers of Swin-Transformer, performing 2x2 feature fusion on the extracted global features through a feature merging layer, and inputting the obtained block feature maps into the next layer of Swin-Transformer.

[0014] S23: repeating step S22 to obtain feature maps of three different scales.

[0015] Further, the feature map output by the last layer of the encoding module is feature map.

[0016] Further, step S4 comprises:

[0017] S41: reconstructing the feature map output by the encoder module using a sub-block reconstruction layer, and then inputting the feature map into the decoder module.

[0018] S42: performing scale restoration on the reconstructed feature map in the decoder module using upsampling and fusing the position correlation weighted feature matrix output by the position correlation network to obtain feature map input into a two-layer residual network structure for feature restoration.

[0019] S43: repeating step S42 three times to output a final feature map, and mapping the final feature map through a softmax function to obtain a final segmentation result map WxHxN, wherein N is the number of segmentation categories.

[0020] Further, the extracting deep layer features according to the feature map output by the last layer of the encoding module and inputting the extracted deep layer features into the decoding module comprises:

[0021] the encoding module reconstructs the reduced to a feature matrix of , where i ∈ {1, 2, 4}, and the reconstructed feature matrix is input into the decoding module.

[0022] Further, step S3 comprises:

[0023] S31: input I feature sub-blocks from the encoder module where f L = {x, y} is the position feature of the sub-block, f C is the image feature of the sub-block, the position weight matrix of the i-th sub-block and other sub-blocks can be calculated as follows:

[0024]

[0025] where V is a transformation matrix, H ij represents the position correlation matrix between the i-th sub-block and the j-th sub-block, which can be calculated as follows:

[0026]

[0027] where, is the feature matrix of sub-block i and j, which can be calculated as follows:

[0028]

[0029] is the position matrix of sub-block i and j, which can be calculated as follows:

[0030]

[0031]

[0032] where K and Q are transformation matrices, which are used to project the original features into a subspace to measure their matching degree, d k is the feature dimension of the subspace, and ε is the sine and cosine function of different wavelengths used to map from a low-dimensional space to a high-dimensional space.

[0033] S32: the feature map output by the position correlation network is the merged feature map of the original feature map and the position weight matrix, which can be calculated as follows:

[0034]

[0035] Further, the position weight matrix is configured to increase the feature weight of the organ region and suppress the feature weight of the non-organ region.

[0036] Further, the method further comprises:

[0037] S5: mapping the final feature map into a first label map through an activation function;

[0038] S6: obtaining a second label map, performing back optimization on parameters of the training model according to a weighted cross-entropy loss and a Dice loss of the first label map and the second label map, and repeating the above steps until the training model converges to determine parameters of an optimal training model; the second label map is a label map containing annotations.

[0039] In a second aspect, the present application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed to implement the method according to the first aspect of the present application.

[0040] In a third aspect, the present application further provides a server, comprising:

[0041] The storage medium is the storage medium according to the second aspect of the present application.

[0042] The processor is electrically connected with the storage medium, and is configured to execute the computer program stored in the storage medium to implement the method according to the first aspect of the present application.

[0043] Compared with the prior art, the present application has the following characteristics:

[0044] The present application provides a medical image small lesion segmentation method, a storage medium and an electronic device, the method comprising: constructing an image segmentation network model; inputting a medical image to be processed into an encoding module to extract global features of the medical image at different scales to obtain feature maps of different scales; inputting the feature maps of different scales into a skip connection module, calculating the position relationship between different sub-blocks by using a position correlation network, constructing a position weight matrix to strengthen the attention degree of the organ region on the medical image; extracting deep features according to the feature map output by the last layer of the encoding module, and inputting the extracted deep features into a decoding module; the decoding module reconstructs the deep features through convolution operation, and restores the feature maps of different scales through up-sampling, and fuses the feature maps of different scales restored and the feature matrix output by the position correlation network to output a final feature map. The above scheme can effectively improve the accuracy of lesion segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the medical image small lesion segmentation method according to the first embodiment of the present application;

[0046] Figure 2 The flowchart of the medical image small lesion segmentation method according to the second embodiment of the present application;

[0047] Figure 3A schematic diagram of a medical image lesion segmentation model according to an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a position correlation network according to an embodiment of the present application;

[0049] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the possible application scenarios, technical principles, specific schemes that can be implemented, purposes and effects that can be achieved of the present application clear, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0051] In this article, the term "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing at various positions in the specification does not necessarily refer to the same embodiment, and does not particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical solution.

[0052] Unless otherwise defined, the meanings of the technical terms used herein are the same as those commonly understood by those skilled in the art to which the present application belongs; the use of related terms in this article is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0053] In the description of the present application, the phrase "and / or" is a description of the logical relationship between objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally represents that the associated objects before and after are a "or" logical relationship.

[0054] In the present application, terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0055] In the present application, the terms "comprise", "contain", "have" or other similar terms used in the statements are intended to cover the non-exclusive inclusion, and these terms do not exclude the presence of other elements in the process, method or product comprising the elements, so that the process, method or product comprising a series of elements can not only include those limited elements, but also include other elements not explicitly listed, or also include the elements inherent to such process, method or product.

[0056] As the same understanding as in the "Guidelines for Examination", in the present application, the expressions "greater than", "less than", "exceed" and the like are understood as not including the number; the expressions "above", "below", "within" and the like are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times" and the like, unless otherwise explicitly specified.

[0057] Please refer to Figure 1 In a first aspect, the present application provides a medical image small lesion segmentation method, the method comprising the following steps:

[0058] S1: constructing an image segmentation network model; the image segmentation network model comprises an encoding module, a decoding module and a skip connection module;

[0059] S2: inputting a medical image to be processed into the encoding module, extracting global features of the medical image at different scales to obtain feature maps of multiple different scales; the medical image is segmented into multiple sub-blocks;

[0060] S3: inputting the feature maps of multiple different scales into the skip connection module, calculating the positional relationship between different sub-blocks using a position correlation network, constructing a position weight matrix to enhance the attention degree of the organ region on the medical image;

[0061] S4: extracting deep features according to the feature map output by the last layer of the encoding module, and inputting the extracted deep features into the decoding module; the decoding module reconstructs the deep features through convolution operation, and restores the feature maps of different scales through up-sampling; the feature maps of different scale restoration and the feature matrix output by the position correlation network are fused to output the final feature map.

[0062] Preferably, the encoding module of the present application comprises a four-layer Transformer structure, which can realize the global feature extraction of the medical image at different scales, thereby avoiding the limitations brought by the global feature modeling of the traditional convolutional neural network and the encoder composed of down-sampling.

[0063] The method provided by the scheme can fully consider the position correlation of a lesion and an organ to which the lesion belongs, can strengthen the attention in a neighborhood of the organ, can effectively improve the segmentation rate of small lesions, and can reduce false segmentation caused by imaging noise in a non-organ region.

[0064] As shown in Figure 3 , in some embodiments, step S2 includes:

[0065] S21: The encoding module divides the medical image to be processed into a plurality of non-overlapping sub-blocks of SxSx3, a total of sub-blocks, and maps to an arbitrary dimension C through a linear embedding layer to obtain feature maps of scales;

[0066] S22: Global features are extracted through a multi-layer Swin-Transformer, and the extracted global features are subjected to 2x2 feature fusion through a feature merging layer, and the obtained block feature maps are input into a next layer of Swin-Transformer;

[0067] S23: Step S22 is repeated to obtain feature maps of three different scales, respectively.

[0068] Preferably, the Swin-Transformer in step S22 is three layers, and the number of times of repeating steps S21-S22 in step S23 is three. The feature map output by the last layer of the encoding module is feature map. The position weight matrix is configured to increase the feature weight of the organ region and suppress the feature weight of the non-organ region.

[0069] Through the above scheme, the feature maps of three scales are input into a skip connection module and the position correlation network as shown in Figure 4 is used to calculate the position weighted feature maps thereof, and the feature map output by the encoding module is subjected to deep-level feature mapping through nine Swin-Transformers and then input into a decoder module.

[0070] Specifically, in some embodiments, step S4 includes:

[0071] S41: The feature map output by the encoder module is reconstructed through a sub-block reconstruction layer and then input into a decoder module;

[0072] S42: The reconstructed feature map is subjected to scale reduction through up-sampling in the decoder module and fused with the position correlation weighted feature matrix output by the position correlation network to obtain The feature maps are fed into a two-layer residual network structure for feature reconstruction;

[0073] S43: Repeat step S42 three times to output the final feature map. Map the final segmentation result map W×H×N using the softmax function, where N is the number of segmentation categories.

[0074] Preferably, the step of extracting deep features from the feature map output by the last layer of the encoding module and inputting the extracted deep features into the decoding module includes:

[0075] The encoding module will use the sub-block reconstruction layer to... Restore to The feature matrix is ​​obtained by reconstructing the feature matrix, where i∈{1,2,4}, and the reconstructed feature matrix is ​​input into the decoding module.

[0076] The location correlation module proposed in this invention is used to learn the location correlation between different sub-blocks in the feature map output by the Transformer encoding module. This enhances the focus on the region where large target organs are located and suppresses the focus on non-organ regions, thereby achieving accurate segmentation of small lesions around organs. This module can be executed concurrently with the Transformer of the encoding module, and its structure is shown in the attached figure. Figure 4 As shown.

[0077] In some embodiments, step S3 includes:

[0078] S31: Input I feature sub-blocks from the encoder module Where f L ={x, y} is the positional feature of the sub-block, f C If the image features are those of a sub-block, then the positional weight matrix of the i-th sub-block and other sub-blocks can be calculated as follows:

[0079]

[0080] Where V is the transformation matrix, H ij Let represent the positional correlation matrix between the i-th sub-block and the j-th sub-block, and its calculation is as follows:

[0081]

[0082] in, Let be the feature matrices of sub-blocks i and j, and its calculation is shown in the following formula:

[0083]

[0084] Let be the position matrix of sub-blocks i and j, which is calculated using the following formula:

[0085]

[0086]

[0087] wherein K and Q are transformation matrices, which function is to project the original features into the subspace to measure their matching degree, dk is the feature dimension of the subspace, and epsilon is the sine and cosine function of different wavelengths for mapping from the low-dimensional space to the high-dimensional space.

[0088] S32: The feature map output by the position correlation network is a combined feature map of the original feature map and the position weight matrix, and the calculation is as shown in the following formula:

[0089]

[0090] The decoder provided by the application is used for restoring image resolution and realizing pixel-level segmentation, and considering that the Transformer does not sufficiently focus on local information and is not good at restoring low-level features and pixels, a convolutional neural network and up-sampling are used in the decoder part to realize pixel restoration of the image.

[0091] As shown in Figure 2 , the method further comprises, after step S4:

[0092] S5: mapping the final feature map into a first label map through an activation function;

[0093] S6: obtaining a second label map, performing reverse optimization on the parameters of the training model according to the weighted cross-entropy loss and the Dice loss of the first label map and the second label map, and repeating the above steps until the training model converges, to determine the parameters of the optimal training model; the second label map is a label map containing annotations.

[0094] Through the above scheme, the label map finally output by the training model can accurately segment the organ and lesion area in the medical image.

[0095] In a second aspect, the application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed to implement the method in the first aspect of the application.

[0096] In a third aspect, as Figure 5 shown, the application further provides an electronic device 10, comprising a storage medium 102 and a processor 101, the storage medium 102 is the storage medium in the second aspect, and the processor 101 is electrically connected with the storage medium 102 and is used for executing the computer program stored in the storage medium to implement the method in the first aspect.

[0097] ​Preferably, the electronic device can be a computer device, including but not limited to: personal computer, server, general-purpose computer, special-purpose computer, network device, embedded device, programmable device, smart mobile terminal, smart home device, wearable smart device, vehicle-mounted smart device, etc. The storage medium includes but is not limited to: RAM, ROM, magnetic disk, magnetic tape, optical disk, flash memory, U disk, mobile hard disk, memory card, memory stick, network server storage, network cloud storage, etc. The processor includes but is not limited to CPU (central processing unit), GPU (graphics processor), MCU (microprocessor), etc.

[0098] Although the above-mentioned embodiments have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic creative concept, so the above is only an embodiment of the present application, and does not limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for segmenting small lesions in medical images, characterized in that, The method includes the following steps: S1: Construct an image segmentation network model; the image segmentation network model includes an encoding module, a decoding module, and a skip connection module; S2: Input the medical image to be processed into the encoding module, extract global features of the medical image at different scales, and obtain feature maps of various scales; the medical image is segmented into multiple sub-blocks; S3: Input the feature maps of different scales into the skip connection module, use a positional correlation network to calculate the positional relationship between different sub-blocks, and construct a positional weight matrix to enhance the attention to organ regions on medical images; S4: Extract deep features from the feature map output by the last layer of the encoding module, and input the extracted deep features into the decoding module. The decoding module reconstructs the deep features through convolution operation, and performs scale restoration on the feature maps at each scale through upsampling. The feature maps restored at different scales are fused with the feature matrix output by the position correlation network to output the final feature map. Step S2 includes: S21: The encoding module segments the medical image to be processed into several non-overlapping segments. The sub-blocks, in total, yielded Each sub-block is mapped to an arbitrary dimension C through a linear embedding layer, resulting in... Feature maps at different scales; S22: Global features are extracted using a multi-layer Swin-Transformer. The extracted global features are then fused using a 2x2 feature merging layer to obtain the desired feature output. The block feature map is fed into the next layer, Swin-Transformer; S23: Repeat step S22 to obtain the following results. , , Feature maps at three different scales; Step S4 includes: S41: Output of the encoder module The feature map is reconstructed using a sub-block reconstruction layer and then fed into the decoder module; S42: The reconstructed feature map is upsampled and scaled in the decoder module, and then fused with the position-related weighted feature matrix output by the position-related network to obtain the final feature map. The feature maps are fed into a two-layer residual network structure for feature reconstruction; S43: Repeat step S42 three times, output the final feature map, and use the softmax function to map it to obtain the final segmentation result map. , where N is the number of segmentation categories.

2. The method for segmenting small lesions in medical images as described in claim 1, characterized in that, The feature map output by the last layer of the encoding module is Feature map.

3. The method for segmenting small lesions in medical images as described in claim 1, characterized in that, The step of extracting deep features from the feature map output by the last layer of the encoding module and inputting the extracted deep features into the decoding module includes: The encoding module will use the sub-block reconstruction layer to... Restore to The feature matrix of , where The reconstructed feature matrix is ​​then input into the decoding module.

4. The method for segmenting small lesions in medical images as described in claim 1, characterized in that, Step S3 includes: S31: Input I feature sub-blocks from the encoder module ,in The positional features of the sub-block If it is an image feature of a sub-block, then the first... i The positional weight matrix of each sub-block and its relative sub-blocks can be calculated as follows: ； Where V is the transformation matrix. Indicates the first i The sub-block and the first j The positional correlation matrix between the sub-blocks is calculated using the following formula: ； in, It is a sub-block i and j The characteristic matrix of is calculated as shown in the following formula: ； It is a sub-block i and j The position matrix is ​​calculated using the following formula: ； ； Where K and Q are transformation matrices, their function is to transform the original features , Projected onto subspace for measurement The degree of matching, For the feature dimension of the subspace, Sine and cosine functions of different wavelengths are used to map from low-dimensional space to high-dimensional space; S32: The feature map output by the location-related network is a merged feature map of the original feature map and the location weight matrix, and its calculation is shown in the formula below: 。 5. The method for segmenting small lesions in medical images as described in claim 4, characterized in that, The location weight matrix is ​​configured to increase the feature weights of organ regions and suppress the feature weights of non-organ regions.

6. The method for segmenting small lesions in medical images as described in claim 1, characterized in that, The method further includes: S5: The final feature map is mapped to the first label map through an activation function; S6: Obtain the second label map, perform reverse optimization on the parameters of the training model based on the weighted cross-entropy loss and Dice loss of the first label map and the second label map, and repeat the above steps until the training model converges to determine the parameters of the optimal training model; the second label map is a label map containing annotations.

7. A storage medium, characterized in that, The storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: The storage medium is the storage medium as described in claim 7; A processor, electrically connected to the storage medium, is configured to execute a computer program stored in the storage medium to implement the method as described in any one of claims 1 to 6.

Images