A method and device for segmenting and recognizing multiple organs in medical images

Abstract

The present disclosure provides a method and device for multi-organ segmentation and recognition of medical images, which belongs to the field of image processing technology. The method includes: extracting features from medical images to obtain multi-layer local feature maps and global feature maps. Adaptive feature extraction is performed on the multi-layer local feature maps respectively, and semantic deepening is performed on the multi-layer local feature maps based on the features obtained by the adaptive feature extraction to obtain multi-layer semantically deepened feature maps. The multi-layer semantically deepened feature maps and the global feature maps are input into a first decoding network to obtain a first fused feature map. The first decoding network is a convolutional network for feature enhancement and feature fusion. The first fused feature map is input into a segmentation network to obtain a segmentation map of the medical image. The method and device for multi-organ segmentation and recognition of medical images provided by the present disclosure can improve the accuracy of medical image segmentation.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, to a method and device for segmenting and identifying multiple organs in medical images. Background Art

[0002] As an important part of computer-aided diagnosis system, medical image segmentation technology aims to accurately identify and separate structures or regions of interest, such as tumors, organs, blood vessels, etc. from medical images. The development of this technology is of great significance for clinical diagnosis, disease prediction and treatment planning, because it can help doctors better understand the patient's condition, thereby improving the accuracy and efficiency of medical decision-making. It plays an indispensable role in the quantitative and qualitative analysis of medical images.

[0003] The ability of network structures represented by U-Net (codec structure) to reconstruct high-definition feature maps using multi-level information in medical image segmentation has attracted widespread attention. In the U-Net++ family, dense and nested skip connections are used to bridge the semantic interval from the encoder to the decoder. However, the skip connection part mainly uses simple addition or cascade operations in the U-Net series of networks, without fully considering the semantic differences between the encoder and the decoder, resulting in blurred boundaries and inaccurate positions of segmented targets. Therefore, a medical image multi-organ segmentation solution is urgently needed to solve the above problems. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method and device for multi-organ segmentation and recognition of medical images to solve the problem of large semantic differences between the encoding end and the decoding end, and to achieve accurate segmentation of medical images.

[0005] A first aspect of the embodiments of the present disclosure provides a method for segmenting and recognizing multiple organs in a medical image, comprising:

[0006] Extract features from medical images to obtain multi-layer local feature maps and global feature maps;

[0007] Performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on features obtained by the adaptive feature extraction to obtain multiple layers of semantic deepening feature maps;

[0008] Inputting the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map; the first decoding network is a convolutional network for feature enhancement and feature fusion;

[0009] The first fused feature map is input into a segmentation network to obtain a segmentation map of the medical image.

[0010] A second aspect of the embodiments of the present disclosure provides a medical image multi-organ segmentation and recognition device, comprising:

[0011] A feature extraction module is used to extract features from medical images to obtain multi-layer local feature maps and global feature maps;

[0012] A semantic deepening module, used for performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on the features obtained by the adaptive feature extraction to obtain multiple layers of semantic deepening feature maps;

[0013] A feature fusion module, used for inputting the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map; the first decoding network is a convolutional network for feature enhancement and feature fusion;

[0014] A data output module is used to input the first fused feature map into a segmentation network to obtain a segmentation map of the medical image.

[0015] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned method for multi-organ segmentation and recognition in medical images when executing the computer program.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for multi-organ segmentation and recognition in medical images are implemented.

[0017] The beneficial effects of the medical image multi-organ segmentation and recognition method and device provided by the embodiments of the present disclosure are:

[0018] The disclosed embodiment performs adaptive feature extraction on the parts of interest in the shallow features (multi-layer local feature maps) and then performs semantic deepening on the multi-layer local feature maps. When this method is used instead of the traditional skip connection, the semantic gap when the shallow features are fused with the deep semantic features in the decoder is alleviated, and more applicable shallow information is provided for the deep semantic information, thereby alleviating the problem of inaccurate positioning and improving the accuracy of medical image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of a flow chart of a method for segmenting and recognizing multiple organs in a medical image provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of a medical image multi-organ segmentation and recognition process according to an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of adaptive feature extraction provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of feature enhancement provided for an embodiment of the present disclosure;

[0024] Figure 5 A graph showing the test results of the Synapse abdominal multi-organ segmentation dataset provided in one embodiment of the present disclosure;

[0025] Figure 6 ACDC heart segmentation dataset test result diagram provided by an embodiment of the present disclosure;

[0026] Figure 7 A structural block diagram of a medical image multi-organ segmentation and recognition device provided by an embodiment of the present disclosure;

[0027] Figure 8 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0029] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0030] Figure 1 A schematic diagram of a flow chart of a method for segmenting and recognizing multiple organs in a medical image provided by an embodiment of the present disclosure. Figure 2 A schematic diagram of a medical image multi-organ segmentation and recognition process according to an embodiment of the present disclosure is provided. Figure 1 and Figure 2 , the method comprising:

[0031] S101: Extract features from medical images to obtain multi-layer local feature maps and global feature maps.

[0032] In this embodiment, a hybrid encoder can be constructed first to extract features from medical images, where the medical images are preprocessed and annotated image data. A hybrid encoder can be constructed using convolutional layers and Transformer network layers, where the convolutional layer is the core component of a convolutional neural network, which is responsible for extracting features from input data (usually image data). The convolutional layer can be composed of multiple convolutional units, each of which contains a series of learnable weights (filters or convolution kernels) and bias items. These weights and bias items are optimized by a back propagation algorithm to effectively capture features in the input data. The convolutional layer can be used to extract local features of different scales from medical images to obtain a multi-layer local feature map.

[0033] The Transformer network layer is a deep learning architecture specifically designed for processing sequence data, especially natural language processing tasks. It can capture long-distance dependencies in the input sequence through self-attention and multi-head attention mechanisms, thereby more effectively understanding the semantic content of the sequence. The Transformer network layer can be used to extract global features of medical images and obtain global feature maps.

[0034] S102: performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on the features obtained by the adaptive feature extraction, so as to obtain multiple layers of semantic deepening feature maps.

[0035] In this embodiment, in order to deepen the semantic features of the multi-layer local feature map, adaptive feature extraction can be performed on the multi-layer local feature map respectively. The extraction process can be performed by constructing a hybrid pyramid adaptive attention network, through which the obtained adaptive features can be used to semantically deepen the multi-layer local feature map, thereby obtaining a multi-layer semantically deepened feature map. Among them, the hybrid pyramid adaptive attention network can perform adaptive feature extraction on the multi-layer local feature map from multiple dimensions, and the problem of inaccurate positioning can be alleviated by selecting useful adaptive features from different dimensions for fusion.

[0036] S103: Input the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map. The first decoding network is a convolutional network for feature enhancement and feature fusion.

[0037] In this embodiment, a cross-order multi-scale fusion decoder can be constructed for feature enhancement and feature fusion processes. The decoder can include a first decoding network. Based on the first decoding network, the multi-layer semantic deepening feature map obtained after semantic deepening can be fused with the feature map of the same scale in the network, and the newly fused multi-layer semantic deep feature map and the global feature map are combined and input into the first decoding network to obtain a first fused feature map. The first fused feature map can clearly and intuitively reflect the features and segmentation positions corresponding to the multiple organs contained in the medical image, thereby overcoming the problem of blurred image edges.

[0038] S104: Input the first fused feature map into a segmentation network to obtain a segmentation map of the medical image.

[0039] In this embodiment, the first fused feature map can be segmented based on a segmentation network, wherein the segmentation network is a deep learning model specifically used to divide an image into multiple parts. The first fused feature map can be input into the segmentation network, and the first fused feature map can be restored to the same size as the original medical image through operations such as sampling or deconvolution on the segmentation network, and the first fused feature map can be segmented into multiple images according to the features corresponding to different organs, thereby obtaining multiple segmentation maps of the medical image.

[0040] For example, the process of segmenting a medical image by a segmentation network can be expressed as: , where Pre represents the final generated segmentation map, Represents the first fused feature map of the input.

[0041] It can be concluded from the above that this embodiment can deepen the semantic features of multi-layer local feature maps, achieve accurate segmentation of medical images through the fusion and enhancement process with the global feature map, solve the problem of blurred image edges, improve the segmentation accuracy of medical images, promote the development of the field of medical image segmentation, and effectively improve the efficiency and accuracy of medical diagnosis.

[0042] In one embodiment of the present disclosure, adaptive feature extraction is performed on multiple layers of local feature maps respectively, and semantic deepening is performed on the multiple layers of local feature maps based on the features obtained by the adaptive feature extraction to obtain multiple layers of semantically deepened feature maps, including:

[0043] For any layer of local feature maps, multi-scale feature extraction is performed on the local feature maps to obtain a multi-scale feature map corresponding to the local feature maps of any layer.

[0044] The multi-scale feature maps corresponding to any layer of local feature maps are connected to obtain a first multi-scale feature map.

[0045] Adaptive feature extraction is performed on the first multi-scale feature map along the channel dimension and the spatial dimension in sequence to obtain a second multi-scale feature map.

[0046] The first multi-scale feature map and the second multi-scale feature map are added to obtain a third multi-scale feature map.

[0047] The third multi-scale feature map is multiplied by any layer of local feature map to obtain a semantic deepening feature map corresponding to any layer of local feature map.

[0048] In this embodiment, the process of performing semantic deepening on the multi-layer local feature map to obtain the multi-layer semantic deepening feature map can be described in detail as follows:

[0049] For semantic deepening of local feature maps of any layer, multi-scale feature extraction can be performed on the local feature maps of this layer first, which means that features can be extracted at different spatial scales (for example, through convolution kernels or pooling operations of different sizes) to capture information of different scales. This method helps the network capture various features from fine-grained to coarse-grained, thereby enhancing the representation ability of the model. Through the above operations, a multi-scale feature map corresponding to the local feature map of this layer can be obtained.

[0050] By connecting the multi-scale feature maps corresponding to the local feature maps of any layer, the feature maps of different scales can be stacked in the channel dimension to obtain a richer feature representation, namely the first multi-scale feature map. Adaptive feature extraction is performed on the connected first multi-scale feature map. This process can usually be achieved through some form of attention mechanism or adaptive convolution / pooling operation, so that the hybrid pyramid adaptive attention network can dynamically emphasize or suppress different features according to its learned weights.

[0051] The adaptive feature extraction process of the first multi-scale feature map can be performed along the channel dimension (i.e., across different feature maps) and the spatial dimension (i.e., at different positions of each feature map) respectively, so that more complex and subtle features can be captured in the hybrid pyramid adaptive attention network. The second multi-scale feature map can be obtained based on the adaptive features extracted along the channel dimension and the spatial dimension respectively.

[0052] The second multi-scale feature map after adaptive feature extraction is added to the original first multi-scale feature map. This residual connection helps to retain the information of the original features and allows the network to learn a smaller and easier to optimize transformation. The third multi-scale feature map obtained after addition is multiplied by the original local feature map of any layer. This operation can be regarded as a feature selection or emphasis because it weights the original feature map according to the value of the third multi-scale feature map. This helps to combine the original features with higher-level feature information to generate a feature map with more semantic information, that is, the semantic deepening feature map corresponding to the local feature map of this layer.

[0053] After the above operations, any layer of local feature map can be converted into a semantically deepened feature map with more semantic information. This process is performed on each layer of feature map in the entire hybrid pyramid adaptive attention network, thus generating a set of multi-layer semantically deepened feature maps.

[0054] From the above, it can be concluded that this embodiment enriches and deepens the semantic information of the local feature map through multi-scale feature extraction and adaptive feature extraction, thereby improving the performance of the entire hybrid pyramid adaptive attention network, thereby solving the problem of mismatch between the semantic information at the encoding end and the decoding end, and improving the accuracy of medical image segmentation.

[0055] In one embodiment of the present disclosure, for any layer of local feature maps, multi-scale feature extraction is performed on the local feature maps of any layer to obtain a multi-scale feature map corresponding to the local feature maps of any layer, including:

[0056] Based on multiple convolutional layers, multi-scale feature extraction is performed on any local feature map to obtain a multi-scale feature map corresponding to any local feature map. The expansion rates of multiple convolutional layers are different.

[0057] In this embodiment, the dilation rate is a parameter involved in the convolution operation, which determines the spacing between adjacent weights in the convolution kernel. When the dilation rate is greater than 1, the convolution kernel skips some input elements, resulting in an effect similar to operating on a larger scale.

[0058] By using different dilation rates, the convolutional layer can operate on different receptive fields, thereby capturing features of different scales. Smaller dilation rates correspond to smaller receptive fields, while larger dilation rates correspond to larger receptive fields.

[0059] The process of extracting multi-scale features from any layer of local feature maps to obtain multi-scale feature maps can be described in detail as follows:

[0060] For each layer of local feature maps, convolutional layers with different expansion rates can be used for each layer of local feature maps. Each convolutional layer can output a feature map, and these feature maps together constitute a multi-scale feature map corresponding to the local feature map of this layer.

[0061] For example, the first layer local feature map, the second layer local feature map, and the third layer local feature map can be processed by convolution with a convolution kernel size of 3×3 and expansion rates of 1, 3, and 5, respectively, which can be specifically expressed as: .in, The feature maps captured by the convolutional layers with different dilation rates are shown. represents a convolution kernel of size 3×3 and dilation rate i, Represents a local feature map of any layer.

[0062] It can be concluded from the above that this embodiment can perform multi-scale feature extraction through multiple convolutional layers with different expansion rates, which helps to improve the effectiveness of feature extraction of medical images at different scales, thereby improving the accuracy of the overall medical image segmentation process.

[0063] Figure 3 A schematic diagram of adaptive feature extraction provided by an embodiment of the present disclosure, referring to Figure 3 In one embodiment of the present disclosure, adaptive feature extraction is performed on the first multi-scale feature map along the channel dimension and the spatial dimension in sequence to obtain a second multi-scale feature map, including:

[0064] Global average pooling, convolution, and normalization are sequentially performed on the first multi-scale feature map along the channel dimension to determine a first scaling factor along the channel dimension.

[0065] A first adaptive feature is determined based on the first scaling factor. The first adaptive feature is an adaptive feature along a channel dimension.

[0066] The first adaptive feature is subjected to global average pooling, convolution and normalization processing in sequence along the spatial dimension to determine a second scaling factor along the spatial dimension.

[0067] A second multi-scale feature map is obtained based on the first adaptive feature and the second scaling factor.

[0068] In this embodiment, the process of performing adaptive feature extraction on the first multi-scale feature map along the channel dimension and the spatial dimension in sequence to obtain the second multi-scale feature map can be described in detail as follows:

[0069] The adaptive feature extraction along the channel dimension of the first multi-scale feature map may first connect all the features in the first multi-scale feature map. For example, the specific connection operation may be: , represents the first multi-scale feature map after connection, , and All are features in the first multi-scale feature map.

[0070] The first multi-scale feature map after connection is subjected to a global average pooling operation and compressed into a C×1×1 feature map along the channel dimension, expressed as .Will Perform convolution processing, and then pass through a fully connected layer and a nonlinear activation function ReLU to obtain ,Will Then, the first scaling factor along the channel dimension is obtained by passing through a fully connected layer and an activation function sigmoid layer and performing normalization to normalize each feature element to between [0, 1]. .Will and Multiply to get the first multi-scale feature map along the channel dimension , which can be specifically expressed as: .

[0071] The adaptive feature extraction process of the first adaptive feature along the spatial dimension is as follows: the adaptive feature along the channel dimension Through the global average pooling operation, it is compressed along the spatial dimension into a new feature map with a dimension of H×W×1, expressed as .Will Perform convolution and normalization operations, and then pass through a fully connected layer and an activation function sigmoid layer to obtain the second scaling factor along the spatial dimension. ,Will and Multiply to get the second multi-scale feature map of the spatial dimension , which can be specifically expressed as: .

[0072] In this embodiment, the first multi-scale feature map With the second multi-scale feature map Adding together can obtain enhanced features processed along the channel dimension and spatial dimension, that is, the third multi-scale feature map , which can be specifically expressed as: Multiplying the third multi-scale feature map with any layer of local feature map can obtain the semantic deepening feature map corresponding to any layer of local feature map, which can be specifically expressed as: ,in, Indicates the semantic deepening feature map of any layer of local feature map, Represents a local feature map of any layer.

[0073] It can be concluded from the above that this embodiment can realize the process of generating a second multi-scale feature map by performing adaptive feature extraction along the channel dimension and the spatial dimension. This method can enhance the adaptability and feature extraction performance of the hybrid pyramid adaptive attention network.

[0074] Figure 4 A schematic diagram of feature enhancement provided by an embodiment of the present disclosure, referring to Figure 2 and Figure 4 In one embodiment of the present disclosure, a multi-layer semantic deepening feature map and a global feature map are input into a first decoding network to obtain a first fused feature map, including:

[0075] Based on the first decoding network, the multi-layer semantic deepening feature map and the target feature map are feature fused to obtain a new multi-layer semantic deepening feature map. The target feature map is a feature map in the first decoding network corresponding to the scale of the multi-layer semantic deepening map.

[0076] The global feature map is reshaped and fused with the new multi-layer semantic deepening feature map in turn. The output of each feature fusion is the input of the next feature fusion to obtain the second fused feature map.

[0077] The output of each feature fusion is enhanced and then added to obtain the third fused feature map.

[0078] The third fused feature is subjected to convolution processing to obtain a fourth fused feature map.

[0079] The reshaped global feature map and the fourth fused feature map are fused to obtain a first fused feature map.

[0080] In this embodiment, the first decoding network can be a convolutional network layer in a cross-order multi-scale fusion decoder, which is used to perform feature enhancement and feature fusion processes. The process of inputting the multi-layer semantic deepening feature map and the global feature map into the first decoding network to obtain the first fused feature map can be described in detail as follows:

[0081] First, the first decoding network can fuse the multi-layer semantic deepening feature map with the target feature map, where the target feature map is the feature map in the network corresponding to the scale of the multi-layer semantic deepening feature map. By fusing the multi-layer semantic deepening feature map, a new multi-layer semantic deepening feature map can be obtained.

[0082] Secondly, the first decoding network can reshape the global features processed by the Transformer layer in the hybrid encoder into features of shape C×H×W, represented as , the reshaped features are enhanced, which can be specifically expressed as , where δ represents the ReLU activation function, Represents a convolution with a convolution kernel of 3. The reshaped features are upsampled using linear interpolation and represented as , and fuse the shallow features generated by the hybrid encoder, and perform feature fusion with the new multi-layer semantic deepening feature map in turn, and the output of each feature fusion is the input of the next feature fusion to obtain the second fused feature map .

[0083] The output of each feature fusion is enhanced and then added to obtain the third fusion feature map. The specific process can be expressed as:

[0084] ,in, represents the features after feature enhancement, Represents the third fused feature map after fusion.

[0085] The fourth fused feature map is obtained by connecting and convolving multiple features in the third fused feature map, which is specifically expressed as: , is the fourth fusion feature map. The reshaped global feature map And the fourth fusion feature map Add and fuse the features to get the first fused feature map for: .

[0086] From the above, it can be concluded that the present embodiment can generate a fused feature map that can capture multiple levels and global information in the input image by combining the information of multi-layer semantic deepening feature maps and global feature maps, and through multiple fusion and enhancement operations, thereby enhancing the feature extraction accuracy of medical images and improving the accuracy and applicability of the image segmentation process.

[0087] In one embodiment of the present disclosure, the multi-layer local feature map includes: a first local feature map, a second local feature map and a third local feature map.

[0088] The first local feature map is the first layer of the multi-layer local feature map, the third local feature map is the last layer of the multi-layer local feature map, and the second local feature map is a feature map between the first local feature map and the third local feature map.

[0089] Adaptive feature extraction is performed on the multi-layer local feature maps respectively, and semantic deepening is performed on the multi-layer local feature maps based on the features obtained by the adaptive feature extraction to obtain a multi-layer semantic deepening feature map, including:

[0090] Adaptive feature extraction is performed on the first local feature map and the third local feature map respectively.

[0091] Semantic deepening is performed on the first local feature map based on the features obtained by adaptive feature extraction to obtain a first semantically deepened feature map.

[0092] Semantic deepening is performed on the third local feature map based on the features obtained by adaptive feature extraction to obtain a third semantically deepened feature map.

[0093] The multi-layer semantic deepening feature map and the target feature map are fused to obtain a new multi-layer semantic deepening feature map, including:

[0094] The first semantic deepening feature map, the second local feature map, and the third semantic deepening feature map are respectively fused with the corresponding target feature map to obtain a new first semantic deepening feature map, a new second local feature map, and a new third semantic deepening feature map.

[0095] In this embodiment, the multi-layer local feature map may include: a first local feature map, a second local feature map and a third local feature map. Among them, the first local feature map may be the first layer of the multi-layer local feature map, the third local feature map may be the last layer of the multi-layer local feature map, and the second local feature map may be a feature map between the first local feature map and the third local feature map. Since the first and last layers of the multi-layer local feature map have the greatest impact on the accuracy of medical image segmentation, only the first local feature map and the third local feature map may be semantically deepened. The semantic deepening process for the intermediate layer may cause the running speed of the overall model to decrease, which is not conducive to improving the efficiency of medical image segmentation.

[0096] Based on the above description, adaptive feature extraction is performed on the multi-layer local feature maps respectively, and semantic deepening is performed on the multi-layer local feature maps based on the features obtained by adaptive feature extraction. The process of obtaining the multi-layer semantic deepening feature maps can be described in detail as follows:

[0097] It is only necessary to perform adaptive feature extraction on the first local feature map and the third local feature map respectively, and perform semantic deepening on the first local feature map according to the extracted features to obtain a first semantic deepening feature map. It is also necessary to perform semantic deepening on the third local feature map according to the extracted features to obtain a third semantic deepening feature map.

[0098] The process of fusing the multi-layer semantic deepening feature map and the target feature map to obtain a new multi-layer semantic deepening feature map can be described in detail as follows:

[0099] The first semantic deepening feature map, the third semantic deepening feature map and the second local feature map that have not undergone semantic deepening are respectively fused with their corresponding target feature maps to obtain a new first semantic deepening feature map, a new second local feature map and a new third semantic deepening feature map.

[0100] It can be concluded from the above that this embodiment can specifically perform semantic deepening on the feature layers that have a greater impact on the accuracy of medical image segmentation, simplify unnecessary semantic deepening processes of local feature maps, and improve processing speed.

[0101] In one embodiment of the present disclosure, the global feature map is reshaped and sequentially fused with the new multi-layer semantic deepening feature map, including:

[0102] After the global feature map is reshaped, it is fused with the new third semantic deepening feature map, the new second local feature map, and the new first semantic deepening feature map in sequence, and the output of each feature fusion is the input of the next feature fusion to obtain the second fused feature map.

[0103] In this embodiment, since not all local feature maps are semantically deepened, the global feature map can be reshaped according to the feature fusion step and then fused with the new third semantically deepened feature map, the new second local feature map, and the new first semantically deepened feature map in sequence. This method considers multi-level feature maps from shallow to deep, including global feature maps and semantically deepened feature maps of different levels, as well as local feature maps in the middle that have not been semantically deepened. This multi-level fusion helps capture rich information from local to global. The global feature map is reshaped so that it can be fused with local feature maps of different scales, which helps to maintain size consistency between feature maps during the fusion process.

[0104] After each feature fusion, the output can be feature enhanced, which helps to emphasize important features and suppress noise, thereby improving the quality of the fused features. Since the output of each feature fusion is used as the input of the next fusion, there is a cumulative effect, so that the features of each layer can be used by subsequent layers, which helps to extract deeper feature information. After obtaining the third fused feature map, it is convolved, which helps to further extract and integrate features and may improve the local structure of the feature map.

[0105] In this embodiment, in order to verify the effect of the medical image multi-organ segmentation and recognition method, the method is integrated into the first model for actual segmentation operations. For example, the overall network framework of the first model can be implemented in PyTorch, and the first model is trained on an NVIDIA RTX3090 GPU with 24GB of memory. The input medical image is adjusted to 224×224, the batch size is 24, and the pre-trained weight initialization using CNN-Transformer on ImageNet is used in the hybrid encoder. The SGD optimizer can be used for optimization, with an initial learning rate of 0.01, a momentum of 0.9, and a weight decay of 1e-4. During the training process, there is no overlap between the training data and the test data, and data enhancement methods such as flipping and rotation can also be used to enhance the quality of medical images.

[0106] Example 1: This example includes 3779 axial abdominal clinical CT images from 30 cases. Each CT volume has 85-198 slices of 512×512 pixels, and the voxel spatial resolution is ([0.54-0.54]×[0.98-0.98]×[2.5-5.0]) mm3. They are randomly divided into 18 training cases (2212 axial slices) and 12 validation cases. Combined with the segmentation example of the Synapse abdominal multi-organ CT image segmentation dataset, the Synapse abdominal multi-organ segmentation dataset test result diagram is obtained, as shown in the figure. Figure 5 shown.

[0107] In order to verify the overall performance of the first model, the segmentation results of the first model and 12 other different models using the above dataset can be compared, and the experimental results can be compared to prove the effectiveness of the model. The comparison results of different models of the Synapse abdominal multi-organ segmentation dataset are shown in Table 1.

[0108] Table 1 Comparison results of different models on the Synapse abdominal multi-organ segmentation dataset

[0109]

[0110] Example 2: This example includes a series of short-axis slices covering the heart from the base of the left ventricle to the apex, with a slice thickness of 5-8mm. The spatial resolution in the short-axis plane ranges from 0.83 to 1.75mm2 pixels. Each patient's scan is manually annotated with the true conditions of the left ventricle (LV), right ventricle (RV) and myocardium (MYO). It is randomly divided into 70 training sets (1930 axial slices) and 20 for testing. Combined with the segmentation example description of the cardiac MRI image segmentation dataset ACDC, the test result diagram of the ACDC heart segmentation dataset is obtained, as shown in the figure. Figure 6 The segmentation results of the first model and other 6 different models using the above datasets are shown in Table 2.

[0111] Table 2 Comparison results of different models on ACDC heart segmentation dataset

[0112]

[0113] It can be concluded from the above that this embodiment has the ability to capture rich information, and the practicability and high accuracy of this method are demonstrated through specific examples.

[0114] Corresponding to a medical image multi-organ segmentation and recognition method in the above embodiment, Figure 7 This is a block diagram of a medical image multi-organ segmentation and recognition device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7The medical image multi-organ segmentation and recognition device 20 includes: a feature extraction module 21, a semantic deepening module 22, a feature fusion module 23 and a data output module 24.

[0115] The feature extraction module 21 is used to extract features from medical images to obtain multi-layer local feature maps and global feature maps.

[0116] The semantic deepening module 22 is used to perform adaptive feature extraction on the multiple layers of local feature maps respectively, and perform semantic deepening on the multiple layers of local feature maps based on the features obtained by the adaptive feature extraction to obtain multiple layers of semantic deepening feature maps.

[0117] The feature fusion module 23 is used to input the multi-layer semantic deepening feature map and the global feature map into the first decoding network to obtain a first fused feature map. The first decoding network is a convolutional network used for feature enhancement and feature fusion.

[0118] The data output module 24 is used to input the first fused feature map into the segmentation network to obtain a segmentation map of the medical image.

[0119] In one embodiment of the present disclosure, the semantic deepening module 22 is specifically used for:

[0120] For any layer of local feature maps, multi-scale feature extraction is performed on the local feature maps to obtain a multi-scale feature map corresponding to the local feature maps of any layer.

[0121] The multi-scale feature maps corresponding to any layer of local feature maps are connected to obtain a first multi-scale feature map.

[0122] Adaptive feature extraction is performed on the first multi-scale feature map along the channel dimension and the spatial dimension in sequence to obtain a second multi-scale feature map.

[0123] The first multi-scale feature map and the second multi-scale feature map are added to obtain a third multi-scale feature map.

[0124] The third multi-scale feature map is multiplied by any layer of local feature map to obtain a semantic deepening feature map corresponding to any layer of local feature map.

[0125] In one embodiment of the present disclosure, the semantic deepening module 22 is specifically used for:

[0126] Based on multiple convolutional layers, multi-scale feature extraction is performed on any local feature map to obtain a multi-scale feature map corresponding to any local feature map. The expansion rates of multiple convolutional layers are different.

[0127] In one embodiment of the present disclosure, the semantic deepening module 22 is specifically used for:

[0128] Global average pooling, convolution, and normalization are sequentially performed on the first multi-scale feature map along the channel dimension to determine a first scaling factor along the channel dimension.

[0129] A first adaptive feature is determined based on the first scaling factor. The first adaptive feature is an adaptive feature along a channel dimension.

[0130] The first adaptive feature is subjected to global average pooling, convolution and normalization processing in sequence along the spatial dimension to determine a second scaling factor along the spatial dimension.

[0131] A second multi-scale feature map is obtained based on the first adaptive feature and the second scaling factor.

[0132] In one embodiment of the present disclosure, the feature fusion module 23 is specifically used to:

[0133] Based on the first decoding network, the multi-layer semantic deepening feature map and the target feature map are feature fused to obtain a new multi-layer semantic deepening feature map. The target feature map is a feature map in the first decoding network corresponding to the scale of the multi-layer semantic deepening map.

[0134] The global feature map is reshaped and fused with the new multi-layer semantic deepening feature map in turn. The output of each feature fusion is the input of the next feature fusion to obtain the second fused feature map.

[0135] The output of each feature fusion is enhanced and then added to obtain the third fused feature map.

[0136] The third fused feature is subjected to convolution processing to obtain a fourth fused feature map.

[0137] The reshaped global feature map and the fourth fused feature map are fused to obtain a first fused feature map.

[0138] In one embodiment of the present disclosure, the multi-layer local feature map includes: a first local feature map, a second local feature map and a third local feature map.

[0139] The first local feature map is the first layer of the multi-layer local feature map, the third local feature map is the last layer of the multi-layer local feature map, and the second local feature map is a feature map between the first local feature map and the third local feature map.

[0140] The semantic deepening module 22 is specifically used for:

[0141] Adaptive feature extraction is performed on the first local feature map and the third local feature map respectively.

[0142] Semantic deepening is performed on the first local feature map based on the features obtained by adaptive feature extraction to obtain a first semantically deepened feature map.

[0143] Semantic deepening is performed on the third local feature map based on the features obtained by adaptive feature extraction to obtain a third semantically deepened feature map.

[0144] The feature fusion module 23 is specifically used for:

[0145] The first semantic deepening feature map, the second local feature map, and the third semantic deepening feature map are respectively fused with the corresponding target feature map to obtain a new first semantic deepening feature map, a new second local feature map, and a new third semantic deepening feature map.

[0146] In one embodiment of the present disclosure, the feature fusion module 23 is specifically used to:

[0147] After the global feature map is reshaped, it is fused with the new third semantic deepening feature map, the new second local feature map, and the new first semantic deepening feature map in sequence, and the output of each feature fusion is the input of the next feature fusion to obtain the second fused feature map.

[0148] Reference Figure 8 , Figure 8 Schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 8 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 7 The functions of modules 21 to 24 are shown.

[0149] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0150] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0151] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0152] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of a medical image multi-organ segmentation and recognition method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0153] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0154] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0155] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0159] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for multi-organ segmentation and recognition in medical images, characterized in that: include: Extract features from medical images to obtain multi-layer local feature maps and global feature maps; Performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on features obtained by the adaptive feature extraction to obtain multiple layers of semantic deepening feature maps; Inputting the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map; the first decoding network is a convolutional network for feature enhancement and feature fusion; Inputting the first fused feature map into a segmentation network to obtain a segmentation map of the medical image; The method of performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on features obtained by the adaptive feature extraction to obtain multiple layers of semantically deepened feature maps, comprises: For any layer of local feature maps, multi-scale feature extraction is performed on the local feature map of any layer to obtain a multi-scale feature map corresponding to the local feature map of any layer; Connecting the multi-scale feature maps corresponding to any layer of local feature maps to obtain a first multi-scale feature map; Performing adaptive feature extraction on the first multi-scale feature map along the channel dimension and the spatial dimension in sequence to obtain a second multi-scale feature map; Adding the first multi-scale feature map and the second multi-scale feature map to obtain a third multi-scale feature map; Multiplying the third multi-scale feature map by the local feature map of any layer to obtain a semantic deepening feature map corresponding to the local feature map of any layer; The step of inputting the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map comprises: Based on the first decoding network, the multi-layer semantic deepening feature map and the target feature map are subjected to feature fusion to obtain a new multi-layer semantic deepening feature map; the target feature map is a feature map in the first decoding network corresponding to the scale of the multi-layer semantic deepening map; The global feature map is reshaped, and feature fused with the new multi-layer semantic deepening feature map in sequence, and the output of each feature fusion is the input of the next feature fusion, so as to obtain a second fused feature map; The output of each feature fusion is enhanced and then added to obtain the third fused feature map; The third fused feature is subjected to convolution processing to obtain a fourth fused feature map; The reshaped global feature map and the fourth fused feature map are subjected to feature fusion to obtain the first fused feature map.

2. A method for segmenting and identifying multiple organs in medical images as claimed in claim 1, characterized in that: For any layer of local feature maps, performing multi-scale feature extraction on the local feature maps of any layer to obtain a multi-scale feature map corresponding to the local feature maps of any layer, includes: Based on multiple convolutional layers, multi-scale feature extraction is performed on a local feature map of any layer to obtain a multi-scale feature map corresponding to the local feature map of any layer; the expansion rates of the multiple convolutional layers are different.

3. A medical image multi-organ segmentation and recognition method as claimed in claim 1, characterized in that: The step of performing adaptive feature extraction on the first multi-scale feature map in sequence along the channel dimension and the spatial dimension to obtain a second multi-scale feature map includes: Performing global average pooling, convolution, and normalization processing on the first multi-scale feature map in sequence along the channel dimension to determine a first scaling factor along the channel dimension; Determining a first adaptive feature based on the first scaling factor; the first adaptive feature is an adaptive feature along a channel dimension; Performing global average pooling, convolution, and normalization processing on the first adaptive feature in sequence along the spatial dimension to determine a second scaling factor along the spatial dimension; A second multi-scale feature map is obtained based on the first adaptive feature and the second scaling factor.

4. A method for segmenting and identifying multiple organs in medical images as claimed in claim 1, characterized in that: The multi-layer local feature map includes: a first local feature map, a second local feature map and a third local feature map; The first local feature map is the first layer of the multi-layer local feature map, the third local feature map is the last layer of the multi-layer local feature map, and the second local feature map is a feature map between the first local feature map and the third local feature map; The method of performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on features obtained by the adaptive feature extraction to obtain multiple layers of semantically deepened feature maps, comprises: Performing adaptive feature extraction on the first local feature map and the third local feature map respectively; Performing semantic deepening on the first local feature map based on the features obtained by adaptive feature extraction to obtain a first semantic deepening feature map; Performing semantic deepening on the third local feature map based on the features obtained by adaptive feature extraction to obtain a third semantic deepening feature map; The step of fusing the multi-layer semantic deepening feature map and the target feature map to obtain a new multi-layer semantic deepening feature map includes: The first semantic deepening feature map, the second local feature map, and the third semantic deepening feature map are respectively fused with the corresponding target feature map to obtain a new first semantic deepening feature map, a new second local feature map, and a new third semantic deepening feature map.

5. A method for segmenting and identifying multiple organs in medical images as claimed in claim 4, characterized in that: The reshaping of the global feature map and sequentially fusing the features with the new multi-layer semantic deepening feature map includes: After the global feature map is reshaped, feature fusion is performed with the new third semantic deepening feature map, the new second local feature map, and the new first semantic deepening feature map in sequence, and the output of each feature fusion is the input of the next feature fusion to obtain a second fused feature map.

6. A medical image multi-organ segmentation and recognition device, characterized in that: include: A feature extraction module is used to extract features from medical images to obtain multi-layer local feature maps and global feature maps; A semantic deepening module, used for performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on the features obtained by the adaptive feature extraction to obtain a multiple layer semantic deepening feature map; A feature fusion module, used for inputting the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map; the first decoding network is a convolutional network for feature enhancement and feature fusion; A data output module, used for inputting the first fused feature map into a segmentation network to obtain a segmentation map of the medical image; The method of performing adaptive feature extraction on the multiple layers of local feature maps respectively, and performing semantic deepening on the multiple layers of local feature maps based on features obtained by the adaptive feature extraction to obtain multiple layers of semantically deepened feature maps, comprises: For any layer of local feature maps, multi-scale feature extraction is performed on the local feature map of any layer to obtain a multi-scale feature map corresponding to the local feature map of any layer; Connecting the multi-scale feature maps corresponding to any layer of local feature maps to obtain a first multi-scale feature map; Performing adaptive feature extraction on the first multi-scale feature map along the channel dimension and the spatial dimension in sequence to obtain a second multi-scale feature map; Adding the first multi-scale feature map and the second multi-scale feature map to obtain a third multi-scale feature map; Multiplying the third multi-scale feature map by the local feature map of any layer to obtain a semantic deepening feature map corresponding to the local feature map of any layer; The step of inputting the multi-layer semantic deepening feature map and the global feature map into a first decoding network to obtain a first fused feature map comprises: Based on the first decoding network, the multi-layer semantic deepening feature map and the target feature map are subjected to feature fusion to obtain a new multi-layer semantic deepening feature map; the target feature map is a feature map in the first decoding network corresponding to the scale of the multi-layer semantic deepening map; The global feature map is reshaped, and feature fused with the new multi-layer semantic deepening feature map in sequence, and the output of each feature fusion is the input of the next feature fusion, so as to obtain a second fused feature map; The output of each feature fusion is enhanced and then added to obtain the third fused feature map; The third fused feature is subjected to convolution processing to obtain a fourth fused feature map; The reshaped global feature map and the fourth fused feature map are subjected to feature fusion to obtain the first fused feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images