Image segmentation method and apparatus

Abstract

The application provides an image segmentation method and device, the image segmentation method comprises the following steps: obtaining a DICOM image to be segmented; performing feature extraction on the DICOM image based on a multi-layer perception machine, obtaining a plurality of sampling feature maps, and performing convolution operation on a target sampling feature map to obtain a multi-scale feature map; and obtaining a segmentation image based on the sampling feature map with the highest image resolution and the multi-scale feature map. The method can obtain the width information and height information of the image by obtaining a plurality of sampling feature maps of the DICOM image through the multi-layer perception machine, and can obtain the abstract information of the image by performing convolution operation on the target sampling feature map to obtain the multi-scale feature map. Finally, the sampling feature map with the highest image resolution and the multi-scale feature map are fused to realize the splicing of the feature information of different dimensions of the image, and the segmentation image is obtained according to the fused feature map, thereby improving the image segmentation accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to an image segmentation method and device. BACKGROUND

[0002] The manner of human hip joint lesions is diverse, such as fracture, arthritis, revision, etc., and accurately obtaining the hip joint lesion is of great significance to the treatment of hip joint lesions.

[0003] In the related art, AI technology is usually used to obtain joint lesion information in a hip joint image, but since the type of hip joint lesions is complex, the AI technology obtains global information of the joint lesion site, and the representation ability of the edge details of the joint lesion information is weak, resulting in low accuracy of hip joint image segmentation. SUMMARY

[0004] The present application provides an image segmentation method and device to solve the problem of the prior art that the representation ability of the edge details of the joint lesion information is weak when using AI technology to obtain global information in a hip joint image, resulting in low accuracy of hip joint image segmentation, and improves the accuracy of hip joint image segmentation.

[0005] The present application provides an image segmentation method, comprising:

[0006] obtaining a DICOM image to be segmented;

[0007] based on a multi-layer perception machine, performing feature extraction on the DICOM image to obtain a plurality of sampling feature maps, and performing convolution operation on a target sampling feature map to obtain a multi-scale feature map, the plurality of sampling feature maps including height information and width information of the DICOM image, and the target sampling feature map being other sampling feature maps in the plurality of sampling feature maps except for a sampling feature map with the highest image resolution;

[0008] based on the sampling feature map with the highest image resolution and the multi-scale feature map, obtaining a segmentation image.

[0009] According to the image segmentation method provided by the present application, the sampling feature map includes an up-sampling feature map and a down-sampling feature map, the multi-layer perception machine includes n encoding layers, n decoding layers and one transition layer, the input of the transition layer is connected with the nth encoding layer, the output of the transition layer is connected with the nth decoding layer, the ith encoding layer and the ith decoding layer are connected by a skip connection, n is a natural number greater than 1, and 1≤i≤n;

[0010] The feature extraction on the DICOM image based on the multi-layer perception machine to obtain a plurality of sampling feature maps comprises:

[0011] Input the DICOM image into a first encoding layer for downsampling to obtain a first down-sampling feature map;

[0012] Input the first down-sampling feature map into a second encoding layer for downsampling to obtain a second down-sampling feature map, until an nth down-sampling feature map is input into a transition layer for downsampling to obtain an (n+1)th down-sampling feature map;

[0013] Input the (n+1)th down-sampling feature map into an nth decoding layer for up-sampling to obtain a first up-sampling image, until an (n-1)th up-sampling feature map is input into a first decoding layer for downsampling to obtain an nth up-sampling feature map.

[0014] According to the image segmentation method provided by the application, the sampling feature map includes an up-sampling feature map and a down-sampling feature map, the multi-layer perception includes n encoding layers, n decoding layers, 1 transition layer and n convolution layers, the input of the transition layer is connected with the nth encoding layer, the output of the transition layer is connected with the nth decoding layer, the ith encoding layer and the ith decoding layer are connected by a skip connection, n is a natural number greater than 1, and 1≤i≤n;

[0015] The convolution operation on the target sampling feature map to obtain a multi-scale feature map includes:

[0016] The nth up-sampling feature map is determined as the sampling feature map with the highest image resolution;

[0017] The (n+1)th down-sampling feature map, the first up-sampling image and the (n-1)th up-sampling image are input into the convolution layer respectively to obtain n convolution feature maps;

[0018] The nth up-sampling feature map and the n convolution feature maps are subjected to a Concat operation to obtain the multi-scale feature image;

[0019] The nth up-sampling feature map is obtained by sequentially performing n+1 encoding and n decoding on the DICOM image, the (n+1)th down-sampling feature map is obtained by sequentially performing n+1 downsampling on the DICOM image, the first up-sampling image is obtained by sequentially performing n+1 downsampling and 1 up-sampling on the DICOM image, the (n-1)th up-sampling image is obtained by sequentially performing n+1 downsampling and n-1 up-sampling on the DICOM image, and the nth up-sampling feature map is obtained by sequentially performing n+1 downsampling and n up-sampling on the DICOM image.

[0020] According to the image segmentation method provided by the application, the multi-layer perception includes a Width channel and a Height channel;

[0021] The multi-layer perceptron performs feature extraction on the DICOM image to obtain a plurality of sampling feature maps, and further comprises:

[0022] A first feature map of the DICOM image is obtained.

[0023] The feature map of the DICOM image is input to a Width channel to perform MLP operation to obtain a second feature map.

[0024] The second feature map is input to a Height channel to perform MLP operation to obtain a third feature map.

[0025] The first feature map and the third feature map are superimposed to obtain the plurality of sampling feature maps.

[0026] According to the image segmentation method provided by the application, the multi-layer perceptron comprises a feature fusion layer, the loss function of the feature fusion layer is a first joint loss function, and the first joint loss function is obtained based on a cross-entropy loss function and a Dice loss function.

[0027] The highest-resolution sampling feature map and the multi-scale feature map are input to the feature fusion layer to obtain a fusion feature map.

[0028] The highest-resolution sampling feature map and the multi-scale feature map are input to the feature fusion layer to obtain a fusion feature map.

[0029] The fusion feature map is obtained based on the fusion feature map.

[0030] According to the image segmentation method provided by the application, the loss function corresponding to the n decoders is a second joint loss function, and the second joint loss function is obtained based on a cross-entropy loss function and a Dice loss function.

[0031] The n+1 down-sampling feature map is input to the n decoding layer to perform up-sampling to obtain a first up-sampling image, and the n-1 up-sampling feature map is input to the first decoding layer to perform down-sampling to obtain the n up-sampling feature map.

[0032] The n+1 down-sampling feature map is input to the n decoding layer to perform up-sampling to obtain a first up-sampling image, and the n-1 up-sampling feature map is input to the first decoding layer to perform down-sampling to obtain the n up-sampling feature map.

[0033] The application further provides an image segmentation device, comprising:

[0034] An image acquisition module is configured to acquire a DICOM image to be segmented.

[0035] A feature fusion module is configured to perform feature extraction on the DICOM image based on a multi-layer perception machine, to obtain a plurality of sampling feature maps, and to perform convolution operation on a target sampling feature map to obtain a multi-scale feature map, wherein the plurality of sampling feature maps include height information and width information of the DICOM image, and the target sampling feature map is one of the plurality of sampling feature maps other than a sampling feature map with the highest image resolution.

[0036] An image segmentation module is configured to obtain a segmented image based on the sampling feature map with the highest image resolution and the multi-scale feature map.

[0037] The present application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image segmentation method according to any one of the above when executing the program.

[0038] The present application also provides a non-transitory computer-readable storage medium, which stores a computer program executable by a processor to implement the image segmentation method according to any one of the above.

[0039] The present application also provides a computer program product, which includes a computer program executable by a processor to implement the image segmentation method according to any one of the above.

[0040] The image segmentation method and device provided by the present application can obtain a plurality of sampling feature maps of a DICOM image through a multi-layer perception machine, can obtain width information and height information of the image, can perform convolution operation on a target sampling feature map to obtain a multi-scale feature map, can obtain abstract information of the image, can perform feature fusion on a sampling feature map with the highest image resolution and the multi-scale feature map, can splice feature information of different dimensions of the image, and can obtain a segmented image according to the fused feature map, thereby improving the image segmentation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0042] Figure 1 is one of the flowcharts of the image segmentation method provided by the present application;

[0043] Figure 2is a structural schematic diagram of a multi-layer perceptron provided by the application;

[0044] Figure 3 is an interface schematic diagram of the MLP Shift Block for convolution of input features provided by the application;

[0045] Figure 4 is a second flow schematic diagram of the image segmentation method provided by the application;

[0046] Figure 5 is an interface schematic diagram of the MLP Block for feature extraction provided by the application;

[0047] Figure 6 is a structural schematic diagram of the image segmentation device provided by the application;

[0048] Figure 7 is an entity structural schematic diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0050] The image segmentation method and device of the present application will be described below. Figures 1-6

[0051] Figure 1 is a first flow schematic diagram of the image segmentation method provided by the application, and the image segmentation method comprises the following steps:

[0052] Step 110, acquiring a DICOM image to be segmented.

[0053] In this step, the DICOM image to be segmented is a human hip joint image, including a lesion part of the human hip joint, and the DICOM image is a kind of 2.5D image, which fully considers the corresponding relationship between different slices of CT images.

[0054] In this embodiment, the DICOM image can be read by software or by using the Python library PyDicom.

[0055] ​In some embodiments, in an image segmentation application scenario with high image resolution requirements, the DICOM image can be directly used as input data, and in an image segmentation application scenario with relatively low image resolution requirements, the DICOM image can be first down-sampled to obtain a down-sampled image of the DICOM image as input data.

[0056] In step 120, feature extraction is performed on the DICOM image based on the multilayer perceptron to obtain a plurality of sampling feature maps, and a target sampling feature map is subjected to convolution operation to obtain a multi-scale feature map. The plurality of sampling feature maps include height information and width information of the DICOM image, and the target sampling feature map is other sampling feature map in the plurality of sampling feature maps except for a sampling feature map with the highest image resolution.

[0057] In this step, the multilayer perceptron (MLP) can encode and decode the input DICOM image, and a new feature map can be obtained in each encoding or decoding process.

[0058] In this embodiment, the multilayer perceptron includes a plurality of encoding layers and a corresponding number of decoding layers. The input data of each layer in the plurality of encoding layers is the output data of the previous encoding layer. For example, the multilayer perceptron includes a 5-layer structure, including 2 encoding layers, 1 transition layer, and 2 decoding layers. The DICOM image is input to the first encoding layer for encoding to obtain the first feature map. The first feature map is input to the second encoding layer to obtain the second feature map. The second feature map is input to the transition layer to obtain the third feature map. The third feature map is input to the first decoding layer to obtain the fourth feature map. The fourth feature map is input to the second decoding layer to obtain the fifth feature map. The fifth feature map is a feature map obtained by 3 up-sampling and 2 down-sampling in succession.

[0059] In this embodiment, since the process of layer-by-layer encoding is also a process of down-sampling the feature map, the size of the corresponding feature map gradually decreases when the DICOM image is encoded in the plurality of encoding layers. For example, the pixel size of the first feature map is 256*256. After being input to the second encoding layer for pooling, the size of the second feature map obtained is 128*128. Correspondingly, the size of the third feature map is 64*64.

[0060] In this embodiment, since the process of layer-by-layer decoding is also a process of up-sampling the feature map, the size of the corresponding feature map gradually increases when the DICOM image is decoded in the plurality of decoding layers. For example, after the third feature map is input to the first decoding layer, the size of the fourth feature map obtained is 128*128. Correspondingly, the size of the fifth feature map is 256*256.

[0061] In this embodiment, the feature map with the highest image resolution corresponds to the feature image obtained when first encoding or last decoding, for example, the feature map with the highest image resolution is the 5th feature map, and the resolution of the corresponding feature map is 256*256.

[0062] In this embodiment, the target sampling feature map can be other feature maps in addition to the feature map with the highest resolution, for example, the target sampling feature map can be the 3rd and 4th feature maps in the above embodiment.

[0063] In this embodiment, the multi-layer perception includes an MLP Block and an MLP Shift Block, wherein the MLP Block is used to encode or decode input image data, and the MLP Shift Block is used to perform convolution operation on the decoded feature map and obtain the feature degree after convolution as a multi-scale feature map.

[0064] Step 130, based on the sampling feature map with the highest image resolution and the multi-scale feature map, obtain a segmentation image.

[0065] In this step, the sampling feature map with the highest image resolution includes global information and edge information of the DICOM image, on the one hand, it can be directly restored to the size of the DICOM image through upsampling and obtain a segmentation image; on the other hand, the sampling feature map with the highest image resolution can be fused with the multi-scale feature map obtained by convolution calculation to obtain a new feature degree, and the new feature map has stronger image content representation ability, and thus the obtained image segmentation effect is better.

[0066] In this embodiment, if the feature map with the highest image resolution is the 5th feature map, the resolution of the corresponding feature map is 256*256, and the multi-scale feature map is the feature map obtained by performing convolution operation on the 3rd and 4th feature maps through the MLP Shift Block, and the 3rd, 4th and 5th feature maps are fused to obtain a fused feature map, which can be used to obtain a segmentation image.

[0067] The image segmentation method provided by the embodiment of the application can obtain a plurality of sampling feature maps of a DICOM image through a multi-layer perception, can obtain width information and height information of the image, can perform convolution operation on a target sampling feature map to obtain a multi-scale feature map, can obtain abstract information of the image, and finally can fuse the sampling feature map with the highest image resolution and the multi-scale feature map to realize feature information splicing of different dimensions of the image, and can obtain a segmentation image according to the fused feature map, thereby improving the image segmentation accuracy.

[0068] In some embodiments, the sampling feature map includes an up-sampling feature map and a down-sampling feature map, the multi-layer perception includes n encoding layers, n decoding layers and 1 transition layer, the input of the transition layer is connected with the n th encoding layer, the output of the transition layer is connected with the n th decoding layer, the i th encoding layer and the i th decoding layer are connected by a skip connection, n is a natural number greater than 1, and 1≤i≤n; based on the multi-layer perception, feature extraction is performed on the DICOM image to obtain a plurality of sampling feature maps, including: inputting the DICOM image into the 1 st encoding layer for down-sampling to obtain a first down-sampling feature map; inputting the first down-sampling feature map into the 2 nd encoding layer for down-sampling to obtain a second down-sampling feature map, until inputting the n th sampling feature map into the transition layer for down-sampling to obtain an n+1 th down-sampling feature map; inputting the n+1 th down-sampling feature map into the n th decoding layer for up-sampling to obtain a first up-sampling image, until inputting the n-1 th up-sampling feature map into the 1 st decoding layer for down-sampling to obtain an n th up-sampling feature map.

[0069] In this embodiment, the multi-layer perception can be an image segmentation model composed of a plurality of encoding layers, a transition layer and a plurality of decoding layers, wherein the encoding layers and the decoding layers are both used to extract feature information in the width and height directions from the input data, the transition layer can be regarded as an encoding layer, connected after the last encoding layer of the plurality of encoding layers and before the first decoding layer of the plurality of decoding layers.

[0070] Figure 2 is a structural schematic diagram of the multi-layer perception provided by the application, in Figure 2 In the embodiment shown, n=3, the multi-layer perception includes 3 encoding layers (corresponding to E1, E2 and E3 respectively), 3 decoding layers (corresponding to D1, D2 and D3 respectively), and a transition layer (corresponding to E4), each decoding layer and each encoding layer are connected by a skip connection, the process of input data through the 3 encoding layers and the transition layer is an image down-sampling process, the size of the obtained down-sampling feature map decreases successively, the process of input data through the 3 decoding layers is an image up-sampling process, the size of the obtained up-sampling feature map increases successively, and the pixel size of the up-sampling feature map output by the decoding layer D1 is adjusted to the original image size by up-sampling.

[0071] In this embodiment, the image down-sampling can reduce the image feature dimension while retaining the height information and width information of the image, and reduce the parameter calculation amount, and the image up-sampling can restore the feature map containing important feature information to the original image size for display on a display device.

[0072] The image segmentation method provided by the embodiment of the present application reduces the parameter calculation amount through n+1 times of downsampling and n times of upsampling of the input image by setting n encoding layers, n decoding layers and 1 transition layer, and each encoding layer and decoding layer is connected by jumping, each encoding layer and decoding layer can extract the width feature and height feature of the image, the final obtained upsampled image contains the global information and edge information of the image, the extraction of the fine-grained features in the upsampled feature map is realized, the representation ability of the image features is improved, and data support is provided for the subsequent multi-scale information fusion process.

[0073] In some embodiments, the sampling feature map includes an upsampled feature map and a downsampled feature map, the multi-layer perception includes n encoding layers, n decoding layers, 1 transition layer and n convolution layers, the input of the transition layer is connected with the n th encoding layer, the output of the transition layer is connected with the n th decoding layer, the i th encoding layer and the i th decoding layer are connected by jumping, n is a natural number greater than 1, and 1≤i≤n; the target sampling feature map is subjected to a convolution operation to obtain a multi-scale feature map, including: determining the n th upsampled feature map as the sampling feature map with the highest image resolution; inputting the n+1 th downsampled feature map, the first upsampled image to the n-1 th upsampled image into the convolution layers respectively to obtain n convolution feature maps; and performing a Concat operation on the n th upsampled feature map and the n convolution feature maps to obtain a multi-scale feature image; wherein the n th upsampled feature map is obtained by sequentially performing n+1 times of encoding and n times of decoding on a DICOM image, the n+1 th downsampled feature map is obtained by sequentially performing n+1 times of downsampling on the DICOM image, the first upsampled image is obtained by sequentially performing n+1 times of downsampling and 1 time of upsampling on the DICOM image, the n-1 th upsampled image is obtained by sequentially performing n+1 times of downsampling and n-1 times of upsampling on the DICOM image, and the n th upsampled feature map is obtained by sequentially performing n+1 times of downsampling and n times of upsampling on the DICOM image.

[0074] In this embodiment, the multi-layer perception can be composed of multiple encoding layers, multiple decoding layers and multiple convolution layers, wherein the convolution layer is used to perform a convolution operation on the feature map output by the decoding layer to obtain a multi-scale feature map.

[0075] In Figure 2 In the embodiment shown in FIG. 4, the sampling feature maps output by the decoding layer D2, the decoding layer D3 and the transition layer E4 are subjected to a convolution operation by the MLP Shift Block to obtain three convolution feature maps, and the three convolution feature maps are subjected to a Concat operation to obtain a multi-scale feature image.

[0076] In this embodiment, the convolution operation on the upsampled images output by the multiple decoding layers is completed by the MLP Shift Block.

[0077] Figure 3 is an interface diagram of the MLP Shift Block provided by the application for convolution of input features, in which Figure 3 In the embodiment shown, each decoding layer can be converted by the MLP to obtain a feature map corresponding to the decoding layer; first, the MLP structure is converted into a convolution layer, then the input feature map is up-sampled by the convolution layer to obtain an up-sampled feature map (corresponding to A), and the up-sampled feature map is subjected to a Conv+BN+Relu operation to complete the channel information fusion and generate a feature map in the transition stage (corresponding to B), and the feature map is up-sampled (Upsampling) to generate a segmentation feature map with the same size as the original image.

[0078] Figure 4 is a second flowchart of the image segmentation method provided by the application, in which Figure 4 In the embodiment shown, first, a DICOM image is obtained, and the DICOM image is input into a multilayer perceptron (MLP) to sequentially perform MLP encoding and MLP decoding, and the up-sampled feature maps output by the transition layer and the plurality of decoding layers are subjected to a Concat operation to obtain an image segmentation result.

[0079] The image segmentation method provided by the embodiment of the application sets a convolution layer in the multilayer perceptron, and performs convolution calculation on the up-sampled feature maps output by the plurality of decoding layers to obtain a plurality of convolution feature maps, and performs a Concat operation on the plurality of convolution feature maps to obtain a multi-scale feature image, which realizes the extraction of coarse-grained feature information in the plurality of up-sampled images, improves the representation ability of image features, and provides new feature information for the subsequent multi-scale information fusion process.

[0080] In some embodiments, the multilayer perceptron includes a Width channel and a Height channel; based on the multilayer perceptron, the DICOM image is subjected to feature extraction to obtain a plurality of up-sampled feature maps, and further includes: obtaining a first feature map of the DICOM image; inputting the feature map of the DICOM image into the Width channel to perform MLP operation to obtain a second feature map; inputting the second feature map into the Height channel to perform MLP operation to obtain a third feature map; and superimposing the first feature map and the third feature map to obtain a plurality of up-sampled feature maps.

[0081] In this embodiment, the decoding layer and the encoding layer of the multilayer perceptron both use the MLP Block module, and the MLP Block module is used to extract Width channel information and Width channel information in the image.

[0082] Figure 5 is an interface diagram of the MLP Block for extracting features, in which Figure 5In the illustrated embodiment, the feature map A is input into the MLP Block module, the MLP operation is performed on the feature map A in the Width channel, then after Layer Normalization (LN) and GELU activation are completed, the MLP operation is performed on the feature map A in the Height channel, and LN and GELU activation are also completed, to obtain the feature map B; finally, A and B are added to generate a new feature map, which is used as the input feature map of the next decoding layer or encoding layer.

[0083] The image segmentation method provided in the embodiment of the application comprises: inputting the first feature map of the DICOM image into a Width channel to perform MLP operation to obtain a second feature map, inputting the second feature map into a Height channel to perform MLP operation to obtain a third feature map, and finally superimposing the first feature map and the third feature map to obtain a plurality of sampling feature maps, so that the plurality of sampling feature maps all contain width information and height information of the image, and the image representation capability of the sampling feature map is improved.

[0084] In some embodiments, the multi-layer perception machine comprises a feature fusion layer, a loss function of the feature fusion layer is a first joint loss function, and the first joint loss function is obtained based on a cross-entropy loss function and a Dice loss function; based on the sampling feature map with the highest image resolution and the multi-scale feature map, a segmentation image is obtained by: performing feature fusion on the sampling feature map with the highest image resolution and the multi-scale feature map in the feature fusion layer to obtain a fused feature map; and based on the fused feature map, the segmentation image is obtained.

[0085] In this embodiment, the sampling feature map with the highest image resolution contains fine-grained feature information of the image, and the multi-scale feature map contains coarse-grained feature information of the image, and after the fine-grained feature information of the image and the coarse-grained feature information of the image are fused, a fused feature map with stronger image representation capability can be obtained.

[0086] In Figure 2 In the illustrated embodiment, the first joint loss function is Loss5, and the sampling feature map with the highest image resolution and the multi-scale feature map are fused in the feature fusion layer (corresponding to the FM layer).

[0087] In this embodiment, the first layer (D1) of the decoding part and the feature fusion layer are classified as fine-grained feature, and the two parts are respectively provided with corresponding weights, and the corresponding loss function is as follows:

[0088] Loss acc = 0.2 · (CELoss1 + DiLoss1) + 0.5 · (CELoss5 + DiLoss5);

[0089] wherein, Loss accLoss is the loss value of the first joint loss function, CEloss1 is the cross entropy loss function (CEloss) of the 1st layer decoder, DiLoss1 is the Dice loss function of the 1st layer decoder, CEloss5 is the cross entropy loss function of the feature fusion layer, and Dioss5 is the Dice loss function of the feature fusion layer.

[0090] The image segmentation method provided in the embodiment of the application can distribute the weights of coarse-grained feature information and fine-grained feature information in the fused feature map by setting the first joint loss function, thereby reasonably adjusting the global information and edge information of the image and improving the accuracy of image edge segmentation.

[0091] In some embodiments, according to the image segmentation method provided by the application, the loss function corresponding to the nth decoder is a second joint loss function, and the second joint loss function is obtained based on a cross entropy loss function and a Dice loss function; the (n+1)th down-sampling feature map is input into the nth decoding layer for up-sampling to obtain a first up-sampling image, and this process is repeated until the (n-1)th down-sampling feature map is input into the 1st decoding layer for down-sampling to obtain the nth up-sampling feature map, including: inputting the (n+1)th down-sampling feature map into the nth decoding layer for up-sampling, and based on the second joint loss function corresponding to the nth decoding layer, a first up-sampling image is obtained, and this process is repeated until the (n-1)th down-sampling feature map is input into the 1st decoding layer for down-sampling, and based on the second joint loss function corresponding to the 1st decoding layer, the nth up-sampling feature map is obtained.

[0092] In Figure 2 In the embodiment shown, the loss functions corresponding to the transition layer E4, the decoding layers D1 to D3 are Loss1 to Loss4, and the loss function of the fused multi-scale features calculated by the FM layer is Loss5. Since the coarse and fine granularity of each feature map is different, and the edge details in the segmentation task are important indicators, different weights can be used for Loss calculation.

[0093] In this embodiment, the transition layer E4, the decoding layer D3 and the encoding layer D3 of the decoding part are defined as coarse-grained features, and the second joint loss function is uniformly processed, and the corresponding second joint loss function is as follows:

[0094]

[0095] In this embodiment, the total loss function of the multi-layer perceptron can be represented by combining the first joint loss function and the second joint loss function, for example, the total loss function is represented as follows:

[0096] Loss = Loss c + Loss acc .

[0097] The image segmentation method provided by the embodiment of the present application can improve the representation ability of the global information and the edge information of the sampling feature map by setting the second joint loss function on the transition layer and the plurality of decoding layers to respectively obtain the corresponding sampling feature maps, thereby improving the accuracy of image edge segmentation.

[0098] The image segmentation device provided by the present application is described below, and the image segmentation device described below can be correspondingly referred to the image segmentation method described above.

[0099] Figure 6 FIG. 1 is a structural schematic diagram of an image segmentation device provided by the present application.

[0100] The image acquisition module 610 is configured to acquire a DICOM image to be segmented.

[0101] The feature fusion module 620 is configured to perform feature extraction on the DICOM image based on a multi-layer perception machine to obtain a plurality of sampling feature maps, and perform convolution operation on a target sampling feature map to obtain a multi-scale feature map.

[0102] The image segmentation module 630 is configured to obtain a segmentation image based on the sampling feature map with the highest image resolution and the multi-scale feature map.

[0103] The image segmentation device provided by the embodiment of the present application can obtain the width information and the height information of the image by obtaining the plurality of sampling feature maps of the DICOM image through the multi-layer perception machine, and can obtain the abstract information of the image by performing convolution operation on the target sampling feature map to obtain the multi-scale feature map.

[0104] Figure 7 FIG. 1 is a structural schematic diagram of an image segmentation device provided by the present application. Figure 7As shown, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute an image segmentation method, which includes: acquiring a DICOM image to be segmented; performing feature extraction on the DICOM image based on a multi-layer perception machine to obtain a plurality of sampling feature maps, and performing convolution operation on a target sampling feature map to obtain a multi-scale feature map, the plurality of sampling feature maps including height information and width information of the DICOM image, and the target sampling feature map being other sampling feature maps in the plurality of sampling feature maps except for a sampling feature map with the highest image resolution; and obtaining a segmentation image based on the sampling feature map with the highest image resolution and the multi-scale feature map.

[0105] In addition, the logical instruction in the memory 730 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0106] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the image segmentation method provided by the above-mentioned methods, which includes: acquiring a DICOM image to be segmented; performing feature extraction on the DICOM image based on a multi-layer perception machine to obtain a plurality of sampling feature maps, and performing convolution operation on a target sampling feature map to obtain a multi-scale feature map, the plurality of sampling feature maps including height information and width information of the DICOM image, and the target sampling feature map being other sampling feature maps in the plurality of sampling feature maps except for a sampling feature map with the highest image resolution; and obtaining a segmentation image based on the sampling feature map with the highest image resolution and the multi-scale feature map.

[0107] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the image segmentation method provided by any of the above methods, and the method comprises: obtaining a DICOM image to be segmented; performing feature extraction on the DICOM image based on a multi-layer perception machine to obtain a plurality of sampling feature maps, and performing convolution operation on a target sampling feature map to obtain a multi-scale feature map, the plurality of sampling feature maps comprising height information and width information of the DICOM image, and the target sampling feature map being any sampling feature map other than a sampling feature map with the highest image resolution among the plurality of sampling feature maps; and obtaining a segmentation image based on the sampling feature map with the highest image resolution and the multi-scale feature map.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0109] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0110] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image segmentation method, characterized in that, include: Obtain the DICOM image to be segmented; Based on a multilayer perceptron, feature extraction is performed on the DICOM image to obtain multiple sampled feature maps. The target sampled feature map is then convolved to obtain a multi-scale feature map. The multiple sampled feature maps include the height and width information of the DICOM image. The target sampled feature map is the other sampled feature maps among the multiple sampled feature maps except for the sampled feature map with the highest image resolution. Based on the highest resolution sampled feature map and the multi-scale feature map, a segmented image is obtained; The process of obtaining a segmented image based on the highest resolution sampled feature map and the multi-scale feature map includes: The highest resolution sampled feature map and the multi-scale feature map of the image are fused in the feature fusion layer to obtain a fused feature map. The segmented image is obtained based on the fused feature map; The multilayer sensor includes a Width channel and a Height channel; The step of extracting features from the DICOM image based on a multilayer perceptron to obtain multiple sampled feature maps also includes: Obtain the first feature map of the DICOM image; The feature map of the DICOM image is input into the Width channel for MLP operation to obtain the second feature map; The second feature map is input into the Height channel for MLP operation to obtain the third feature map; The first feature map and the third feature map are superimposed to obtain the plurality of sampled feature maps.

2. The image segmentation method according to claim 1, characterized in that, The sampling feature map includes an upsampled feature map and a downsampled feature map. The multilayer perceptron includes n encoding layers, n decoding layers and 1 transition layer. The input of the transition layer is connected to the nth encoding layer, and the output of the transition layer is connected to the nth decoding layer. There is a skip connection between the ith encoding layer and the ith decoding layer. n is a natural number greater than 1, and 1 ≤ i ≤ n. The DICOM image is subjected to feature extraction based on a multilayer perceptron to obtain multiple sampled feature maps, including: The DICOM image is input into the first coding layer for downsampling to obtain the first downsampled feature map; The first downsampled feature map is input into the second coding layer for downsampling to obtain the second downsampled feature map, until the nth sampled feature map is input into the transition layer for downsampling to obtain the (n+1)th downsampled feature map; The (n+1)th downsampled feature map is input to the nth decoding layer for upsampling to obtain the first upsampled image, until the (n-1)th upsampled feature map is input to the 1st decoding layer for downsampling to obtain the nth upsampled feature map.

3. The image segmentation method according to claim 1, characterized in that, The sampled feature map includes an upsampled feature map and a downsampled feature map. The multilayer perceptron includes n encoding layers, n decoding layers, 1 transition layer and n convolutional layers. The input of the transition layer is connected to the nth encoding layer, and the output of the transition layer is connected to the nth decoding layer. There is a skip connection between the ith encoding layer and the ith decoding layer. n is a natural number greater than 1, and 1≤i≤n. The step of performing a convolution operation on the target sampled feature map to obtain a multi-scale feature map includes: The nth upsampled feature map is determined to be the sampled feature map with the highest image resolution; The (n+1)th downsampled feature map and the first upsampled image to the (n-1)th upsampled image are respectively input into the convolutional layer to obtain n convolutional feature maps; The nth upsampled feature map and the nth convolutional feature map are subjected to a concat operation to obtain the multi-scale feature image; Wherein, the nth upsampled feature map is obtained by sequentially encoding and decoding the DICOM image (n+1 times); the (n+1)th downsampled feature map is obtained by sequentially downsampling the DICOM image (n+1 times); the first upsampled image is obtained by sequentially downsampling and upsampling the DICOM image (n+1 times); the (n-1)th upsampled image is obtained by sequentially downsampling and upsampling the DICOM image (n+1 times); and the nth upsampled feature map is obtained by sequentially downsampling and upsampling the DICOM image (n+1 times).

4. The image segmentation method according to claim 1, characterized in that, The multilayer perceptron includes a feature fusion layer, and the loss function of the feature fusion layer is a first joint loss function, which is obtained based on the cross-entropy loss function and the Dice loss function.

5. The image segmentation method according to claim 2, characterized in that, The loss function corresponding to the n decoding layers is the second joint loss function, which is obtained based on the cross-entropy loss function and the Dice loss function. The step of inputting the (n+1)th downsampled feature map into the nth decoding layer for upsampling to obtain the first upsampled image, until the (n-1)th upsampled feature map is input into the 1st decoding layer for downsampling to obtain the nth upsampled feature map, includes: The (n+1)th downsampled feature map is input to the nth decoding layer for upsampling, and a first upsampled image is obtained based on the second joint loss function corresponding to the nth decoding layer, until the (n-1)th upsampled feature map is input to the 1st decoding layer for downsampling, and an nth upsampled feature map is obtained based on the second joint loss function corresponding to the 1st decoding layer.

6. An image segmentation apparatus, characterized in that, include: The image acquisition module is used to acquire the DICOM image to be segmented; The feature fusion module is used to extract features from the DICOM image based on a multilayer perceptron to obtain multiple sampled feature maps, and to perform a convolution operation on the target sampled feature map to obtain a multi-scale feature map. The multiple sampled feature maps include the height and width information of the DICOM image, and the target sampled feature map is the other sampled feature maps in the multiple sampled feature maps except for the sampled feature map with the highest image resolution. The image segmentation module is used to obtain a segmented image based on the highest resolution sampled feature map and the multi-scale feature map of the image; The process of obtaining a segmented image based on the highest resolution sampled feature map and the multi-scale feature map includes: The highest resolution sampled feature map and the multi-scale feature map of the image are fused in the feature fusion layer to obtain a fused feature map. The segmented image is obtained based on the fused feature map; The multilayer sensor includes a Width channel and a Height channel; The step of extracting features from the DICOM image based on a multilayer perceptron to obtain multiple sampled feature maps also includes: Obtain the first feature map of the DICOM image; The feature map of the DICOM image is input into the Width channel for MLP operation to obtain the second feature map; The second feature map is input into the Height channel for MLP operation to obtain the third feature map; The first feature map and the third feature map are superimposed to obtain the plurality of sampled feature maps.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image segmentation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method as described in any one of claims 1 to 5.

Images