Abdominal organ segmentation method and system based on random super-resolution learning assistance and multilayer feature mapping

Abstract

The invention discloses an abdominal organ segmentation method and system based on random super-resolution learning assistance and multi-layer feature mapping. The method comprises the following steps: acquiring an image to be segmented; the to-be-segmented image is input to a segmentation model, a semantic segmentation label under the low-resolution image dimension is obtained, the segmentation model is obtained through training of a training set and is trained in an auxiliary mode through a multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used for reconstructing the semantic segmentation label under the low-resolution image dimension. The decoding module is used for decoding the advanced feature representation of the to-be-segmented image to obtain a multi-resolution reconstructed image and reconstructed features; and processing the semantic segmentation tag under the low-resolution image dimension to obtain a semantic segmentation tag under the high-resolution image. According to the method, anatomical structure information of images with various resolutions can be extracted, potential association between multi-resolution learning and semantic segmentation tasks is enhanced, and the performance of the medical super-resolution image semantic segmentation tasks is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical super-resolution image semantic segmentation, and in particular relates to an abdominal organ segmentation method and system based on random super-resolution learning assistance and multi-layer feature mapping. Background Art

[0002] Automated and accurate organ segmentation based on medical images (i.e., semantic segmentation) is an important research field in clinical imaging. It helps to quickly extract the anatomical structure of the lesion area, reduce the burden on radiologists, and improve the efficiency of clinical data analysis. Currently, the segmentation results of high-resolution medical images are highly accurate, but the acquisition and calculation are time-consuming and resource-intensive. The acquisition and calculation of low-resolution images are fast, but the segmentation results are less accurate. Therefore, in order to obtain excellent segmentation results without relying on high-resolution input, how to extract more features from low-resolution medical images to complete the high-resolution output of semantic segmentation labels is a difficult and hot topic in research.

[0003] With the development of deep learning technology, researchers have recently proposed a dual-stream task framework, which takes image segmentation as the main task and super-resolution generation as the auxiliary task. Through the constraint relationship between tasks, the super-resolution task guides the image segmentation task and improves the segmentation performance of low-resolution images. That is, the main task of the framework takes low-resolution images as input and high-resolution segmentation labels as output, and the auxiliary task takes high-resolution images as output. Although the dual-stream framework has achieved good performance in the field of image segmentation, there are still problems such as difficult training of super-resolution tasks, little single-resolution information, and incomplete dual-task feature mapping.

[0004] Specifically, the existing two-stream framework directly maps a single low-resolution image to a high-resolution image. The high-resolution reconstruction task is difficult to train, and the model has less resolution information to learn. If the model can be provided with multiple resolution information and a simple image reconstruction task is used instead of a super-resolution image generation task, it will be possible to extract more image details from image reconstruction tasks of different resolutions, improve the model's understanding of different hierarchical structures of the image, and reduce the difficulty of model training. In addition, most of the existing two-stream frameworks perform feature similarity mapping between super-resolution tasks and segmentation tasks on a certain layer of features in the model decoder, but different layers of the model extract features from different angles. Constraining feature similarity at multiple levels will hopefully extract image information more comprehensively.

[0005] Considering the importance of different image resolution information and model-level features to the extraction of fine-grained structural information of images, there is an urgent need for a medical super-resolution image semantic segmentation method and system based on random super-resolution learning assistance and multi-layer feature mapping to address the shortcomings of the existing technology. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes an abdominal organ segmentation method and system based on random super-resolution learning assistance and multi-layer feature mapping to solve the problems existing in the above-mentioned prior art and realize accurate super-resolution segmentation of medical images.

[0007] The present invention provides an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping, comprising:

[0008] Obtain the image to be segmented;

[0009] Inputting the image to be segmented into a segmentation model to obtain a semantic segmentation label under a low-resolution image dimension, wherein the segmentation model is trained using a training set and assisted by a multi-resolution image reconstruction module, wherein the multi-resolution image reconstruction module is used to decode a high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstruction features;

[0010] The semantic segmentation labels under the low-resolution image dimension are processed to obtain the semantic segmentation labels under the high-resolution image.

[0011] Optionally, obtaining the training set includes:

[0012] Obtain original high-resolution image data;

[0013] The original high-resolution image data is cropped and scaled, and the dimensions are unified to obtain a low-resolution image;

[0014] Based on the low-resolution image, the training set is acquired.

[0015] Optionally, the segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0016] The multi-resolution image data acquisition module is used to acquire low-resolution image data of uniform dimension;

[0017] The image encoder module is used to encode the image data to obtain high-level feature representation;

[0018] The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features;

[0019] The semantic output module is used to output semantic tags.

[0020] Optionally, the multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels;

[0021] The semantic segmentation decoder unit is used to decode the high-level feature representation to generate a semantic segmentation label;

[0022] The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label;

[0023] The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception.

[0024] Optionally, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for reconstructed image and original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0025] The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image;

[0026] The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image;

[0027] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through space and channel attention perception to obtain reconstruction features.

[0028] Optionally, fusing different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain the reconstruction features includes:

[0029] The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ..., G n ;

[0030] For G n The layer performs a channel global average pooling operation on G n-1 The spatial global average pooling operation is performed on the layer, and the two feature layers after the operation are fused to obtain the reconstructed features.

[0031] Optionally, using a multi-resolution image reconstruction module to assist in training to obtain the segmentation model includes:

[0032] Obtaining a mapping loss according to the reconstruction features and the semantic features;

[0033] According to the mapping loss, combined with the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label, the total loss of the segmentation model is calculated;

[0034] The segmentation model is obtained according to the total loss of the segmentation model.

[0035] The present invention also provides an abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping, comprising: an image acquisition module, an image processing module and an image output module;

[0036] The image acquisition module is used to acquire low-resolution images;

[0037] The image processing module is used to input the low-resolution image into a segmentation model to obtain a semantic segmentation label under the low-resolution image dimension, wherein the segmentation model is obtained by auxiliary training using a multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used to decode the high-level feature representation of the low-resolution image to obtain a multi-resolution reconstructed image and reconstruction features;

[0038] The image output module is used to output the semantic segmentation label under the high-resolution image according to the semantic segmentation label under the low-resolution image dimension.

[0039] Optionally, a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0040] The multi-resolution image data acquisition module is used to acquire image data with different resolutions;

[0041] The image encoder module is used to encode the image data to obtain high-level feature representation;

[0042] The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features;

[0043] The semantic output module is used to output semantic tags.

[0044] Optionally, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for reconstructed image and original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0045] The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image;

[0046] The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image;

[0047] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through space and channel attention perception to obtain reconstruction features.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] 1. The present invention obtains training data of different resolutions by random cropping and downsampling, and can extract anatomical structure information of images with multiple resolutions. This method can quickly achieve resolution enhancement and improve the ability to fuse and utilize information with different resolutions;

[0050] 2. The present invention maps and fuses the features of different layers of the image reconstruction task decoder and the semantic segmentation task decoder through spatial and channel attention perception, directing the channel attention to the shallow features to capture a wider background; directing the spatial attention to the deep features to identify detailed patterns and textures;

[0051] 3. This paper designs four loss functions to evaluate the intra-task and inter-task losses of image reconstruction and semantic segmentation tasks from multiple perspectives, allowing the model to capture more high-resolution information and semantic segmentation information, and strengthen the potential correlation between image reconstruction and semantic segmentation tasks;

[0052] In general, the present invention improves the performance of medical super-resolution image semantic segmentation tasks through methods such as fast resolution enhancement, spatial and channel attention perception, image reconstruction and semantic segmentation mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0054] Figure 1 It is a flow chart of an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to an embodiment of the present invention;

[0055] Figure 2 It is a model framework diagram of an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a decoder multi-layer feature fusion method in an embodiment of the present invention;

[0057] Figure 4 4 is a comparison chart of the segmentation effects of different algorithm models in the embodiments of the present invention. DETAILED DESCRIPTION

[0058] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0059] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] The present invention proposes an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping. Figure 1 As shown, the specific steps include:

[0061] Obtain the image to be segmented;

[0062] The image to be segmented is input into the segmentation model to obtain the semantic segmentation label under the low-resolution image dimension, wherein the segmentation model is trained using the training set and assisted by the multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used to decode the high-level feature representation of the image to be segmented and obtain the multi-resolution reconstructed image and reconstruction features;

[0063] The semantic segmentation labels under the low-resolution image dimension are processed to obtain the semantic segmentation labels under the high-resolution image.

[0064] Specifically, the input data is a low-resolution image dataset with an image size of (H / q, W / q) at any q value, and the output is a semantic segmentation label under a high-resolution image with an image size of (H, W).

[0065] Furthermore, obtaining a training set includes:

[0066] Obtain original high-resolution image data;

[0067] The original high-resolution image data is cropped and scaled, and the dimensions are unified to obtain a low-resolution image;

[0068] Based on low-resolution images, obtain a training set.

[0069] Specifically, for the original high-resolution image dataset X_high, the image size is (H, W), and each image is randomly cropped into a cropped dataset X_crop with an image size of (H / p, W / p), where the p value is greater than or equal to 1, and the value is different each time the crop is cropped, representing the scaling ratio of the random resolution of the image. The dataset X_crop is downsampled by a coefficient of q / p to generate a low-resolution image dataset X_low with an image size of (H / q, W / q), where the q value is greater than 1 and the value is determined according to the input image resolution of the test data, representing the image reduction ratio in the process of constructing the low-resolution dataset. In the process of comparing models of different methods, the model goal is to perform semantic segmentation of low-resolution images with an image size of (H / q, W / q) into high-resolution images with an image size of (H, W).

[0070] Further, the segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0071] A multi-resolution image data acquisition module, used to acquire low-resolution image data of uniform dimension;

[0072] The image encoder module is used to encode image data and obtain high-level feature representation;

[0073] Multi-resolution image semantic segmentation module, which is used to decode the high-level feature representation of image data, generate semantic segmentation labels under multi-resolution images, and obtain semantic features;

[0074] The semantic output module is used to output semantic labels.

[0075] Furthermore, the multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels;

[0076] The semantic segmentation decoder unit is used to decode the high-level feature representation and generate semantic segmentation labels;

[0077] The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label;

[0078] The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception.

[0079] Specifically, the multi-resolution image semantic segmentation module includes a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and a semantic segmentation generated label and an original label error calculation unit, the output end of the image encoder module is connected to the semantic segmentation decoder unit, the output end of the semantic segmentation decoder unit is connected to the semantic segmentation decoder multi-layer feature fusion unit and the semantic segmentation generated label and the original label error calculation unit, the output end of the semantic segmentation decoder multi-layer feature fusion unit is connected to the dual-task feature mapping module, and the output end of the semantic segmentation generated label and the original label error calculation unit is connected to the model total loss calculation module;

[0080] The semantic segmentation decoder unit is used to decode the high-level feature representation of the data output by the image encoder module to generate semantic segmentation labels;

[0081] The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow features of the semantic segmentation decoder unit through spatial and channel attention perception;

[0082] Dual-task feature mapping module, used to calculate the similarity of multi-layer fusion feature maps for multi-resolution image reconstruction tasks and semantic segmentation tasks;

[0083] The error calculation unit for the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation generated label and the original label output by the semantic segmentation decoder unit.

[0084] More specifically, the error calculation unit for semantic segmentation generated labels and original labels includes:

[0085] Generate the value of each pixel in the label image based on the output of the semantic segmentation decoder unit And the value y of each pixel in the original true label image n , calculate the Dice coefficient loss L of the two images Dice :

[0086]

[0087] Where N represents the total number of pixels and ε represents a minimum value to avoid the denominator being 0.

[0088] Further, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for the reconstructed image and the original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0089] An image reconstruction decoder unit, for decoding the high-level feature representation and reconstructing a multi-resolution image;

[0090] The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image;

[0091] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain the reconstruction features.

[0092] Specifically, the multi-resolution image reconstruction module includes an image reconstruction decoder unit, an error calculation unit for reconstructed images and original images, and an image reconstruction decoder multi-layer feature fusion unit, the output end of the image encoder module is connected to the image reconstruction decoder unit, the output end of the image reconstruction decoder unit is connected to the error calculation unit for reconstructed images and original images and the image reconstruction decoder multi-layer feature fusion unit, the output end of the image reconstruction decoder multi-layer feature fusion unit is connected to the dual-task feature mapping module, and the output end of the error calculation unit for reconstructed images and original images is connected to the model total loss calculation module;

[0093] An image reconstruction decoder unit, for decoding the high-level feature representation of the data output by the image encoder module and reconstructing a multi-resolution image;

[0094] The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image output by the image reconstruction decoder unit and the original image;

[0095] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception.

[0096] Furthermore, the different deep and shallow features of the image reconstruction decoder unit are fused through spatial and channel attention perception to obtain the reconstruction features including:

[0097] The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ..., G n ;

[0098] For G n The layer performs a channel global average pooling operation on G n-1 The spatial global average pooling operation is performed on the layer, and the two feature layers after the operation are fused to obtain the reconstructed features.

[0099] Specifically, the multi-layer feature fusion unit of the image reconstruction decoder includes:

[0100] The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ... from deep to shallow. The multi-layer feature fusion process is carried out in each feature layer in turn. Each fusion performs spatial global average pooling (i.e. spatial attention) on the previous deep layer feature, and performs channel global average pooling (i.e. channel attention) on the current layer feature. The calculated features are weighted and fused as the fused features. Taking the fusion of G1 and G2 layers as an example, the G1 layer feature is obtained by spatial global average pooling. 1s , G 1s Through convolution (Conv), Sigmoid (δ), and G1 weighted convolution, we get G 11 , the G2 layer features are pooled globally through the channel to obtain G 2c , G 2c After linear transformation (Linear), ReLU (θ), linear transformation, and Sigmoid calculation, the channel weight G is obtained. 22 , G 11 and G 22 After weighted upsampling (Up) and adding it to the G2 layer, the feature G fused by the G1 and G2 layers is obtained. 12 :

[0101]

[0102] G 11 =Conv(δ(Conv(G 1s ))⊙G1)

[0103]

[0104] G 22 =δ(Linear(θ(Linear(G2c))))

[0105] G 12 =(Up(G 11 ⊙G 22 ))⊕G2

[0106] Among them, C represents the number of channels, ⊙ represents the Hadamard product, and ⊕ represents the addition operation at the pixel level.

[0107] More specifically, the error calculation unit of the reconstructed image and the original image includes:

[0108] The value of each pixel in the generated image based on the output of the image reconstruction decoder unit And the value x of each pixel in the original image ij , calculate the mean square error loss L of the two images MSE :

[0109]

[0110] Furthermore, the segmentation model is obtained by using a multi-resolution image reconstruction module to assist in training, including:

[0111] Obtain mapping loss based on reconstruction features and semantic features;

[0112] According to the mapping loss, the total loss of the segmentation model is calculated by combining the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label;

[0113] Get the segmentation model based on the total segmentation model loss.

[0114] Specifically, in addition, the pixel-level cross entropy loss L is calculated CE and region-level mutual information loss L RMI , which constitutes the total loss L at the pixel level and region level CE-RMI :

[0115]

[0116] Where λ represents the weight factor, B represents the number of images processed in each batch, Represents the region-level mutual information between the original true labels and the generated labels.

[0117] The dual-task feature mapping module includes:

[0118] Based on the output of the multi-layer feature fusion unit of the image reconstruction decoder, the feature maps of different channels are inner-producted pairwise to obtain the Gram matrix H G Based on the output of the multi-layer feature fusion unit of the semantic segmentation decoder, the feature maps of different channels are inner-producted pairwise to obtain the Gram matrix H S . We further calculate the dual-task mapping loss L for the multi-resolution image reconstruction task and the semantic segmentation task GS :

[0119]

[0120] Where u and v are the dimension coordinates of the Gram matrix.

[0121] More specifically, the model total loss calculation module includes:

[0122] According to L MSE , L Dice and L CE-RMI and L GS , calculate the total model loss L:

[0123] L=L MSE +L Dice +μ(L CE-RMI +L GS )

[0124] Where μ is a weight coefficient with a value less than 1, which is used to generate the total loss L at the pixel level and region level for the semantic segmentation label and the original label CE-RMI And the dual-task mapping loss L for image reconstruction and semantic segmentation GS Weight reduction is performed to ensure that the model training process is not overly affected by these two parts of loss and to achieve a trade-off between the contributions of different losses.

[0125] The present invention also provides an abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping, comprising: an image acquisition module, an image processing module and an image output module;

[0126] An image acquisition module, used for acquiring low-resolution images;

[0127] An image processing module is used to input the low-resolution image into the segmentation model to obtain the semantic segmentation label under the low-resolution image dimension, wherein the segmentation model is obtained by auxiliary training using a multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used to decode the high-level feature representation of the low-resolution image to obtain a multi-resolution reconstructed image and reconstruction features;

[0128] The image output module is used to output the semantic segmentation label under the high-resolution image according to the semantic segmentation label under the low-resolution image dimension.

[0129] Further, a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0130] A multi-resolution image data acquisition module, used to acquire image data with different resolutions;

[0131] The image encoder module is used to encode image data and obtain high-level feature representation;

[0132] Multi-resolution image semantic segmentation module, which is used to decode the high-level feature representation of image data, generate semantic segmentation labels under multi-resolution images, and obtain semantic features;

[0133] The semantic output module is used to output semantic labels.

[0134] Furthermore, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for the reconstructed image and the original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0135] An image reconstruction decoder unit, for decoding the high-level feature representation and reconstructing a multi-resolution image;

[0136] The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image;

[0137] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain the reconstruction features.

[0138] The present embodiment is described in detail below with reference to the accompanying drawings:

[0139] This embodiment provides an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping. The overall structure is as follows: Figure 1 As shown, it includes obtaining an image to be segmented, a segmentation model, generating a low-resolution semantic segmentation label, and outputting a high-resolution semantic segmentation label, wherein the segmentation model includes original training data, multi-resolution training data, an image encoder, a high-level feature representation, image reconstruction-assisted semantic segmentation decoding, model loss calculation, model training, and parameter preservation. In this embodiment, the Synapse dataset is taken as an example. The data is an abdominal CT image, involving the segmentation of eight organs including the aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas, and stomach. The spatial dimension resolution of the original high-resolution image is (512, 512), and the spatial dimension resolution of the low-resolution image is (256, 256).

[0140] Model framework such as Figure 2 As shown in the figure, the image to be segmented is input into the segmentation model to generate a low-resolution semantic segmentation label, and the high-resolution semantic segmentation label is obtained after upsampling. The details of the segmentation model include: for the original high-resolution image dataset X_high, the image size is (512,512), and each image is randomly cropped into a cropped dataset X_crop with an image size of (512 / p,512 / p), where the p value is greater than or equal to 1 and less than or equal to 2, and the value of each cropping is different. The dataset X_crop is downsampled by a coefficient of 2 / p to generate a low-resolution image dataset X_low with an image size of (256,256). In the process of comparing different method models, the model goal is to perform semantic segmentation of a low-resolution image with an image size of (256,256) with a high-resolution image with an image size of (512,512).

[0141] A low-resolution image dataset X_low with an image size of (256, 256) is input into the image encoder module to form a high-level feature representation of the image data, where the encoders of the multi-resolution image reconstruction task and the multi-resolution image semantic segmentation task share weights.

[0142] The high-level feature representations output by the encoder are input into the decoders of the image reconstruction task and the semantic segmentation task, respectively, to generate multi-resolution reconstructed images and organ segmentation labels.

[0143] For the different layers of features of the decoder of the image reconstruction task and the semantic segmentation task, the features of different depths and shallow layers are fused through spatial attention and channel attention. The feature layer fusion algorithm of the two tasks is the same. Take the feature fusion of the image reconstruction task as an example:

[0144] The feature layers of the decoder unit are G1, G2, G3, G4, ... from deep to shallow. The multi-layer feature fusion process is carried out in each feature layer in turn. Each fusion performs spatial global average pooling (i.e. spatial attention) on the previous deep layer feature, and channel global average pooling (i.e. channel attention) on the current layer feature. The calculated features are weighted and fused as the fused features. Figure 3 As shown in the figure, taking the fusion of G1 layer and G2 layer as an example, the G1 layer features are obtained by spatial global average pooling. 1s , G 1s Through convolution (Conv), Sigmoid (δ), and G1 weighted convolution, we get G 11 , the G2 layer features are pooled globally through the channel to obtain G 2c , G 2c After linear transformation (Linear), ReLU (θ), linear transformation, and Sigmoid calculation, the channel weight G is obtained. 22 , G 11 and G 22 After weighted upsampling (Up) and adding it to the G2 layer, the feature G fused by the G1 and G2 layers is obtained. 12 :

[0145]

[0146] G 11 =Conv(δ(Conv(G 1s ))⊙G1)

[0147]

[0148] G 22 =δ(Linear(θ(Linear(G2c))))

[0149] G 12 =(Up(G 11 ⊙G 22 ))⊕G2

[0150] Where C represents the number of channels (3 in this embodiment), H and W represent image dimensions, which are 512 in this embodiment, ⊙ represents the Hadamard product, ⊕ represents the addition operation at the pixel level, and the convolution kernel size is 3×3 in this embodiment.

[0151] After fusing the decoder multi-layer features of the image reconstruction task and the semantic segmentation task, the Gram matrix H of the image reconstruction task is calculated.G and the Gram matrix H for the semantic segmentation task S . Further calculate the dual-task mapping loss L for the image reconstruction task and the semantic segmentation task GS :

[0152]

[0153] Wherein u and v are the dimension coordinates of the Gram matrix, and the value is 16 in this embodiment.

[0154] The value of each pixel in the generated image based on the output of the image reconstruction task decoder And the value x of each pixel in the original image ij , calculate the mean square error loss L of the two images MSE :

[0155]

[0156] The value of each pixel in the generated label image based on the output of the semantic segmentation task decoder And the value y of each pixel in the original true label image n , calculate the Dice coefficient loss L of the two images Dice :

[0157]

[0158] Where N represents the total number of pixels (the average value in this embodiment is 256×256), and ε represents a minimum value (the average value in this embodiment is 1e-6) to avoid the denominator being 0. In addition, the cross entropy loss L at the pixel level is calculated CE and region-level mutual information loss L RMI , which constitutes the total loss L at the pixel level and region level CE-RMI :

[0159]

[0160] Where λ represents the weight factor (the value in this embodiment is 0.5), B represents the number of images processed in each batch (the value in this embodiment is 8), Represents the region-level mutual information between the original true labels and the generated labels.

[0161] Further calculate the total model loss L:

[0162] L=L MSE +L Dice +μ(L CE-RMI +L GS )

[0163] Where μ is the weight coefficient (the value in this embodiment is 0.1), which is used to generate the total loss L at the pixel level and region level for the semantic segmentation label and the original label CE-RMI And the dual-task mapping loss L for image reconstruction and semantic segmentation GS Weight reduction is performed to ensure that the model training process is not overly affected by these two parts of loss and to achieve a trade-off between the contributions of different losses.

[0164] Finally, the model parameters are trained and updated, and the performance of the model in the abdominal organ segmentation task is tested. Figure 4 As shown, compared with ResNet-50 (average Dice coefficient 76.81%), the Dice coefficient of the model in this embodiment reaches 83.18%.

[0165] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping, characterized in that: include: Obtain the image to be segmented; Inputting the image to be segmented into a segmentation model to obtain a semantic segmentation label under a low-resolution image dimension, wherein the segmentation model is trained using a training set and assisted by a multi-resolution image reconstruction module, wherein the multi-resolution image reconstruction module is used to decode a high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstruction features; The semantic segmentation labels under the low-resolution image dimension are processed to obtain the semantic segmentation labels under the high-resolution image.

2. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 1, characterized in that: Acquiring the training set includes: Obtain original high-resolution image data; The original high-resolution image data is cropped and scaled, and the dimensions are unified to obtain a low-resolution image; Based on the low-resolution image, the training set is acquired.

3. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 1, characterized in that: The segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module; The multi-resolution image data acquisition module is used to acquire low-resolution image data of uniform dimension; The image encoder module is used to encode the image data to obtain high-level feature representation; The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features; The semantic output module is used to output semantic tags.

4. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 3 is characterized in that: The multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels; The semantic segmentation decoder unit is used to decode the high-level feature representation to generate a semantic segmentation label; The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label; The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception.

5. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 4, characterized in that: The multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for reconstructed image and original image, and an image reconstruction decoder multi-layer feature fusion unit; The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image; The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image; The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through space and channel attention perception to obtain reconstruction features.

6. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 5, characterized in that: The different deep and shallow features of the image reconstruction decoder unit are fused through spatial and channel attention perception to obtain reconstruction features including: The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ..., G n ; For G n The layer performs a channel global average pooling operation on G n-1 The spatial global average pooling operation is performed on the layer, and the two feature layers after the operation are fused to obtain the reconstructed features.

7. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 5, characterized in that: Auxiliary training using the multi-resolution image reconstruction module includes: Obtaining a mapping loss according to the reconstruction features and the semantic features; According to the mapping loss, combined with the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label, the total loss of the segmentation model is calculated; The segmentation model is obtained according to the total loss of the segmentation model.

8. An abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping, characterized in that: include: Image acquisition module, image processing module and image output module; The image acquisition module is used to acquire low-resolution images; The image processing module is used to input the low-resolution image into the segmentation model to obtain the semantic segmentation label under the low-resolution image dimension, wherein the segmentation model is obtained by training with a training set and assisted by a multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used to decode the high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstruction features; The image output module is used to output the semantic segmentation label under the high-resolution image according to the semantic segmentation label under the low-resolution image dimension.

9. The abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping according to claim 8, characterized in that: Multi-resolution image data acquisition module, image encoder module, multi-resolution image semantic segmentation module and semantic output module; The multi-resolution image data acquisition module is used to acquire image data with different resolutions; The image encoder module is used to encode the image data to obtain high-level feature representation; The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features; The semantic output module is used to output semantic tags.

10. The abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping according to claim 8, characterized in that: The multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for reconstructed image and original image, and an image reconstruction decoder multi-layer feature fusion unit; The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image; The error calculation unit of the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image; The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through space and channel attention perception to obtain reconstruction features.

Images