A method and system for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping

Abstract

The present invention discloses a method and system for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping. The method comprises: obtaining an image to be segmented; inputting the image to be segmented into a segmentation model to obtain semantic segmentation labels in the low-resolution image dimension, wherein the segmentation model is trained using a training set and assisted by a multi-resolution image reconstruction module, the multi-resolution image reconstruction module being used to decode the high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstructed features; and processing the semantic segmentation labels in the low-resolution image dimension to obtain semantic segmentation labels in the high-resolution image. The present invention can extract anatomical structure information from images of multiple resolutions, strengthen the potential association between multi-resolution learning and semantic segmentation tasks, and improve the performance of semantic segmentation tasks of medical super-resolution images.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical super-resolution image semantic segmentation, and in particular relates to a method and system for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping. Background Art

[0002] Automated and precise organ segmentation based on medical images (i.e., semantic segmentation) is an important research area in clinical imaging. It helps to quickly extract the anatomical structure of the lesion area, reduce the burden on radiologists, and improve the efficiency of clinical data analysis. Currently, the segmentation results of high-resolution medical images are highly accurate, but the acquisition and calculation are time-consuming and resource-intensive. Low-resolution images are quickly acquired and calculated, but the segmentation results are less accurate. Therefore, in order to achieve excellent segmentation results without relying on high-resolution input, how to extract more features from low-resolution medical images to complete the high-resolution output of semantic segmentation labels is a difficult and hot research topic.

[0003] With the development of deep learning technology, researchers have recently proposed a two-stream task framework that uses image segmentation as the primary task and super-resolution generation as an auxiliary task. By leveraging constraints between these tasks, the super-resolution task guides the image segmentation task, improving the segmentation performance of low-resolution images. Specifically, the framework's primary task takes a low-resolution image as input and outputs a high-resolution segmentation label, while the auxiliary task outputs a high-resolution image. While the two-stream framework has achieved promising results in the field of image segmentation, it still faces challenges such as the difficulty of training the super-resolution task, limited single-resolution information, and incomplete feature mapping for both tasks.

[0004] Specifically, existing two-stream frameworks directly map a single low-resolution image to a high-resolution image. High-resolution reconstruction tasks are difficult to train, and the model has less resolution information to learn. If the model can be provided with information at multiple resolutions and a simpler image reconstruction task replaces the super-resolution image generation task, it will be possible to extract more image details from image reconstruction tasks at different resolutions, improving the model's understanding of the different hierarchical structures of the image while reducing the difficulty of model training. Furthermore, most existing two-stream frameworks perform feature similarity mapping between super-resolution and segmentation tasks at a single layer of the model decoder. However, different layers of the model extract features from different angles. Constraining feature similarity at multiple levels is expected to more comprehensively extract image information.

[0005] Considering the importance of different image resolution information and model-level features to the extraction of fine-grained structural information of images, there is an urgent need for a medical super-resolution image semantic segmentation method and system based on random super-resolution learning assistance and multi-layer feature mapping to address the shortcomings of existing technologies. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes an abdominal organ segmentation method and system based on random super-resolution learning assistance and multi-layer feature mapping to solve the problems existing in the above-mentioned prior art and achieve accurate super-resolution segmentation of medical images.

[0007] The present invention provides an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping, comprising:

[0008] Obtain the image to be segmented;

[0009] Inputting the image to be segmented into a segmentation model to obtain semantic segmentation labels in the low-resolution image dimension, wherein the segmentation model is trained using a training set and assisted by a multi-resolution image reconstruction module, wherein the multi-resolution image reconstruction module is used to decode the high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstructed features;

[0010] The semantic segmentation labels under the low-resolution image dimension are processed to obtain semantic segmentation labels under the high-resolution image dimension.

[0011] Optionally, obtaining the training set includes:

[0012] Obtain original high-resolution image data;

[0013] Cropping and scaling the original high-resolution image data, and unifying the dimensions to obtain a low-resolution image;

[0014] The training set is obtained based on the low-resolution image.

[0015] Optionally, the segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0016] The multi-resolution image data acquisition module is used to acquire low-resolution image data of uniform dimension;

[0017] The image encoder module is used to encode the image data to obtain high-level feature representation;

[0018] The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features;

[0019] The semantic output module is used to output semantic labels.

[0020] Optionally, the multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels;

[0021] The semantic segmentation decoder unit is used to decode the high-level feature representation to generate semantic segmentation labels;

[0022] The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label;

[0023] The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception.

[0024] Optionally, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for the reconstructed image and the original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0025] The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image;

[0026] The error calculation unit between the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image;

[0027] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain reconstruction features.

[0028] Optionally, fusing different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain the reconstruction features includes:

[0029] The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ..., G n ;

[0030] To G n The layer performs a global average pooling operation on the channel, n-1 The spatial global average pooling operation is performed on the layer, and the two feature layers after the operation are fused to obtain the reconstructed features.

[0031] Optionally, using a multi-resolution image reconstruction module to assist in training to obtain the segmentation model includes:

[0032] Obtaining a mapping loss based on the reconstruction features and the semantic features;

[0033] Calculating the total loss of the segmentation model based on the mapping loss, combining the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label;

[0034] The segmentation model is obtained according to the total loss of the segmentation model.

[0035] The present invention also provides an abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping, comprising: an image acquisition module, an image processing module and an image output module;

[0036] The image acquisition module is used to acquire low-resolution images;

[0037] The image processing module is used to input the low-resolution image into a segmentation model to obtain semantic segmentation labels in the low-resolution image dimension, wherein the segmentation model is obtained by auxiliary training using a multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used to decode the high-level feature representation of the low-resolution image to obtain a multi-resolution reconstructed image and reconstructed features;

[0038] The image output module is used to output the semantic segmentation label under the high-resolution image according to the semantic segmentation label under the low-resolution image dimension.

[0039] Optionally, a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module, and a semantic output module;

[0040] The multi-resolution image data acquisition module is used to acquire image data with different resolutions;

[0041] The image encoder module is used to encode the image data to obtain high-level feature representation;

[0042] The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features;

[0043] The semantic output module is used to output semantic labels.

[0044] Optionally, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for the reconstructed image and the original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0045] The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image;

[0046] The error calculation unit between the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image;

[0047] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain reconstruction features.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] 1. This method obtains training data of different resolutions through random cropping and downsampling, and can extract anatomical structure information from images of multiple resolutions. This method can quickly achieve resolution enhancement and improve the ability to fuse and utilize information of different resolutions.

[0050] 2. This paper uses spatial and channel attention perception to perform feature mapping and fusion on different layers of features in the image reconstruction task decoder and the semantic segmentation task decoder. It directs channel attention to shallow features to capture a wider range of backgrounds, and directs spatial attention to deep features to recognize detailed patterns and textures.

[0051] 3. This paper designs four loss functions to evaluate the intra-task and inter-task losses of image reconstruction and semantic segmentation tasks from multiple perspectives. This allows the model to capture more high-resolution and semantic segmentation information, strengthening the potential connection between image reconstruction and semantic segmentation tasks.

[0052] In general, the present invention improves the performance of medical super-resolution image semantic segmentation tasks through methods such as fast resolution enhancement, spatial and channel attention perception, image reconstruction and semantic segmentation mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0054] Figure 1 This is a flow chart of a method for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping according to an embodiment of the present invention;

[0055] Figure 2 This is a model framework diagram of an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of a decoder multi-layer feature fusion method according to an embodiment of the present invention;

[0057] Figure 4 This is a comparison chart of the segmentation effects of different algorithm models in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0059] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] This paper proposes a method for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping. Figure 1 As shown, the specific steps include:

[0061] Obtain the image to be segmented;

[0062] The image to be segmented is input into the segmentation model to obtain semantic segmentation labels in the low-resolution image dimension. The segmentation model is trained using the training set and assisted by the multi-resolution image reconstruction module. The multi-resolution image reconstruction module is used to decode the high-level feature representation of the image to be segmented and obtain a multi-resolution reconstructed image and reconstructed features.

[0063] The semantic segmentation labels under the low-resolution image dimension are processed to obtain the semantic segmentation labels under the high-resolution image.

[0064] Specifically, the input data is a low-resolution image dataset with an image size of (H / q, W / q) at any q value, and the output is a semantic segmentation label under a high-resolution image with an image size of (H, W).

[0065] Furthermore, obtaining a training set includes:

[0066] Obtain original high-resolution image data;

[0067] Crop and scale the original high-resolution image data and unify the dimensions to obtain a low-resolution image;

[0068] Based on low-resolution images, obtain a training set.

[0069] Specifically, for the original high-resolution image dataset X_high, with an image size of (H, W), each image is randomly cropped into a cropped dataset X_crop of image size (H / p, W / p), where the p value is greater than or equal to 1 and the value is different for each crop, representing the random image resolution scaling ratio. The dataset X_crop is downsampled by a factor of q / p to generate a low-resolution image dataset X_low of image size (H / q, W / q), where the q value is greater than 1 and is determined based on the input image resolution of the test data, representing the image reduction ratio during the construction of the low-resolution dataset. In the process of comparing different method models, the model goal is to perform semantic segmentation of low-resolution images of image size (H / q, W / q) on high-resolution images of image size (H, W).

[0070] Furthermore, the segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0071] A multi-resolution image data acquisition module is used to acquire low-resolution image data of uniform dimension;

[0072] Image encoder module, used to encode image data and obtain high-level feature representation;

[0073] Multi-resolution image semantic segmentation module, which is used to decode the high-level feature representation of image data, generate semantic segmentation labels under multi-resolution images, and obtain semantic features;

[0074] Semantic output module, used to output semantic labels.

[0075] Furthermore, the multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels;

[0076] Semantic segmentation decoder unit, which decodes high-level feature representations to generate semantic segmentation labels;

[0077] The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label;

[0078] The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception.

[0079] Specifically, the multi-resolution image semantic segmentation module includes a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and a semantic segmentation generated label and original label error calculation unit. The output end of the image encoder module is connected to the semantic segmentation decoder unit. The output end of the semantic segmentation decoder unit is connected to the semantic segmentation decoder multi-layer feature fusion unit and the semantic segmentation generated label and original label error calculation unit. The output end of the semantic segmentation decoder multi-layer feature fusion unit is connected to the dual-task feature mapping module. The output end of the semantic segmentation generated label and original label error calculation unit is connected to the model total loss calculation module.

[0080] The semantic segmentation decoder unit is used to decode the high-level feature representation of the data output by the image encoder module to generate semantic segmentation labels;

[0081] The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception;

[0082] Dual-task feature mapping module, used to calculate the similarity of multi-layer fusion feature maps for multi-resolution image reconstruction tasks and semantic segmentation tasks;

[0083] The error calculation unit for semantic segmentation generated labels and original labels is used to calculate the error between the semantic segmentation generated labels and original labels output by the semantic segmentation decoder unit.

[0084] More specifically, the error calculation unit for semantic segmentation generated labels and original labels includes:

[0085] Generate the value of each pixel in the label image based on the output of the semantic segmentation decoder unit And the value y of each pixel in the original true label image n , calculate the Dice coefficient loss L of the two images Dice :

[0086]

[0087] Where N represents the total number of pixels and ε represents a minimum value to avoid the denominator being 0.

[0088] Furthermore, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for the reconstructed image and the original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0089] An image reconstruction decoder unit, which is used to decode the high-level feature representation and reconstruct a multi-resolution image;

[0090] An error calculation unit for the reconstructed image and the original image, used for calculating the error between the multi-resolution reconstructed image and the original image;

[0091] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain the reconstruction features.

[0092] Specifically, the multi-resolution image reconstruction module includes an image reconstruction decoder unit, an error calculation unit for reconstructed images and original images, and an image reconstruction decoder multi-layer feature fusion unit. The output end of the image encoder module is connected to the image reconstruction decoder unit. The output end of the image reconstruction decoder unit is connected to the error calculation unit for reconstructed images and original images and the image reconstruction decoder multi-layer feature fusion unit. The output end of the image reconstruction decoder multi-layer feature fusion unit is connected to the dual-task feature mapping module. The output end of the error calculation unit for reconstructed images and original images is connected to the model total loss calculation module.

[0093] An image reconstruction decoder unit, which is used to decode the high-level feature representation of the data output by the image encoder module and reconstruct a multi-resolution image;

[0094] a reconstructed image and original image error calculation unit, used to calculate the error between the multi-resolution reconstructed image output by the image reconstruction decoder unit and the original image;

[0095] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception.

[0096] Furthermore, the deep and shallow features of the image reconstruction decoder unit are fused through spatial and channel attention perception to obtain the following reconstruction features:

[0097] The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ..., G n ;

[0098] To G n The layer performs a global average pooling operation on the channel, n-1 The spatial global average pooling operation is performed on the layer, and the two feature layers after the operation are fused to obtain the reconstructed features.

[0099] Specifically, the multi-layer feature fusion unit of the image reconstruction decoder includes:

[0100] The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ... from deep to shallow. The multi-layer feature fusion process is carried out in each feature layer in sequence. Each fusion performs spatial global average pooling (i.e. spatial attention) on the previous deep layer feature, and performs channel global average pooling (i.e. channel attention) on the current layer feature. The calculated feature weighted fusion is used as the fusion feature. Taking the fusion of G1 layer and G2 layer as an example, the G1 layer feature is obtained by spatial global average pooling to obtain G 1s , G 1s Through convolution (Conv), Sigmoid (δ), and G1 weighted convolution, we get G 11 , G2 layer features are obtained by channel global average pooling G 2c , G 2c After linear transformation (Linear), ReLU (θ), linear transformation, and Sigmoid calculation, the channel weight G is obtained. 22 , G 11 and G 22 Weighted upsampling (Up) and adding it to the G2 layer yields the fusion feature G of the G1 and G2 layers. 12 :

[0101]

[0102] G 11 =Conv(δ(Conv(G 1s ))⊙G1)

[0103]

[0104] G 22 =δ(Linear(θ(Linear(G2c))))

[0105] G 12 =(Up(G 11 ⊙G 22 ))⊕G2

[0106] Among them, C represents the number of channels, ⊙ represents the Hadamard product, and ⊕ represents the addition operation at the pixel level.

[0107] More specifically, the error calculation unit of the reconstructed image and the original image includes:

[0108] The value of each pixel in the generated image based on the output of the image reconstruction decoder unit And the value x of each pixel in the original image ij , calculate the mean square error loss L of the two images MSE :

[0109]

[0110] Furthermore, the segmentation model is obtained by using a multi-resolution image reconstruction module to assist in training, including:

[0111] Obtain mapping loss based on reconstruction features and semantic features;

[0112] Based on the mapping loss, the total loss of the segmentation model is calculated by combining the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label;

[0113] Get the segmentation model based on the total loss of the segmentation model.

[0114] Specifically, in addition, the pixel-level cross entropy loss L is calculated CE and region-level mutual information loss L RMI , which constitutes the total loss L at the pixel level and region level CE-RMI :

[0115]

[0116] Where λ represents the weight factor, B represents the number of images processed in each batch, Represents the region-level mutual information between the original true labels and the generated labels.

[0117] The dual-task feature mapping module includes:

[0118] Based on the output of the multi-layer feature fusion unit of the image reconstruction decoder, the feature maps of different channels are inner-producted pairwise to obtain the Gram matrix H G Based on the output of the multi-layer feature fusion unit of the semantic segmentation decoder, the feature maps of different channels are inner-producted pairwise to obtain the Gram matrix H S . Further calculate the dual-task mapping loss L for the multi-resolution image reconstruction task and the semantic segmentation task GS :

[0119]

[0120] Where u and v are the dimensions of the Gram matrix.

[0121] More specifically, the model total loss calculation module includes:

[0122] According to L MSE , L Dice and L CE-RMI and L GS , calculate the total model loss L:

[0123] L=L MSE +L Dice +μ(L CE-RMI +L GS )

[0124] Where μ is the weight coefficient, which is less than 1, and is used to generate the total loss L at the pixel level and region level for the semantic segmentation label and the original label. CE-RMI And the dual-task mapping loss L for image reconstruction and semantic segmentation GS Weight reduction is performed to ensure that the model training process is not overly affected by these two parts of loss, and to achieve a trade-off between the contributions of different losses.

[0125] The present invention also provides an abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping, comprising: an image acquisition module, an image processing module and an image output module;

[0126] An image acquisition module, used for acquiring low-resolution images;

[0127] An image processing module, configured to input a low-resolution image into a segmentation model to obtain semantic segmentation labels in the low-resolution image dimension. The segmentation model is trained with the assistance of a multi-resolution image reconstruction module, which is configured to decode the high-level feature representation of the low-resolution image to obtain a multi-resolution reconstructed image and reconstructed features.

[0128] The image output module is used to output the semantic segmentation label under the high-resolution image based on the semantic segmentation label under the low-resolution image dimension.

[0129] Further, a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module;

[0130] A multi-resolution image data acquisition module is used to acquire image data of different resolutions;

[0131] Image encoder module, used to encode image data and obtain high-level feature representation;

[0132] Multi-resolution image semantic segmentation module, which is used to decode the high-level feature representation of image data, generate semantic segmentation labels under multi-resolution images, and obtain semantic features;

[0133] Semantic output module, used to output semantic labels.

[0134] Furthermore, the multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for the reconstructed image and the original image, and an image reconstruction decoder multi-layer feature fusion unit;

[0135] An image reconstruction decoder unit, which is used to decode the high-level feature representation and reconstruct a multi-resolution image;

[0136] An error calculation unit for the reconstructed image and the original image, used for calculating the error between the multi-resolution reconstructed image and the original image;

[0137] The multi-layer feature fusion unit of the image reconstruction decoder is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain the reconstruction features.

[0138] The present embodiment will be described in detail below with reference to the accompanying drawings:

[0139] This embodiment provides an abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping. The overall structure is as follows: Figure 1 As shown, the process includes acquiring an image to be segmented, developing a segmentation model, generating low-resolution semantic segmentation labels, and outputting high-resolution semantic segmentation labels. The segmentation model includes original training data, multi-resolution training data, an image encoder, high-level feature representation, image reconstruction-assisted semantic segmentation decoding, model loss calculation, model training, and parameter storage. In this embodiment, the Synapse dataset is used as an example. This dataset consists of abdominal CT images and involves segmentation of eight organs: the aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas, and stomach. The spatial resolution of the original high-resolution image is (512, 512), and the spatial resolution of the low-resolution image is (256, 256).

[0140] Model framework such as Figure 2 As shown in the figure, the image to be segmented is input into the segmentation model to generate low-resolution semantic segmentation labels. After upsampling, high-resolution semantic segmentation labels are obtained. The segmentation model details include: for the original high-resolution image dataset X_high, with an image size of (512, 512), each image is randomly cropped into a cropped dataset X_crop with an image size of (512 / p, 512 / p), where the value of p is greater than or equal to 1 and less than or equal to 2, and the value of p is different for each crop. The dataset X_crop is downsampled by a factor of 2 / p to generate a low-resolution image dataset X_low with an image size of (256, 256). In the process of comparing different method models, the model goal is to perform semantic segmentation on the low-resolution image of image size (256, 256) with the high-resolution image of image size (512, 512).

[0141] The low-resolution image dataset X_low with image size (256, 256) is input into the image encoder module to form a high-level feature representation of the image data, where the encoders of the multi-resolution image reconstruction task and the multi-resolution image semantic segmentation task share weights.

[0142] The high-level feature representations output by the encoder are input into the decoders of the image reconstruction task and semantic segmentation task, respectively, to generate multi-resolution reconstructed images and organ segmentation labels.

[0143] For the different layers of decoder features in image reconstruction and semantic segmentation tasks, spatial attention and channel attention are used to fuse features of different depths and shallow layers. The feature layer fusion algorithm for the two tasks is the same. Take the image reconstruction task feature fusion as an example:

[0144] The feature layers of the decoder unit are G1, G2, G3, G4, ... from deep to shallow. The multi-layer feature fusion process is carried out in each feature layer in sequence. Each fusion performs spatial global average pooling (i.e. spatial attention) on the previous deep layer feature, and performs channel global average pooling (i.e. channel attention) on the current layer feature. The calculated features are weighted and fused as the fused features. Figure 3 As shown, taking the fusion of G1 layer and G2 layer as an example, the G1 layer features are obtained by spatial global average pooling. 1s , G 1s Through convolution (Conv), Sigmoid (δ), and G1 weighted convolution, we get G 11 , G2 layer features are obtained by channel global average pooling G 2c , G 2c After linear transformation (Linear), ReLU (θ), linear transformation, and Sigmoid calculation, the channel weight G is obtained. 22 , G 11 and G 22 Weighted upsampling (Up) and adding it to the G2 layer yields the fusion feature G of the G1 and G2 layers. 12 :

[0145]

[0146] G 11 =Conv(δ(Conv(G 1s ))⊙G1)

[0147]

[0148] G 22 =δ(Linear(θ(Linear(G2c))))

[0149] G 12 =(Up(G 11 ⊙G 22 ))⊕G2

[0150] Where C represents the number of channels (3 in this embodiment), H and W represent the image dimensions, which are 512 in this embodiment, ⊙ represents the Hadamard product, ⊕ represents the addition operation at the pixel level, and the convolution kernel size is 3×3 in this embodiment.

[0151] After fusing the decoder multi-layer features of the image reconstruction task and the semantic segmentation task, the Gram matrix H of the image reconstruction task is calculated.G and the Gram matrix H for the semantic segmentation task S . Further calculate the dual-task mapping loss L for the image reconstruction task and the semantic segmentation task GS :

[0152]

[0153] Where u and v are the dimension coordinates of the Gram matrix, and in this embodiment, the value is 16.

[0154] The value of each pixel in the generated image based on the output of the image reconstruction task decoder And the value x of each pixel in the original image ij , calculate the mean square error loss L of the two images MSE :

[0155]

[0156] The value of each pixel in the generated label image based on the output of the semantic segmentation task decoder And the value y of each pixel in the original true label image n , calculate the Dice coefficient loss L of the two images Dice :

[0157]

[0158] Where N represents the total number of pixels (the average value in this embodiment is 256×256), and ε represents a minimum value (the average value in this embodiment is 1e-6) to avoid the denominator being 0. In addition, the cross entropy loss L at the pixel level is calculated CE and region-level mutual information loss L RMI , which constitutes the total loss L at the pixel level and region level CE-RMI :

[0159]

[0160] Where λ represents the weight factor (the value in this embodiment is 0.5), B represents the number of images processed in each batch (the value in this embodiment is 8), Represents the region-level mutual information between the original true labels and the generated labels.

[0161] Further calculate the total loss L of the model:

[0162] L=L MSE +L Dice +μ(L CE-RMI +L GS )

[0163] Where μ is the weight coefficient (the value in this embodiment is 0.1), which is used to calculate the total loss L at the pixel level and region level for the semantic segmentation generated labels and the original labels. CE-RMI And the dual-task mapping loss L for image reconstruction and semantic segmentation GS Weight reduction is performed to ensure that the model training process is not overly affected by these two parts of loss, and to achieve a trade-off between the contributions of different losses.

[0164] Finally, the model parameters are trained and updated, and the model’s performance in the abdominal organ segmentation task is tested. Figure 4 As shown, compared with ResNet-50 (average Dice coefficient 76.81%), the Dice coefficient of the model in this embodiment reaches 83.18%.

[0165] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for abdominal organ segmentation based on random super-resolution learning assistance and multi-layer feature mapping, characterized in that: include: Obtain the image to be segmented; Inputting the image to be segmented into a segmentation model to obtain semantic segmentation labels in the low-resolution image dimension, wherein the segmentation model is trained using a training set and assisted by a multi-resolution image reconstruction module, wherein the multi-resolution image reconstruction module is used to decode the high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstructed features; The segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module; The multi-resolution image data acquisition module is used to acquire low-resolution image data of uniform dimension; The image encoder module is used to encode the image data to obtain high-level feature representation; The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features; The semantic output module is used to output semantic labels; The multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels; The semantic segmentation decoder unit is used to decode the high-level feature representation to generate semantic segmentation labels; The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label; The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception to obtain semantic features; The multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for reconstructed image and original image, and an image reconstruction decoder multi-layer feature fusion unit; The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image; The error calculation unit between the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image; The image reconstruction decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain reconstruction features; Auxiliary training using the multi-resolution image reconstruction module includes: Obtaining a mapping loss based on the reconstruction features and the semantic features; Calculating the total loss of the segmentation model based on the mapping loss, combining the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label; Obtaining the segmentation model according to the total loss of the segmentation model; The semantic segmentation labels under the low-resolution image dimension are processed to obtain semantic segmentation labels under the high-resolution image dimension.

2. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 1 is characterized in that: Obtaining the training set includes: Obtain original high-resolution image data; Cropping and scaling the original high-resolution image data, and unifying the dimensions to obtain a low-resolution image; The training set is obtained based on the low-resolution image.

3. The abdominal organ segmentation method based on random super-resolution learning assistance and multi-layer feature mapping according to claim 1, characterized in that: The different deep and shallow layer features of the image reconstruction decoder unit are fused through spatial and channel attention perception to obtain the reconstruction features including: The feature layers of the image reconstruction decoder unit are G1, G2, G3, G4, ..., G n ; To G n The layer performs a global average pooling operation on the channel, n-1 The spatial global average pooling operation is performed on the layer, and the two feature layers after the operation are fused to obtain the reconstructed features.

4. An abdominal organ segmentation system based on random super-resolution learning assistance and multi-layer feature mapping, characterized in that: include: Image acquisition module, image processing module and image output module; The image acquisition module is used to acquire low-resolution images; The image processing module is used to input the low-resolution image into a segmentation model to obtain semantic segmentation labels in the low-resolution image dimension, wherein the segmentation model is trained using a training set and assisted by a multi-resolution image reconstruction module, and the multi-resolution image reconstruction module is used to decode the high-level feature representation of the image to be segmented to obtain a multi-resolution reconstructed image and reconstructed features; The segmentation model includes: a multi-resolution image data acquisition module, an image encoder module, a multi-resolution image semantic segmentation module and a semantic output module; The multi-resolution image data acquisition module is used to acquire low-resolution image data of uniform dimension; The image encoder module is used to encode the image data to obtain high-level feature representation; The multi-resolution image semantic segmentation module is used to decode the high-level feature representation of the image data, generate semantic segmentation labels under the multi-resolution image, and obtain semantic features; The semantic output module is used to output semantic labels; The multi-resolution image semantic segmentation module includes: a semantic segmentation decoder unit, a semantic segmentation decoder multi-layer feature fusion unit, and an error calculation unit for semantic segmentation generated labels and original labels; The semantic segmentation decoder unit is used to decode the high-level feature representation to generate semantic segmentation labels; The error calculation unit of the semantic segmentation generated label and the original label is used to calculate the error between the semantic segmentation label and the original label; The semantic segmentation decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the semantic segmentation decoder unit through spatial and channel attention perception to obtain semantic features; The multi-resolution image reconstruction module includes: an image reconstruction decoder unit, an error calculation unit for reconstructed image and original image, and an image reconstruction decoder multi-layer feature fusion unit; The image reconstruction decoder unit is used to decode the high-level feature representation and reconstruct a multi-resolution image; The error calculation unit between the reconstructed image and the original image is used to calculate the error between the multi-resolution reconstructed image and the original image; The image reconstruction decoder multi-layer feature fusion unit is used to fuse the different deep and shallow layer features of the image reconstruction decoder unit through spatial and channel attention perception to obtain reconstruction features; Auxiliary training using the multi-resolution image reconstruction module includes: Obtaining a mapping loss based on the reconstruction features and the semantic features; Calculating the total loss of the segmentation model based on the mapping loss, combining the error between the multi-resolution reconstructed image and the original image, and the error between the semantic segmentation label and the original label; Obtaining the segmentation model according to the total loss of the segmentation model; The image output module is used to output the semantic segmentation label under the high-resolution image according to the semantic segmentation label under the low-resolution image dimension.

Images