Image processing method and device, and computer readable storage medium

Abstract

The present disclosure provides an image processing method, device and computer-readable storage medium; the method comprises: extracting features of an image to be processed by a main image segmentation model to obtain a feature map of the image to be processed; according to the feature map, performing image segmentation on the segmentation target by a target image segmentation model to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; the image coding model is used to impose shape constraints on the target segmentation result. The present disclosure can improve the accuracy of image segmentation.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, and a computer-readable storage medium. Background Art

[0002] Current deep learning segmentation algorithms usually only perform pixel-level segmentation based on image features, while ignoring the shape of the segmentation results, which may lead to outliers in the segmentation results. For example, when performing image segmentation of organs at risk in medical images, the organ shapes predicted by current deep learning segmentation algorithms are different from medical prior knowledge. For example, the optic chiasm has an X-shaped anatomical structure, but most current deep learning networks that use cross entropy loss functions or Dice loss functions will predict it as a smooth ellipse-like shape, resulting in image segmentation results that are inconsistent with prior knowledge, reducing the accuracy of image segmentation. Summary of the invention

[0003] The embodiments of the present disclosure are intended to provide an image processing method and apparatus and a computer-readable storage medium, which can improve the accuracy of image segmentation.

[0004] The technical solution of the present disclosure is achieved as follows:

[0005] The present disclosure provides an image processing method, the method comprising:

[0006] Extracting features of the image to be processed by the main image segmentation model to obtain a feature map of the image to be processed;

[0007] According to the feature map, the segmentation target is segmented by a target image segmentation model to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; the image coding model is used to perform shape constraints on the target segmentation result.

[0008] In the above solution, before performing image segmentation on the segmentation target by using a target image segmentation model according to the feature map to obtain the target segmentation result, the method further includes:

[0009] Acquire a training sample image set, wherein the training sample image set includes at least one training sample image corresponding to the segmentation target, and the training sample image is labeled with the segmentation target;

[0010] Perform image segmentation on each training sample image through the initial target image segmentation model to obtain a predicted segmentation image;

[0011] According to the labeled segmentation target and the predicted segmentation image, in combination with the initial image coding model, adversarial training is performed on the initial target image segmentation model to obtain a target image segmentation model.

[0012] In the above scheme, the initial target image segmentation model is subjected to adversarial training based on the labeled segmentation target and the predicted segmentation image in combination with the initial image coding model to obtain the target image segmentation model, including:

[0013] Using the initial image coding model, the annotated segmentation target and the predicted segmentation image are encoded to obtain a first spatial shape code and a second spatial shape code; wherein the first spatial shape code is a low-dimensional manifold space representation of the annotated segmentation target; and the second spatial shape code is a low-dimensional manifold space representation of the predicted segmentation image;

[0014] Calculating the coding distance between the first spatial shape coding and the second spatial shape coding by using a preset minimum square error loss function to obtain a prediction error;

[0015] Based on the shape constraint of the prediction error, the network parameters of the initial target image segmentation model are updated, and the above-mentioned adversarial training process is iterated until the final prediction error is less than the preset segmentation error threshold, and then the training process is terminated to obtain the target image segmentation model.

[0016] In the above scheme, it is characterized in that the initial image coding model includes a decoding sub-model, and after the first spatial shape coding and the second spatial shape coding are processed by a preset minimum square error loss function to obtain a prediction error, the method further includes:

[0017] The first spatial shape code and the second spatial shape code are decoded and graphically reconstructed respectively by the decoding sub-model to obtain a first reconstructed image and a second reconstructed image respectively;

[0018] By presetting a reconstruction loss function, the error between the labeled segmentation target and the first reconstructed image, and the error between the predicted segmentation image and the second reconstructed image are combined to obtain a reconstruction error of the initial image coding model.

[0019] In the above scheme, it is characterized in that the network parameters of the initial target image segmentation model are updated based on the shape constraint of the prediction error, and the above adversarial training process is iterated until the final prediction error is less than the preset segmentation error threshold, and the training process is terminated to obtain the target image segmentation model, which includes:

[0020] In the Mth training process, the network parameters of the initial target image segmentation model are updated using the prediction error; wherein M is an integer greater than or equal to 1;

[0021] In the M+1th training, a comprehensive coding error is obtained based on the reconstruction error and the prediction error;

[0022] Using the comprehensive coding error, updating the network parameters of the initial image coding model;

[0023] The above-mentioned adversarial training process is iterated until the final prediction error is less than the preset coding error threshold, thereby obtaining the target coding model and the target image segmentation model.

[0024] In the above solution, the initial image coding model includes a coding sub-model, and the coding sub-model includes at least one convolutional layer and a fully connected layer;

[0025] The encoding of the annotated segmentation target and the predicted segmentation image by using the initial image encoding model to obtain a first spatial shape encoding and a second spatial shape encoding includes:

[0026] According to the preset step size of the at least one convolutional layer, the labeled segmentation target and the predicted segmentation image are compressed by the at least one convolutional layer, and the labeled segmentation target and the compressed predicted segmentation image in the compressed training sample image are encoded by the fully connected layer to obtain the first spatial shape code and the second spatial shape code.

[0027] In the above scheme, the decoding sub-model includes at least one deconvolution layer, a linear rectification function and a batch normalization layer.

[0028] In the above scheme, the image segmentation of the segmented target is performed by a target image segmentation model according to the feature map positioning result to obtain the target segmentation result, including:

[0029] Using a target positioning model, positioning the segmented target according to the feature map to obtain a positioning result of the segmented target;

[0030] According to the positioning result, the segmentation target is image segmented by a target image segmentation model to obtain a target segmentation result.

[0031] In the above solution, the main image segmentation model includes a compression activation module, and the feature extraction of the image to be processed by the main image segmentation model to obtain the feature map of the image to be processed includes:

[0032] Downsampling the image to be processed twice to obtain a downsampled image;

[0033] According to a preset convolution layer expansion rate, convolution processing is performed on the downsampled image to obtain a convolution result;

[0034] Performing residual and compression activation processing on the convolution result through the compression activation module to obtain an activation result;

[0035] Perform multi-scale feature extraction and deconvolution processing on the activation result to obtain a feature map of the image to be processed.

[0036] In the above scheme, the feature map includes N layers of feature maps, where N is an integer greater than 1, and the segmentation target is located according to the feature map through the target positioning model to obtain the positioning result of the segmentation target, including:

[0037] By using the target positioning model, feature recognition and normalization processing are performed on the N-th layer feature map to obtain the probability that each pixel in the N-th layer feature map belongs to the center position of the segmented target;

[0038] Image segmentation is performed on the N-th layer feature map according to the probability to obtain a positioning result of the segmented target.

[0039] In the above scheme, the image segmentation of the segmented target is performed by a target image segmentation model according to the positioning result to obtain the target segmentation result, including:

[0040] Predicting the shape of the segmented target by using a target image segmentation model to obtain a cutting range of the segmented target;

[0041] According to the positioning result of the segmented target and the cropping range, the Nth layer feature map is cropped to obtain the target segmentation result.

[0042] In the above solution, after extracting features from the image to be processed by the main image segmentation model to obtain a feature map of the image to be processed, the method further includes:

[0043] According to a preset main segmentation target, the feature map is positioned and segmented by the main image segmentation model to obtain a main segmentation result corresponding to the main segmentation target;

[0044] After performing image segmentation on the segmented target by using a target image segmentation model according to the positioning result and obtaining the target segmentation result, the method further comprises:

[0045] The main segmentation result and the target segmentation result are fused in the image to be processed to obtain an image segmentation result of the image to be processed.

[0046] The present disclosure provides an image processing device, the device comprising:

[0047] A main image segmentation model is used to extract features of the image to be processed to obtain a feature map of the image to be processed;

[0048] A target image segmentation model is used to perform image segmentation on the segmentation target according to the feature map to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; the image coding model is used to perform shape constraints on the target segmentation result.

[0049] In the above device, the image processing device further includes a training unit, wherein:

[0050] The training unit is used to obtain a set of training sample images before performing image segmentation on the segmentation target through the target image segmentation model according to the feature map to obtain the target segmentation result, wherein the training sample image set includes at least one training sample image corresponding to the segmentation target, and the training sample image has a labeled segmentation target; perform image segmentation on each training sample image through the initial target image segmentation model to obtain a predicted segmentation image; and perform adversarial training on the initial target image segmentation model according to the labeled segmentation target and the predicted segmentation image in combination with the initial image coding model to obtain a target image segmentation model.

[0051] In the above-mentioned device, the training unit is also used to use the initial image coding model to encode the labeled segmentation target and the predicted segmentation image to obtain a first spatial shape code and a second spatial shape code; wherein the first spatial shape code is a low-dimensional manifold space representation of the labeled segmentation target; the second spatial shape code is a low-dimensional manifold space representation of the predicted segmentation image; by using a preset minimum square error loss function, the coding distance between the first spatial shape code and the second spatial shape code is calculated to obtain a prediction error; based on the shape constraint of the prediction error, the network parameters of the initial target image segmentation model are updated, and the above-mentioned adversarial training process is iterated until the final prediction error is less than a preset segmentation error threshold, and then the training process is terminated to obtain the target image segmentation model.

[0052] In the above-mentioned device, the initial image coding model includes a decoding sub-model, and the training unit is also used to, after the first spatial shape code and the second spatial shape code are processed by a preset minimum square error loss function to obtain a prediction error, decode and graphically reconstruct the first spatial shape code and the second spatial shape code respectively through the decoding sub-model to obtain a first reconstructed image and a second reconstructed image respectively; and combine the error between the labeled segmentation target and the first reconstructed image, and the error between the predicted segmentation image and the second reconstructed image through a preset reconstruction loss function to obtain a reconstruction error of the initial image coding model.

[0053] In the above-mentioned device, the training unit is also used to update the network parameters of the initial target image segmentation model using the prediction error during the M-th training process; wherein M is an integer greater than or equal to 1; in the M+1-th training, a comprehensive coding error is obtained based on the reconstruction error and the prediction error; the network parameters of the initial image coding model are updated using the comprehensive coding error; and the above-mentioned adversarial training process is iterated until the final prediction error is less than a preset coding error threshold, thereby obtaining the target coding model and the target image segmentation model.

[0054] In the above-mentioned device, the initial image coding model includes a coding sub-model, and the coding sub-model includes at least one convolutional layer and a fully connected layer; the training unit is also used to compress the training sample image and the predicted segmentation image through the at least one convolutional layer according to the preset step size of the at least one convolutional layer, and encode the labeled segmentation targets in the compressed training sample image and the compressed predicted segmentation image through the fully connected layer to obtain the first spatial shape code and the second spatial shape code.

[0055] In the above device, the decoding sub-model includes at least one deconvolution layer, a linear rectification function and a batch normalization layer.

[0056] In the above-mentioned device, the image processing device also includes a target positioning model, which is used to locate the segmentation target according to the feature map to obtain the positioning result of the segmentation target; based on the positioning result, the segmentation target is image segmented by the target image segmentation model to obtain the target segmentation result.

[0057] In the above-mentioned device, the main image segmentation model includes a compression activation module, and the main image segmentation model is also used to downsample the image to be processed twice to obtain a downsampled image; convolve the downsampled image according to a preset convolution layer expansion rate to obtain a convolution result; perform residual and compression activation processing on the convolution result through the compression activation module to obtain an activation result; perform multi-scale feature extraction and deconvolution processing on the activation result to obtain a feature map of the image to be processed.

[0058] In the above device, the feature map includes N layers of feature maps, N is an integer greater than 1, and the target positioning model is also used to perform feature recognition and normalization processing on the Nth layer of feature maps to obtain the probability that each pixel in the Nth layer of feature maps belongs to the center position of the segmented target; the Nth layer of feature maps is segmented according to the probability to obtain the positioning result of the segmented target.

[0059] In the above-mentioned device, the target positioning model is also used to predict the shape of the segmented target through the target image segmentation model to obtain the cropping range of the segmented target; according to the positioning result of the segmented target and the cropping range, the Nth layer feature map is cropped to obtain the target segmentation result.

[0060] In the above device, the main image segmentation model is further used to extract features from the image to be processed, obtain the feature map of the image to be processed, and then locate and segment the feature map according to a preset main segmentation target to obtain a main segmentation result corresponding to the main segmentation target;

[0061] The image processing device also includes a fusion unit, which is used to perform image segmentation on the segmented target according to the positioning result in the target image segmentation model, and after obtaining the target segmentation result, fuse the main segmentation result and the target segmentation result in the image to be processed to obtain the image segmentation result of the image to be processed.

[0062] The present disclosure provides an image processing device, the device comprising:

[0063] Memory for storing computer programs;

[0064] The processor is used to implement any of the above-mentioned image processing methods when executing the computer program stored in the memory.

[0065] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for causing a processor to execute the program to implement any of the above-mentioned image processing methods.

[0066] The disclosed embodiments provide an image processing method and device and a computer-readable storage medium. Since the image coding model can impose shape constraints on the target segmentation result, the target image segmentation model obtained by adversarial training with the image coding model can learn the accurate shape of the segmented target. In this way, when the image processing device uses the target image segmentation model to perform image segmentation on the segmented target, a more accurate target segmentation result can be obtained, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 1 ;

[0068] Figure 2 A schematic diagram of the structure of a main image segmentation model provided in an embodiment of the present disclosure;

[0069] Figure 3 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 2 ;

[0070] Figure 4 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 3 ;

[0071] Figure 5 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 4 ;

[0072] Figure 6 A schematic diagram of the training process of the image coding model provided in the embodiment of the present disclosure;

[0073] Figure 7 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 5 ;

[0074] Figure 8 A schematic diagram of the structure of a compression activation module provided in an embodiment of the present disclosure;

[0075] Fig. 9 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 6 ;

[0076] Fig.10 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 7

[0077] Fig.11 A schematic diagram of a network architecture of an image processing device provided by an embodiment of the present disclosure;

[0078] Fig.12Schematic diagram of the image segmentation effect of the image processing method provided in the embodiment of the present disclosure Figure 1 ;

[0079] Fig.13 Schematic diagram of the image segmentation effect of the image processing method provided in the embodiment of the present disclosure Figure 2 ;

[0080] Fig.14 A schematic diagram of the structure of the image processing device provided in the embodiment of the present disclosure Figure 1 ;

[0081] Fig.15 A schematic diagram of the structure of the image processing device provided in the embodiment of the present disclosure Figure 2 . DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

[0083] At present, for accurate image segmentation methods, especially in the medical field, when planning radiotherapy clinically, there are more than twenty organs at risk that need to be taken into account. Doctors are usually required to outline them on the CT images based on their medical prior knowledge. However, labeling on three-dimensional CT images is usually time-consuming and laborious. Due to the complex anatomical structure of the head and neck, it usually takes a professional doctor more than 2.5 hours to outline a patient's organ. At the same time, due to subjective factors, different doctors may not outline the same patient. In addition, due to the insensitivity of CT to soft tissue, many organs have low contrast with surrounding tissues and unclear boundaries, which may cause the segmentation algorithm to produce incorrect segmentation results.

[0084] Therefore, a fast, efficient, high-performance, and robust computer-assisted segmentation algorithm can greatly reduce the workload of doctors, improve the speed and quality of radiotherapy planning, and improve the effectiveness of radiotherapy.

[0085] Current deep learning segmentation algorithms usually only perform pixel-level segmentation based on image features, while ignoring the shape of the segmentation results. This may lead to outliers in the segmentation results, and the predicted organ shape is different from the medical prior knowledge. For example, the optic chiasm has an X-shaped anatomical structure, but most deep learning networks that use the cross entropy loss function or the Dice loss function currently predict it as a smooth ellipse-like shape, resulting in the organ shape obtained by the current image segmentation method not being consistent with the medical prior knowledge, and the image segmentation accuracy is low.

[0086] Figure 1is an optional flow chart of the method provided by the embodiment of the present disclosure, which will be combined with Figure 1 The steps shown are explained.

[0087] S101, extracting features of the image to be processed by using a main image segmentation model to obtain a feature map of the image to be processed.

[0088] The image processing method provided in the embodiments of the present disclosure is applicable to image segmentation scenarios, such as medical image segmentation, picture effect processing, or other scenarios requiring image segmentation, and the embodiments of the present disclosure are not limited thereto.

[0089] In the disclosed embodiment, the image processing device can extract features of the image to be processed through the main image segmentation model to obtain a feature map of the image to be processed. The main image segmentation model is a trained neural network model, which is used to identify the features of each pixel in the image to be processed, and mark the recognition result, usually the probability of a certain category to which the pixel belongs, on the corresponding pixel, thereby obtaining a feature map of the image to be processed.

[0090] In the disclosed embodiment, the main image segmentation model can use relevant feature extraction technology to extract the feature map of the image to be processed. For example, the feature map of the image to be processed can be extracted based on artificially designed features such as local brightness features of the image, organ shape features, etc.

[0091] In some embodiments, the image to be processed may be a medical image containing organs at risk (OAR), and the main image segmentation model may be a fully convolutional neural network modified from a 3D U-Net based on an encoder-decoder architecture, which is used to learn multi-scale features of medical images. The image processing device inputs a medical image, such as a CT original image, into the main image segmentation model, and the output of the main image segmentation model is a probability tensor of each pixel belonging to each organ.

[0092] In the disclosed embodiment, for segmentation targets with clear boundaries and large volumes, such as large organs in CT images, the segmentation results can be output through the main image segmentation model.

[0093] In the disclosed embodiment, in order to improve the segmentation accuracy of the main image segmentation model, the image processing device can use a residual module (Residue Block) including a convolution layer, a linear rectification function, and a batch normalization layer as the backbone network of the main image segmentation model, and further add a squeeze-and-excitation module (Squeeze-and-excitationModule, SE module) as a feature-level attention mechanism. Furthermore, in order to reduce the loss of details caused by downsampling, only two downsamplings can be used in the main image segmentation model.

[0094] In the disclosed embodiment, in order to learn multi-scale features, the image processing device may use a densely connected atrous spatial pyramid pooling (DenseASPP) as the network architecture of the main image segmentation model.

[0095] In some embodiments, the network architecture of the main image segmentation model can be as follows Figure 2 As shown. Figure 2 In the example, the main image segmentation model first downsamples the input image to be processed twice to obtain a downsampled image of smaller size; the downsampled image is convolved once with a preset dilation rate d of 2 to expand the receptive field of the downsampled image, and the features of different scales of the downsampled image are learned through different preset convolution layer dilation rates d preset in DenseASPP, such as d of 3, 6, 12 and 18; it can be seen that the larger the preset convolution layer dilation rate (dilation rate), the larger the scale of the features that the main image segmentation model can learn. The main image segmentation model upsamples the learned features of different scales through a deconvolution, and fuses the upsampled features with the features of the same scale through a cross-layer connection, and then performs the same upsampling and fusion process again to obtain the feature map finally extracted from the image to be processed. Through the above process, the main image segmentation model can capture and learn features of different scales to fuse multi-scale features and ensure a sufficiently large receptive field of the convolution kernel by setting the preset convolution layer dilation rate, thereby enhancing the feature expression ability of the main image segmentation model.

[0096] S102. According to the feature map, image segmentation is performed on the segmentation target through a target image segmentation model to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; and the image coding model is used to perform shape constraints on the target segmentation result.

[0097] In the disclosed embodiment, the image processing device can identify the required segmentation target from the feature map and perform image segmentation based on the obtained feature map through the target image segmentation model to obtain the target segmentation result.

[0098] In the disclosed embodiment, the target image segmentation model is a neural network model obtained by adversarial training with an image coding model.

[0099] In the disclosed embodiment, since the shape of the segmentation target is difficult to describe quantitatively with numerical values, the image processing device can encode the shape of the segmentation target annotated in the training sample image during the adversarial training stage through the image coding model, and characterize the shape of the annotated segmentation target in the form of coding. Among them, the shape boundary of the segmentation target annotated in the training sample image can be manually outlined based on prior knowledge. In this way, for some segmentation targets with unclear boundaries in the actual image, the image coding model uses the shape of the segmentation target in the learned prior knowledge to impose shape constraints on the target segmentation result output by the target image segmentation model that is jointly adversarially trained with it, so that the target segmentation result output by the target image segmentation model conforms to the prior knowledge.

[0100] In the disclosed embodiment, the image coding model can be jointly trained with the target image segmentation model for adversarial training. During the adversarial training, the similarity loss function between the network predicted shape and the true shape is minimized as the objective function of the initial target image segmentation model, and the similarity loss function between the network predicted shape and the true shape is maximized as the objective function of the initial image coding model; in this way, through adversarial training, the initial image coding model can better capture the difference between the network predicted shape and the true shape, and provide a more effective supervisory signal to the initial target image segmentation model. When the initial target image segmentation model can make the initial image coding model unable to distinguish whether the input is a shape predicted by the network or a true annotated shape, it can be regarded as achieving the goal of adversarial training, and the trained image coding model and target image segmentation model are obtained. In this way, when the target image segmentation model is applied for image segmentation, the shape constraints learned by the target image segmentation model during the adversarial training process can be used to improve the accuracy of the target segmentation result, thereby improving the accuracy of the segmentation of the target image segmentation model.

[0101] It is understandable that, since the target image segmentation model learns the shape constraints on the segmentation target during the adversarial training with the image coding model, a more accurate target segmentation result can be obtained, thereby improving the accuracy of image segmentation.

[0102] In the embodiment of the present disclosure, for the adversarial training process between the target image segmentation model and the image coding model, before S102, the following can also be done: Figure 3 As shown, including S201-S203, as follows:

[0103] S201 . Obtain a training sample image set, wherein the training sample image set includes at least one training sample image corresponding to a segmentation target, and the training sample image carries a pre-labeled segmentation target.

[0104] S202: Perform image segmentation on each training sample image using the initial target image segmentation model to obtain a predicted segmented image.

[0105] In the disclosed embodiment, the image processing device can establish an initial target image segmentation model, wherein the weights of the network parameters of each layer in the initial target image segmentation model are initial values. In one round of training, the image processing device uses the initial target image segmentation model to perform image segmentation on a training sample image in the training sample image set to obtain a predicted segmented image, thereby starting adversarial training of the initial target image segmentation model.

[0106] S203. According to the labeled segmentation target and the predicted segmentation image, combined with the initial image coding model, adversarial training is performed on the initial target image segmentation model to obtain a target image segmentation model.

[0107] In the disclosed embodiment, the predicted segmented image characterizes the effect of image segmentation using the initial target image segmentation model. The image processing device may compare the predicted segmented image with the labeled segmentation target on the training sample image to understand the training effect of the initial target image segmentation model after each round of training, and before the next training starts, the network weights of each layer of the initial target image segmentation model are updated according to the shape constraints given by the initial image coding model, so that the predicted segmented image output by the initial target image segmentation model in the next training can be closer to the labeled segmentation target. The image processing device continues the above iterative training process until the image coding model cannot distinguish between the predicted segmented image and the labeled segmentation target. The image processing device considers that the training has reached the preset training goal, ends the iterative training and obtains the target image segmentation model.

[0108] In the disclosed embodiment, the main function of the image coding model is to capture the difference between the predicted segmented image and the annotated segmented target as much as possible. As long as the image coding model can distinguish the predicted segmented image from the annotated segmented target, it means that the image segmentation accuracy of the initial target image segmentation model has not yet reached the requirements of the preset training target, and it is necessary to continue the next round of training. Therefore, the image coding model can supervise the training process of the initial target image segmentation model. To this end, in order to improve the ability of the image coding model to capture the difference between the predicted segmented image and the annotated segmented target, the image coding model will also first use the initial image coding model to start adversarial training with the initial target image coding model, and during the adversarial training process, it will alternately update its respective network model parameters with the initial target image coding model, and obtain the target image coding model when the preset training target is finally reached.

[0109] In the embodiments of the present disclosure, see Figure 4 , Figure 3The specific implementation process of S203 shown in the figure can be implemented through S2031-S2033, which will be explained in combination with each step.

[0110] S2031. Encode the labeled segmentation target and the predicted segmentation image using the initial image coding model to obtain a first spatial shape code and a second spatial shape code; wherein the first spatial shape code is a low-dimensional manifold space representation of the labeled segmentation target; and the second spatial shape code is a low-dimensional manifold space representation of the predicted segmentation image.

[0111] In the embodiment of the present disclosure, in order to accurately characterize the shape of the pre-labeled segmentation target and the shape of the segmentation result contained in the predicted segmentation image, the image processing device can first use the initial image coding model to encode the labeled segmentation target in the sample image to obtain a first spatial shape code; the image processing device uses the initial image coding model to encode the predicted segmentation image to obtain a second spatial shape code.

[0112] In the disclosed embodiment, the initial image coding model includes a coding sub-model, which includes at least one convolutional layer and a fully connected layer; the coding sub-model can compress the sample image and the predicted segmentation image through at least one convolutional layer according to a preset step size of the at least one convolutional layer, and encode the labeled segmentation targets in the compressed sample image and the compressed predicted segmentation image through the fully connected layer to obtain a first spatial shape code and a second spatial shape code.

[0113] In the disclosed embodiment, when the input of the initial image coding model is a sample image, the output is a low-dimensional manifold representation of the shape of the annotated segmentation target, so that the initial image coding model can learn the latent space representation of the shape of the annotated segmentation target through training. In some embodiments, when the sample image is a medical image with manually annotated organs at risk, the shape of the manually annotated organs at risk can be learned in the initial image coding model, so as to constrain the shape of the predicted segmentation image output by the initial image coding model through the shape of the organs at risk in the prior knowledge, thereby integrating the shape constraint into the learning and training of the initial image coding model.

[0114] S2032. Calculate the coding distance between the first spatial shape coding and the second spatial shape coding by using a preset minimum square error loss function to obtain a prediction error.

[0115] In the disclosed embodiment, since the first spatial shape code can characterize the shape of the labeled segmentation target, and the second spatial shape code can characterize the shape of the predicted segmentation image, the image processing device can calculate the coding distance between the first spatial shape code and the second spatial shape code by presetting the minimum square error loss function as the prediction error, which is used to characterize the gap between the predicted segmentation image output by the initial target image segmentation model and the labeled segmentation target.

[0116] In the embodiment of the present disclosure, the preset minimum square error loss function can be shown as formula (1), as follows:

[0117]

[0118] In formula (1), x is the sample image, p data is the distribution function of the sample image, y is the labeled segmentation target, p gt is the distribution function of the labeled segmentation target, D is the initial image encoding model, G is the initial target image segmentation model, G(x) is the predicted segmentation image, and D latent (y) is the first spatial shape code, D latent (G(x)) is the second spatial shape code, L shape is the prediction error. During the training process, the initial image coding model needs to maximize the distance between the first spatial shape coding and the second spatial shape coding through formula (1) to distinguish the difference between the predicted segmented image and the labeled segmented target to the greatest extent, while the initial target image segmentation model needs to minimize the distance between the first spatial shape coding and the second spatial shape coding through formula (1) to make the initial image coding model unable to distinguish the predicted segmented image from the labeled segmented target in adversarial training, so as to achieve the preset training goal.

[0119] S2033. Based on the shape constraint of the prediction error, the network parameters of the initial target image segmentation model are updated, and the above adversarial training process is iterated until the final prediction error is less than the preset segmentation error threshold, and the training process is terminated to obtain the target image segmentation model.

[0120] In the disclosed embodiment, since the prediction error characterizes the difference between the predicted segmented image and the annotated segmented target, the image processing device can adjust the update step size based on the shape constraint of the prediction error during the process of updating the network parameters of the initial target image segmentation model. The image processing device iterates in the above manner during each training process until the final prediction error is less than the preset segmentation error threshold, indicating that the predicted segmented image and the annotated segmented target are very close at this time, that is, the image segmentation accuracy of the initial target image segmentation model has reached the preset training target, so the image processing device can end the training process and obtain the target image segmentation model.

[0121] In the embodiment of the present disclosure, for the stage of training the image coding model in adversarial training, see Figure 5 ,exist Figure 4 After S2032 shown in FIG. 1 , S301-S302 may also be performed, and each step will be described in conjunction with the following steps:

[0122] S301 . Decode and reconstruct the first spatial shape code and the second spatial shape code respectively through a decoding sub-model, and obtain a first reconstructed image and a second reconstructed image accordingly.

[0123] In the disclosed embodiment, the initial image coding model also includes a decoding sub-model. In order to improve the learning ability of the initial image coding model for the shape features of the labeled segmentation target during training, the image processing device can decode and graphically reconstruct the first spatial shape coding and the second spatial shape coding respectively through the decoding sub-model, and obtain the first reconstructed image and the second reconstructed image accordingly. By comparing the difference between the first reconstructed image and the second reconstructed image obtained after encoding and decoding and the original labeled segmentation target and the predicted segmentation image, the network parameters of the initial image coding model are updated, and the learning ability of the initial image coding model for the shape of the labeled segmentation target is gradually improved during training.

[0124] In the disclosed embodiment, the decoding submodel includes at least one deconvolution layer, a linear rectification function, and a batch normalization layer. The image processing device can reconstruct the annotated segmentation target corresponding to the first spatial shape encoding and the predicted segmentation image corresponding to the second spatial shape encoding from the latent space representation through at least one deconvolution layer, a linear rectification function, and a batch normalization layer in the decoding submodel, thereby obtaining a first reconstructed image and a second reconstructed image.

[0125] In the disclosed embodiment, the first reconstructed image is an image obtained by encoding, decoding and reconstructing the marked segmented target by the initial image coding model, and the second reconstructed image is an image obtained by encoding, decoding and reconstructing the predicted segmented image by the initial image coding model.

[0126] S302: By using a preset reconstruction loss function, the error between the labeled segmentation target and the first reconstructed image and the error between the predicted segmentation image and the second reconstructed image are combined to obtain a reconstruction error of the initial image coding model.

[0127] In the disclosed embodiment, the image processing device can calculate the error between the labeled segmentation target and the first reconstructed image and the error between the predicted segmentation image and the second reconstructed image through a preset reconstruction loss function, and combine the two errors to obtain the reconstruction error of the initial image coding model.

[0128] In the embodiment of the present disclosure, the preset reconstruction loss function may be as shown in formula (2), as follows:

[0129]

[0130] In formula (2), D(y) is the first reconstructed image, D(G(x)) is the second reconstructed image, and L rec The image processing device calculates the error between the labeled segmentation target and the first reconstructed image, and the error between the predicted segmentation image and the second reconstructed image respectively through formula (2), and adds them together to obtain the reconstruction error.

[0131] In the embodiments of the present disclosure, see Figure 5 , Figure 4 S2033 shown in the figure can be implemented through S303-S306, which will be described in combination with each step:

[0132] S303. In the Mth training process, the prediction error is used to update the network parameters of the initial target image segmentation model; wherein M is an integer greater than or equal to 1.

[0133] In the disclosed embodiment, in the actual adversarial training process, the updating of network parameters is performed alternately. The image processing device will first update the network parameters of the initial target image segmentation model, use the initial target image segmentation model with updated network parameters for the next round of training, and update the network parameters of the initial image coding model according to the results of the next round of training.

[0134] In the disclosed embodiment, for the Mth training process, the image processing device may update the network parameters of the initial target image segmentation model using the prediction error according to the preset optimization function corresponding to the initial target image segmentation model, where M is an integer greater than or equal to 1.

[0135] In the embodiment of the present disclosure, the preset optimization function corresponding to the initial target image segmentation model can be shown as formula (3), as follows:

[0136]

[0137] L seg =-L focal +λL dice (4)

[0138] In formula (3), λ1 is the balance factor used to balance the prediction error L shape and segmentation error L seg The two loss terms can be set to 5 according to empirical values. segThe loss function of the main image segmentation model can be obtained by formula (4). The image processing device can use formula (3) to minimize the sum of the prediction error and the segmentation error L G As the goal, the network parameters of the initial target image segmentation model are updated so that the predicted segmentation image output by the initial target image segmentation model in the next round of training, i.e., the M+1th training, is further close to the labeled segmentation target.

[0139] In formula (4), L focal Ld is the focal loss function used when training the main image segmentation model, which is used to increase the weight of the misclassified samples in training and play a focusing role; ice is a generalized Dice loss function used when training the main image segmentation model, λ is a weight for balancing the proportions of the two losses in the total loss, and in some embodiments, λ may be 1.

[0140] S304. In the M+1th training, a comprehensive coding error is obtained based on the reconstruction error and the prediction error.

[0141] S305: Using the comprehensive coding error, update the network parameters of the initial image coding model.

[0142] In the disclosed embodiment, after the image processing device completes updating the network parameters of the initial image segmentation model, it enters the M+1th training. In the M+1th training, the image processing device obtains a comprehensive coding error based on the reconstruction error and prediction error calculated in the M+1th training and according to a preset optimization function corresponding to the initial image coding model, and uses the comprehensive coding error to update the network parameters of the initial image coding model.

[0143] In the embodiment of the present disclosure, the preset optimization function corresponding to the initial image coding model may be as shown in formula (5), as follows:

[0144]

[0145] In formula (5), λ2 is the disturbance factor. Since the predicted segmented image output by the initial target image segmentation model will gradually approach the labeled segmented target during the training process, if the initial image encoding model cannot effectively distinguish the predicted segmented image from the labeled segmented target, it will not be able to provide effective supervision for the training process of the target image segmentation model. The addition of the disturbance factor can appropriately increase the difference between the predicted segmented image and the labeled segmented target, which is conducive to improving the effect of adversarial training. In some embodiments, the value of λ2 is generally not more than 0.001, so as to avoid excessive λ2 increasing the instability of training. L Dis the comprehensive coding error, which reflects the gap between the prediction error and the reconstruction error. The image processing device uses formula (5) to maximize the comprehensive error L D As the goal, the network parameters of the initial image segmentation model are adjusted. So far, through two consecutive rounds of adversarial training, the network parameters of the initial target image segmentation model and the initial image coding model are updated respectively.

[0146] In some embodiments, the adversarial training process of the initial image coding model can be as follows Figure 6 As shown. Figure 6 In the example, the initial image coding model encodes the labeled segmentation target and the predicted segmentation image respectively to obtain the first spatial shape code and the second spatial shape code. The initial image coding model obtains the reconstruction error by the method in S302 and obtains the prediction error by the method in S2032. Based on the reconstruction error and the prediction error, the comprehensive coding error is obtained by formula 5 to update the network parameters of the initial image coding model.

[0147] S306, iterating the above adversarial training process until the final prediction error is less than a preset coding error threshold, thereby obtaining a target coding model and a target image segmentation model.

[0148] In the disclosed embodiment, the image processing device iterates the above-mentioned adversarial training process until the prediction error obtained in the final round of training process is less than a preset coding error threshold, then ends the adversarial training and obtains the target coding model and the target image segmentation model.

[0149] It can be understood that in the embodiment of the present disclosure, during the process of training the target image segmentation model, the image processing device uses the image coding model to add shape constraints to the prediction results of the target image segmentation model. The target image segmentation model can encode the input shape into the latent space, and by minimizing the distance between the network predicted shape and the true shape in the latent space, the shape predicted by the network is made to conform to prior knowledge, thereby constraining the prediction results of the target image segmentation model and improving the accuracy of image segmentation.

[0150] In the embodiment of the present disclosure, the main image segmentation model includes a compression activation module, see Figure 7 , Figure 1 S101 shown in the figure can be implemented through S1011-S1014, which will be explained in combination with each step.

[0151] S1011 , downsample the image to be processed twice to obtain a downsampled image.

[0152] In the disclosed embodiment, when the image processing device uses the main image segmentation model for feature extraction, in order to reduce detail loss caused by downsampling, only two downsamplings are used in the main image segmentation model to obtain a downsampled image.

[0153] In the disclosed embodiment, since the image segmentation model reduces the number of downsampling times, the loss of high-resolution information in the downsampled image is reduced.

[0154] S1012. Perform convolution processing on the downsampled image according to a preset convolution layer expansion rate to obtain a convolution result.

[0155] In the embodiment of the present disclosure, the image processing device can use the main image segmentation model such as Figure 2 The densely connected atrous convolution spatial pyramid module DenseASPP shown in the figure performs convolution processing on the downsampled image according to the preset convolution layer expansion rate to obtain the convolution result.

[0156] In some embodiments, Figure 2 The deconvolution upsampling process in the above method can also be implemented using a trilinear difference upsampling process. The specific selection is made according to the actual situation and is not limited in the embodiments of the present disclosure.

[0157] S1013. Perform residual and compression activation processing on the convolution result through the compression activation module to obtain an activation result.

[0158] S1014, performing multi-scale feature extraction and deconvolution processing on the activation result to obtain a feature map of the image to be processed.

[0159] In the disclosed embodiment, in order to further enhance the feature expression capability of the network, the image processing device further adds a squeeze-and-excitation module (SE module) as a feature-level attention mechanism in the main image segmentation model based on the residual module. The image processing device can perform residual and squeeze-and-excitation processing on the convolution result through the squeeze-and-excitation module to obtain an activation result. Then, multi-scale feature extraction is performed on the activation result through DenseASPP, and then a feature map of the image to be processed is obtained through deconvolution processing.

[0160] It can be understood that in the embodiment of the present disclosure, the main image segmentation model enhances the feature expression ability of the network through the densely connected atrous convolution spatial pyramid module DenseASPP and the added compression-activation module, improves the accuracy of feature extraction of the main image segmentation model, and thus improves the accuracy of image segmentation of the image processing device.

[0161] In the embodiments of the present disclosure, see Fig. 9 , Figure 1 S102 shown in the figure can be implemented through S1021-S1022, which will be explained in combination with each step.

[0162] S1021. Using the target positioning model, the segmentation target is positioned according to the feature map to obtain a positioning result of the segmentation target.

[0163] At present, the image to be processed contains many target images of different volumes. For example, the organs at risk in the CT image contain many organs of different volumes. For large organs, such as the parotid gland, its volume is more than 250 times that of the smallest organ, the lens. There is a huge sample imbalance problem for segmentation targets with smaller volumes, which may lead to low segmentation accuracy. Therefore, the image processing device in the embodiment of the present disclosure can use a two-stage network of first positioning and then segmentation: first, through the target positioning model, the Nth layer feature map output by the main image segmentation model is subjected to feature recognition and normalization processing to obtain the probability that each pixel in the Nth layer feature map belongs to the center position of the segmentation target, thereby locating the position of the segmentation target, and then based on the probability that each pixel belongs to the center position of the segmentation target, the Nth layer feature map is segmented through the target image segmentation model to obtain the positioning result of the segmentation target.

[0164] In the disclosed embodiment, the positioning result of the segmented target is a feature map region containing the segmented target, and the image processing device can perform the next step of image segmentation specifically for the segmented target based on the positioning result of the segmented target.

[0165] In some embodiments, for the Nth layer feature map output by the main image segmentation model for the CT image, the image processing device can perform feature processing through two compression activation modules SEResBlock in the target positioning model, and then through an S-type activation function Sigmoid layer, normalize the calculation result to the interval of 0 to 1, thereby outputting the probability map of the center position of each small organ in the CT image. Based on the probability map of the center position of each small organ, the image processing device cuts out the image area where the probability of the center position of the small organ is concentrated in a certain range from the Nth layer feature map as the positioning result of the segmentation target.

[0166] In some embodiments, the structure of the compression activation module can be as follows Figure 8 As shown in the figure, the current convolutional layer network module directly enters the scale module after the residual module Residue Block. In order to learn different depths of features with different strengths, the convolutional layer network module method in the embodiment of the present disclosure introduces a compression activation module after the residual layer module to improve the attention mechanism of the network. Specifically, the compression activation module may include a global pooling layer, a convolution layer, a linear rectification function layer, and an S-type activation function layer. Figure 8In the figure, C is the number of channels of the convolution, D is the depth of the feature map, H is the height of the feature map, and W is the width of the feature map. When the feature map enters the compression activation module through the residual module, the size of the feature map is reduced to 1*1*1 through global pooling, and then the reduced feature map is convolved through the convolution layer, and then the number of channels is compressed r times through the linear rectifier function (Rectified Linear Unit, ReLU), and the feature map is activated. After that, the activation result is activated by one convolution and one S-type function, and the output of the S-type function activation module is output through the scale module. After being processed by the above compression activation module. The convolutional network module can focus on the more important features in the feature map and improve the accuracy of image segmentation.

[0167] S1022. According to the positioning result, perform image segmentation on the segmentation target through the target image segmentation model to obtain a target segmentation result.

[0168] In the disclosed embodiment, the image processing device can predict the shape of the segmented target through the target image segmentation model based on the positioning result to obtain the cropping range of the segmented target; and based on the positioning result and the cropping range of the segmented target, find the corresponding area in the high-resolution Nth layer feature map, perform cropping processing on the Nth layer feature map, and obtain the target segmentation result.

[0169] It is understandable that in the disclosed embodiment, the image processing device can first locate the segmentation target with a smaller volume, cut the feature map, and then finely segment the cut feature map part, thereby solving the sample imbalance problem of small segmentation targets, reducing the segmentation difficulty, and improving the accuracy of image segmentation. In addition, since the target image segmentation model adds shape constraints during the training process, the shape prediction performed by the target image segmentation model is more accurate, thereby further improving the accuracy of image segmentation.

[0170] In the embodiments of the present disclosure, see Fig.10 , Fig. 9 After S101 shown in the figure, S401 may be executed, and after S1022, S402 may be executed. The description will be made in combination with each step.

[0171] S401. According to a preset main segmentation target, the feature map is positioned and segmented by a main image segmentation model to obtain a main segmentation result corresponding to the main segmentation target.

[0172] In the disclosed embodiment, since the feature map output by the main image segmentation model is a probability map of each pixel belonging to various segmentation targets, for some main segmentation targets with large volume and clear boundaries, positioning and segmentation processing can be directly performed in the feature map output by the main image segmentation model to obtain the main segmentation result corresponding to the main segmentation target.

[0173] S402: Fusing the main segmentation result and the target segmentation result in the image to be processed to obtain an image segmentation result of the image to be processed.

[0174] In the disclosed embodiment, the image processing device can fuse the main segmentation result obtained by the main image segmentation model and the target segmentation result obtained by the target image segmentation model in the image to be processed, and finally obtain the image segmentation result of the image to be processed that can reflect multiple segmentation targets.

[0175] In some embodiments, the main segmentation result may be the segmentation result of large organs in the CT image, and the target segmentation result may be the segmentation result of small organs in the CT image. The image processing device fuses the main segmentation result with the target segmentation result to obtain an image segmentation result of the image to be processed including the segmentation results of large organs and the segmentation results of small organs.

[0176] Next, an exemplary application of the embodiment of the present disclosure in a practical application scenario will be described.

[0177] In some embodiments, for the segmentation of medical images, the present disclosure provides a network architecture of an image processing device, such as Fig.11 shown. Fig.11The small and medium organ segmentation network 140 is equivalent to the target image segmentation network. In the training stage of the small organ segmentation network 140, the image processing device can perform adversarial training with the small organ segmentation network 140 through the adversarial autoencoder 160. Specifically, the predicted segmentation image output by the small organ segmentation network 140 and the labeled segmentation target in the training sample image are encoded by the encoder 160_1 contained in the adversarial autoencoder 160 to obtain the first spatial shape code and the second spatial shape code. Among them, the encoder 160_1 is equivalent to the encoding sub-model, and the training sample image contains the labeled segmentation target. The image processing device calculates the prediction error through the first spatial shape code and the second spatial shape code, and updates the network parameters of the small organ segmentation network 140 in training through the prediction error to complete one training. In the next training process, the image processing device decodes and reconstructs the first spatial shape code and the second spatial shape code obtained by the current training through the decoder 160_2 contained in the adversarial autoencoder 160, wherein the decoder 160_2 is equivalent to the decoding sub-model, and obtains the first reconstructed image and the second reconstructed image. The image processing device calculates the reconstruction error based on the error between the annotated segmentation target and the first reconstructed image, the predicted segmentation image and the second reconstructed image, and calculates the comprehensive coding error through the reconstruction error and the prediction error obtained by the current training, and updates the network parameters of the adversarial autoencoder 160 through the comprehensive coding error, thereby realizing the alternating adversarial training of the small organ segmentation network 140 and the adversarial autoencoder 160. The image processing device iterates the above training process until the adversarial autoencoder 160 cannot distinguish the difference between the predicted segmentation image output by the small organ segmentation network 140 and the annotated segmentation target, and then ends the adversarial training to obtain the trained small organ segmentation network 140.

[0178] In the application stage of the small organ segmentation network 140, the image processing device first extracts features from the input original CT image through the main image segmentation model 110 to obtain a feature map, and directly obtains the large organ segmentation result based on the feature map, wherein the large organ is equivalent to the main segmentation target, and the large organ segmentation result is equivalent to the main segmentation result. Then, based on the Nth layer feature map in the feature map, the image processing device locates the image range of the probability of belonging to the small organ in the feature map through the small organ localization network 120, and performs regional segmentation on the feature map according to the localization position to obtain the small organ probability map. Among them, the small organ localization network 120 is equivalent to the target localization model, and the small organ probability map is equivalent to the localization result. In order to reduce the data dimension and improve the processing efficiency, the image processing device can perform ROI (region of interest) pooling on the small organ probability map, and segment the Nth layer feature map in the feature map based on the small organ position on the small organ probability map and the small organ shape predicted by the small organ segmentation network 140 through the trained small organ segmentation network 140, and obtain the small organ segmentation result, which is equivalent to the target segmentation result. The image processing device fuses the large organ segmentation result and the small organ segmentation result through the fusion unit 150 to obtain a final image segmentation result, which is equivalent to the image segmentation result to be processed.

[0179] In some embodiments, when the image processing method in the disclosed embodiment is used to segment organs in a head CT image, the comparison effect diagram between the image processing method and the atlas-based segmentation algorithm Multi-Atlas, the deep annotation network DeepLavV3+, and the anatomical network AnatomyNet for CT image segmentation can be shown as follows: Fig.12 It can be seen that the segmented shape of small organs obtained by the method in the embodiment of the present disclosure is closer to the manual annotation result, and the image segmentation accuracy is higher.

[0180] In some embodiments, Fig.13 As shown, when the target segmentation result obtained by using the image processing method in the embodiment of the present disclosure to perform organ segmentation on the head CT image is overlapped on the image of the manual annotation result for comparison, it can be seen that the target segmentation result obtained in the embodiment of the present disclosure has a high degree of image overlap with the manual annotation result, indicating that the shape of the target segmentation result obtained in the embodiment of the present disclosure is basically consistent with the manual annotation result, and the image segmentation accuracy is high.

[0181] The present disclosure provides an image processing device, such as Fig.14 As shown, Fig.14 A structural schematic diagram of an image processing device 455 provided in an embodiment of the present disclosure, wherein the image processing device 455 includes a main image segmentation model 4551 and a target image segmentation model 4552, wherein:

[0182] The main image segmentation model 4551 is used to extract features of the image to be processed to obtain a feature map of the image to be processed;

[0183] The target image segmentation model 4552 is used to perform image segmentation on the segmentation target according to the feature map to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; the image coding model is used to perform shape constraints on the target segmentation result.

[0184] In some embodiments, the image processing device 455 further includes a training unit, wherein:

[0185] The training unit is used to obtain a set of training sample images before performing image segmentation on the segmentation target through the target image segmentation model according to the feature map to obtain the target segmentation result, wherein the training sample image set includes at least one training sample image corresponding to the segmentation target, and the training sample image has a labeled segmentation target; perform image segmentation on each training sample image through the initial target image segmentation model to obtain a predicted segmentation image; and perform adversarial training on the initial target image segmentation model according to the labeled segmentation target and the predicted segmentation image in combination with the initial image coding model to obtain a target image segmentation model.

[0186] In some embodiments, the training unit is also used to use the initial image coding model to encode the labeled segmentation target and the predicted segmentation image to obtain a first spatial shape code and a second spatial shape code; wherein the first spatial shape code is a low-dimensional manifold space representation of the labeled segmentation target; the second spatial shape code is a low-dimensional manifold space representation of the predicted segmentation image; by using a preset minimum square error loss function, the coding distance between the first spatial shape code and the second spatial shape code is calculated to obtain a prediction error; based on the shape constraint of the prediction error, the network parameters of the initial target image segmentation model are updated, and the above-mentioned adversarial training process is iterated until the final prediction error is less than a preset segmentation error threshold, and then the training process is terminated to obtain the target image segmentation model.

[0187] In some embodiments, the initial image coding model includes a decoding sub-model, and the training unit is further used to, after the first spatial shape code and the second spatial shape code are processed by a preset minimum square error loss function to obtain a prediction error, decode and graphically reconstruct the first spatial shape code and the second spatial shape code respectively through the decoding sub-model to obtain a first reconstructed image and a second reconstructed image respectively; and combine the error between the labeled segmentation target and the first reconstructed image, and the error between the predicted segmentation image and the second reconstructed image through a preset reconstruction loss function to obtain a reconstruction error of the initial image coding model.

[0188] In some embodiments, the training unit is also used to update the network parameters of the initial target image segmentation model using the prediction error during the Mth training process; wherein M is an integer greater than or equal to 1; in the M+1th training, a comprehensive coding error is obtained based on the reconstruction error and the prediction error; the network parameters of the initial image coding model are updated using the comprehensive coding error; the above-mentioned adversarial training process is iterated until the final prediction error is less than a preset coding error threshold, thereby obtaining the target coding model and the target image segmentation model.

[0189] In some embodiments, the initial image coding model includes a coding sub-model, which includes at least one convolutional layer and a fully connected layer; the training unit is also used to compress the training sample image and the predicted segmentation image through the at least one convolutional layer according to a preset step size of the at least one convolutional layer, and encode the labeled segmentation targets in the compressed training sample image and the compressed predicted segmentation image through the fully connected layer to obtain the first spatial shape code and the second spatial shape code.

[0190] In some embodiments, the decoding sub-model includes at least one deconvolution layer, a linear rectification function and a batch normalization layer.

[0191] In some embodiments, the image processing device 455 also includes a target positioning model, which is used to locate the segmentation target according to the feature map to obtain the positioning result of the segmentation target; based on the positioning result, the segmentation target is segmented by the target image segmentation model 4552 to obtain the target segmentation result.

[0192] In some embodiments, the main image segmentation model 4551 includes a compression activation module, and the main image segmentation model 4551 is also used to downsample the image to be processed twice to obtain a downsampled image; convolve the downsampled image according to a preset convolution layer expansion rate to obtain a convolution result; perform residual and compression activation processing on the convolution result through the compression activation module to obtain an activation result; perform multi-scale feature extraction and deconvolution processing on the activation result to obtain a feature map of the image to be processed.

[0193] In some embodiments, the feature map includes N layers of feature maps, where N is an integer greater than 1, and the target positioning model is also used to perform feature recognition and normalization processing on the Nth layer of feature maps to obtain the probability that each pixel in the Nth layer of feature maps belongs to the center position of the segmented target; the Nth layer of feature maps is segmented according to the probability to obtain the positioning result of the segmented target.

[0194] In some embodiments, the target positioning model is also used to predict the shape of the segmented target through a target image segmentation model to obtain a cropping range of the segmented target; based on the positioning result of the segmented target and the cropping range, the Nth layer feature map is cropped to obtain the target segmentation result.

[0195] In some embodiments, the main image segmentation model 4551 is further used to extract features from the image to be processed, obtain a feature map of the image to be processed, and then locate and segment the feature map according to a preset main segmentation target to obtain a main segmentation result corresponding to the main segmentation target;

[0196] The image processing device 455 also includes a fusion unit, which is used to perform image segmentation on the segmented target according to the positioning result in the target image segmentation model, and after obtaining the target segmentation result, fuse the main segmentation result and the target segmentation result in the image to be processed to obtain the image segmentation result of the image to be processed.

[0197] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0198] It should be noted that in the embodiments of the present disclosure, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a terminal, server, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific combination of hardware and software.

[0199] Correspondingly, an embodiment of the present disclosure further provides a computer program product, which includes computer executable instructions, and the computer executable instructions are used to implement the steps in the image processing method provided by the embodiment of the present disclosure.

[0200] Accordingly, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is used to implement the steps of the image processing method provided in the above embodiment.

[0201] The present disclosure also provides an image processing device, Fig.15 A schematic diagram of an optional structure of an image processing device provided in an embodiment of the present disclosure is shown in FIG. Fig.15 As shown, the image processing device 110 includes:

[0202] Memory 1102, used for storing computer programs;

[0203] The processor 1103 is used to implement the steps of the image processing method provided in the above embodiment when executing the computer program stored in the memory 1102.

[0204] The image processing device 110 further includes a communication bus 1104. The communication bus 1104 is configured to realize connection and communication between these components.

[0205] In the embodiment of the present disclosure, the memory 1102 is configured to store computer programs and applications performed by the processor 1103, and can also cache data to be processed or already processed by various modules in the processor 1103 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0206] When the processor 1103 executes the program, the steps of any one of the above-mentioned image processing methods are implemented.

[0207] The processor 1103 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the above-mentioned processor function may also be other, and the embodiments of the present disclosure are not limited thereto.

[0208] The above-mentioned computer-readable storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0209] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0210] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0211] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0212] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0213] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present disclosure.

[0214] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0215] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for the automatic test line of the device to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0216] The methods disclosed in several method embodiments provided in the present disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0217] The features disclosed in several method or device embodiments provided in the present disclosure may be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0218] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Extracting features of the image to be processed by the main image segmentation model to obtain a feature map of the image to be processed; According to the feature map, the segmentation target is segmented by a target image segmentation model to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; the image coding model is used to impose shape constraints on the target segmentation result; Before performing image segmentation on the segmentation target by using a target image segmentation model according to the feature map to obtain a target segmentation result, the method further includes: Acquire a training sample image set, wherein the training sample image set includes at least one training sample image corresponding to the segmentation target, and the training sample image is labeled with the segmentation target; Perform image segmentation on each training sample image through the initial target image segmentation model to obtain a predicted segmentation image; Using the initial image coding model, the annotated segmentation target and the predicted segmentation image are encoded to obtain a first spatial shape code and a second spatial shape code; wherein the first spatial shape code is a low-dimensional manifold space representation of the annotated segmentation target; and the second spatial shape code is a low-dimensional manifold space representation of the predicted segmentation image; Calculating the coding distance between the first spatial shape coding and the second spatial shape coding by using a preset minimum square error loss function to obtain a prediction error; Based on the shape constraint of the prediction error, the network parameters of the initial target image segmentation model are updated, and the above-mentioned adversarial training process is iterated until the final prediction error is less than the preset segmentation error threshold, and then the training process is terminated to obtain the target image segmentation model.

2. The method according to claim 1, characterized in that: The initial image coding model includes a decoding sub-model. After the first spatial shape coding and the second spatial shape coding are processed by a preset minimum square error loss function to obtain a prediction error, the method further includes: The first spatial shape code and the second spatial shape code are decoded and graphically reconstructed respectively by the decoding sub-model to obtain a first reconstructed image and a second reconstructed image respectively; By presetting a reconstruction loss function, the error between the labeled segmentation target and the first reconstructed image, and the error between the predicted segmentation image and the second reconstructed image are combined to obtain a reconstruction error of the initial image coding model.

3. The method according to claim 2, characterized in that The network parameters of the initial target image segmentation model are updated based on the shape constraint of the prediction error, and the adversarial training process is iterated until the final prediction error is less than a preset segmentation error threshold, and the training process is terminated to obtain the target image segmentation model. In the Mth training process, the network parameters of the initial target image segmentation model are updated using the prediction error; wherein M is an integer greater than or equal to 1; In the M+1th training, a comprehensive coding error is obtained based on the reconstruction error and the prediction error; Using the comprehensive coding error, updating the network parameters of the initial image coding model; The above-mentioned adversarial training process is iterated until the final prediction error is less than a preset coding error threshold, thereby obtaining the image coding model and the target image segmentation model.

4. The method according to claim 1, characterized in that: The initial image coding model includes a coding sub-model, and the coding sub-model includes at least one convolutional layer and a fully connected layer; The encoding of the annotated segmentation target and the predicted segmentation image by using the initial image encoding model to obtain a first spatial shape encoding and a second spatial shape encoding includes: According to the preset step size of the at least one convolutional layer, the training sample image and the predicted segmentation image are compressed by the at least one convolutional layer, and the annotated segmentation targets in the compressed training sample image and the compressed predicted segmentation image are encoded by the fully connected layer to obtain the first spatial shape code and the second spatial shape code.

5. The method according to claim 2, characterized in that: The decoding sub-model includes at least one deconvolution layer, a linear rectification function and a batch normalization layer.

6. The method according to claim 1, characterized in that The step of performing image segmentation on the segmentation target by using a target image segmentation model according to the feature map to obtain a target segmentation result includes: Using a target positioning model, positioning the segmented target according to the feature map to obtain a positioning result of the segmented target; According to the positioning result, the segmentation target is image segmented by a target image segmentation model to obtain a target segmentation result.

7. The method according to any one of claims 1 to 6, characterized in that: The main image segmentation model includes a compression activation module, and extracting features from the image to be processed by the main image segmentation model to obtain a feature map of the image to be processed includes: Downsampling the image to be processed twice to obtain a downsampled image; According to a preset convolution layer expansion rate, convolution processing is performed on the downsampled image to obtain a convolution result; Performing residual and compression activation processing on the convolution result through the compression activation module to obtain an activation result; Perform multi-scale feature extraction and deconvolution processing on the activation result to obtain a feature map of the image to be processed.

8. The method according to claim 6, characterized in that The feature map includes N layers of feature maps, where N is an integer greater than 1. The target positioning model is used to locate the segmented target according to the feature map to obtain a positioning result of the segmented target, including: By using the target positioning model, feature recognition and normalization processing are performed on the N-th layer feature map to obtain the probability that each pixel in the N-th layer feature map belongs to the center position of the segmented target; Image segmentation is performed on the N-th layer feature map according to the probability to obtain a positioning result of the segmented target.

9. The method according to claim 8, characterized in that The step of performing image segmentation on the segmented target by using a target image segmentation model according to the positioning result to obtain a target segmentation result includes: Predicting the shape of the segmented target by using a target image segmentation model to obtain a cutting range of the segmented target; According to the positioning result of the segmented target and the cropping range, the Nth layer feature map is cropped to obtain the target segmentation result.

10. The method according to claim 6 or 9, characterized in that: After extracting features of the image to be processed by the main image segmentation model to obtain a feature map of the image to be processed, the method further includes: According to a preset main segmentation target, the feature map is positioned and segmented by the main image segmentation model to obtain a main segmentation result corresponding to the main segmentation target; According to the positioning result, after performing image segmentation on the segmented target by using a target image segmentation model and obtaining the target segmentation result, the method further comprises: The main segmentation result and the target segmentation result are fused in the image to be processed to obtain an image segmentation result of the image to be processed.

11. An image processing device, characterized in that: include: A main image segmentation model is used to extract features of the image to be processed to obtain a feature map of the image to be processed; A target image segmentation model, used to perform image segmentation on the segmentation target according to the feature map to obtain a target segmentation result; wherein the target image segmentation model is obtained by adversarial training with an image coding model; the image coding model is used to perform shape constraints on the target segmentation result; A training unit is used to obtain a set of training sample images, wherein the training sample image set includes at least one training sample image corresponding to the segmentation target, and the training sample image has a labeled segmentation target; by using an initial target image segmentation model, each training sample image is segmented to obtain a predicted segmentation image; by using an initial image coding model, the labeled segmentation target and the predicted segmentation image are encoded to obtain a first spatial shape code and a second spatial shape code; wherein the first spatial shape code is a low-dimensional manifold space representation of the labeled segmentation target; the second spatial shape code is a low-dimensional manifold space representation of the predicted segmentation image; by using a preset minimum square error loss function, the coding distance between the first spatial shape code and the second spatial shape code is calculated to obtain a prediction error; based on the shape constraint of the prediction error, the network parameters of the initial target image segmentation model are updated, and the adversarial training process is iterated until the final prediction error is less than a preset segmentation error threshold, and then the training process is terminated to obtain the target image segmentation model.

12. An image processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 10 when executing the computer program stored in the memory.

13. A computer-readable storage medium, characterized in that: A computer program is stored, which is used to cause a processor to execute and implement the method according to any one of claims 1 to 10.

Images