An image segmentation method, device, terminal equipment and readable storage medium

Abstract

Embodiments of the present application provide an image segmentation method and device, the method comprising: obtaining a first image to be segmented; the first image is of a first resolution, and the first image includes a target object; inputting the first image into a pre-trained double-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; the second resolution is greater than the first resolution; determining a target image of the second resolution corresponding to the target object according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution; by using the double-path super-resolution segmentation structure model provided by the embodiments of the present application, the segmentation effect of a small nodule in a low-resolution CT image can be greatly improved by referring to the idea of image super-resolution, a high-resolution image is generated, and accurate segmentation of target objects of different scales is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image segmentation method, apparatus, terminal equipment and readable storage medium. Background Art

[0002] Lung nodule screening using computed tomography (CT) is an important tool for diagnosing early-stage lung cancer. Accurately segmenting lung nodules in CT images during screening can provide doctors with more accurate and quantitative nodule features, such as precise volume, major and minor diameters, and HU values, significantly assisting in determining benign or malignant status, as well as subsequent follow-up and treatment.

[0003] Deep learning algorithms are widely used in nodule detection and segmentation tasks due to their superior performance compared to traditional image algorithms. The deep learning-based small lung nodule segmentation method mainly uses neural networks and their simple deformations to extract two-dimensional or three-dimensional information from CT images and generate segmentation result maps.

[0004] CT scans generally have low resolution, while lung nodules vary greatly in size. Consequently, tiny nodules in CT images can only display a small number of pixels on a slice and may be lost in large-scale model sampling. These nodules are too small to be captured by the model. Alternatively, some nodules can be very large, with a long axis exceeding 30 mm, requiring a larger receptive field in the neural network to capture all the information. Existing small nodule segmentation methods cannot strike a good balance between small and large nodules, and cannot simultaneously segment lung nodules of different sizes. Furthermore, they perform poorly for segmenting tiny nodules in low-resolution CT images. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide an image segmentation method and apparatus that overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, an embodiment of the present invention provides an image segmentation method, the method comprising:

[0007] Acquire a first image to be segmented; wherein the first image has a first resolution and includes a target object;

[0008] Inputting the first image into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image at a second resolution and a segmentation mask of the target object at a second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; and the second resolution is greater than the first resolution;

[0009] A target image with a second resolution corresponding to the target object is determined according to the predicted image with the second resolution and the segmentation mask of the target object with the second resolution.

[0010] Optionally, the dual-path super-resolution segmentation structure model is obtained by:

[0011] Acquire a sample image of a third resolution, and acquire a sample image of a fourth resolution corresponding to the sample image of the third resolution;

[0012] The high-resolution segmentation neural network and the image super-resolution neural network are trained respectively according to the sample image to obtain a segmented image with a fifth resolution; the fifth resolution is greater than the third resolution, and the fifth resolution is the same as the fourth resolution;

[0013] Determining a loss function of a high-resolution segmentation neural network and a loss function of an image super-resolution neural network based on the sample image of the third resolution, the sample image of the fourth resolution, and the segmented image of the fifth resolution;

[0014] When the loss function of the high-resolution segmentation neural network is less than a first preset value and / or the loss function of the image super-resolution neural network is less than a second preset value, the high-resolution segmentation neural network and the image super-resolution neural network are determined as the dual-path super-resolution segmentation structure model.

[0015] Optionally, the method further includes:

[0016] According to the pre-trained dilated convolutional neural network, a receptive field image corresponding to the target object is determined.

[0017] Optionally, the method further includes:

[0018] According to the size of the target object, the scale-aware dilated convolutional neural network in the dual-path super-resolution segmentation structure model is trained to obtain the dilated convolutional neural network; the dilated convolutional neural network includes multiple parallel dilated convolution kernels with different dilation rates, wherein each dilated convolution kernel corresponds to a scale coefficient map.

[0019] Optionally, the method further includes:

[0020] The first image is segmented according to a pre-trained association learning model to obtain a segmented image corresponding to the target object, wherein the association learning model is obtained by association learning the high-resolution segmentation neural network and the image super-resolution neural network.

[0021] Optionally, the association learning model is obtained by:

[0022] Performing feature association learning on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model;

[0023] Scale association learning is performed on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model.

[0024] Optionally, the performing feature association learning on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model includes:

[0025] Determining a first feature map and a second feature map respectively according to an input vector of the sample image, the high-resolution segmentation neural network, and the image super-resolution neural network;

[0026] Calculate the Gram matrix of the first feature map and the second feature map according to the calculation method of the correlation coefficient;

[0027] Calculate the correlation loss function between the Gram matrix of the first feature map and the Gram matrix of the second feature map;

[0028] Training the dual-path super-resolution segmentation structure model according to the association loss function;

[0029] Correspondingly, the scale association learning of the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model includes:

[0030] Scaling the scale coefficient maps corresponding to the dilated convolution kernels at different levels to obtain multiple sub-scale maps, where the sub-scale maps have the same size as the sample image;

[0031] Connecting the multiple sub-scale maps to determine a multi-channel scale coefficient map; wherein the scale coefficient map includes a first scale coefficient map corresponding to the high-resolution segmentation neural network and a second scale coefficient map corresponding to the image super-resolution neural network;

[0032] determining a scaled Gram matrix according to the first scale coefficient map and the second scale coefficient map;

[0033] determining an association loss function based on the scale Gram matrix;

[0034] The dual-path super-resolution segmentation structure model is trained according to the association loss function.

[0035] In a second aspect, an embodiment of the present invention provides an image segmentation device, comprising:

[0036] An acquisition module, configured to acquire a first image to be segmented; wherein the first image has a first resolution and includes a target object;

[0037] a segmentation module, configured to input the first image into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image at a second resolution and a segmentation mask of the target object at the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; and the second resolution is greater than the first resolution;

[0038] A determination module is configured to determine a target image with a second resolution corresponding to the target object based on the predicted image with the second resolution and a segmentation mask of the target object with the second resolution.

[0039] Optionally, the device further comprises a model building module, wherein the model building module is configured to:

[0040] Acquire a sample image of a third resolution, and acquire a sample image of a fourth resolution corresponding to the sample image of the third resolution;

[0041] The high-resolution segmentation neural network and the image super-resolution neural network are trained respectively according to the sample image to obtain a segmented image with a fifth resolution; the fifth resolution is greater than the third resolution, and the fifth resolution is the same as the fourth resolution;

[0042] Determining a loss function of a high-resolution segmentation neural network and a loss function of an image super-resolution neural network based on the sample image of the third resolution, the sample image of the fourth resolution, and the segmented image of the fifth resolution;

[0043] When the loss function of the high-resolution segmentation neural network is less than a first preset value and / or the loss function of the image super-resolution neural network is less than a second preset value, the high-resolution segmentation neural network and the image super-resolution neural network are determined as the dual-path super-resolution segmentation structure model.

[0044] Optionally, the segmentation module is further configured to:

[0045] According to the pre-trained dilated convolutional neural network, a receptive field image corresponding to the target object is determined.

[0046] Optionally, the model building module is used to:

[0047] According to the size of the target object, the scale-aware dilated convolutional neural network in the dual-path super-resolution segmentation structure model is trained to obtain the dilated convolutional neural network; the dilated convolutional neural network includes multiple parallel dilated convolution kernels with different dilation rates, wherein each dilated convolution kernel corresponds to a scale coefficient map.

[0048] Optionally, the segmentation module is used to:

[0049] The first image is segmented according to a pre-trained association learning model to obtain a segmented image corresponding to the target object, wherein the association learning model is obtained by association learning the high-resolution segmentation neural network and the image super-resolution neural network.

[0050] Optionally, the model building module is used to:

[0051] Performing feature association learning on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model;

[0052] Scale association learning is performed on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model.

[0053] Optionally, the model building module is used to:

[0054] Determining a first feature map and a second feature map respectively according to an input vector of the sample image, the high-resolution segmentation neural network, and the image super-resolution neural network;

[0055] Calculate the Gram matrix of the first feature map and the second feature map according to the calculation method of the correlation coefficient;

[0056] Calculate the correlation loss function between the Gram matrix of the first feature map and the Gram matrix of the second feature map;

[0057] Training the dual-path super-resolution segmentation structure model according to the association loss function;

[0058] The model building module is also used to:

[0059] Scaling the scale coefficient maps corresponding to the dilated convolution kernels at different levels to obtain multiple sub-scale maps, where the sub-scale maps have the same size as the sample image;

[0060] Connecting the multiple sub-scale maps to determine a multi-channel scale coefficient map; wherein the scale coefficient map includes a first scale coefficient map corresponding to the high-resolution segmentation neural network and a second scale coefficient map corresponding to the image super-resolution neural network;

[0061] determining a scaled Gram matrix according to the first scale coefficient map and the second scale coefficient map;

[0062] determining an association loss function based on the scale Gram matrix;

[0063] The dual-path super-resolution segmentation structure model is trained according to the association loss function.

[0064] In a third aspect, an embodiment of the present invention provides a terminal device, comprising: at least one processor and a memory;

[0065] The memory stores a computer program; the at least one processor executes the computer program stored in the memory to implement the image segmentation method provided in the first aspect.

[0066] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the image segmentation method provided in the first aspect is implemented.

[0067] The embodiments of the present invention include the following advantages:

[0068] The image segmentation method, apparatus, terminal device and readable storage medium provided by the embodiments of the present invention obtain a first image to be segmented; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and the target image of the second resolution corresponding to the target object is determined according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution. The dual-path super-resolution segmentation structure model provided by the embodiments of the present invention can draw on the idea of ​​image super-resolution, and to a large extent improve the segmentation effect of tiny nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flowchart of steps of an embodiment of an image segmentation method of the present invention;

[0070] Figure 2 It is a flowchart of the steps of model training of the present invention;

[0071] Figure 3 1 is a schematic diagram of another step of model training of the present invention;

[0072] Figure 4 is a structural block diagram of an embodiment of an image segmentation device of the present invention;

[0073] Figure 5 It is a structural schematic diagram of a terminal device of the present invention. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] An embodiment of the present invention provides an image segmentation method for segmenting a target object in an image. The embodiment is performed by an image segmentation device, which is provided on a terminal device.

[0076] Reference Figure 1 , shows a flowchart of an embodiment of an image segmentation method of the present invention, which may specifically include the following steps:

[0077] S101, obtaining a first image to be segmented; wherein the first image has a first resolution and includes a target object;

[0078] Specifically, the terminal device acquires a first image to be segmented, which may be a nodule CT image. The first image has a low resolution and includes target objects, such as various nodules.

[0079] S102. Input the first image into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; and the second resolution is greater than the first resolution.

[0080] S103 : Determine a target image with a second resolution corresponding to the target object according to the predicted image with the second resolution and the segmentation mask of the target object with the second resolution.

[0081] Specifically, since the resolution of the acquired first image is low, when segmenting the nodules in the first image, the resolution of the image obtained is also low, and the specific details cannot be seen clearly. Therefore, an embodiment of the present invention provides a dual-path super-resolution segmentation structure model, that is, a dual-path super-resolution segmentation structure model pre-trained on a terminal device, the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network;

[0082] When the terminal device acquires the first image, the first image is input into the dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution, wherein the second resolution is greater than the first resolution.

[0083] For example, if the resolution of the first image is 480P, then by inputting it into the above-mentioned dual-path super-resolution segmentation structure model, a second image of the nodule can be obtained, and the resolution of the second image is 1080P, so that a high-resolution image of the nodule can be generated.

[0084] The image segmentation method provided by the embodiment of the present invention obtains a first image to be segmented; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and the target image of the second resolution corresponding to the target object is determined according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution. The dual-path super-resolution segmentation structure model provided by the embodiment of the present invention can draw on the idea of ​​image super-resolution to greatly improve the segmentation effect of tiny nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales.

[0085] Another embodiment of the present invention further supplements the image segmentation method provided in the above embodiment.

[0086] Figure 2 It is a flow chart of the steps of model training of the present invention, such as Figure 2 As shown in the figure, the dual-path super-resolution segmentation structure model includes a nodule super-resolution segmentation channel (NSSR) and a nodule image super-resolution channel. The two channels share the same encoder and use two decoders with the same structure and scale-aware hole convolution modules to calculate the feature association loss between the features of the two paths and the scale association loss between the scale coefficient maps of the scale-aware hole convolution modules in the two paths.

[0087] Optionally, the dual-path super-resolution segmentation structure model is obtained by:

[0088] Acquire a sample image of a third resolution, and acquire a sample image of a fourth resolution corresponding to the sample image of the third resolution;

[0089] The high-resolution segmentation neural network and the image super-resolution neural network are trained respectively according to the sample image to obtain a segmented image with a fifth resolution; the fifth resolution is greater than the third resolution, and the fifth resolution is the same as the fourth resolution;

[0090] Determining a loss function of a high-resolution segmentation neural network and a loss function of an image super-resolution neural network based on the sample image of the third resolution, the sample image of the fourth resolution, and the segmented image of the fifth resolution;

[0091] When the loss function of the high-resolution segmentation neural network is less than a first preset value and / or the loss function of the image super-resolution neural network is less than a second preset value, the high-resolution segmentation neural network and the image super-resolution neural network are determined as the dual-path super-resolution segmentation structure model.

[0092] Specifically, in order to better recover the information of tiny nodules from relatively low-resolution CT images, super-resolution image learning is added to the nodule segmentation network. Therefore, a dual-path structure is constructed, including a nodule high-resolution segmentation channel and a nodule CT image super-resolution channel. These two branches simultaneously learn high-resolution reconstruction and segmentation tasks from low-resolution images.

[0093] Both CNN branches use the same Unet CNN architecture and share encoder weights. The encoder consists of three CNN modules, each of which includes multiple 3D convolution operations and a 3D downsampling operation. The two-channel decoder uses the same network architecture but does not share parameters.

[0094] The feature map output by the super-resolution channel of the nodule CT image is then input into an enhanced super-resolution module (EDSR). EDSR is a super-resolution convolution module. Similar to the residual convolution module, EDSR consists of convolution layers, activation layers, and skip connections. Its main function is to optimize the low-level feature extraction capability in the super-resolution task.

[0095] The enhanced super-resolution module generates a 2x resolution CT nodule image, while the feature image output by the nodule high-resolution segmentation channel is upsampled by a factor of 2 and output as the nodule segmentation result. Therefore, both channels use the low-resolution image as input to the network and output a segmentation result image and a super-resolution reconstructed image, each twice the size of the input image.

[0096] Due to the lack of high-resolution CT images during training, the original CT volume was first downsampled to half its original resolution to obtain low-resolution input. The original CT and its segmentation mask were used as the gold standard. The DICE loss function was used for the high-resolution segmentation of nodules. This loss function calculates the intersection-over-union ratio between the predicted segmentation result and the gold standard, making the model more sensitive to the segmentation of small nodules. The mean square error loss function was used for the super-resolution channel of the nodule CT images.

[0097] Optionally, the method further includes:

[0098] According to the pre-trained dilated convolutional neural network, the receptive field image corresponding to the target object is determined.

[0099] Due to the large differences in the volume of lung nodules, the segmentation network needs to adjust its receptive field according to the different input nodule sizes to obtain better multi-scale segmentation results. To this end, a scale-aware dilated convolutional module, namely a dilated convolutional neural network, is proposed to achieve dynamic receptive field adjustment.

[0100] Optionally, the method further includes:

[0101] According to the size of the target object, the scale-aware dilated convolutional neural network in the dual-path super-resolution segmentation structure model is trained to obtain a dilated convolutional neural network; the dilated convolutional neural network includes multiple parallel dilated convolution kernels with different dilation rates, where each dilated convolution kernel corresponds to a scale coefficient map.

[0102] like Figure 3 As shown in the figure, the ordinary convolution module in the decoder of the dual-path super-resolution segmentation structure model is replaced by a scale-aware dilated convolution module. Each scale-aware dilated convolution module consists of four dilated convolutions, and a scale coefficient map is generated to assign different weighted values ​​to different convolution output features and calculate the weighted sum.

[0103] The scale-aware dilated convolution includes multiple parallel dilated convolution kernels with different dilation rates. The stacking of dilated convolution kernels enables the model to support feature extraction operations with various receptive fields. However, due to the difference in nodule size, the features output by the multi-scale receptive field convolution operation do not contribute equally to the final segmentation structure. Therefore, a scale coefficient map is used to measure the contribution rate of the convolution output with different receptive fields at each pixel in the feature image. That is, the scale coefficient map is used to measure the size of objects at different positions in the feature, thereby using different receptive fields.

[0104] The scale factor map is generated by a convolution operation with a kernel of 1*1*1 and a Sigmoid activation function, and the number of channels corresponds to the number of stacked dilated convolutions with different dilation rates.

[0105] Exemplarily, the dilated convolutions used include convolutions with a kernel of 1*1*1 and a dilation rate of 1, convolutions with a kernel of 3*3*3 and a dilation rate of 1, convolutions with a kernel of 3*3*3 and a dilation rate of 2, and convolutions with a kernel of 3*3*3 and a dilation rate of 5. In the embodiment of the present invention, a scale-aware dilated convolution module is used to replace the ordinary convolution module in the original decoder of the nodule high-resolution segmentation channel and the nodule CT image super-resolution channel, and each dilated convolution module outputs a scale coefficient map. After scaling, the scale coefficient maps output by convolution modules at different levels have the same size as the low-resolution CT image input of the entire model.

[0106] In order to further enhance the robustness and accuracy of super-resolution segmentation, two association learning modules are introduced in the embodiment of the present invention, which use the features and scale coefficient map generated by the nodule CT image super-resolution channel to guide the learning of the nodule super-resolution segmentation channel.

[0107] Optionally, the method further includes:

[0108] The first image is segmented according to a pre-trained association learning model to obtain a segmented image corresponding to the target object, wherein the association learning model is obtained by association learning a high-resolution segmentation neural network and an image super-resolution neural network.

[0109] Optionally, the association learning model is obtained by:

[0110] Perform feature association learning on the high-resolution segmentation neural network and image super-resolution neural network in the dual-path super-resolution segmentation structure model;

[0111] Scale-related learning is performed on the high-resolution segmentation neural network and image super-resolution neural network in the dual-path super-resolution segmentation structure model.

[0112] Optionally, feature association learning is performed on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model, including:

[0113] Determine a first feature map and a second feature map according to an input vector of the sample image, a high-resolution segmentation neural network, and an image super-resolution neural network;

[0114] Calculate the Gram matrix of the first feature map and the second feature map according to the calculation method of the correlation coefficient;

[0115] Calculate the correlation loss function between the Gram matrix of the first feature map and the Gram matrix of the second feature map;

[0116] According to the associated loss function, the dual-path super-resolution segmentation structure model is trained;

[0117] Correspondingly, scale-related learning is performed on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model, including:

[0118] The scale coefficient maps corresponding to the dilated convolution kernels at different levels are scaled to obtain multiple sub-scale maps, where the sub-scale maps are the same size as the sample image.

[0119] Connecting the multiple sub-scale maps to determine a multi-channel scale coefficient map; wherein the scale coefficient map includes a first scale coefficient map corresponding to the high-resolution segmentation neural network and a second scale coefficient map corresponding to the image super-resolution neural network;

[0120] determining a scale Gram matrix based on the first scale coefficient map and the second scale coefficient map;

[0121] Determine the associated loss function based on the scale Gram matrix;

[0122] The dual-path super-resolution segmentation structure model is trained according to the associated loss function.

[0123] Specifically, the first is feature association learning. Compared with the nodule super-resolution segmentation branch with prediction task, the nodule CT image super-resolution branch performs the reconstruction task, which contains more complete high-resolution information.

[0124] This embodiment of the present invention uses a feature association module to guide learning using the segmentation branch. Applied after the final layer of the dual-channel decoder, the feature association module first passes the different feature maps output by the two branches through a convolutional layer, a batch normalization layer, and a ReLU activation layer to generate two new feature images. The association learning loss function is then optimized by minimizing the Gram matrix of these two feature maps.

[0125] Assuming a feature map of size W*H*D and C channels, it is first reconstructed into a C*(W*H*D) vector. Then, based on the correlation coefficient calculation method, the Gram matrix is ​​calculated by multiplying the vector and its transpose. During training, the correlation loss function is calculated between the Gram matrices of the dual-channel input vectors. The correlation loss function uses the mean square error function. By optimizing the dual-channel Gram matrix, the nodule segmentation channel can better extract high-resolution features, thereby accurately segmenting small nodules.

[0126] Similarly, such an association similarity hypothesis can be applied to the training of scale coefficient maps, because the nodule segmentation channel and the nodule super-resolution channel should share similar receptive field sizes in the convolution operation. Therefore, the embodiment of the present invention proposes a scale association learning module to enhance the robustness of the scale-aware hole convolution module. First, the scale coefficient maps output by the convolution modules at different levels are scaled to the size of the input image, and then all the scale maps are connected together to form a multi-channel scale coefficient map. The Gram matrix is ​​calculated for the scale coefficient map output by the nodule super-resolution segmentation channel and the scale coefficient map output by the nodule CT image super-resolution channel, and finally the association loss function is calculated for the Gram matrices output by the two channels to optimize the model. The mean square error function is also used as the loss function for training.

[0127] In the specific implementation process, only the nodule super-resolution segmentation branch can be calculated, which can reduce the amount of calculation to a certain extent and shorten the calculation time.

[0128] An embodiment of the present invention provides a dual-path super-resolution segmentation framework with scale perception capability, including a nodule image super-resolution branch and a nodule super-resolution segmentation branch to improve the segmentation of small nodules in low-resolution CT images. A scale-aware adaptive dilated convolution block is proposed, which can automatically adjust the receptive field size in the dilated convolution according to the scale of the input segmentation target, thereby achieving accurate segmentation of objects of different scales. A dual association learning module is also proposed, including feature association learning and scale association learning, to improve the robustness of high-resolution feature extraction and nodule super-resolution segmentation branches.

[0129] This paper proposes an accurate segmentation algorithm for lung nodules in low-resolution CT images. Using a dual-path architecture of image super-resolution and segmentation super-resolution, the model generates high-resolution features from low-resolution images, thereby improving the segmentation of small nodules. Furthermore, a scale-aware dilated convolutional module enhances the model's adaptability to lung nodules of varying scales, achieving multi-scale segmentation. Furthermore, an associative learning module is employed to ensure the robustness of the entire model.

[0130] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0131] The image segmentation method provided by an embodiment of the present invention obtains a first image to be segmented; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution, a target image of the second resolution corresponding to the target object is determined. The dual-path super-resolution segmentation structure model provided by the embodiment of the present invention can draw on the idea of ​​image super-resolution to greatly improve the segmentation effect of tiny nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales.

[0132] Another embodiment of the present invention provides an image segmentation device for executing the image segmentation method provided in the above embodiment.

[0133] Reference Figure 4, shows a structural block diagram of an embodiment of an image segmentation device of the present invention. The device may specifically include the following modules: an acquisition module 401 and a segmentation module 402, wherein:

[0134] The acquisition module 401 is used to acquire a first image to be segmented; wherein the first image is of a first resolution and includes a target object;

[0135] The segmentation module 402 is configured to input the first image into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image at a second resolution and a segmentation mask of the target object at the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; and the second resolution is greater than the first resolution.

[0136] The determination module 403 is configured to determine a target image with the second resolution corresponding to the target object according to the predicted image with the second resolution and the segmentation mask of the target object with the second resolution.

[0137] An image segmentation device provided by an embodiment of the present invention obtains a first image to be segmented; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution, a target image of the second resolution corresponding to the target object is determined. Through the dual-path super-resolution segmentation structure model provided by the embodiment of the present invention, the idea of ​​image super-resolution can be used to greatly improve the segmentation effect of small nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales.

[0138] Another embodiment of the present invention further supplements the image segmentation device provided in the above embodiment.

[0139] Optionally, the device further comprises a model building module, the model building module being configured to:

[0140] Acquire a sample image of a third resolution, and acquire a sample image of a fourth resolution corresponding to the sample image of the third resolution;

[0141] The high-resolution segmentation neural network and the image super-resolution neural network are trained respectively according to the sample image to obtain a segmented image with a fifth resolution; the fifth resolution is greater than the third resolution, and the fifth resolution is the same as the fourth resolution;

[0142] Determining a loss function of a high-resolution segmentation neural network and a loss function of an image super-resolution neural network based on the sample image of the third resolution, the sample image of the fourth resolution, and the segmented image of the fifth resolution;

[0143] When the loss function of the high-resolution segmentation neural network is less than a first preset value and / or the loss function of the image super-resolution neural network is less than a second preset value, the high-resolution segmentation neural network and the image super-resolution neural network are determined as the dual-path super-resolution segmentation structure model.

[0144] Optionally, the segmentation module is further configured to:

[0145] According to the pre-trained dilated convolutional neural network, the receptive field image corresponding to the target object is determined.

[0146] Optionally, the model building module is used to:

[0147] According to the size of the target object, the scale-aware dilated convolutional neural network in the dual-path super-resolution segmentation structure model is trained to obtain a dilated convolutional neural network; the dilated convolutional neural network includes multiple parallel dilated convolution kernels with different dilation rates, where each dilated convolution kernel corresponds to a scale coefficient map.

[0148] Optionally, the segmentation module is used to:

[0149] The first image is segmented according to a pre-trained association learning model to obtain a segmented image corresponding to the target object, wherein the association learning model is obtained by association learning a high-resolution segmentation neural network and an image super-resolution neural network.

[0150] Optionally, the model building module is used to:

[0151] Perform feature association learning on the high-resolution segmentation neural network and image super-resolution neural network in the dual-path super-resolution segmentation structure model;

[0152] Scale-related learning is performed on the high-resolution segmentation neural network and image super-resolution neural network in the dual-path super-resolution segmentation structure model.

[0153] Optionally, the model building module is used to:

[0154] Determine a first feature map and a second feature map according to an input vector of the sample image, a high-resolution segmentation neural network, and an image super-resolution neural network;

[0155] Calculate the Gram matrix of the first feature map and the second feature map according to the calculation method of the correlation coefficient;

[0156] Calculate the correlation loss function between the Gram matrix of the first feature map and the Gram matrix of the second feature map;

[0157] According to the associated loss function, the dual-path super-resolution segmentation structure model is trained;

[0158] The Model Building Module is also used to:

[0159] The scale coefficient maps corresponding to the dilated convolution kernels at different levels are scaled to obtain multiple sub-scale maps, where the sub-scale maps are the same size as the sample image.

[0160] Connecting the multiple sub-scale maps to determine a multi-channel scale coefficient map; wherein the scale coefficient map includes a first scale coefficient map corresponding to the high-resolution segmentation neural network and a second scale coefficient map corresponding to the image super-resolution neural network;

[0161] determining a scale Gram matrix based on the first scale coefficient map and the second scale coefficient map;

[0162] Determine the associated loss function based on the scale Gram matrix;

[0163] The dual-path super-resolution segmentation structure model is trained according to the associated loss function.

[0164] It should be noted that each implementable method in this embodiment can be implemented separately, or can be implemented in combination in any combination without conflict, and this application does not limit it.

[0165] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0166] An image segmentation device provided by an embodiment of the present invention obtains a first image to be segmented; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution, a target image of the second resolution corresponding to the target object is determined. Through the dual-path super-resolution segmentation structure model provided by the embodiment of the present invention, the idea of ​​image super-resolution can be used to greatly improve the segmentation effect of small nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales.

[0167] Yet another embodiment of the present invention provides a terminal device for executing the image segmentation method provided in the above embodiment.

[0168] Figure 5 This is a schematic diagram of the structure of a terminal device of the present invention, such as Figure 5 As shown, the terminal device includes: at least one processor 501 and a memory 502;

[0169] The memory stores a computer program; and at least one processor executes the computer program stored in the memory to implement the image segmentation method provided in the above embodiment.

[0170] The terminal device provided by this embodiment obtains a first image to be segmented; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution, a target image of the second resolution corresponding to the target object is determined. The dual-path super-resolution segmentation structure model provided by the embodiment of the present invention can draw on the idea of ​​image super-resolution to greatly improve the segmentation effect of tiny nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales.

[0171] Yet another embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the image segmentation method provided by any of the above embodiments is implemented.

[0172] According to the computer-readable storage medium of this embodiment, a first image to be segmented is obtained; wherein the first image is of a first resolution and includes a target object; the first image is input into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image of a second resolution and a segmentation mask of the target object of the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; the second resolution is greater than the first resolution, and according to the predicted image of the second resolution and the segmentation mask of the target object of the second resolution, a target image of the second resolution corresponding to the target object is determined. The dual-path super-resolution segmentation structure model provided by the embodiment of the present invention can draw on the idea of ​​image super-resolution to greatly improve the segmentation effect of tiny nodules in low-resolution CT images, generate high-resolution images, and achieve accurate segmentation of target objects of different scales.

[0173] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0174] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The embodiments of the present invention are described with reference to flowcharts and / or block diagrams of methods, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing electronic device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing electronic device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing electronic device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing electronic device so that a series of operating steps are executed on the computer or other programmable electronic device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable electronic device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0178] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0179] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or electronic device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements that are inherent to such process, method, article, or electronic device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or electronic device that includes the element.

[0180] The above is a detailed introduction to an image segmentation method and an image segmentation device provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An image segmentation method, characterized in that: The method comprises: Acquire a first image to be segmented; wherein the first image has a first resolution and includes a target object; Inputting the first image into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image at a second resolution and a segmentation mask of the target object at a second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; and the second resolution is greater than the first resolution; Determining a target image at a second resolution corresponding to the target object based on the predicted image at the second resolution and the segmentation mask of the target object at the second resolution, the method comprising: Determining a receptive field image corresponding to the target object based on a pre-trained dilated convolutional neural network; The method comprises: According to the size of the target object, the scale-aware dilated convolutional neural network in the dual-path super-resolution segmentation structure model is trained to obtain the dilated convolutional neural network; the dilated convolutional neural network includes multiple parallel dilated convolution kernels with different dilation rates, wherein each dilated convolution kernel corresponds to a scale coefficient map; Characterized in that the method comprises: Segmenting the first image according to a pre-trained association learning model to obtain a segmented image corresponding to the target object, wherein the association learning model is obtained by association learning the high-resolution segmentation neural network and the image super-resolution neural network; The association learning model is obtained in the following way: Performing feature association learning on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model; Scale association learning is performed on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model.

2. The method according to claim 1, wherein the dual-path super-resolution segmentation structure model is obtained by: Acquire a sample image of a third resolution, and acquire a sample image of a fourth resolution corresponding to the sample image of the third resolution; Training the high-resolution segmentation neural network and the image super-resolution neural network respectively according to the sample image to obtain a segmented image of the fifth resolution; The fifth resolution is greater than the third resolution, and the fifth resolution is the same as the fourth resolution; Determining a loss function of a high-resolution segmentation neural network and a loss function of an image super-resolution neural network based on the sample image of the third resolution, the sample image of the fourth resolution, and the segmented image of the fifth resolution; When the loss function of the high-resolution segmentation neural network is less than a first preset value and / or the loss function of the image super-resolution neural network is less than a second preset value, the high-resolution segmentation neural network and the image super-resolution neural network are determined as the dual-path super-resolution segmentation structure model.

3. The method according to claim 1, characterized in that The performing feature association learning on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model includes: Determining a first feature map and a second feature map respectively according to an input vector of the sample image, the high-resolution segmentation neural network, and the image super-resolution neural network; Calculate the Gram matrix of the first feature map and the second feature map according to the calculation method of the correlation coefficient; Calculate the correlation loss function between the Gram matrix of the first feature map and the Gram matrix of the second feature map; Training the dual-path super-resolution segmentation structure model according to the association loss function; Correspondingly, the scale association learning of the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model includes: Scaling the scale coefficient maps corresponding to the dilated convolution kernels at different levels to obtain multiple sub-scale maps, where the sub-scale maps have the same size as the sample image; Connecting the multiple sub-scale maps to determine a multi-channel scale coefficient map; wherein the scale coefficient map includes a first scale coefficient map corresponding to the high-resolution segmentation neural network and a second scale coefficient map corresponding to the image super-resolution neural network; determining a scaled Gram matrix according to the first scale coefficient map and the second scale coefficient map; determining an association loss function based on the scale Gram matrix; The dual-path super-resolution segmentation structure model is trained according to the association loss function.

4. An image segmentation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first image to be segmented; wherein the first image has a first resolution and includes a target object; a segmentation module, configured to input the first image into a pre-trained dual-path super-resolution segmentation structure model to obtain a predicted image at a second resolution and a segmentation mask of the target object at the second resolution; wherein the dual-path super-resolution segmentation structure model includes at least a high-resolution segmentation neural network and an image super-resolution neural network; and the second resolution is greater than the first resolution; a determination module, configured to determine a target image at the second resolution corresponding to the target object based on the predicted image at the second resolution and the segmentation mask of the target object at the second resolution; Determining a receptive field image corresponding to the target object based on a pre-trained dilated convolutional neural network; include: According to the size of the target object, the scale-aware dilated convolutional neural network in the dual-path super-resolution segmentation structure model is trained to obtain the dilated convolutional neural network; the dilated convolutional neural network includes multiple parallel dilated convolution kernels with different dilation rates, wherein each dilated convolution kernel corresponds to a scale coefficient map; include: Segmenting the first image according to a pre-trained association learning model to obtain a segmented image corresponding to the target object, wherein the association learning model is obtained by association learning the high-resolution segmentation neural network and the image super-resolution neural network; The association learning model is obtained in the following way: Performing feature association learning on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model; Scale association learning is performed on the high-resolution segmentation neural network and the image super-resolution neural network in the dual-path super-resolution segmentation structure model.

5. A terminal device, characterized in that: include: at least one processor and memory; The memory stores a computer program; The at least one processor executes the computer program stored in the memory to implement the image segmentation method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed, implements the image segmentation method according to any one of claims 1 to 3.

Images