Low-resolution image segmentation method
By establishing and training two models for super-resolution conversion and image segmentation, combining self-supervised and pseudo-supervised training, and performing multiple optimization steps, the problem of difficulty in obtaining and labeling of high-resolution magnetic resonance images is solved, and efficient high-resolution image segmentation is achieved.
Patent Information
- Application Number
- CN202510178550.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-18
- Publication Date
- 2025-06-13
AI Technical Summary
In the prior art, high-resolution magnetic resonance images are difficult to obtain, and high-resolution image labeling is time-consuming and labor-intensive, so it is impossible to directly obtain high-resolution image segmentation results through low-resolution magnetic resonance images.
By establishing and training two models: the first model is used to convert low-resolution images into high-resolution images, the second model is used to segment low-resolution images, trained in combination with self-supervised and pseudo-supervised methods, and improve segmentation performance through multiple optimization steps.
With only low-resolution images and labeling, high-resolution segmentation results can be effectively obtained, solving the problem of time-consuming acquisition and labeling of high-resolution data, and improving segmentation performance.
Smart Images
Figure CN120147327A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning, and particularly to a method for low-resolution image segmentation. Background Art
[0002] High-resolution 3D magnetic resonance imaging (MRI) can provide detailed anatomical structure information, enabling precise segmentation of regions of interest in various medical image analysis tasks, which is beneficial to clinical tasks such as high-precision surgical planning and prosthesis preparation.
[0003] Existing deep learning-based high-resolution segmentation models are the mainstream methods for such tasks, with good generalization and accuracy. These methods usually require high-resolution data and their segmentation annotations to effectively train the model. However, such high-resolution data is often difficult to collect in actual clinical scenarios. First, due to the high requirements of the acquisition equipment, scanning time, and signal-to-noise ratio limitations, 2D scanned magnetic resonance images are usually used clinically. Although these images have high in-plane resolution, the inter-plane resolution is low. Second, the segmentation annotation of magnetic resonance images usually requires layer-by-layer delineation of the region of interest, which is a time-consuming and labor-intensive process. For such data, existing methods usually can only obtain low-resolution and low-accuracy segmentation results, often unable to meet the requirements of subsequent tasks. Summary of the Invention
[0004] This application provides a method for low-resolution image segmentation to solve the problems in the prior art that a large number of high-resolution magnetic resonance images are difficult to obtain, the annotation of high-resolution images is time-consuming and labor-intensive, and high-resolution image segmentation results cannot be directly obtained from low-resolution magnetic resonance images.
[0005] This application provides a method for low-resolution image segmentation, which specifically includes an image acquisition step, a first model establishment step, a first model training step, a second model establishment step, a second model training step, and an input step.
[0006] The image acquisition step is used to acquire at least one set of low-resolution images, and the low-resolution images include a first annotation; the first model establishment step is used to establish a first model, and the first model can convert the low-resolution images into high-resolution images, and the high-resolution images include a second annotation; the first model training step is to interpolate the low-resolution images and their first annotations, perform convolution and a first downsampling operation on the interpolated images to obtain an image pair, and use the image pair as supervision to train the high-resolution images and the second annotations output by the first model in a self-supervised manner to obtain a trained first model; the second model establishment step is used to establish a second model, input a low-resolution image into the second model, and perform segmentation on the low-resolution image through LR segmentation and HR segmentation to obtain a low-resolution segmentation result and a high-resolution segmentation result; the second model training step is to input a pseudo low-resolution image into the second model to obtain a pseudo high-resolution segmentation result and a pseudo low-resolution segmentation result, and use the pseudo high-resolution segmentation result and the pseudo low-resolution segmentation result as pseudo supervision to train the high-resolution segmentation result and the low-resolution segmentation result output by the second model in a pseudo-supervised manner; the input step is to input the low-resolution image into the trained second model to obtain a high-resolution segmentation result corresponding to the low-resolution image.
[0007] Further, the first model training step includes an interpolation step, a convolution step, a first downsampling step, and an image output step.
[0008] The interpolation step is to perform interpolation processing on the low-resolution image and its first annotation based on the SMORE technology; the convolution step is to perform convolution on the interpolated low-resolution image with a 1D convolutional Gaussian filter h(x; r) as a slice section, where x represents the object on which the filter acts, and r represents the full width at half maximum of the Gaussian filter; the first downsampling step is to downsample the low-resolution image and its first annotation by r times to obtain a simulated low-resolution image and a simulated first annotation, where the low-resolution image and the simulated low-resolution image form the image pair; the image output step is to input the simulated low-resolution image and the simulated first annotation into the first model to output a first image and a third annotation, and its calculation formula is
[0009]
[0010] where, represents the first image, represents the third annotation, SelfSR represents the first model, I LLR represents the simulated low-resolution image, Y LLR0 represents the simulated first annotation, 1 represents an output terminal of the first model, and 2 represents another output terminal of the first model.
[0011] Further, in the first model training step, there are two loss functions. The first loss function is used to train the first image, and the calculation formula of the first loss function is
[0012] Lsr_ I =||SelfSR(I LLR ,Y LLR ) 1 -I LR || 2
[0013] Wherein, Lsr_ I is the first loss function, and I LR represents the low-resolution image.
[0014] The second loss function is used to train the third annotation, and the calculation formula of the second loss function is
[0015]
[0016] Wherein, L D ice is the second loss function, and Y LR represents the first annotation.
[0017] Further, after the first model training step, there is also a first optimization step. In the first optimization step, the low-resolution image is input into the trained first model, and a first feature map can be obtained. The first feature map is used to extract an uncertainty map, and a fifth loss function is obtained based on the uncertainty map, which is used to accurately represent the high reconstruction error region in the high-resolution image.
[0018] Further, the first optimization step includes a first feature map acquisition step, an intermediate image acquisition step, an attention map acquisition step, a high-resolution image acquisition step, an uncertainty map generation step, and a fifth loss function calculation step.
[0019] In the first feature map acquisition step, the low-resolution image is input into the first model to obtain the first feature map output by the first model. The calculation formula of the first feature map is
[0020] Fm=SelfSR(I LR ,Y LR )m 1
[0021] Wherein, Fm represents the first feature map, I LR represents the low-resolution image, and Y LRdenote the first annotation, and m1 denote an intermediate output feature of the first model.
[0022] The intermediate image acquisition step is to obtain N intermediate images based on the first feature map through convolution and activation functions. The calculation formula of the intermediate image output step is
[0023]
[0024] where, denote 3D convolution applied to the intermediate features at the low-resolution image level, denote 3D convolution applied to the segmentation features at the first annotation level, and tanh denote the activation function, denote any one of the intermediate images, denote the annotation corresponding to any one of the intermediate images.
[0025] The attention map acquisition step is to obtain N attention maps through convolution and the channel softmax function. The calculation formula of the attention map acquisition step is
[0026]
[0027] where, denote any one of the attention maps, and Softmax denote the channel softmax function, denote 3D convolution applied to the intermediate features of the attention map.
[0028] The high-resolution image acquisition step is to obtain the high-resolution image based on the attention map using element-wise multiplication. The calculation formula is
[0029]
[0030] where, denote the high-resolution image, denote the second annotation.
[0031] The uncertainty map generation step is to generate an uncertainty map based on the attention map. The calculation formula of the uncertainty map generation step is
[0032]
[0033] where U denote the uncertainty map, σ denote the Sigmoid activation function, conv denote convolution, and [] denote concatenation along the channel dimension.
[0034] The fifth loss function calculation step is to obtain a fifth loss function based on the uncertainty map, which is used to accurately represent the high reconstruction error region in the high-resolution image. The calculation formula of the fifth loss function calculation step is
[0035]
[0036] Among them, is the original pixel-level loss function of the first model, is the fifth loss function.
[0037] Furthermore, the second model training step includes a second downsampling step and a pseudo-data input step.
[0038] The second downsampling step is to obtain the pseudo low-resolution image by downsampling the high-resolution image. The formula for the second downsampling step is
[0039]
[0040] Among them, represents the pseudo low-resolution image, represents the pseudo low-resolution segmentation result, represents the high-resolution image, represents the second annotation, *z represents the 1D convolution operation of the high-resolution image along the z-axis in the three-dimensional coordinate system, represents the downsampling operation of the high-resolution image along the z-axis with a magnification factor of r in the three-dimensional coordinate system, and h(z; r) represents that the object of the Gaussian filter is the z-axis.
[0041] The pseudo-data input step is to input the pseudo low-resolution image into the second model, and obtain the pseudo low-resolution segmentation result and the pseudo high-resolution segmentation result through LR segmentation and HR segmentation.
[0042] Furthermore, there are two loss functions in the second model training step. The third loss function is used to train the LR segmentation, and the calculation formula of the third loss function is
[0043]
[0044] Among them, Seg represents the segmentation network, represents the pseudo first annotation, and also represents the pseudo low-resolution segmentation result.
[0045] The fourth loss function is used to train the HR segmentation, and the calculation formula of the fourth loss function is
[0046]
[0047] Among them, represents the pseudo high-resolution segmentation result.
[0048] Further, after the second model training step, a second optimization step and a third optimization step are further included. The second optimization step is to use the uncertainty map as the loss weight of the second model, obtain a sixth loss function, and optimize the second model training step. The third optimization step is to input the low-resolution image into the trained first model to obtain a second feature map. After inputting the low-resolution image into the trained first model, perform a second downsampling, and input the second downsampled image into the second model to obtain a third feature map. Using the second feature map as supervision, perform supervised training on the third feature map output by the second model to optimize the second model training step and obtain an optimized second model.
[0049] Further, the second optimization step includes a normalization step and a regularization constraint step. The normalization step is to normalize the numerical range of the uncertainty map to [0, 1]. The regularization constraint step is to perform a regularization constraint on the fourth loss function based on the uncertainty map. The calculation formula of the regularization constraint step is
[0050]
[0051] where is the fourth loss function after regularization constraint, that is, the sixth loss function.
[0052] Further, the third optimization step includes a second feature map acquisition step, a third feature map acquisition step, a fourth feature map acquisition step, a similarity calculation step, and a seventh loss function calculation step.
[0053] The second feature map acquisition step is to input the low-resolution image into the first model to obtain the second feature map output by the first model. The calculation formula of the second feature map is
[0054] Fsr = SelfSR(I LR , Y LR )m 2
[0055] where Fsr represents the second feature map and m2 represents an intermediate output feature of the first model.
[0056] The third feature map acquisition step is to input the low-resolution image into the second model to obtain the third feature map output by the second model. The calculation formula of the third feature map is
[0057] Fseg = Seg(I LR , Y LR ) M2
[0058] Among them, Fseg represents the second feature map, and M2 represents an intermediate output feature of the second model.
[0059] The step of obtaining the fourth feature map is based on the method of bilinear interpolation to align the shape of the second feature map with the shape of the third feature map, obtaining a feature map of C'×D'×H'×W', where C represents the channel, D represents the depth, H represents the height, and W represents the width.
[0060] The similarity calculation step is used to obtain a fully connected affinity graph. The granularity of the affinity graph is β, and the affinity graph includes fully connected nodes, and calculate the similarity aij between the i-th node and the j-th node. The calculation formula is:
[0061]
[0062] Among them, T represents the transpose, p represents the model, Fp,j represents the value of the j-th node in a certain model, and Fp,i represents the value of the j-th node in a certain model.
[0063] The seventh loss function calculation step is used to calculate the squared difference of the similarity to represent the seventh loss function. The calculation formula of the seventh loss function calculation step is
[0064]
[0065] Among them, represents the similarity between the i-th and j-th nodes in the second feature map, represents the similarity between the i-th and j-th nodes in the third feature map, and Lcorr is the seventh loss function.
[0066] This application provides a low-resolution image segmentation method. The high-resolution image is obtained through the first model, the high-resolution segmentation result is obtained through the second model, and the method is optimized from the data level, loss function level, and network parameter level through three optimization steps, effectively improving the performance of high-resolution segmentation, enabling the second model to obtain an effective high-resolution segmentation result with only low-resolution images and annotations, and solving the problems in the prior art that a large number of high-resolution magnetic resonance images are difficult to obtain, high-resolution annotation is time-consuming and laborious, and the high-resolution segmentation result cannot be directly obtained from low-resolution magnetic resonance images. Description of the Drawings
[0067] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0068] Figure 1 is the flowchart of the low-resolution image segmentation method described in the embodiments of the present invention;
[0069] Figure 2 is the flowchart of the first model training step described in the embodiments of the present invention;
[0070] Figure 3 is the flowchart of the first optimization step described in the embodiments of the present invention;
[0071] Figure 4 is the flowchart of the second model training step described in the embodiments of the present invention;
[0072] Figure 5 is the flowchart of the second optimization step described in the embodiments of the present invention;
[0073] Figure 6 is the flowchart of the third optimization step described in the embodiments of the present invention;
[0074] Figure 7 is the quantitative comparison table of high-resolution segmentation of the present invention and existing methods on the public dataset;
[0075] Figure 8 is the qualitative comparison diagram of high-resolution segmentation of the present invention and existing methods on the public dataset;
[0076] Figure 9 is the qualitative comparison diagram of high-resolution segmentation of the present invention and the nnUNet method on the private dataset. Detailed implementation manners
[0077] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.
[0078] As Figure 1As shown in the figure, the present application provides a low-resolution image segmentation method, which specifically includes step S1) image acquisition step, step S2) first model establishment step, step S3) first model training step, step S4) first optimization step, step S5) second model establishment step, step S6) second model training step, step S7) second optimization step, step S8) third optimization step, and step S9) input step.
[0079] Step S1) Image acquisition step: acquire at least one set of low-resolution images, and the low-resolution images include first annotations.
[0080] In this embodiment, the low-resolution image is a 2D scanned magnetic resonance image, and the first annotation is a low-resolution annotation corresponding to the 2D scanned magnetic resonance image.
[0081] Step S2) First model establishment step: establish a first model, and the first model can convert the low-resolution image into a high-resolution image, and the high-resolution image includes second annotations.
[0082] In this embodiment, the first model is a super-resolution model, the high-resolution image is a 3D scanned magnetic resonance image, and the second annotation is a high-resolution annotation corresponding to the 3D scanned magnetic resonance image.
[0083] Step S3) First model training step: interpolate the low-resolution image and its first annotation, perform convolution and first downsampling operations on the interpolated image to obtain an image pair, and use the image pair as supervision to train the high-resolution image and the second annotation output by the first model in a self-supervised manner to obtain the trained first model.
[0084] In this embodiment, for the low-resolution image I LR and its corresponding first annotation Y LR , both have a spatial resolution of a×a×c. Assuming that the spatial resolutions in the frequency encoding and phase encoding directions are the same, the super-resolution model uses a magnification factor r to generate a high-resolution image with a spatial resolution of a×a×c. In the prior art, if a large number of high-resolution magnetic resonance images need to be acquired, high-precision acquisition equipment is required. The acquired high-resolution images occupy a large amount of storage space, and the hardware costs of cameras, hard disks, etc. are relatively high; in addition, high-resolution magnetic resonance images require a long scanning time during shooting and a long transmission time during transmission, and their signal-to-noise ratio is relatively high. It is time-consuming and laborious to make annotations on high-resolution magnetic resonance images.
[0085] In the proposed solution, only low-resolution magnetic resonance images need to be collected and labeled. After image processing by computer software, high-resolution images with labels can be obtained without directly collecting and labeling high-resolution magnetic resonance images, solving the problems that it is difficult to obtain a large number of high-resolution magnetic resonance images and that high-resolution labeling is time-consuming and laborious.
[0086] As Figure 2 shown, step S3) the first model training step includes step S31) the interpolation step, step S32) the convolution step, step S33) the first downsampling step, and step S34) the image output step.
[0087] In step S31) the interpolation step, based on the SMORE technique, interpolation processing is performed on the low-resolution image and its first label to achieve isotropic voxel spacing between the low-resolution image and the first label, thereby ensuring spatial consistency in the first model training stage and the first model inference stage.
[0088] In step S32) the convolution step, a 1D convolutional Gaussian filter h(x; r) is used as a slice section to perform convolution on the interpolated low-resolution image, where x represents the object on which the filter acts, and r represents the full width at half maximum of the Gaussian filter, and r also represents the magnification factor adopted by the super-resolution model.
[0089] In step S33) the first downsampling step, the low-resolution image and its first label are downsampled by a factor of r to obtain a simulated low-resolution image and a simulated first label. Among them, the low-resolution image and the simulated low-resolution image form the image pair. In this embodiment, the low-resolution image and the simulated low-resolution image are used as the supervision to train the first model in a self-supervised manner.
[0090] In step S34) the image output step, the simulated low-resolution image and the simulated first label are input into the first model, and a first image and a third label are output. The calculation formula is
[0091]
[0092] where represents the first image, represents the third label, SelfSR represents the first model, I LLR represents the simulated low-resolution image, Y LLR represents the simulated first label, 1 represents one output end of the first model, and 2 represents the other output end of the first model.
[0093] In this embodiment, the simulated low-resolution image I LLR compared with the first image The simulated low-resolution image I LLR is a low-resolution image, and the first image is a high-resolution image; the simulated first annotation Y LLR Compared with the third annotation The simulated first annotation Y LLR is a low-resolution annotation, and the third annotation is a high-resolution annotation. Therefore, during the training of the first model, the real 3D scanned magnetic resonance image is not acquired, and the training process of obtaining the high-resolution image from the low-resolution image by the super-resolution model can be achieved without the real 3D scanned magnetic resonance image.
[0094] Furthermore, there are two loss functions in the first model training step. The first loss function is used to train the first image, and the calculation formula of the first loss function is
[0095] L sr_I = ||SelfSR(I LLR , Y LLR ) 1 - I LR || 2
[0096] where Lsr_ I is the first loss function, I LR represents the low-resolution image, and the first loss function is used to implement the training process from the simulated low-resolution image I LLR to the first image .
[0097] The second loss function is used to train the third annotation, and the calculation formula of the second loss function is
[0098]
[0099] where L D ice is the second loss function, Y LR represents the first annotation, and the second loss function is used to implement the training process from the simulated first annotation Y LLR to the third annotation .
[0100] In this embodiment, during the process of obtaining high-resolution images through the super-resolution model, tissues in medical images often overlap, and their boundaries become blurred, resulting in a high degree of uncertainty in the segmentation of these regions. This problem not only leads to unreliable segmented regions but also causes poorly reconstructed regions in the super-resolution task. Therefore, we estimate uncertainty in the self-supervised super-resolution model by designing an uncertainty-aware super-resolution module, thereby identifying regions with high reconstruction errors and providing guidance for the segmentation task. Thus, it is necessary to optimize the first model training step in step S3).
[0101] Step S4) First optimization step: Input the low-resolution image into the trained first model, and a first feature map can be obtained. The first feature map is used to extract an uncertainty map, and a fifth loss function is obtained based on the uncertainty map to accurately represent the regions with high reconstruction errors in the high-resolution image.
[0102] As Figure 3 shown, step S4) First optimization step includes step S41) First feature map acquisition step, step S42) Intermediate image acquisition step, step S43) Attention map acquisition step, step S44) High-resolution image acquisition step, step S45) Uncertainty map generation step, and step S46) Fifth loss function calculation step.
[0103] Step S41) First feature map acquisition step: Input the low-resolution image into the first model to obtain the first feature map output by the first model. The calculation formula for the first feature map is
[0104] Fm = SelfSR(I LR , Y LR )m 1
[0105] where Fm represents the first feature map, I LR represents the low-resolution image, Y LR represents the first annotation, and m1 represents an intermediate output feature of the first model.
[0106] Step S42) Intermediate image acquisition step: Based on the first feature map, N intermediate images are obtained through convolution and activation functions. The calculation formula for the intermediate image output step is
[0107]
[0108] where represents 3D convolution applied to the intermediate features at the low-resolution image level, represents 3D convolution applied to the segmentation features at the first annotation level, and tanh represents the activation function. Denote any intermediate image, Denote the annotation corresponding to any intermediate image.
[0109] Step S43) Attention map acquisition step, obtaining N attention maps through convolution and channel softmax function, and the calculation formula of the attention map acquisition step is
[0110]
[0111] where, Denote any attention map, Softmax denotes the channel softmax function, Denote the 3D convolution of the intermediate features applied to the attention map.
[0112] Step S44) High-resolution image acquisition step, based on the attention map, obtaining the high-resolution image using element-wise multiplication, and the calculation formula is
[0113]
[0114] where, Denote the high-resolution image, Denote the second annotation, That is, denote taking the low-resolution image I LR and its corresponding first annotation Y LR as inputs, and obtaining the high-resolution image and the second annotation after inputting them into the first model.
[0115] Step S45) Uncertainty map generation step, generating an uncertainty map based on the attention map, and the calculation formula of the uncertainty map generation step is
[0116]
[0117] where U denotes the uncertainty map, σ denotes the Sigmoid activation function, conv denotes convolution, and [] denotes concatenation along the channel dimension.
[0118] Step S46) Fifth loss function calculation step, obtaining a fifth loss function based on the uncertainty map, which is used to accurately represent the high reconstruction error region in the high-resolution image, and the calculation formula of the fifth loss function calculation step is
[0119]
[0120] where, is the original pixel-level loss function of the first model, is the fifth loss function, and the fifth loss function will replace the original pixel-level loss function of the first model when using the low-resolution image I LRand its corresponding first annotation Y LR as the input, after inputting into the first model, the fifth loss function can accurately represent the high-resolution image in the high reconstruction error region, making the high-resolution image obtained through the first model more accurate.
[0121] Step S5) Second model establishment step: Establish a second model. Input a low-resolution image into the second model. Through LR segmentation and HR segmentation, the low-resolution image is segmented to obtain a low-resolution segmentation result and a high-resolution segmentation result. At this time, there is still a large error in the obtained high-resolution segmentation result, so the second model needs to be trained to obtain the final high-resolution segmentation result.
[0122] Step S6) Second model training step: Input a pseudo low-resolution image into the second model to obtain a pseudo high-resolution segmentation result and a pseudo low-resolution segmentation result. Use the pseudo high-resolution segmentation result and the pseudo low-resolution segmentation result as pseudo supervision, and train the high-resolution segmentation result and the low-resolution segmentation result output by the second model in a pseudo-supervised manner.
[0123] As Figure 4 shown, step S6) Second model training step includes step S61) Second downsampling step and step S62) Pseudo data input step.
[0124] Step S61) Second downsampling step: Obtain the pseudo low-resolution image by downsampling the high-resolution image. The formula of the second downsampling step is
[0125]
[0126] where represents the pseudo low-resolution image, represents the pseudo low-resolution segmentation result, represents the high-resolution image, represents the second annotation, *z represents the 1D convolution operation of the high-resolution image along the z-axis in the three-dimensional coordinate system, represents the downsampling operation of the high-resolution image along the z-axis with a magnification factor of r in the three-dimensional coordinate system, and h(z; r) represents that the object of the Gaussian filter is the z-axis.
[0127] Step S62) Pseudo data input step: Input the pseudo low-resolution image into the second model, and obtain the pseudo low-resolution segmentation result and the pseudo high-resolution segmentation result through LR segmentation and HR segmentation.
[0128] Further, in the second model training step of step S6), there are two loss functions. The third loss function is used to train the LR segmentation, and the calculation formula of the third loss function is
[0129]
[0130] where Seg represents the segmentation network, that is, the second model, represents the pseudo first annotation, and also represents the pseudo low-resolution segmentation result. Through the third loss function, the low-resolution segmentation result of the low-resolution image can be obtained. The purpose of the present invention is to obtain the high-resolution segmentation result, so the third loss function is no longer optimized.
[0131] The fourth loss function is used to train the HR segmentation, and the calculation formula of the fourth loss function is
[0132]
[0133] where, represents the pseudo high-resolution segmentation result. Through the fourth loss function, the high-resolution segmentation result of the low-resolution image can be obtained. However, due to the frequent overlap of tissues in medical images and the blurring of their boundaries, the segmentation of these regions has high uncertainty. This problem not only leads to unreliable segmentation regions. Therefore, we estimate the uncertainty in the self-supervised super-resolution model by designing an uncertainty-aware super-resolution head to provide guidance for the segmentation task and optimize the fourth loss function to improve the accuracy of the high-resolution segmentation result.
[0134] Step S7) Second optimization step: Use the uncertainty map as the loss weight of the second model to obtain a sixth loss function and optimize the second model training step of step S6).
[0135] As Figure 5 shown, the second optimization step of step S7) includes step S71) normalization step and step S72) regularization constraint step.
[0136] Step S71) normalization step: Normalize the numerical range of the uncertainty map to [0, 1].
[0137] Step S72) regularization constraint step: Based on the uncertainty map, perform regularization constraint on the fourth loss function. The calculation formula of the regularization constraint step is
[0138]
[0139] where, is the fourth loss function after regularization constraint, that is, the sixth loss function.
[0140] In this embodiment, given the differences between the image reconstruction and segmentation tasks, the traditional distillation method that constrains features through pixel alignment can only provide limited help. Instead, the structural associations between regions in the feature maps generated by different tasks show similar patterns, which are called association patterns. To improve the accuracy of the high-resolution segmentation results, it is necessary to further optimize the second model training step in step S6). By distilling these association patterns from the first model, a seventh loss function is obtained to improve the accuracy of the second model's segmentation task.
[0141] Step S8) Third optimization step: Input the low-resolution image into the trained first model to obtain a second feature map. After inputting the low-resolution image into the trained first model, perform a second downsampling, and then input the second downsampled image into the second model to obtain a third feature map. Using the second feature map as supervision, perform supervised training on the third feature map output by the second model to optimize the second model training step in step S6) and obtain an optimized second model.
[0142] As Figure 6 shown, step S8) third optimization step includes step S81) second feature map acquisition step, step S82) third feature map acquisition step, step S83) fourth feature map acquisition step, step S84) similarity calculation step, and step S85) seventh loss function calculation step.
[0143] Step S81) Second feature map acquisition step: Input the low-resolution image into the first model to obtain the second feature map output by the first model. The calculation formula for the second feature map is
[0144] Fsr = SelfSR(I LR , Y LR )m 2
[0145] where Fsr represents the second feature map and m2 represents an intermediate output feature of the first model.
[0146] Step S82) Third feature map acquisition step: Input the low-resolution image into the second model to obtain the third feature map output by the second model. The calculation formula for the third feature map is
[0147] Fseg = Seg(I LR , Y LR ) M2
[0148] where Fseg represents the second feature map and M2 represents an intermediate output feature of the second model.
[0149] Step S83) Fourth feature map acquisition step: Based on the bilinear interpolation method, align the shape of the second feature map with the shape of the third feature map to obtain a feature map of C'×D'×H'×W', that is, the fourth feature map, where C represents the channel, D represents the depth, H represents the height, and W represents the width.
[0150] Step S84) Similarity calculation step: Obtain a fully connected affinity graph, where the affinity graph is the affinity matrix of the fourth feature map, the granularity of the affinity graph is β, and the affinity graph includes fully connected nodes, and calculate the similarity aij between the i-th node and the j-th node. The calculation formula is:
[0151]
[0152] where T represents transpose, p represents the model, Fp,j represents the value of the j-th node in a certain model, and Fp,i represents the value of the j-th node in a certain model.
[0153] Step S85) Seventh loss function calculation step: Calculate the squared difference of the similarities to represent the seventh loss function. The calculation formula of the seventh loss function calculation step is
[0154]
[0155] where represents the similarity between the i-th and j-th nodes in the second feature map, represents the similarity between the i-th and j-th nodes in the third feature map, Lcorr is the seventh loss function, and the seventh loss function is the loss of distilling these association patterns in the first model.
[0156] Step S9) Input step: Input the low-resolution image into the trained second model to obtain the high-resolution segmentation result corresponding to the low-resolution image.
[0157] As Figure 7 shown, we evaluate the performance of the present invention in high-resolution segmentation from low-resolution middle images by comparing with other methods. The other high-resolution segmentation methods include 3D methods (PFSeg, DS2F) or 2D methods (ISDNet) and the training protocol of B-spline interpolation data, and the training protocol of B-spline interpolation data enables the trained model (nnUNet) to use the upsampled low-resolution image as input to generate high-resolution segmentation results. Due to the increase in the input size, this will consume more computing resources during the training and inference processes. We also compare the performance of fully supervised learning that requires high-resolution data for training and input.
[0158] Figure 7 The quantitative results in show that the present invention achieves top performance without any high-resolution data and annotations. Notably, our HD95 metric even outperforms fully supervised methods, which can be attributed to our effective super-resolution assisted segmentation design.
[0159] As Figure 8 shown, the qualitative results of three different subjects also support the reported metrics. For example, 2D methods such as ISDNet show better results in the sagittal plane, but the overall performance is inferior to 3D methods such as PFSeg. In addition, training with interpolated data combined with a powerful segmentation framework provides a strong baseline for this task; however, the boundary accuracy of these interpolated results is low, possibly due to misalignment between the interpolated images and labels. In contrast, the segmentation results of REHRSeg have the boundaries closest to the ground truth annotations and can provide excellent performance even in challenging cases (such as the coronal plane view) where it is difficult for other methods to obtain satisfactory results.
[0160] As Figure 9 shown, we conducted a qualitative evaluation on a private dataset. Since there is currently no method to train a high-resolution segmentation model with only low-resolution data, we can only compare by training a low-resolution segmentation model (nnUNet) and inputting high-resolution data in the test phase. As can be seen from the following figure, the proposed method can effectively reconstruct high-resolution tumor segmentation results from low-resolution data, while nnUNet cannot obtain accurate results.
[0161] This application provides a low-resolution image segmentation method, including an image acquisition step, a first model establishment step, a first model training step, a first optimization step, a second model establishment step, a second model training step, a second optimization step, a third optimization step, and an input step. The high-resolution image is obtained through the first model, and the high-resolution segmentation result is obtained through the second model. The method is optimized from the data level, loss function level, and network parameter level through three optimization steps, effectively improving the performance of high-resolution segmentation, enabling the second model to obtain an effective high-resolution segmentation result with only low-resolution images and annotations, and solving the problems in the prior art that a large number of high-resolution magnetic resonance images are difficult to obtain, high-resolution annotation is time-consuming and laborious, and it is impossible to directly obtain high-resolution segmentation results from low-resolution magnetic resonance images.
[0162] The above has introduced in detail a low-resolution image segmentation method provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.
Claims
1. A low-resolution image segmentation method, characterized in that: The specific steps include: An image acquisition step of acquiring at least one set of low-resolution images, wherein the low-resolution images include a first annotation; A first model building step, building a first model, wherein the first model can convert the low-resolution image into a high-resolution image, wherein the high-resolution image includes a second annotation; A first model training step, interpolating the low-resolution image and the first annotation thereof, performing convolution and a first downsampling operation on the interpolated image to obtain an image pair, and using the image pair as supervision, training the high-resolution image and the second annotation output by the first model in a self-supervised manner to obtain a trained first model; A second model building step, building a second model, inputting a low-resolution image into the second model, segmenting the low-resolution image through LR segmentation and HR segmentation, and obtaining a low-resolution segmentation result and a high-resolution segmentation result; a second model training step, inputting a pseudo low-resolution image into the second model to obtain a pseudo high-resolution segmentation result and a pseudo low-resolution segmentation result, using the pseudo high-resolution segmentation result and the pseudo low-resolution segmentation result as pseudo supervision, and training the high-resolution segmentation result and the low-resolution segmentation result output by the second model in a pseudo-supervisory manner; as well as The input step is to input the low-resolution image into the trained second model to obtain a high-resolution segmentation result corresponding to the low-resolution image.
2. The low-resolution image segmentation method according to claim 1, characterized in that: The first model training step comprises the following steps: An interpolation step, based on the SMORE technology, interpolating the low-resolution image and the first annotation thereof; A convolution step, using a 1D convolution Gaussian filter h(x; r) as a slice section to convolve the interpolated low-resolution image, wherein x represents the object of the filter, and r represents the full width at half height of the Gaussian filter; a first downsampling step of downsampling the low-resolution image and the first annotation by a factor of r to obtain a simulated low-resolution image and a simulated first annotation, wherein the low-resolution image and the simulated low-resolution image constitute the image pair; and The image output step inputs the simulated low-resolution image and the simulated first annotation into the first model, and outputs the first image and the third annotation, and the calculation formula is: in, represents the first image, represents the third annotation, SelfSR represents the first model, and I LLR Represents a simulated low-resolution image, Y LLR Indicates the simulation of the first label, 1 represents an output end of the first model, and 2 represents the other output end of the first model.
3. The low-resolution image segmentation method according to claim 2, characterized in that: The first model training step includes two loss functions. The first loss function is used to train the first image. The calculation formula of the first loss function is: L sr_I =|‖SelfSR(I LLR ,Y LLR )1-I LR ‖|2 Among them, L sr_I is the first loss function, I LR Represents a low-resolution image; The second loss function is used to train the third annotation. The calculation formula of the second loss function is: Among them, L Dice is the second loss function, Y LR Indicates the first annotation.
4. The low-resolution image segmentation method according to claim 1, characterized in that: After the first model training step, the following steps are also included: In the first optimization step, the low-resolution image is input into the trained first model to obtain a first feature map, which is used to extract an uncertainty map. A fifth loss function is obtained based on the uncertainty map to accurately represent the high reconstruction error area in the high-resolution image.
5. The low-resolution image segmentation method according to claim 4, characterized in that: The first optimization step comprises the following steps: The first feature map acquisition step is to input the low-resolution image into the first model to obtain a first feature map output by the first model. The calculation formula of the first feature map is: F m =SelfSR(I LR ,Y LR ) m1 Among them, F m represents the first feature map, I LR represents a low-resolution image, Y LR represents the first annotation, and m1 represents an intermediate output feature of the first model; The intermediate image acquisition step is to acquire N intermediate images through convolution and activation functions based on the first feature map. The calculation formula of the intermediate image output step is: in, represents the 3D convolution applied to the intermediate features at the low-resolution image level, represents the 3D convolution applied to the segmentation features of the first annotation level, tanh represents the activation function, represents any intermediate image, Indicates the annotation corresponding to any intermediate image; Attention map acquisition step, N attention maps are obtained through convolution and channel soft maximization function. The calculation formula of the attention map acquisition step is: in, represents any attention map, Softmax represents the channel soft maximization function, represents the 3D convolution applied to the intermediate features of the attention map; The high-resolution image acquisition step uses element-by-element multiplication to acquire the high-resolution image based on the attention map. The calculation formula is: in, represents the high-resolution image, represents the second annotation; The uncertain graph generation step generates an uncertain graph based on the attention graph. The calculation formula of the uncertain graph generation step is: Where U represents the uncertainty map, σ represents the Sigmoid activation function, conv represents convolution, and [] represents concatenation along the channel dimension; and A fifth loss function calculation step is to obtain a fifth loss function based on the uncertainty map, which is used to accurately represent the high reconstruction error area in the high-resolution image. The calculation formula of the fifth loss function calculation step is: in, is the original pixel-level loss function of the first model, is the fifth loss function.
6. The low-resolution image segmentation method according to claim 1, characterized in that: The second model training step comprises the following steps: The second downsampling step is to obtain the pseudo low-resolution image by downsampling the high-resolution image. The formula of the second downsampling step is: in, represents a pseudo low-resolution image, represents the pseudo low-resolution segmentation result, represents a high-resolution image, Indicates the second annotation, * z Represents the 1D convolution operation of the high-resolution image along the z-axis in a three-dimensional coordinate system. represents the downsampling operation of the high-resolution image along the z-axis at a magnification of r in the three-dimensional coordinate system, and h(z; r) represents that the Gaussian filter acts on the z-axis; and The pseudo data input step is to input the pseudo low-resolution image into the second model, and obtain the pseudo low-resolution segmentation result and the pseudo high-resolution segmentation result through LR segmentation and HR segmentation.
7. The low-resolution image segmentation method according to claim 1, characterized in that: The second model training step includes two loss functions. The third loss function is used to train the LR segmentation. The calculation formula of the third loss function is: Among them, Seg represents the segmentation network, represents a pseudo first annotation, also represented as the pseudo low-resolution segmentation result; The fourth loss function is used to train the HR segmentation. The calculation formula of the fourth loss function is: in, Represented as the pseudo high-resolution segmentation result.
8. The low-resolution image segmentation method according to claim 1, characterized in that: After the second model training step, the following steps are also included: A second optimization step, using the uncertainty map as the loss weight of the second model, obtaining a sixth loss function, and optimizing the second model training step; as well as In the third optimization step, the low-resolution image is input into the trained first model to obtain a second feature map, and the low-resolution image is input into the trained first model and then subjected to a second downsampling, and the second downsampled image is input into the second model to obtain a third feature map, and the third feature map output by the second model is subjected to supervised training using the second feature map as supervision, and the second model training step is optimized to obtain an optimized second model.
9. The low-resolution image segmentation method according to claim 8, characterized in that: The second optimization step comprises the following steps: A normalization step, normalizing the value range of the uncertainty graph to [0, 1]; and A regularization constraint step is performed on the fourth loss function based on the uncertainty graph. The calculation formula of the regularization constraint step is: in, is the fourth loss function after regularization constraints, that is, the sixth loss function.
10. The low-resolution image segmentation method according to claim 8, characterized in that: The third optimization step comprises the following steps: The second feature map acquisition step is to input the low-resolution image into the first model to obtain the second feature map output by the first model. The calculation formula of the second feature map is: F sr =SelfSR(I LR ,Y LR ) m2 Among them, F sr represents the second feature map, and m2 represents an intermediate output feature of the first model; The third feature map acquisition step is to input the low-resolution image into the second model to obtain the third feature map output by the second model. The calculation formula of the third feature map is: F seg =Seg(I LR ,Y LR ) M2 Among them, F seg represents the second feature map, M2 represents an intermediate output feature of the second model; In a fourth feature map acquisition step, based on a bilinear interpolation method, the shape of the second feature map is aligned with the shape of the third feature map to obtain a C'×D'×H'×W' feature map, where C represents channel, D represents depth, H represents height, and W represents width; The similarity calculation step obtains a fully connected affinity graph, the granularity of the affinity graph is β, and the affinity graph includes fully connected nodes, calculate the similarity a between the i-th node and the j-th node ij , the calculation formula is: Among them, T represents transpose, p represents the model, and F p,j represents the value of the jth node in a model, F p,i represents the value of the jth node in a model; and The seventh loss function calculation step is to calculate the square difference of the similarity to represent the seventh loss function. The calculation formula of the seventh loss function calculation step is: in, represents the similarity between the i-th and j-th nodes in the second feature graph, represents the similarity between the i-th and j-th nodes in the third feature graph, L corr It is the seventh loss function.