High-resolution image acquisition method

Abstract

The invention provides a high-resolution image acquisition method, which can convert a low-resolution image and a label thereof into a high-resolution image and a label thereof by establishing a super-resolution model, the generated high-resolution image and the label thereof can be used for a subsequent segmentation task, and the high-resolution image and the label thereof can be acquired by pre-training a first model. The convergence effect and reconstruction performance of the super-resolution model on a specified task are improved, then a self-supervised learning strategy is designed, so that the super-resolution model only needs data in a 2D scanning mode to complete training, and simultaneous modeling of a gray level image and a binary mask image is proposed, so that the super-resolution model is obtained. Therefore, the super-resolution model can process the grayscale image and the binary mask image at the same time. The method has the advantages that the problem that an existing super-resolution model can only process a grayscale image and cannot directly process a binary segmentation mask image is solved, and the time cost and the labor cost for obtaining a large amount of high-resolution magnetic resonance image data are reduced.

Description

Technical Field

[0001] This application relates to the field of machine learning, and particularly to a method for obtaining high-resolution images. Background Art

[0002] High-resolution 3D magnetic resonance imaging (MRI) can provide detailed anatomical structure information, enabling precise segmentation of regions of interest in various medical image analysis tasks, which is beneficial for clinical tasks such as high-precision surgical planning and prosthesis preparation.

[0003] Existing deep learning-based high-resolution segmentation models are the mainstream methods for such tasks, with good generalization and accuracy. Such methods usually require high-resolution data and their segmentation annotations to effectively train the model. However, such high-resolution data is often difficult to collect in actual clinical scenarios. First, due to the high requirements of the acquisition equipment, the limitations of the scanning time and signal-to-noise ratio, 2D scanned magnetic resonance images are usually used clinically. Although these images have a relatively high in-plane resolution, the inter-plane resolution is low. Second, the segmentation annotation of magnetic resonance images usually requires layer-by-layer delineation of the regions of interest, which is a time-consuming and laborious process that requires a lot of human resources. For such data, existing methods usually can only obtain low-resolution and low-accuracy segmentation results, often unable to meet the requirements of subsequent tasks. Summary of the Invention

[0004] This application provides a method for obtaining high-resolution images to solve the problems that it is difficult to obtain a large number of high-resolution magnetic resonance images and that the annotation of high-resolution images is time-consuming and laborious.

[0005] This application provides a method for obtaining high-resolution images, including a first data acquisition step, a first model establishment step, a pre-training step, a second data acquisition step, a first model training step, and an input step.

[0006] The first data acquisition step is used to acquire the dataset publicly available on Vimeo-90K video frame interpolation; the first model establishment step is used to establish a first model that can convert the low-resolution image into a high-resolution image, and the high-resolution image includes a second annotation. The first model is constructed based on a 3D U-Net-based model through a 2D U-Net. Specifically, all 2D convolutions in the encoder and decoder of the first model are replaced with 3D convolutions, and the encoder features are directly fused with the corresponding decoder features by channel through skip connections; the pre-training step is to input the dataset into the first model and pre-train the first model to initialize the first model; the second data acquisition step is used to acquire at least one set of low-resolution images, and the low-resolution images include a first annotation; the first model training step is to interpolate the low-resolution image and its first annotation, perform convolution and the first downsampling operation on the interpolated image to obtain an image pair, and use the image pair as supervision to train the high-resolution image and the second annotation output by the first model in a self-supervised manner to obtain the trained first model; the input step is to input the low-resolution image and the first annotation into the trained first model and output the high-resolution image and the second annotation.

[0007] Further, in the first model, there is at least one 3D filter, and the 3D filter is a 5D filter with a size of ci×co×t×h×w, where t is the time size, (h, w) is the spatial dimension of the convolution kernel, ci represents the number of input channels, and co represents the number of output channels.

[0008] Further, in the first model, the backbone network of the first model is a ResNet-3D network structure with 18 layers. Specifically, the last layer of the backbone network of the first model is removed and all stride operations in the time dimension are removed, so that the backbone network of the first model contains 5 convolutional blocks, namely the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, and the fifth convolutional block. Each convolutional block consists of two 3D convolutional layers and a skip connection, and the spatial stride used in the first convolutional block, the third convolutional block, and the fourth convolutional block is 2.

[0009] Further, in the first model, there are five 3D transposed convolutional layers with a stride of 2, namely the first 3D transposed convolutional layer, the second 3D transposed convolutional layer, the third 3D transposed convolutional layer, the fourth 3D transposed convolutional layer, and the fifth 3D transposed convolutional layer. Specifically, a 3D convolutional layer is added after the fifth 3D transposed convolutional layer.

[0010] Further, the input channels of the first model are set to 2, and the output channels of the first model are set to 2.

[0011] Further, the pre-training step specifically includes an input step, a connection fusion step, an output step, and a segmentation step.

[0012] The 3D feature map acquisition step is to input the data set into the first model, and each decoder outputs a 3D feature map; the connection fusion step is to add a temporal fusion layer after each decoder, and the temporal fusion layer is composed of 2D convolutions. Through the temporal fusion layer, the features in the temporal dimension of the 3D feature map are concatenated and fused along the channels into a 2D spatial feature map; the output step is to input the 2D spatial feature map into a 7×7 2D convolution kernel to output a first output with a predicted size of H×W×3(k - 1); the segmentation step is to segment the first output along the channel dimension to obtain (k - 1) output frames.

[0013] Further, the first model training step specifically includes an interpolation step, a convolution step, a downsampling step, and an image output step.

[0014] The interpolation step is to perform interpolation processing on the low-resolution image and its first annotation based on the SMORE technique; the convolution step is to use a 1D convolutional Gaussian filter h(x; r) as a slice section to perform convolution on the interpolated low-resolution image, where x represents the object on which the filter acts, and r represents the full width at half maximum of the Gaussian filter; the downsampling step is to downsample the low-resolution image and its first annotation by a factor of r to obtain a simulated low-resolution image and a simulated first annotation, where the low-resolution image and the simulated low-resolution image form the image pair; the image output step is to input the simulated low-resolution image and the simulated first annotation into the first model to output a first image and a third annotation, and its calculation formula is

[0015]

[0016] where represents the first image, represents the third annotation, SelfSR represents the first model, I LLR represents the simulated low-resolution image, Y LLR represents the simulated first annotation, 1 represents an output end of the first model, and 2 represents an output end of the first model.

[0017] Further, in the first model training step, when the input low-resolution image is a grayscale image, there is a first loss function for training the grayscale image; when the input low-resolution image is a binary mask image, there is a second loss function for training the binary mask image.

[0018] Further, the calculation formula of the first loss function is

[0019]

[0020] where, Lsr_ I is the first loss function, represents the high-resolution image, I LR represents the low-resolution image, and 2 represents an output end of the first model.

[0021] Further, the calculation formula of the second loss function is

[0022]

[0023] where, L D ice represents the second loss function, Y LR represents the first annotation, represents the second annotation.

[0024] This application provides a method for obtaining a high-resolution image. By establishing a super-resolution model, it can convert a low-resolution image and its annotation into a high-resolution image and its annotation. The generated high-resolution image and its annotation can be used for subsequent segmentation tasks. By pre-training the first model and using a large-scale video dataset of natural scenes as pre-training data, it helps to improve the convergence effect and reconstruction performance of the super-resolution model on the specified task. Then, by designing a self-supervised learning strategy, the super-resolution model only needs data in the 2D scanning mode, that is, the low-resolution image, to complete the training. And it is proposed to simultaneously model the grayscale image and the binary mask image, so that the super-resolution model can process both of them at the same time. The present invention does not require a large amount of high-resolution magnetic resonance image data to train the first model, nor does it need to additionally annotate the obtained high-resolution image, which can greatly reduce the cost of data collection in the actual clinical scenario, solve the problem that the existing super-resolution model can only process grayscale images and cannot directly process binary segmentation mask images, restricting its application in tasks such as 3D printing and prosthesis preparation. At the same time, it reduces the time cost and labor cost of obtaining a large amount of high-resolution magnetic resonance image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those skilled in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0026] Figure 1 is a flowchart of the high-resolution image acquisition method according to the embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of the pre-training step according to the embodiment of the present invention;

[0028] Figure 3 is a flowchart of the pre-training step according to the embodiment of the present invention;

[0029] Figure 4 is a flowchart of the first model training step according to the embodiment of the present invention; Detailed implementation manners

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0031] As Figure 1 - Figure 2 shown, the present application provides a high-resolution image acquisition method, including step S1) the first data acquisition step, step S2) the first model establishment step, step S3) the pre-training step, step S4) the second data acquisition step, step S5) the first model training step, and step S6) the input step.

[0032] Step S1) The first data acquisition step, acquiring the dataset publicly available on Vimeo-90K video frame interpolation.

[0033] Step S2) The first model establishment step, establishing a first model, which can convert the low-resolution image into a high-resolution image. The high-resolution image includes a second annotation. The first model is based on a 3D U-Net-based model and is constructed by extending the 2D U-Net widely used in pixel generation tasks. Among them, all 2D convolutions in the encoder and decoder of the first model are replaced with 3D convolutions to facilitate accurate modeling of the temporal dynamics between input frames, thereby significantly improving the interpolation quality. The encoder features are directly fused with the corresponding decoder features by channel through skip connections, so as to fuse low-level and high-level information to achieve accurate and clear interpolation.

[0034] Furthermore, in the first model, there is at least one 3D filter. The 3D filter is a 5D filter with a size of ci×co×t×h×w, where t is the temporal size, (h, w) is the spatial size of the convolutional kernel, ci represents the number of input channels, and co represents the number of output channels. The additional temporal dimension is very useful for modeling temporal abstractions, such as motion trajectories, actions, or correspondences between video frames. Therefore, in this embodiment, setting the temporal size of each 3D filter to t can better capture the correspondences between each video frame in the dataset publicly available for Vimeo-90K video frame interpolation.

[0035] Furthermore, in the first model, the backbone network of the first model is a ResNet-3D network structure with 18 layers. Among them, the last layer of the backbone network of the first model is removed, and all stride operations in the temporal dimension are removed. Removing all stride operations in the temporal dimension is to avoid losing details crucial for generating clearer images due to downsampling operations such as strides and pooling. The backbone network of the first model contains 5 convolutional blocks, namely the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, and the fifth convolutional block. Each convolutional block consists of two 3D convolutional layers and a skip connection. The spatial stride used in the first convolutional block, the third convolutional block, and the fourth convolutional block is 2 to keep the computational cost within a controllable range.

[0036] Furthermore, in the first model, there are five 3D transposed convolutional layers. The stride of the 3D transposed convolutional layer is 2, and its purpose is to facilitate the upsampling operation. The 3D transposed convolutional layers are respectively the first 3D transposed convolutional layer, the second 3D transposed convolutional layer, the third 3D transposed convolutional layer, the fourth 3D transposed convolutional layer, and the fifth 3D transposed convolutional layer. Among them, a 3D convolutional layer is added after the fifth 3D transposed convolutional layer, and its purpose is to facilitate the handling of common checkerboard effects.

[0037] Furthermore, the input channels of the first model are set to 2, and the output channels of the first model are set to 2 to simultaneously model grayscale images and binary mask images, enabling the first model to perform super-resolution reconstruction on the grayscale images and binary mask images of the region of interest simultaneously.

[0038] In this embodiment, the first model has the slice interpolation feature, that is, the intermediate slice is reconstructed according to the upper and lower layer slices to improve the interlayer resolution of the image. This feature highly coincides with the video frame interpolation task. Therefore, a large-scale video dataset of natural scenes can be used as pre-training data to help improve the convergence effect and reconstruction performance of the super-resolution model on the specified task. Therefore, the dataset publicly available on Vimeo-90K video frame interpolation is obtained to pre-train the first model, so that the high-resolution images and their annotations obtained through the first model are more accurate.

[0039] Step S3) Pre-training step: Input the dataset into the first model to pre-train the first model, initialize the first model, and improve the convergence effect and reconstruction performance of the first model on the super-resolution task.

[0040] As Figure 3 shown, step S3) The pre-training step specifically includes step S31) 3D feature map acquisition step, step S32) connection and fusion step, step S33) output step, and step S34) segmentation step.

[0041] Step S31) 3D feature map acquisition step: Input the dataset into the first model, and each decoder outputs a 3D feature map. In this embodiment, the decoder essentially constructs the output frame from the deep latent representation captured by the encoder through step-by-step, multi-scale feature upsampling and feature fusion.

[0042] Step S32) Connection and fusion step: Add a temporal fusion layer after each decoder. The temporal fusion layer consists of 2D convolutions. Through the temporal fusion layer, the features in the temporal dimension of the 3D feature map are concatenated and fused by channels into a 2D spatial feature map to help aggregate and integrate the information in multiple output frames for prediction;

[0043] Step S33) Output step: Input the 2D spatial feature map into a 7×7 2D convolution kernel to output a first output with a predicted size of H×W×3(k - 1).

[0044] Step S34) Segmentation step: Segment the first output along the channel dimension to obtain (k - 1) output frames, realizing the initialization of the first model.

[0045] In this embodiment, the purpose of pre-training is that the first model does not consider the slice gap when performing the super-resolution task. Since many segmentation datasets lack this information, in this case, pre-training can be performed for the slice interpolation and deblurring tasks of 3D data to further improve the performance of the super-resolution network.

[0046] Step S4) Second data acquisition step, acquiring at least one set of low-resolution images, where the low-resolution images include first annotations.

[0047] In this embodiment, the low-resolution images are 2D scanned magnetic resonance images, and the first annotations are low-resolution annotations corresponding to the 2D scanned magnetic resonance images.

[0048] In this embodiment, for the low-resolution image I LR and its corresponding first annotation Y LR , both have a spatial resolution of a×a×c. Assuming that the spatial resolutions in the frequency encoding and phase encoding directions are the same, the super-resolution model uses a scaling factor r to generate a high-resolution image with a spatial resolution of a×a×(c / r). In the prior art, if a large number of high-resolution magnetic resonance images need to be acquired, high-precision acquisition equipment is required. The acquired high-resolution images occupy a large storage space, and the hardware costs of cameras, hard disks, etc. are relatively high; in addition, high-resolution magnetic resonance images require a long scanning time during shooting and a long transmission time during transmission, and their signal-to-noise ratio is relatively high. It is time-consuming and laborious to make annotations on high-resolution magnetic resonance images.

[0049] In the proposed solution, only low-resolution magnetic resonance images need to be acquired and annotated, and after the computer software processes the images, high-resolution images with annotations can be obtained, without directly acquiring and annotating high-resolution magnetic resonance images, solving the problems that it is difficult to obtain a large number of high-resolution magnetic resonance images and that high-resolution annotation is time-consuming and laborious.

[0050] Step S5) First model training step, interpolating the low-resolution image and its first annotation, performing convolution and a first downsampling operation on the interpolated image to obtain an image pair, and using the image pair as supervision to train the high-resolution image and the second annotation output by the first model in a self-supervised manner to obtain the trained first model.

[0051] As Figure 4 shown, step S5) First model training step specifically includes step S51) Interpolation step, step S52) Convolution step, step S53) Downsampling step, and step S54) Image output step.

[0052] Step S51) Interpolation step, based on the SMORE technique, performing interpolation processing on the low-resolution image and its first annotation to achieve isotropic voxel spacing of the low-resolution image and the first annotation, thereby ensuring spatial consistency in the first model training stage and the first model inference stage.

[0053] Step S52) Convolution step: Convolve the interpolated low-resolution image with a 1D convolutional Gaussian filter h(x; r) as a slice section, where x represents the object on which the filter acts, and r represents the full width at half maximum of the Gaussian filter, and r also represents the scaling factor size adopted by the super-resolution model.

[0054] Step S53) Downsampling step: Downsample the low-resolution image and its first annotation by a factor of r to obtain a simulated low-resolution image and a simulated first annotation. Among them, the low-resolution image and the simulated low-resolution image form the image pair. In this embodiment, the image pair composed of the low-resolution image and the simulated low-resolution image is used as supervision to train the first model in a self-supervised manner.

[0055] Step S54) Image output step: Input the simulated low-resolution image and the simulated first annotation into the first model, and output a first image and a third annotation. Its calculation formula is

[0056]

[0057] Among them, represents the first image, represents the third annotation, SelfSR represents the first model, I LLR represents the simulated low-resolution image, Y LLR represents the simulated first annotation, 1 represents an output end of the first model, and 2 represents an output end of the first model.

[0058] In this embodiment, the simulated low-resolution image I LLR compared with the first image the simulated low-resolution image I LLR is a low-resolution image, and the first image is a high-resolution image; the simulated first annotation Y LLR compared with the third annotation the simulated first annotation Y LLR is a low-resolution annotation, and the third annotation is a high-resolution annotation. Therefore, during the training of the first model, a real 3D scanned magnetic resonance image is not obtained, and the training of the process of the super-resolution model obtaining a high-resolution image from a low-resolution image can be achieved without a real 3D scanned magnetic resonance image.

[0059] Furthermore, in the first model training step, when the input low-resolution image is a grayscale image, there is a first loss function to train the grayscale image; when the input low-resolution image is a binary mask image, there is a second loss function to train the binary mask image.

[0060] Further, the calculation formula of the first loss function is

[0061]

[0062] where Lsr_ I is the first loss function, represents the high-resolution image, I LR represents the low-resolution image, and 2 represents an output end of the first model.

[0063] Further, the calculation formula of the second loss function is

[0064]

[0065] where L D ice represents the second loss function, Y LR represents the first annotation, represents the second annotation.

[0066] Step S6) Input step: Input the low-resolution image and the first annotation into the trained first model, and output the high-resolution image and the second annotation.

[0067] This application provides a method for obtaining a high-resolution image. By establishing a super-resolution model, the low-resolution image and its annotation can be converted into a high-resolution image and its annotation. The generated high-resolution image and its annotation can be used for subsequent segmentation tasks. By pre-training the first model and using a large-scale video dataset of natural scenes as pre-training data, it helps to improve the convergence effect and reconstruction performance of the super-resolution model on the specified task. Then, by designing a self-supervised learning strategy, the super-resolution model only needs data in the 2D scanning mode, that is, the low-resolution image, to complete the training. And it is proposed to model the grayscale image and the binary mask image simultaneously, so that the super-resolution model can process both of them at the same time. The present invention does not require a large amount of high-resolution magnetic resonance image data to train the first model, nor does it need to additionally annotate the obtained high-resolution image, which can greatly reduce the cost of data collection in the actual clinical scenario, solve the problem that the existing super-resolution model can only process grayscale images and cannot directly process binary segmentation mask images, which limits its application in tasks such as 3D printing and prosthesis preparation. At the same time, it reduces the time cost and labor cost of obtaining a large amount of high-resolution magnetic resonance image data.

[0068] The above has introduced in detail a high-resolution image acquisition method provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation on this application.

Claims

1. A high-resolution image acquisition method, characterized in that: The steps include: The first data acquisition step is to acquire a data set publicly available on Vimeo-90K video interpolation frames; A first model building step is to build a first model, wherein the first model can convert the low-resolution image into a high-resolution image, wherein the high-resolution image includes a second annotation, and the first model is based on a 3D U-Net-based model and is built through a 2D U-Net, wherein all 2D convolutions in the encoder and the decoder in the first model are replaced with 3D convolutions, and encoder features are directly fused with corresponding decoder features by channel through skip connections; A pre-training step, inputting the data set into a first model, pre-training the first model, and initializing the first model; A second data acquisition step, acquiring at least one set of low-resolution images, wherein the low-resolution images include the first annotation; A first model training step of interpolating the low-resolution image and the first annotation thereof, performing convolution and a first downsampling operation on the interpolated image to obtain an image pair, and training the high-resolution image and the second annotation output by the first model in a self-supervised manner using the image pair as supervision to obtain a trained first model; and The input step is to input the low-resolution image and the first annotation into the trained first model, and output the high-resolution image and the second annotation.

2. The high-resolution image acquisition method according to claim 1, characterized in that: In the first model, including At least one 3D filter, the 3D filter is a 5-dimensional filter, and the size of the 3D filter is ci×co×t×h×w, wherein t is the time size, (h, w) is the spatial size of the convolution kernel, ci represents the number of input channels, and co represents the number of output channels.

3. The high-resolution image acquisition method according to claim 1, characterized in that: In the first model, The backbone network of the first model is a ResNet-3D network structure with 18 layers, wherein the last layer of the backbone network of the first model and the stride operations of all time dimensions are removed, so that the backbone network of the first model includes 5 convolution blocks, which are respectively a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, and a fifth convolution block, each convolution block consists of two 3D convolution layers and a skip connection, and the spatial stride used in the first convolution block, the third convolution block and the fourth convolution block is 2.

4. The high-resolution image acquisition method according to claim 3, characterized in that: In the first model, including Five 3D transposed convolutional layers, the stride of the 3D transposed convolutional layer is 2, and the 3D transposed convolutional layers are respectively a first 3D transposed convolutional layer, a second 3D transposed convolutional layer, a third 3D transposed convolutional layer, a fourth 3D transposed convolutional layer, and a fifth 3D transposed convolutional layer, wherein a 3D convolutional layer is added after the fifth 3D transposed convolutional layer.

5. The high-resolution image acquisition method according to claim 1, characterized in that: The input channel of the first model is set to 2, and the output channel of the first model is set to 2.

6. The high-resolution image acquisition method according to claim 1, characterized in that: The pre-training step specifically includes the following steps: A 3D feature map acquisition step, inputting the data set into the first model, and each decoder outputs a 3D feature map; A connection and fusion step, in which a time fusion layer is added after each decoder, wherein the time fusion layer is composed of 2D convolutions, and the features of the time dimension in the 3D feature map are connected by channels and fused into a 2D spatial feature map through the time fusion layer; In the output step, the 2D spatial feature map inputs a 7×7 2D convolution kernel to output a first output with a predicted size of H×W×3(k-1); as well as The segmentation step segments the first output along the channel dimension to obtain (k-1) output frames.

7. The high-resolution image acquisition method according to claim 1, characterized in that: The first model training step specifically includes the following steps: An interpolation step, based on the SMORE technology, interpolating the low-resolution image and the first annotation thereof; A convolution step, using a 1D convolution Gaussian filter h(x; r) as a slice section to convolve the interpolated low-resolution image, wherein x represents the object of the filter, and r represents the full width at half height of the Gaussian filter; a downsampling step of downsampling the low-resolution image and the first annotation by a factor of r to obtain a simulated low-resolution image and a simulated first annotation, wherein the low-resolution image and the simulated low-resolution image constitute the image pair; and The image output step inputs the simulated low-resolution image and the simulated first annotation into the first model, and outputs the first image and the third annotation, and the calculation formula is: in, represents the first image, represents the third annotation, SelfSR represents the first model, and I LLR Represents a simulated low-resolution image, Y LLR Indicates the simulation of the first annotation, 1 represents an output terminal of the first model, and 2 represents an output terminal of the first model.

8. The high-resolution image acquisition method according to claim 1, characterized in that: In the first model training step, When the input low-resolution image is a grayscale image, there is a first loss function to train the grayscale image; When the input low-resolution image is a binary mask image, there is a second loss function for training the binary mask image.

9. The high-resolution image acquisition method according to claim 8, characterized in that: The first loss function calculation formula is: Among them, Lsr_ I is the first loss function, represents a high-resolution image, I LR represents a low-resolution image, and 2 represents an output terminal of the first model.

10. The high-resolution image acquisition method according to claim 8, characterized in that: The second loss function calculation formula is: Among them, L D ice represents the second loss function, Y LR Indicates the first annotation, Indicates the second annotation.

Images