Method and device for segmenting fine objects based on super-resolution technology

Abstract

The present invention discloses a method and device for segmenting fine objects based on super-resolution technology, which can be used for segmenting three-dimensional micro-vessels. The method comprises the following steps: 1) constructing a super-resolution network, proposing a structure-preserving super-resolution network, and performing super-resolution operations on micro-vessels on the basis of maintaining the vascular structure information to obtain more detailed information; 2) constructing a three-dimensional vascular segmentation network, which adopts a U-shaped network structure, extracting features layer by layer in the convolution layer while connecting the two symmetrical layers at the head and tail to achieve full feature extraction; 3) using super-resolution to process data, and using the super-resolution data to train the segmentation network, so as to achieve better segmentation effect. The present invention enhances the vascular characteristic information through super-resolution technology, thereby significantly improving the original network and achieving efficient and accurate segmentation effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation in computer vision, and in particular relates to a method and device for segmenting fine objects based on super-resolution technology. Background Art

[0002] Image segmentation technology is an important research direction in the field of computer vision and an important part of image semantic understanding. Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. It has broad application prospects in the medical field.

[0003] Since convolutional neural networks have excellent feature extraction and expression capabilities, and do not require manual extraction of image features or excessive image preprocessing, they have made great achievements in the field of medical image segmentation in recent years. For example, the segmentation accuracy of larger objects such as the liver and lungs has reached 90%. However, in the microscopic field, the segmentation of tiny objects (such as obtaining accurate brain microscopic 3D blood vessels from brain image data) is still a huge challenge.

[0004] There are many difficulties in brain microvascular segmentation: 1. The proportion of vascular tissue in the entire brain is relatively low. In addition to cerebral blood vessels, brain MRA images also contain large areas of non-vascular tissue such as white matter, gray matter, and tissue fluid, which are easily affected by other impurities; 2. The blood vessels are tiny. The diameter of the brain's micro-blood vessels is only 1-2 voxels (about 50-100 microns), and it is impossible to accurately extract the local features of the blood vessels; 3. The geometric shapes of blood vessels are complex and changeable. The blood vessels intersect and twist into clusters, and the blood vessel segmentation is easily affected by geometric features.

[0005] Single Image Super Resolution (SISR) aims to restore a high-resolution image (HR) from a low-resolution image (LR). However, super-resolution is an ill-posed problem, and each LR input may have multiple HR solutions. With the development of deep learning, many super-resolution algorithms have been proposed, most of which optimize the pixel distance between the super-resolution image and the high-resolution image by measuring the mean square error. However, this optimization goal causes the image generated by the deep model to be the statistical average of multiple super-resolution solutions. Therefore, the images generated by such methods will have some structural distortion problems.

[0006] Therefore, the existing technology has the problem of being unable to accurately extract the local features of blood vessels and the problem of structural distortion that is prone to occur in the super-resolution of a single image. Summary of the invention

[0007] In view of the above-mentioned deficiencies of the prior art, the present invention provides a method and device for segmenting fine objects based on super-resolution technology, which improves the accuracy of blood vessel segmentation of the segmentation network model. At the same time, the super-resolution network with added gradient branches focuses on structural features compared with the traditional super-resolution algorithm, which can effectively solve the problem of structural distortion in traditional super-resolution images and facilitate the subsequent segmentation work.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] According to a first aspect of the present specification, a method for segmenting a subtle object based on super-resolution technology is provided, comprising the following steps:

[0010] S1, acquiring three-dimensional brain MRA image data with blood vessel annotations, randomly dividing the three-dimensional brain MRA image data into a plurality of three-dimensional image block data, and constructing a three-dimensional image segmentation data set;

[0011] S2, converting each three-dimensional image block data in the three-dimensional image segmentation data set into two-dimensional image data to obtain high-resolution image data;

[0012] S3, down-sampling the high-resolution image data to generate low-resolution image data, and constructing a super-resolution dataset from the low-resolution image data and the high-resolution image data;

[0013] S4, using the super-resolution dataset, trains a super-resolution network model based on image gradient;

[0014] S5, using the trained super-resolution network model to perform super-resolution processing on the three-dimensional image segmentation dataset to obtain super-resolution image data with a resolution twice as large as the original one, and construct a super-resolution image segmentation dataset;

[0015] S6, using the super-resolution image segmentation dataset to train a three-dimensional segmentation network to obtain an image segmentation network model;

[0016] S7, use the trained image segmentation network model to perform blood vessel segmentation.

[0017] Further, in step S4, the super-resolution network model includes a super-resolution branch network model and a gradient branch network model;

[0018] First, the low-resolution image data in the super-resolution data set is input into the super-resolution branch network model to extract features and output a feature map; and the low-resolution image data in the super-resolution data set is subjected to gradient transformation to obtain a gradient map, and the gradient map is input into the gradient branch network model to extract structural features and output a feature map, wherein part of the feature map output by the super-resolution branch network model is input into the gradient branch network model for fusion to assist in the generation of a super-resolution gradient map;

[0019] Then, the feature map output by the gradient branch network model is input into the super-resolution branch network model, the feature map output by the super-resolution branch network model and the feature map output by the gradient branch network model are fused, and a fused feature map is output;

[0020] Finally, the fused feature map is upsampled in the super-resolution branch network model to obtain super-resolution image data.

[0021] Furthermore, the super-resolution branch network model includes a plurality of super-resolution modules, a fusion module and an upsampling module;

[0022] The low-resolution image data in the super-resolution dataset is input into the super-resolution branch network model. The super-resolution branch network model first uses a 3×3 convolution kernel to extract shallow features of the low-resolution image data in the super-resolution dataset and outputs a feature map. Then, multiple super-resolution modules are used to extract deep features of the low-resolution image data in the super-resolution dataset and output a feature map. Then, a last fusion module is used to fuse the feature map output by the super-resolution branch network model and the feature map output by the gradient branch network model. Finally, an upsampling module is used to expand the size of the fused feature map to twice its original size to obtain super-resolution image data.

[0023] Furthermore, the gradient branch network model includes a plurality of gradient modules;

[0024] The low-resolution image data in the super-resolution data set is subjected to gradient transformation to obtain a gradient map, and the gradient map is input into the gradient branch network model to extract structural features using multiple gradient modules and output a feature map.

[0025] Furthermore, the super-resolution module includes convolutional layers and activation layers stacked in sequence;

[0026] The fusion module includes a connection layer and a convolution layer stacked in sequence, the convolution kernel size is 3×3, and the number of convolution kernels is 32;

[0027] The gradient module includes convolutional layers and activation layers stacked in sequence, which are connected in series through a residual structure.

[0028] Furthermore, step S5 is specifically as follows:

[0029] S51, performing data conversion on the 3D image segmentation data set in step S1, performing data conversion on each 3D image block data, and converting it into 2D image data;

[0030] S52, using the trained super-resolution network model to perform super-resolution processing on the two-dimensional image data in step S51, to obtain super-resolution image data with a resolution doubled;

[0031] S53, performing data conversion on the super-resolution image data in step S52 to generate three-dimensional super-resolution image block data;

[0032] S54, collecting and generating three-dimensional super-resolution image block data, and summarizing and organizing them into a super-resolution image segmentation data set.

[0033] Furthermore, the function of the super-resolution image data is:

[0034] I SR =G(I LR θ G )

[0035] Where G represents the generator, θ G Represents the parameters of the generator, I LR Represents a low-resolution image, I SR Represents super-resolution image data.

[0036] Furthermore, the gradient map of the image is obtained by calculating the difference between adjacent pixels, and the function of the gradient map is:

[0037] I x (Z) = I (x + 1, y) - I (x, y)

[0038] I y (Z) = I (x, y + 1) - I (x, y)

[0039]

[0040] Where Z = (x, y) represents the pixel coordinates in the image, I x (Z) represents the difference of pixel point Z in the horizontal direction x, I y (Z) represents the difference of pixel Z in the horizontal direction y, represents the difference coordinates of the pixel point, M(·) represents the operation of extracting the gradient map, and ||·||2 represents the L2 norm. Since the amplitude information is sufficient to represent the sharpness of the local area of ​​the image, only the amplitude information of the gradient is used here, and the direction information is not considered.

[0041] Furthermore, the objective function of the super-resolution network model is:

[0042]

[0043] Among them, α1, α2, α3, β1, β2 represent the trade-off parameters of different losses; represents the loss of the super-resolution branch network model, represents the loss of the gradient branch network model, represents the image perceptual loss, Represents the pixel loss of the image on the super-resolution branch network model, represents the adversarial loss of the image on the super-resolution branch network model, represents the pixel loss of the gradient map on the super-resolution branch network model, represents the adversarial loss of the gradient map on the super-resolution branch network model, Represents the pixel loss of the gradient map on the gradient branch network model;

[0044]

[0045] in, Represents the pixel-level loss of the image, I LR Represents a low-resolution image, I HR represents a high-resolution image, G(·) represents the super-resolution network generator, ||·|| represents the use of L1 paradigm, represents the operation of averaging it, represents the pixel-level loss of the gradient image, and M(·) represents the gradient image after gradient change;

[0046]

[0047] in, represents the perceptual loss, implemented by the pre-trained VGG network, φ i Represents the output features of the i-th layer of the VGG model. Perceptual loss is used to measure the difference between HR image features and SR image features;

[0048]

[0049] in represents the adversarial loss, and D(·) represents the discriminator in the GAN network.

[0050] According to a second aspect of the present specification, a device for segmenting fine objects based on super-resolution technology is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the method for segmenting fine objects based on super-resolution technology as described in the first aspect.

[0051] The beneficial effects of the present invention are as follows: first, super-resolution technology is applied to subtle objects, the image is enlarged, and the image resolution is improved. Image resolution represents the amount of information stored in the image. The higher the resolution, the more information. We apply super-resolution technology to subtle objects and improve the image resolution, which increases the feature information of subtle objects, thereby improving the subsequent segmentation accuracy. In the example of this article, the proportion of pixels occupied by cerebral blood vessels in the entire brain image is relatively low, and the local features of blood vessels cannot be accurately extracted. Super-resolution technology is applied to blood vessel segmentation, and the brain image is enlarged using super-resolution technology, thereby further increasing the blood vessel detail information, improving the accuracy of blood vessel segmentation, and solving the problem that the proportion of blood vessel pixels is low and the local features of blood vessels cannot be accurately extracted; second, it is proposed to apply image gradients to super-resolution, and through image gradients, the characteristics of blood vessel structure are enhanced, the generation of super-resolution images is assisted, and the sharpness and the areas that should be paid attention to in the structure are highlighted, and the problem of structural distortion that is prone to occur in super-resolution is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A flowchart of a method for segmenting a fine object based on super-resolution technology provided in one embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of data preprocessing in one embodiment of the present invention;

[0055] Figure 3 A diagram showing a super-resolution network structure in one embodiment of the present invention;

[0056] Figure 4 A detailed module diagram of a super-resolution network in one embodiment of the present invention;

[0057] Figure 5 A three-dimensional segmentation network structure diagram in one embodiment of the present invention;

[0058] Figure 6 A brain image gradient comparison diagram in one embodiment of the present invention;

[0059] Figure 7 A comparison diagram of cerebral vascular effects in one embodiment of the present invention;

[0060] Figure 8 4 is a structural diagram of a fine object segmentation device based on super-resolution technology provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0062] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0063] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0064] like Figure 1 As shown, an embodiment of the present invention provides a method for segmenting a subtle object based on super-resolution technology, the method comprising the following steps:

[0065] S1, acquiring three-dimensional brain MRA image data with blood vessel annotations, randomly dividing the three-dimensional brain MRA image data into a plurality of three-dimensional image block data, and constructing a three-dimensional image segmentation data set;

[0066] In one embodiment, the TubeTK dataset is selected as the experimental database. The data downloaded from the official website of the TubeTK database is presented in the MetaImage format and cannot be used directly. First, ITKTubeTK is used to process it, and the vascular label data (.tre) in the TubeTK database is converted into a binary three-dimensional image; in addition to the brain vascular data required for the experiment, the MRA image also contains other tissue structures, such as the skull, which will bring unnecessary noise to the experiment. Therefore, the MRA image is preprocessed using FSL software to remove the brain skull and retain the brain vascular data; then the processed data is randomly divided into n 128x128x16 three-dimensional image block data. This operation can increase the amount of experimental data while avoiding large-size data processing during training, which slows down the efficiency of the entire training process. At this point, the three-dimensional image segmentation dataset is constructed.

[0067] S2, converting each three-dimensional image block data in the three-dimensional image segmentation data set into two-dimensional image data to obtain high-resolution image data;

[0068] like Figure 2 As shown, in one embodiment, a 128×128×16 three-dimensional image block data is divided into 16 128×128 two-dimensional image data, ie, high-resolution image data.

[0069] S3, down-sampling the high-resolution image data to generate low-resolution image data, and constructing a super-resolution dataset from the low-resolution image data and the high-resolution image data;

[0070] In one embodiment, the two-dimensional data (128×128) mentioned above is downsampled into the required low-resolution data (64×64) by a downsampling method, and the two correspond one to one. The super-resolution image dataset consists of two parts, namely, the high-resolution image HR (128×128) and the low-resolution image LR (64×64).

[0071] S4, using the super-resolution dataset, trains a super-resolution network model based on image gradient;

[0072] like Figure 3 As shown, in one embodiment, the super-resolution network model in step S4 includes a super-resolution branch network model and a gradient branch network model;

[0073] First, the low-resolution image data in the super-resolution dataset is input into the super-resolution branch network model for feature extraction and output of the feature map; and the low-resolution image data in the super-resolution dataset is subjected to gradient transformation to obtain a gradient map, and the gradient map is input into the gradient branch network model for structural feature extraction and output of the feature map, wherein part of the feature map output by the super-resolution branch network model is input into the gradient branch network model, specifically, the feature maps extracted by the 5th, 10th, 15th, and 20th super-resolution modules in the super-resolution branch network model are incorporated into the gradient branch to assist in reconstructing the super-resolution gradient map;

[0074] Then, the feature map output by the gradient branch network model is input into the super-resolution branch network model, the feature map output by the super-resolution branch network model and the feature map output by the gradient branch network model are fused, and a fused feature map is output;

[0075] Finally, the fused feature map is upsampled in the super-resolution branch network model to obtain super-resolution image data.

[0076] like Figure 4 As shown, in one embodiment, the super-resolution branch network model includes multiple super-resolution modules, a fusion module and an upsampling module;

[0077] The low-resolution image data in the super-resolution dataset is input into the super-resolution branch network model. The super-resolution branch network model adopts the basic architecture of SRResNet. It first uses a 3×3 convolution kernel to extract the shallow features of the low-resolution image data in the super-resolution dataset and outputs the feature map. Then, multiple super-resolution modules are used to extract the deep features of the low-resolution image data in the super-resolution dataset and output the feature map. The last fusion module is used to fuse the feature map output by the super-resolution branch network model and the feature map output by the gradient branch network model. Finally, an upsampling module is used to expand the size of the fused feature map to twice its original size to obtain super-resolution image data. The entire generation network is performed in the low-resolution feature space, making full use of the high-frequency features of the image.

[0078] In one embodiment, the gradient branch network model includes a plurality of gradient modules;

[0079] The low-resolution image data in the super-resolution data set is subjected to gradient transformation to obtain a gradient map, and the gradient map is input into the gradient branch network model to extract structural features using multiple gradient modules and output feature maps. In one embodiment, in the gradient branch network model, a gradient map of the low-resolution image is first obtained through a gradient operation, and then a 3×3 convolution layer is used to extract shallow features of the gradient map, and then multiple gradient modules are used to extract higher-level features.

[0080] In one embodiment, the super-resolution module includes convolutional layers and activation layers stacked in sequence. According to the DenseNet concept, the features of all layers are connected in series to achieve feature reuse; and then multiple Dense Blocks are connected together in the form of a residual structure to form a super-resolution module.

[0081] The fusion module includes a connection layer and a convolution layer stacked in sequence, with a convolution kernel size of 3×3 and 32 convolution kernels.

[0082] The gradient module consists of sequentially stacked convolutional layers and activation layers, which are connected in series through a residual structure for feature extraction.

[0083] The goal of the super-resolution branch network model is to take low-resolution image data as input and generate super-resolution image data. The corresponding high-resolution image data is used as the real image. By comparing the high-resolution image data with the generated super-resolution image data and calculating the difference between the two, the super-resolution network is optimized to generate more similar and realistic super-resolution image data. The function of the super-resolution image data is:

[0084] I SR =G(I LR θ G )

[0085] Where G represents the generator, θ G Represents the parameters of the generator, I LR Represents a low-resolution image, I SR Represents super-resolution image data.

[0086] The gradient branch network model aims to convert the gradient map of a low-resolution image into a gradient map of a high-resolution image, obtain the deep structural information of the image through multiple gradient modules, and then integrate it into the super-resolution branch network model to provide structural priors for super-resolution image generation. The gradient map of an image is obtained by calculating the difference between adjacent pixels. The function of the gradient map is:

[0087] I x (Z) = I (x + 1, y) - I (x, y)

[0088] I y (Z) = I (x, y + 1) - I (x, y)

[0089]

[0090] Where Z = (x, y) represents the pixel coordinates in the image, I x (Z) represents the difference of pixel point Z in the horizontal direction x, I y (Z) represents the difference of pixel Z in the horizontal direction y, represents the difference coordinates of the pixel point, M(·) represents the operation of extracting the gradient map, and ||·||2 represents the L2 norm. Since the amplitude information is sufficient to represent the sharpness of the local area of ​​the image, only the amplitude information of the gradient is used here, and the direction information is not considered.

[0091] From the network structure model, we can see that the gradient branch network model incorporates several intermediate layer features from the super-resolution branch network model. Taking the features of the super-resolution intermediate layer as a strong prior condition can greatly reduce the number of parameters of the gradient network, thereby improving the performance of the gradient network. At the end of the gradient network, the obtained gradient feature map is fused into the super-resolution branch network model to guide the super-resolution reconstruction. It is easier to extract image geometric features in the gradient space. Therefore, the geometric structure of the image can be effectively preserved through the gradient branch network model, thereby producing a more realistic super-resolution image. The image gradient-based super-resolution algorithm consists of two parts: the gradient branch network model and the super-resolution branch network model. The objective function of the super-resolution network model is:

[0092]

[0093] Among them, α1, α2, α3, β1, β2 represent the trade-off parameters of different losses; represents the loss of the super-resolution branch network model, represents the loss of the gradient branch network model, represents the image perceptual loss, Represents the pixel loss of the image on the super-resolution branch network model, represents the adversarial loss of the image on the super-resolution branch network model, represents the pixel loss of the gradient map on the super-resolution branch network model, represents the adversarial loss of the gradient map on the super-resolution branch network model, Represents the pixel loss of the gradient map on the gradient branch network model;

[0094] Pixelwis Loss: Based on pixel-level loss, L1 loss is used here to speed up model convergence. Its expression is as follows:

[0095]

[0096] in, Represents the pixel-level loss of the image, I LR Represents a low-resolution image, I HR represents a high-resolution image, G(·) represents the super-resolution network generator, ||·|| represents the use of L1 paradigm, represents the operation of averaging it, represents the pixel-level loss of the gradient image, and M(·) represents the gradient image after gradient change;

[0097]

[0098] in, represents the perceptual loss, implemented by the pre-trained VGG network, φ i Represents the output features of the i-th layer of the VGG model. Perceptual loss is used to measure the difference between HR image features and SR image features;

[0099]

[0100] in represents the adversarial loss, and D(·) represents the discriminator in the GAN network.

[0101] S5, first use the trained super-resolution network model to perform super-resolution processing on the 3D image segmentation dataset, obtain super-resolution image data with a resolution twice as large as the original, and construct a super-resolution image segmentation dataset;

[0102] In one embodiment, step S5 is specifically as follows:

[0103] S51, performing data conversion on the 3D image segmentation data set in step S1, performing data conversion on each 3D image block data, and converting it into 2D image data; in one embodiment, performing data conversion on the 3D image segmentation data set in S1, performing data conversion on each 3D image block data, and converting it into 2D image data, that is, dividing a 128×128×16 3D image block into 16 128×128 2D image slices;

[0104] S52, using the trained super-resolution network model to perform super-resolution processing on the two-dimensional image data in step S51, to obtain super-resolution image data with a resolution twice as large as the original one; in one embodiment, the image originally having a size of 128×128 is super-resolution processed to generate a super-resolution image having a size of 256×256;

[0105] S53, performing data conversion on the super-resolution image data in step S52 to generate three-dimensional super-resolution image block data; in one embodiment, 16 128×128 super-resolution images are converted into three-dimensional super-resolution image blocks with a size of 256×256×16;

[0106] S54, collecting and generating three-dimensional super-resolution image block data, and summarizing and organizing them into a super-resolution image segmentation data set.

[0107] S6, using the super-resolution image segmentation dataset to train a three-dimensional segmentation network to obtain an image segmentation network model;

[0108] like Figure 5 As shown in the figure, the 3D-Unet network is used as the segmentation network of the algorithm. The network mainly includes structures such as contraction path, expansion path and jump connection. The encoder path on the left is composed of two 3×3×3 convolution blocks and corresponding rectified linear units (ReLU), and then down-sampled by a 2×2×2 convolution block. The data size is continuously reduced, and the number of data channels is expanded from 3 to 256, thereby capturing the high-dimensional information of the 3D image. The decoder path on the right is also composed of two 3×3×3 convolution blocks and corresponding rectified linear units (ReLU) to obtain image features, and then up-sampled by a 2×2×2 convolution block to gradually restore the image to its original size. In the left and right paths, the jump connection module is used to fuse the down-sampled feature map into the up-sampled feature map and output the final feature map. Finally, the number of data channels is reduced from 64 to 1 through two 1×1×1 convolution blocks, and finally the extracted feature map is classified to achieve segmentation.

[0109]

[0110] in Represents the loss function of the segmentation network, P i represents the predicted value, G i Represents the true label value, N represents all the image patches in the dataset, and ε represents a small smoothing constant to avoid the denominator being zero.

[0111] S7, use the trained image segmentation network model to perform blood vessel segmentation.

[0112] like Figure 6 As shown in the figure, in the MRA image, the grayscale value of the pixel at the edge of the blood vessel changes in a jumpy manner compared to the grayscale value of the pixel at other positions. This grayscale change can effectively identify the contour information of the blood vessel. The greater the change in grayscale value, the more the gradient map can reflect the structural contour here, so the contour information can be extracted by calculating the image gradient. The left figure is an original MRA image. The grayscale values ​​of the pixels on both sides of the blood vessel edge are quite different. After the gradient transformation, an MRA gradient map with a clear contour can be obtained. The image gradient map is fused with the original image to obtain the gradient-enhanced MRA image on the right. As can be seen from the right figure, the edge structure of the gradient-enhanced MRA image is more obvious than the original image, which deepens the blood vessel contour structure and is conducive to extracting edge structure information.

[0113] like Figure 7 As shown in the figure, the segmentation effect image using super-resolution refers to the method we proposed - the blood vessel segmentation method based on super-resolution technology; the segmentation effect image without super-resolution refers to the effect image of the basic blood vessel segmentation method without super-resolution; the standard segmentation effect image refers to the real and accurate blood vessel segmentation effect image.

[0114] The dashed circles in the figure highlight the differences in visualization results between different methods, and circle the areas where our proposed method is superior to other methods. It can be seen that the segmentation effect of our method in local areas is close to the real result.

[0115] Therefore, this method for segmenting fine objects based on super-resolution technology proposes to apply image gradients to super-resolution technology and add a gradient branch to provide structural prior information for the super-resolution branch to constrain the generation of super-resolution images. Furthermore, the cerebral blood vessels have rich geometric structures, and the super-resolution technology based on image gradients focuses on the geometric structure, which can effectively increase the vascular structure information and help improve the accuracy of vascular segmentation.

[0116] The fine object segmentation method based on super-resolution technology applies super-resolution technology to fine objects, enlarges the image, and improves the image resolution. Image resolution represents the amount of information stored in the image. The higher the resolution, the more information. We apply super-resolution technology to fine objects and improve the image resolution, which increases the feature information of fine objects, thereby improving the subsequent segmentation accuracy. In the example of this article, the proportion of pixels occupied by cerebral blood vessels in the entire brain image is relatively low, and the local features of blood vessels cannot be accurately extracted. Super-resolution technology is applied to blood vessel segmentation, and brain images are enlarged using super-resolution technology, thereby further increasing blood vessel detail information and improving blood vessel segmentation accuracy, solving the problem of low proportion of blood vessel pixels and inability to accurately extract local features of blood vessels; secondly, it is proposed to apply image gradients to super-resolution, and through image gradients, the characteristics of blood vessel structure are enhanced, assisting the generation of super-resolution images, highlighting the sharpness and areas that should be paid attention to in the structure, and solving the problem of structural distortion that is easy to occur in super-resolution.

[0117] See also Figure 8 An embodiment of the present invention provides a device for segmenting a fine object based on super-resolution technology, including a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement the fine object segmentation method based on super-resolution technology in the above embodiment.

[0118] The embodiment of the fine object segmentation device based on super-resolution technology of the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 8 As shown, it is a hardware structure diagram of any device with data processing capability where the fine object segmentation device based on super-resolution technology of the present invention is located. Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0119] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0120] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0121] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for segmenting fine objects based on super-resolution technology in the above embodiment is implemented.

[0122] The computer-readable storage medium may be an internal storage unit of any device with data processing capability in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store computer programs and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0123] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0124] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0125] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "a", "an", and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0126] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0127] The above are only preferred embodiments of one or more embodiments of this specification and are not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for segmenting fine objects based on super-resolution technology, characterized in that: The following steps are involved: S1, acquiring three-dimensional brain MRA image data with blood vessel annotations, randomly dividing the three-dimensional brain MRA image data into a plurality of three-dimensional image block data, and constructing a three-dimensional image segmentation data set; S2, converting each three-dimensional image block data in the three-dimensional image segmentation data set into two-dimensional image data to obtain high-resolution image data; S3, down-sampling the high-resolution image data to generate low-resolution image data, and constructing a super-resolution dataset from the low-resolution image data and the high-resolution image data; S4, using a super-resolution dataset to train a super-resolution network model based on image gradients; the super-resolution network model includes a super-resolution branch network model and a gradient branch network model; First, the low-resolution image data in the super-resolution data set is input into the super-resolution branch network model to extract features and output a feature map; and the low-resolution image data in the super-resolution data set is subjected to gradient transformation to obtain a gradient map, and the gradient map is input into the gradient branch network model to extract structural features and output a feature map, wherein part of the feature map output by the super-resolution branch network model is input into the gradient branch network model for fusion to assist in the generation of a super-resolution gradient map; Then, the feature map output by the gradient branch network model is input into the super-resolution branch network model, the feature map output by the super-resolution branch network model and the feature map output by the gradient branch network model are fused, and a fused feature map is output; Finally, the fused feature map is upsampled in the super-resolution branch network model to obtain super-resolution image data; S5, using the trained super-resolution network model to perform super-resolution processing on the three-dimensional image segmentation dataset to obtain super-resolution image data with a resolution twice as large as the original one, and construct a super-resolution image segmentation dataset; S6, using the super-resolution image segmentation dataset to train a three-dimensional segmentation network to obtain an image segmentation network model; S7, use the trained image segmentation network model to perform blood vessel segmentation.

2. The method for segmenting fine objects based on super-resolution technology according to claim 1, characterized in that: The super-resolution branch network model includes multiple super-resolution modules, a fusion module and an up-sampling module; The low-resolution image data in the super-resolution dataset is input into the super-resolution branch network model. The super-resolution branch network model first uses a 3×3 convolution kernel to extract shallow features of the low-resolution image data in the super-resolution dataset and outputs a feature map. Then, multiple super-resolution modules are used to extract deep features of the low-resolution image data in the super-resolution dataset and output a feature map. Then, a last fusion module is used to fuse the feature map output by the super-resolution branch network model and the feature map output by the gradient branch network model. Finally, an upsampling module is used to expand the size of the fused feature map to twice its original size to obtain super-resolution image data.

3. The method for segmenting fine objects based on super-resolution technology according to claim 2, characterized in that: The gradient branch network model includes a plurality of gradient modules; The low-resolution image data in the super-resolution data set is subjected to gradient transformation to obtain a gradient map, and the gradient map is input into the gradient branch network model to extract structural features using multiple gradient modules and output a feature map.

4. The method for segmenting fine objects based on super-resolution technology according to claim 3, characterized in that: The super-resolution module includes convolutional layers and activation layers stacked in sequence; The fusion module includes a connection layer and a convolution layer stacked in sequence, the convolution kernel size is 3×3, and the number of convolution kernels is 32; The gradient module includes convolutional layers and activation layers stacked in sequence, which are connected in series through a residual structure.

5. The method for segmenting fine objects based on super-resolution technology according to claim 1, characterized in that: Step S5 is specifically as follows: S51, performing data conversion on the 3D image segmentation data set in step S1, performing data conversion on each 3D image block data, and converting it into 2D image data; S52, using the trained super-resolution network model to perform super-resolution processing on the two-dimensional image data in step S51, to obtain super-resolution image data with a resolution doubled; S53, performing data conversion on the super-resolution image data in step S52 to generate three-dimensional super-resolution image block data; S54, collecting and generating three-dimensional super-resolution image block data, and summarizing and organizing them into a super-resolution image segmentation data set.

6. The method for segmenting fine objects based on super-resolution technology according to claim 1, characterized in that: The function of the super-resolution image data is: I SR =G(I LR ;θ G ) Where G represents the generator, θ G Represents the parameters of the generator, I LR Represents a low-resolution image, I SR Represents super-resolution image data.

7. The method for segmenting fine objects based on super-resolution technology according to claim 6, characterized in that: The gradient map of an image is obtained by calculating the difference between adjacent pixels. The function of the gradient map is: I x (Z)=I(x+1,y)-I(x,y) I y (Z)=I(x,y+1)-I(x,y) Where Z = (x, y) represents the pixel coordinates in the image, I x (Z) represents the difference of pixel point Z in the horizontal direction x, I y (Z) represents the difference of pixel Z in the horizontal direction y, represents the difference coordinates of the pixel point, M(·) represents the operation of extracting the gradient map, and ||·||2 represents the L2 norm. Since the amplitude information is sufficient to represent the sharpness of the local area of ​​the image, only the amplitude information of the gradient is used here, and the direction information is not considered.

8. The method for segmenting fine objects based on super-resolution technology according to claim 7, characterized in that: The objective function of the super-resolution network model is: Among them, α1, α2, α3, β1, β2 represent the trade-off parameters of different losses; represents the loss of the super-resolution branch network model, represents the loss of the gradient branch network model, represents the image perceptual loss, Represents the pixel loss of the image on the super-resolution branch network model, represents the adversarial loss of the image on the super-resolution branch network model, represents the pixel loss of the gradient map on the super-resolution branch network model, represents the adversarial loss of the gradient map on the super-resolution branch network model, Represents the pixel loss of the gradient map on the gradient branch network model; in, Represents the pixel-level loss of the image, I LR Represents a low-resolution image, I HR represents a high-resolution image, G(·) represents the super-resolution network generator, ||·|| represents the use of L1 paradigm, represents the operation of averaging it, represents the pixel-level loss of the gradient image, and M(·) represents the gradient image after gradient change; in, represents the perceptual loss, implemented by the pre-trained VGG network, φ i Represents the output features of the i-th layer of the VGG model. Perceptual loss is used to measure the difference between HR image features and SR image features; in represents the adversarial loss, and D(·) represents the discriminator in the GAN network.

9. A device for segmenting fine objects based on super-resolution technology, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the subtle object segmentation method based on super-resolution technology as described in any one of claims 1 to 8.

Images