3D medical image compression system for human-computer vision

Abstract

The present invention discloses a three-dimensional medical image compression system for human-machine vision, wherein an inter-frame auxiliary feature cache module caches inter-frame auxiliary features; an auxiliary feature encoder network extracts inter-frame potential features from the auxiliary features and transmits them to a current image encoder network, a context entropy model, and an auxiliary feature decoder network; the current image encoder network extracts image potential features from the current image based on the inter-frame potential features; the current image decoder network decodes the image potential features based on the inter-frame decoding features; and a feature fusion module is used to fuse features output by the current image decoder network and the auxiliary feature decoder network to obtain fused features. The three-dimensional medical image compression system for human-machine vision of the present invention utilizes inter-frame information to assist in the reconstruction of the current image frame, which can achieve a compression effect superior to that of traditional image compression algorithms for human vision, and can also improve performance indicators for downstream medical tasks for machine vision.

Description

Technical Field

[0001] The present invention specifically relates to a three-dimensional medical image compression system oriented to human-machine vision. Background Art

[0002] Digital images such as CT and MRI have been widely used in clinical auxiliary diagnosis. With the development of digital imaging technology and the improvement of the accuracy of medical equipment, the size and bit depth of digital images are gradually increasing, which brings huge challenges to the storage and transmission of medical images. In recent years, with the development of the new generation of universal video compression standard VVC and the popularization of neural network technology, lossy compression methods can achieve higher compression ratios than lossless compression while retaining sufficient medical image details and textures.

[0003] However, most of the existing deep learning lossy image compression models are designed for 2D images or videos, and are not suitable for three-dimensional medical images such as CT and MRI. There are also related studies that use 3D convolution to extract inter-frame information in three-dimensional medical images, but this method is computationally intensive and time-consuming, so the efficiency of extracting inter-frame information is low. Secondly, traditional image compression algorithms are all oriented towards human vision, with the goal of preserving the details and content of the original image. In many downstream application scenarios, compared with the original image as input, the compressed and reconstructed images often cannot maintain high performance indicators. Summary of the invention

[0004] The present invention provides a three-dimensional medical image compression system for human-machine vision to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0005] A three-dimensional medical image compression system for human-machine vision, comprising: a current image encoder network, a current image decoder network, a context entropy model, an inter-frame auxiliary feature cache module, an auxiliary feature encoder network, an auxiliary feature decoder network, a feature fusion module and a reconstruction network;

[0006] The inter-frame auxiliary feature cache module caches the inter-frame auxiliary features;

[0007] The auxiliary feature encoder network extracts inter-frame potential features from the auxiliary features stored in the inter-frame auxiliary feature cache module and transmits them to the current image encoder network, the context entropy model and the auxiliary feature decoder network;

[0008] The current image encoder network extracts image potential features from the current image based on the inter-frame potential features extracted by the auxiliary feature encoder network;

[0009] The auxiliary feature decoder network decodes the inter-frame potential features extracted by the auxiliary feature encoder network into inter-frame decoding features and transmits them to the current image decoder network and the feature fusion module;

[0010] The context entropy model encodes the potential features of the image;

[0011] The current image decoder network decodes the image latent features based on the inter-frame decoding features extracted by the auxiliary feature decoder network;

[0012] The feature fusion module is used to fuse the features output by the current image decoder network and the auxiliary feature decoder network to obtain a fused feature. The fused feature is cached in the inter-frame auxiliary feature cache module as a new inter-frame auxiliary feature. The fused feature is used as an input for downstream machine vision tasks and is input into the reconstruction network. The reconstruction network reconstructs an image that conforms to human vision based on the fused feature.

[0013] Furthermore, the current image encoder network includes three layers of first Block networks, each of which includes a convolutional network layer, a GDN normalization network layer, and a ResNet residual network layer;

[0014] The auxiliary feature encoder network includes three layers of second Block networks, each of which includes a convolutional network layer, a nonlinear activation layer GELU and a ResNet residual network layer.

[0015] Furthermore, the current image decoder network includes a third Block network and two fourth Block networks, the third Block network includes a convolutional layer, an IGDN denormalization network layer and a Pixel upsampling network layer, and each of the fourth Block networks includes an IGDN denormalization network layer, a ResNet residual network layer and a Pixel upsampling network layer;

[0016] The auxiliary feature decoding network includes a fifth Block network and two sixth Block networks, the fifth Block network includes a convolutional layer, a GELU non-linear activation layer and a TConv deconvolutional network layer, and each of the sixth Block networks includes a GELU non-linear activation layer, a ResNet residual network layer and a TConv deconvolutional network layer.

[0017] Furthermore, the context entropy model models the quantized image potential features into a Gaussian distribution, and then obtains the final binary code stream file through an arithmetic encoder.

[0018] Furthermore, the context entropy model obtains Gaussian distribution parameters of potential features of the current image based on spatial context, inter-channel context and inter-frame potential feature context. The spatial context, inter-channel context and inter-frame potential feature context perform feature aggregation through inter-channel merging, and then obtains Gaussian distribution parameters of potential features of the current image through a probability prediction network layer.

[0019] Furthermore, the feature fusion module adds the features output by the current image decoder network and the auxiliary feature decoder network, and then obtains the fused features through a hyperbolic tangent activation function.

[0020] Furthermore, the fused features are used as the input of the downstream medical image segmentation task and the fused features are input into the TransUnet segmentation network.

[0021] Furthermore, when processing three-dimensional medical images through the three-dimensional medical image compression system for human-computer vision, an initialization parameter N is set for the inter-frame auxiliary feature cache module, and the inter-frame auxiliary features stored in the inter-frame auxiliary feature cache module are initialized every time N pictures are processed.

[0022] Furthermore, when the number of slices of the three-dimensional medical image to be processed is greater than a preset value M, an initialization parameter N is set for the inter-frame auxiliary feature buffer module.

[0023] Furthermore, the range of N is greater than or equal to 5 and less than or equal to 15.

[0024] The benefit of the present invention lies in the fact that the provided three-dimensional medical image compression system for human-machine vision uses inter-frame information to assist in the reconstruction of the current image frame. It can not only achieve a compression effect that is superior to traditional image compression algorithms for human vision, but also improve performance indicators for downstream medical tasks for machine vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 is a schematic diagram of a three-dimensional medical image compression system for human-machine vision of the present invention;

[0027] Figure 2 is a schematic diagram of a current image encoder network and an auxiliary feature encoder network of the present invention;

[0028] Figure 3 is a schematic diagram of a current image decoder network and an auxiliary feature decoder network of the present invention;

[0029] Figure 4 is a schematic diagram of the entropy model of multi-feature aggregation of the present invention;

[0030] Figure 5 It is a schematic diagram comparing the compression results of the human-machine vision-oriented three-dimensional medical image compression system of the present invention and the existing compression method on the MRNet data set;

[0031] Figure 6 It is a schematic diagram comparing the human-machine vision-oriented three-dimensional medical image compression system of the present invention and the existing compression method in performing segmentation tasks on the CHAOS data set. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] like Figure 1 The present application shows a three-dimensional medical image compression system for human-machine vision, which includes: a current image encoder network, a current image decoder network, a context entropy model, an inter-frame auxiliary feature cache module, an auxiliary feature encoder network, an auxiliary feature decoder network, a feature fusion module and a reconstruction network.

[0034] The inter-frame auxiliary feature cache module caches inter-frame auxiliary features. The inter-frame auxiliary feature dimension is C*H*W, where C represents the number of channels (such as 16, 32, etc.), and H and W represent the height and width of the input image size.

[0035] The auxiliary feature encoder network extracts inter-frame latent features from the auxiliary features stored in the inter-frame auxiliary feature cache module and transmits them to the current image encoder network, the context entropy model and the auxiliary feature decoder network. The current image encoder network extracts image latent features for the current image based on the inter-frame latent features extracted by the auxiliary feature encoder network. The auxiliary feature decoder network decodes the inter-frame latent features extracted by the auxiliary feature encoder network into inter-frame decoding features and transmits them to the current image decoder network and the feature fusion module. The current image decoder network decodes the image latent features based on the inter-frame decoding features extracted by the auxiliary feature decoder network.

[0036] Specifically, Figure 2 and 3As shown, the current image encoder network includes three layers of first Block networks, each of which includes a convolutional network layer, a GDN normalization network layer, and a ResNet residual network layer. The auxiliary feature encoder network includes three layers of second Block networks, each of which includes a convolutional network layer, a nonlinear activation layer GELU, and a ResNet residual network layer. The main function of the convolutional network layer is to reduce the dimension of the feature, while the GDN network is a normalization method widely used in image reconstruction tasks, and the residual network is used to extract deep potential features. It can be seen that for every group of Block networks, the auxiliary feature encoder network passes the features of the current Block to the current image encoder network. In this way, inter-frame information at multiple scales is input into the current image encoder network in a concatenated manner. Similarly, the auxiliary feature decoding network also passes the inter-frame decoding features to the current image decoder network.

[0037] The current image decoder network includes the third Block network and two fourth Block networks. The third Block network includes a convolutional layer, an IGDN denormalization network layer, and a pixel upsampling network layer. The fourth Block network includes an IGDN denormalization network layer, a ResNet residual network layer, and a pixel upsampling network layer. The role of the convolutional network layer here is mainly to change the number of channels of the current image decoding feature. The IGDN network layer corresponds to the GDN network layer in the current image encoder network and plays the role of denormalization of the decoding feature. The pixel upsampling network layer is used to change the size of the current decoding feature. The output size of this network layer is usually twice the input size.

[0038] The auxiliary feature decoding network includes the fifth Block network and two sixth Block networks. The fifth Block network includes a convolutional layer, a GELU nonlinear activation layer, and a TConv deconvolutional network layer. The sixth Block network includes a GELU nonlinear activation layer, a ResNet residual network layer, and a TConv deconvolutional network layer. Similar to the current image decoder network, the convolutional network layer is used to convert the number of channels of the decoding features, and the ResNet residual network layer is used to extract potential features. The functions of the TConv deconvolutional network layer and the Pixel upsampling layer are both to enlarge the feature size. The TConv layer has higher computational efficiency and faster speed, while the Pixel layer can restore clearer image features and is often used for pixel-level image restoration tasks. At the same time, in the auxiliary feature decoding network, each Block outputs multi-scale inter-frame decoding information, which is input into the current image decoder network in a concatenated manner.

[0039] When compressing the first frame of a 3D medical image volume, the inter-frame auxiliary features are initialized to zero values. As the first frame is encoded, the inter-frame auxiliary features are updated once. When compressing the second frame of the 3D image volume, the inter-frame auxiliary features that have just been updated will be read. By analogy, the cyclic iteration and update of the recurrent neural network are realized.

[0040] In an embodiment of the present application, when a three-dimensional medical image is processed by a three-dimensional medical image compression system for human-machine vision, an initialization parameter N is set for the inter-frame auxiliary feature cache module, and the inter-frame auxiliary features stored in the inter-frame auxiliary feature cache module are initialized every time N images are processed. In an embodiment of the present application, the range of N is greater than or equal to 5 and less than or equal to 15. In the present application, N takes a value of 10, which means that for a group of three-dimensional image slices, the inter-frame auxiliary features used when compressing the first frame are initialized to all zeros, and the inter-frame auxiliary features of the current GOP group can be used when compressing the 2nd to 10th frames, and the inter-frame auxiliary features need to be reinitialized when the next GOP group arrives.

[0041] Preferably, when the number of slices of the three-dimensional medical image to be processed is greater than a preset value M, an initialization parameter N is set for the inter-frame auxiliary feature cache module. Preferably, the value of M is 50.

[0042] The context entropy model encodes the potential features of the image. Specifically, the context entropy model models the quantized potential features of the image into a Gaussian distribution, and then obtains the final binary code stream file through an arithmetic encoder to decode the binary code stream file.

[0043] In order to further improve the prediction accuracy of the context entropy model and reduce the size of the binary file after compression of the three-dimensional medical image, this application provides an entropy model framework for multi-feature aggregation. Figure 4 As shown. In addition to spatial context and inter-channel context, the entropy model designed in this application also aggregates inter-frame latent features. The context entropy model obtains the Gaussian distribution parameters of the current image's latent features based on the spatial context, inter-channel context, and inter-frame latent feature context. The spatial context, inter-channel context, and inter-frame latent feature context perform feature aggregation by inter-channel merging, and then obtain the Gaussian distribution parameters mu and sigma of the current image's latent features through the probability prediction network layer.

[0044] The feature fusion module is used to fuse the features output by the current image decoder network and the auxiliary feature decoder network to obtain a fused feature. In an embodiment of the present application, the feature fusion module adds the features output by the current image decoder network and the auxiliary feature decoder network, and then obtains a fused feature with a dimension of C*H*W through a hyperbolic tangent activation function tanh.

[0045] On the one hand, the fused features are cached as new inter-frame auxiliary features in the inter-frame auxiliary feature cache module, so that the inter-frame auxiliary features will be updated once each time a frame of image is encoded.

[0046] On the one hand, the fused features are input into the reconstruction network, and the reconstruction network reconstructs an image that conforms to human vision based on the fused features. Specifically, if you want to reconstruct the original image, you only need to input the fused features into a reconstruction network consisting of a simple convolutional network with several layers.

[0047] Figure 5 The image reconstruction performance of this system for human vision is demonstrated. Comparing the compression performance with traditional methods on the MRNet dataset, it can be seen that the compression method provided by this application has achieved the best results on the three sub-datasets. Measured by BD-Rate (PSNR), the method of this application has 40%, 35% and 8% BD-Rate (PSNR) improvements compared to traditional methods such as JPEG-XL, HEVC, and VVC, respectively.

[0048] On the one hand, the fused features serve as the input of downstream machine vision tasks. If you want to perform further machine vision analysis on the original image, you only need to input the fused features into the corresponding analysis network and modify the network head. Taking the medical image segmentation task as an example, the fused features are input into the TransUnet segmentation network, and the number of input channels of the first layer of the convolutional network of the segmentation model is changed to the number of output channels of the fused features C = 16.

[0049] Figure 6 The image segmentation results of this system for machine vision are shown. Compared with the JP2K method on the CHAOS dataset, the higher the Dice index, the more accurate the segmentation, so the higher the index, the better. The lower the HD95, the lower the probability of error in extreme cases, so the lower the better. As can be seen from the result graph, the processing method of this application has greatly improved the segmentation performance compared with JP2K.

[0050] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A three-dimensional medical image compression system for human-machine vision, characterized in that: Contains: current image encoder network, current image decoder network, context entropy model, inter-frame auxiliary feature cache module, auxiliary feature encoder network, auxiliary feature decoder network, feature fusion module and reconstruction network; The inter-frame auxiliary feature cache module caches the inter-frame auxiliary features; The auxiliary feature encoder network extracts inter-frame potential features from the auxiliary features stored in the inter-frame auxiliary feature cache module and transmits them to the current image encoder network, the context entropy model and the auxiliary feature decoder network; The current image encoder network extracts image potential features from the current image based on the inter-frame potential features extracted by the auxiliary feature encoder network; The auxiliary feature decoder network decodes the inter-frame potential features extracted by the auxiliary feature encoder network into inter-frame decoding features and transmits them to the current image decoder network and the feature fusion module; The context entropy model encodes the potential features of the image; The current image decoder network decodes the image latent features based on the inter-frame decoding features extracted by the auxiliary feature decoder network; The feature fusion module is used to fuse the features output by the current image decoder network and the auxiliary feature decoder network to obtain a fused feature. On the one hand, the fused features are cached as new inter-frame auxiliary features in the inter-frame auxiliary feature cache module. On the other hand, the fused features serve as input for downstream machine vision tasks, and on the other hand, the fused features are input into the reconstruction network, and the reconstruction network reconstructs an image that conforms to human vision based on the fused features.

2. The three-dimensional medical image compression system for human-machine vision according to claim 1, characterized in that: The current image encoder network includes three layers of first Block networks, each of which includes a convolutional network layer, a GDN normalization network layer, and a ResNet residual network layer; The auxiliary feature encoder network includes three layers of second Block networks, each of which includes a convolutional network layer, a nonlinear activation layer GELU and a ResNet residual network layer.

3. The human-machine vision-oriented three-dimensional medical image compression system according to claim 1, characterized in that: The current image decoder network includes a third Block network and two fourth Block networks, the third Block network includes a convolutional layer, an IGDN denormalization network layer and a Pixel upsampling network layer, and each of the fourth Block networks includes an IGDN denormalization network layer, a ResNet residual network layer and a Pixel upsampling network layer; The auxiliary feature decoder network includes a fifth Block network and two sixth Block networks, the fifth Block network includes a convolutional layer, a GELU non-linear activation layer and a TConv deconvolutional network layer, and each of the sixth Block networks includes a GELU non-linear activation layer, a ResNet residual network layer and a TConv deconvolutional network layer.

4. The three-dimensional medical image compression system for human-machine vision according to claim 1, characterized in that: The context entropy model models the quantized image potential features into a Gaussian distribution, and then obtains the final binary code stream file through an arithmetic encoder.

5. The three-dimensional medical image compression system for human-machine vision according to claim 1, characterized in that: The context entropy model obtains Gaussian distribution parameters of the potential features of the current image based on spatial context, inter-channel context and inter-frame potential feature context. The spatial context, inter-channel context and inter-frame potential feature context perform feature aggregation by inter-channel merging, and then obtains Gaussian distribution parameters of the potential features of the current image through a probability prediction network layer.

6. The human-machine vision-oriented three-dimensional medical image compression system according to claim 1, characterized in that: The feature fusion module adds the features output by the current image decoder network and the auxiliary feature decoder network, and then obtains the fused features through a hyperbolic tangent activation function.

7. The three-dimensional medical image compression system for human-machine vision according to claim 1, characterized in that: The fused features are used as the input of the downstream medical image segmentation task and are input into the TransUnet segmentation network.

8. The human-machine vision-oriented three-dimensional medical image compression system according to claim 1, characterized in that: When processing three-dimensional medical images through the human-machine vision-oriented three-dimensional medical image compression system, an initialization parameter N is set for the inter-frame auxiliary feature cache module, and the inter-frame auxiliary features stored in the inter-frame auxiliary feature cache module are initialized every time N pictures are processed.

9. The human-machine vision-oriented three-dimensional medical image compression system according to claim 8, characterized in that: When the number of slices of the three-dimensional medical image to be processed is greater than a preset value M, an initialization parameter N is set for the inter-frame auxiliary feature buffer module.

10. The human-machine vision-oriented three-dimensional medical image compression system according to claim 8, characterized in that: The range of N is greater than or equal to 5 and less than or equal to 15.

Images