Three-dimensional biomedical video compression method, system, device and storage medium

Abstract

The present invention discloses a 3D biomedical video compression method, system, device and storage medium. Through a 3D biomedical video compression framework including 3D motion estimation, motion vector coding, 3D motion compensation and residual coding, the 3D biomedical video compression performance can be greatly improved and important information in the 3D biomedical video can be retained.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional biomedical video compression, and in particular to a three-dimensional biomedical video compression method, system, device and storage medium. Background Art

[0002] Biomedical videos depict the changes in three-dimensional biomedical images along the fourth dimension, time. Redundancy in the temporal dimension is generally higher than redundancy in the spatial dimension. Among various biomedical video formats, functional magnetic resonance imaging (fMRI) is the most representative, a powerful tool for measuring changes in neuronal activity induced by hemodynamics. Furthermore, biological videos can be used to study structural changes in organisms; for example, three-dimensional biological videos captured using optical microscopy can help observe activities such as cell division.

[0003] Captured 3D biomedical videos typically have hundreds of frames in the temporal dimension, posing significant storage and transmission challenges. To address these challenges, efficient biomedical video coding methods are urgently needed. These methods must support high-bitrate lossy or even lossless coding to preserve all important information.

[0004] Currently, the following two types of technical solutions are mainly used:

[0005] 1. Three-dimensional video coding method based on wavelet transform.

[0006] Wavelet-based methods primarily draw on the concept of transform coding, aiming to transform the original information into another domain with more concentrated energy. The wavelet transform was chosen primarily for its highly regarded excellent decomposition capabilities and reversibility. For this reason, wavelet-based methods are also frequently used for both lossy and lossless 3D video coding. The most commonly used wavelet-based method is JP3D (3D Image Extension of the JPEG-2000 Standard), part 10 of the renowned JPEG-2000 (an image coding standard created and maintained by the JPEG organization). However, JP3D cannot remove redundant information beyond three dimensions, limiting its performance. Recent work has investigated the 4D wavelet transform and its improved variants for 3D biomedical video coding. These methods typically treat 3D video as a 4D image; that is, they do not distinguish between the temporal dimension and other spatial dimensions. Because the computational process for both temporal and spatial dimensions is identical, they cannot completely remove redundant spatiotemporal information. However, it is important to note that the temporal dimension has different characteristics from other dimensions. For example, correlation in the temporal dimension is often greater than correlation in the spatial dimension. Therefore, special treatment of the temporal dimension is required, such as estimating the motion between adjacent frames. In addition to these issues, similar to motion-based methods, these methods also hand-design the transformation and entropy modules independently instead of jointly optimizing them.

[0007] 2. Traditional motion-based 3D video coding method.

[0008] The core idea of ​​motion-based methods is predictive coding, which focuses on modeling the motion vector (MV) between two frames by designing motion estimation (ME) and motion compensation (MC) modules. Previous work has directly applied motion-based methods designed for 2D natural videos, such as Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC), to 3D biomedical videos. However, these methods fail to exploit correlation in the third dimension, the axial direction. Consequently, their coding performance is significantly limited. Unlike these methods, the multi-view extension of HEVC (MV-HEVC) supports simultaneous coding of multiple 2D videos using inter-layer prediction. Therefore, MV-HEVC has the potential to exploit correlation in all dimensions. However, MV-HEVC for 3D biomedical video coding still faces several challenges. First, the prediction, transform, and entropy coding modules are designed and optimized separately, resulting in poor performance. Second, the ME module of MV-HEVC is designed based on motion in natural videos, such as rigid body motion. In contrast, motion in biomedical videos is characterized by the irregular scaling of some biological tissues. This mismatch between these two types of motion leads to inefficient ME modules. Third, to be compatible with HEVC decoders, the multi-view information in MV-HEVC is only referenced during the prediction process. The transform and entropy coding modules ignore this valuable information for further use, resulting in incomplete removal of redundant information. Summary of the Invention

[0009] The purpose of the present invention is to provide a three-dimensional biomedical video compression method, system, device and storage medium, which can improve the three-dimensional biomedical video compression performance.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] A three-dimensional biomedical video compression method, comprising:

[0012] The three-dimensional biomedical video is encoded frame by frame. When encoding the current frame image, motion estimation is performed using the current frame image and the reconstructed reference frame image corresponding to the previous frame through a three-dimensional motion estimation network to obtain a motion vector; the motion vector is encoded and decoded through a motion vector encoding network to obtain a reconstructed motion vector; motion compensation is performed using the reconstructed motion vector and the reference frame image corresponding to the previous frame through a three-dimensional motion compensation module to obtain a predicted frame image; the predicted frame image and the current frame image are combined to calculate the residual, and the residual is encoded and decoded through a residual encoding network to obtain a reconstructed residual; the reconstructed residual and the predicted frame image are used to generate a reconstructed reference frame image corresponding to the current frame, and applied to the encoding of the next frame image; wherein, an entropy model is provided in both the motion vector encoding network and the residual encoding network, and the three-dimensional motion estimation network, the motion vector encoding network, the three-dimensional motion compensation module and the residual encoding network are jointly optimized in an end-to-end manner.

[0013] A three-dimensional biomedical video compression system includes: a three-dimensional motion estimation network, a motion vector coding network, a three-dimensional motion compensation module, a residual coding network and a reference frame reconstruction module;

[0014] The three-dimensional biomedical video is encoded frame by frame through a three-dimensional biomedical video compression system. When the current frame image is encoded, the three-dimensional motion estimation network uses the current frame image and the reconstructed reference frame image corresponding to the previous frame to perform motion estimation to obtain a motion vector; the motion vector encoding network encodes and decodes the motion vector to obtain a reconstructed motion vector; the three-dimensional motion compensation network uses the reconstructed motion vector and the reference frame image corresponding to the previous frame to perform motion compensation to obtain a predicted frame image; the residual coding network combines the predicted frame image and the current frame image to calculate the residual, and encodes and decodes the residual to obtain a reconstructed residual; the reference frame reconstruction module uses the reconstructed residual and the predicted frame image to generate a reconstructed reference frame image corresponding to the current frame, and applies it to the encoding of the next frame image; wherein, an entropy model is provided in the motion vector encoding network and the residual coding network, and the three-dimensional motion estimation network, the motion vector encoding network, the three-dimensional motion compensation module and the residual coding network are jointly optimized in an end-to-end manner.

[0015] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0017] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0018] It can be seen from the technical solution provided by the present invention that a three-dimensional biomedical video compression framework is designed, which includes a three-dimensional motion estimation network, a motion vector coding network, a three-dimensional motion compensation module and a residual coding network. The framework can jointly optimize each part end-to-end, and performs different processing on the time dimension and the spatial dimension. Through this framework, the three-dimensional biomedical video compression performance can be greatly improved, and important information in the three-dimensional biomedical video can be retained. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a framework of a three-dimensional biomedical video compression method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a three-dimensional motion estimation network provided by an embodiment of the present invention;

[0022] Figure 3 A flowchart of Example 1 provided in accordance with an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a motion vector coding network based on an autoencoder provided in an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a residual coding network based on learnable wavelets provided in an embodiment of the present invention;

[0025] Figure 6 This is a flowchart of Example 2 provided by an embodiment of the present invention;

[0026] Figure 7 This is a flowchart of Example 3 provided by an embodiment of the present invention;

[0027] Figure 8 A schematic diagram illustrating the beneficial effects of a three-dimensional biomedical video coding framework provided by an embodiment of the present invention;

[0028] Figure 9 A schematic diagram of a three-dimensional biomedical video compression system provided by an embodiment of the present invention;

[0029] Figure 10 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] First, the following terms may be used in this article:

[0032] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0033] The following describes in detail a three-dimensional biomedical video compression method provided by the present invention. Any information not described in detail in the embodiments of the present invention is well known to those skilled in the art. For any unspecified conditions in the embodiments of the present invention, the process is performed in accordance with conventional conditions in the art or the conditions recommended by the manufacturer.

[0034] Example 1

[0035] The embodiment of the present invention provides a three-dimensional biomedical video compression method, such as Figure 1 As shown, the overall framework of this method is as follows: during the encoding process, a three-dimensional biomedical video is encoded frame by frame. When encoding the current frame image, a three-dimensional motion estimation network uses the current frame image and the reconstructed reference frame image corresponding to the previous frame to perform motion estimation to obtain a motion vector; the motion vector encoding network encodes and decodes the motion vector to obtain a reconstructed motion vector; the three-dimensional motion compensation module uses the reconstructed motion vector and the reference frame image corresponding to the previous frame to perform motion compensation to obtain a predicted frame image; the predicted frame image and the current frame image are combined to calculate the residual, and the residual encoding network encodes and decodes the residual to obtain a reconstructed residual; the reconstructed residual and the predicted frame image are used to generate a reconstructed reference frame image corresponding to the current frame (the result of encoding the current frame image), which is then applied to the encoding of the next frame image. In the above encoding process, both the motion vector and the residual need to be encoded and transmitted to the relevant decoder for reconstruction. The motion vector encoding network and the residual encoding network are both equipped with an entropy model. The three-dimensional motion estimation network, the motion vector encoding network, the three-dimensional motion compensation module, and the residual encoding network are jointly optimized in an end-to-end manner.

[0036] In the embodiment of the present invention, it can be used for both lossy and lossless coding. Figure 1 Path 2 in the example is used for lossless coding, while path 1 is used for lossy coding. Specifically, for lossy coding, the motion compensation result is the predicted frame image, and the residual is obtained by subtracting the predicted frame image from the current frame image. For lossless coding, the motion compensation result is rounded to an integer to obtain the predicted frame image, mainly because lossless coding of floating-point numbers consumes a large number of bits.

[0037] In an embodiment of the present invention, a three-dimensional biomedical video contains four dimensions, namely, a time dimension t and spatial dimensions x, y, and z. In MV-HEVC, a two-dimensional video along the z dimension of a three-dimensional video is called a view, and videos of different z dimensions constitute multi-view information. There is a lot of redundancy between these views, but MV-HEVC has not designed an effective technology to remove this redundant information during the transformation and entropy coding process. In an embodiment of the present invention, a three-dimensional motion estimation network, a motion vector coding network, a three-dimensional motion compensation module, and a residual coding network form a three-dimensional biomedical video compression framework. The residual coding network and the entropy model are both three-dimensional, thereby removing redundancy between views, and the entire framework can be jointly optimized end-to-end.

[0038] In order to more clearly demonstrate the technical solutions and technical effects provided by the present invention, the following specific embodiments are given. Figure 1 A detailed description of each part of the framework is given below.

[0039] 1. 3D motion estimation network.

[0040] In the embodiment of the present invention, the 3D motion estimation network is mainly responsible for performing motion estimation using the current frame image and the reconstructed reference frame image corresponding to the previous frame to obtain a motion vector.

[0041] It is observed that over time, only some areas in a 3D video move dramatically, while other areas contain a lot of imaging noise and are mostly static. 3D biomedical videos have common features, such as noticeable motion of small areas over time and large background noise. Therefore, performing motion estimation on the entire frame is complex and inefficient. In an embodiment of the present invention, based on these observations, a simple and efficient 3D motion estimation network is proposed. First, the most intensely moving areas in the 3D video are predicted using a segmentation network. Second, the motion vectors between frames of the predicted areas are obtained using an unsupervised ME network.

[0042] like Figure 2 As shown, the 3D motion estimation network provided by the embodiment of the present invention includes a segmentation network and an unsupervised motion estimation network.

[0043] 1. Segment the network.

[0044] In an embodiment of the present invention, the input of the segmentation network is the current frame image, and the foreground area of ​​the current frame is segmented by the segmentation network.

[0045] For example, the segmentation network can adopt a three-dimensional U-Net structure. Figure 2 The upper part shows the relevant structure. The 3D U-Net consists of multiple 3D convolutional layers, maximum pooling layers, trilinear upsampling layers and multiple skip connections. Specifically, Figure 2 Among the 21 boxes in the upper part: the 3rd, 6th and 9th represent maximum pooling layers, the 12th, 15th and 18th represent trilinear upsampling layers, and the rest represent 3D convolutional layers. The dotted arrows represent skip connections.

[0046] 2. Unsupervised motion estimation network.

[0047] In an embodiment of the present invention, the foreground area of ​​the current frame, the current frame image and the reconstructed reference frame image corresponding to the previous frame are input into the unsupervised motion estimation network, the current frame image and the reconstructed reference frame image corresponding to the previous frame are merged and multiplied with the foreground area of ​​the current frame to obtain the area where the motion vector needs to be predicted, and the area where the motion vector needs to be predicted is predicted to obtain the motion vector.

[0048] Figure 2 The lower part shows the structure of the unsupervised motion estimation network, which includes multiple 3D convolutional layers and multiple trilinear upsampling layers. First, several 3D convolutional layers are connected in sequence, and then several 3D linear upsampling layers and 3D convolutional layers are alternately set. The output part of each 3D convolutional layer is equipped with a leaky relu layer as a nonlinear activation function. Specifically, Figure 2 The dotted boxes in the lower half represent the 3D convolutional layer + leakyrelu layer, and the solid boxes represent the trilinear upsampling layer.

[0049] In order to facilitate training, a large number of skip connections are added to the 3D motion estimation network ( Figure 2 dashed arrow in the figure).

[0050] 2. Motion vector coding network.

[0051] In an embodiment of the present invention, a motion vector encoding network based on an autoencoder is used to encode and decode motion vectors. The motion vector encoding network based on an autoencoder includes: a motion vector encoding forward transform network, a first quantization module, a first entropy model, a first arithmetic encoder, a first arithmetic decoder, a first inverse quantization module, and a motion vector encoding inverse transform network.

[0052] Among them, the motion vector is forward transformed by the motion vector coding forward transformation network, and then quantized by the first quantization module to obtain a quantization feature, and the probability parameter of the quantization feature is calculated by the first entropy model, which is called the first probability parameter; the quantization feature is arithmetically encoded by the first arithmetic encoder in combination with the first probability parameter to obtain a first encoding code stream; the first encoding code stream is arithmetically decoded by the first arithmetic decoder in combination with the first probability parameter to obtain a quantization feature, and is dequantized by the first dequantization module, and then detransformed by the motion vector coding inverse transformation network to obtain a reconstructed motion vector.

[0053] 3. Three-dimensional motion compensation network.

[0054] In the embodiment of the present invention, the input of the three-dimensional motion compensation network is the reference frame image corresponding to the previous frame of the reconstructed motion vector.

[0055] When lossy coding is used, the reconstructed motion vector is used to perform a warp operation on the reference frame image corresponding to the previous frame to obtain a predicted frame image; when lossless coding is used, the reconstructed motion vector is used to perform a warp operation on the reference frame image corresponding to the previous frame, and then a rounding operation is performed to obtain a predicted frame image.

[0056] Those skilled in the art will understand that the warp operation refers to: transforming an image to the perspective of another image, or moving an image to a target position according to relative motion.

[0057] 4. Residual coding network.

[0058] In an embodiment of the present invention, a residual coding network is used to encode and decode the residual; wherein the residual is obtained by subtracting the current frame image from the predicted frame image.

[0059] In an embodiment of the present invention, the residual coding network includes: a residual coding forward transform network, a second quantization module, a second entropy model, a second arithmetic encoder, a second arithmetic decoder, a second inverse quantization module and a residual coding inverse transform network.

[0060] When lossless coding is used, the residual is forward transformed through the residual coding forward transform network to obtain residual features, and the probability parameters of the residual features, called second probability parameters, are calculated through the second entropy model; the residual features are arithmetically encoded by the second arithmetic encoder in combination with the second probability parameters to obtain a second encoded code stream; the second encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the second probability parameters to obtain residual features, and then inversely transformed through the residual coding inverse transform network to obtain a reconstructed residual.

[0061] When lossy coding is used, the residual is forward transformed through the residual coding forward transform network, and quantized through the second quantization module to obtain quantized residual features, and the probability parameters of the quantized residual features are calculated through the second entropy model, which are called third probability parameters; the quantized residual features are arithmetically encoded by the second arithmetic encoder in combination with the third probability parameters to obtain a third encoded code stream; the third encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the third probability parameters to obtain quantized residual features, which are dequantized through the second dequantization module and then detransformed through the residual coding inverse transform network to obtain a reconstructed residual.

[0062] In an embodiment of the present invention, the residual coding network may use an end-to-end learnable wavelet solution.

[0063] To facilitate understanding, the following three examples are used to introduce the method.

[0064] Example 1

[0065] Figure 3 This is the flowchart of Example 1, which uses a three-dimensional motion estimation network, a motion vector coding network based on an autoencoder, and a residual coding network based on a learnable wavelet.

[0066] The input is the current frame image and the reconstructed reference frame image, and the output is the reconstructed frame image. 3D biomedical video is encoded frame by frame. Specifically:

[0067] 1) The current frame image and the reconstructed reference frame image are fed into a 3D motion estimation network to obtain motion vectors. The motion vectors need to be compressed and transmitted to the decoder for reconstruction.

[0068] Figure 4 This article provides a schematic diagram of an autoencoder-based motion vector encoding network. The forward motion vector encoding network consists of multiple convolutional layers (Conv) arranged in sequence, with the outputs of the first to second-to-last convolutional layers each using a Reinforced Luminance (ReLU) activation function. The inverse motion vector encoding network consists of multiple deconvolutional layers (Deconv) arranged in sequence, with the outputs of the second to last deconvolutional layers each using a Reinforced Luminance (ReLU) activation function. The relevant processes in the autoencoder-based motion vector encoding network have been previously described in detail and will not be repeated here.

[0069] 2) The motion compensation module uses the reconstructed motion vectors to perform a warp operation on the reference frame. For lossy coding, the output of the motion compensation module is the predicted frame, and the residual is obtained by subtracting the predicted frame from the current frame. For lossless coding, the result of the warp operation must first be rounded to obtain the predicted frame.

[0070] 3) Subtract the current frame from the predicted frame to obtain the residual. After obtaining the residual, the residual codec network performs the encoding and decoding operations on the residual.

[0071] In this example, the residual encoding and decoding network uses a residual coding network based on learnable packets, such as Figure 5 The relevant processes in the residual coding network have been introduced in detail before. The main difference is that when using the learnable wavelet solution, the residual coding forward transform network and the residual coding inverse transform network in the residual coding network based on the learnable wavelet correspond to the learnable wavelet forward transform and learnable wavelet inverse transform. The rest of the parts are the same, so they will not be repeated here. Figure 5 In the

[15] , path 1 is used for lossy coding and path 2 is used for lossless coding. In lossy coding, the learnable inverse wavelet transform result also needs to be input into the post-processing module, which is composed of multiple layers of convolution to compensate for quantization loss.

[0072] 4) Finally, the reconstructed residual is added to the predicted frame to obtain the reconstructed frame.

[0073] During the entire process above, the motion vector and residual need to be encoded and transmitted to the decoder for reconstruction.

[0074] Example 2

[0075] Figure 6 This is a flowchart of Example 2. Unlike Example 1, the motion vector coding and residual coding used in Example 2 both use corresponding modules in traditional coding (such as HEVC). The rest of the process is the same as in Example 1.

[0076] Example 3

[0077] Figure 7 This is the flowchart of Example 2. Unlike Example 1, Example 3 uses a traditional motion estimation network, and the rest of the process is consistent with that of Example 1.

[0078] In order to illustrate the effect of the above solution of the present invention, relevant experiments were carried out.

[0079] 1. Beneficial effects of a three-dimensional biomedical video coding framework.

[0080] On typical 3D biomedical video datasets, including the ACDC dataset and the Fluo dataset, the experimental results are as follows: Figure 8As shown in the figure, Bit per pixel (BPP) represents pixel depth, and PSNR represents peak signal-to-noise ratio; ours and ours_fine represent the solutions of the present invention, except that ours_fine performs additional multi-frame backhaul fine-tuning during training; aiWave is a 3D image coding method (D.Xue, H.Ma, L.Li, D.Liu, and Z.Xiong, “aiWave: VolumetricImageCompression with 3-D Trained Affine Wavelet-like Transform,” arXiv preprint arXiv:2203.05822, 2022); MV-HEVC and HEVC are traditional motion-based 3D video coding methods mentioned in the background technology section. JP3D is a wavelet transform-based method. In 3D video coding, the dimensions {x, y, z} or {x, y, t} can be selected to produce a 2D video for encoding. The corresponding curves are marked with -xyz and -zyt, where xyz refers to the three spatial dimensions of the 3D video and xyt refers to the two spatial dimensions and the time dimension of the 3D video. From the curves corresponding to the experimental results of each solution, it can be seen that the present invention achieves the highest performance, that is, better reconstruction quality is achieved at the same bit rate, or in other words, more bit rate is saved at the same reconstruction quality.

[0081] 2. Beneficial effects of 3D motion estimation network.

[0082] Table 1 shows the warp PSNR under different settings of the 3D motion estimation network. The warp PSNR is defined as the PSNR (peak signal-to-noise ratio) between the predicted frame and the current frame. As can be seen from Table 1, when the segmentation network is removed (i.e., the last 5 rows marked with the symbol "×" in the Segmentation part), the warp PSNR drops significantly. Experimental results show that due to the irregular motion and severe background noise in biomedical videos, it is difficult to accurately estimate the motion vector for full-frame 3D motion estimation. The experiment also tried to increase the number of channels in the unsupervised motion estimation network. The warp PSNR gradually increases, but there is still a large gap with the performance using the segmentation network. The experimental results demonstrate the effectiveness of the proposed 3D motion estimation network.

[0083] Table 1: Results of 3D motion estimation network

[0084]

[0085] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0086] Example 2

[0087] The present invention also provides a three-dimensional biomedical video compression system, which is mainly implemented based on the method provided in the above embodiment. Figure 9 As shown, the system mainly includes: 3D motion estimation network, motion vector coding network, 3D motion compensation module, residual coding network and reference frame reconstruction module;

[0088] The three-dimensional biomedical video is encoded frame by frame through a three-dimensional biomedical video compression system. When the current frame image is encoded, the three-dimensional motion estimation network uses the current frame image and the reconstructed reference frame image corresponding to the previous frame to perform motion estimation to obtain a motion vector; the motion vector encoding network encodes and decodes the motion vector to obtain a reconstructed motion vector; the three-dimensional motion compensation network uses the reconstructed motion vector and the reference frame image corresponding to the previous frame to perform motion compensation to obtain a predicted frame image; the residual coding network combines the predicted frame image and the current frame image to calculate the residual, and encodes and decodes the residual to obtain a reconstructed residual; the reference frame reconstruction module uses the reconstructed residual and the predicted frame image to generate a reconstructed reference frame image corresponding to the current frame, and applies it to the encoding of the next frame image; wherein, an entropy model is provided in the motion vector encoding network and the residual coding network, and the three-dimensional motion estimation network, the motion vector encoding network, the three-dimensional motion compensation module and the residual coding network are jointly optimized in an end-to-end manner.

[0089] In the embodiment of the present invention, the three-dimensional motion estimation network includes: a segmentation network and an unsupervised motion estimation network

[0090] The input of the segmentation network is the current frame image, and the segmentation result is generated by the segmentation network; the segmentation result, the current frame image and the reconstructed reference frame image corresponding to the previous frame are input into the unsupervised motion estimation network, the current frame image and the reconstructed reference frame image corresponding to the previous frame are merged and multiplied with the segmentation result, the area where the motion vector needs to be predicted in the current frame image and the reconstructed reference frame image corresponding to the previous frame is segmented, the area where the motion vector needs to be predicted is predicted, and the motion vector is obtained.

[0091] In an embodiment of the present invention, the motion vector coding network is a motion vector coding network based on an autoencoder;

[0092] The motion vector coding network based on the autoencoder includes: a motion vector coding forward transform network, a first quantization module, a first entropy model, a first arithmetic encoder, a first arithmetic decoder, a first inverse quantization module and a motion vector coding inverse transform network;

[0093] Among them, the motion vector is forward transformed by the motion vector coding forward transformation network, and then quantized by the first quantization module to obtain a quantization feature, and the probability parameter of the quantization feature is calculated by the first entropy model, which is called the first probability parameter; the quantization feature is arithmetically encoded by the first arithmetic encoder in combination with the first probability parameter to obtain a first encoding code stream; the first encoding code stream is arithmetically decoded by the first arithmetic decoder in combination with the first probability parameter to obtain a quantization feature, and is dequantized by the first dequantization module, and then detransformed by the motion vector coding inverse transformation network to obtain a reconstructed motion vector.

[0094] In an embodiment of the present invention, the residual coding network includes: a residual coding forward transform network, a second quantization module, a second entropy model, a second arithmetic encoder, a second arithmetic decoder, a second inverse quantization module and a residual coding inverse transform network;

[0095] When lossless coding is used, the residual is forward transformed through the residual coding forward transform network to obtain residual features, and the probability parameters of the residual features, called second probability parameters, are calculated through the second entropy model; the residual features are arithmetically encoded by the second arithmetic encoder in combination with the second probability parameters to obtain a second encoded code stream; the second encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the second probability parameters to obtain residual features, and then inversely transformed through the residual coding inverse transform network to obtain a reconstructed residual.

[0096] When lossy coding is used, the residual is forward transformed through the residual coding forward transform network, and quantized through the second quantization module to obtain quantized residual features, and the probability parameters of the quantized residual features are calculated through the second entropy model, which are called third probability parameters; the quantized residual features are arithmetically encoded by the second arithmetic encoder in combination with the third probability parameters to obtain a third encoded code stream; the third encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the third probability parameters to obtain quantized residual features, which are dequantized through the second dequantization module and then detransformed through the residual coding inverse transform network to obtain a reconstructed residual.

[0097] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0098] Example 3

[0099] The present invention also provides a processing device, such as Figure 10 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0100] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0101] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0102] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0103] The output device may be a display terminal;

[0104] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0105] Example 4

[0106] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0107] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0108] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A three-dimensional biomedical video compression method, characterized in that: include: The three-dimensional biomedical video is encoded frame by frame. When encoding the current frame image, the motion estimation is performed using the current frame image and the reconstructed reference frame image corresponding to the previous frame through the three-dimensional motion estimation network to obtain a motion vector; the motion vector is encoded and decoded through the motion vector encoding network to obtain a reconstructed motion vector; the three-dimensional motion compensation module uses the reconstructed motion vector and the reconstructed reference frame image corresponding to the previous frame to perform motion compensation to obtain a predicted frame image; the predicted frame image and the current frame image are combined to calculate the residual, and the residual coding network is used to encode and decode the residual to obtain a reconstructed residual; the reconstructed residual and the predicted frame image are used to generate a reconstructed reference frame image corresponding to the current frame, and applied to the encoding of the next frame image; wherein, an entropy model is provided in both the motion vector encoding network and the residual coding network, and the three-dimensional motion estimation network, the motion vector encoding network, the three-dimensional motion compensation module and the residual coding network are jointly optimized in an end-to-end manner; The calculating of the residual by combining the predicted frame image and the current frame image, and encoding and decoding the residual by a residual coding network to obtain a reconstructed residual includes: Subtracting the current frame image from the predicted frame image to obtain a residual, and using a residual coding network to encode and decode the residual; the residual coding network includes: a residual coding forward transform network, a second quantization module, a second entropy model, a second arithmetic encoder, a second arithmetic decoder, a second inverse quantization module and a residual coding inverse transform network; When lossless coding is used, the residual is forward transformed by the residual coding forward transform network to obtain residual features, and a probability parameter of the residual features is calculated by the second entropy model, which is called a second probability parameter; the residual features are arithmetically encoded by the second arithmetic encoder in combination with the second probability parameter to obtain a second encoded code stream; the second encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the second probability parameter to obtain residual features, and then inversely transformed by the residual coding inverse transform network to obtain a reconstructed residual; When lossy coding is used, the residual is forward transformed by the residual coding forward transform network, and quantized by the second quantization module to obtain quantized residual features, and the probability parameters of the quantized residual features are calculated by the second entropy model, which are called third probability parameters; the quantized residual features are arithmetically encoded by the second arithmetic encoder in combination with the third probability parameters to obtain a third encoded code stream; the third encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the third probability parameters to obtain quantized residual features, which are dequantized by the second dequantization module, and then detransformed by the residual coding inverse transform network, and input into the post-processing module to obtain a reconstructed residual.

2. A three-dimensional biomedical video compression method according to claim 1, characterized in that: The step of performing motion estimation using a current frame image and a reconstructed reference frame image corresponding to a previous frame through a three-dimensional motion estimation network to obtain a motion vector includes: A three-dimensional motion estimation network including a segmentation network and an unsupervised motion estimation network is used for motion estimation; the input of the segmentation network is the current frame image, and the foreground area of ​​the current frame is segmented by the segmentation network; the foreground area of ​​the current frame, the current frame image and the reconstructed reference frame image corresponding to the previous frame are input to the unsupervised motion estimation network, the current frame image and the reconstructed reference frame image corresponding to the previous frame are merged and multiplied with the foreground area of ​​the current frame to obtain the area where the motion vector needs to be predicted, and the area where the motion vector needs to be predicted is predicted to obtain the motion vector.

3. The three-dimensional biomedical video compression method according to claim 1, characterized in that: The encoding and decoding of the motion vector by a motion vector coding network to obtain a reconstructed motion vector includes: A motion vector coding network based on an autoencoder is used to encode and decode the motion vector; the motion vector coding network based on the autoencoder includes: a motion vector coding forward transform network, a first quantization module, a first entropy model, a first arithmetic encoder, a first arithmetic decoder, a first inverse quantization module and a motion vector coding inverse transform network; Among them, the motion vector is forward transformed by the motion vector coding forward transformation network, and then quantized by the first quantization module to obtain a quantization feature, and the probability parameter of the quantization feature is calculated by the first entropy model, which is called the first probability parameter; the quantization feature is arithmetically encoded by the first arithmetic encoder in combination with the first probability parameter to obtain a first encoding code stream; the first encoding code stream is arithmetically decoded by the first arithmetic decoder in combination with the first probability parameter to obtain a quantization feature, and is dequantized by the first dequantization module, and then detransformed by the motion vector coding inverse transformation network to obtain a reconstructed motion vector.

4. A three-dimensional biomedical video compression system, characterized in that: Includes: 3D motion estimation network, motion vector coding network, 3D motion compensation module, residual coding network and reference frame reconstruction module; The three-dimensional biomedical video is encoded frame by frame by a three-dimensional biomedical video compression system. When the current frame image is encoded, the three-dimensional motion estimation network uses the current frame image and the reconstructed reference frame image corresponding to the previous frame to perform motion estimation to obtain a motion vector; the motion vector encoding network encodes and decodes the motion vector to obtain a reconstructed motion vector; the three-dimensional motion compensation module uses the reconstructed motion vector and the reconstructed reference frame image corresponding to the previous frame to perform motion compensation to obtain a predicted frame image; the residual encoding network combines the predicted frame image and the current frame image to calculate the residual, and encodes and decodes the residual to obtain a reconstructed residual; the reference frame reconstruction module uses the reconstructed residual and the predicted frame image to generate a reconstructed reference frame image corresponding to the current frame, and applies it to the encoding of the next frame image; wherein, an entropy model is provided in both the motion vector encoding network and the residual encoding network, and the three-dimensional motion estimation network, the motion vector encoding network, the three-dimensional motion compensation module and the residual encoding network are jointly optimized in an end-to-end manner; The residual coding network includes: a residual coding forward transform network, a second quantization module, a second entropy model, a second arithmetic encoder, a second arithmetic decoder, a second inverse quantization module and a residual coding inverse transform network; When lossless coding is used, the residual is forward transformed by the residual coding forward transform network to obtain residual features, and a probability parameter of the residual features is calculated by the second entropy model, which is called a second probability parameter; the residual features are arithmetically encoded by the second arithmetic encoder in combination with the second probability parameter to obtain a second encoded code stream; the second encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the second probability parameter to obtain residual features, and then inversely transformed by the residual coding inverse transform network to obtain a reconstructed residual; When lossy coding is used, the residual is forward transformed by the residual coding forward transform network, and quantized by the second quantization module to obtain quantized residual features, and the probability parameters of the quantized residual features are calculated by the second entropy model, which are called third probability parameters; the quantized residual features are arithmetically encoded by the second arithmetic encoder in combination with the third probability parameters to obtain a third encoded code stream; the third encoded code stream is arithmetically decoded by the second arithmetic decoder in combination with the third probability parameters to obtain quantized residual features, which are dequantized by the second dequantization module, and then detransformed by the residual coding inverse transform network, and input into the post-processing module to obtain a reconstructed residual.

5. A three-dimensional biomedical video compression system according to claim 4, characterized in that: The three-dimensional motion estimation network includes: a segmentation network and an unsupervised motion estimation network The input of the segmentation network is the current frame image, and the segmentation result is generated by the segmentation network; the segmentation result, the current frame image and the reconstructed reference frame image corresponding to the previous frame are input into the unsupervised motion estimation network, the current frame image and the reconstructed reference frame image corresponding to the previous frame are merged and multiplied with the segmentation result, the area where the motion vector needs to be predicted in the current frame image and the reconstructed reference frame image corresponding to the previous frame is segmented, the area where the motion vector needs to be predicted is predicted, and the motion vector is obtained.

6. The three-dimensional biomedical video compression system according to claim 4, characterized in that: The motion vector coding network is a motion vector coding network based on an autoencoder; The motion vector coding network based on the autoencoder includes: a motion vector coding forward transform network, a first quantization module, a first entropy model, a first arithmetic encoder, a first arithmetic decoder, a first inverse quantization module and a motion vector coding inverse transform network; Among them, the motion vector is forward transformed by the motion vector coding forward transformation network, and then quantized by the first quantization module to obtain a quantization feature, and the probability parameter of the quantization feature is calculated by the first entropy model, which is called the first probability parameter; the quantization feature is arithmetically encoded by the first arithmetic encoder in combination with the first probability parameter to obtain a first encoding code stream; the first encoding code stream is arithmetically decoded by the first arithmetic decoder in combination with the first probability parameter to obtain a quantization feature, and is dequantized by the first dequantization module, and then detransformed by the motion vector coding inverse transformation network to obtain a reconstructed motion vector.

7. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images