Multi-hypothesis motion compensated enhanced video compression snapshot imaging reconstruction method and system

Abstract

The application discloses a multi-hypothesis motion compensation enhanced video compression snapshot imaging reconstruction method and system, relates to the technical field of image processing, and inputs an initial reconstruction value into a trained multi-hypothesis motion compensation enhanced Transform reconstruction network for cascade calculation of multiple stages; in each stage, an overall reconstruction branch utilizes a multi-scale Transform coding and decoding network based on mask guidance to extract deep features and output an overall reconstruction result; a motion compensation enhancement branch uses a multi-hypothesis prediction module to perform motion estimation on a reference frame to generate a predicted frame; a residual error between the mask-modulated predicted frame and an original measurement value is calculated, and a residual reconstruction module reconstructs the calculated residual error; reconstruction information is used to optimize the predicted frame result through a jump connection to obtain a motion compensation value; the motion compensation value and the overall reconstruction result are summed to obtain a reconstruction result of the current stage; and a high-quality video after reconstruction is output.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for video compression snapshot imaging reconstruction enhanced by multi-hypothesis motion compensation. Background Technology

[0002] Video compression snapshot imaging is an emerging imaging method based on compressed sensing theory. It utilizes a mask matrix to compress and encode multiple consecutive video frames into a single-frame snapshot measurement, which is then used to reconstruct the original high-speed video using a reconstruction algorithm. This imaging method boasts advantages such as low hardware cost and high efficiency. However, constructing an efficient reconstruction algorithm remains a challenging problem. Effectively utilizing the spatiotemporal correlation of image sequences in video is key to improving reconstruction quality. However, existing deep network reconstruction methods are insufficient in representing long-range spatial context within video frames and fail to explicitly utilize the spatiotemporal correlation of moving targets between video frames for reconstruction, thus requiring further improvement in reconstruction quality. Summary of the Invention

[0003] To address the shortcomings mentioned in the background art, the present invention aims to provide a video compression snapshot imaging reconstruction method and system with enhanced multi-hypothesis motion compensation.

[0004] The objective of this invention can be achieved through the following technical solution: a video compression snapshot imaging reconstruction method enhanced by multi-hypothesis motion compensation, the method comprising the following steps:

[0005] Receive multiple consecutive video frames, compress and encode the multiple consecutive video frames using a mask matrix, and obtain a single-frame snapshot measurement;

[0006] Calculate the initial reconstruction data from the single-frame snapshot measurement;

[0007] The initial reconstruction data is input into a pre-established Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and the high-quality reconstructed video is output.

[0008] Preferably, the calculation process for the initial reconstruction data is as follows:

[0009]

[0010] Among them, X 0 output The initial reconstruction data obtained from the calculation, y represents compressed measurement data. ⊙ represents matrix division, and ⊙ represents matrix multiplication. C represents the mask after summing along the time dimension. k Represents the mask matrix.

[0011] Preferably, the multi-hypothesis motion compensation enhanced Transformer reconstruction network model includes multiple cascaded stages.

[0012] Preferably, the cascade stage includes an overall reconstruction branch and a motion compensation enhancement branch.

[0013] Preferably, the initial reconstruction data is input into the overall reconstruction branch to obtain the overall reconstruction result, and the initial reconstruction data is input into the motion compensation enhancement branch to obtain the motion compensation value;

[0014] The overall reconstruction result is summed with the motion compensation value to output the reconstructed high-quality video.

[0015] Preferably, the overall reconstruction branch includes encoding operations, an M-Trans module, downsampling operations, upsampling operations, a first convolutional layer, a second convolutional layer, and decoding operations;

[0016] After the initial reconstructed data is input into the encoding operation to output high-dimensional features, the high-dimensional features are input into the M-Trans module to output the first layer of depth features;

[0017] The first layer depth features are input into the downsampling operation, and the M-trans module outputs the second layer depth features.

[0018] The second layer depth features are input into the downsampling operation, and the M-trans module outputs the third layer depth features.

[0019] The third layer depth features are input to the upsampling operation, and after the second layer depth features are concatenated, they are sequentially passed through the first convolutional layer, the M-Trans module, and the upsampling operation to output the second layer output features.

[0020] After performing a concatenation operation between the second layer output features and the first layer depth features, the depth features are sequentially passed through the second convolutional layer, the M-Trans module, and the decoding operation to output the depth features.

[0021] The depth features are summed with the initial reconstruction data to obtain the overall reconstruction result.

[0022] Preferably, the M-Trans module includes two multi-head attention computation modules, each of which includes a layer normalization layer, a mask-guided multi-head self-attention layer, a layer normalization layer, and a multilayer perceptron layer.

[0023] The high-dimensional features are passed sequentially through the layer normalization layer and the mask-guided multi-head self-attention layer, and summed with the high-dimensional features to output attention features. The attention features are then sequentially input into the layer normalization layer and the multilayer perceptron layer, and summed with the attention features to output mask attention features.

[0024] The masked attention features are sequentially passed through the layer normalization layer and the mask-guided multi-head self-attention layer, and then summed with the masked attention features to output the attention features. These attention features are then sequentially input into the layer normalization layer and the multilayer perceptron layer, and summed with the attention features to output the current layer depth features. The computation process in a multi-head attention calculation module is as follows:

[0025] X M-MSA l =M-MSA(LN(X) l-1 ))+X l-1

[0026] X l =MLP(LN(X) M-MSA l ))+X M-MSA l

[0027] Where X l-1 The features are represented by downsampling, LN(·) represents layer normalization, M-MSA(·) represents a mask-guided multi-head self-attention layer, MLP(·) represents multi-layer perceptron processing, and X represents the features after downsampling. l This represents the computational output feature of a multi-head attention computation module within the M-Trans module;

[0028] The mask-guided multi-head self-attention layer includes an encoding layer, a projection matrix, and a mask guidance module;

[0029] The input feature X is reshaped by the encoding layer and linearly projected into a query matrix Q, a key matrix K, and a value matrix. The calculation formula is as follows:

[0030] Q = XW q K = XW k V = XW v

[0031] in Let represent the learnable projection matrices, and let H represent the number of heads in the multi-head attention mechanism. The dimension of each head is represented as... In the head h Attention calculation in (h=1,...,H) is shown below:

[0032]

[0033] head h =(M a h ⊙V h A h

[0034] in Represents the learnable parameters. K represents h The transpose of the matrix, A represents the mask attention map generated by the mask guidance module. h Representing intermediate features, for the matrix Perform weighted calculations, concatenate the H values, and then perform linear projection to output multi-head attention:

[0035] M-MSA(X) = (Concat(head) h ))W+f p (V)

[0036] in f is a learnable parameter. p (·) is the function that generates the position embedding, which consists of two Conv3×3 convolutional layers and a GeLU activation function. The output is resized to obtain the output features of the attention layer.

[0037] Preferably, the motion compensation enhancement branch includes a multi-hypothesis prediction module and a residual reconstruction module;

[0038] The initial reconstruction data is input into the multi-hypothesis prediction module and outputs a prediction frame; the prediction frame is modulated by a mask matrix, and the difference between the prediction frame and the original measurement value is obtained to obtain the measurement residual value; the measurement residual value is input into the residual reconstruction module and outputs the residual reconstruction value; the residual reconstruction value is summed with the prediction frame and outputs the motion compensation value.

[0039] The multi-hypothesis prediction module includes a first convolutional layer, a residual feature extraction module, a second convolutional layer, a ReLU layer, a third convolutional layer, a fourth convolutional layer, a Sigmoid layer, and a fifth convolutional layer.

[0040] The image data is input into the first convolutional layer to obtain the first inter-frame features;

[0041] The first inter-frame features are input into two residual feature extraction modules to obtain depth features;

[0042] The deep features are input into the second convolutional layer, the ReLU layer, and the third convolutional layer to obtain the hypothesis set;

[0043] The depth features are input into the fourth convolutional layer and the sigmoid layer to obtain the hypothetical weights;

[0044] The hypothesis weights are then multiplied by the hypothesis set and input into the fifth convolutional layer to obtain the prediction frame.

[0045] The residual feature extraction module includes a first convolutional layer, a ReLU layer, and a second convolutional layer.

[0046] The first inter-frame features are sequentially input into the first convolutional layer, the ReLU layer, and the second convolutional layer, and summed with the first inter-frame features to obtain residual features.

[0047] In one embodiment, the residual reconstruction module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer.

[0048] The measured residual values ​​are sequentially input into the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer to obtain the first residual reconstruction value;

[0049] The first residual reconstruction value is summed with the predicted frame to obtain the residual reconstruction value.

[0050] Preferably, the expression for the multi-hypothesis motion compensation enhanced Transformer reconstruction network model is:

[0051] X = F(X) 0 output )

[0052] Where F(·) is the Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and X 0 output X represents the initial reconstruction data, and X represents the high-quality reconstructed video.

[0053] In another aspect of the present invention, in order to achieve the above-mentioned objective, a video compression snapshot imaging reconstruction system with multi-hypothesis motion compensation enhancement is disclosed, comprising:

[0054] Video compression module: used to receive multiple consecutive video frames, compress and encode the multiple consecutive video frames using a mask matrix, and obtain a single-frame snapshot measurement;

[0055] Initial Reconstruction Module: Used to calculate the initial reconstruction data from single-frame snapshot measurements;

[0056] Video Reconstruction Module: This module takes the initial reconstruction data and feeds it into a pre-built Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and outputs a high-quality reconstructed video.

[0057] The beneficial effects of this invention are:

[0058] This invention compresses high-speed video snapshots into single-frame measurements; performs initial reconstruction calculations on these measurements; inputs the initial reconstruction values ​​into a trained multi-hypothesis motion compensation enhanced Transformer reconstruction network, which is then cascaded through multiple stages. In each stage, the overall reconstruction branch uses a mask-guided multi-scale Transformer encoder-decoder network to extract depth features and output the overall reconstruction result; the motion compensation enhancement branch uses a multi-hypothesis prediction module to estimate motion on a reference frame and generate a prediction frame; the residual between the mask-modulated prediction frame and the original measurements is calculated, and the residual reconstruction module reconstructs the calculated residual; through skip connections, the reconstruction information is used to optimize the prediction frame result to obtain motion compensation values; the motion compensation values ​​are summed with the overall reconstruction result to obtain the reconstruction result for the current stage; and the output of the previous stage is used as the input for the next stage to achieve cascaded calculations, ultimately outputting a high-quality reconstructed video, effectively improving both reconstruction quality and efficiency. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic flowchart of the method of the present invention;

[0061] Figure 2 This is a schematic diagram of the structure of a Transformer reconstruction network model with multiple assumptions motion compensation enhancement in one embodiment.

[0062] Figure 3 This is a schematic diagram of the M-Trans module in one embodiment;

[0063] Figure 4 This is a schematic diagram of the structure of the M-MSA module in one embodiment;

[0064] Figure 5 This is a schematic diagram of the structure of a multi-hypothesis prediction module in one embodiment;

[0065] Figure 6 This is a schematic diagram of the residual reconstruction module in one embodiment;

[0066] Figure 7 This is a schematic diagram of the reconstruction results on the simulation dataset;

[0067] Figure 8 This is a schematic diagram of the reconstruction results on a real dataset.

[0068] Figure 9 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] like Figure 1 As shown, a video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement is presented, comprising the following steps:

[0071] Receive multiple consecutive video frames, compress and encode the multiple consecutive video frames using a mask matrix, and obtain a single-frame snapshot measurement;

[0072] Calculate the initial reconstruction data from the single-frame snapshot measurement;

[0073] The initial reconstruction data is input into a pre-established Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and the high-quality reconstructed video is output.

[0074] The training method for the multi-hypothesis motion-compensated enhanced Transformer reconstruction network model is as follows: Video frames with a resolution of 480×894 from 90 different scenes in the DAVIS2017 dataset are cropped to obtain 13,000 training data pairs in 8×256×256 data cubes, which serve as the training set. A shifted binary random mask is used. Eight consecutive frames were modulated and compressed into single-frame measurements to serve as the training set. Kobe, Traffic, Runner, Drop, Crash, and Aerial were selected as the simulation test set, with video segment sizes of 256×256. Meanwhile, Wheel data captured by an SCI camera was used as the real test set.

[0075] A Transformer reconstruction network model with multi-hypothesis motion compensation enhancement is constructed. The model consists of cascaded reconstruction stages, each containing an overall reconstruction branch and a motion compensation enhancement branch. The overall reconstruction branch extracts deep features from a mask-guided multi-scale Transformer encoder-decoder network to obtain the overall reconstruction. The motion compensation enhancement branch includes a multi-hypothesis prediction module and a residual reconstruction module to generate motion compensation values. The overall reconstruction and the motion compensation values ​​are summed to obtain the reconstruction result of the current stage, which is then input into the next cascaded stage.

[0076] A loss function is used to optimize the Transformer reconstruction network model with multiple hypotheses motion compensation enhancement. The training hyperparameters of the Transformer reconstruction network model with multiple hypotheses motion compensation enhancement are preset. Image data from the training set is input into the Transformer reconstruction network model with multiple hypotheses motion compensation enhancement. Based on the loss function, the Adam optimizer is used to iteratively optimize and update the model parameters until the network model converges, thus obtaining the Transformer reconstruction network model with multiple hypotheses motion compensation enhancement. Image data from simulation and real test sets are respectively input into the Transformer reconstruction network model with multiple hypotheses motion compensation enhancement for testing, obtaining the reconstructed video frame sequence.

[0077] First, video frames with a resolution of 480×894 were selected from 90 different scenes in the DAVIS2017 dataset and cropped to obtain 13,000 training data pairs in 8×256×256 data cubes as the training set. Then, eight consecutive frames were modulated using a shifted binary random mask and compressed to a single frame measurement value as the training set.

[0078] The aforementioned multi-hypothesis motion compensation enhanced video compression snapshot imaging reconstruction method modulates video frames using a mask matrix and integrates them to obtain measured image data for a single frame. The measured values ​​of the single frame are then used for initial reconstruction calculation. This initial reconstructed image data is input into a trained multi-hypothesis motion compensation enhanced Transformer reconstruction network model, which outputs the reconstructed original video sequence. The trained multi-hypothesis motion compensation enhanced Transformer reconstruction network model consists of multiple cascaded reconstruction stages. Each stage's overall reconstruction branch utilizes a mask-guided multi-scale Transformer network for overall reconstruction, while the motion compensation enhancement branch uses a multi-hypothesis prediction module and a residual reconstruction module to generate motion compensation values. These motion compensation values ​​are summed with the overall reconstruction to obtain the reconstruction result for the current stage, which is then input into the next stage for cascaded operation, thereby improving reconstruction quality and outputting a high-quality reconstructed video in the final stage.

[0079] like Figure 2 As shown, in one embodiment, the overall reconstruction branch includes an encoding operation, an M-Trans module, a downsampling operation, an upsampling operation, a first convolutional layer, a second convolutional layer, and a decoding operation;

[0080] The image data is input and encoded, and then the M-Trans module outputs high-dimensional features. Then, the input M-Trans module outputs the first layer of depth features. The first layer depth feature is input for downsampling operation, and the M-trans module outputs the second layer depth feature. The second layer depth feature input is downsampled, and the M-trans module outputs the third layer depth feature. The third-layer depth feature input upsampling operation and the second-layer depth feature concatenation operation are then passed sequentially through the first convolutional layer and the M-Trans module to output the second-layer output feature. The second layer output features, after being upsampled, are concatenated with the first layer depth features, and then sequentially passed through the second convolutional layer, the M-Trans module, and the decoding operation to output depth features X0'. The depth features are then summed with the image data to obtain the overall reconstruction result X. re1 .

[0081] like Figure 3 As shown, in one embodiment, the M-Trans module includes two multi-head attention computation modules, each module including a layer normalization layer, a mask-guided multi-head self-attention layer, a layer normalization layer, and a multi-layer perceptron layer;

[0082] The high-dimensional features are sequentially passed through the layer normalization layer and the mask-guided multi-head self-attention layer, and summed with the high-dimensional features to output attention features. The attention features are then sequentially input into the layer normalization layer and the multilayer perceptron layer, and summed with the attention features to output mask attention features.

[0083] The masked attention features are sequentially passed through the layer normalization layer and the mask-guided multi-head self-attention layer, and then summed with the masked attention features to output the attention features. These attention features are then sequentially input into the layer normalization layer and the multilayer perceptron layer, and summed with the attention features to output the final masked attention features. The calculation is as follows:

[0084] X M-MSA l =M-MSA(LN(X) l-1 ))+X l-1

[0085]

[0086] Where X l-1 The features are represented by downsampling, LN(·) represents layer normalization, M-MSA(·) represents a mask-guided multi-head self-attention layer, MLP(·) represents multi-layer perceptron processing, and X represents the features after downsampling. l This represents the computational output feature of a multi-head attention computation module within the M-Trans module;

[0087] like Figure 4As shown, in one embodiment, the mask-guided multi-head self-attention layer includes an encoding layer, a projection matrix, and a mask guidance module;

[0088] The input features are reshaped by the encoding layer and linearly projected into a query matrix Q, a key matrix K, and a value matrix. The calculation formula is as follows:

[0089] Q = XW q K = XW k V = XW v

[0090] in Let represent the learnable projection matrices, and let H represent the number of heads in the multi-head attention mechanism. The dimension of each head is represented as... In the head h Attention calculation in (h=1,...,H) is shown below:

[0091]

[0092] head h =(M a h ⊙V h A h

[0093] in Represents the learnable parameters. K represents h The transpose of the matrix, A represents the mask attention map generated by the mask guidance module. h Representing intermediate features, for the matrix Perform weighted calculations, concatenate the H results, and then perform linear projection to output multi-head attention:

[0094] M-MSA(X) = (Concat(head) h ))W+f p (V)

[0095] in f is a learnable parameter. p (·) is the function that generates the position embedding, which consists of two Conv3×3 convolutional layers and a GeLU activation function. The output is resized to obtain the output features of the attention layer.

[0096] like Figure 4 As shown, in one embodiment, the mask guidance module includes a downsampling operation, a first convolutional layer, a second convolutional layer, a ReLU layer, a third convolutional layer, and a Sigmoid activation layer;

[0097] The mask is downsampled in the same way as the feature map and then input into the first convolutional layer to obtain the first feature map; the first feature map is input into the second convolutional layer, the ReLU layer, the third convolutional layer and the Sigmoid activation layer to obtain the second feature map; the second feature map and the first feature map are inner products and summed with the first feature map to output the mask attention map.

[0098] like Figure 2 As shown, in one embodiment, the motion compensation enhancement branch includes a multi-hypothesis prediction module and a residual reconstruction module;

[0099] The image data is input into the multi-hypothesis prediction module, which then outputs a prediction frame. The predicted frame The measurement residual value is obtained by modulating the mask matrix and subtracting it from the original measurement value. The measured residual value is input into the residual reconstruction module, which outputs the residual reconstruction value; the residual reconstruction value is summed with the predicted frame to output the motion compensation value.

[0100] like Figure 5 As shown, in one embodiment, the multi-hypothesis prediction module includes a first convolutional layer, a residual feature extraction module, a second convolutional layer, a ReLU layer, a third convolutional layer, a fourth convolutional layer, a Sigmoid layer, and a fifth convolutional layer;

[0101] The image data is input into the first convolutional layer to obtain the first inter-frame features; the first inter-frame features are then input into two residual feature extraction modules to obtain depth features. The deep features are input into the second convolutional layer, the ReLU layer, and the third convolutional layer to obtain the hypothesis set. The depth features are input into the fourth convolutional layer and the Sigmoid layer to obtain hypothesis weights. The hypothesis weights are then multiplied by the hypothesis set, and the result is fed into the fifth convolutional layer to obtain the prediction frame.

[0102] The residual feature extraction module includes a first convolutional layer, a ReLU layer, and a second convolutional layer. The first inter-frame features are sequentially input into the first convolutional layer, the ReLU layer, and the second convolutional layer, and summed with the first inter-frame features to obtain the residual features.

[0103] like Figure 6 As shown, in one embodiment, the residual reconstruction module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer.

[0104] The measurement residual value The first residual reconstruction value is obtained by sequentially inputting the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer; the first residual reconstruction value is then compared with the predicted frame. Sum the results to obtain the reconstructed residual values.

[0105] In one embodiment, the loss function is:

[0106]

[0107] Where X output,k To reconstruct the final output of the k-th frame of the video in the network model, X k This is the original video, where B represents the total number of video frames, nx represents the video length, and ny represents the video width.

[0108] In one embodiment, the images in the training set are compressed into single-frame measurements using a mask, an initial reconstruction of the single-frame measurements is calculated, and the initial reconstructed image data is input into the multi-hypothesis motion-compensated enhanced Transformer reconstruction network model. Based on the loss function, the Adam optimizer is used to iteratively optimize and update the weight parameters of the network model until the multi-hypothesis motion-compensated enhanced Transformer reconstruction network model converges, obtaining the network model. The steps include:

[0109] Select m image data {X1,...,X} from the training set m}, calculate the corresponding compression measurement Y i =ΦX i i = 1, 2, ..., m;

[0110] Calculate the initial reconstruction X 0 output The formula is:

[0111]

[0112] in Represents matrix dot division. This represents the mask after summing along the time dimension.

[0113] The calculated data is fed into a Transformer reconstruction network model enhanced with multi-hypothesis motion compensation to update the network model's weight parameters. Where Adam is the Adam optimizer, t is the time step initialized to 0, and θ t+1 Let θ be the weight parameter at time step t+1. t Here, represents the weight parameters at time step t, and α represents the learning rate of the Transformer reconstruction network model with multi-hypothesis motion compensation enhancement. The gradient of the weight parameters at time step t;

[0114] The training process continues until the multi-hypothesis motion-compensated enhanced Transformer reconstruction network model converges. Training is also complete if the preset number of iterations is reached, even if the preset convergence condition has not yet been met. The preset number of iterations can be set according to actual conditions, such as epoch = 100 iterations.

[0115] In one embodiment, the expression for the multi-hypothesis motion-compensated enhanced Transformer reconstruction network model is:

[0116] X = F(X) 0 output )

[0117] Where F(·) is the Transformer reconstruction network model with multiple hypothesis motion compensation enhancement, and X is the final reconstruction result.

[0118] In one embodiment, image data from the test set is input into a multi-hypothesis motion-compensated enhanced Transformer reconstruction network model for testing, including:

[0119] The image data in the test set is modulated using a mask matrix; the initial reconstruction of the snapshot measurement is calculated; the initial reconstructed image data is imported into a trained Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, the reconstructed video is output, and its PSNR and SSIM values ​​are calculated based on the original data.

[0120] To demonstrate the effectiveness of this invention, comparative and ablation experiments were conducted. First, the dataset and training details are introduced. Then, comparative experimental results of different algorithms on the dataset are presented. Finally, a series of ablation experiments were implemented to evaluate the effectiveness of the mask guidance module and the motion compensation enhancement module.

[0121] The model was trained on the training set and tested on the test set. The parameters were set as follows: α = 0.0002, and the learning rate was reduced by 10% every 10 iterations. The epoch count was 100, and the number of cascaded stages was n = 4. Quantitative analysis methods were used to evaluate the experiment.

[0122] Experiments on the test set yielded final reconstructed average PSNR and SSIM values ​​of 32.92 dB and 0.946, respectively.

[0123] The present invention was compared with several representative methods, and the comparative experimental results are shown in Table 1. The present invention achieved the best performance on the Runner, Kobe, and Traffic datasets, as well as on the average reconstructed PSNR values ​​for each dataset. Traditional methods based on iterative optimization, such as the GAP-TV algorithm, GMM-TP algorithm, and MMLE-MFA algorithm, generally performed worse than deep learning-based methods in terms of reconstruction results. MMLE-GMM can improve the video frame reconstruction quality by adaptively learning the parameters in the model. PnP-FFDNet, E2E-CNN, Saidini's algorithms, and RE2-Net, as deep learning methods, showed some improvement in reconstruction quality. RE2-Net effectively utilized the inter-frame spatiotemporal correlation to represent complex video scenes through complementary auxiliary computation between various sub-networks, achieving the highest reconstructed PSNR value on the Crash dataset, and also achieving optimal results on the Drop dataset, which has many static scenes. This invention focuses on using a multi-hypothesis motion compensation structure to represent the spatiotemporal correlation of moving targets between video frames. By enhancing the mining and utilization of their spatiotemporal correlation through motion compensation, the overall reconstruction quality is improved. Therefore, it achieves the highest PSNR values ​​on the Kobe, Runner, and Traffic datasets and performs best in the average reconstruction PSNR values ​​of each dataset.

[0124] Table 1

[0125]

[0126] Similarly, ablation experiments were conducted on the test set to verify their effectiveness by removing the mask guidance mechanism and motion compensation structure. The results of the ablation experiments are shown in Table 2.

[0127] Table 2

[0128]

[0129] As can be seen from Table 1, compared with existing deep learning-based reconstruction methods, our proposed method can significantly improve reconstruction quality.

[0130] As can be seen from Table 2, the motion compensation structure plays a key role in improving the quality of video reconstruction. At the same time, the mask guidance mechanism enables the Transformer network to perform more targeted self-attention calculations, thereby improving the overall reconstruction quality.

[0131] like Figure 7 The image shows the visualized prediction results of the reconstruction on a simulation test dataset. Based on the reconstruction results, it can be seen that this method can effectively reconstruct objects in fast-moving scenes, and it significantly improves upon other algorithms in reconstructing digital edges and human contours.

[0132] like Figure 8 As shown, this is a visualization of the reconstruction results on a real dataset. It can be seen that the network model of this patent has clearer reconstruction results, smoother reconstructed video frames, and more accurate reconstruction of letter outlines and moving wheel edges.

[0133] On the other hand, such as Figure 9 As shown, this embodiment of the invention discloses a video compression snapshot imaging reconstruction system with multi-hypothesis motion compensation enhancement, comprising:

[0134] Video compression module: used to receive multiple consecutive video frames, compress and encode the multiple consecutive video frames using a mask matrix, and obtain a single-frame snapshot measurement;

[0135] Initial Reconstruction Module: Used to calculate the initial reconstruction data from single-frame snapshot measurements;

[0136] Video Reconstruction Module: This module takes the initial reconstruction data and feeds it into a pre-built Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and outputs a high-quality reconstructed video.

[0137] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0138] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0139] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0140] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement, characterized in that, The method includes the following steps: Receive multiple consecutive video frames, compress and encode the multiple consecutive video frames using a mask matrix, and obtain a single-frame snapshot measurement; Calculate the initial reconstruction data from the single-frame snapshot measurement; The initial reconstruction data is input into a pre-established Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and the high-quality reconstructed video is output. The multi-hypothesis motion compensation enhanced Transformer reconstruction network model includes multiple cascaded stages; The cascade phase includes an overall reconstruction branch and a motion compensation enhancement branch; The initial reconstruction data is input into the overall reconstruction branch to obtain the overall reconstruction result, and the initial reconstruction data is input into the motion compensation enhancement branch to obtain the motion compensation value; The overall reconstruction result is summed with the motion compensation value to output the reconstructed high-quality video; The motion compensation enhancement branch includes a multi-hypothesis prediction module and a residual reconstruction module; The initial reconstruction data is input into the multi-hypothesis prediction module, which then outputs a prediction frame. The predicted frame is modulated by a mask matrix, and the difference between the predicted frame and the original measurement value is used to obtain the measurement residual value; the measurement residual value is input into the residual reconstruction module to output the residual reconstruction value; the residual reconstruction value is summed with the predicted frame to output the motion compensation value; The multi-hypothesis prediction module includes a first convolutional layer, a residual feature extraction module, a second convolutional layer, a ReLU layer, a third convolutional layer, a fourth convolutional layer, a Sigmoid layer, and a fifth convolutional layer. The initial reconstructed data is input into the first convolutional layer to obtain the first inter-frame features; The first inter-frame features are input into two residual feature extraction modules to obtain depth features; The deep features are input into the second convolutional layer, the ReLU layer, and the third convolutional layer to obtain the hypothesis set; The depth features are input into the fourth convolutional layer and the sigmoid layer to obtain the hypothetical weights; The hypothesis weights are then multiplied by the hypothesis set and input into the fifth convolutional layer to obtain the prediction frame. The residual feature extraction module includes a first convolutional layer, a ReLU layer, and a second convolutional layer; The first inter-frame features are sequentially input into the first convolutional layer, the ReLU layer, and the second convolutional layer, and summed with the first inter-frame features to obtain residual features; The residual reconstruction module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer; The measured residual values ​​are sequentially input into the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer to obtain the first residual reconstruction value; The motion compensation value is obtained by summing the first residual reconstruction value with the predicted frame.

2. The video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement according to claim 1, characterized in that, The calculation process for the initial reconstruction data is as follows: in, The initial reconstruction data obtained from the calculation, , y This indicates compressed measurement data. Represents matrix dot division. Represents matrix dot product. This represents the mask after summing along the time dimension. Represents the mask matrix.

3. The video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement according to claim 1, characterized in that, The overall reconstruction branch includes encoding operations, M-Trans modules, downsampling operations, upsampling operations, a first convolutional layer, a second convolutional layer, and decoding operations; After the initial reconstructed data is input into the encoding operation to output high-dimensional features, the high-dimensional features are input into the M-Trans module to output the first layer of depth features; The first layer depth features are input into the downsampling operation, and the M-trans module outputs the second layer depth features. The second layer depth features are input into the downsampling operation, and the M-trans module outputs the third layer depth features. The third layer depth features are input to the upsampling operation, and after the second layer depth features are concatenated, they are sequentially passed through the first convolutional layer, the M-Trans module, and the upsampling operation to output the second layer output features. After performing a concatenation operation between the second layer output features and the first layer depth features, the depth features are sequentially passed through the second convolutional layer, the M-Trans module, and the decoding operation to output the depth features. The depth features are summed with the initial reconstruction data to obtain the overall reconstruction result.

4. The video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement according to claim 3, characterized in that, The M-Trans module includes two multi-head attention computation modules, each comprising a layer normalization layer, a mask-guided multi-head self-attention layer, a layer normalization layer, and a multilayer perceptron layer. The high-dimensional features are passed sequentially through the layer normalization layer and the mask-guided multi-head self-attention layer, and summed with the high-dimensional features to output attention features. The attention features are then sequentially input into the layer normalization layer and the multilayer perceptron layer, and summed with the attention features to output mask attention features. The masked attention features are sequentially passed through the layer normalization layer and the mask-guided multi-head self-attention layer, and then summed with the masked attention features to output the attention features. These attention features are then sequentially input into the layer normalization layer and the multilayer perceptron layer, and summed with the attention features to output the current layer depth features. The calculation process in a multi-head attention calculation module is as follows: in This indicates features that have undergone downsampling. Representative layer normalization processing, This represents a mask-guided multi-head self-attention layer. This indicates multi-layer perceptron processing. This represents the computational output feature of a multi-head attention computation module within the M-Trans module; The mask-guided multi-head self-attention layer includes an encoding layer, a projection matrix, and a mask guidance module; Input features X The encoding layer performs a size transformation, reshapes the matrix, and projects it linearly into a query matrix. Key matrix Sum matrix , The calculation formula is as follows: in Let represent the learnable projection matrices, and let . The number of heads representing multi-head attention, with each head dimension represented as... , in the head The attention calculation in the code is as follows: in Represents the learnable parameters. express The transpose of the matrix, This represents the mask attention map generated by the mask guidance module. Representing intermediate features, for the matrix Perform weighted calculation and The results are processed Concat Perform operations and linear projection to output multi-head attention: in For learnable parameters, The function for generating positional embeddings consists of two... The convolutional layer and GeLU activation function are used to resize the output to obtain the output features of the attention layer.

5. The video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement according to claim 1, characterized in that, The expression for the multi-hypothesis motion compensation enhanced Transformer reconstruction network model is as follows: in, For a Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, X represents the initial reconstruction data, and X represents the high-quality reconstructed video.

6. A video compression snapshot imaging reconstruction system with multi-hypothesis motion compensation enhancement, employing the video compression snapshot imaging reconstruction method with multi-hypothesis motion compensation enhancement as described in any one of claims 1 to 5, characterized in that, include: Video compression module: used to receive multiple consecutive video frames, compress and encode the multiple consecutive video frames using a mask matrix, and obtain a single-frame snapshot measurement; Initial Reconstruction Module: Used to calculate the initial reconstruction data from single-frame snapshot measurements; Video Reconstruction Module: This module takes the initial reconstruction data and feeds it into a pre-built Transformer reconstruction network model with multi-hypothesis motion compensation enhancement, and outputs a high-quality reconstructed video.

Images