A method for compressed sensing reconstruction of color video snapshots based on quad-Bayer array

Abstract

The present invention discloses a method for compressed sensing reconstruction of color video snapshots based on a quad-Bayer array, comprising the following steps: A. obtaining a preview quad-Bayer video and a quad-Bayer sampling value associated with the preview image, and obtaining a quad-Bayer sampling measurement value associated with the preview video through a given mask, wherein the sampling value is obtained according to a compression ratio selected by a user, and the preview image is obtained by scaling the original video according to the compression ratio. The present invention proposes the first color video SCI reconstruction technology based on a quad-Bayer array. Compared with traditional reconstruction methods, this method can significantly reduce artifacts and color distortion and achieve high-quality color video reconstruction. This technology effectively overcomes the difficulties in traditional methods by optimizing algorithms and reconstruction processes, and provides excellent detail performance, color restoration and dynamic effects, which not only improves video quality, but also brings new breakthroughs in the field of compressed sampling and reconstruction of color video signals.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and signal transmission, and in particular to a method for color video snapshot compression sensing reconstruction based on a quad-Bayer array. Background Art

[0002] Video compression sensing is a mathematical algorithm in the field of information processing. It points out that when the signal is sparse, the original signal can be efficiently reconstructed by sampling only less data. Due to its advantages such as high acquisition efficiency and low storage requirements, video compression sensing is widely used in video compression, medical imaging and other fields. Specifically, given the original video signal, its sampling value can be directly obtained by a random linear sampling. Since the linear sampling matrix integrates signal acquisition and signal compression, the sampling process becomes simple and has a high sampling efficiency. At the same time, since the scale of the sampling value is much smaller than the original signal, compression sensing also has the advantages of fast sampling speed and small storage requirements. But at the same time, the limited sampling scale makes image reconstruction an extremely pathological problem.

[0003] Since restoring the original signal from restricted sampling values ​​is essentially an underdetermined set of equations, existing snapshot compressed sensing algorithms often introduce additional prior information to constrain the solution space. Common priors include sparse prior, low-rank prior and deep prior. At the same time, the compressed sensing reconstruction problem with priors can be solved by many nonlinear iterative algorithms. Several popular iterative algorithms include iterative shrinkage-thresholding algorithm (ISTA), alternating direction method of multipliers (ADMM) and approximate message passing (AMP). These algorithms have good interpretability and reliable mathematical convergence proofs, but their optimization-based iterative method is very time-consuming and cannot meet the needs of practical applications.

[0004] With the rise of deep learning, data-driven deep models can reconstruct the sampled values ​​into the original signal using a single forward propagation. Therefore, some researchers have proposed end-to-end networks as an emerging concept in the field of deep learning. These models abandon the complexity of traditional iterative algorithms and directly learn the direct mapping from input to output through deep networks. The design of end-to-end networks usually does not include intermediate steps in the iterative process, but directly optimizes the entire process from raw data to final results. Through the learning ability of deep networks, end-to-end networks can automatically capture the complex relationship between input and output, thereby achieving efficient signal reconstruction or task processing without manually setting hyperparameters. However, due to the end-to-end network trial Figure 1 The entire task is processed in one go, and the model complexity is often high. This design may lead to a significant increase in the number of model parameters, making it difficult to deploy and run the model on lightweight devices such as mobile terminals. Nevertheless, with the continuous advancement of deep learning technology and the development of model compression technology, the potential of end-to-end networks in lightweight applications is still worthy of further exploration and research.

[0005] As people's requirements for smartphone cameras have become higher and higher in recent years, the development of smartphone camera sensors has accelerated. As one of the improvement trends, the quad-Bayer color filter array (CFA) pattern achieves a good trade-off between pixel size and sensor sensitivity, thereby minimizing the decrease in sensor sensitivity when the pixels are small. Compared with the traditional Bayer array pattern (Bayer CFA pattern), the quad-Bayer pattern can prevent resolution loss in low light conditions, thereby obtaining high-quality images with less noise. Due to these advantages, more and more people have become mobile photographers and use cameras with quad-Bayer for video shooting. Therefore, most of the videos nowadays are in quad-Bayer format.

[0006] State Space Models (SSMs), represented by the Mamba model, are generally considered to be time-invariant systems that map a one-dimensional function or sequence x(t) to y(t) through a hidden state h(t). The advantage of Mamba is that it can model global attention with a linear complexity related to the size of the input image, thereby achieving efficient modeling with low parameters and computational complexity.

[0007] These end-to-end network structures have many problems: (1) Existing end-to-end (E2E) methods are all designed based on the Bayer format, and cannot adapt to the current quad-Bayer format videos, which will cause corresponding color distortion and greatly affect the reconstruction quality, and thus cannot achieve corresponding high-quality reconstruction; (2) Existing E2E methods often require large computational complexity and parameter quantity, and the E2E method divides the video into spatial and temporal dimensions, and after performing self-attention calculations in the two dimensions, the features are fused through a residual network, which may cause a lack of spatiotemporal consistency, resulting in the inability to achieve high-quality reconstruction. Summary of the invention

[0008] The object of the present invention is to provide a method for color video snapshot compressed sensing reconstruction based on a quad-Bayer array to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: a method for color video snapshot compressed sensing reconstruction based on a quad-Bayer array, comprising the following steps:

[0010] A. obtaining a preview four-Bayer video and four-Bayer sampling values ​​associated with the preview image, and obtaining four-Bayer sampling measurement values ​​associated with the preview video through a given mask, wherein the sampling values ​​are obtained according to a compression ratio selected by a user, and the preview image is obtained by scaling the original video according to the compression ratio;

[0011] B. Input the preview measurement value and the predefined mask into the video snapshot compressed sensing reconstruction model of the deep neural network to calculate the output reconstructed video, wherein the deep neural network includes: initialization of the reconstruction initial value, a shallow feature extraction layer, the extraction layer is composed of a deep separable convolution layer; a deep feature encoding layer, the encoding layer includes a convolution, a linear layer, a linear attention module and a downsampling layer; the bottleneck layer includes a convolution, a linear layer and a linear attention module; a deep feature decoding layer, the decoding layer includes a convolution layer, a residual connection and an upsampling layer; a video reconstruction layer, the reconstruction layer includes multiple convolution layers;

[0012] C. After the deep neural network model training is completed, the parameters of the trained model are obtained to obtain an end-to-end video snapshot compressed sensing reconstruction model;

[0013] D. Input the preview image into the deep neural network model to calculate the output reconstructed video.

[0014] Preferably, the step A is composed of a T-frame color video, denoted as x∈R H×W×C×T , where H, W, C, and T represent the height, width, number of channels, and number of frames of the color video, respectively, through the mask, denoted as Φ∈RH×W×C×T , where R represents a real number set. The encoded color video is acquired through a quad-Bayer array camera to obtain a two-dimensional quad-Bayer-based measurement value, which is recorded as a matrix y∈R H×W , the sampling value is obtained by a one-time random sampling, and the sampling value y is calculated as follows: y = Φx, where Φ represents the linear sampling matrix, Φ∈R m×n , x represents the original image, x∈R n , the obtained y and the predefined Φ are input into the reconstruction algorithm to obtain the final reconstructed color image

[0015] Preferably, the objective function of the reconstruction model of the video snapshot compressed sensing is calculated as follows: In the formula, Represents the currently obtained reconstructed image.

[0016] Preferably, the deep neural network model training includes: calculating the loss value of the deep neural network through a loss function, which is used to update the model parameters of the video snapshot compressed sensing reconstruction model during the model iterative training process, and stopping the training until the loss value after training is less than a set iteration threshold.

[0017] Preferably, the loss function L rec The calculation is as follows: In the formula, represents the reconstructed image, x is the corresponding original image, N represents the number of elements in the training set, Represents the difference between the reconstructed image and the original image.

[0018] Preferably, the number of the depth feature encoding layers and the number of the depth feature decoding layers are both three.

[0019] Preferably, the number of Mamba modules in the deep feature coding layer is two, three, or four, respectively.

[0020] Preferably, the number of Mamba modules in the deep feature encoding layer is six, and the number of Mamba modules in the bottleneck layer is six.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention proposes the first color video SCI reconstruction technology based on quad-Bayer array. Compared with the traditional reconstruction method, this method can significantly reduce artifacts and color distortion and achieve high-quality color video reconstruction. This technology effectively overcomes the difficulties in traditional methods by optimizing algorithms and reconstruction processes, and provides excellent detail performance, color restoration and dynamic effects, which not only improves the video quality, but also brings new breakthroughs in the field of compressed sampling and reconstruction of color video signals. The mamba model is used as the linear attention mechanism. Compared with the existing E2E method, the present invention only requires 31% of its parameters and 4.5% of the floating-point operations per second (FLOPS), but can achieve more than 3d in peak signal-to-noise ratio (PSNR). B's performance is improved. This efficient balance between computational complexity and reconstruction quality makes the present invention more practical and valuable for promotion in terms of lightweight. Compared with the existing model-based method and PnP method, the present invention does not need to be post-processed with a demosaicing algorithm during color video reconstruction. This innovative technology not only simplifies the algorithm flow, but also greatly improves the reasoning speed, making the present invention more real-time and efficient in practical applications. Since the demosaicing algorithm is removed, the reconstruction result of the present invention is more accurate and stable, further improving the quality of the image or video. This color reconstruction technology without the need for a demosaicing algorithm enables the present invention to complete the reconstruction of high-quality images more quickly and meet the needs of more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart showing the differences between the present invention and the previous method;

[0024] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The existing E2E methods are all designed based on the Bayer format, and cannot adapt to the videos shot in the current quad-Bayer format, which will cause corresponding color distortion, greatly affecting the reconstruction quality, and thus cannot achieve corresponding high-quality reconstruction. At the same time, the existing E2E methods all use self-attention models (transformer) and convolutional neural networks (CNN), which often require large computational complexity and parameter quantity. In addition, the E2E method divides the video into spatial and temporal dimensions, and fuses the features through a residual network after performing self-attention calculations in the two dimensions, which may cause a lack of spatiotemporal consistency, resulting in the inability to achieve high-quality reconstruction. Therefore, the current E2E method cannot achieve efficient and high-quality reconstruction based on quad-Bayer.

[0027] Reference Figure 1 to Figure 2 In some embodiments, a method for color video snapshot compressed sensing reconstruction based on a quad-Bayer array according to the present invention comprises at least the following steps:

[0028] A. obtaining a preview quad-Bayer video and quad-Bayer sampling values ​​associated with the preview image, wherein the sampling values ​​are obtained according to a compression ratio selected by a user, and the preview image is obtained by scaling the original video according to the compression ratio;

[0029] B. Input the preview image into the image compression sensing reconstruction model of the deep neural network to calculate the output reconstructed image.

[0030] Specific implementation of step A

[0031] See also Figure 1 The reconstruction model of video snapshot compressed sensing of the present invention obtains a reconstruction model for end-to-end video snapshot compressed sensing after model training through a training data set, and the original quad-bayer video can be obtained by the user taking a photo or from an album stored in a mobile device, thereby completing the collection of the original image. After the user obtains the original image and selects the compression ratio, the user modulates the original image through a given mask and according to the compression ratio to generate a preview measurement value, and then associates the measurement with the given mask and stores it in the mobile device, thereby meeting the terminal demand of resource priority. It should be noted that the compression ratio can be set to multiple values ​​such as 1%, 4%, 10%, 50%, etc., and the user can sample and store the original signal according to actual needs.

[0032] Specific implementation of step B

[0033] See also Figure 2The present invention adopts a reconstruction model of video compression and compressed sensing based on a deep neural network, which is much smaller than all current E2E methods in terms of computational complexity and model scale.

[0034] Specifically, a T-frame color video is denoted as x∈R H×W×C×T , where H, W, C, and T represent the height, width, number of channels, and number of frames of the color video, respectively, through the mask, denoted as Φ∈R H×W×C×T , where R represents a real number set. The encoded color video is acquired through a quad-Bayer array camera to obtain a two-dimensional quad-Bayer-based measurement value, which is recorded as a matrix y∈R H×W ,The sampling value is obtained by a one-time random sampling, and the sampling value y is calculated as follows: y = Φx, where Φ represents the linear sampling matrix;

[0035] a) Quad-Bayer mode color video SCI reconstruction process: Input y and Φ, and get the initial reconstructed video X by inputting y and Φ into the initialization module at the same time in ∈R H×W×1×T , where X in represents the original (raw) quad-Bayer video with only 1 channel, and uses it as the input of the proposed reconstruction algorithm MambaSCI and obtains the final reconstructed color video

[0036] X out ∈R H×W×3×T ;

[0037] b) MambaSCI framework: X in First, the feature F∈R is obtained through a shallow feature extraction module composed of deep separable convolution (DWConv) H×W×C×T , where C is the current number of feature channels, and F then passes through three encoder layers, each of which contains a residual-STMamba-Block (RSTMamba) and a downsampling operation to obtain the output feature Where i∈{1, 2, 3} represents three encoder layers connected in sequence, indicating that after passing through each encoder layer, the length and width of the feature are halved and the number of feature channels is doubled, resulting in the final feature in

[0038] at the same time It will pass through a bottleneck layer consisting of multiple RSTMambas, and the spatial dimension and number of channels will not change. After three decoding layers (decoder), it is used to decode the feature map and restore the image to its original resolution and number of channels. The input of each decoder is obtained by adding the output features of the encoder at the same layer to the output features of the previous decoder. Each decoder contains a residual DWConv block and an upsampling operation. The features of each decoder output Finally, the characteristics

[0039] F final ∈R H×W×C×T , and finally F final The final X is obtained through the final reconstruction convolution block (ReConv) out , which is the reconstructed color video. This process can be briefly described as: F = DWConv (X in ), F1=encoder(F)F2=encoder(F1)F3=encoder(F2),

[0040]

[0041]

[0042]

[0043] c) RSTMamba Detailed Design: In order to efficiently capture spatiotemporal consistency and complete high-quality reconstruction of edge details, RSTMamba was customized as the core module of MambaSCI.

[0044] The STMamba, Edge-Detail-Reconstruction (EDR) and Channel Attention (CA) modules are connected by residual connection to complete efficient and high-quality reconstruction. STMamba is used to capture long-term spatiotemporal consistency, EDR is used to better complete edge detail reconstruction, and CA is to make up for the lack of channel information interaction in the Mamba model. The three are connected by residual connection.

[0045] To achieve high-quality reconstruction together, the specific process is as follows: the input feature F first undergoes layer normalization operation (LayerNorm) and then serves as the input of STMamba and obtains new features through residual connection with F with learnable scale, and then passes through LayerNorm and learnable residual EDR module, then passes through a linear layer (Projection), and then passes through LayerNorm and learnable residual CA module to obtain the final features. This process can be briefly described as:

[0046] F1=F*scale1+STMamba(LN(F)),

[0047] F2=F1*scale2+EDR(LN(F1)),F2=Projection(F2),

[0048] F3=F2*scale3+CA(LN(F2)), where LN represents LayerNorm,

[0049] scale1, scale2 and scale3 represent the jump scales of three learnable residual connections respectively;

[0050] d) STMamba detailed process: The input feature F is first divided into two branches. In the first branch, F first passes through a linear layer (Projection) to double the feature channel to obtain X. First, X is expanded along the time dimension T to obtain X. s ∈R T(HW)×C , by X s By forward scanning and reverse scanning, we can obtain the feature X of the spatial dimension respectively. sf and X sb , and X is expanded along the spatial dimension to obtain X t ∈R (HW)T×C And obtain the time dimension features through forward scanning. At the same time, the other branch also doubles the feature channel through Projection to obtain Z, and then multiplies it with the above three features through the ReLU activation function, accumulates it element by element, and passes through Projection.

[0051] After converting the feature channels to their original number, the final features are obtained. This process can be briefly described as follows:

[0052] X=Projection(F)Z=Projection(F),

[0053] X sf =Conv1d(ForwardSSM(X s )),X bf =Conv1d(BorwardSSM(X s )),

[0054] X t =Conv1d(ForwardSSM(X t )),

[0055] F out =Projection(X sf ☉Z+X sb ☉Z+X t ☉Z);

[0056] e) Detailed process of EDR and CA: EDR realizes lightweight and efficient edge detail reconstruction through DWConv and linear layer (Linear), while CA transforms the features into a 1×1×1 shape through average pooling, maps them to the (0, 1) interval through three-dimensional convolution (Conv3d) and sigmoid activation function, and then multiplies them with the original feature map to realize channel attention modeling and make up for the lack of channel information interaction in the Mamba model.

[0057] The deep neural network can be trained end-to-end, given the original image training set The model can be trained using the minimum L2 loss: In the formula, represents the reconstructed video, x is the corresponding original video; N represents the number of elements in the training set; is the square of the Euclidean function of the matrix, which represents the difference between the two images, that is, the difference between the reconstructed image and the original image.

[0058] It should be recognized that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose, the program can be run on a programmed dedicated integrated circuit.

[0059] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context, and the processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes collectively on one or more processors, by hardware, or a combination thereof, where a computer program includes multiple instructions that can be executed by one or more processors.

[0060] Further, the methods may be implemented in and operably connected to any suitable type of computing platform, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, RAM, ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network when such media includes instructions or programs for implementing the above steps in conjunction with a microprocessor or other data processor. The invention herein includes these and other different types of non-transitory computer-readable storage media, and the present invention may also include the computer itself when programmed according to the methods and techniques of the present invention.

[0061] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory, and the output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0062] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for color video snapshot compressed sensing reconstruction based on a quad-Bayer array, characterized in that: The following steps are involved: A. obtaining a preview four-Bayer video and a four-Bayer sampling value associated with the preview image, and obtaining the four-Bayer sampling value associated with the preview video through a given mask, wherein the four-Bayer sampling value is obtained according to a compression ratio selected by a user, and the preview image is obtained by scaling the original video according to the compression ratio; B. Inputting the four-Bayer sampling values ​​and the predefined mask into a video snapshot compressed sensing reconstruction model of a deep neural network to calculate the output reconstructed video, wherein the deep neural network includes: Initialization of the reconstruction initial value, a shallow feature extraction layer, the shallow feature extraction layer is composed of a depth-separable convolutional layer, A deep feature coding layer, wherein the deep feature coding layer includes a convolutional layer, a linear layer, a linear attention module and a downsampling layer, A bottleneck layer, comprising a convolutional layer, a linear layer, and a linear attention module, A deep feature decoding layer, wherein the deep feature decoding layer includes a convolutional layer and a residual connection and an upsampling layer, A video reconstruction layer, wherein the video reconstruction layer includes multiple convolutional layers; C. After the deep neural network model training is completed, the parameters of the trained model are obtained to obtain an end-to-end video snapshot compressed sensing reconstruction model; D. Input the preview image into the deep neural network model to calculate the output reconstructed video; The original video can be obtained by the user taking a photo, thereby completing the collection of the original image, and the original image is recorded as x, x∈R H ×W×C×T , where H, W, C, and T represent the height, width, number of channels, and number of frames of the color video, respectively, and R represents a real number set. The encoded color video is obtained by masking, and the two-dimensional quad-Bayer-based measurement value is obtained by collecting the quad-Bayer array camera, which is recorded as the sampling value y. The sampling value y is obtained by one-time random sampling. The sampling value calculation is as shown in formula (1): y=Φx (1), In formula (1), Φ represents the linear sampling matrix, Φ∈R H×W×C×T , x represents the original image; the obtained y and the predefined Φ are input into the reconstruction algorithm to obtain the reconstructed image The objective function of the reconstruction model of the video snapshot compression sensing is calculated as shown in formula (2): In formula (2), represents the reconstructed image; The deep neural network model training includes: calculating the loss value of the deep neural network through a loss function, which is used to update the model parameters of the reconstruction model of the video snapshot compressed sensing during the model iteration training process, until the loss value after training is less than a set iteration threshold, then the training is stopped; The loss function L rec The calculation is as follows: In formula (3), represents the reconstructed image, x is the corresponding original image, N represents the number of elements in the training set, Represents the difference between the reconstructed image and the original image, and the number of the deep feature encoding layer and the number of the deep feature decoding layer are both three.

Images