Video compressed sensing reconstruction method based on frequency domain separation

Abstract

The invention relates to the technical field of video compression transmission methods, in particular to a video compression sensing reconstruction method based on frequency domain separation, which comprises the following steps: acquiring a video frame sequence and performing compression sensing sampling on each frame of the video frame sequence to obtain a measurement value; obtaining an initial reconstruction result of the current frame through initial reconstruction based on the measurement value; calculating the hidden layer state of the current frame by using a hidden layer state calculation module, wherein the hidden layer state comprises a low-frequency hidden layer state and a high-frequency hidden layer state; and fusing the hidden layer state with the initial reconstruction result to obtain a reconstruction result of the video frame, and independently processing low-frequency and high-frequency components by adopting a frequency domain separation strategy. In low-frequency hidden layer state calculation, inter-frame low-frequency information is globally associated through an attention mechanism, so that the fidelity and stability of the low-frequency information are improved; in high-frequency hidden layer state calculation, optical flow calculation is introduced to carry out alignment and weighting processing on inter-frame high-frequency components, and propagation of high-frequency noise is effectively restrained.

Description

Technical Field

[0001] The present invention relates to the technical field of video compression and transmission methods, and particularly relates to a video compressive sensing reconstruction method based on frequency domain separation. Background Art

[0002] With the development of technologies such as video communication, intelligent monitoring, and unmanned systems, the demand for efficient video compression and high-quality reconstruction is increasing day by day. In traditional video compressive sensing technology, the original video data is sparsely sampled and reconstructed at the decoding end to reduce the transmission and storage costs. However, in the face of high dynamic scenes, complex backgrounds, and low signal-to-noise ratio environments, the reconstruction quality and efficiency of existing methods are severely restricted.

[0003] A video compressive sensing system includes two parts, namely an encoding end and a decoding end. The encoding end can be implemented by a single-pixel camera, which is equivalent to the following formula: y = Φx (1), where Φ is a non-adaptive measurement matrix, the video frame signal to be observed is x ∈ R N ×1 , and the measurement value is y ∈ R M×1 , then the measurement matrix Φ ∈ R M×N , and M << N. M / N is the sampling rate, and this sampling rate is much smaller than the Nyquist sampling rate. It can be seen from formula (1) that sampling and compression are integrated, and the computational complexity of the encoding end is extremely low.

[0004] In order to recover the complete video signal x to be observed from very few sampling samples y, an efficient and accurate compressive sensing reconstruction algorithm is required.

[0005] Reconstruction algorithms can be divided into two categories: 1) traditional algorithms based on optimization, and 2) deep learning methods based on neural networks. Traditional algorithms usually have a high computational complexity because they often require iterative solutions and are currently less used.

[0006] The current mainstream architectures of deep learning methods can also be divided into two categories: 1) networks with window sliding as input, and 2) recurrent neural networks. Networks with window sliding as input currently mainly use recurrent neural networks because of the small visual range in the time dimension. The core principle of recurrent neural networks is to transfer the reconstruction result or hidden state of the current frame to the next frame, thereby realizing the interaction of time-domain information and helping with reconstruction.

[0007] In the prior art, common methods are reconstructions based on sparse representation and optimization algorithms, such as the iterative shrinkage threshold algorithm (ISTA) and block matching methods. These methods usually rely on strong sparse assumptions and recover the original video data by optimizing the loss function. However, these methods have the following problems:

[0008] 1. High noise sensitivity: In low signal-to-noise ratio environments, the optimization algorithm is easily affected by noise when recovering high-frequency information, resulting in blurred or even lost edge details. This phenomenon is particularly evident in dynamic video scenes, resulting in poor temporal consistency of the reconstruction results and a significant decrease in visual quality.

[0009] 2. High computational complexity: Traditional methods, such as iterative methods based on sparse optimization, require a large amount of computing resources to solve nonlinear optimization problems, resulting in low reconstruction efficiency and difficulty meeting the requirements of real-time applications. This problem is particularly prominent in high-resolution video processing scenarios.

[0010] 3. Insufficient Utilization of Inter-Frame Correlation: Existing methods often ignore the temporal correlation between video frames during decoding, reconstructing each frame independently. This lack of temporal correlation fails to fully utilize the motion information between video frames, resulting in low reconstruction quality, especially for moving objects in complex scenes.

[0011] 4. Insufficient frequency domain information fusion: In practical applications, the processing of high-frequency and low-frequency information is often unified, ignoring the different impacts of high-frequency and low-frequency components on reconstruction quality. High-frequency components are crucial for details and edges, while low-frequency components provide the structural foundation of the scene. Existing technologies fail to distinguish and optimize the characteristics of these two components, resulting in the mixing of noise and high-frequency details, limiting reconstruction quality.

[0012] In summary, the reconstruction quality and efficiency of existing methods in high-dynamic, low-SNR environments are difficult to meet the requirements of practical applications. These issues significantly limit the application of video compression sensing technology in resource-constrained scenarios such as smart transportation, drone monitoring, and public safety. Summary of the Invention

[0013] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provide a video compressed sensing reconstruction method based on frequency domain separation, which realizes high-quality reconstruction of the video through frequency domain separation, efficient hidden layer state calculation and the introduction of optical flow information.

[0014] The present invention protects a video compression sensing reconstruction method based on frequency domain separation, comprising:

[0015] Acquire a video frame sequence and perform compressed sensing sampling on each frame of the video frame sequence to obtain a measurement value;

[0016] Based on the measurement value, obtaining an initial reconstruction result of the current frame through initial reconstruction;

[0017] Calculating the hidden layer state of the current frame using a hidden layer state calculation module, wherein the hidden layer state includes a low-frequency hidden layer state and a high-frequency hidden layer state;

[0018] The hidden layer state is fused with the initial reconstruction result to obtain a reconstruction result of the video frame.

[0019] Preferably, the calculation of the low-frequency hidden layer state includes:

[0020] Through the attention mechanism and Fusion is performed to obtain the low-frequency hidden layer component The features of each pixel in It can be calculated by the following formula:

[0021]

[0022] in, for The i-th pixel, for The j-th pixel point, f(·,·) is a binary function mapping, through , where θ(·) and φ(·) are two unary linear functions implemented by convolutional layers, used to calculate the correlation between two points, g(·) is a linear mapping function implemented by a convolutional layer, and C(·) is a normalization function, set to It can be implemented with softmax in the network;

[0023] and The similarity is measured by , where θ(·) and φ(·) are two unary linear functions implemented by the convolutional layer. The convolution kernel size of the convolutional layer is 1×1, the stride is 1, and the padding is 0.

[0024] In the attention mechanism, The linear mapping of is Value, is the Key, For Query.

[0025] Preferably, the low-frequency component is obtained by a linear low-pass filter, and the low-pass filter includes but is not limited to a Gaussian blur filter or other linear filtering methods.

[0026] Preferably, the calculation of the high-frequency hidden layer state includes:

[0027] According to the optical flow F t-1 , through motion compensation Align to The result is calculate and The similarity at each point

[0028]

[0029] in, for and The point product of , p(·) and q(·) are two linear functions, which are implemented by two convolutional layers respectively. Sigmoid(·) is the sigmoid function used to limit the output to [0,1]. As weight, with Perform pixel-by-pixel multiplication, that is, multiply the corresponding position elements to obtain the high-frequency hidden layer components

[0030] Preferably, the fusion method includes: calculating the hidden layer state Depend on and Adding them together we get:

[0031]

[0032] Fusion of the hidden layer state and the current frame reconstruction state, With S t The cascade is then performed, and the fused features are obtained through a convolutional layer and used as the input of the next reconstruction module.

[0033] Preferably, the optical flow information F t-1 Obtained through the SpyNet network or other optical flow estimation algorithms.

[0034] Preferably, the fused hidden layer state is cascaded with the initial reconstruction result of the current frame, processed by a convolution layer, and input into the next reconstruction module to obtain the final reconstruction result of the frame.

[0035] Preferably, the method further comprises:

[0036] The hidden layer states of multiple video frames are calculated and fused in sequence, and processed frame by frame to obtain the reconstruction results of continuous video.

[0037] Preferably, the method is applicable to resource-constrained video sensor network scenarios, including but not limited to smart transportation, public safety monitoring, field exploration, and emergency rescue.

[0038] Specifically, the present invention solves the problems in the prior art by:

[0039] 1. Frequency Domain Separation Strategy: This paper adopts a frequency domain separation strategy, processing low-frequency and high-frequency components independently. In the calculation of low-frequency hidden layer states, an attention mechanism is used to globally correlate low-frequency information between frames, thereby improving the fidelity and stability of low-frequency information. In the calculation of high-frequency hidden layer states, optical flow calculation is introduced to align and weight high-frequency components between frames, effectively suppressing the propagation of high-frequency noise.

[0040] 2. Optical flow estimation and motion compensation: To address the underutilization of temporal information between frames, this paper introduces optical flow estimation and uses motion compensation to align the high-frequency components of the previous frame. This method significantly improves the consistency of information between frames, ensuring accurate reconstruction of moving objects in dynamic scenes.

[0041] 3. Fusion of Hidden Layer States: This invention resolves the conflict between high-frequency noise and low-frequency structural information by weighted fusion of low- and high-frequency hidden layer states. The fused hidden layer states incorporate both global and local correlation information between frames, significantly improving the detail and overall consistency of the reconstructed frames.

[0042] 4. Efficient computing and real-time performance: In the algorithm implementation, the present invention uses a lightweight convolutional neural network to implement key steps such as feature extraction of low-frequency and high-frequency components and optical flow estimation. At the same time, the Sigmoid function is used to constrain the dynamic weights to ensure that the computational complexity is controlled within an acceptable range and meet real-time processing requirements.

[0043] Through the synergy of the above technical features, the present invention has significant beneficial effects in the following aspects:

[0044] (1) Improved reconstruction quality: Through frequency domain separation and efficient optical flow alignment, low-frequency components provide clear structural information, while high-frequency components achieve a balance between detail preservation and noise suppression. Compared with traditional methods, this method can achieve higher PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) indicators in complex dynamic scenes.

[0045] (2) Enhanced temporal consistency: Optical flow estimation and motion compensation ensure the motion consistency between frames, solve the problem of frame jumping caused by noise or motion distortion, and improve the visual smoothness of video sequences.

[0046] (3) Computational efficiency optimization: Through lightweight network design and efficient dynamic weight calculation, the present invention significantly reduces computational complexity. Compared with traditional sparse optimization methods, the processing time is shortened by about 50%, achieving real-time processing of high-resolution video.

[0047] (4) Adaptability and robustness: The present invention can still guarantee the quality of video reconstruction in low-bandwidth and high-noise scenarios. This makes it highly practical in resource-constrained environments such as drone monitoring, disaster relief, and smart cities.

[0048] In summary, the present invention solves the core problems in the existing technology from the algorithm level. Through the frequency domain separation strategy, optical flow estimation and optimization processing of hidden layer states, it provides a video compression sensing reconstruction method with both high quality and high efficiency, bringing breakthrough improvements to the fields of video communication and intelligent monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the video compression sensing method flow of the present invention.

[0050] Figure 2 Schematic diagram of the hidden layer state calculation process of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0052] like Figure 1-2 As shown, the present invention provides a video compression sensing reconstruction method based on frequency domain separation, including acquiring a video frame sequence and performing compression sensing sampling on each frame to obtain measurement values. Preferably, these measurement values ​​can be generated by a single-pixel camera or other low-sampling device to reduce the complexity and cost of the sampling hardware. After obtaining the measurement values, an initial reconstruction result of the current frame is obtained through an initial reconstruction module. The initial reconstruction can use a simple linear interpolation method or other fast reconstruction algorithm.

[0053] The hidden layer state calculation module then calculates the hidden layer state for the current frame. The hidden layer state is divided into low-frequency and high-frequency hidden layer states. This separation strategy effectively reduces the impact of noise on high-frequency details while preserving low-frequency structural information. Finally, the hidden layer state is fused with the initial reconstruction result to obtain the final reconstruction of the video frame.

[0054] In one embodiment of the present invention, the encoding end of the present invention: adopts an existing method to perform compressed sensing sampling to obtain the measurement value y of each frame t ∈R M×1 , M is the number of measurements.

[0055] Decoding end: First, through the initial reconstruction module, from the measurement value y of each frame t Restore the initial reconstruction Where h×w is the spatial dimension of the video frame. Subsequent reconstructions will maintain the h×w spatial dimension until the final result is output. Generally, subsequent reconstructions are usually composed of multiple network blocks (such as residual blocks). For one or more reconstruction states in the subsequent reconstruction, the following hidden state calculation and fusion are performed:

[0056] Assume that the current reconstruction state and the previous frame reconstruction state are S t and S t-1 The goal is to t and S t-1 Calculate the hidden layer state containing the valid time domain information of the previous frame Then compare it with S t In order to reduce the impact of noise on this step, the present invention processes the high-frequency component and the low-frequency component separately. The specific steps are as follows:

[0057] First, calculate S t-1 to S t The optical flow F t-1 , the methods that can be used include but are not limited to the SpyNet network.

[0058] Then, calculate S t and S t-1 The high frequency component and low-frequency components The calculation method is: first calculate the low-frequency component. The methods that can be used include but are not limited to linear low-pass filter, Gaussian blur, etc. Calculate the low-frequency component and After that, the high frequency components can be and get.

[0059] Then, the low-frequency hidden layer components are calculated. and Fusion is performed to obtain the low-frequency hidden layer component The features of each pixel in It can be calculated by the following formula:

[0060]

[0061] in, for The i-th pixel, for The j-th pixel point, f(·,·) is a binary function mapping, which can be obtained by , where θ(·) and φ(·) are two linear functions implemented by convolutional layers, used to calculate the correlation between two points. g(·) is a linear mapping function that can be implemented by a convolutional layer, and C(·) is a normalization function that can be set to It can be implemented using softmax in the network.

[0062] and The similarity is measured by , where θ(·) and φ(·) are two unary linear functions implemented by the convolutional layer. The convolution kernel size of the convolutional layer is 1×1, the stride is 1, and the padding is 0.

[0063] Next, calculate the high-frequency hidden layer components. According to the optical flow F t-1 , through motion compensation Align to The result is calculate and The similarity at each point

[0064]

[0065] in, for and The point product of , p(·) and q(·) are two linear functions, which can be implemented by two convolutional layers respectively. sigmoid(·) is a sigmoid function used to limit the output to [0,1]. As weight, with Perform pixel-by-pixel multiplication, that is, multiply the corresponding position elements to obtain the high-frequency hidden layer component

[0066] Next, calculate the hidden layer state Depend on and Adding them together we get:

[0067]

[0068] Fusion of hidden layer state and current frame reconstruction state. With S t Cascade, then obtain the fused features through a convolutional layer and use it as the input of the next reconstruction module.

[0069] For example, in a smart traffic scenario, video frame data can be transmitted from a roadside camera to a cloud server, and high-quality video reconstruction with low-bandwidth transmission can be achieved through the method of the present invention.

[0070] In the present invention, the calculation of the low-frequency hidden layer state is one of the key links in the reconstruction process. Specifically, pixel-level feature information is extracted from the low-frequency components of the current frame and the previous frame, and the low-frequency hidden layer state of the current frame is calculated using formula (2):

[0071] Preferably, the Softmax function can be dynamically adjusted according to computing resources, and the empirical value range is usually selected between [0.1, 1].

[0072] In the process of obtaining the low-frequency components, the present invention preferably uses a linear low-pass filter to filter the video frame. The low-pass filter can include various methods such as Gaussian blur and mean filtering. The standard deviation σ of the Gaussian blur filter parameters is preferably 1.5 to ensure that the main structural information of the video frame is retained while effectively filtering out high-frequency noise.

[0073] Taking public security monitoring as an example, in complex nighttime monitoring scenarios, low-frequency components can clearly retain the outlines of large objects while filtering out small light spots and noise, helping to improve the accuracy of target recognition.

[0074] The high-frequency hidden layer state is realized by aligning the optical flow information. The optical flow information can be calculated by the SpyNet network. After alignment, the high-frequency hidden layer state is calculated by formula (3)

[0075] The high frequency component is calculated using the following formula:

[0076]

[0077]

[0078] Among them, S t is the initial reconstruction feature of the current frame, S t-1 Reconstruct the state of the previous frame, and represents the high frequency component, and Preferably, in different application scenarios, the present invention can dynamically adjust the weight ratio of the initial reconstruction. For example, in a field exploration environment, a higher weight can be given to the high-frequency component to enhance the edge details of small targets.

[0079] First, to improve the accuracy of video frame reconstruction, the present invention separates the hidden layer states into low-frequency and high-frequency hidden layer states and solves them separately. This separation strategy significantly reduces the distortion of high-frequency information caused by noise accumulation while ensuring the stability of the low-frequency components.

[0080] Specifically, based on the initial reconstruction results of the current and previous frames, high-frequency and low-frequency components are calculated, respectively. The low-frequency component is obtained through a low-pass filter, such as Gaussian blur or mean filtering, to ensure that the main structural information in the frame is extracted. The high-frequency component is obtained by subtracting the low-frequency component from the original frame, thereby preserving edge details and texture information.

[0081] When calculating the hidden layer state, the present invention combines the attention mechanism with the motion compensation strategy. Figure 2 As shown in , the optical flow information obtained by the optical flow calculation module provides the basis for motion compensation between frames. Optical flow can accurately capture the movement of objects between frames, thereby supporting the alignment of high-frequency components.

[0082] In processing high-frequency components, the present invention uses motion compensation to align the high-frequency components of the previous frame, obtaining the aligned high-frequency hidden layer state. Furthermore, a dynamic weighting strategy is employed to calculate a weight matrix based on the similarity of pixels between frames. This dynamic weighting mechanism significantly suppresses the propagation of high-frequency noise and effectively improves the fidelity of edge details in the reconstructed frame.

[0083] Unlike high-frequency processing, low-frequency hidden layer states are derived through global pixel-level correlation. Each pixel in the current frame is then fused with all pixels from the previous frame. This method captures the dynamic characteristics of low-frequency information through global correlation, enhancing the consistency of information between frames while maintaining the overall image structure.

[0084] This invention significantly improves the quality and efficiency of compressed sensing video reconstruction through a frequency-domain separation strategy and efficient hidden-layer state calculation. Separate processing of low-frequency and high-frequency information effectively suppresses noise propagation while retaining more useful time-domain information. These innovations make this invention particularly suitable for applications such as smart cities, security surveillance, and resource-constrained video sensor networks, providing high-quality reconstructed video at low bandwidths.

[0085] A key aspect of the present invention is modeling inter-frame motion using optical flow information. Preferably, the optical flow information is obtained using the SpyNet network or other lightweight optical flow estimation algorithms. The SpyNet network, due to its high efficiency and low computational complexity, has significant advantages in resource-constrained scenarios.

[0086] Optical flow information F t-1 The purpose of optical flow is to capture the displacement and deformation relationships between frames, providing a basis for aligning high-frequency components. In urban smart transportation systems, cameras are installed on moving vehicles, resulting in significant motion and displacement between video frames. Optical flow information can be used to precisely align video details from the previous frame to the current frame, ensuring that the reconstruction process is unaffected by motion.

[0087] For example, in a scene where a vehicle is traveling from west to east, the road edge and traffic signs in the previous frame will shift in position in the current frame. The role of optical flow information is to accurately capture this displacement, thereby minimizing the motion error between frames.

[0088] In the present invention, the fusion of hidden layer states is the core guarantee of reconstruction quality. The low-frequency hidden layer states and the high-frequency hidden layer states are weighted superimposed using formula (4):

[0089] The low-frequency hidden layer states provide the main structural information of the video frame, while the high-frequency hidden layer states enhance the detail expression and edge clarity.

[0090] The calculated high-frequency hidden layer state and low-frequency hidden layer state are fused. This fusion operation ensures the full integration of high-frequency and low-frequency information. The final hidden layer state With the initial reconstruction result S t After cascading, it is input into the subsequent convolutional layer to further extract features and obtain the optimized reconstruction result.

[0091] This method significantly improves the quality of video frame reconstruction by combining frequency domain separation, optical flow estimation, and a dynamic weighting strategy. The separation of low-frequency and high-frequency components and differentiated treatment of different noise sources enhance the robustness of the algorithm. This method ensures high-quality video transmission even in low-bandwidth environments, providing effective technical support for smart transportation, public safety monitoring, and other fields.

[0092] In practical applications, such as in smart transportation scenarios, the processing method of the present invention can reconstruct high-quality video streams under limited network bandwidth conditions, thereby improving the accuracy of target detection and behavior analysis.

[0093] Preferably, during fusion, the present invention can dynamically adjust the weights of low and high frequencies based on the specific scenario. For example, in public security monitoring, low-frequency components can be used to identify activities within a large area, while high-frequency components can be used to capture subtle anomalies, such as the specific posture or gestures of a suspect.

[0094] The hidden layer state after fusion and the initial reconstruction result S of the current frame t Cascade processing is performed, which specifically includes the following processes:

[0095] First, the concatenated features are subjected to dimensionality reduction and feature extraction through the convolutional layer to reduce the impact of redundant information and obtain streamlined reconstructed features. Subsequently, this reconstructed feature is input into the subsequent reconstruction module to generate the final video frame reconstruction result.

[0096] For example, in disaster response scenarios, the image quality captured by cameras deployed on drones can be unstable due to altitude and wind. The proposed fusion strategy combines the rough features in the initial reconstruction results with the temporal information in the hidden layer states, significantly improving the clarity and accuracy of frame reconstruction.

[0097] A key feature of this invention is its support for frame-by-frame processing of video sequences. Each frame in the sequence undergoes the complete process, from compressed sensing sampling to reconstruction, ensuring video continuity and quality. By fully utilizing the temporal information between frames, inconsistencies between consecutive frames caused by low sampling rates or noise are eliminated.

[0098] For example, when multiple cameras are used to monitor a mining environment in real time during field exploration, bandwidth limitations can lead to video quality degradation. This invention uses frame-by-frame processing to provide high-quality reconstructed video even under limited bandwidth conditions, ensuring continuous safety monitoring of the mining area.

[0099] The present invention is applicable to resource-constrained video sensor network scenarios, including but not limited to smart transportation, public safety monitoring, field exploration, and emergency rescue. The following are some examples of specific application scenarios:

[0100] Smart Transportation: Camera systems deployed in smart vehicles require real-time video data transmission. However, due to network bandwidth limitations, traditional video transmission methods cannot meet the requirements for high-resolution video. This invention uses compressed sensing and frequency domain separation techniques to achieve high-quality video reconstruction under low bandwidth conditions, providing support for smart transportation.

[0101] Public safety monitoring: In cities, thousands of cameras record large amounts of video data in real time. The method of the present invention can significantly reduce the bandwidth requirements for video transmission, while improving the clarity of reconstructed video, facilitating subsequent event backtracking and analysis.

[0102] Field exploration: In remote areas such as mining areas and forests, weak communication network infrastructure poses challenges in transmitting and storing video data. This invention can complete data acquisition at a low sampling rate and, through an efficient reconstruction algorithm, ensure the quality of exploration videos, providing technical support for environmental monitoring and natural resource management.

[0103] Emergency rescue: During disaster response, drones and ground-based sensors need to transmit real-time video. This invention can provide high-quality reconstructed video within limited bandwidth, helping rescue command centers to understand disaster site conditions in a timely manner and improve decision-making efficiency.

[0104] As demonstrated above, this invention demonstrates significant technical advantages in multiple application scenarios. The strategy of combining optical flow estimation with frequency domain separation effectively improves video reconstruction quality. Frame-by-frame optimization and fusion of hidden layer states ensures temporal consistency and spatial detail in the video. In low-bandwidth, high-noise environments, this invention demonstrates exceptional robustness and adaptability, providing advanced video processing solutions for smart cities, security surveillance, and other fields.

[0105] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video compression sensing reconstruction method based on frequency domain separation, characterized in that: include: Acquire a video frame sequence and perform compressed sensing sampling on each frame of the video frame sequence to obtain a measurement value; Based on the measurement value, obtaining an initial reconstruction result of the current frame through initial reconstruction; Calculating the hidden layer state of the current frame using a hidden layer state calculation module, wherein the hidden layer state includes a low-frequency hidden layer state and a high-frequency hidden layer state; The hidden layer state is fused with the initial reconstruction result to obtain a reconstruction result of the video frame.

2. The video compression sensing reconstruction method based on frequency domain separation according to claim 1 is characterized in that: The calculation of the low-frequency hidden layer state includes: Through the attention mechanism and Fusion is performed to obtain the low-frequency hidden layer component The features of each pixel in It can be calculated by the following formula: in, for The i-th pixel point, for The j-th pixel point of , f(·,·) is a binary function mapping, through , where θ(·) and φ(·) are two one-dimensional linear functions implemented by convolutional layers, used to calculate the correlation between two points, g(·) is a linear mapping function implemented by a convolutional layer, and C(·) is a normalization function, set to It can be implemented with softmax in the network; and The similarity is measured through , where θ(·) and φ(·) are two unary linear functions implemented by the convolutional layer. The convolution kernel size of the convolutional layer is 1×1, the stride is 1, and the padding is 0.

3. The method according to claim 2, characterized in that In the attention mechanism, The linear mapping of is Value, is the Key, For Query.

4. The method according to claim 2, characterized in that: The low-frequency component is obtained by a linear low-pass filter, and the low-pass filter includes but is not limited to a Gaussian blur filter or other linear filtering methods.

5. The video compression sensing reconstruction method based on frequency domain separation according to claim 2 is characterized in that: The calculation of the high frequency hidden layer state includes: According to the optical flow F t-1 , through motion compensation Align to The result is calculate and The similarity at each point in, for and , p(·) and q(·) are two linear functions, which are implemented by two convolutional layers respectively. sigmoid(·) is the sigmoid function, which is used to limit the output to [0,1]. As weight, with Perform pixel-by-pixel multiplication, that is, multiply the elements at corresponding positions to obtain the high-frequency hidden layer components 6. The method according to claim 5, characterized in that The fusion method includes: calculating the hidden layer state Depend on and Adding them together gives: Fusion of the hidden layer state and the current frame reconstruction state, With S t The fused features are then cascaded through a convolutional layer and used as the input of the next reconstruction module.

7. The method according to any one of claims 1 to 6, characterized in that The optical flow information F t-1 Obtained through the SpyNet network or other optical flow estimation algorithms.

8. The method according to claim 6, characterized in that The fused hidden layer state is cascaded with the initial reconstruction result of the current frame, processed by the convolution layer, and input into the next reconstruction module to obtain the final reconstruction result of the frame.

9. The method according to claim 1, characterized in that: The method further includes: calculating and fusing hidden layer states of multiple video frames in sequence, processing frame by frame, and obtaining reconstruction results of continuous videos.

10. The video compression sensing reconstruction method based on frequency domain separation according to any one of claims 1 to 6, characterized in that: The method is applicable to resource-constrained video sensor network scenarios, including but not limited to smart transportation, public safety monitoring, field exploration and emergency rescue.

Images