Pulse camera video reconstruction method based on time multi-resolution and fine-grained attention

Abstract

The invention discloses a pulse camera video reconstruction method based on time multi-resolution and fine-grained attention, and the method comprises the following steps: pulse data is divided into a plurality of data blocks, each data block comprises multiple frames of data, and parts of frames in the adjacent data blocks are overlapped; performing feature extraction on each data block; encoding the extracted features by using an encoder to obtain encoded features; processing the coding features by adopting an attention mechanism to obtain fine codes; and after decoding the fine code, outputting a video sequence through an output block. According to the pulse camera video reconstruction method based on time multi-resolution and fine-grained attention disclosed by the invention, the reconstruction quality is improved while the operation efficiency is improved.

Description

Technical Field

[0001] The invention relates to a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention, and belongs to the technical field of image processing. Background Art

[0002] In traditional digital cameras, the photosensitive chip accumulates photons within a fixed exposure time and converts the light changes during this period into a static image. This method sometimes results in blurred images when shooting high-speed moving objects. Compared with traditional cameras, pulse cameras use an innovative photography technology. It does not use the traditional concept of shutter and exposure time, but allows each pixel to continuously collect photons. Once the set number of photons is reached, it is converted into a digital signal to generate a pulse. The shorter the generation time of these pulses, the stronger the light, and vice versa. In this way, the density and sparseness of the pulses directly reflect the changes in light intensity.

[0003] Most of the existing image reconstruction methods are traditional digital image designs and are not suitable for pulse camera images. The existing pulse camera image reconstruction methods mainly include TFI method and TFP method. Among them, the TFI method estimates the light intensity of the scene by measuring the pulse interval of each pixel. This method is direct and fast, but it is easily affected by noise because of the short observation time and only relying on a single interval; the TFP method evaluates the light intensity by counting the number of pulses within a set short exposure time. Its reconstruction effect is significantly affected by the size of the time window: a small window is easily disturbed by noise, and a large window may introduce motion blur.

[0004] In addition, there are methods in the prior art for processing pulse camera images based on deep learning methods. For example, Spk2Img adopts a multi-time window strategy to explore short-term and long-term temporal correlations and performs motion alignment through deformable convolution, thereby significantly improving the clarity of the image; WGSE improves the quality of reconstructed images by using discrete wavelet transform to represent binary pulse streams more robustly. However, there are still some artifacts and blurs in the images reconstructed by Spk2Img and WGSE.

[0005] In addition, existing pulse camera image reconstruction methods can only perform single-frame reconstruction, that is, only a single image can be generated for a given pulse stream at a time. When restoring a continuous image sequence, such methods will cause a large amount of repeated calculations between consecutive frames, thereby increasing the computational cost. On the other hand, the time window length of the existing methods is fixed and short, which theoretically leads to unsatisfactory reconstruction quality, and it is impossible to flexibly change the time window size of inference.

[0006] Therefore, it is necessary to conduct in-depth research on the existing pulse camera image reconstruction methods to solve the above problems. Summary of the invention

[0007] In order to overcome the above problems, an in-depth study was conducted and a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention was proposed, which includes the following steps:

[0008] S1, dividing the pulse data into multiple data blocks, each data block contains multiple frames of data, and some frames in adjacent data blocks overlap;

[0009] S2, extract features from each data block;

[0010] S3, using an encoder to encode the extracted features to obtain encoded features;

[0011] S4, use the attention mechanism to process the encoding features and obtain fine encoding;

[0012] S5. After decoding the fine coding, the video sequence is output through the output block.

[0013] In a preferred embodiment, in S2, a two-dimensional convolutional layer is used to extract features from each data block, which is expressed as:

[0014] F i =Conv2D(B i )

[0015] Among them, Conv2D represents a two-dimensional convolutional layer, which means B i represents the i-th data block, F i Represents the features extracted from the i-th data block.

[0016] In a preferred embodiment, in S3, the encoder includes multiple layers, the first layer is an initialization layer, and the subsequent layers are all temporal downsampling layers.

[0017] In a preferred embodiment, the features extracted from different data blocks are connected in series to obtain the total features, which are input into the first level, and the first level outputs the initialization features, which are expressed as:

[0018] D 1 =Init(F)

[0019] Among them, Init represents the first level, D 1 Represents the initialization features of the first layer output.

[0020] In a preferred embodiment, in the subsequent layer, the output of the previous layer is used as input, and its output is the encoded feature, which is expressed as:

[0021] D l =Downl (D l-1 ),l=2,…,L,

[0022] Among them, D l represents the output of layer l, D l-1 Represents the output of the l-1 layer, Down l represents the downsampling layer of the lth subsequent time dimension, and L represents the total number of encoder levels.

[0023] In a preferred embodiment, the attention mechanism adopts fine-grained attention, including a local attention module and a Mamba module.

[0024] In the local attention module, the encoded features output by each level of the encoder are divided into multiple blocks along the spatial dimension, expressed as:

[0025] P l = reshape(D l )

[0026] Among them, P l Represents a set of multiple blocks after division, each block is of size p×p, reshape represents the division process,

[0027] P l Attention characteristics It is expressed as:

[0028]

[0029] Q l =Query l (P l ),

[0030] K l =Key l (P l ),

[0031] V l =Value l (P l ),

[0032] Among them, Softmax represents the normalized exponential function, Q l Indicates P l The query vector K l Indicates P l The key vector, V l Indicates P l The value vector of d k K l The last dimension, Query l Indicates the query conversion process, Keyl Indicates the key conversion process, Value l Represents the value conversion process.

[0033] In a preferred embodiment, a residual connection is set after the fine-grained attention module, and the result of the residual connection is used as a new fine code for subsequent decoding, which is expressed as:

[0034] E l =A l +D l ,l=1,…,L

[0035] Among them, E l represents the new fine code of layer l, A l It is the refined encoding obtained after processing by the l-th layer fine-grained attention module.

[0036] In a preferred embodiment, in S5, the decoder comprises a plurality of layers, wherein the Lth layer to the second layer are temporal upsampling blocks, and the first layer is an output block.

[0037] Among them, the L-th layer of the decoder takes the L-th layer of fine coding as input, which is expressed as:

[0038] U L =E L

[0039] The remaining layers take the fine encoding of different layers and the output of the previous layer as input, expressed as:

[0040] U l =E l +Up l+1 (U l+1 ),l=L-1,…,2.

[0041] The present invention also provides an electronic device, comprising:

[0042] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any of the above methods.

[0043] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any of the above-mentioned methods.

[0044] The beneficial effects of the present invention include:

[0045] (1) It can directly reconstruct multiple frames of images from the pulse stream and flexibly change the time window size of the input data without retraining;

[0046] (2) Improve the reconstruction quality while improving the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic flow chart of a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to a preferred embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the encoder structure in a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to a preferred embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a decoder structure in a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to a preferred embodiment of the present invention;

[0050] Figure 4 Schematic diagram of the structure of a fine-grained attention module in a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present invention will become more clear and distinct.

[0052] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0053] Pulse camera technology originates from the study of biological retinas, and its design is inspired by the neural coding method in animal retinas. Unlike traditional frame-based cameras, pulse cameras do not rely on fixed exposure time to capture images, but instead use the changes in light intensity at each pixel to emit pulses when the accumulation reaches a certain threshold, thereby achieving high temporal resolution imaging. This feature gives pulse cameras a clear advantage in processing high-speed dynamic scenes.

[0054] According to a pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention provided by the present invention, a pulse sequence is reconstructed into continuous video frames, such as Figure 1 As shown, including:

[0055] S1, dividing the pulse data into multiple data blocks, each data block contains multiple frames of data, and some frames in adjacent data blocks overlap;

[0056] S2, extract features from each data block;

[0057] S3, using an encoder to encode the extracted features to obtain encoded features;

[0058] S4, use the attention mechanism to process the encoding features and obtain fine encoding;

[0059] S5, after decoding the fine coding, output the video sequence through the output block. In S1, preferably, the number of data blocks into which the pulse data is divided is set to:

[0060]

[0061] Where N represents the number of data blocks, M represents the total number of pulse data, K represents the number of frames in each data block, and stride represents the step size.

[0062] In S2, preferably, a two-dimensional convolutional layer is used to extract features from each data block, expressed as:

[0063] F i =Conv2D(B i )

[0064] Among them, Conv2D represents a two-dimensional convolutional layer, which means B i represents the i-th data block, F i Represents the features extracted from the i-th data block.

[0065] According to the present invention, in S3-S5, spatial and temporal domains are modeled separately, and temporal features are downsampled step by step to extract multi-temporal resolution features while maintaining spatial resolution. Spatial modeling is performed at each level to supplement spatial domain information to model and reconstruct a clear video.

[0066] Specifically, in S3, the encoder includes multiple layers, the first layer is an initialization layer, and the subsequent layers are all temporal downsampling layers.

[0067] According to the present invention, the features extracted from different data blocks are connected in series to obtain the total feature, which is expressed as Among them, T 1 is the time dimension of the first layer, and C is the initial channel of the total feature. Furthermore, the time dimension of the first layer is set to T 1 =N.

[0068] The total features are input into the first level, and the features are initialized by the output of the first level, which can be expressed as:

[0069] D 1 =Init(F)

[0070] Among them, Init represents the first level, D 1 Represents the initialization features of the first layer output.

[0071] In the present invention, by initializing features, a standardized starting point is provided for feature extraction and subsequent processing.

[0072] Furthermore, in the subsequent layers, the output of the previous layer is used as input, and its output is the encoded feature, which is expressed as:

[0073] D l =Down l (D l-1 ),l=2,…,l,

[0074] Among them, D l represents the output of layer l, D l-1 Represents the output of the l-1 layer, Down l represents the lth subsequent layer, L represents the total number of encoder levels, and further, C l and T l are the number of output channels and the time dimension of the lth layer respectively.

[0075] Preferably, each of the downsampling levels includes a plurality of 3D convolutional layers and LeakyReLU layers, the number of the LeakyReLU layers is the same as the number of the 3D convolutional layers, and a LeakyReLU layer is arranged after each 3D convolutional layer, such as Figure 2 shown.

[0076] Among them, the first 3D convolutional layer performs downsampling in the time dimension with a step size of 2, reducing the time dimension to half of the original, and the step size in the spatial dimension is 1, keeping the spatial dimension unchanged. Subsequent 3D convolutional layers are used to learn more complex features.

[0077] In the present invention, through multiple levels of S3 encoding, the spatial domain resolution is maintained and the time domain is downsampled step by step to reduce the redundancy of time domain information and obtain multi-time resolution representation. At the same time, the time domain information is fully utilized to supplement the spatial domain information and make up for the problem of insufficient spatial domain information.

[0078] In S4, preferably, the attention mechanism adopts a fine-grained attention (FGA) module, including a local attention module and a Mamba module.

[0079] In the local attention module, the encoded features output by each level of the encoder are divided into multiple blocks along the spatial dimension, expressed as:

[0080] P l = reshape(D l )

[0081] Among them, P l Represents a set of multiple blocks after division, each block is of size p×p, and reshape represents the division process.

[0082] Furthermore, P l Attention characteristics It is expressed as:

[0083]

[0084] Q l =Query l (P l ),

[0085] K l =Key l (P l ),

[0086] V l =Value l (P l ),

[0087] Among them, Softmax represents the normalized exponential function, Q l Indicates P l The query vector K l Indicates P l The key vector, V l Indicates P l The value vector of d k K l The last dimension, Query l Indicates the query conversion process, Key l Indicates the key conversion process, Value l Represents the value conversion process.

[0088] The query conversion process, key conversion process, and value conversion process are all common processes of the attention mechanism and are not described in detail in the present invention. For example, a combination of a 1D convolutional layer and a LeakyReLU layer is used.

[0089] Preferably, in the query conversion process, the key conversion process, and the value conversion process, the kernel size of the 1D convolution is 3.

[0090] The Mamba module is a model proposed by Albert Gu and Tri Dao in December 2023. It can maintain linear time complexity when processing long sequence data and significantly improves computational efficiency. In the present invention, its specific structure is not described in detail.

[0091] Furthermore, the attention feature Through the output block Out l After and P l Perform residual connection and then pass through the Mamba module to obtain fine coding.

[0092] Furthermore, the encoding features of each layer output are processed by the attention mechanism, which can be expressed as:

[0093] A l =FGA l (D l ),l=1,…,L

[0094] Among them, FGA l represents the l-th layer fine-grained attention module, A l It is the refined encoding obtained after processing by the l-th layer fine-grained attention module.

[0095] According to the present invention, each pixel in the fine-grained attention module is used as an independent marker, and the model can analyze the dynamic relationship between each pixel in more detail, thereby achieving more accurate reconstruction of temporal and spatial information in complex dynamic scenes. By subdividing the feature map into multiple patches, the balanced distribution of computational burden is further achieved, so that the model can run efficiently even in an environment with limited resources.

[0096] In a preferred embodiment, a residual connection is set after the fine-grained attention module, and the result of the residual connection is used as a new fine code for subsequent decoding, which is expressed as:

[0097] E l =A l +D l ,l=1,…,L

[0098] Among them, E l Represents the new fine code of layer l.

[0099] In S5, the decoder includes multiple levels, wherein the Lth level to the 2nd level are temporal upsampling blocks, and the 1st level is an output block, integrating new fine coding from bottom to top.

[0100] Preferably, each of the upsampling blocks comprises a plurality of 3D convolutional layers, a transposed 3D convolutional layer and a plurality of LeakyReLU layers, wherein the LeakyReLU layer is arranged between the 3D convolutional layer and the transposed 3D convolutional layer, such as Figure 3 shown.

[0101] Among them, the L-th layer of the decoder takes the L-th layer of fine coding as input, which is expressed as:

[0102] U L =E L

[0103] The remaining layers take the fine encoding of different layers and the output of the previous layer as input, expressed as:

[0104] U l =E l +Up l+1 (U l+1 ),l=L-1,…,2.

[0105] According to the present invention, the output block is a 3D convolution layer, and the output video segment is represented as:

[0106]

[0107] in, represents the output video segment, Flat represents the output block, and E 1 Indicates the new fine coding of layer 1, Up 2 Indicates the decoder layer 2 level, U 2 Represents the output of the second layer.

[0108] According to the present invention, the decoder restores the time scale layer by layer and integrates the spatiotemporal features. This bottom-up integration strategy ensures the flow of information from details to the whole, and finally outputs a high-quality video sequence.

[0109] According to a preferred embodiment of the present invention, during the training process, the loss function loss is set to:

[0110]

[0111] Where N is the number of image frames in the video clip, represents the predicted video segment, represents a real video clip, ||·|| 1 represents the L1 norm.

[0112] Furthermore, according to the present invention, the spatial resolution and time window size of the input data can be flexibly changed without the need for retraining, which has great application potential.

[0113] Various embodiments of the method described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document is not limited here.

[0115] Example

[0116] Example 1

[0117] The video reconstruction experiment is conducted on a certain dataset, including the following steps:

[0118] S1, dividing the pulse data into multiple data blocks, each data block contains multiple frames of data, and some frames in adjacent data blocks overlap;

[0119] S2, extract features from each data block;

[0120] S3, using an encoder to encode the extracted features to obtain encoded features;

[0121] S4, use the attention mechanism to process the encoding features and obtain fine encoding;

[0122] S5, after decoding the fine coding, output the video sequence through the output block. In S1, the number of data blocks into which the pulse data is divided is set to:

[0123]

[0124] In S2, a two-dimensional convolutional layer is used to extract features from each data block, expressed as:

[0125] F i =Conv2D(B i )

[0126] In S3, the encoder includes multiple layers, the first layer is an initialization layer, and the subsequent layers are all downsampling layers.

[0127] The features extracted from different data blocks are concatenated to obtain the total features, which are input into the first level, and the first level outputs the initialization features, which can be expressed as:

[0128] D 1 =Init(F)

[0129] In the subsequent layers, the output of the previous layer is used as input, and its output is the encoded feature, which is expressed as:

[0130] D l =Down l (D l-1 ),l=2,…,L,

[0131] Each of the downsampling levels includes multiple 3D convolutional layers and LeakyReLU layers, the number of the LeakyReLU layers is the same as the number of the 3D convolutional layers, and a LeakyReLU layer is set after each 3D convolutional layer. The first 3D convolutional layer performs downsampling in the time dimension with a step size of 2, reducing the time dimension to half of the original, and the step size in the spatial dimension is 1.

[0132] In S4, the attention mechanism adopts a fine-grained attention module, including a local attention module and a Mamba module, such as Figure 4 In the local attention module, the encoded features output by each level of the encoder are divided into multiple blocks along the spatial dimension, which are expressed as:

[0133] P l = reshape(D l )

[0134] P l Attention characteristics It is expressed as:

[0135]

[0136] Q l =Query l (P l ),

[0137] K l =Key l (P l ),

[0138] V l =Value l (P l ),

[0139] The query conversion process, key conversion process, and value conversion process use a combination of 1D convolutional layer and LeakyReLU layer, where the kernel size of the 1D convolution is 3.

[0140] Attention Characteristics Through the output block Out l After and P l Perform residual connection and then pass through the Mamba module to obtain fine coding.

[0141] A residual connection is set after the fine-grained attention module, and the result of the residual connection is used as a new fine code for subsequent decoding, which is expressed as:

[0142] E l =A l +D l ,l=1,…,L

[0143] The decoder comprises multiple layers, wherein the Lth layer to the 2nd layer are upsampling blocks, the 1st layer is an output block, and new fine coding is integrated from bottom to top, each of the upsampling blocks comprises multiple 3D convolutional layers, a transposed 3D convolutional layer and multiple LeakyReLU layers, and the LeakyReLU layer is arranged between the 3D convolutional layer and the transposed 3D convolutional layer.

[0144] The L-th layer of the decoder takes the L-th layer’s fine code as input, expressed as:

[0145] U L =E L

[0146] The remaining layers take the fine encoding of different layers and the output of the previous layer as input, expressed as:

[0147] U l =E l +Up l+1 (U l+1 ),l=L-1,…,2.

[0148] The output block is a 3D convolutional layer, and the output video segment is represented as:

[0149]

[0150] The peak signal-to-noise ratio (PSNR) is used to characterize the quality of video reconstruction, and the video memory consumed by the average single-frame reconstruction, inference time, and FLOPs indicators are used to characterize the consumption of computing resources by reconstruction.

[0151] Example 2

[0152] The same experiment as Example 1 was carried out, except that real pulse data of three different motion scenes (car, fan rotation and bridge) were used, and the deep learning-based blind image quality evaluation index MANIQA was used to evaluate the images in the reconstructed video, and the average performance of the reconstructed image in each motion process was obtained, where the larger the MANIQA value, the better the quality of the image reconstructed by the model.

[0153] Comparative Example 1

[0154] The same experiment as Example 1 was performed, except that WGSE and Spk2Img methods were used respectively, wherein the WGSE method refers to the document Y. Dong, J. Zhao, R. Xiong and T. Huang, "High-Speed ​​Scene Reconstruction from Low-Light Spike Streams," 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP), Suzhou, China, 2022, pp. 1-5, doi: 10.1109 / VCIP56404.2022.10008850.;

[0155] For the Spk2Img method, please refer to the literature Zhao J, Xiong R, Liu H, et al. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 11996-12005.

[0156] The results of Example 1 and Comparative Example 1 are shown in Table 1.

[0157] Table 1

[0158]

[0159]

[0160] It can be seen from the table that the peak signal-to-noise ratio of the method in Example 1 is as high as 45.6, which is improved by more than 4dB compared with the best method in Comparative Example 1.

[0161] The average inference time for a single photo of the method in Example 1 is much shorter than that of the method in Comparative Example 1, and the FLOPs and video memory consumption are also smaller.

[0162] Comparative Example 2

[0163] The same experiment as in Example 2 was performed, except that TFI, TFP, TFSTP, HMSR, SpikeFormer, WGSE and Spk2Img methods were used respectively.

[0164] Among them, the TFI method and the TFP method refer to the literature Zhu L, Dong S, Huang T, et al. A retina-inspired sampling method for visual texture reconstruction [C] / / 2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2019: 1432-1437;

[0165] For the TFSTP method, please refer to the literature Zheng Y, Zheng L, Yu Z, et al. High-speed imagereconstruction through short-term plasticity for spiking cameras[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2021:6358-6367;

[0166] For the HMSR method, please refer to the literature Zhao J, Xiong R, Huang T. High-speed motion scenereconstruction for spike camera via motion aligned filtering[C] / / 2020IEEEInternational Symposium on Circuits and Systems(ISCAS).IEEE, 2020:1-5;

[0167] For the SpikeFormer method, please refer to the literature She C, Qing L.SpikeFormer:Image Reconstruction from the Sequence of Spike Camera Based on Transformer[C] / / Proceedings of the2022 5th International Conference on Image and Graphics Processing.2022:72-78.

[0168] The results of Example 2 and Comparative Example 2 are shown in Table 2.

[0169]

[0170] It can be seen from the table that the method in Example 2 has excellent video reconstruction results in different scenarios. It is better than all methods in Comparative Example 2 in the car and bridge scenarios, and also has a high score in the fan scenario. The average values ​​in different scenarios are better than different methods in Comparative Example 2, showing high stability and high reconstruction quality.

[0171] The present invention has been described above in conjunction with preferred embodiments, but these embodiments are only exemplary and serve only as an illustration. On this basis, the present invention may be subjected to a variety of substitutions and improvements, all of which fall within the scope of protection of the present invention.

Claims

1. A method for pulse camera video reconstruction based on temporal multi-resolution and fine-grained attention, characterized in that The following steps are involved: S1, dividing the pulse data into multiple data blocks, each data block contains multiple frames of data, and some frames in adjacent data blocks overlap; S2, extract features from each data block; S3, using an encoder to encode the extracted features to obtain encoded features; S4, use the attention mechanism to process the encoding features and obtain fine encoding; S5. After decoding the fine coding, the video sequence is output through the output block.

2. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 1, characterized in that: In S2, a two-dimensional convolutional layer is used to extract features from each data block, expressed as: F i =Conv2D(B i ) Among them, Conv2D represents a two-dimensional convolutional layer, which means B i represents the i-th data block, F i Represents the features extracted from the i-th data block.

3. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 1, characterized in that: In S3, the encoder includes multiple layers, the first layer is an initialization layer, and the subsequent layers are all temporal downsampling layers.

4. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 3, characterized in that: The features extracted from different data blocks are concatenated to obtain the total features, which are input into the first level, and the first level outputs the initialization features, which can be expressed as: D 1 =Init(F) Among them, Init represents the first level, D 1 Represents the initialization features of the first layer output.

5. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 4, characterized in that: In the subsequent layers, the output of the previous layer is used as input, and its output is the encoded feature, which is expressed as: D l =Down l (D l-1 ),l=2,…,L, Among them, D l represents the output of layer l, D l-1 Represents the output of the l-1 layer, Down l represents the downsampling layer of the lth subsequent time dimension, and L represents the total number of encoder levels.

6. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 1, characterized in that: The attention mechanism adopts fine-grained attention, including local attention module and Mamba module. In the local attention module, the encoded features output by each level of the encoder are divided into multiple blocks along the spatial dimension, expressed as: P l =reshape(D l ) Among them, P l Represents a set of multiple blocks after division, each block is of size p×p, reshape represents the division process, P l Attention characteristics It is expressed as: Q l =Query l (P l ), K l =Key l (P l ), V l =Value l (P l ), Among them, Softmax represents the normalized exponential function, Q l Indicates P l The query vector K l Indicates P l The key vector, V l Indicates P l The value vector of d k K l The last dimension, Query l Indicates the query conversion process, Key l Indicates the key conversion process, Value l Represents the value conversion process.

7. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 6, characterized in that: A residual connection is set after the fine-grained attention module, and the result of the residual connection is used as a new fine code for subsequent decoding, which is expressed as: AND l =A l +D l ,l=1,…,L Among them, E l represents the new fine code of layer l, A l It is the refined encoding obtained after processing by the l-th layer fine-grained attention module.

8. The pulse camera video reconstruction method based on temporal multi-resolution and fine-grained attention according to claim 7, characterized in that: In S5, the decoder comprises a plurality of layers, wherein the Lth layer to the second layer are temporal upsampling blocks, and the first layer is an output block. Among them, the L-th layer of the decoder takes the L-th layer of fine coding as input, which is expressed as: The L =E L The remaining layers take the fine encoding of different layers and the output of the previous layer as input, expressed as: U l =E l +Up l+1 (U l+1 ),l=L-1,…,2。 9. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

Images