Method for reconstructing high dynamic range video using high-speed rolling mixed bit pattern pulses

Abstract

The method of reconstructing high dynamic range video by high-speed rolling mixed bit pattern pulse belongs to the field of modern digital camera, including: using the method of superimposing pulses in the time domain to reconstruct images from the captured single-bit pulse stream and multi-bit pulse stream in segments, obtaining single-bit and multi-bit pulse reconstructed images; using all the information of single-bit pulses in the time domain and space domain to calculate the optical flow information between multi-bit pulse frames, using the motion information provided by the optical flow information to correct the deformation of the multi-bit pulse reconstructed image, using the optical flow information between the multi-bit pulse frames to insert the multi-bit pulse reconstructed image; using the deep learning method to fuse the single-bit pulse reconstructed image and the multi-bit pulse reconstructed image. The present invention reduces data redundancy, speeds up the reading speed, retains the high frame rate attribute of the pulse camera, and improves the quality of the fused image; at the same time, single-bit pulse data is used to estimate the optical flow, and the optical flow is used to correct the deformation of the multi-bit pulse plane.

Description

Technical Field

[0001] The invention belongs to the technical field of modern digital camera, and in particular relates to a method for reconstructing high dynamic range video with high-speed rolling mixed bit mode pulse. Background Art

[0002] In the field of modern digital camera technology, the demand for high dynamic range and high frame rate imaging technology is increasing. High dynamic range imaging can capture a wider range of brightness, making the details of the dark and bright parts of the picture clearly visible; high frame rate imaging can capture rapidly changing scenes. Traditional cameras use timed exposure to obtain images or videos formed by arranging image sequences. Shorter exposure times will make the captured pictures darker and more susceptible to noise. Although longer exposure times can well restore the details of the dark parts of the scene and avoid certain noise, they will cause overexposure of the bright parts of the scene. Therefore, some scholars have proposed some improvement methods for the problems of traditional cameras.

[0003] There are two main classical high dynamic range imaging methods. The main idea of ​​these two methods is to use short-exposure images to restore the details of the bright parts of the scene, and use long-exposure images to restore the details of the dark parts of the scene. The two methods complement each other, so that the picture has richer details in both dark and bright parts. The first method uses a single camera to shoot images of different exposure times of the same scene in a short time, aligns these images and then weights them with certain weights to obtain HDR (High Dynamic Range) images. The second method is to set up an optical path system inside the camera to divert the light input to the camera to multiple photosensitive devices. These photosensitive devices are set with different exposure times. After taking images with different exposure times, they are weighted according to certain weights to obtain HDR images. The development of deep learning technology has provided new ideas for HDR imaging of traditional cameras. One method is to use a single LDR (Low Dynamic Range) image and design a neural network to predict the lost details in the dark and overexposed parts of the scene; the other method is to follow the idea of ​​complementary different exposure times, design a neural network algorithm to more effectively align images, and obtain more reasonable weighting parameters, thereby further improving the dynamic range and imaging quality of the image. These methods effectively improve the dynamic range of images taken by traditional cameras, but are limited by the performance and exposure mode of traditional cameras themselves, making it difficult to improve the frame rate of images taken by traditional cameras.

[0004] As a new type of high-speed and high-dynamic photosensor, the pulse camera has the ability to capture high dynamic range images in high-speed scenes. The pulse camera uses pulse asynchronous emission technology to continuously integrate the photon signal of each pixel. When the accumulated number of photons reaches a predetermined threshold, a single-bit pulse signal is emitted. The high-speed pulse emission mechanism of the pulse camera enables it to achieve the function of continuous imaging in the time domain, bringing a new breakthrough in the field of photography. Most existing technologies reconstruct images and videos based on single-bit data of pulse cameras. However, since the dynamic range of the pulse camera depends on the number of bits of the pulse data, multi-bit pulses have obvious advantages in expanding the dynamic range of the pulse camera. However, due to the limitation of the transmission bandwidth, multi-bit pulses inevitably reduce the efficiency of data readout.

[0005] In the method disclosed in the document "Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spikestream. In Proc. of Computer Vision and Pattern Recognition, pages 11996–12005, 2021.", an end-to-end neural network structure Spk2ImgNet is designed, which uses a layered architecture to progressively exploit temporal correlation, thereby effectively dealing with motion blur and noise problems; the network uses the pulse stream captured by the pulse camera as input, groups it through a three-layer structure, estimates the light intensity, extracts features, and aligns and fuses to reconstruct high-quality images. The method in the above document only designs a reconstruction algorithm for single-bit pulse data. Due to the limited bit position of the original data, the image or video that can be reconstructed is difficult to have high dynamic range characteristics. Therefore, this method is not designed for video reconstruction and fails to fully take into account the continuity between video frames.

[0006] In the method disclosed in the document "Yakun Chang, Chu Zhou, Yuchen Hong, Liwen Hu, Chao Xu, Tiejun Huang, and Boxin Shi. 1000FPS HDR video with a Spike-RGB hybrid camera. In Proc. of Computer Vision and Pattern Recognition, pages 22180–22190, 2023.", a hybrid camera system is constructed by using a spectroscope to synchronously shoot with a pulse camera and an alternating exposure RGB camera. The method uses the accurate motion information contained in the pulse data to deblur the medium exposure and long exposure video frames, interpolate the lost information between the RGB frames, and finally obtain a high dynamic range color video of 1000 frames per second by fusing the alternating exposure RGB video frames and the pulse reconstruction frames. The performance of the method disclosed in the above document is limited by the frame rate of 60 frames per second of the RGB camera. When the scene moves faster, the deblurring module designed is not robust enough, and the reconstructed video may still be blurred; and the method relies on a spectroscope, which causes the photons reaching the camera to be weakened, which is not conducive to the perception of low-light scenes. Summary of the invention

[0007] The object of the present invention is to provide a method for reconstructing high dynamic range video by high speed rolling mixed bit pattern pulse.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] The present invention provides a method for reconstructing high dynamic range video using a high-speed rolling mixed bit pattern pulse, which specifically comprises the following steps:

[0010] (1) Reconstructing images segment by segment from the captured single-bit pulse stream and multi-bit pulse stream by using a pulse superposition method in the time domain to obtain single-bit pulse reconstructed images and multi-bit pulse reconstructed images;

[0011] (2) using all the information of the single-bit pulse in the time domain and the spatial domain to calculate the optical flow information between the multi-bit pulse frames, using the motion information provided by the optical flow information to correct the deformation of the multi-bit pulse reconstructed image, and using the optical flow information between the multi-bit pulse frames to interpolate the multi-bit pulse reconstructed image;

[0012] (3) Use deep learning methods to fuse single-bit pulse reconstructed images and multi-bit pulse reconstructed images.

[0013] Furthermore, in step (1), the process of acquiring the multi-bit pulse stream is expressed as:

[0014]

[0015] Where S L (x, y, t) represents the multi-bit pulse signal of the pixel at the coordinate (x, y) at the tth time, L represents the quantization level, and the relationship between the quantization level L and the number of bits m is L = 2 m -1, H represents an integer in [0, L-1]; Q c (t) represents the number of accumulated electrons at time t, Q th represents the threshold electron number.

[0016] Furthermore, in step (1), the mathematical expression of the method of superimposing pulses in the time domain is:

[0017]

[0018]

[0019] Among them I s (i) and I m (i) represents the single-bit pulse reconstructed image and the multi-bit pulse reconstructed image, m represents the number of bits, i represents the segment index, τ represents the interval between two frames, n represents the sum index, t i Indicates the starting time point of the i-th segment.

[0020] Furthermore, in step (3), each single-bit pulse reconstructed image and multi-bit pulse reconstructed image is expanded left and right in the time domain to form a reconstructed image sequence:

[0021]

[0022]

[0023] in ω is a positive integer that determines the length of the extension of the reconstructed image sequence.

[0024] Furthermore, in step (3), the corrected reconstructed image sequence is input into a neural network-based encoder to extract features; a two-stage fusion process is designed. In the first stage, the extracted single-bit features and multi-bit features are fused; in the second stage, the time domain sequence is fused and the fused sequence is averaged; finally, a decoder is designed to obtain a reconstructed high-quality image through the fusion result.

[0025] Furthermore, the neural network includes: a feature extraction module, an XBA module, an XTA module and a fusion decoding module; the feature extraction module is used to extract features of single-bit and multi-bit pulse reconstruction image groups respectively; the XBA module is used to spatially associate the extracted features; the XTA module is used to temporally fuse the features; and the fusion decoding module is used to reconstruct the HDR image using the fused features.

[0026] Furthermore, the operation of the XBA module is: represents the connection operation, which connects the features of the single-bit reconstructed image indexed by q and the features of the multi-bit reconstructed image indexed by k. represents the projection matrix composed of multiple layers of convolutional networks, F s (q) represents the feature of the single-bit reconstructed image with index q, F m (k) represents the feature of the multi-bit reconstructed image with index k.

[0027] Furthermore, the operation of the XTA module is: represents the connection operation, connecting the mixed bit feature with index q and the mixed bit feature with index k. represents the projection matrix composed of a multi-layer convolutional network, B(i) represents the bias value, and B(i) is equal to the mixed bit feature sequence F ms (i) Averaging in the time domain.

[0028] Furthermore, the end of the neural network includes a three-layer ConvLSTM structure, which transforms the previous state S in the time domain p (i-1), sequence The average and mixed bit feature F ms (i) Input into the decoder to obtain the reconstructed HDR image I HDR (i).

[0029] Furthermore, the compressed real HDR image G i With the reconstructed HDR image I HDR (i), the compression formula is μ represents the degree of compression; at the same time, the loss function is designed in Take β1=β2=1,φ h Represents the feature extractor of the pre-trained network used in LPIPS, calculates the loss function value of the true value and the predicted value, and then uses the back-propagation algorithm to continuously update the weights of each layer of each module of the network.

[0030] The beneficial effects of the present invention are:

[0031] The method of the present invention for reconstructing high dynamic range video by high-speed rolling mixed bit pattern pulse mainly comprises three steps: preprocessing of pulse signal, upsampling to obtain dense multi-bit pulses and fusing single-bit pulse reconstructed image and multi-bit pulse reconstructed image; each step is realized by designing corresponding modules in a neural network.

[0032] Compared with pure multi-bit pulse cameras, the RMB pulse imaging in the present invention greatly reduces data redundancy, speeds up reading, and retains the high frame rate attribute of the pulse camera. At the same time, the present invention uses single-bit pulse data to estimate optical flow, and uses optical flow to correct the deformation of the multi-bit pulse plane, solving the problem of italics of moving objects caused by the rolling of multi-bit pulse information. The deep learning method used in the present invention designs a network module for extracting single-bit and multi-bit pulse reconstructed image features and fusion, which effectively improves the quality of the fused image.

[0033] Compared with the existing HDR technology based on pulse cameras, the present invention can output 2000FPS HDR video frames, which can better capture the transient changes of high-speed moving objects. At the same time, the present invention introduces a new rolling mixed bit pulse data mode, which can effectively improve the dynamic range of the reconstructed video through rolling multi-bit information while efficiently reading data in high-speed scenes. In addition, the present invention introduces cross-bit attention fusion and cross-time attention fusion modules, successfully expanding the high dynamic range information recorded by rolling multi-bit pulse data to full-frame video frames, and obtaining high-quality high dynamic range video. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flow chart of a method for reconstructing high dynamic range video using high-speed rolling mixed bit pattern pulses provided by the present invention.

[0035] Figure 2 Flowchart of the acquisition and processing of synthetic data for testing. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0037] See also Figure 1 To illustrate, the present invention provides a method for high-speed rolling mixed bit pattern pulse reconstruction of high dynamic range video, which mainly consists of the following steps:

[0038] (1) Preprocessing of pulse signals;

[0039] The pulse camera integrates photons in the time domain and converts them into electrons. When the accumulated electrons exceed the threshold number of electrons, a pulse signal is emitted. The process of pulse signal generation can be expressed by the following formula:

[0040]

[0041] Where S(x, y, t) represents the pulse signal of the pixel at the (x, y) coordinate at the tth moment, Q c (t) represents the number of accumulated electrons at time t, Q th represents the threshold electron number.

[0042] The multi-bit pulse camera can be used to make more detailed quantization processing of the pulse signal. The process of generating a multi-bit pulse signal can be expressed by the following formula:

[0043]

[0044] Where S L (x, y, t) represents the multi-bit pulse signal of the pixel at the coordinate (x, y) at the time t, L represents the quantization level, and its relationship with the number of bits m is L = 2 m -1, H represents an integer in [0, L-1].

[0045] Reconstruct the image segment by segment from the captured single-bit pulse stream and multi-bit pulse stream by superimposing the pulses in the time domain:

[0046]

[0047]

[0048] Among them I s (i) and I m (i) represents the single-bit pulse reconstructed image and the multi-bit pulse reconstructed image, m represents the number of bits, i represents the segment index, τ represents the interval between two frames, n represents the sum index, t i Indicates the starting time point of the i-th segment.

[0049] (2) Upsampling to obtain dense multi-bit spikings;

[0050] RMB (rolling mixed bit) pulse mixes multi-bit pulses line by line within a single-bit pulse. After a scanning period T, a complete frame of multi-bit pulse data is reconstructed. Since the line-by-line scanning is performed asynchronously in the time domain, a jello effect appears in the captured moving objects. The single-bit pulse data has all the information in the time domain and the space domain, and can be used to calculate the optical flow information between multi-bit pulse frames. The motion information provided by the optical flow information is used to correct the deformation of the multi-bit pulse reconstructed image. The optical flow information between the multi-bit pulse frames is then used to interpolate the multi-bit pulse data, thereby obtaining a multi-bit pulse plane with the same number of frames as the single-bit pulse data.

[0051] (3) Fusion of single-bit pulse reconstructed images and multi-bit pulse reconstructed images;

[0052] Each single-bit pulse reconstructed image and multi-bit pulse reconstructed image is expanded left and right in the time domain to form a reconstructed image sequence:

[0053]

[0054]

[0055] in ω is a positive integer that determines the length of the extended reconstructed image sequence. The corrected reconstructed image sequence is input into the neural network-based encoder to extract features. A two-stage fusion process is designed. In the first stage, the extracted single-bit features and multi-bit features are fused; in the second stage, the time domain sequence is fused and the fused sequence is averaged; finally, a decoder is designed to obtain a reconstructed high-quality image through the fusion result.

[0056] In addition, the present invention specially designs a neural network for training, and uses synthetic data for testing, and finally outputs the test results. The specific process of network training, test data collection, image reconstruction and network testing is as follows:

[0057] 1. Training of neural networks;

[0058] a) The entire neural network mainly consists of a feature extraction module, an XBA (Cross-Bit Attention) module, an XTA (Cross-Time Attention) module and a fusion decoding module. Among them, the feature extraction module is mainly used to extract the features of the single-bit and multi-bit pulse reconstruction image groups respectively; the XBA module is mainly used to associate the extracted features in space; the XTA module is mainly used to fuse features in time; and the fusion decoding module is mainly used to reconstruct HDR images using fused features.

[0059] b) Reconstructing the image sequence I from a group of single-bit pulses sand multi-bit pulse reconstruction image sequence I m Input into the encoder, the extracted features correspond to the reconstructed image, respectively recorded as and Among them, F s (j) represents the feature extracted from the single-bit reconstructed image with index j, F m (j) represents the features extracted from the multi-bit reconstructed image with index j.

[0060] c) F s (i) and F m (i) Input to the XBA module, and remove the spatial correlation caused by motion through spatial correlation weighting;

[0061] Among them, the operations in the XBA module are: represents the connection operation, which connects the features of the single-bit reconstructed image indexed by q and the features of the multi-bit reconstructed image indexed by k. represents the projection matrix composed of multiple layers of convolutional networks, F s (q) represents the feature of the single-bit reconstructed image with index q, F m (k) represents the feature of the multi-bit reconstructed image with index k.

[0062] Use the SoftMax layer for normalization, that is, XBA(q)←SoftMax(XBA(q)). Use the normalized weight mask and F m (k) Perform dot product summation to obtain the modulated multi-bit reconstruction features Will and F s (q) are connected together and a 1×1 convolution layer is used to reduce the channel of the feature to obtain the mixed bit feature F ms (i), the mixed bit feature sequence is recorded as

[0063] d) Processing mixed bit feature sequence F ms (i) Mixing in the time domain: Input the mixed bit signature sequence into the XTA module. The operation of the XTA module is similar to that of the XBA: in represents the connection operation, connecting the mixed bit feature with index q and the mixed bit feature with index k. represents the projection matrix composed of a multi-layer convolutional network, B(i) represents the bias value, and B(i) is equal to the mixed bit feature sequence F ms (i) Averaging in the time domain.

[0064] Similar to the operation in the XBA module, XTA(q) is normalized using the SoftMax layer and compared with F ms (k) Do the dot product sum: The corresponding sequence is obtained as

[0065] e) A three-layer ConvLSTM structure is designed at the end of the neural network to convert the previous state S in the time domain into p (i-1), sequence The average and F ms (i) Input into the decoder to obtain the reconstructed HDR image I HDR (i).

[0066] f) Compress the real HDR image G i and network prediction value I HDR (i), the compression formula is μ represents the degree of compression; design loss function in Take β1=β2=1,φ h Represents the feature extractor of the pre-trained network used in LPIPS, calculates the loss function value of the true value and the predicted value, and then uses the back-propagation algorithm to continuously update the weights of each layer of each module of the network.

[0067] 2. Test the collection and processing of synthetic data. The specific operation process is as follows: Figure 2 As shown;

[0068] a) Capture single-bit spikings: Use a 1000×1000 pixel pulse camera, set the capture time to 1 second, and the frequency to 20,000 Hz to capture high-speed motion scenes.

[0069] b) Synthesized RMB pulse data (rolling-mixed-bit spikings): Take 10 sets of real single-bit pulse data streams with a size of 1000×1000, divide the pulses into 2×2×2 (height×width×time) pixel boxes, sum the 8 pixel pulse values ​​to get 3-bit data, and get single-bit data from one of the specific pixels. Scroll down row by row, take 4 consecutive rows as 3-bit pulse data rows, and the other rows as single-bit pulse data rows, and get 10 sets of RMB pulse stream data with a size of 500×500.

[0070] A mixed method of replacing multi-bit pulses may also be used, such as replacing the row-by-row scrolling method with column-by-column scrolling, or changing the number of rows scrolled each time and the scrolling cycle.

[0071] c) Preprocessing RMB pulse stream data: Take the scanning period as T = 12.5ms, every 125 frames of single-bit pulse stream correspond to one frame of 3-bit pulse stream, use optical flow information to interpolate the 3-bit pulse stream, and insert 124 frames between every two frames to be consistent with the number of single-bit pulse data.

[0072] 3. Image reconstruction;

[0073] Take N=10, the scanning period is T=12.5ms, and reconstruct the single-bit pulse image and the multi-bit pulse image. Take N=10 to superimpose the pulse data, estimate the optical flow, correct the italic effect in the multi-bit pulse reconstructed image, and insert the multi-bit pulse reconstructed image to match the number of single-bit pulse images. Take ω=3 to expand each reconstructed image in both directions to obtain a reconstructed image sequence, and finally obtain an extended single-bit pulse reconstructed image sequence and a multi-bit pulse reconstructed image sequence, each group of 7 reconstructed images, containing 70 frames of pulse data information.

[0074] 4. The above-mentioned extended single-bit pulse reconstructed image sequence and multi-bit pulse reconstructed image sequence are input into a trained neural network (such as RMB-Net). The features are first extracted through a shared encoder, and then the spatial and temporal correlation problems of the feature sequences are processed through modules such as XBA and XTA. Finally, high-frame-rate and high-resolution HDR and HFR videos are output through a decoder.

[0075] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for reconstructing high dynamic range video using high-speed rolling mixed bit pattern pulses, characterized in that: The following steps are involved: (1) Reconstructing images segment by segment from the captured single-bit pulse stream and multi-bit pulse stream by using a pulse superposition method in the time domain to obtain single-bit pulse reconstructed images and multi-bit pulse reconstructed images; (2) using all the information of the single-bit pulse in the time domain and the spatial domain to calculate the optical flow information between the multi-bit pulse frames, using the motion information provided by the optical flow information to correct the deformation of the multi-bit pulse reconstructed image, and using the optical flow information between the multi-bit pulse frames to interpolate the multi-bit pulse reconstructed image; (3) Use deep learning methods to fuse single-bit pulse reconstructed images and multi-bit pulse reconstructed images.

2. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 1, characterized in that: In step (1), the process of acquiring the multi-bit pulse stream is as follows: The pulse camera integrates photons in the time domain and converts them into electrons. When the accumulated electrons exceed the threshold number of electrons, a pulse signal is emitted. The process of pulse signal generation is expressed as: Where S(x,y,t) represents the pulse signal of the pixel at the (x,y) coordinate at the tth moment, Q c (t) represents the number of accumulated electrons at time t, Q th represents the threshold electron number; The pulse signal is quantized by a multi-bit pulse camera to generate a multi-bit pulse signal. This process is expressed as: Where S L (x, y, t) represents the multi-bit pulse signal of the pixel at the coordinate (x, y) at the time t, L represents the quantization level, and the relationship between the quantization level L and the number of bits m is L = 2 m -1, H represents an integer in [0, L-1].

3. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 2, characterized in that: In step (1), the mathematical expression of the method of superimposing pulses in the time domain is: Among them I s (i) and I m (i) represents the single-bit pulse reconstructed image and the multi-bit pulse reconstructed image, m represents the number of bits, i represents the segment index, τ represents the interval between two frames, n represents the sum index, and t i Indicates the starting time point of the i-th segment.

4. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 3, characterized in that: In step (3), each single-bit pulse reconstructed image and multi-bit pulse reconstructed image is expanded left and right in the time domain to form a reconstructed image sequence: Among them I s (i) and I m (i) represents the single-bit pulse reconstruction image and the multi-bit pulse reconstruction image, respectively. ω is a positive integer that determines the length of the extension of the reconstructed image sequence.

5. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 4, characterized in that: In step (3), the corrected reconstructed image sequences are respectively input into a neural network-based encoder to extract features; A two-stage fusion process is designed. In the first stage, the extracted single-bit features and multi-bit features are fused; In the second stage, the time domain sequences are fused and the fused sequences are averaged; finally, a decoder is designed to obtain a reconstructed high-quality image through the fusion results.

6. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 5, characterized in that: The neural network includes: a feature extraction module, an XBA module, an XTA module and a fusion decoding module; the feature extraction module is used to extract the features of the single-bit and multi-bit pulse reconstruction image groups respectively; the XBA module is a cross-bit attention mechanism module, which is used to spatially associate the extracted features; the XTA module is a cross-time attention mechanism module, which is used to temporally fuse the features; the fusion decoding module is used to reconstruct the HDR image using the fused features.

7. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 6, characterized in that: The operations of the XBA module are: represents the connection operation, which connects the features of the single-bit reconstructed image indexed by q and the features of the multi-bit reconstructed image indexed by k. represents the projection matrix composed of multiple layers of convolutional networks, F s (q) represents the feature of the single-bit reconstructed image with index q, F m (k) represents the feature of the multi-bit reconstructed image with index k.

8. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 6, characterized in that: The operations of the XTA module are: represents the connection operation, connecting the mixed bit feature with index q and the mixed bit feature with index k. represents the projection matrix composed of a multi-layer convolutional network, B(i) represents the bias value, and B(i) is equal to the mixed bit feature sequence F ms (i) Averaged in the time domain, the mixed bit feature sequence is recorded as 9. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 6, characterized in that: The end of the neural network includes a three-layer ConvLSTM structure, which transforms the previous state S in the time domain p (i-1), sequence The average and mixed bit signature sequence F ms (i) Input into the decoder to obtain the reconstructed HDR image I HDR (i); XTA(q) is normalized using the SoftMax layer and compared with F ms (k) Do the dot product sum: The corresponding sequence is obtained as The mixed bit feature sequence is recorded as 10. The method for reconstructing high dynamic range video using high speed rolling mixed bit pattern pulse according to claim 9, characterized in that: Compressed real HDR image G i With the reconstructed HDR image I HDR (i), the compression formula is μ represents the degree of compression; at the same time, the loss function is designed in Take β1=β2=1,φ h Represents the feature extractor of the pre-trained network used in LPIPS, calculates the loss function value of the true value and the predicted value, and then uses the back-propagation algorithm to continuously update the weights of each layer of each module of the network.

Images