A super-resolution enhancement method for motion images by fusing pulse data and frame data

Abstract

A method for super-resolution improvement of a motion image by fusing pulse data and frame data, comprising: S1. Data input; S2. Synchronous data: Synchronize pulse data and frame data according to timestamps, the synchronized timestamps are based on frame data, and the pulse data contains all data from the previous frame to the current frame; S3. Pulse optical flow: Extract the optical flow features of pulse data; S4. Frame data optical flow: Extract the optical flow features of frame data; S5. Data correction: Use the checkerboard information data obtained in step S1 to correct the pulse data after calculation; S6. Data fusion: Fuse the optical flow features corrected in step S5; S7. Generate recovery: Restore the data after data fusion in step S6 to high frame rate and high quality frame data; S8. Data output: Output high frame rate video data. The method of the present invention improves the frame rate of traditional cameras through an artificial intelligence fusion method, so that it can have a good presentation effect on high-speed moving targets.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a super-resolution enhancement method for a motion image by fusing pulse data and frame data. Background Art

[0002] As a new type of camera technology, pulse camera mainly provides pulse signals by sensing changes in the environment. Thanks to its high-speed acquisition rate and wide dynamic capability, it has been widely used in the acquisition of moving targets and other fields. Traditional cameras can provide relatively high-resolution frame data, but the main problem is that traditional cameras are affected by settings such as shutter exposure and cannot capture moving targets well, especially high-speed targets, which often leads to motion blur. Although there are professional high-speed cameras that can capture high-speed targets, most high-speed cameras are expensive, and high-speed cameras will generate a lot of data redundancy and fixed frame rate. The difficulty of solving the above problems and defects is: It is very difficult to solve this problem with traditional methods. This is an inherent defect based on the principle of traditional cameras. When capturing moving targets, the use of global shutter will inevitably bring a lot of redundancy of background information.

[0003] There is a big difference between pulse data and frame data in data expression. Pulse cameras output asynchronous pulse signals with high time resolution, while traditional cameras output continuous frame data. It is of great significance to effectively integrate the two. The significance of solving the above problems and defects is: it can effectively reduce the data redundancy caused by high-speed target acquisition, and complete the data acquisition of high-speed moving targets with less data volume. Summary of the invention

[0004] The present invention provides a super-resolution enhancement method for motion images by fusing pulse data and frame data, which improves the frame rate of a traditional camera through an artificial intelligence fusion method, so that it can have a good presentation effect on high-speed moving targets.

[0005] The technical solution of the present invention is as follows:

[0006] The super-resolution enhancement method of a motion image by fusing pulse data and frame data of the present invention comprises the following steps: S1. data input: a pulse camera and a conventional camera are fixed on the same plane, dynamic correction checkerboard information data are obtained each time they are used, and pulse data and frame data are output simultaneously; S2. data synchronization: pulse data and frame data are synchronized according to a timestamp, the synchronized timestamp is based on the frame data, and the pulse data contains all data in a time period from a previous frame to a current frame; S3. pulse optical flow: optical flow features of pulse data are extracted; S4. frame data optical flow: optical flow features of frame data are extracted; S5. data correction: the checkerboard information data obtained in step S1 is used to correct the pulse data after calculation so that it can be mapped to the coordinate system of the frame data; S6. data fusion: the optical flow features corrected in step S5 are fused; S7. generation and recovery: the data after data fusion in step S6 is restored to high frame rate and high quality frame data; S8. data output: high frame rate video data is output.

[0007] Preferably, in the above-mentioned method for super-resolution enhancement of a moving image by fusing pulse data and frame data, in step S2, the synchronization data includes time alignment of the pulse data and the frame data, and space alignment of the pulse data and the frame data.

[0008] Preferably, in the above-mentioned super-resolution enhancement method of a motion image that fuses pulse data and frame data, in step S2, the time alignment of the pulse data and the frame data is achieved by setting an absolute timestamp, that is, the data at the current time of the frame data and all the data from this moment of the pulse data to the previous frame time are data at the same moment.

[0009] Preferably, in the above-mentioned super-resolution enhancement method of a motion image that fuses pulse data and frame data, in step S2, the pulse data and frame data are spatially aligned by a moving checkerboard. First, the checkerboard is set to a circle and rotated uniformly at a certain speed. Data extraction is performed according to the requirements of time alignment for 1 s. The pulses are mapped into image frames at time intervals of 20 ms. The candidate coordinate systems of the two cameras are calculated for 1 s of data using the checkerboard, and the candidate coordinate systems of each camera are averaged. The mean result is used as the final output of the two coordinate systems, and a mapping relationship between the two coordinate systems is established based on the coordinate systems.

[0010] Preferably, in the above-mentioned super-resolution enhancement method of motion images that fuses pulse data and frame data, in step S3, collection is performed in the form of a sliding window. When the pulse camera outputs a pulse signal, the signal is accumulated in a matrix window. When a certain number is accumulated, the window is saved and accumulated again. When the new accumulation is completed, the L1-optical flow features of the two adjacent windows are calculated, and so on.

[0011] Preferably, in the above-mentioned method for super-resolution enhancement of a moving image by fusing pulse data and frame data, in step S4, the optical flow is directly extracted from adjacent frame data using the L1-optical flow method.

[0012] Preferably, in the above-mentioned super-resolution enhancement method of a motion image that fuses pulse data and frame data, in step S6, after the frame data and pulse data are respectively mapped to the optical flow feature space, the optical flow feature contains motion vector information, and before data fusion, the pulse data and frame data in step S2 need to be time-aligned and space-aligned. The time alignment corresponds the data of the pulse camera time period to the data of the frame camera time point. The space alignment requires a coordinate system for mapping. The optical flow features of the pulse camera are mapped to the coordinate system under the frame data through the conversion of the two coordinate systems. Then, the fusion can be completed by a direct addition method, and the optical flow features of the pulse data are superimposed on the optical flow features of the frame data.

[0013] Preferably, in the above-mentioned super-resolution enhancement method of motion images that fuses pulse data and frame data, in step S7, the fused optical flow features are used to repair the original video, and the fused optical flow features and the source video frames are simultaneously input into the trained generative adversarial network, and the corresponding output parameters are set. The network generates and adds frames to the video frames through the optical flow features to restore the original video.

[0014] Preferably, in the above-mentioned method for super-resolution enhancement of a moving image by fusing pulse data and frame data, in step S8, the data generated in step S7 is encoded according to AVS3 and then output.

[0015] According to the technical solution of the present invention, the beneficial effects produced are:

[0016] The super-resolution enhancement method of motion images by fusing pulse data and frame data of the present invention can effectively utilize pulse cameras to assist ordinary cameras in achieving the capture effect of high-speed targets, thereby achieving the capture of high-speed targets at a relatively low cost. The present invention uses relatively low-cost pulse cameras and ordinary cameras to achieve the capture of high-speed targets, generate output and input videos of arbitrary frame rates, and improve the capture performance of traditional cameras.

[0017] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0019] Figure 1It is a flow chart of the method for super-resolution enhancement of a moving image by fusing pulse data and frame data of the present invention;

[0020] Figure 2 The present invention relates to a method for acquiring frame data and a pulse camera;

[0021] Figure 3 It is the chess and card grid used for calibration in the method of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0023] In video data acquisition, problems such as blurred images caused by rapid changes in light, fast target speed, or camera shake are often encountered. To address this problem, the present invention proposes two modal fusion and repair methods. With the assistance of a pulse camera, traditional cameras can effectively solve this problem, thereby obtaining clear high-frame rate images under harsh conditions.

[0024] The method for super-resolution enhancement of motion images by fusing pulse data and frame data of the present invention is a method for super-resolution enhancement that effectively fuses pulse data and frame data. It mainly uses a multimodal fusion method of a traditional camera and a pulse camera to collect motion data, giving full play to the advantages of two types of modal data (i.e., pulse data and frame data), and using pulse data to repair and improve the frame data image. Specifically, by extracting optical flow features from both types of data, pulse data and frame data, and after superimposing and fusing the optical flow feature layer, the original video is repaired using the fused optical flow features, thereby completing the process of upgrading a video with a normal frame rate to a video with any high frame rate.

[0025] The working principle of the present invention is: the present invention fuses high-speed pulse data and frame data, and the fused data will be presented in the form of high frame rate. It mainly adopts four main steps: data alignment, optical flow conversion, optical flow fusion and super-resolution recovery. After the two modal data are aligned, they are converted into the form of optical flow features for fusion. The fused data performs super-resolution repair on the original frame data, so that the original video can be upgraded from low frame rate to high frame rate data.

[0026] like Figure 1 As shown, the super-resolution enhancement method of a motion image that fuses pulse data and frame data includes the following steps:

[0027] S1. Data input: Fix the pulse camera and the traditional camera on the same plane, obtain dynamic correction checkerboard information data each time they are used, and output the pulse data and frame data at the same time. Figure 2 These are two different ways of acquiring data by the two modal sensors involved in the present invention. The left side shows the image of continuous frames acquired by a traditional camera, and the right side shows the output data of a pulse camera. Each pixel of the pulse camera can be asynchronously triggered at high speed.

[0028] S2. Synchronize data: Synchronize the two modal data (pulse data and frame data) according to the timestamp. The synchronized timestamp is based on the frame data, that is, the frame data contains the image of the current frame, and the pulse data contains all the data from the previous frame to the current frame. Synchronize data includes the time alignment of pulse data and frame data, and the spatial alignment of pulse data and frame data. Since frame data and pulse data belong to two modal data, there are significant differences in the two data structures. When the data is projected onto the optical flow feature layer for fusion, to achieve a good fusion effect, the data of the two modalities need to be aligned in time and space.

[0029] Data alignment is divided into spatial alignment and temporal alignment.

[0030] The time alignment scheme is: the time alignment of pulse data and frame data is achieved by setting an absolute timestamp, that is, the data of the current time of the frame data and all the data of the pulse data from this moment to the previous frame are the same moment data. For example, if the timestamp of the current frame is 1648623012.207346 and the timestamp of the previous frame is 1648623006.0445027, then the pulse data corresponding to the current frame contains all the data between 1648623006.0445027 and 1648623012.207346.

[0031] The spatial alignment scheme is as follows: the pulse data and frame data are spatially aligned by moving the checkerboard (such as Figure 3 As shown in the figure, since the pulse camera can only capture moving targets, the traditional static correction is no longer applicable. Therefore, the present invention proposes a dynamic chessboard. First, the chessboard is set to a circle and rotated at a constant speed. Data extraction is performed according to the requirements of time alignment. The time lasts for 1 second. The pulse is mapped into an image frame at a time interval of 20ms. The candidate coordinate systems of the two cameras are obtained using the chessboard for the 1 second data, and the candidate coordinate systems of each camera are averaged. The average result is used as the final output of the two coordinate systems, and the mapping relationship between the two coordinate systems is established based on the coordinate system.

[0032] S3. Pulse optical flow: Extract the optical flow features of pulse data. Since the pulse signal is a binary signal, it cannot be directly converted into an optical flow feature, and the pulse camera only has feedback for moving targets. The present invention collects in the form of a sliding window. When the pulse camera outputs a pulse signal, the signal is accumulated in a matrix window. When a certain number is accumulated, the window is saved and accumulated again. When the new accumulation is completed, the L1-optical flow features of the two adjacent windows are calculated, and so on.

[0033] S4. Frame data optical flow: extract the optical flow features of the frame data. For the frame data, directly use the L1-optical flow method to extract the optical flow of adjacent frame data.

[0034] S5. Data correction: Using the checkerboard information data obtained in step S1, the pulse data is corrected through calculation so that it can be mapped to the coordinate system of the frame data.

[0035] S6. Data fusion: Fuse the optical flow data (i.e., optical flow features) corrected in step S5. Specifically, after the frame data and pulse data are mapped to the optical flow feature space respectively, the optical flow features contain motion vector information. Because they are two modal data obtained from different angles, the pulse data and frame data in step S2 need to be aligned before data fusion. The alignment is divided into time alignment and space alignment. Time alignment corresponds the data of the pulse camera time period to the data of the frame camera time point. In space alignment, a coordinate system is required for mapping. The optical flow features of the pulse camera are transformed through the two coordinate systems, and the coordinates are mapped to the coordinate system under the frame data. Then, the fusion can be completed by using the direct superposition method, that is, the optical flow features of the pulse data are superimposed on the optical flow features of the frame data.

[0036] S7. Generate and restore: The fused data, i.e., the data fused in step S6, is restored to high frame rate and high quality frame data. The fused optical flow features are used to repair the original video. The fused optical flow features and the source video frames are simultaneously input into the trained generative adversarial network, and the corresponding output parameters are set. The network generates and adds frames to the video frames through the optical flow features to restore the original video.

[0037] S8. Data output: output high frame rate video data. Specifically, the data generated in step S7 is encoded according to AVS3 and then output, and the output video data has the characteristics of high frame rate and high resolution.

[0038] The present invention uses two evaluation methods to evaluate the method proposed by the present invention. PNSR, SSIM and VMAF are commonly used in the field for image quality evaluation. In a high-speed scene, 50 1-minute videos are shot by the method of the present invention at the same time, and these videos are fused. 500 pictures are extracted from the fused videos for evaluation, that is, 10 pictures are extracted from each video, and these pictures are evaluated. The comparison method includes simple superposition fusion Map1, that is, directly superimposing event data and frame data for fusion, and proportional fusion method Map2, that is, fusion is performed according to the weight ratio, event data accounts for 20%, and frame data accounts for 80%. The following Table 1 shows the comparison results, and the larger the value, the better.

[0039] Table 1

[0040] method PNSR SSIM VMAF Map1 23.74214515 0.929508289 37.6672338 Map2 22.82433229 0.813766524 22.58379643 The present invention 34.1826 0.946972 59.2076329

[0041] The present invention proposes a dual-modal video super-resolution method, one modality is pulse camera data, the other modality is traditional camera data, and the data of the two modalities are fused by an artificial intelligence method, specifically, the two modalities are mapped to features of the same data type, namely optical flow features for fusion, and the fused features are used to repair the original camera modality data to obtain a restored high-speed video. Since the temporal resolution and dynamic perception ability of pulse data are much greater than those of traditional cameras, but the spatial resolution is very low, the fused and repaired video has the characteristics of high temporal resolution, wide dynamic range, and high spatial resolution.

[0042] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for super-resolution enhancement of a moving image by fusing pulse data and frame data, characterized in that: The following steps are involved: S1. Data input: Fix the pulse camera and the traditional camera on the same plane, obtain dynamic correction checkerboard information data each time they are used, and output the pulse data and frame data at the same time; S2. Synchronizing data: synchronizing the pulse data and the frame data according to the timestamp, the synchronized timestamp is based on the frame data, and the pulse data contains all the data from the previous frame to the current frame time period; S3. Pulse optical flow: extracting the optical flow features of the pulse data; S4. Frame data optical flow: extracting optical flow features of the frame data; S5. Data correction: using the checkerboard information data obtained in step S1, the optical flow characteristics of the pulse data are corrected by calculation so that it can be mapped to the coordinate system of the frame data; S6 data fusion: step S5 corrected pulse data optical flow features fusion; S7. Generate and restore: The data after data fusion in step S6 is restored to high frame rate and high quality frame data, and the original video is repaired using the fused optical flow features. The fused optical flow features and the source video frames are simultaneously input into the trained generative adversarial network, and the corresponding output parameters are set. The network generates and adds frames to the video frames through the optical flow features to restore the original video; S8. Data output: Output high frame rate video data.

2. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 1, characterized in that: In step S2, the synchronization data includes time alignment of the pulse data and the frame data, and space alignment of the pulse data and the frame data.

3. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 2, characterized in that: In step S2, the time alignment of the pulse data and the frame data is achieved by setting an absolute timestamp, that is, the data at the current time of the frame data and all the data from this moment of the pulse data to the previous frame time are data at the same moment.

4. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 2, characterized in that: In step S2, the pulse data and the frame data are spatially aligned by moving a chessboard. First, the chessboard is set to a circle and rotated at a constant speed. Data extraction is performed according to the requirements of time alignment. The time lasts for 1 second. The pulses are mapped to image frames at time intervals of 20 ms. The candidate coordinate systems of the two cameras are calculated for the 1 second data using the chessboard, and the candidate coordinate systems of each camera are averaged. The mean result is used as the final output of the two coordinate systems, and the mapping relationship between the two coordinate systems is established based on the coordinate systems.

5. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 1, characterized in that: In step S3, the data is collected in the form of a sliding window. When the pulse camera outputs a pulse signal, the signal is accumulated in a matrix window. When a certain number of signals are accumulated, the window is saved and accumulated again. When the new accumulation is completed, the L1-optical flow features of the two adjacent windows are calculated, and so on.

6. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 1, characterized in that: In step S4, the L1-optical flow method is directly used to extract the optical flow from the adjacent frame data.

7. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 2, characterized in that: In step S6, after the frame data and the pulse data are respectively mapped to the optical flow feature space, the optical flow feature contains motion vector information. Before data fusion, the pulse data and frame data in step S2 need to be time-aligned and space-aligned. The time alignment corresponds the data of the pulse camera time period to the data of the frame camera time point. The space alignment requires a coordinate system for mapping. The optical flow features of the pulse camera are mapped to the coordinate system under the frame data through the conversion of the two coordinate systems. Then, the fusion can be completed by directly adding the optical flow features of the pulse data, and the optical flow features of the pulse data are superimposed on the optical flow features of the frame data.

8. The method for super-resolution enhancement of a moving image by fusing pulse data and frame data according to claim 1, characterized in that: In step S8, the data generated in step S7 is encoded according to AVS3 and output.

Images