Pulse domain image distortion correction method based on pulse camera and FPGA acceleration method

Abstract

The invention discloses a pulse domain image distortion correction method based on a pulse camera and an FPGA acceleration method, and the method comprises the steps: carrying out the calibration of a corner camera through a checkerboard image obtained through the shooting of the pulse camera, obtaining a reverse mapping coordinate matrix through calculation, directly carrying out the position and pixel value mapping of single-bit pulse data obtained through real-time sampling in a pulse domain, and carrying out the correction of the distortion of the single-bit pulse data. Therefore, image distortion correction is completed. The FPGA hardware acceleration structure comprises a source pulse frame memory RAM, a mapping coordinate matrix memory ROM, a mapping coordinate calculation module and a corrected pulse frame memory RAM; and an FPGA hardware acceleration structure is utilized to realize pulse domain image distortion correction, and the characteristics of high frame rate and high real-time performance of the pulse camera are fully exerted.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to image distortion correction and acceleration optimization technologies, and particularly relates to a method for correcting image distortion in the pulse domain based on a pulsed camera and an FPGA acceleration method. Background Art

[0002] Image distortion widely exists in many imaging devices in our daily life, and these distortions will have a serious impact on many vision applications, especially in scenarios with high precision requirements, such as computer vision, robot vision, and measurement systems.

[0003] As a new type of camera that simulates the encoding of the brain retina and records spatio-temporal light intensity information by independent photosensing and asynchronous triggering of each pixel, the sampling frequency of the pulsed camera can reach up to 40,000 Hz at most. However, the images captured by it will inevitably have a certain degree of distortion. Distortion is likely to mislead the human eye's discrimination and will also have a certain adverse impact on subsequent vision processing such as detection and tracking.

[0004] When correcting image distortion in the traditional image domain, the pixel value expression of each pixel point is at least 8 bits. If high-resolution and high-frame-rate image requirements are involved, the required high bandwidth and a large amount of storage resources involved are difficult to meet in an actual hardware system, thus affecting the performance of image resolution and frame rate real-time performance, etc. Summary of the Invention

[0005] In order to address the deficiencies of the above-mentioned existing technologies, the present invention provides a method for correcting image distortion in the pulse domain based on a pulsed camera and an FPGA acceleration design method. By proposing a new method for real-time distortion correction of images in the pulse domain, distortion correction is directly performed in the pulse domain where each pixel point is expressed as a single bit. At the same time, hardware acceleration design is carried out for implementation and application on the FPGA, overcoming the limitations in aspects such as the processing speed and storage resources of the actual hardware system of the pulsed camera during correction, and giving full play to its characteristics of high frame rate and high real-time performance.

[0006] The technical solution of the present invention is as follows:

[0007] A method for correcting image distortion in the pulse domain based on a pulsed camera includes the following steps:

[0008] 1) Obtain a checkerboard image through a pulsed camera;

[0009] Each pixel point within the resolution range of the pulsed camera independently collects light intensity. The pulsed camera samples the light intensity information of all pixel points at each fixed interval time, accumulates the light intensity, and issues asynchronous single-bit pulse data at different speeds. The image is calculated through an image reconstruction algorithm according to the pulse data transmission rate. Therefore, capture the checkerboard pattern and reconstruct the image;

[0010] 2) Use a checkerboard image for corner camera calibration to obtain camera parameters;

[0011] 3) Combine the distortion model to obtain the inverse mapping coordinate matrix;

[0012] Due to central symmetry, the inverse mapping coordinate matrix can be calculated only for a quarter-resolution range of the entire image.

[0013] 4) The pulsed camera samples to obtain single-bit pulsed data, and directly performs distortion correction in the 1-bit pulse domain expressed at each pixel point;

[0014] For each pixel position in the pulsed data frame within the resolution range, search for the coordinates provided by the inverse mapping matrix, perform rearrangement from the source pulse frame to the corrected pulse frame, and directly obtain the pixel value at the pixel position.

[0015] 5) Calculate the corrected image by using an image reconstruction algorithm for the corrected pulse frame;

[0016] Each pixel point in the pulse array after position rearrangement independently records its own time-domain information and then emits a pulse, and the corrected image is obtained after reconstruction.

[0017] This method makes full use of the sampling characteristics of the pulsed camera, can reduce the pixel value expression and control bandwidth at each pixel point, and the distortion correction is clear and simple, and can well play the high frame rate feature of the pulsed camera.

[0018] The present invention also provides an FPGA hardware acceleration structure for the above-mentioned pulsed domain image distortion correction method based on a pulsed camera, including: a source pulse frame memory RAM, a mapping coordinate matrix memory ROM, a mapping coordinate calculation module, and a corrected pulse frame memory RAM. In order to make full use of the precious on-chip resources on the FPGA, the present invention divides the source pulse frame into small blocks in a region-by-region manner, and stores them in their respective memory RAMs respectively, so that the regions can perform point-by-point correction in parallel, improving the frame rate of the correction process. The mapping coordinate matrix memory ROM and the corrected pulse frame storage RAM are also reasonably divided accordingly, so that the source pulse frame RAM, the mapping matrix ROM, and the corrected pulse frame RAM after region division correspond to each other, and parallel processing of distortion correction is performed. The mapping coordinate matrix memory ROM is used to store the mapping coordinate matrix, and the mapping coordinate matrix adopts the form of relative coordinates and base addresses within the mapping partition. The mapping coordinate calculation module is responsible for calculating and converting the relative coordinates and base addresses within the mapping partition into the relative coordinates within the corresponding partition of the source pulse frame, and obtaining the mapping coordinates of the remaining three quarter regions, fully saving the storage resources of the FPGA. Pixel-level pipelining, inter-module pipelining mode, and ping-pong operation are also adopted to achieve real-time processing and output in a block-parallel manner. The specific steps of the FPGA acceleration method are as follows:

[0019] 1) In state0, the source pulse frames are input row by row and stored in the pingRAM of the segmented source pulse frame memory RAM under the condition of meeting the row range condition until the entire pingRAM stores a whole source pulse frame, that is, the previous frame.

[0020] 2) In statel, the source pulse frames are input and a whole frame is stored in the pongRAM of the segmented source pulse frame memory RAM, that is, the next frame. At the same time, the mapping coordinates stored in the mapping coordinate matrix memory ROM are sequentially searched and retrieved, and the remaining three quarter-region mapping coordinates are calculated through processing by the mapping coordinate calculation module. Then, the single-bit pulse values at the mapping coordinate positions of the previous frame in the source pulse frame pingRAM are read out and written in the pingRAM of the corrected pulse frame memory RAM to complete the distortion correction of the previous frame.

[0021] 3) In state2, the source pulse frames are input into the pingRAM of the source pulse frame RAM to overwrite the original data and become the new previous frame. At the same time, the next frame in the source pulse frame pongRAM is corrected, and the previous frame in the corrected pulse frame pingRAM is output row by row.

[0022] 4) In state3, the source pulse frames are input into the pongRAM of the source pulse frame RAM to overwrite the original data and become the new next frame. The previous frame in the source pulse frame pingRAM is corrected, and the next frame of the corrected pulse frame RAM is output. After that, it always jumps between state2 and state3 to achieve seamless real-time processing.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] (1) The present invention proposes a method for pulse-domain image distortion correction based on a pulse camera, which makes full use of the characteristics of the pulse camera, significantly reduces the number of bits involved in each pixel point, relieves the bandwidth pressure, and ensures the high frame rate of the image.

[0025] (2) The present invention proposes an FPGA acceleration method for pulse-domain distortion correction. The characteristics of the pulse domain and the sub-region design greatly reduce the storage requirements for the input source pulse frames and the mapping matrix, promoting the deployment of the actual application of the hardware system; the pixel-by-pixel correction processing pipeline, the high parallelism of the correction processing between regions and between modules, and the ping-pong operation between modules greatly improve the real-time performance and processing speed without affecting the correction effect, fully retaining the advantages of the pulse camera with a high sampling frame rate and good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1It is the block diagram of the pulse domain image distortion correction method based on a pulsed camera provided by the present invention.

[0027] Figure 2 It is the block diagram of the FPGA hardware acceleration for pulse domain image distortion correction provided by the present invention. Specific embodiments

[0028] The present invention will be further described below in conjunction with the accompanying drawings through embodiments, but it is not limited to the scope of the present invention in any way.

[0029] The present invention proposes a new pulse domain image distortion correction method, which makes full use of the characteristics of the pulsed camera, reduces the bandwidth pressure while ensuring the correction effect, and realizes the FPGA hardware acceleration design, saving storage space and improving the real-time performance and processing speed.

[0030] Reference Figure 1 , the block diagram of the pulse domain image distortion correction method based on a pulsed camera of the present invention is given. The pulse pixel points of the pulsed camera are independent and will asynchronously collect the light intensity at their respective pixel positions. Use the pulsed camera to take pictures of the checkerboard, calibrate the camera corner points using the obtained checkerboard image, and calculate and store the inverse mapping coordinate matrix of the upper left quarter offline in combination with the distortion model. The mapping coordinates of the remaining three quarters are obtained through mapping coordinate calculation. When the pulsed camera samples in real time, for each pixel position in the pulsed data frame within the resolution range, search for the coordinates provided by the inverse mapping matrix, rearrange from the input source pulsed frame to the corrected pulsed frame, and directly obtain the pixel value at the pixel position. Then, through the image reconstruction algorithm, the corrected image can be calculated from the corrected pulsed frame for display and subsequent processing.

[0031] Reference Figure 2, for the FPGA implementation, the hardware acceleration design of the proposed method is completed. This figure shows the block diagram of the FPGA hardware acceleration for pulse domain distortion correction. The FPGA hardware acceleration structure includes: a source pulse frame memory RAM, a mapping coordinate matrix memory ROM, a mapping coordinate calculation module, and a corrected pulse frame memory RAM. During the inverse mapping of distortion correction, a whole frame of the input source pulse frame is required. Therefore, a whole frame of single-bit source pulse frame must be cached. The resolution of the pulse camera is 1920*1080, and the sampling frequency in the pulse domain can be up to nearly 20000Hz, that is, it will collect and receive a huge amount of data of up to nearly 20000 frames per second. Without changing the clock frequency and using the traditional per-pixel serial correction, the number of frames of the corrected pulse frame will decrease by hundreds of times, which is a great waste of the high sampling rate characteristic of the pulse camera. Therefore, the camera frame rate is guaranteed by dividing the area and increasing the parallelism. After partitioning, the minimum coordinate in the mapped absolute coordinates within each area is used as the partition base coordinate, and all the mapped coordinates within each area are subtracted from the corresponding area's partition base coordinate to obtain the new partition relative coordinates. The relative coordinates within each partition are stored in the corresponding number of ROMs, and the partition base coordinate matrix is stored in 1 ROM. And the bit width represented by the relative coordinates within each partition can be further reduced. At the same time, the source pulse frame also needs to be divided into areas. Similarly, the source pulse image base coordinates of each area can be obtained, and the difference between the mapped absolute coordinates and the source pulse image base coordinates is used as the relative coordinates within the corresponding partition of the source pulse image for addressing and reading. In state0, the source pulse frame is input row by row and stored in the pingRAM of the segmented source pulse frame memory RAM when the row range condition is met until the entire pingRAM stores a whole source pulse frame, that is, the previous frame. In state1, the source pulse frame is input and a whole frame is stored in the pongRAM of the segmented source pulse frame memory RAM, that is, the next frame. At the same time, the mapped coordinates stored in the mapping coordinate matrix memory ROM are sequentially searched and retrieved, and the mapped coordinates of the remaining three quarter areas are processed and calculated through the mapping coordinate calculation module. Then, the single-bit pulse value at the mapped coordinate position of the previous frame in the source pulse frame pingRAM is read out and written in the pingRAM of the corrected pulse frame memory RAM to complete the distortion correction of the previous frame. In state2, the source pulse frame is input to the pingRAM of the source pulse frame RAM to overwrite the original data and become the new previous frame. At the same time, the next frame in the source pulse frame pongRAM is corrected, and the previous frame in the corrected pulse frame pingRAM is output row by row.In state3, the source pulse frame is input into the pongRAM of the source pulse frame RAM to overwrite the original data and become the new next frame. The previous frame in the source pulse frame pingRAM is corrected, and the next frame of the corrected pulse frame RAM is output. After that, it always jumps between state2 and state3 to achieve seamless real-time processing.

[0032] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. For those skilled in the art, various substitutions and modifications are possible without departing from the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention shall be defined by the scope defined in the claims.

Claims

1. A pulse-domain image distortion correction method based on a pulse camera, characterized in that Perform corner camera calibration on the checkerboard image obtained by using a pulsed camera, calculate and obtain the inverse mapping coordinate matrix, and directly perform position and pixel value mapping on the single-bit pulsed data obtained by real-time sampling in the pulse domain, thereby completing image distortion correction; the steps are as follows: 1) Obtain the checkerboard image through a pulsed camera; Use the pulsed camera to sample the light intensity information of all pixel points within the picture resolution range at each fixed interval time, accumulate the light intensity and emit single-bit pulsed data, calculate the image through an image reconstruction algorithm according to the pulsed data transmission rate, and shoot and reconstruct the checkerboard pattern to obtain the image; 2) Use the checkerboard image for corner camera calibration to obtain camera parameters; 3) Combine the distortion model to obtain the inverse mapping coordinate matrix; 4) The pulsed camera samples to obtain single-bit pulsed data, and directly performs distortion correction in the pulse domain where each pixel point is expressed as 1 bit; For each pixel position in the pulsed data frame within the resolution range, search for the coordinates provided by the inverse mapping matrix, perform rearrangement from the source pulse frame to the corrected pulse frame, and directly obtain the pixel value at the pixel position; 5) Calculate the corrected image through an image reconstruction algorithm for the corrected pulse frame.

2. The pulse domain image distortion correction method based on a pulse camera according to claim 1, characterized in that Due to central symmetry, obtain the inverse mapping coordinate matrix for a quarter of the entire image resolution range.

3. The pulse domain image distortion correction method based on a pulse camera according to claim 1, wherein In step 4), the pixel value is a single-bit asynchronous pulse value.

4. An FPGA hardware acceleration structure for implementing the pulse-domain image distortion correction method based on a pulse camera described in claim 1, comprising: Source pulse frame memory RAM, mapping coordinate matrix memory ROM, mapping coordinate calculation module, corrected pulse frame memory RAM.

5. The FPGA hardware acceleration structure according to claim 4, characterized in that, Also adopt pixel-level pipelining, inter-module pipelining mode and ping-pong operation, and realize real-time processing and output in a block-parallel manner.

6. The FPGA hardware acceleration structure according to claim 4, characterized in that, The mapping coordinate matrix memory ROM is used to store the mapping coordinate matrix. The mapping coordinate matrix adopts the form of relative coordinates and base addresses within the mapping partition. The mapping coordinate calculation module is responsible for calculating and converting the relative coordinates and base addresses within the mapping partition into the relative coordinates within the corresponding partition of the source pulse frame, and obtaining the mapping coordinates of the remaining three quarter regions, thereby saving storage resources.

7. An FPGA acceleration method for pulse-domain image distortion correction based on a pulsed camera, characterized in that Use the FPGA hardware acceleration structure described in claim 5 to realize real-time distortion correction of the image; Including the following states: 1) state0 state, the source pulse frame is input row by row and stored in the pingRAM of the block-based source pulse frame memory RAM under the condition of meeting the row range condition until the entire pingRAM stores a whole source pulse frame, that is, the previous frame; 2) state1 state, the source pulse frame is input and a whole frame is stored in the pongRAM of the block-based source pulse frame memory RAM, that is, the next frame. At the same time, the mapping coordinates stored in the mapping coordinate matrix memory ROM are sequentially searched and retrieved, and processed and calculated by the mapping coordinate calculation module to obtain the mapping coordinates of the remaining three quarter regions. Then, the single-bit pulse value at the mapping coordinate position of the previous frame in the source pulse frame pingRAM is read out and written in the pingRAM of the corrected pulse frame memory RAM to complete the distortion correction of the previous frame; 3) In state2, the source pulse frame is input into the ping RAM of the source pulse frame RAM to overwrite the original data and become the new previous frame. At the same time, the next frame in the pong RAM of the source pulse frame is corrected, and the previous frame of the corrected pulse frame ping RAM is output row by row. 4) In state3, the source pulse frame is input into the pong RAM of the source pulse frame RAM to overwrite the original data and become the new next frame. The previous frame in the ping RAM of the source pulse frame is corrected, and the next frame of the corrected pulse frame RAM is output. After that, it always jumps between state2 and state3 to achieve seamless real-time processing.

Images