A device for geometric correction of a real-time image sequence

Abstract

The present application relates to a kind of geometric correction device of real-time image sequence, wherein, pipeline real-time computing module is used to calculate the position coordinate of the output image pixel point relative to the position coordinate of input image pixel point according to the camera parameter of calibration;Input buffer unit is used to buffer input image data;Input buffer management module is used to manage input image data in input buffer unit according to the integer part of the position coordinate of input image pixel point;Interpolation calculation module is used to take the decimal part of the position coordinate of input image pixel point as the weight of interpolation calculation, and input image data is carried out interpolation calculation, and obtains correction image value;Correction system control module is used to start pipeline real-time computing module and control the data processing of input buffer management module, to complete the timing control from input distortion map to output correction map.The present application can realize the correction of high real-time image sequence with less embedded hardware resources.

Description

Technical Field

[0001] This invention relates to the field of image correction technology, and in particular to a geometric correction device for real-time image sequences. Background Technology

[0002] In fields such as autonomous navigation, autonomous driving, and industrial inspection, the selection and design of sensor subsystems directly affect the final performance of the entire system. Compared to active vision sensors such as LiDAR, passive vision sensors, represented by image sensors, have irreplaceable advantages due to their rich texture information, and are therefore receiving increasing research and application. However, the massive amount of image data leads to increased computational costs and system power consumption, as well as reduced real-time performance, which hinders their widespread application. Binocular or multi-view image sensors amplify this problem, further increasing the difficulty of application. Therefore, establishing high real-time calibration methods and realizing low-power, low-cost calibration devices are of great significance for promoting the application of vision sensors.

[0003] Image correction is a fundamental operation of passive vision sensors. Especially in binocular stereo vision applications, the calibration and correction of binocular image sequences are the foundation and prerequisite for binocular stereo vision sensors to carry out back-end vision algorithms. The basic principle of image correction is based on the optical lens model to establish the mapping relationship between the output image pixels and the input image pixels. The conventional image correction processing method is to first cache the entire frame of image data, and then, according to the calculated position of the target pixel in the original image, retrieve the image data from the storage medium sequentially for interpolation calculation to obtain the corrected output image. Existing image sequence mapping technology solutions include: (1) the GPU solution of general-purpose CPUs, which includes the method used by general-purpose computing units such as personal computers, industrial computers, and servers. This solution itself is expensive and consumes a lot of power, and requires caching the entire frame of image data and performing serial mapping processing, resulting in low real-time performance; (2) the traditional FPGA mapping solution, which requires pre-calculating the mapping table and pre-processing and storing the mapping table. Since the mapping table data is large, it often relies on DDR memory accessed in different forms, which not only greatly increases the demand for cache resources and system bandwidth, but also increases hardware costs and power consumption.

[0004] Therefore, the existing solutions are difficult to implement for embedded applications with limited hardware resources. Specifically, this is because: 1) If the entire frame of image data is cached using on-chip storage media, it will consume a lot of storage resources; 2) If the entire frame of image data is cached to DDR storage media, it will occupy a lot of DDR bandwidth; 3) Cache the entire frame of image before performing image correction, which increases the latency of image processing and reduces real-time performance. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a geometric correction device for real-time image sequences, which can achieve high real-time correction of image sequences with less embedded hardware resources.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows: A geometric correction device for real-time image sequences is provided, comprising: a correction system control module, a pipelined real-time calculation module, an input buffer management module, an input buffer unit, and an interpolation calculation module. The pipelined real-time calculation module is used to calculate the position coordinates of the output image pixels relative to the position coordinates of the input image pixels in real time according to calibrated camera parameters. The input end of the input buffer unit is connected to an input interface and is used to buffer input image data. The input buffer management module is used to manage the input image data in the input buffer unit according to the integer part of the position coordinates of the input image pixels. The input end of the interpolation calculation module is connected to the output end of the input buffer unit, and the output end is connected to an output interface. It is used to use the fractional part of the position coordinates of the input image pixels as the weight for interpolation calculation to perform interpolation calculation on the input image data to obtain the corrected image value. The correction system control module is used to start the pipelined real-time calculation module and control the data processing of the input buffer management module to complete the timing control from the input distorted image to the output corrected image.

[0007] The input and output interfaces use the stream interface as the primary mode, while also supporting the AXI interface as a backup mode.

[0008] The pipeline real-time calculation module, through The system calculates the position coordinates of the output image pixels relative to the input image pixels in real time, where x′, y′ are the normalized position coordinates of the output image pixels, and r... 2 =x′ 2 +y ′2 x″, y″ are the position coordinates of the pixels in the input image, k1~k6 are the radial distortion parameters of the camera, and p1, p2 are the tangential distortion parameters of the camera.

[0009] The pipeline real-time calculation module is a 7-stage pipeline real-time calculation module, which performs calculations on... The decomposition process involves the following first-level operations: Where x and y are the coordinates of the pixels in the output image, and z is the normalization coefficient; the second-level operation includes: r 2 =x′ 2 +y′ 2 m = 2x′*y′ and Third-level operations include: and Level 4 operations include: and Level 5 operations include: and Level 6 operations include: k r =k′ / k″, Level 7 operations include: and

[0010] The input cache management module dynamically updates the row range of input image data in the input cache unit. Specifically, after the row range of the input image data cache corresponding to the new output row is generated, its maximum and minimum values ​​are obtained by sorting and compared with the maximum and minimum values ​​corresponding to the previous row range. If the minimum value of the current row range is equal to the minimum value of the previous row range, the data of the minimum row is retained. If the minimum value of the current row range is greater than the minimum value of the previous row range, data smaller than the current minimum row is deleted. If the maximum value of the current row range is equal to the maximum value of the previous row range, no new input image data needs to be cached. If the maximum value of the current row range is greater than the maximum value of the previous row range, new input image data needs to be cached until the data in the maximum row range has been cached, and then the data reading of the next row is started.

[0011] The input cache unit updates the input image data cache using a loop-loop update method, which has four states: In the first state, the data is cached starting from the lowest address and the storage requirement of the maximum row range is met. When the storage row range is updated, the system enters the second state, where useless data is deleted and low address space is released. When the cached data reaches the maximum value of the cache address, the system enters the third state, where the storage address of the maximum row jumps to the starting address of the cache to continue caching the subsequent data, and the system enters the fourth state.

[0012] An output image buffer is also provided between the output end and the output interface of the interpolation calculation module. The output image buffer is provided to the input buffer management module as a feedback signal through a nearly full marker.

[0013] When the "almost full" flag is low, the input buffer management module retrieves the input image data from the input buffer unit, calculates the output data in the interpolation calculation module, and outputs it through the output image buffer; when the "almost full" flag is high, the input buffer management module stops retrieving data.

[0014] When the remaining space in the output image buffer is less than half the depth, the "almost full" flag is low; when the remaining space in the output image buffer is greater than or equal to half the depth, the "almost full" flag is high. The depth is twice the maximum value of the first clock cycle n and the second clock cycle m. The first clock cycle n is the clock cycle from when the "almost full" flag goes low until the output image buffer receives the first data. The second clock cycle m is the clock cycle from when the "almost full" flag goes high until data stops being written to the output image buffer.

[0015] Beneficial effects

[0016] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art:

[0017] This invention addresses the requirements for correcting camera-distorted images in machine vision and computer vision applications. First, it establishes the internal parameters, external parameters, and lens distortion parameters of the image acquisition device through a calibration method. Then, it uses a hardware acceleration module to calculate the position of the target pixel in the original image in real time and uses bilinear interpolation to obtain the target pixel value in real time. This allows for the correction of high-real-time image sequences with fewer embedded hardware resources.

[0018] This invention comprehensively considers the usage scenarios of the modules, providing a choice of stream interface and AXI interface scheme. During algorithm execution, it does not exchange data with DDR, reducing reliance on DDR and DDR bandwidth usage, minimizing external memory usage, and lowering hardware costs. The hardware design employs pipeline technology to achieve real-time computation of the image geometric correction algorithm, significantly reducing the clock requirements of the algorithm's main clock. Furthermore, the hardware design utilizes adaptive optimization technology for image caching, reducing the algorithm's demand for cache resources and bringing computation latency close to the limit corresponding to the cache line cycle. The real-time image sequence geometric correction method used in this invention can serve as an IP implementation solution applicable to large SoCs, significantly reducing chip area and achieving good economic benefits. Simultaneously, this invention also provides a high-energy-efficiency implementation method for deploying real-time image correction algorithms in embedded systems. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating the implementation process of an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a 7-stage pipeline processing multiple pixels in parallel according to an embodiment of the present invention;

[0022] Figure 4This is a row range mapping diagram in the embodiment of the present invention, in which a single row of the output diagram corresponds to multiple rows of the input diagram;

[0023] Figure 5 This is a schematic diagram of the data update principle of the input cache unit in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram illustrating the optimized principle of output image buffering in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0026] Embodiments of the present invention relate to a geometric correction device for real-time image sequences, such as... Figure 1 As shown, it includes: a correction system control module, a pipeline real-time calculation module, an input buffer management module, an input buffer unit, and an interpolation calculation module.

[0027] The correction system control module is the overall control module of the core computing unit inside the geometric correction system. Based on the input and output data information, it starts the pipeline real-time calculation module and controls the data processing of the input buffer management module to complete the timing control from the input distortion map to the output correction map.

[0028] The pipeline real-time calculation module calculates the position coordinates of the output image pixels relative to the input image pixels in real time, based on the calibrated camera internal parameters, external parameters, and lens distortion parameters, using the row and column coordinates of the output image (i.e., the correction image) as variables.

[0029] The input cache management module manages the input image data based on the integer values ​​of the position coordinates of the pixels in the input image. Specifically, it saves the input image data according to the row range, retrieves the image data when the output correction image is satisfied, and deletes the input image data that is no longer used.

[0030] A buffer unit is used to buffer input image data.

[0031] The interpolation calculation module uses the small values ​​of the position coordinates of the pixels in the input image as the weights for interpolation calculation to perform interpolation calculation on the input image data and obtain the corrected image value, thereby achieving a high-precision correction result.

[0032] The input and output interfaces support a streaming interface as the primary input mode, while also supporting an AXI interface as a backup mode. The former significantly reduces DDR bandwidth usage through the streaming interface, while the latter provides more flexible usage. An output image buffer is provided between the interpolation calculation module and the output interface.

[0033] The key data flow of a real-time image sequence geometric correction system is as follows: Figure 2 As shown. First, using the camera's internal parameters, external parameters, and lens distortion parameters obtained from calibration, a pipeline is used to calculate in real time the position coordinates of the output image pixels relative to the input image pixels. These coordinates are fixed-point. Then, the integer part of the input coordinate values ​​is provided to the input buffer management module for storing and retrieving the input image data; finally, the fractional part of the input coordinate values ​​is provided to the interpolation calculation module for weighting the interpolation calculation.

[0034] The above modules work together independently to perform distortion coordinate calculations, input image caching and management, and input image data interpolation calculations for output. The entire process is executed in a pipelined, real-time manner. Image data is stored in the cache, eliminating the need for DDR memory and DDR bandwidth. Intermediate processing does not require data interaction with other devices, achieving low-latency, high-real-time processing from distorted input image to corrected output image.

[0035] In this embodiment, the pipelined real-time calculation module uses camera calibration methods to obtain the internal parameters, external parameters, and lens distortion parameters of the binocular image, and obtains the corrected binocular image through correction calculations. The image correction algorithm based on the pinhole camera model establishes a mapping relationship between pixel coordinates before and after image correction using normalized pixel coordinates.

[0036]

[0037]

[0038] Where x′, y′ are the normalized position coordinates of the pixels in the output image, and r 2 =x ′2 +y′ 2 x″, y″ are the position coordinates of the pixels in the input image, k1~k6 are the radial distortion parameters of the camera, and p1, p2 are the tangential distortion parameters of the camera.

[0039] In this embodiment, the real-time pipeline calculation module needs to decompose the above formula during calculation, ensuring that each step is a basic operation and the output of the previous stage becomes the input of the next stage, thereby completing the real-time calculation of the multi-stage pipeline. Figure 3 As shown.

[0040] The pipeline design in this implementation is 7 stages, and the decomposed computation is shown in the table below:

[0041]

[0042] Assuming a divider takes 12 clock cycles, a multiplier takes 1 clock cycle, and an adder takes 1 clock cycle, the 7-stage operation requires a total of 33 clock cycles. Without a pipelined architecture, the correction algorithm would need 33 clock cycles to complete the calculation for a single pixel, requiring the main clock frequency to be 33 times the pixel clock frequency of the image data. This poses a significant challenge for the design of application-specific integrated circuits (ASICs).

[0043] The processing capacity of a pipeline is limited by the minimum processing capacity of a single-stage pipeline. When a pipelined architecture is used in the divider, a single-stage pipeline requires a maximum of two algorithm clock cycles to complete the processing of one pixel. This significantly reduces the clock requirements of the algorithm's main clock while meeting the algorithm's real-time requirements.

[0044] The biggest challenge for implementing image correction algorithms in dedicated hardware circuits comes from the need for data caching, which mainly consists of two parts: input image caching and output image caching.

[0045] Since the camera's intrinsic and extrinsic parameters and distortion parameters corresponding to the image are obtained in advance, pre-calculation is initiated after reading one row of input image data to obtain the mapping points of the input image required for the next output row. The integer parts of the row coordinates of the mapping points are sorted to obtain the minimum and maximum values ​​of the row coordinates, thereby obtaining the row range of the input image data required for the next output row. Figure 4 As shown, saving the input image data to the buffer according to the row range satisfies the input conditions for calculating the output image of the next row.

[0046] The range of input image data cached rows is dynamically updated. After a new output row's corresponding input image data cached row range is generated, its maximum and minimum values ​​are obtained by sorting, and compared with the maximum and minimum values ​​corresponding to the previous row range. If the minimum value of the current row range is equal to the minimum value of the previous row range, the data of the smallest row is retained. If the minimum value of the current row range is greater than the minimum value of the previous row range, data smaller than the current minimum row is deleted. If the maximum value of the current row range is equal to the maximum value of the previous row range, no new input image data needs to be cached. If the maximum value of the current row range is greater than the maximum value of the previous row range, new input image data needs to be cached until the data in the maximum row range has been cached, and then the data reading of the next row is initiated.

[0047] The input image cache uses a loop-based update method, such as... Figure 5As shown, the loopback update method can be divided into four states: State 1 caches data starting from the lowest address and satisfies the storage requirement of the maximum row range. When the storage row range is updated, it enters State 2, where useless data is deleted and low address space is released. When the cached data reaches the maximum value of the cache address, corresponding to State 3, the storage address of the maximum row jumps to the starting address of the cache, and continues to cache the subsequent data, while entering State 4. Since row image data smaller than the current minimum row has been deleted, State 4 will not overwrite valid row data.

[0048] An adaptive optimization method for input image caching pre-computes input image caches for on-demand retrieval, promptly deleting useless input image data and freeing up cache space for caching new data, thus optimizing cache space utilization. This adaptive optimization method can automatically adapt to changes in distortion levels and support correction at higher resolutions even with low distortion.

[0049] The output image buffer is implemented using a FIFO (First-In, First-Out) mechanism, and a "Almost Full" (50%) marker is provided as a feedback signal to the input buffer management module to achieve dynamic control of the output data stream. Figure 6 As shown.

[0050] When Almost Full is low, it indicates that the FIFO can buffer output data. In this case, the input buffer management module retrieves the data from the input buffer and calculates the output data in the interpolation calculation module, which is then output through the FIFO. When Almost Full is high, it indicates that the FIFO is almost full and data reception needs to be paused. In this case, the input buffer management module stops retrieving data, and the flow of data into the FIFO stops.

[0051] If Almost Full is low for n clock cycles until the FIFO receives the first data, and Almost Full is high for m clock cycles until data stops being written to the FIFO, then the FIFO depth is twice the larger of n and m. Almost Full is low when the FIFO margin is less than half the depth, and high when the FIFO margin is greater than or equal to half the depth.

[0052] By using a small-capacity FIFO with an Almost Full tag, the geometric correction system can avoid pre-allocating the maximum capacity FIFO according to the row data size of its maximum supported resolution, and achieve on-demand retrieval and adaptive caching of the output image, thereby saving cache capacity.

[0053] It is easy to see that this invention comprehensively considers the usage scenarios of the modules, providing both stream interfaces and AXI as selectable interface solutions. Furthermore, during algorithm execution, no data exchange occurs with DDR, reducing reliance on DDR and DDR bandwidth usage, minimizing external memory usage, and lowering hardware costs. The invention employs pipeline technology in its hardware design, enabling real-time computation of the image geometric correction algorithm and significantly reducing the clock requirements of the algorithm's main clock. The invention also utilizes adaptive optimization technology for image caching, reducing the algorithm's demand for cache resources and bringing computation latency close to the limit corresponding to the cache line cycle. The real-time image sequence geometric correction method used in this invention can serve as an IP implementation solution applicable to large-scale SoCs, significantly reducing chip area in the SoC and achieving good economic benefits. Simultaneously, this invention also provides a high-energy-efficiency implementation method for deploying real-time image correction algorithms in embedded systems.

Claims

1. A geometric correction device for real-time image sequences, characterized in that, include: The system comprises a calibration system control module, a pipelined real-time calculation module, an input buffer management module, an input buffer unit, and an interpolation calculation module. The pipelined real-time calculation module is used to calculate the position coordinates of the output image pixels relative to the input image pixels in real time based on the calibrated camera parameters. The input terminal of the input buffer unit is connected to the input interface and is used to buffer input image data. The input buffer management module is used to manage the input image data in the input buffer unit based on the integer part of the position coordinates of the input image pixels. The input terminal of the interpolation calculation module is connected to the output terminal of the input buffer unit, and the output terminal is connected to the output interface. It is used to perform interpolation calculations on the input image data by using the fractional part of the position coordinates of the input image pixels as the weight for interpolation calculations, thereby obtaining the corrected image value. The correction system control module is used to start the pipeline real-time calculation module and control the data processing of the input buffer management module to complete the timing control from the input distorted image to the output corrected image. The pipeline real-time calculation module... The position coordinates of the output image pixels relative to the position coordinates of the input image pixels are calculated in real time. To output the normalized position coordinates of the image pixels, , The input image contains the position coordinates of the pixels. For the radial distortion parameters of the camera, This represents the tangential distortion parameter of the camera.

2. The geometric correction device for real-time image sequences according to claim 1, characterized in that, The input and output interfaces use the stream interface as the primary mode, while also supporting the AXI interface as a backup mode.

3. The geometric correction device for real-time image sequences according to claim 1, characterized in that, The pipeline real-time calculation module is a 7-stage pipeline real-time calculation module, which performs calculations on... The decomposition process involves the following first-level operations: ,in, To output the coordinates of the pixels in the image, These are the normalization coefficients; the second-level operations include: , and The third-level operations include: , and The fourth level of operations includes: and Level 5 operations include: and Level 6 operations include: Level 7 operations include: and .

4. The geometric correction device for real-time image sequences according to claim 1, characterized in that, The input cache management module dynamically updates the row range of input image data in the input cache unit. Specifically, after the row range of the input image data cache corresponding to the new output row is generated, its maximum and minimum values ​​are obtained by sorting and compared with the maximum and minimum values ​​corresponding to the previous row range. If the minimum value of the current row range is equal to the minimum value of the previous row range, the data of the minimum row is retained. If the minimum value of the current row range is greater than the minimum value of the previous row range, data smaller than the current minimum row is deleted. If the maximum value of the current row range is equal to the maximum value of the previous row range, no new input image data needs to be cached. If the maximum value of the current row range is greater than the maximum value of the previous row range, new input image data needs to be cached until the data in the maximum row range has been cached, and then the data reading of the next row is started.

5. The geometric correction device for real-time image sequences according to claim 4, characterized in that, The input cache unit updates the input image data cache using a loop-loop update method, which has four states: In the first state, the data is cached starting from the lowest address and the storage requirement of the maximum row range is met. When the storage row range is updated, the system enters the second state, where useless data is deleted and low address space is released. When the cached data reaches the maximum value of the cache address, the system enters the third state, where the storage address of the maximum row jumps to the starting address of the cache to continue caching the subsequent data, and the system enters the fourth state.

6. The geometric correction device for real-time image sequences according to claim 1, characterized in that, An output image buffer is also provided between the output end and the output interface of the interpolation calculation module. The output image buffer is provided to the input buffer management module as a feedback signal through a near-full mark.

7. The geometric correction device for real-time image sequences according to claim 6, characterized in that, When the "almost full" flag is low, the input buffer management module retrieves the input image data from the input buffer unit, calculates the output data in the interpolation calculation module, and outputs it through the output image buffer; when the "almost full" flag is high, the input buffer management module stops retrieving data.

8. The geometric correction device for real-time image sequences according to claim 6, characterized in that, When the remaining space in the output image buffer is less than half the depth, the "almost full" flag is low; when the remaining space in the output image buffer is greater than or equal to half the depth, the "almost full" flag is high. The depth is twice the maximum value of the first clock cycle n and the second clock cycle m. The first clock cycle n is the clock cycle from when the "almost full" flag goes low until the output image buffer receives the first data. The second clock cycle m is the clock cycle from when the "almost full" flag goes high until data stops being written to the output image buffer.

Images