Multi-image sensor image acquisition alignment method and system and medium

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a multi-image sensor image acquisition alignment method and system and a medium, and the alignment method comprises the steps: correcting the coordinate data of an image to be corrected to reduce the spatial deviation of the image to be corrected relative to a reference; the sequence of the corrected image coordinate data is adjusted, and the corrected image coordinate data and the to-be-corrected image coordinate data are made to be the same in sequence according to the mapping relation between the corrected image coordinate data and the address; the corrected image coordinate data is cached to the minimum pixel row, and down-sampling is started to form an aligned image; according to the invention, the correction logic of the correction formula is utilized, and the line number of the image coordinate data occupied in the corresponding originally collected image is obtained through the first line of image coordinate data of the corrected image by utilizing the correction formula in a back-stepping manner, so that the delay time of image correction and output is reduced, the image processing speed is improved, and the real-time performance of image correction is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method, system and medium for aligning image acquisition of multiple image sensors. Background Art

[0002] Multi-image sensor cameras are usually used in multi-spectral fusion application scenarios. For example, a camera has both a visible light image sensor and an infrared image sensor inside, and the camera can collect data from both sensors at the same time. This type of camera application requires that the field of view of different sensors must be precisely overlapped. However, due to the installation errors of multiple sensors inside the camera, the images of the objects on multiple sensors cannot be precisely overlapped, which requires spatial correction to align the images of different sensors.

[0003] The solutions in the prior art usually include two processes: calibration and correction. The calibration process can obtain the spatial difference parameters between different sensors, including scaling, rotation and translation parameters. The correction process uses the scaling, rotation and translation parameters to perform algorithmic processing on images collected by different sensors, so that the images of different sensors are aligned after algorithmic processing.

[0004] However, in the prior art, if the installation errors between several image sensors in a multi-image sensor camera are large or due to the limitations of the algorithm itself, the memory needs to cache the complete image data before performing the image correction and output steps, which greatly increases the delay time of image correction and output and reduces the image correction efficiency. In particular, when there are a large number of image sensors, it is necessary to wait for several images to be completely input or cached, which results in a significant increase in the waiting or delay time for image processing and poor real-time performance.

[0005] A Chinese patent discloses a real-time video image magnification method based on FPGA (application number CN202010180019.4), wherein the method includes obtaining and storing original image data; reading the stored image data as needed and caching it in the RAM of the FPGA chip; and amplifying the image data cached in the RAM of the FPGA chip by a bilinear interpolation algorithm. While ensuring the real-time and smoothness of image magnification, image magnification can be achieved with only an entry-level FPGA chip, which reduces hardware costs and improves resource utilization efficiency compared to traditional GPU or DSP implementations. However, the bilinear interpolation used for image magnification in this scheme is different from the purpose of using bilinear interpolation to improve image resolution to improve image correction accuracy in this scheme, so the technical problems solved by the two are not the same.

[0006] A Chinese patent discloses a multi-sensor real-time image registration method and system (application number CN201410505388.0), wherein the method includes: setting a laser rangefinder and three sensors and setting one of the sensors as a reference sensor; determining the horizontal offset pixels and vertical offset pixels and magnification of the output images of the two non-reference sensors relative to the output image of the reference sensor; and performing real-time registration processing on the image information collected by the three sensors in real time according to the horizontal offset pixels, vertical offset pixels and magnification. By using this method and system, the image registration process can be simplified, the use of complex image registration methods can be avoided, the processing speed of image registration can be improved, and the rapid implementation of the system real-time registration process can be ensured; the problem of continuous zoom of visible light cameras and the image registration problem caused by the dual field of view switching of medium-wave thermal imagers and long-wave thermal imagers can also be effectively solved. However, in this scheme, the method of directly correcting the image at the original resolution produces a rounding error of the unit pixel, which is significantly higher than the method in this scheme and has poor accuracy.

[0007] Therefore, in order to solve the above-mentioned problems, the present invention provides a multi-image sensor image acquisition and alignment method, system and medium, which can take into account both the real-time and high-precision of correction. Summary of the invention

[0008] The purpose of the present invention is to overcome the above problems existing in the prior art and to provide a method, system and medium for aligning images collected by multiple image sensors.

[0009] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions: A method for aligning images collected by multiple image sensors is provided. An image collected by any image sensor is selected as a reference, and images collected by the remaining image sensors are corrected one by one and then aligned with the reference. The alignment method comprises: Correcting coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference; Adjusting the order of the coordinate data of the corrected image according to the mapping relationship between the coordinate data of the corrected image and the address so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; Buffer the rectified image coordinate data to the minimum pixel row and start downsampling to form an aligned image; Wherein, the image to be corrected is formed by interpolation operation of the originally acquired image; The correction is achieved by obtaining a correction formula constructed based on the correction parameters obtained by alignment calibration; The minimum pixel row is a pixel row of the image to be corrected that includes the first row of pixels of the corrected image.

[0010] Furthermore, the correction formula constructed by obtaining correction parameters based on alignment calibration includes: The feature point extraction benchmark and the image to be corrected are used to obtain the coordinates of the feature corner points, and the correction parameters are calculated to construct the correction formula.

[0011] Furthermore, the interpolation operation includes: The adjustment coefficient is set to interpolate the image to be corrected into an image with a resolution multiple corresponding to the adjustment coefficient.

[0012] Further, the adjusting the order of the coordinate data of the corrected image includes: A mapping formula is constructed based on the adjustment coefficient to map the two-dimensional rectified image coordinate data to a one-dimensional indexed memory write address.

[0013] Furthermore, if the corrected image coordinate data is not cached to the minimum pixel row, the corrected image coordinate data continues to be cached.

[0014] Furthermore, before the step of caching the corrected image coordinate data to the minimum pixel rows, the method further comprises: Calculate the pixel row of the image to be rectified that includes the first row of pixels of the rectified image, so that the output aligned image coordinate data is complete.

[0015] Furthermore, the calculation of the pixel row of the image to be corrected including the first row of pixels of the corrected image comprises: Based on the correction formula, the corresponding coordinates of the first and last two pixel points in the first row of the corrected image coordinates in the image to be corrected are obtained, and the maximum value of the vertical coordinate is divided by the adjustment coefficient and rounded up.

[0016] Furthermore, the downsampling comprises: Adjust the rectified image resolution to output the aligned image at the same resolution as the originally acquired image.

[0017] A multi-image sensor image acquisition and alignment system, comprising: A calculation module, used for correcting the coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference, wherein the correction is implemented by a correction formula constructed by obtaining correction parameters based on alignment calibration, and the image to be corrected is formed by interpolation operation of the original acquired image; A processing module, used for adjusting the order of the coordinate data of the corrected image, according to the mapping relationship between the coordinate data of the corrected image and the address, so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; The buffer module is used to buffer the coordinate data of the corrected image to the minimum pixel row and start downsampling to form an aligned image. The minimum pixel row is the pixel row of the image to be corrected including the first row of pixels of the corrected image.

[0018] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned method for aligning image acquisition of multiple image sensors.

[0019] The beneficial effects of the present invention are: 1. In the present invention, interpolation calculation is performed on the original captured image to improve the image resolution for correction, thereby improving the image correction accuracy, obtaining a sub-pixel high-precision image space correction effect, and mapping the corrected image coordinate data to the memory write address one by one, so as to sort the corrected image coordinate data according to the data sequence of the original captured image coordinate data, thereby realizing real-time image space correction, so that the image correction has both high accuracy and real-time performance. More importantly, in the present invention, the correction logic of the correction formula is utilized, and the number of image coordinate data rows occupied in the corresponding original captured image is obtained by using the correction formula through the first row of image coordinate data of the corrected image, so as to reduce the delay time of image correction and output, improve the image processing speed, and further improve the real-time performance of image correction.

[0020] 2. In the present invention, the horizontal and vertical resolutions of the original captured image are increased by k times respectively by bilinear interpolation, and the image to be corrected with a resolution of (k×W)×(k×H) is obtained, and then the correction operation is performed. The correction formula is used to perform correction calculation on the interpolated image to be corrected. Since the calculation result must be an integer coordinate, and the result calculated according to the correction formula may be a decimal, a unit pixel error will be generated in the process of rounding from decimal to integer. It is worth noting that the unit pixel error is relative to the image to be corrected. The error ratio in the width direction is 1 / (k×W); compared with the method used in the prior art of directly correcting an image with a W×H resolution, a rounding error of a unit pixel is also generated, and the error ratio in the width direction is 1 / W. Therefore, the correction accuracy of the present invention is improved by k times compared with the prior art. Therefore, the equivalent correction accuracy after downsampling to the aligned image with a W×H resolution after correction can achieve sub-pixel accuracy, and the parameter k can be used to flexibly adjust the amount of FPGA resources consumed and the accuracy to be obtained, with high accuracy and strong practicality.

[0021] 3. In the present invention, the corresponding relationship between the coordinates of the first and last two pixel points in the first row of the corrected image coordinate data and the corresponding position coordinates in the image coordinate data before correction is utilized, and the correction formula is used to reversely deduce the minimum pixel rows that need to be cached for the image coordinate data before correction after the first row of coordinate data of the corrected image coordinate data is complete, so as to save the delay time of waiting for the complete image coordinate data to be cached and improve the image processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a flowchart of a multi-image sensor image acquisition and alignment method in the present invention; Figure 2 It is a system block diagram of a multi-image sensor image acquisition and alignment system in the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the present invention first provides a method for aligning images collected by multiple image sensors, wherein an image collected by any image sensor is selected as a reference, and the images collected by the remaining image sensors are corrected one by one and then aligned with the reference. The alignment method comprises: Correcting coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference; Adjusting the order of the coordinate data of the corrected image according to the mapping relationship between the coordinate data of the corrected image and the address so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; Buffer the rectified image coordinate data to the minimum pixel row and start downsampling to form an aligned image; Wherein, the image to be corrected is formed by interpolation operation of the originally acquired image; The correction is achieved by obtaining a correction formula constructed based on the correction parameters obtained by alignment calibration; The minimum pixel row is a pixel row of the image to be corrected that includes the first row of pixels of the corrected image.

[0025] In the present invention, the multi-image sensor camera includes at least two image sensors, any one image sensor is selected as a reference image sensor, and the image captured by it is used as a benchmark, and the remaining image sensors are reference image sensors, and the images captured by them are the original captured images.

[0026] In the present invention, the spatial deviation of the image to be corrected relative to the reference is corrected by correcting the coordinate data of the image to be corrected. The spatial deviation type includes one or more combinations of image translation, scaling or rotation. After correction, the image coordinate data of several images to be corrected correspond to each other relative to the reference and can be kept consistent in spatial position.

[0027] In the present invention, the image coordinate data is pixel values ​​with coordinate order, and the correction process is essentially an operation to restore the correct coordinate order of pixel values ​​that deviate from the correct coordinate order, while the image data cached, interpolated and outputted are pixel values ​​of the image.

[0028] In the present invention, by aligning and calibrating a multi-image sensor camera, the angle or distance difference between different image sensors is obtained, and correction parameters are obtained to determine the translation, scaling or rotation defects of the image sensor during the installation process, thereby facilitating targeted correction of images collected by different image sensors and overcoming errors generated in the later stage of installation or use. After the alignment and calibration is completed, before correcting the image to be corrected, in order to ensure the correction accuracy of the image to be corrected, it is necessary to perform interpolation calculation on the original collected image to improve the resolution of the original collected image, and then use the image to be corrected after the resolution is adjusted to greatly improve the correction accuracy of the image.

[0029] In the present invention, since the originally captured image has rotation or translation errors, there must be a difference in the data order between the corrected image coordinate data and the originally captured image coordinate data. Therefore, if the corrected image coordinate data order is directly output, the image with the correct coordinate order of the originally captured image cannot be obtained. Therefore, it is necessary to reorder the corrected image coordinate data according to the data order of the originally captured image coordinate data to restore the correct image of the originally captured image, and map them with the memory write address to store them in the memory according to the data order of the originally captured image coordinate data and wait for output.

[0030] In the present invention, the degree of image translation, scaling and rotation caused by the deviation generated after the installation or use of the image sensor is relatively small, the overlap between the corrected image and the originally collected image is relatively concentrated, and the difference in the number of rows between the corrected image coordinate data and the image coordinate data to be corrected is small. Therefore, there is no need to wait for complete or more rows of data to be corrected to be input before image correction and output. It is only necessary for the memory to cache the first few rows of the image coordinate data to be corrected before starting to output the corrected image data. In the prior art, if the installation difference errors between several image sensors in a multi-image sensor camera are large or due to the limitations of the algorithm itself, the memory needs to cache the complete image data before performing the image correction and output steps, which greatly increases the delay time of image correction and output and reduces the image correction efficiency. In particular, when there are a large number of image sensors, it is necessary to wait for several images to be completely input or cached, which results in a significant increase in the waiting or delay time for image processing.

[0031] When the first row of image coordinate data of the corrected image is completely output, the subsequent image coordinate data can be completely output. Therefore, the memory needs to cache a specific number of rows of originally acquired image coordinate data so that the memory can completely output the corrected image in the order of the originally acquired image coordinate data. The number of rows of image coordinate data occupied in the corresponding image to be corrected can be obtained by using the correction formula through the first row of image coordinate data of the corrected image. This allows the memory to start image correction without waiting for the complete image to be corrected to be input, thereby reducing the delay time of image correction and output and improving the image processing speed.

[0032] As described above, in the present invention, interpolation calculation is performed on the original captured image to improve the image resolution for correction, thereby improving the image correction accuracy, obtaining a sub-pixel high-precision image space correction effect, and mapping the corrected image coordinate data to the memory write address one by one, so as to sort the corrected image coordinate data according to the data order of the original captured image coordinate data, thereby realizing real-time image space correction, so that the image correction has both high accuracy and real-time performance. More importantly, in the present invention, the correction logic of the correction formula is utilized, and the number of image coordinate data rows occupied in the corresponding original captured image is obtained by using the correction formula through the first row of image coordinate data of the corrected image, so as to reduce the delay time of image correction and output, improve the image processing speed, and further improve the real-time performance of image correction.

[0033] In the embodiment of the present invention, the correction formula constructed by obtaining the correction parameters based on alignment calibration includes: The feature point extraction benchmark and the image to be corrected are used to obtain the coordinates of the feature corner points, and the correction parameters are calculated to construct the correction formula.

[0034] In the present invention, firstly, one of at least two image sensors is set as a reference image sensor, so as to calibrate the reference image sensor using the reference image sensor, and then the correction parameters of several image sensors to be corrected relative to the reference image sensor are respectively obtained to complete the calibration.

[0035] The specific calibration process is: S1. Place a camera with N (N>=2) image sensors, a calibration object, and a light source so that the calibration object to be photographed is clearly imaged in the center of the field of view of each image sensor; S2, use the camera to perform a calibration object shooting, and obtain N images corresponding to the image sensor one by one; S3. For each of the above images, extract M feature corner point coordinates (M>=6) using a feature point extraction method; S4. Select an image captured by any image sensor as a calibration reference image, select another image as a calibration reference image, and use the correction formula for calibration. The correction formula is: The following will combine the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] like Figure 1 As shown, the present invention first provides a method for aligning images collected by multiple image sensors, wherein an image collected by any image sensor is selected as a reference, and the images collected by the remaining image sensors are corrected one by one and then aligned with the reference. The alignment method comprises: Correcting coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference; Adjusting the order of the coordinate data of the corrected image according to the mapping relationship between the coordinate data of the corrected image and the address so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; Buffer the rectified image coordinate data to the minimum pixel row and start downsampling to form an aligned image; Wherein, the image to be corrected is formed by interpolation operation of the originally acquired image; The correction is achieved by obtaining a correction formula constructed based on the correction parameters obtained by alignment calibration; The minimum pixel row is a pixel row of the image to be corrected that includes the first row of pixels of the corrected image.

[0037] In the present invention, the multi-image sensor camera includes at least two image sensors, any one image sensor is selected as a reference image sensor, and the image captured by it is used as a benchmark, and the remaining image sensors are reference image sensors, and the images captured by them are the original captured images.

[0038] In the present invention, the spatial deviation of the image to be corrected relative to the reference is corrected by correcting the coordinate data of the image to be corrected. The spatial deviation type includes one or more combinations of image translation, scaling or rotation. After correction, the image coordinate data of several images to be corrected correspond to each other relative to the reference and can be kept consistent in spatial position.

[0039] In the present invention, the image coordinate data is pixel values ​​with coordinate order, and the correction process is essentially an operation to restore the correct coordinate order of pixel values ​​that deviate from the correct coordinate order, while the image data cached, interpolated and outputted are pixel values ​​of the image.

[0040] In the present invention, by aligning and calibrating a multi-image sensor camera, the angle or distance difference between different image sensors is obtained, and correction parameters are obtained to determine the translation, scaling or rotation defects of the image sensor during the installation process, thereby facilitating targeted correction of images collected by different image sensors and overcoming errors generated in the later stage of installation or use. After the alignment and calibration is completed, before correcting the image to be corrected, in order to ensure the correction accuracy of the image to be corrected, it is necessary to perform interpolation calculation on the original collected image to improve the resolution of the original collected image, and then use the image to be corrected after the resolution is adjusted to greatly improve the correction accuracy of the image.

[0041] In the present invention, since the originally captured image has rotation or translation errors, there must be a difference in the data order between the corrected image coordinate data and the originally captured image coordinate data. Therefore, if the corrected image coordinate data order is directly output, the image with the correct coordinate order of the originally captured image cannot be obtained. Therefore, it is necessary to reorder the corrected image coordinate data according to the data order of the originally captured image coordinate data to restore the correct image of the originally captured image, and map them with the memory write address to store them in the memory according to the data order of the originally captured image coordinate data and wait for output.

[0042] In the present invention, the degree of image translation, scaling and rotation caused by the deviation generated after the installation or use of the image sensor is relatively small, the overlap between the corrected image and the originally collected image is relatively concentrated, and the difference in the number of rows between the corrected image coordinate data and the image coordinate data to be corrected is small. Therefore, there is no need to wait for complete or more rows of data to be corrected to be input before image correction and output. It is only necessary for the memory to cache the first few rows of the image coordinate data to be corrected before starting to output the corrected image data. In the prior art, if the installation difference errors between several image sensors in a multi-image sensor camera are large or due to the limitations of the algorithm itself, the memory needs to cache the complete image data before performing the image correction and output steps, which greatly increases the delay time of image correction and output and reduces the image correction efficiency. In particular, when there are a large number of image sensors, it is necessary to wait for several images to be completely input or cached, which results in a significant increase in the waiting or delay time for image processing.

[0043] When the first row of image coordinate data of the corrected image is completely output, the subsequent image coordinate data can be completely output. Therefore, the memory needs to cache a specific number of rows of originally acquired image coordinate data so that the memory can completely output the corrected image in the order of the originally acquired image coordinate data. The number of rows of image coordinate data occupied in the corresponding image to be corrected can be obtained by using the correction formula through the first row of image coordinate data of the corrected image. This allows the memory to start image correction without waiting for the complete image to be corrected to be input, thereby reducing the delay time of image correction and output and improving the image processing speed.

[0044] As described above, in the present invention, interpolation calculation is performed on the original captured image to improve the image resolution for correction, thereby improving the image correction accuracy, obtaining a sub-pixel high-precision image space correction effect, and mapping the corrected image coordinate data to the memory write address one by one, so as to sort the corrected image coordinate data according to the data order of the original captured image coordinate data, thereby realizing real-time image space correction, so that the image correction has both high accuracy and real-time performance. More importantly, in the present invention, the correction logic of the correction formula is utilized, and the number of image coordinate data rows occupied in the corresponding original captured image is obtained by using the correction formula through the first row of image coordinate data of the corrected image, so as to reduce the delay time of image correction and output, improve the image processing speed, and further improve the real-time performance of image correction.

[0045] In the embodiment of the present invention, the correction formula constructed by obtaining the correction parameters based on alignment calibration includes: The feature point extraction benchmark and the image to be corrected are used to obtain the coordinates of the feature corner points, and the correction parameters are calculated to construct the correction formula.

[0046] In the present invention, firstly, one of at least two image sensors is set as a reference image sensor, so as to calibrate the reference image sensor using the reference image sensor, and then the correction parameters of several image sensors to be corrected relative to the reference image sensor are respectively obtained to complete the calibration.

[0047] The specific calibration process is: S1. Place a camera with N (N>=2) image sensors, a calibration object, and a light source so that the calibration object to be photographed is clearly imaged in the center of the field of view of each image sensor; S2, use the camera to perform a calibration object shooting, and obtain N images corresponding to the image sensor one by one; S3. For each of the above images, extract M feature corner point coordinates (M>=6) using a feature point extraction method; S4, select an image collected by any image sensor as a calibration reference image, select another image as a calibration reference image, and calibrate using a correction formula, the correction formula is:

[0048] The meanings of the correction parameters are: x , t y represents the overall translation of the image, a 00 , a 01 , a 10 , a 11 , the matrix composed represents the rotation and scaling of the image. The purpose of calculating the correction parameters is to align the calibrated reference image to the calibrated reference image by translation, rotation, and scaling.

[0049] The calculation process is as follows: (x, y) represents the coordinates of the corner points of the calibrated reference image, (x', y') represents the coordinates of the correction points of the calibrated reference image, and the 6 correction parameters t in the above correction formula are calculated by the least squares method. x , t y , a 00 , a 01 , a 10 , a 11 .

[0050] S5. Continue to select the remaining uncalibrated calibration reference images, repeat step S4, and finally obtain the correction parameters of the reference image sensor relative to the reference image sensor.

[0051] In the present invention, the feature point extraction method includes but is not limited to image corner point extraction algorithms such as Harris, SIFT, SURF, and ORB, and the above algorithms can be implemented through software tools such as Matlab or OpenCV.

[0052] In the embodiment of the present invention, the interpolation operation includes: The adjustment coefficient is set to interpolate the image to be corrected into an image with a resolution multiple corresponding to the adjustment coefficient.

[0053] In the present invention, the interpolation operation includes but is not limited to interpolation algorithms such as linear interpolation, bilinear interpolation or trilinear interpolation. Preferably, the present invention adopts a bilinear interpolation algorithm for image correction to improve the accuracy of image correction.

[0054] The present invention includes but is not limited to using FPGA as a memory and processor, or an external SRAM as a processor to implement interpolation operations. Preferably, the present invention uses FPGA as a processor to receive image coordinate data from an image sensor. The specific interpolation process is: The adjustment coefficient is set to k, and clock 1 is used to receive image coordinate data from the reference image sensor. Clock 2 is used to generate bilinear interpolation data. The frequency of clock 2 is at least k×k times the frequency of clock 1. When the FPGA receives the first row of data from the reference image sensor, it caches it. When the second data of the second row of data from the reference image sensor is received, clock 2 runs k×k times to calculate the bilinear interpolation result data of k×k pixels in the enclosed area of ​​(1,1)(1,k)(k,1)(k,k); when the third data of the second row of data from the reference image sensor is received, clock 2 runs k×k times to calculate the bilinear interpolation result data of k×k pixels in the enclosed area of ​​(k,1)(k,2k)(k,k)(k,2k); and so on. Finally, the original captured image with a resolution of W×H is interpolated into an image to be corrected with a resolution of (k×W)×(k×H).

[0055] The core idea of ​​bilinear interpolation in the present invention is: suppose the original image has four adjacent pixels A / B / C / D forming a square in space, the goal is to obtain k×k pixels by interpolation in the enclosed space ABCD, in the enclosed area of ​​these four adjacent pixels, first interpolate in the X direction with the distance as the weight, interpolate A and B pixels to obtain the intermediate interpolation result X1, interpolate C and D pixels to obtain the interpolation result X2, and then interpolate X1 and X2 with the Y direction distance as the weight to obtain the final result.

[0056] Since the above bilinear interpolation algorithm process is a prior art, it will not be described in detail here.

[0057] As described above, in the present invention, the horizontal and vertical resolutions of the original captured image itself are respectively increased by k times by bilinear interpolation, and the image to be corrected with a resolution of (k×W)×(k×H) is obtained and then the correction operation is performed. The correction formula is used to perform correction calculation on the interpolated image to be corrected. Since the calculation result must be an integer coordinate, and the result calculated according to the correction formula may be a decimal, a unit pixel error will be generated in the process of rounding from decimal to integer. It is worth noting that the unit pixel error is relative to the image to be corrected, and its error ratio in the width direction is 1 / (k×W); compared with the prior art that directly corrects the image with a resolution of W×H, a unit pixel rounding error is also generated, and its error ratio in the width direction is 1 / W. Therefore, the correction accuracy of the present invention is improved by k times compared with the prior art. Therefore, the equivalent correction accuracy after downsampling to the aligned image with a resolution of W×H after correction can reach sub-pixel accuracy, and the parameter k can be flexibly adjusted between the amount of FPGA resources consumed and the accuracy to be obtained, with high accuracy and strong practicality.

[0058] Correction after interpolation operation includes: A1. In a shooting scenario, a camera with N (N>=2) image sensors is used to shoot a target scene. A2, the reference image sensor captures the reference image without correction and is directly output; A3, select any image captured by the reference image sensor as the original captured image; A4. The originally acquired image is bilinearly interpolated in the FPGA to interpolate the W×H resolution image into a (k×W)×(k×H) resolution image to obtain the image to be corrected; A5. Calculate the corrected coordinates of each pixel in the image to be corrected after conversion by the correction formula, so as to obtain the corrected image of the reference image sensor. x , t y The parameter represents translation. After the original image resolution is enlarged by interpolation, the translation coefficient needs to be enlarged accordingly, while the rotation and scaling coefficients do not need to change. Therefore, k×t is used for specific correction. x , k×t y Replace t in the correction formula x , t y parameter; A6. Downsample the rectified image to convert (k×W)×(k×H) into an image with W×H resolution and output it. A7. Select the remaining images captured by the reference image sensor as the original captured images, and repeat the operations of step A4 to step A5 to finally complete the image correction.

[0059] In the embodiment of the present invention, the step of adjusting the order of the coordinate data of the corrected image includes: A mapping formula is constructed based on the adjustment coefficient to map the two-dimensional rectified image coordinate data to a one-dimensional indexed memory write address.

[0060] In the present invention, a mapping formula between the corrected image coordinate data and the memory write address is constructed based on the adjustment coefficient, so that the coordinate order of the corrected image coordinate data can be reordered according to the coordinate order of the originally acquired image coordinate data through the formula, so as to facilitate the output of the corrected image coordinate data.

[0061] The mapping formula is: ADDR=x+y×k×W, where ADDR represents the memory write address, x represents the horizontal coordinate of the corrected image coordinate data, y represents the vertical coordinate of the corrected image coordinate data, k×W represents the width of the corrected image, and therefore y×k×W represents the number of rows of the corrected image. According to the mapping formula, the coordinate positions of the images to be corrected can be sorted one by one so as to be sorted in the order of the coordinate data of the original acquired image and wait for output.

[0062] The mapping formula is used to convert the corrected image coordinate data (x, y) into the memory write address ADDR, thereby mapping the index coordinates x and y of the two-dimensional image to the one-dimensional index storage device, and writing the pixel value corresponding to the coordinate into the memory address and waiting for sequential output.

[0063] In the embodiment of the present invention, if the corrected image coordinate data is not cached to the minimum pixel row, the corrected image coordinate data continues to be cached.

[0064] In the present invention, the corrected image coordinate data mapped to the memory is cached. If the cache reaches the minimum pixel rows, that is, the number of pixel rows of the image to be corrected including the first row of pixels of the corrected image, downsampling output can be started. If the cache does not reach the minimum pixel rows, the corrected image coordinate data continues to be cached and downsampling output is not started.

[0065] In the embodiment of the present invention, before the step of caching the corrected image coordinate data to the minimum pixel row, the step further includes: The pixel row of the image to be rectified including the first row of pixels of the image to be rectified is calculated to make the output aligned image coordinate data complete.

[0066] In the present invention, calculation is required to obtain the minimum pixel rows. Specifically, the minimum pixel rows can be calculated after alignment calibration and determination of the adjustment coefficient. The purpose of calculating the minimum pixel rows is to start the downsampling output operation without waiting for the memory to cache all the corrected image coordinate data, thereby reducing the delay time of waiting for the cache. Compared with the prior art, which needs to wait for the complete image input before outputting the image, the present invention can effectively improve the image processing efficiency.

[0067] In the embodiment of the present invention, the calculation of the pixel row of the image to be corrected including the first row of pixels of the corrected image includes: Based on the correction formula, the corresponding coordinates of the first and last two pixel points in the first row of the corrected image coordinates in the image to be corrected are obtained, and the maximum value of the vertical coordinate is divided by the adjustment coefficient and rounded up.

[0068] In the present invention, since the degree of image translation, scaling and rotation caused by sensor installation deviation is small, the corrected image can be output after the first few rows of corrected image coordinate data arrive, without waiting for the completion of caching of all corrected image coordinate data, and can be executed after the corrected image coordinate data of the minimum number of pixel rows are cached, so the delay is low. When the first row of pixel coordinate data of the output aligned image is complete, the corrected image coordinate data can be downsampled and output in sequence. The calculation process can reversely calculate the pixel coordinate position corresponding to the image to be corrected through the leftmost pixel coordinate (0, 0) and the rightmost pixel coordinate (0, k×W) of the first row of pixels of the corrected image coordinate data.

[0069] The specific calculation method of the minimum pixel row is: Substitute the origin (0,0) of the output corrected image coordinate data into the correction formula, that is,

[0070] Reverse deduction can get the first coordinate for:

[0071] Output the last pixel of the first row of the corrected image coordinate data Substitute into the correction formula, that is

[0072] The second coordinate can be obtained by reverse deduction for:

[0073] That is to say, the coordinate (x1, y1) in the coordinate data of the image to be corrected corresponds to the coordinate origin (0, 0) of the coordinate data of the corrected image, and the coordinate (x2, y2) in the coordinate data of the image to be corrected corresponds to the coordinate of the last pixel in the first row of the coordinate data of the corrected image. Therefore, when the pixel cache of the coordinate data of the image to be corrected for the coordinates (x1, y1) and (x2, y2) is complete, the first row of the corrected image coordinate data is cached completely and can be output. Subsequent output will ensure the integrity of the corrected image coordinate data. Obviously, the number of cached rows of the corrected image coordinate data corresponds to the longitudinal coordinate in the coordinates. The number of rows that need to be cached for the image coordinate data to be corrected can be calculated as L1=max(y2, y1), which is the maximum value of the longitudinal coordinates in (x1, y1) and (x2, y2).

[0074] Then the number of rows of the original acquired image coordinate data that needs to be cached before the interpolation operation of the image to be corrected is L=L1 / k rounded up, that is, L=max(y2,y1) / k rounded up.

[0075] Furthermore, since the image resolution and size of the corrected image are consistent with those of the image to be corrected, in order to increase the calculation speed of the minimum pixel rows, the size parameter of the image to be corrected, that is, W, can be directly used to calculate the minimum pixel rows, so as to perform the calculation in advance to improve the speed.

[0076] As described above, in the present invention, the corresponding relationship between the coordinates of the first and last two pixel points in the first row of the corrected image coordinate data and the corresponding position coordinates in the image coordinate data before correction is utilized, and the correction formula is used to reversely deduce the minimum pixel rows that need to be cached for the image coordinate data before correction after the first row of coordinate data of the corrected image coordinate data is complete, so as to save the delay time of waiting for the complete image coordinate data to be cached and improve the image processing speed.

[0077] In the embodiment of the present invention, the downsampling includes: Adjust the rectified image resolution to output the aligned image at the same resolution as the originally acquired image.

[0078] In the present invention, when the memory caches the coordinate data of the corrected image to the minimum pixel row, downsampling output can be started, and the corrected image is downsampled to restore (k×W)×(k×H) to an image with a resolution of W×H, that is, an image with the same resolution as the original captured image is output, so that the equivalent correction accuracy after downsampling can reach sub-pixel accuracy, thereby improving the correction accuracy.

[0079] like Figure 2 As shown, based on the same inventive concept, the present application also proposes a multi-image sensor image acquisition and alignment system, comprising: A calculation module, used for correcting the coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference, wherein the correction is implemented by a correction formula constructed by obtaining correction parameters based on alignment calibration, and the image to be corrected is formed by interpolation operation of the original acquired image; A processing module, used for adjusting the order of the coordinate data of the corrected image, according to the mapping relationship between the coordinate data of the corrected image and the address, so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; The buffer module is used to buffer the coordinate data of the corrected image to the minimum pixel row and start downsampling to form an aligned image. The minimum pixel row is the pixel row of the image to be corrected including the first row of pixels of the corrected image.

[0080] Based on the same inventive concept, the present application also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the multi-image sensor image acquisition and alignment method as described above is implemented.

[0081] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0082] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for aligning images collected by multiple image sensors, wherein an image collected by any image sensor is selected as a reference, and the images collected by the remaining image sensors are corrected one by one and then aligned with the reference, wherein: The alignment method comprises: Correcting coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference; Adjusting the order of the coordinate data of the corrected image according to the mapping relationship between the coordinate data of the corrected image and the address so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; Buffer the rectified image coordinate data to the minimum pixel row and start downsampling to form an aligned image; Wherein, the image to be corrected is formed by interpolation operation of the originally acquired image; The correction is achieved by obtaining a correction formula constructed based on the correction parameters obtained by alignment calibration; The minimum pixel row is a pixel row of the image to be corrected that includes the first row of pixels of the corrected image.

2. A method for aligning multiple image sensors according to claim 1, characterized in that: The correction formula constructed by obtaining correction parameters based on alignment calibration includes: The feature point extraction benchmark and the image to be corrected are used to obtain the coordinates of the feature corner points, and the correction parameters are calculated to construct the correction formula.

3. A method for aligning multiple image sensors according to claim 1, characterized in that: The interpolation operation comprises: The adjustment coefficient is set to interpolate the image to be corrected into an image with a resolution multiple corresponding to the adjustment coefficient.

4. A method for aligning multiple image sensors according to claim 3, characterized in that: The step of adjusting the order of the coordinate data of the corrected image includes: A mapping formula is constructed based on the adjustment coefficient to map the two-dimensional rectified image coordinate data to a one-dimensional indexed memory write address.

5. A method for aligning multiple image sensors according to any one of claims 1 to 4, characterized in that: If the corrected image coordinate data is not cached to the minimum pixel row, the corrected image coordinate data continues to be cached.

6. A method for aligning multiple image sensors according to claim 5, characterized in that: Before the step of caching the corrected image coordinate data in the minimum pixel row, the method further comprises: Calculate the pixel row of the image to be rectified that includes the first row of pixels of the rectified image, so that the output aligned image coordinate data is complete.

7. A method for aligning multiple image sensors according to claim 6, characterized in that: The calculation includes the pixel row of the image to be corrected including the first row of pixels of the corrected image, including: Based on the correction formula, the corresponding coordinates of the first and last two pixel points in the first row of the corrected image coordinates in the image to be corrected are obtained, and the maximum value of the vertical coordinate is divided by the adjustment coefficient and rounded up.

8. The method for aligning multiple image sensors according to claim 3, characterized in that: The down sampling comprises: Adjust the rectified image resolution to output the aligned image at the same resolution as the originally acquired image.

9. A multi-image sensor image acquisition and alignment system, characterized in that: include: A calculation module, used for correcting the coordinate data of the image to be corrected to reduce the spatial deviation of the image to be corrected relative to the reference, wherein the correction is implemented by a correction formula constructed by obtaining correction parameters based on alignment calibration, and the image to be corrected is formed by interpolation operation of the original acquired image; A processing module, used for adjusting the order of the coordinate data of the corrected image, according to the mapping relationship between the coordinate data of the corrected image and the address, so that the order of the coordinate data of the corrected image is the same as that of the coordinate data of the image to be corrected; The buffer module is used to buffer the coordinate data of the corrected image to the minimum pixel row and start downsampling to form an aligned image. The minimum pixel row is the pixel row of the image to be corrected including the first row of pixels of the corrected image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a multi-image sensor image acquisition and alignment method as described in any one of claims 1 to 8 is implemented.

Images