A real-time correction method and storage medium for video image data

Abstract

The present invention discloses a real-time correction method for video image data and a storage medium, comprising: inputting a reference image (standard checkerboard H×V) according to an optical system design model to obtain a distorted image; establishing an approximate elliptical distortion correction model between the reference image and the distorted image, wherein the distortion model is divided into an X-direction approximate elliptical distortion correction model and a Y-direction approximate elliptical distortion correction model; outputting X-direction and Y-direction correction mapping tables based on the X-direction approximate elliptical distortion correction model and the Y-direction approximate elliptical distortion correction model, wherein data from one-eighth of the correction mapping table is extracted for mapping; mapping the input real-time video one-to-one with the correction mapping tables to obtain a mapped video; and finally, performing bilinear interpolation on the mapped video to output a final corrected video. The method of the present invention has high real-time performance, low hardware resource consumption, low latency, and is equally applicable to lenses with different fields of view, thus having broad application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time correction method for video image data and a storage medium. Background Art

[0002] With the rapid development of computer technology and the widespread adoption of artificial intelligence, wide-angle cameras (lenses) are widely used in military, road traffic, industry, and residential areas due to their large field of view. However, wide-angle lenses also present a number of problems, one of which is severe distortion. This phenomenon is caused by factors such as the design and processing of the camera lens itself, as well as axial spacing errors during lens assembly.

[0003] Currently, a common implementation method in hardware circuits is to use a correction mapping table. The correction result corresponding to the image is first calculated and stored in the hardware circuit. Based on the input real-time video, the corresponding corrected video output is found from the correction mapping table. While the correction mapping table is a very straightforward calculation, its disadvantage is that storing a correction mapping table for a single image requires a large amount of hardware circuit resources. While traditional distortion correction schemes can achieve different correction effects based on different distortion correction coefficients, they suffer from the following two issues:

[0004] (1) It occupies a large amount of storage resources and has poor real-time performance.

[0005] (2) Reduced field of view. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to provide a real-time correction method and storage medium for video image data, which can achieve real-time display without loss of field of view video while using relatively small storage resources, thereby reducing its delay.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for real-time correction of video image data and a storage medium thereof, comprising the following steps:

[0009] Step (1) generates a checkerboard image (H×V) as a reference image based on the resolution H×V (width×height) of the lens. The lens in the camera is designed using optical software to create an optical system design model. The reference image is input into the optical system design model to obtain a distorted image.

[0010] Step (2) establishes an approximate ellipse distortion correction model in the X direction according to formula (13) and an approximate ellipse distortion correction model in the Y direction according to formula (14) based on the distorted image and the reference image, including:

[0011] In step (2.1), image distortion mainly occurs when the pixel position of the image undergoes geometric distortion after imaging. Geometric distortion is divided into linear distortion and nonlinear distortion. The causes of nonlinear distortion are mainly divided into three types: radial distortion, centrifugal distortion, and thin prism distortion. The formula is as follows:

[0012] x distorted =x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) (4)

[0013] y distorted =y(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2y 2 ) (5)

[0014] x 2 +y 2 =r 2 (6)

[0015] In the above formula, x and x distorted The reference image and distorted image in the X direction, y and y distorted are the reference image and distorted image in the Y direction respectively. k1, k2, k3, p1, and p2 are distortion correction parameters. If there are too many distortion coefficients, the distortion correction effect will be poor.

[0016] In step (2.2), distortion correction usually considers radial distortion and ignores the influence of tangential distortion. This part of distortion can describe the nonlinear distortion of the lens. Step (2.1) can optimize the standard concentric circle distortion model as follows:

[0017]

[0018]

[0019] x 2 +y 2 =r 2 (9)

[0020] In step (2.3), the image resolution ratio H:V≠1:1 is used. The standard concentric circle distortion model in step (2.2) has poor image edge distortion correction effect. Therefore, step (2.2) is further improved to an elliptical distortion correction model. The formula is as follows:

[0021]

[0022]

[0023]

[0024] in, H is the height of the image and V is the width of the image.

[0025] The elliptical distortion correction model in step (2.4) and step (2.3) cannot completely cover the H×V image. Therefore, the elliptical distortion correction model is further improved to an approximate elliptical distortion correction model. The formula is as follows:

[0026]

[0027]

[0028]

[0029] Where k1 and k2 are distortion coefficients, x, x distorted The corrected image and distorted image in the X direction, y and y distorted They are the corrected image and distorted image in the Y direction respectively.

[0030] Step 3: Obtain an X-direction correction model using the X-direction approximate elliptical distortion correction model, and obtain a Y-direction correction model using the Y-direction approximate elliptical distortion correction model; crop the white areas appearing in the X-direction correction model and the Y-direction correction model, respectively, and interpolate the cropped X-direction correction model and the Y-direction correction model into the H×V X-direction correction model and the Y-direction correction model, respectively, to ensure that the field of view in the X-direction and Y-direction horizontal directions remains unchanged.

[0031] Step 4: Based on the processing results of step 3, extract one-eighth of the data from the processed correction model in the X direction, convert the one-eighth of the data into the stored COE data format and use it as the correction mapping table for the X direction; extract one-eighth of the data from the processed correction model in the Y direction, convert the one-eighth of the data into the stored COE data format and use it as the correction mapping table for the Y direction.

[0032] Step 5, distortion correction mapping is performed on the input image, including: the input image is respectively subjected to distortion correction mapping for the correction mapping table in the X direction and the correction mapping table in the Y direction, to obtain a mapped image, wherein the quality of the mapped image is poor and aliasing occurs, so the quality of the mapped image is subjected to grayscale correction. The input distorted image is subjected to distortion correction using the correction mapping table, and the distorted image f(x,y) is bilinearly interpolated, wherein bilinear interpolation can be performed according to the characteristics of the image, and finally the corrected image f(xd ,y d ).

[0033] The formula for bilinear interpolation is:

[0034] f(x d ,y d )=(x+1-x d )(y+1-y d )f(x,y)+(x+1-x d )(y d -y)f(x,y+1)+(xx d )(y+1-y d )f(x+1,y)+(x d -x)(y d -y)f(x+1,y+1) (16)

[0035] Where f(x,y) is the image after the correction mapping in step 4, f(x d ,y d ) is the image after bilinear interpolation. Let δ x =x d -x,δ y =y d -y, and 0≤δ x ≤1,0≤δ y ≤1, the above formula can be expressed as:

[0036] f(x d ,y d )=(1-δ x )(1-δ y )f(x,y)+(1-δ x )δ y f(x,y+1)+δ x (1-δ y )f(x+1,y)+δ x δ y f(x+1,y+1) (17)

[0037] Step 6: Perform grayscale correction on the image output in step 5 to obtain a corrected video image.

[0038] Furthermore, in step 2, the distortion coefficients k1 and k2 are obtained by designing an active infrared radiation calibration checkerboard based on the Portier principle. The camera lens is aligned with the active infrared radiation calibration checkerboard based on the Portier principle, multiple distorted images are collected, and the multiple distorted images are processed by the Camera Calibrator of the MATLAB software to obtain the values ​​of the distortion coefficients k1 and k2. The distortion coefficients of different camera lenses are obtained in this way.

[0039] Furthermore, in step 3, the interpolation is enlarged to a size of H×V by bicubic interpolation to ensure that the fields of view of the corrected image in the X horizontal direction and the Y horizontal direction remain unchanged.

[0040] Furthermore, in step 4, an approximate elliptical distortion correction model is used, which approximates the elliptical distortion model to the size of an H×V image through the coefficient k2. The model is used to extract one-half, one-quarter or one-eighth of the correction mapping table in the X direction and the Y direction for video distortion correction mapping, thereby saving hardware storage resources.

[0041] Furthermore, the interpolation amplification in step 3 and the grayscale correction in step 6 are performed by bilinear interpolation, dual three-point interpolation or four-point interpolation.

[0042] A computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for real-time correction of video image data of the present invention.

[0043] Beneficial effects of the present invention:

[0044] The method of the present invention first generates a distorted image by inputting a reference image (standard checkerboard) according to an optical system design model; then establishes an approximate elliptical distortion correction model between the reference image and the distorted image, wherein the distortion model is divided into X and Y directions; outputs X and Y correction mapping tables respectively according to the approximate elliptical distortion correction models in the X and Y directions, wherein the data of one-eighth correction mapping table is stored in a hardware circuit memory; wherein the input real-time video is mapped one-to-one to the correction mapping table data in the hardware circuit memory; finally, the mapped image is interpolated and the corrected video is output. The present application has high real-time performance, low memory resource consumption and low latency for large field of view video data correction, and is equally applicable to lenses with different fields of view, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 : Approximate elliptical distortion correction flow chart.

[0046] Figure 2 : Matlab software simulates reference image and distorted image: Figure 2 Middle: (a) is the reference image, (b) is the distorted image.

[0047] Figure 3 : Active infrared radiation calibration chessboard is designed based on the Portier principle (the chessboard grid size is 50CM).

[0048] Figure 4 : Calibrate checkerboard distorted image and corrected image.

[0049] Figure 5: The corrected distortion image and the cropped and enlarged correction image.

[0050] Figure 6 : The correction map stores one eighth.

[0051] Figure 7 : Hardware circuit interpolation method.

[0052] Figure 8 : Real-time video distortion image and correction image; Figure 8 Middle: (a) is the distorted image, and (b) is the corrected image. DETAILED DESCRIPTION

[0053] The present invention is described in further detail below with reference to the embodiments.

[0054] Those skilled in the art will understand that the following examples are intended only to illustrate the present invention and should not be construed as encompassing the scope of the present invention. Where specific techniques or conditions are not specified in the examples, the techniques or conditions described in the literature in the art or in the product specifications were used. Materials or equipment used without manufacturer identification are commercially available conventional products.

[0055] To illustrate the specific real-time process, assume that the resolution of the distorted image is H×V=640×512. The specific implementation steps are as follows:

[0056] Step 1: Based on the image resolution of 640×512 (height×width), use Matlab software to generate a 640×512 reference image, such as Figure 2 (a) shows that the optical system is designed by condev software to generate an optical system design model, and the generated checkerboard is input as a reference image into the condev optical system design model to obtain a distorted image, as shown in FIG. Figure 2 (b) shown.

[0057] Step 2: Based on the reference image and distorted image obtained in step 1, Matlab software is used to establish the X-direction approximate ellipse distortion correction model (Formula 18) and the Y-direction approximate ellipse distortion correction model (Formula 19). The calculation formula is as follows:

[0058]

[0059]

[0060]

[0061] Where a is 320, b is 256, and R is 320.

[0062] Step 3: Obtain the values ​​of distortion coefficients k1 and k2 in formula (18) and formula (19) in step 2. Use the camera to collect different angle calibration checkerboards to obtain multiple distorted images. The checkerboard is an active infrared radiation calibration checkerboard designed based on the Portier principle (such as Figure 3 By processing multiple distorted images with matlab software, the values ​​of distortion coefficients k1 and k2 are obtained. In this way, the distortion coefficients of different cameras can be obtained. Figure 4 It is the distortion coefficient effect obtained based on the calibration checkerboard.

[0063] Step 4: Substitute the distortion coefficients k1 and k2 obtained in step 3 into formulas (18) and (19) in step 2. The approximate elliptical distortion correction model in the X direction obtained by formula (18) in step 2 is used to obtain the correction model in the X direction; the approximate elliptical distortion correction model in the Y direction obtained by formula (19) in step 2 is used to obtain the correction model in the Y direction.

[0064] There will be white areas in the X-direction and Y-direction calibration models. The white areas in the X-direction and Y-direction calibration models are cropped and then enlarged to 640×512 (height×width) through bi-triple interpolation. This processing method can ensure that the X-direction and Y-direction calibration model fields of view remain unchanged. Figure 5 shown.

[0065] Step 5: Output the approximate elliptical distortion correction model for the X direction and the approximate elliptical distortion correction model for the Y direction according to the correction model obtained in step 4. Use MATLAB to extract one-eighth of the data from the correction model after processing in the X direction, and convert the one-eighth of the data into the stored COE data format, which is called the correction mapping table for the X direction; use MATLAB to extract one-eighth of the data from the correction model after processing in the Y direction, and convert the one-eighth of the data into the stored COE data format, which is called the correction mapping table for the Y direction, as shown in the following figure: Figure 6 shown.

[0066] Step 6: According to the result of step 5, the image quality after mapping is poor, and aliasing distortion is generated, and grayscale correction is required. The present invention adopts bilinear interpolation, and the display effect is as follows: Figure 7 As shown. The formula for bilinear interpolation is:

[0067] f(x d ,y d )=(x+1-x d )(y+1-y d)f(x,y)+(x+1-x d )(y d -y)f(x,y+1)+(xx d )(y+1-y d )f(x+1,y)+(x d -x)(y d -y)f(x+1,y+1) (21)

[0068] Where f(x,y) is the image after the correction mapping in step 5, f(x d ,y d ) is the image after bilinear interpolation. Let δ x =x d -x,δ y =y d -y, and 0≤δ x ≤1,0≤δ y ≤1, the above formula can be expressed as:

[0069] σx=x d -x,σy=y d -y, and 0≤σx≤1,0≤σy≤1, from the above formula we can know:

[0070] f(x d ,y d )=(1-σx)(1-σy)f(x,y)+(1-σx)σyf(x,y+1)+σx(1-σy)f(x+1,y)+σxσyf(x+1,y+1) (22)

[0071] Step 7: After step 6, the real-time distortion correction video is finally obtained, as shown in Figure 8 shown.

Claims

1. A method for real-time correction of video image data, characterized in that: The following steps are involved: Step 1: Generate an H×V checkerboard image as a reference image based on the image resolution, and input the reference image into the optical system design model to obtain a distorted image, where H and V represent width and height, respectively. Step 2: Based on the distorted image and the reference image, an approximate ellipse distortion correction model in the X direction is established according to formula (1), and an approximate ellipse distortion correction model in the Y direction is established according to formula (2): Among them, k1 and k2 are image distortion coefficients, x, x distorted The corrected image and distorted image in the X direction, y and y distorted They are the corrected image and distorted image in the Y direction respectively; Step 3: Obtain an X-direction correction model using the X-direction approximate elliptical distortion correction model, and obtain a Y-direction correction model using the Y-direction approximate elliptical distortion correction model; crop the white areas appearing in the X-direction correction model and the Y-direction correction model, respectively, and interpolate the cropped X-direction correction model and the Y-direction correction model to the H×V X-direction correction model and the Y-direction correction model, respectively, to ensure that the field of view in the X-direction and Y-direction horizontal directions remains unchanged; Step 4: Based on the processing results of step 3, extract data from the processed correction model in the X direction, convert the extracted data into the stored COE data format and use it as the correction mapping table for the X direction; extract data from the processed correction model in the Y direction, convert the extracted data into the stored COE data format and use it as the correction mapping table for the Y direction; Step 5, performing distortion correction mapping on the input image, including: performing distortion correction mapping on the input image one by one with the correction mapping table in the X direction and the correction mapping table in the Y direction, to obtain a mapped image, which generates aliasing distortion and requires grayscale correction; Step 6: Perform grayscale correction on the image output in step 5 to obtain a corrected video image.

2. The method for real-time correction of video image data according to claim 1, characterized in that: In step 2, the distortion coefficients k1 and k2 are obtained by designing an active infrared radiation calibration checkerboard based on the Portier principle. A camera lens is aligned with the active infrared radiation calibration checkerboard based on the Portier principle to collect multiple distorted images. The multiple distorted images are processed by the Camera Calibrator of the MATLAB software to obtain the values ​​of the distortion coefficients k1 and k2. In this way, the distortion coefficients of different camera lenses are obtained.

3. The real-time correction method for video image data according to claim 1, characterized in that: In step 3, the interpolation is performed by upscaling the image to a size of H×V by bicubic interpolation to ensure that the fields of view of the corrected image in the X horizontal direction and the Y horizontal direction remain unchanged.

4. The method for real-time correction of video image data according to claim 1, wherein: In step 4, an approximate elliptical distortion correction model is used. The model approximates the elliptical distortion model to the size of an H×V image through the coefficient k2. The model is used to extract one-half, one-quarter, or one-eighth of the correction mapping table in the X and Y directions for video distortion correction mapping, thereby saving hardware storage resources.

5. The method for real-time correction of video image data according to claim 1, wherein: The interpolation in step 3 and the grayscale correction in step 6 are performed by bilinear interpolation, double three-point interpolation or four-point interpolation.

6. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for real-time correction of video image data as described in any one of claims 1 to 5 are implemented.

Images