Camera distortion removing anomaly detection method and system

Abstract

The invention discloses a camera distortion-removing anomaly detection method and system, and the method comprises the following steps: obtaining 3D coordinates of all corner points on a calibration plate, calibrating the 3D coordinates, and obtaining an internal reference matrix and a distortion coefficient of a calibration model; obtaining pixel coordinates of each angular point in the calibration plate image after distortion removal; the pixel coordinates of each angular point in the calibration plate image after distortion removal are sequenced according to rows and columns, and the straightness of the angular points in each row and the straightness of the angular points in each column are calculated; and obtaining a distortion-removed rendered image and a view-amplified distortion-removed rendered image, carrying out image analysis processing on the distortion-removed rendered image and / or the view-amplified distortion-removed rendered image, and judging whether distortion-removed abnormity exists or not. According to the camera distortion removal anomaly detection method and system provided by the invention, the distortion removal condition can be comprehensively judged by combining the straightness and the distortion removal anomaly judgment algorithm, so that the accuracy of internal reference calibration of a pinhole model can be better ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of lens detection, and particularly relates to a method and system for detecting abnormal distortion removal of a camera. Background Art

[0002] A camera is an essential component in mobile phones, consumer drones, or monitoring devices, and is also a very important component. Currently, cameras are widely used in the automotive field, such as ADAS (Advanced Driver Assistance System), AVM (Around View Monitor), RVC (Rear View Camera), DMS (Driver Monitoring System), DVR (Dashcam), etc. For ADAS, AVM, etc., there are often large distortions, and relevant internal parameter calibrations are required to correct the distortions, such as using pinhole models, fisheye models, polynomial models, etc. However, even if calibration and correction are performed, if the effect is not good, subsequent computer vision tasks may be affected. Therefore, it is also of great significance to determine the distortion removal effect after the camera distortion removal.

[0003] At the current stage, the determination of the camera distortion removal effect often considers aspects such as the internal parameter matrix, reprojection error, corner coverage rate, and distortion removal situation. Among them, for the internal parameter matrix and reprojection error, they are relatively easy to detect. However, for the distortion removal situation, such as edge fission, distortion rings, and abnormal distortion of the entire image that may occur after distortion correction, it is currently difficult to detect through quantitative indicators. Summary of the Invention

[0004] Aiming at the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: to propose a method and system for detecting abnormal distortion removal of a camera, which can comprehensively evaluate the distortion removal situation by combining straightness and an abnormal distortion removal judgment algorithm, thereby better ensuring the accuracy of the internal parameter calibration of the pinhole model.

[0005] One technical solution adopted by the present invention is: to provide a method for detecting abnormal distortion removal of a camera, including the following steps:

[0006] S1: Obtain the 3D coordinates of each corner point on the calibration board, calibrate the 3D coordinates to obtain the internal parameter matrix and distortion coefficients of the calibration model;

[0007] S2: Obtain the pixel coordinates of each corner point in the calibration board image after distortion removal;

[0008] S3: Sort the pixel coordinates of each corner point in the calibration board image by row and column, and calculate the straightness of the corner points in each row and each column;

[0009] S4: Obtain the undistorted rendering image and the undistorted rendering image with enlarged field of view, perform image analysis and processing on the undistorted rendering image and / or the undistorted rendering image with enlarged field of view, and determine whether there is undistortion abnormality.

[0010] Further, in the step S1, the following sub-steps are included:

[0011] S11: Project the 3D coordinates onto the 2D image plane:

[0012]

[0013] where (x, y) represents the 2D image coordinates, (X, Y, Z) represents the 3D coordinates, and f x represents the focal length in the x direction, and f y represents the focal length in the y direction;

[0014] S12: Construct a pinhole distortion model:

[0015]

[0016] where (x d , y d ) represents the distorted image coordinates, [k1, k2, k3, k4, k5, k6] represents the radial distortion coefficients, [p1, p2] represents the tangential distortion coefficients, and r represents the distance from the 3D coordinate point (x, y) to the image center and r 2 = x 2 + y 2 ;

[0017] S13: Pixel coordinate conversion:

[0018]

[0019] where represents the internal parameter matrix of the camera, (c x , c y ) represents the principal point coordinates, and (u', v') represents the re-projected pixel coordinates obtained by converting the 3D coordinate point (x, y) through the internal parameter matrix;

[0020] S14: Iterate the internal parameter matrix with the goal of minimizing the error value between the re-projected pixel coordinates of the corner points and the actual pixel coordinates.

[0021] Further, in the step S2, the following sub-steps are included:

[0022] S21: Convert the corner pixel coordinates to normalized image coordinates:

[0023]

[0024] where (x'ij , y' ij ) represents the normalized image coordinates, (u ij , v ij ) represents the corner pixel coordinates, i represents the row, and j represents the column;

[0025] S22: Using (x' i＝1j＝1 , y' i＝1j＝1 ) as the starting point, sort the normalized image coordinates by row and column, and initialize the normalized image coordinates to (x' n , y' n ), where n = (i - 1)L + j, and L represents the total number of columns;

[0026] S23: Undistort the normalized image coordinates according to the intrinsic matrix:

[0027]

[0028] where (x', y') represents the normalized image coordinates, and iterate (x' n+1 , y' n+1 ) until convergence to obtain the undistorted normalized coordinates (x correctij , y correctij );

[0029] S24: Convert the undistorted normalized coordinates to pixel coordinates:

[0030]

[0031] where (u correct , v correct ) represents the pixel coordinates converted from the undistorted normalized coordinates.

[0032] Furthermore, the S3 step includes the following sub-steps:

[0033] S31: According to the slope formula and the pixel coordinates of the corner points in each row, calculate the slope and the oblique distance of the corner points in each row:

[0034]

[0035] where a i represents the slope of the i-th row, b i represents the oblique distance of the i-th row, H represents the total number of rows, and L represents the total number of columns;

[0036] S32: According to the slope formula and the pixel coordinates of the corner points in each column, calculate the slope and the oblique distance of the corner points in each column:

[0037]

[0038] where aj represents the slope of the j-th column, b j represents the oblique distance of the j-th column;

[0039] S33: Calculate the straightness of the corner points in each row of the calibration plate image:

[0040]

[0041] where, R i represents the straightness of the i-th row, represents the maximum value of u in the i-th row correctij and represents the minimum value of u in the i-th row correctij ;

[0042] S34: Calculate the straightness of the corner points in each column of the calibration plate image:

[0043]

[0044] where, R j represents the straightness of the j-th column, represents the maximum value of u in the j-th column correctij and represents the minimum value of u in the j-th column correctij ;

[0045] Furthermore, in the step S3, the following sub-steps are further included:

[0046] S35: Compare the calculated straightness of the corner points in each row and each column with the preset straightness threshold. If it does not meet the preset straightness threshold range, it is determined that the undistortion is abnormal, otherwise, enter the step S4.

[0047] Furthermore, the obtaining of the undistorted rendering image and the undistorted rendering image with enlarged field of view includes the following sub-steps:

[0048] S41: According to the getOptimalNewCameraMatrix function of OpenCV, enlarge the field of view of the undistorted calibration plate image to obtain a calibration plate image with an enlarged field of view;

[0049] S42: Render the calibration plate image and / or the calibration plate image with an enlarged field of view white, and generate a distortion mapping according to the internal parameter matrix, radial distortion coefficient, and tangential distortion coefficient through the initUndistortRectifyMap function of OpenCV;

[0050] S43: According to the distortion mapping, remap the calibration board image and / or the magnified field-of-view calibration board image through the remap function of OpenCV to obtain the undistorted rendering image and / or the magnified field-of-view undistorted rendering image.

[0051] Further, performing image analysis processing on the undistorted rendering image and / or the magnified field-of-view undistorted rendering image to determine whether there is undistortion abnormality includes the following sub-steps:

[0052] S44: According to pixel value detection, obtain the boundary coordinate values of the first region in the undistorted rendering image, and determine whether they meet the preset boundary coordinate values. If so, enter S45; otherwise, determine that there is undistortion abnormality in the undistorted rendering image.

[0053] S45: Traverse the pixel values of each pixel point in the first region, and determine whether the pixel value meets the preset pixel value. If so, enter S46; otherwise, determine that there is undistortion abnormality in the undistorted rendering image.

[0054] S46: According to pixel value detection, obtain the boundary coordinate values of the second region in the magnified field-of-view undistorted rendering image, and determine whether they meet the preset boundary coordinate values. If so, determine that there is no undistortion abnormality in the undistorted rendering image; otherwise, determine that there is undistortion abnormality in the undistorted rendering image.

[0055] Further, the obtaining the boundary coordinate values of the first region in the undistorted rendering image according to pixel value detection includes the following sub-steps:

[0056] S441: Let Traverse the pixel values pixel(i,j) of all pixel points that satisfy (i,j), and when pixel(i,j)=0, take the maximum value of i as the first left boundary of the first region;

[0057] S442: Let Traverse the pixel values pixel(i,j) of all pixel points that satisfy (i,j), and when pixel(i,j)=0, take the minimum value of i as the first right boundary of the first region;

[0058] S443: Let Traverse the pixel values pixel(i,j) of all pixel points that satisfy (i,j), and when pixel(i,j)=0, take the maximum value of j as the first upper boundary of the first region;

[0059] S444: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 0, take the minimum value of j as the first lower boundary of the first region;

[0060] Where hight represents the height of the undistorted rendering image, width represents the width of the undistorted rendering image, and α represents the offset.

[0061] Further, the obtaining of the boundary coordinate values of the second region in the undistorted rendering image with enlarged field of view according to pixel value detection includes the following sub-steps:

[0062] S461: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 255, take the minimum value of i as the second left boundary of the second region;

[0063] S462: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 255, take the maximum value of i as the second right boundary of the second region;

[0064] S463: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 255, take the minimum value of j as the second upper boundary of the second region;

[0065] S464: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 255, take the maximum value of j as the second lower boundary of the second region;

[0066] Where hight' represents the height of the undistorted rendering image with enlarged field of view, width' represents the width of the undistorted rendering image with enlarged field of view, and α represents the offset.

[0067] To solve the above technical problems, the second technical solution adopted by the present invention is: to provide a camera undistortion anomaly detection system, including:

[0068] A calibration module for obtaining the 3D coordinates of each corner point on the calibration board, calibrating the 3D coordinates, and obtaining the internal parameter matrix and distortion coefficient of the calibration model;

[0069] A corner point positioning module for obtaining the pixel coordinates after undistortion of each corner point in the calibration board image;

[0070] The straightness calculation module is used to sort the pixel coordinates of each corner point in the calibrated board image after distortion removal by rows and columns, and calculate the straightness of the corner points in each row and each column.

[0071] The distortion removal anomaly determination module is used to obtain the distortion-removed rendering image and the distortion-removed rendering image with enlarged field of view, perform image analysis and processing on the distortion-removed rendering image and / or the distortion-removed rendering image with enlarged field of view, and determine whether there is a distortion removal anomaly.

[0072] The camera distortion removal anomaly detection method and system of the present invention can comprehensively evaluate the distortion removal situation by combining the straightness and the distortion removal anomaly judgment algorithm, so as to better ensure the accuracy of the internal parameter calibration of the pinhole model. Brief Description of the Drawings

[0073] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0074] Figure 1 It is a flowchart of an embodiment of the camera distortion removal anomaly detection method of the present invention.

[0075] Figure 2 For the present invention Figure 1 It is a sub-flowchart of step S1 in the present invention.

[0076] Figure 3 For the present invention Figure 1 It is a sub-flowchart of step S2 in the present invention.

[0077] Figure 4 For the present invention Figure 1 It is a sub-flowchart of step S3 in the present invention.

[0078] Figure 5 For the present invention Figure 1 It is a sub-flowchart of step S4 in the present invention.

[0079] Figure 6 It is a distortion-removed rendering image of an embodiment of the present invention.

[0080] Figure 7 It is a distortion-removed rendering image with enlarged field of view of an embodiment of the present invention.

[0081] Figure 8 It is a structural block diagram of an embodiment of the camera distortion removal anomaly detection system of the present invention. Detailed Embodiments

[0082] The present invention will be further described below with reference to the drawings.

[0083] Please refer to Figure 1, is a flowchart of an implementation manner of the camera undistortion anomaly detection method of the present invention. This implementation manner may specifically include the following steps:

[0084] S1: Obtain the 3D coordinates of each corner point on the calibration board, calibrate the 3D coordinates to obtain the internal parameter matrix and distortion coefficients of the calibration model.

[0085] Specifically, this implementation manner of the camera undistortion anomaly detection is mainly for detecting undistortion anomalies proposed for the pinhole model, and can effectively judge whether the undistortion after calibration is abnormal. After calibration through the pinhole model, the internal parameter matrix can be used, and the straightness of the overall image can be judged by the straightness; secondly, image processing is used to judge whether there are anomalies in undistortion. For the calibration of the pinhole model, the corner points of the calibration board used need to be determined through the original calibration pictures taken. The corner points can be directly obtained by using the corner point acquisition function in OpenCV. The calibration board can generally use a checkerboard or a circular grid, and the purpose is to obtain the coordinates of each corner point of the grid (the corner points of the checkerboard are the positions where black and white meet, and the circular grid is the center position of the circle), and arrange them in order, that is, arrange them from left to right and from top to bottom in sequence, establish the corresponding corner point coordinates (two-dimensional projection points) of the known three-dimensional world coordinate points and their positions on the image plane, and solve the internal parameter and distortion coefficients of the camera through mathematical methods (least squares method, gradient descent method, etc.) (which can be obtained by directly calling the calibrateCamera function provided by opencv).

[0086] Please refer to Figure 2 , this step S1 may include the following sub-steps:

[0087] S11: Project the 3D coordinates onto the 2D image plane:

[0088]

[0089] Among them, (x, y) represents the 2D image coordinates, (X, Y, Z) represents the 3D coordinates, f x represents the focal length in the x direction, f y represents the focal length in the y direction.

[0090] S12: Construct a pinhole distortion model:

[0091]

[0092] Among them, (x d , y d ) represents the distorted image coordinates, [k1, k2, k3, k4, k5, k6] represents the radial distortion coefficients, [p1, p2] represents the tangential distortion coefficients, r represents the distance from the 3D coordinate point (x, y) to the image center and r 2 = x2 +y 2 。

[0093] S13: Pixel coordinate conversion:

[0094]

[0095] Among them, represents the internal parameter matrix of the camera, (c x , c y ) represents the principal point coordinates, and (u', v') represents the reprojected pixel coordinates of the 3D coordinate point (x, y) obtained by conversion through the internal parameter matrix.

[0096] S14: Iterate the internal parameter matrix with the goal of minimizing the error value between the reprojected pixel coordinates of the corner points and the actual pixel coordinates.

[0097] Specifically, the above steps S11 - S14 are the detailed process of using the pinhole model to calibrate the internal parameters of the calibration plate image, and it is also a common calibration process in this field. By iterating the internal parameter matrix with the goal of minimizing the error value between the reprojected pixel coordinates of the corner points and the actual pixel coordinates, the final internal parameter matrix can be obtained. The difference between this embodiment and the common calibration process is that the general radial distortion model contains three radial distortion parameters k1, k2, and k3, while in this embodiment, the established distortion model has six radial distortion parameters k1 - k6, so that it can be applied to cameras with relatively large distortion or large FOV, and obtain a better effect of correcting radial distortion, while ensuring the distortion correction of the edge part of the image. In addition, it can also be found in actual applications that the accuracy of the actual shooting distance after undistortion with six radial distortion parameters is higher.

[0098] S2: Obtain the undistorted pixel coordinates of each corner point in the calibration plate image.

[0099] Specifically, since this solution determines whether there is an abnormality in the undistortion of the calibration plate by calculating the straightness of the corner points in the calibration plate image, and the calculation of the straightness also requires obtaining the pixel coordinates of each corner point in the calibration plate image first. Therefore, in this embodiment, the undistorted pixel coordinates of each intersection point in the calibration plate image are obtained through the calibrated internal parameter matrix and distortion coefficients.

[0100] In some embodiments, please refer to Figure 3 , this step S2 may include the following sub - steps:

[0101] S21: Convert the corner point pixel coordinates into normalized image coordinates:

[0102]

[0103] Among them, (x' ij, y' ij ) represents the normalized image coordinates, (u ij , v ij ) represents the corner pixel coordinates, i represents the row, and j represents the column.

[0104] S22: Taking (x' i＝1j＝1 , y' i＝1j＝1 ) as the initial point, sort the normalized image coordinates by row and column, and initialize the normalized image coordinates as (x' n , y' n ), where n = (i - 1)L + j, and L represents the total number of columns.

[0105] S23: Undistort the normalized image coordinates according to the intrinsic matrix:

[0106]

[0107] Among them, (x', y') represents the normalized image coordinates. Iterate (x' n+1 , y' b+1 ) until convergence to obtain the undistorted normalized coordinates (x correctij , y correctij ).

[0108] S24: Convert the undistorted normalized coordinates to pixel coordinates:

[0109]

[0110] Among them, (u correct , v correct ) represents the pixel coordinates converted from the undistorted normalized coordinates.

[0111] Specifically, the above steps S21 - S24 are the detailed process of obtaining the pixel coordinates of the corner points in the undistorted calibration board according to the calibrated internal parameter matrix and distortion coefficients. Among them, the conversion of the pixel coordinates of the calibration board corner points to the normalized image coordinates in step S21 is a conventional step in undistortion. In this step, the pixel coordinates of the calibration board corner points are the pixel coordinates of the corner points in the calibration board image, that is, the undistorted pixel coordinates, which can be directly obtained through the corner pixel detection function of OpenCV; in step S22, it should be noted that the corner points need to be sorted row by row and column by column from left to right and from top to bottom, and the normalized image coordinates are initialized with the top - left corner point as the initial corner point. That is, in this step, the row and column numbers of the normalized image coordinates need to be initialized in the arrangement with n as the serial number, so as to facilitate the subsequent undistortion calculation. Therefore, let i = 1 represent the corner points in the first row, j = 1 represent the corner points in the first column, and n=(i - 1)L + j, then the above serial number conversion can be achieved; step S23 is to calculate the undistorted normalized coordinates according to the calibrated internal parameter matrix and distortion coefficients obtained from steps S11 - S14 through an iterative method; furthermore, in S24, the undistorted normalized coordinates are converted into pixel coordinates, so as to complete the subsequent straightness calculation.

[0112] S3: Sort the pixel coordinates of each corner point in the calibration board image after undistortion row by row and column by column, and calculate the straightness of the corner points in each row and each column.

[0113] Specifically, since in the above steps S21 - S24, the pixel coordinates of the corner points after undistortion have been obtained, which means that the current image is a calibrated image, therefore, by calculating the straightness of the pixel coordinates of the corner points in each row and each column, the effect of undistortion can be initially determined.

[0114] Please refer to Figure 4 , this step S3 can include the following sub - steps:

[0115] S31: According to the slope formula and the pixel coordinates of the corner points in each row, calculate the slope and the oblique distance of the corner points in each row:

[0116]

[0117] Among them, a i represents the slope of the i - th row, b i represents the oblique distance of the i - th row, H represents the total number of rows, and L represents the total number of columns.

[0118] S32: According to the slope formula and the pixel coordinates of the corner points in each column, calculate the slope and the oblique distance of the corner points in each column:

[0119]

[0120] Among them, aj represents the slope of the j-th column, b j represents the skew distance of the j-th column.

[0121] S33: Calculate the straightness of the corner points in each row of the calibration plate image:

[0122]

[0123] where, R i represents the straightness of the i-th row, represents the maximum value of u in the i-th row correctij and represents the minimum value of u in the i-th row correctij and

[0124] S34: Calculate the straightness of the corner points in each column of the calibration plate image:

[0125]

[0126] where, R j represents the straightness of the j-th column, represents the maximum value of u in the j-th column correctij and represents the minimum value of u in the j-th column correctij and

[0127] Specifically, in this embodiment, the calculation formulas for the slope and skew distance in steps S31 and S32 are actually obtained by converting the slope formula y = ax + b. By using the pixel coordinates of the corner points in each row and each column, the slope and skew distance of the pixel coordinates of the corner points in each row and each column can be calculated. Thus, according to the straightness calculation formula in steps S33 and S34, the straightness of the pixel coordinates of the corner points in each row and each column is calculated.

[0128] In some embodiments, this step S3 may further include the following sub-steps:

[0129] S35: Compare the calculated straightness of the corner points in each row and each column with a preset straightness threshold. If it does not meet the preset straightness threshold range, it is determined that the distortion removal is abnormal; otherwise, it enters step S4.

[0130] Specifically, after calculating the straightness of the undistorted corner pixel coordinates in each row and column, an appropriate straightness threshold can be selected as the judgment criterion according to the actual undistortion correction requirements, and the calculated straightness can be compared with it to preliminarily judge the undistortion effect of the calibration image. However, it should be noted that through the straightness judgment, only the undistortion effect of the corner area in the calibration image can be judged. For the edge position of the image or the edge fission, distortion ring, and abnormal distortion of the whole image after distortion correction, image processing is still needed to judge.

[0131] S4: Obtain the undistorted rendering image and the undistorted rendering image with enlarged field of view, and perform image analysis processing on the undistorted rendering image and / or the undistorted rendering image with enlarged field of view to determine whether there is undistortion abnormality.

[0132] Specifically, in this embodiment, by performing image analysis on the undistorted rendering image of the calibration board image and the undistorted rendering image with enlarged field of view, it is determined whether there is undistortion abnormality.

[0133] In some embodiments, please refer to Figure 5 、 Figure 6 、 Figure 7 , in this step S4, "obtain the undistorted rendering image and the undistorted rendering image with enlarged field of view" may include the following sub-steps:

[0134] S41: According to the getOptimalNewCameraMatrix function of OpenCV, enlarge the field of view of the undistorted calibration board image to obtain a calibration board image with enlarged field of view.

[0135] Specifically, in this embodiment, the image resolution can be reduced through the getOptimalNewCameraMatrix function of OpenCV, so as to enlarge the field of view of the undistorted calibration board image. For example, if the resolution of the calibration image is 3840*2160, the alpha parameter of getOptimalNewCameraMatrix is set to 1, imageSize is set to 3840*2160, and newImgSize is set to 1920*1080, the field of view of the calibration image can be enlarged by 4 times. The purpose of obtaining the calibration board image with enlarged field of view is to obtain the undistorted rendering image with enlarged field of view through the calibration board image with enlarged field of view, so as to detect the undistortion abnormality with edge fission in the undistorted rendering image with enlarged field of view.

[0136] S42: Render the calibration plate image and / or the magnified calibration plate image of the field of view white, and generate a distortion map through the initUndistortRectifyMap function of OpenCV according to the internal parameter matrix, radial distortion coefficients, and tangential distortion coefficients.

[0137] S43: Remap the calibration plate image and / or the magnified calibration plate image of the field of view according to the distortion map through the remap function of OpenCV to obtain the undistorted rendered image and / or the magnified undistorted rendered image of the field of view.

[0138] Specifically, the image processing in this embodiment is mainly achieved through pixel value detection. Therefore, in order to meet the requirements of pixel value detection, it is necessary to render the calibration plate image with the same resolution and / or the magnified calibration plate image of the field of view white, and then, according to the internal parameter matrix and distortion coefficients obtained above, the undistorted rendered image and / or the magnified undistorted rendered image of the field of view can be obtained. The way of undistorting is mainly through the initUndistortRectifyMap function and remap function of OpenCV. Among them, the initUndistortRectifyMap function is used to generate the distortion map, and the remap function can remap the calibration plate image and / or the magnified calibration plate image of the field of view according to this distortion map, so as to obtain the undistorted rendered image and / or the magnified undistorted rendered image of the field of view.

[0139] In some embodiments, in this step S4, "perform image analysis processing on the undistorted rendered image and / or the magnified undistorted rendered image of the field of view to determine whether there is undistortion abnormality" may include the following sub-steps:

[0140] S44: According to pixel value detection, obtain the boundary coordinate values of the first area in the undistorted rendered image, and determine whether they meet the preset boundary coordinate values. If so, enter S45; otherwise, determine that there is undistortion abnormality in the undistorted rendered image.

[0141] Specifically, this step S44 mainly obtains the first area of the undistorted rendered image through pixel value detection of the undistorted rendered image. This first area is actually the largest cropped display area in the undistorted rendered image, that is, the image in this area is valid image information. Intuitively from the undistorted rendered image, this first area is actually the largest inscribed rectangle of the black border in the undistorted rendered image, and the boundary coordinate values are the coordinate values of the four sides of this first area. Furthermore, after obtaining the boundary coordinate values of the first area in the undistorted rendered image, compare them with the preset boundary coordinate values. If they do not meet, it can be directly determined that there is undistortion abnormality in this undistorted rendered image. If they meet, it is still necessary to enter step S45 to further detect this undistorted rendered image.

[0142] In some embodiments, "obtaining the boundary coordinate values of the first region in the undistorted rendering image according to the pixel value detection" in this step S44 may include the following sub-steps:

[0143] S441: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the maximum value of i as the first left boundary of the first region;

[0144] S442: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the minimum value of i as the first right boundary of the first region;

[0145] S443: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the maximum value of j as the first upper boundary of the first region;

[0146] S444: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the minimum value of j as the first lower boundary of the first region;

[0147] Wherein, hight represents the height of the undistorted rendering image, width represents the width of the undistorted rendering image, α represents the offset, pixel(i, j) = 0 represents a point with coordinates (i, j) and a pixel value of 0, and a pixel value of 0 means white.

[0148] Specifically, steps S441 - S444 are the detailed calculation processes of the boundaries in the above-mentioned first region. Among them, the first left boundary is the left boundary of the first region, the first right boundary is the right boundary of the first region, the first upper boundary is the upper boundary of the first region, and the first lower boundary is the lower boundary of the first region. In this embodiment, the preset boundary coordinate values can be set as:

[0149]

[0150] Wherein, y u represents the ordinate of the first upper boundary, y d represents the ordinate of the first lower boundary, x l represents the abscissa of the first left boundary, x rRepresents the abscissa of the first right boundary. When the obtained first left boundary, first right boundary, first upper boundary, and first lower boundary simultaneously satisfy the preset boundary coordinate values, it is determined that there is no de-distortion anomaly in the de-distorted rendering image. At this time, step S45 can be jumped to and the pixel values in the above first region can be further detected. Otherwise, there is a de-distortion anomaly.

[0151] S45: Traverse the pixel values of each pixel point in the first region, and determine whether the pixel value satisfies the preset pixel value. If so, enter S46. Otherwise, it is determined that there is a de-distortion anomaly in the de-distorted rendering image.

[0152] Specifically, in step S44 above, only the boundaries of the first region are determined. For whether there are anomalies inside the first region, pixel value detection needs to be performed on each pixel point inside it. If the pixel value of each pixel point in the first region is 0 (i.e., white), it means that there is a distortion anomaly in the de-distorted rendering image, and step S46 can be entered to further determine the de-distorted rendering image with magnified field of view. Otherwise, it is determined that there is a de-distortion anomaly in the de-distorted rendering image. It should be noted that through steps S44, S45, and steps S441 - S444, only the de-distortion anomaly of the de-distorted rendering image can be determined. For the de-distorted rendering image with magnified field of view, it is also necessary to further change the pixel value detection algorithm and the preset boundary coordinate values.

[0153] S46: According to the pixel value detection, obtain the boundary coordinate values of the second region in the de-distorted rendering image with magnified field of view, and determine whether they satisfy the preset boundary coordinate values. If so, it is determined that there is no de-distortion anomaly in the de-distorted rendering image. Otherwise, it is determined that there is a de-distortion anomaly in the de-distorted rendering image.

[0154] Specifically, the calculation of the boundary coordinate values of the second region in the de-distorted rendering image with magnified field of view in this step S46 is similar to the calculation of the first region in step S44 above, and the determination principle is also similar. The difference is that the purpose of pixel value detection for the de-distorted rendering image with magnified field of view is to determine the non-effective image region in the calibration plate image (mainly to check whether there is de-distortion edge fission).

[0155] In some embodiments, the "obtain the boundary coordinate values of the second region in the de-distorted rendering image with magnified field of view according to the pixel value detection" in this step S46 may include the following sub-steps:

[0156] S461: Let Traverse the pixel values pixel(i,j) of all pixel points that satisfy (i,j), and when pixel(i,j) = 255, take the minimum value of i as the second left boundary of the second region;

[0157] S462: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 255, take the maximum value of i as the second right boundary of the second region;

[0158] S463: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 255, take the minimum value of j as the second upper boundary of the second region;

[0159] S464: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 255, take the maximum value of j as the second lower boundary of the second region;

[0160] Wherein, hight' represents the height of the de - distorted rendering image with magnified field of view, width' represents the width of the de - distorted rendering image with magnified field of view, α represents the offset, pixel(i, j) = 255 represents the point with coordinates (i, j) and pixel value 255, and the pixel value 255 represents black.

[0161] Specifically, the steps of S461 - S464 are the detailed calculation process of the coordinate values of each boundary in the above - mentioned second region. Among them, the second left boundary is the left boundary of the second region, the second right boundary is the right boundary of the second region, the second upper boundary is the upper boundary of the second region, and the second lower boundary is the lower boundary of the second region. In this embodiment, the preset boundary coordinate values can be set as:

[0162]

[0163] Wherein, y' u represents the ordinate of the first upper boundary, y' d represents the ordinate of the first lower boundary, x' l represents the abscissa of the first left boundary, x' r represents the abscissa of the first right boundary, M and M' both represent the preset error. When the obtained second left boundary, second right boundary, second upper boundary and second lower boundary simultaneously satisfy the preset boundary coordinate values, it is determined that there is no de - distortion abnormality in the de - distorted rendering image with magnified field of view, otherwise there is a de - distortion abnormality.

[0164] Please refer to Figure 8, which is a structural block diagram of an embodiment of the camera distortion removal anomaly detection system of the present invention. The camera distortion removal anomaly detection system of this embodiment is used to implement the camera distortion removal anomaly detection method described in the above embodiment. Specifically, the camera distortion removal anomaly detection system of this embodiment includes a calibration module 100, a corner point positioning module 200, a straightness calculation module 300, and a distortion removal anomaly determination module 400. Among them:

[0165] The calibration module 100 is used to obtain the 3D coordinates of each corner point on the calibration board, calibrate the 3D coordinates, and obtain the internal parameter matrix and distortion coefficient of the calibration model;

[0166] The corner point positioning module 200 is used to obtain the pixel coordinates of each corner point in the calibration board image after distortion removal;

[0167] The straightness calculation module 300 is used to sort the pixel coordinates of each corner point in the calibration board image after distortion removal by row and column, and calculate the straightness of each row of corner points and each column of corner points;

[0168] The distortion removal anomaly determination module 400 is used to obtain the distortion removal rendering image and the distortion removal rendering image with enlarged field of view, perform image analysis and processing on the distortion removal rendering image and / or the distortion removal rendering image with enlarged field of view, and determine whether there is a distortion removal anomaly.

[0169] The present invention comprehensively evaluates the distortion removal situation by combining the straightness and the distortion removal anomaly judgment algorithm, so as to better ensure the accuracy of the internal parameter calibration of the pinhole model.

[0170] The above content only expresses the preferred embodiments of the present invention, and its description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A method for detecting abnormal distortion removal of a camera, characterized in that, It includes the following steps: S1: Obtain the 3D coordinates of each corner point on the calibration board, calibrate the 3D coordinates to obtain the internal parameter matrix and distortion coefficients of the calibration model; S2: Obtain the undistorted pixel coordinates of each corner point in the calibration board image; S3: Sort the undistorted pixel coordinates of each corner point in the calibration board image by row and column, and calculate the straightness of the corner points in each row and each column; S4: Obtain the undistorted rendering image and the undistorted rendering image with enlarged field of view, perform image analysis and processing on the undistorted rendering image and / or the undistorted rendering image with enlarged field of view, and determine whether there is undistortion abnormality.

2. The camera distortion removal anomaly detection method according to claim 1, wherein, In the step S1, it includes the following sub-steps: S11: Project the 3D coordinates onto the 2D image plane: where (x, y) represents 2D image coordinates, and (X, Y, Z) represents 3D coordinates, and f x represents the focal length in the x direction, and f y represents the focal length in the y direction; S12: Construct a pinhole distortion model: Among them, (x d , y d ) represents the image coordinates after distortion, [k1, k2, k3, k4, k5, k6] represents the radial distortion coefficients, [p1, p2] represents the tangential distortion coefficients, r represents the distance from the 3D coordinate point (x, y) to the image center and r 2 = x 2 + y 2 ; S13: Pixel coordinate conversion: Among them, represents the internal parameter matrix of the camera, (c x , c y ) represents the principal point coordinates, and (u', v') represents the re-projected pixel coordinates obtained by transforming the 3D coordinate points (x, y) through the internal parameter matrix; S14: Iterate the internal parameter matrix with the goal of minimizing the error value between the re-projected pixel coordinates of the corner points and the actual pixel coordinates.

3. The camera distortion removal anomaly detection method according to claim 1, wherein In the step S2, it includes the following sub-steps: S21: Convert the corner point pixel coordinates into normalized image coordinates: Among them, (x' ij , y' ij ) represents the normalized image coordinates, (u ij , v ij ) represents the corner pixel coordinates, i represents the row, and j represents the column; S22: Starting from (x' i＝1j＝1 , y' i＝1j＝1 ), sort the normalized image coordinates row by row and column by column, and initialize the normalized image coordinates to (x' n , y' n ), where n = (i - 1)L + j, and L represents the total number of columns; S23: Undistort the normalized image coordinates according to the internal parameter matrix: Among them, (x', y') represents the normalized image coordinates, and (x' n+1 , y' n+1 ) is iterated until convergence to obtain the undistorted normalized coordinates (x correctij , y correctij ); S24: Convert the undistorted normalized coordinates into pixel coordinates: where (u correct , v correct ) represents the pixel coordinates converted from the normalized coordinates after distortion removal.

4. The camera distortion removal anomaly detection method according to claim 1, wherein The step S3 includes the following sub-steps: S31: Calculate the slope and oblique distance of the corner points in each row according to the slope formula and the pixel coordinates of the corner points in each row: where a i represents the slope of the i-th row, b i represents the offset of the i-th row, H represents the total number of rows, and L represents the total number of columns; S32: Calculate the slope and oblique distance of the corner points in each column according to the slope formula and the pixel coordinates of the corner points in each column: Among them, a j represents the slope of the j-th column, and b j represents the offset of the j-th column; S33: Calculate the straightness of the corner points in each row in the calibration board image: Among them, R i represents the straightness of the i-th row, represents the maximum value of u in the i-th row, correctij and represents the minimum value of u in the i-th row. correctij ​ S34: Calculate the straightness of the corner points in each column in the calibration board image: Among them, R j represents the straightness of the j-th column, represents the maximum value of u in the j-th column correctij , represents the minimum value of u in the j-th column correctij .

5. The camera distortion removal anomaly detection method according to claim 4, wherein, In the step S3, it also includes the following sub-steps: S35: Compare the calculated straightness of the corner points in each row and each column with the preset straightness threshold. If it does not meet the preset straightness threshold range, it is determined that there is undistortion abnormality, otherwise, enter the step S4.

6. The camera distortion removal anomaly detection method according to claim 2, wherein, The obtaining of the undistorted rendering image and the undistorted rendering image with enlarged field of view includes the following sub-steps: S41: Enlarge the field of view of the undistorted calibration board image according to the getOptimalNewCameraMatrix function of OpenCV to obtain the calibration board image with enlarged field of view; S42: Render the calibration board image and / or the calibration board image with enlarged field of view white, and generate a distortion map according to the internal parameter matrix, radial distortion coefficient and tangential distortion coefficient through the initUndistortRectifyMap function of OpenCV; S43: Remap the calibration board image and / or the calibration board image with enlarged field of view according to the distortion map through the remap function of OpenCV to obtain the undistorted rendering image and / or the undistorted rendering image with enlarged field of view.

7. The camera distortion removal anomaly detection method according to claim 6, wherein, The performing of image analysis and processing on the undistorted rendering image and / or the undistorted rendering image with enlarged field of view to determine whether there is undistortion abnormality includes the following sub-steps: S44: Based on pixel value detection, obtain the boundary coordinate values of the first region in the de-distorted rendering image, and determine whether the preset boundary coordinate values are satisfied. If so, proceed to S45; otherwise, determine that there is a de-distortion anomaly in the de-distorted rendering image. S45: Traverse the pixel values of each pixel point in the first region, and determine whether the pixel values satisfy the preset pixel values. If so, proceed to S46; otherwise, determine that there is a de-distortion anomaly in the de-distorted rendering image. S46: Based on pixel value detection, obtain the boundary coordinate values of the second region in the de-distorted rendering image with enlarged field of view, and determine whether the preset boundary coordinate values are satisfied. If so, determine that there is no de-distortion anomaly in the de-distorted rendering image; otherwise, determine that there is a de-distortion anomaly in the de-distorted rendering image.

8. The camera distortion removal anomaly detection method according to claim 7, wherein, The step of obtaining the boundary coordinate values of the first region in the de-distorted rendering image based on pixel value detection includes the following sub-steps: S441: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the maximum value of i as the first left boundary of the first region; S442: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the minimum value of i as the first right boundary of the first region; S443: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 0, take the maximum value of j as the first upper boundary of the first region; S444: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 0, take the minimum value of j as the first lower boundary of the first region; Where hight represents the height of the de-distorted rendering image, width represents the width of the de-distorted rendering image, and α represents the offset.

9. The camera distortion removal anomaly detection method according to claim 7, characterized in that, The step of obtaining the boundary coordinate values of the second region in the de-distorted rendering image with enlarged field of view based on pixel value detection includes the following sub-steps: S461: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 255, take the minimum value of i as the second left boundary of the second region; S462: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 255, take the maximum value of i as the second right boundary of the second region; S463: Let Traverse the pixel values pixel(i, j) of all pixel points satisfying (i, j), and when pixel(i, j) = 255, take the minimum value of j as the second upper boundary of the second region; S464: Let Traverse the pixel values pixel(i, j) of all pixel points that satisfy (i, j), and when pixel(i, j) = 255, take the maximum value of j as the second lower boundary of the second region; Where hight' represents the height of the de-distorted rendering image with enlarged field of view, width' represents the width of the de-distorted rendering image with enlarged field of view, and α represents the offset.

10. A camera distortion removal anomaly detection system, characterized in that, It includes: A calibration module, which is used to obtain the 3D coordinates of each corner point on the calibration board, calibrate the 3D coordinates, and obtain the internal parameter matrix and distortion coefficient of the calibration model; A corner point positioning module, which is used to obtain the pixel coordinates of each corner point in the calibration board image after de-distortion; A straightness calculation module, which is used to sort the pixel coordinates of each corner point in the calibration board image after de-distortion by row and column, and calculate the straightness of each row of corner points and each column of corner points; A de-distortion anomaly determination module, which is used to obtain the de-distorted rendering image and the de-distorted rendering image with enlarged field of view, perform image analysis and processing on the de-distorted rendering image and / or the de-distorted rendering image with enlarged field of view, and determine whether there is a de-distortion anomaly.

Images