Image correction method and device for vehicle camera, terminal and storage medium

Abstract

The application relates to the field of vehicle vision technology, in particular to an image correction method and device for a vehicle camera, a terminal and a storage medium. A standard intrinsic parameter matrix and a standard extrinsic parameter matrix corresponding to a standard camera after calibration on a standard vehicle and an actual intrinsic parameter matrix and an actual extrinsic parameter matrix corresponding to an actual camera on an actual vehicle are acquired; perspective transformation data is determined according to the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix and the actual extrinsic parameter matrix; an actual image collected by the actual camera is acquired, and a target image is determined according to the perspective transformation data and the actual image. The application can project a photograph taken by the actual camera at a corresponding position on the same vehicle model to a view plane corresponding to the standard camera, so as to solve the technical problem that the camera has a view angle error caused by installation error in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle vision technology, in particular to an image correction method and device for vehicle cameras, a terminal and a storage medium. BACKGROUND

[0002] At present, computer vision methods have been widely applied to intelligent auxiliary driving systems. In the task of automatic driving based on vision, deep learning technology provides a powerful solution for target detection and recognition. Generally speaking, the installation positions of each camera on the same vehicle model should be exactly the same, but in actual situations, there will be errors in the orientation angle and position offset, resulting in different images of the same object taken by multiple cameras at the corresponding positions on the same vehicle model. The image data collected in this case with perspective errors is not conducive to the training of subsequent deep learning networks.

[0003] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0004] The present application provides an image correction method and device for vehicle cameras, a terminal and a storage medium to solve the technical problem of perspective error of cameras at corresponding positions on the same vehicle model due to installation errors in related technologies.

[0005] To achieve the above-mentioned purpose, the following technical solutions are adopted in the present application:

[0006] The first aspect of the present application provides an image correction method for vehicle cameras, comprising the following steps:

[0007] Obtaining a standard intrinsic parameter matrix, a standard extrinsic parameter matrix corresponding to a standard camera after calibration on a standard vehicle, and an actual intrinsic parameter matrix and an actual extrinsic parameter matrix corresponding to an actual camera on an actual vehicle, wherein the actual vehicle and the standard vehicle are of the same vehicle model, and the actual camera and the standard camera are installed at corresponding positions on the vehicle;

[0008] According to the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix and the actual extrinsic parameter matrix, determine the perspective transformation data, wherein the perspective transformation data is used to reflect the transformation relationship between the image coordinates of the standard camera and the actual camera;

[0009] Obtaining an actual image collected by the actual camera, and determining a target image according to the perspective transformation data and the actual image, wherein the target image is used to reflect the shooting effect of the actual image corresponding to the standard camera.

[0010] According to the technical means, the embodiment of the present application can project the actual camera shooting photos of the same vehicle model at the corresponding position to the standard camera view plane, so as to solve the technical problem of the existing technology that the camera has a view angle error due to installation error.

[0011] Optionally, in an embodiment of the present application, the method for obtaining the standard intrinsic matrix and the standard extrinsic matrix comprises:

[0012] Obtaining a plurality of shooting images obtained by the standard camera shooting the checkerboard calibration board, wherein each of the shooting images corresponds to a different shooting angle;

[0013] Obtaining a plurality of corner point coordinates of the checkerboard calibration board in each of the shooting images, and determining the intrinsic matrix and the extrinsic matrix corresponding to the standard camera according to the checkerboard calibration board and the corner point coordinates;

[0014] Iteratively optimizing the intrinsic matrix and the extrinsic matrix according to a target optimization algorithm to obtain the standard intrinsic matrix and the standard extrinsic matrix.

[0015] According to the technical means, the embodiment of the present application can calibrate and optimize the intrinsic matrix and the extrinsic matrix of the standard camera, so as to improve the correction effect of the subsequent view angle error.

[0016] Optionally, in an embodiment of the present application, the method for determining the perspective transformation data according to the standard intrinsic matrix, the standard extrinsic matrix, the actual intrinsic matrix and the actual extrinsic matrix comprises:

[0017] Obtaining a standard offset vector according to the standard extrinsic matrix, and obtaining an actual offset vector according to the actual extrinsic matrix, wherein the standard offset vector and the actual offset vector are both used to reflect the offset amount between the world coordinate system and the camera coordinate system;

[0018] Obtaining first product data corresponding to the actual intrinsic matrix and the actual offset vector, and second product data corresponding to the standard intrinsic matrix and the standard offset vector;

[0019] Determining the perspective transformation data according to the first product data and the second product data.

[0020] According to the technical means, the embodiment of the present application can determine the perspective transformation data of the actual camera and the standard camera, and project the actual image to the view plane of the standard camera through the perspective transformation data. This projection method is simple and convenient, and does not need to change the upstream and downstream modules, is easy to deploy, and does not have a significant impact on the calculation amount and the time delay of the system.

[0021] Optionally, in an embodiment of the present application, the acquiring the actual image captured by the actual camera comprises:

[0022] acquiring a target distortion parameter corresponding to the standard camera;

[0023] acquiring an original image captured by the actual camera, and performing de-distortion on the original image according to the target distortion parameter to obtain the actual image.

[0024] According to the above technical means, the embodiment of the present application can correct the image distortion caused by the inherent properties of the actual camera, and further improve the image quality.

[0025] Optionally, in an embodiment of the present application, the acquiring the target distortion parameter corresponding to the standard camera comprises:

[0026] taking each of the corner point coordinates as a plurality of distortion corner point coordinates, and determining, according to the checkerboard calibration board, an un-distorted corner point coordinate corresponding to each of the distortion corner point coordinates;

[0027] determining a distortion model according to each of the distortion corner point coordinates and the un-distorted corner point coordinate corresponding to each of the distortion corner point coordinates, wherein the distortion model is used to reflect a mapping relationship between the distortion corner point coordinates and the un-distorted corner point coordinates;

[0028] determining a target distortion parameter according to the distortion model.

[0029] According to the above technical means, the embodiment of the present application can construct a distortion model for reflecting the mapping relationship between the distortion corner point coordinates and the un-distorted corner point coordinates, and determine the target distortion parameter corresponding to the standard camera through the distortion model, which is used for subsequent de-distortion of the actual image captured by the actual camera.

[0030] Optionally, in an embodiment of the present application, the determining the target distortion parameter according to the distortion model comprises:

[0031] iteratively optimizing model parameters of the distortion model according to a loss function of the distortion model until a minimum function value of the loss function is obtained;

[0032] determining the target distortion parameter according to the model parameters of the distortion model corresponding to the minimum function value.

[0033] According to the above technical means, the embodiment of the present application can optimize the distortion parameter, and the target distortion parameter obtained after optimization can achieve better de-distortion effect.

[0034] Optionally, in an embodiment of the present application, the determining the target image according to the perspective transformation data and the actual image comprises:

[0035] obtaining first image coordinates corresponding to each pixel point in the actual image respectively;

[0036] determining second image coordinates corresponding to each pixel point respectively according to the perspective transformation data and the first image coordinates of each pixel point;

[0037] determining the target image according to each pixel point and the second image coordinates of each pixel point.

[0038] According to the above technical means, the embodiment of the present application can project each pixel point in the actual image to the view plane of the standard camera to obtain the target image after the perspective error is eliminated.

[0039] The second aspect embodiment of the present application provides an image correction device for a vehicle camera, comprising:

[0040] an obtaining module, configured to obtain a standard intrinsic parameter matrix and a standard extrinsic parameter matrix corresponding to a standard camera after calibration on a standard vehicle, and an actual intrinsic parameter matrix and an actual extrinsic parameter matrix corresponding to an actual camera on an actual vehicle, wherein the actual vehicle and the standard vehicle are of the same vehicle type, and the actual camera and the standard camera are installed at corresponding positions of the vehicle;

[0041] a determining module, configured to determine perspective transformation data according to the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix and the actual extrinsic parameter matrix, wherein the perspective transformation data is used to reflect the transformation relationship between the image coordinates of the standard camera and the actual camera;

[0042] a correction module, configured to obtain an actual image collected by the actual camera, and determine a target image according to the perspective transformation data and the actual image, wherein the target image is used to reflect the shooting effect of the actual image under the standard camera.

[0043] The third aspect embodiment of the present application provides a terminal device, comprising a memory, a processor and an image correction program for a vehicle camera stored in the memory and executable on the processor, and when the processor executes the image correction program for a vehicle camera, the steps of the image correction method for a vehicle camera according to any one of the above are implemented.

[0044] The fourth aspect embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores an image correction program for a vehicle camera, and when the processor executes the image correction program for a vehicle camera, the steps of the image correction method for a vehicle camera according to any one of the above are implemented.

[0045] Advantages of the present application:

[0046] (1) The present application can project the actual camera's shooting photos of the corresponding position on the same vehicle model to the standard camera's view plane, thereby reducing the actual camera's view angle error caused by installation error.

[0047] (2) The present application calibrates and optimizes the intrinsic matrix, extrinsic matrix and distortion parameters of the standard camera, thereby improving the subsequent view angle error correction effect and distortion removal effect.

[0048] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0050] Figure 1 A flowchart of the image correction method for vehicle camera of the present application;

[0051] Figure 2 A perspective transformation diagram of the present application;

[0052] Figure 3 A common camera layout structure diagram for autonomous driving of the present application;

[0053] Figure 4 A structure diagram of the image correction device for vehicle camera of the embodiment of the present application;

[0054] Figure 5 An internal structure principle block diagram of the terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0055] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0056] The image correction method for vehicle camera, device, terminal and storage medium of the embodiment of the present application are described below with reference to the accompanying drawings. In view of the network congestion caused by the whole vehicle power-on mentioned in the above background art, the present application provides an image correction method for vehicle camera, in which the standard intrinsic parameter matrix, the standard extrinsic parameter matrix corresponding to the standard camera after calibration on the standard vehicle, and the actual intrinsic parameter matrix, the actual extrinsic parameter matrix corresponding to the actual camera on the actual vehicle are obtained, wherein the actual vehicle and the standard vehicle are of the same vehicle model, and the actual camera and the standard camera are installed at corresponding positions of the vehicle. The perspective transformation data is determined according to the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix and the actual extrinsic parameter matrix, wherein the perspective transformation data is used to reflect the transformation relationship between the image coordinates of the standard camera and the actual camera. The actual image collected by the actual camera is obtained, and the target image is determined according to the perspective transformation data and the actual image, wherein the target image is used to reflect the shooting effect of the actual image under the standard camera. Thus, the shooting photo of the actual camera at the corresponding position on the same vehicle model is projected to the corresponding view plane of the standard camera, so as to solve the technical problem of the view angle error of the camera caused by the installation error in the prior art.

[0057] For example, the standard camera on the standard vehicle is calibrated to obtain the standard intrinsic parameter matrix K s and the standard extrinsic parameter matrix R s . The actual intrinsic parameter matrix K and the actual extrinsic parameter matrix R of the actual camera on the actual vehicle are measured. The actual image shot by the actual camera is obtained, and the perspective transformation data between the actual camera and the standard camera is calculated through K s , R s , K and R. The actual image is projected from the view plane of the actual camera to the view plane of the standard camera through the perspective transformation data, i.e. the target image is obtained. The view angle error caused by the installation error of the actual camera is corrected in the target image.

[0058] Specifically, Figure 1 is a flowchart of an image correction method for vehicle camera provided by the embodiment of the present application.

[0059] As shown in Figure 1 , the image correction method for vehicle camera comprises the following steps:

[0060] S100, obtaining the standard intrinsic parameter matrix, the standard extrinsic parameter matrix corresponding to the standard camera after calibration on the standard vehicle, and the actual intrinsic parameter matrix, the actual extrinsic parameter matrix corresponding to the actual camera on the actual vehicle, wherein the actual vehicle and the standard vehicle are of the same vehicle model, and the actual camera and the standard camera are installed at corresponding positions of the vehicle.

[0061] Specifically, the installation position of the standard camera on the standard vehicle is a standard position, which can be used as the installation standard of the camera at the corresponding position of the same vehicle model. The actual vehicle and the standard vehicle are the same vehicle model, and the actual camera and the standard camera are located at the corresponding positions of the vehicle, Figure 2 An example of a common camera layout structure for autonomous driving is shown. Due to slight installation errors in the production process, there may be deviations between the actual camera installation position and the standard position, resulting in a certain visual angle error of the actual camera relative to the standard camera. For example Figure 3 As shown, due to the different positions of the observation points A and B, the images observed by A and B for the same target object are inconsistent, i.e., there is a visual angle error between A and B. To solve this problem, the embodiment needs to obtain the intrinsic matrix and extrinsic matrix of the standard camera and the actual camera, i.e., the standard intrinsic matrix, the standard extrinsic matrix, the actual intrinsic matrix, and the actual extrinsic matrix.

[0062] In one embodiment, the method for obtaining the standard intrinsic matrix and the standard extrinsic matrix comprises:

[0063] Obtaining a plurality of shooting images obtained by shooting the checkerboard calibration board by the standard camera, wherein each shooting image corresponds to a different shooting angle;

[0064] Obtaining a plurality of corner point coordinates of the checkerboard calibration board in each shooting image, and determining the intrinsic matrix and the extrinsic matrix corresponding to the standard camera according to the checkerboard calibration board and the corner point coordinates;

[0065] Iteratively optimizing the intrinsic matrix and the extrinsic matrix according to a target optimization algorithm to obtain the standard intrinsic matrix and the standard extrinsic matrix.

[0066] Specifically, the embodiment calibrates the standard camera by using the checkerboard calibration board. The world coordinate system and the grid size of the checkerboard calibration board are pre-set, so the physical coordinates of each corner point of the checkerboard calibration board in the world coordinate system can be calculated. A group of shooting images are obtained by shooting the checkerboard calibration board from different angles by using the standard camera. The pixel coordinates of the feature points such as the corner points of the checkerboard calibration board in each shooting image are detected, i.e., a plurality of corner point coordinates are obtained. The intrinsic and extrinsic matrices of the standard camera can be obtained through the physical coordinates of each corner point and the corner point coordinates. Since the calibration results of the standard camera are used to correct the visual errors of other cameras, the reliability of the intrinsic and extrinsic parameter matrices of the standard camera is closely related to the correction effect. In order to improve the reliability of the intrinsic and extrinsic parameter matrices, the embodiment also iteratively optimizes the intrinsic and extrinsic parameter matrices by using a pre-set target optimization algorithm.

[0067] For example, a checkerboard calibration board of Zhang Zhengyou calibration method is prepared, and a standard camera is used to capture a group of images from different angles. Feature points such as corner points in the images are detected, and pixel coordinates of the corner points are obtained, that is, a plurality of corner point coordinates are obtained. An intrinsic matrix is solved according to the corner point coordinates (where f x , f y and c x , c y respectively represent focal length and pixel coordinate translation), and an extrinsic matrix (where R is a rotation matrix, and t is a translation vector). Then, a preset target optimization algorithm is used to optimize the solved intrinsic matrix and extrinsic matrix.

[0068] In an embodiment, the checkerboard calibration board is a checkerboard calibration board of Zhang Zhengyou calibration method. Zhang Zhengyou calibration method mainly includes three steps: solving the product of the intrinsic matrix and the extrinsic matrix, solving the intrinsic matrix, and solving the extrinsic matrix.

[0069] In an embodiment, the target optimization algorithm is L-M (Levenberg-Marquardt) algorithm.

[0070] Specifically, L-M algorithm is an optimization algorithm widely used in nonlinear least squares problems. It combines the advantages of Gauss-Newton and gradient descent method, has fast convergence speed characteristics, is not sensitive to initial value points, and has good global approximation performance. In the L-M algorithm, for each parameter to be optimized, the Jacobian matrix of the parameter to be optimized is calculated in each iteration optimization. When the size of the Jacobian matrix is less than the preset size, it means that the current point is relatively flat, and the Gauss-Newton algorithm is used as the target optimization algorithm at this time. When the size of the Jacobian matrix is greater than or equal to the preset size, it means that the current point is relatively curved, and the gradient descent method is used as the target optimization algorithm at this time.

[0071] In S200, perspective transformation data is determined according to the standard intrinsic matrix, the standard extrinsic matrix, the actual intrinsic matrix, and the actual extrinsic matrix, where the perspective transformation data is used to reflect the transformation relationship between the image coordinates of the standard camera and the actual camera.

[0072] Specifically, the intrinsic matrix of the camera can reflect the transformation relationship from the camera coordinate system to the image coordinate system, and the extrinsic matrix of the camera can reflect the transformation relationship between the world coordinate system and the camera coordinate system. Therefore, the standard intrinsic matrix and the standard extrinsic matrix can reflect the transformation relationship between the image coordinate system corresponding to the standard camera and the world coordinate system, and the actual intrinsic matrix and the actual extrinsic matrix can reflect the transformation relationship between the image coordinate system corresponding to the actual camera and the world coordinate system. Through the transformation relationship between the two image coordinate systems and the world coordinate system respectively, the transformation relationship between the two image coordinate systems can be analyzed, and the perspective transformation data between the standard camera and the actual camera is obtained.

[0073] For example, the image coordinates (X img , Y img ) of a pixel point A in an actual image captured by an actual camera are obtained, and the pixel point A can be converted from the image coordinate system of the actual camera to the world coordinate system through the actual intrinsic matrix and the actual extrinsic matrix, that is, the world coordinates of the pixel point A are (X world , Y world ). The pixel point A can be converted from the world coordinate system to the image coordinate system of the standard camera through the standard intrinsic matrix and the standard extrinsic matrix, that is, the image coordinates of the corresponding point of the pixel point A under the standard camera are (X imgs , Y imgs ), thereby realizing the transformation of the pixel point A from the image coordinate system of the actual camera to the image coordinate system of the standard camera. Therefore, the perspective transformation data between the standard camera and the actual camera can be determined according to the standard intrinsic matrix, the standard extrinsic matrix, the actual intrinsic matrix, and the actual extrinsic matrix.

[0074] In one embodiment, the perspective transformation data between the standard camera and the actual camera is determined according to the standard intrinsic matrix, the standard extrinsic matrix, the actual intrinsic matrix, and the actual extrinsic matrix, including:

[0075] obtaining a standard offset vector according to the standard extrinsic matrix and an actual offset vector according to the actual extrinsic matrix, wherein the standard offset vector and the actual offset vector are both used to reflect the offset amount between the world coordinate system and the camera coordinate system;

[0076] obtaining first product data corresponding to the actual intrinsic matrix and the actual offset vector and second product data corresponding to the standard intrinsic matrix and the standard offset vector;

[0077] determining the perspective transformation data according to the first product data and the second product data.

[0078] Specifically, the role of the perspective transformation data in the embodiment is to project the actual camera image to the standard camera view plane, and two perspective transformations are needed, one perspective transformation is realized based on the product data of the actual intrinsic matrix and the actual offset vector, and the other perspective transformation is realized based on the product data of the standard intrinsic matrix and the standard offset vector. The transformation relationship between the image coordinates of the standard camera and the actual camera can be obtained through the two product data, that is, the perspective transformation data is obtained. Through the perspective transformation data, the actual image collected by the actual camera can be projected to the view plane of the standard camera to correct the view angle error of the actual camera.

[0079] For example, assuming that the actual distance of the standard camera and the actual camera to a far target is the same (the slight difference can be ignored), the actual image can be projected to the view plane of the standard camera through two perspective transformations, and the perspective transformation data is as follows:

[0080]

[0081] wherein, k s , t s is the standard intrinsic matrix and the standard offset vector; k, t is the actual intrinsic matrix and the actual offset vector; p img is the image coordinate corresponding to the actual camera; p s is the image coordinate corresponding to the standard camera.

[0082] S300, acquiring an actual image collected by an actual camera, and determining a target image according to the perspective transformation data and the actual image, wherein the target image is used to reflect the shooting effect of the actual image under the standard camera.

[0083] Specifically, since the actual camera may have a view angle error due to installation error, it is necessary to project the actual image to the view plane where the standard camera is located through the perspective transformation data to correct the view angle error of the actual camera, and obtain the projection image of the actual image under the standard camera, that is, the target image.

[0084] In one embodiment, acquiring the actual image collected by the actual camera includes:

[0085] Acquiring a target distortion parameter corresponding to the standard camera;

[0086] Acquiring an original image collected by the actual camera, and performing de-distortion on the original image according to the target distortion parameter to obtain the actual image.

[0087] Specifically, distortion is an inherent property of a camera, and the existence of distortion will cause the image quality to decrease. Therefore, after obtaining the original image of the actual camera, the original image needs to be de-distorted according to the pre-determined target distortion parameter, and the de-distorted image is taken as the actual image collected by the actual camera, and then other subsequent operations are performed. The embodiment can eliminate the deformation of the captured image caused by the actual camera itself through de-distortion, and further improve the image quality of the captured image.

[0088] In one embodiment, the target distortion parameter is a radial distortion parameter.

[0089] In one embodiment, obtaining the target distortion parameter corresponding to the standard camera comprises:

[0090] Taking each corner point coordinate as a plurality of distortion corner point coordinates, determining the non-distortion corner point coordinates corresponding to each distortion corner point coordinate according to the checkerboard calibration board;

[0091] Determining a distortion model according to each distortion corner point coordinate and the non-distortion corner point coordinates corresponding to each distortion corner point coordinate, wherein the distortion model is used to reflect the mapping relationship between the distortion corner point coordinates and the non-distortion corner point coordinates;

[0092] Determining the target distortion parameter according to the distortion model.

[0093] Specifically, since distortion is an inherent property of a camera, the captured images of the checkerboard calibration board usually have a certain degree of distortion. The corner point coordinates detected by corner point detection on these captured images are the distortion corner point coordinates. Since the world coordinate system and the grid size of the checkerboard calibration board are pre-set, the normal coordinates of each corner point, i.e., the non-distortion corner point coordinates, can be determined according to the checkerboard calibration board. A distortion model reflecting the mapping relationship between the distortion corner point coordinates and the non-distortion corner point coordinates can be constructed by the detected plurality of distortion corner point coordinates and the corresponding non-distortion corner point coordinates. The model parameters of the distortion model contain the distortion parameters, so the target distortion parameter corresponding to the standard camera can be determined according to the obtained distortion model, which is used for subsequent de-distortion of the actual image collected by the actual camera.

[0094] For example, the checkerboard corner point positions in the captured image are detected by the corner point detection algorithm cv2.goodFeaturesToTrack. And the Brown-Conrady model is used to construct the distortion model:

[0095] x distorted = x (1 + k1 * r2+ k2 * r4), y distorted = y (1 + k1 * r2+ k2 * r4);

[0096] Where x, y are the coordinates of the undistorted corner point; r^2 = x^2 + y^2; x distorted y distorted The coordinates of the detected distorted corner points are given; k1 and k2 are the radial distortion parameters.

[0097] In one embodiment, determining the target distortion parameters based on the distortion model includes:

[0098] The model parameters of the distortion model are iteratively optimized based on the loss function of the distortion model until the minimum function value of the loss function is obtained.

[0099] The target distortion parameters are determined based on the model parameters of the distortion model corresponding to the minimum function value.

[0100] Specifically, to obtain the optimal distortion parameters, this embodiment requires optimizing the model parameters of the distortion model. The loss function of the distortion model reflects the difference between the output data of the distortion model and the real data. Therefore, the model parameters can be iteratively optimized using the loss function of the distortion model until the minimum function value of the loss function is reached. Based on the model parameters of the distortion model at this point, the optimal distortion parameters, i.e., the target distortion parameters, can be obtained.

[0101] For example, the loss function of the distortion model is shown below:

[0102]

[0103] Where, x i ,y i For the ideal, undistorted corner coordinates, x distorted,i and y distorted,i These are the coordinates of the detected distorted corner points. The optimal parameters k1 and k2 can be obtained by minimizing the loss function of the distortion model using a preset optimization algorithm.

[0104] In one embodiment, the preset optimization algorithm is either the LM algorithm or the gradient descent algorithm.

[0105] In one embodiment, the method further includes:

[0106] The new captured image is distorted according to the target distortion parameters, and the corner coordinates of the checkerboard calibration board in the distorted captured image are used as the corner coordinates to be verified.

[0107] Based on the difference between the coordinates of the corner point to be verified and its corresponding undistorted corner point coordinates, determine whether to continue optimizing the target distortion parameters.

[0108] Specifically, to improve the accuracy of the target distortion parameters, this embodiment also adds verification and refinement steps. After obtaining the target distortion parameters, they are used to distort a newly captured image, and then the corner coordinates in the distorted image are detected. If the difference between the detected corner coordinates and the ideal undistorted corner coordinates is greater than a preset threshold, it indicates that the two coordinates cannot be aligned, meaning the target distortion parameters are not accurate enough, and the target distortion parameters are further optimized. If the difference between the detected corner coordinates and the ideal undistorted corner coordinates is less than or equal to the preset threshold, it indicates that the two coordinates are aligned, the target distortion parameters are accurate enough, and the target distortion parameters are taken as the optimal distortion parameters.

[0109] In one embodiment, determining the target image based on perspective transformation data and the actual image includes:

[0110] Obtain the first image coordinates corresponding to each pixel in the actual image;

[0111] Based on the perspective transformation data and the first image coordinates of each pixel, determine the second image coordinates corresponding to each pixel.

[0112] The target image is determined based on each pixel and its second image coordinates.

[0113] Specifically, for each pixel in the actual image, the current image coordinates of that pixel are obtained, i.e., the first image coordinates are obtained. Then, based on the perspective transformation data and the first image coordinates of the pixel, the second image coordinates of the pixel in the coordinate system corresponding to the standard camera are calculated. The projection point of the pixel under the standard camera is determined based on the second image coordinates. The target image can be obtained by using the projection points of each pixel in the actual image under the standard camera. This embodiment, by perspective transforming the pixels in the actual image to the view plane of the standard camera, can eliminate the viewing angle error caused by the installation error of the actual camera, thus improving the data consistency of the captured images.

[0114] In one embodiment, the method further includes using the target image as input data for a preset visual model.

[0115] Specifically, the visual model can be an object detection / recognition model. Since the target image is an image after eliminating viewpoint errors, using the target image as input data for the visual model can ensure data consistency, thereby improving the stability and reliability of the visual model. Especially for situations involving camera installation deviations, using the target image after eliminating visual errors as input data can reduce the learning difficulty of the visual model and avoid adding expensive manual annotation data.

[0116] In summary, this application embodiment can project photos taken by the actual camera at a corresponding location on the same vehicle model onto the viewing plane of the standard camera, thereby reducing the viewing angle error of the actual camera caused by installation errors. Furthermore, this application embodiment also improves image quality and the subsequent viewing angle error correction effect by calibrating and optimizing the intrinsic and extrinsic parameter matrices of the standard camera, and correcting image distortion caused by the inherent properties of the actual camera. Moreover, the method of viewing angle error correction implemented by perspective transformation data in this application embodiment is simple and convenient, requires no changes to upstream or downstream modules, is easy to deploy, and does not significantly affect the system's computational load or latency.

[0117] Next, referring to the accompanying drawings, an image correction device for a vehicle camera according to an embodiment of this application is described.

[0118] like Figure 4 As shown, the image correction device 10 for a vehicle camera includes: an acquisition module 100, a determination module 200, and a correction module 300.

[0119] Specifically, the acquisition module 100 is used to acquire the standard intrinsic parameter matrix and standard extrinsic parameter matrix corresponding to the standard camera on the standard vehicle after calibration, and the actual intrinsic parameter matrix and actual extrinsic parameter matrix corresponding to the actual camera on the actual vehicle. The actual vehicle and the standard vehicle are of the same model, and the actual camera and the standard camera are installed at corresponding positions on the vehicle.

[0120] The determining module 200 is used to determine perspective transformation data based on the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix, and the actual extrinsic parameter matrix, wherein the perspective transformation data is used to reflect the transformation relationship between the image coordinates of the standard camera and the actual camera;

[0121] The correction module 300 is used to acquire the actual image captured by the actual camera, and determine the target image based on the perspective transformation data and the actual image, wherein the target image is used to reflect the shooting effect corresponding to the actual image under the standard camera.

[0122] It should be noted that the foregoing explanation of the image correction method embodiment for a vehicle camera also applies to the image correction device for a vehicle camera in this embodiment, and will not be repeated here.

[0123] Figure 5 A schematic diagram of the structure of a terminal device provided in an embodiment of this application. The terminal device may include:

[0124] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0125] When the processor 502 executes the program, it implements the image correction method for vehicle cameras provided in the above embodiments.

[0126] Furthermore, the terminal equipment also includes:

[0127] Communication interface 503 is used for communication between memory 501 and processor 502.

[0128] The memory 501 is used to store computer programs that can run on the processor 502.

[0129] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0130] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0131] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0132] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0133] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image correction method for a vehicle camera as described above.

[0134] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0135] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0136] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0137] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, followed by editing, interpreting or otherwise processing as necessary, and then stored in computer memory.

[0138] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0139] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0141] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An image correction method for a vehicle camera, characterized in that, Includes the following steps: Obtain the standard intrinsic parameter matrix and standard extrinsic parameter matrix corresponding to the standard camera on the standard vehicle after calibration, and the actual intrinsic parameter matrix and actual extrinsic parameter matrix corresponding to the actual camera on the actual vehicle, wherein the actual vehicle and the standard vehicle have the same model, and the actual camera and the standard camera are installed at corresponding positions on the vehicle; Based on the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix, and the actual extrinsic parameter matrix, perspective transformation data is determined. This perspective transformation data reflects the transformation relationship between the image coordinates of the standard camera and the actual camera, and includes: obtaining a standard offset vector based on the standard extrinsic parameter matrix; obtaining an actual offset vector based on the actual extrinsic parameter matrix; where both the standard offset vector and the actual offset vector reflect the offset between the world coordinate system and the camera coordinate system; obtaining a first product of the actual intrinsic parameter matrix and the actual offset vector, and a second product of the standard intrinsic parameter matrix and the standard offset vector; and determining the perspective transformation data based on the first product data and the second product data. Acquiring the actual image captured by the actual camera, and determining the target image based on the perspective transformation data and the actual image, wherein the target image is used to reflect the shooting effect corresponding to the actual image under the standard camera, includes: acquiring the first image coordinates corresponding to each pixel in the actual image; determining the second image coordinates corresponding to each pixel based on the perspective transformation data and the first image coordinates of each pixel; and determining the target image based on each pixel and the second image coordinates of each pixel.

2. The method as described in claim 1, characterized in that, The methods for obtaining the standard intrinsic parameter matrix and the standard extrinsic parameter matrix include: Acquire several images of the chessboard calibration board captured by the standard camera, wherein each captured image corresponds to a different shooting angle; Obtain the coordinates of several corner points of the chessboard calibration board in each of the captured images, and determine the intrinsic parameter matrix and extrinsic parameter matrix corresponding to the standard camera based on the chessboard calibration board and the coordinates of each corner point; The standard intrinsic parameter matrix and the standard extrinsic parameter matrix are obtained by iteratively optimizing the intrinsic parameter matrix and the extrinsic parameter matrix according to the objective optimization algorithm.

3. The method as described in claim 2, characterized in that, The acquisition of the actual image captured by the actual camera includes: Obtain the target distortion parameters corresponding to the standard camera; The original image captured by the actual camera is obtained, and the original image is distorted according to the target distortion parameters to obtain the actual image.

4. The method as described in claim 3, characterized in that, The acquisition of the target distortion parameters corresponding to the standard camera includes: Each corner point coordinate is used as a number of distorted corner point coordinates, and the coordinates of the undistorted corner point corresponding to each of the distorted corner point coordinates are determined according to the chessboard calibration plate. The distortion model is determined based on the coordinates of each distorted corner point and the coordinates of the undistorted corner point corresponding to each distorted corner point coordinate, wherein the distortion model is used to reflect the mapping relationship between the coordinates of the distorted corner point and the coordinates of the undistorted corner point; The target distortion parameters are determined based on the distortion model.

5. The method as described in claim 4, characterized in that, Determining the target distortion parameters based on the distortion model includes: The model parameters of the distortion model are iteratively optimized based on the loss function of the distortion model until the minimum function value of the loss function is obtained; The target distortion parameters are determined based on the model parameters of the distortion model corresponding to the minimum function value.

6. An image correction device for a vehicle camera, characterized in that, include: The acquisition module is used to acquire the standard intrinsic parameter matrix and standard extrinsic parameter matrix corresponding to the standard camera on the standard vehicle after calibration, and the actual intrinsic parameter matrix and actual extrinsic parameter matrix corresponding to the actual camera on the actual vehicle. The actual vehicle and the standard vehicle are of the same model, and the actual camera and the standard camera are installed at corresponding positions on the vehicle. A determination module is used to determine perspective transformation data based on the standard intrinsic parameter matrix, the standard extrinsic parameter matrix, the actual intrinsic parameter matrix, and the actual extrinsic parameter matrix. The perspective transformation data reflects the transformation relationship between the image coordinates of the standard camera and the actual camera. This includes: obtaining a standard offset vector based on the standard extrinsic parameter matrix; obtaining an actual offset vector based on the actual extrinsic parameter matrix; where both the standard offset vector and the actual offset vector reflect the offset between the world coordinate system and the camera coordinate system; obtaining a first product of the actual intrinsic parameter matrix and the actual offset vector, and a second product of the standard intrinsic parameter matrix and the standard offset vector; and determining the perspective transformation data based on the first product data and the second product data. A correction module is used to acquire the actual image captured by the actual camera, and determine a target image based on the perspective transformation data and the actual image. The target image is used to reflect the shooting effect corresponding to the actual image under the standard camera. The module includes: acquiring the first image coordinates corresponding to each pixel in the actual image; determining the second image coordinates corresponding to each pixel based on the perspective transformation data and the first image coordinates of each pixel; and determining the target image based on each pixel and the second image coordinates of each pixel.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an image correction program for a vehicle camera stored in the memory and executable on the processor. When the processor executes the image correction program for the vehicle camera, it implements the steps of the image correction method for a vehicle camera as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image correction program for a vehicle camera, which, when executed by a processor, implements the steps of the image correction method for a vehicle camera as described in any one of claims 1-5.

Images