On-vehicle Surround View Camera Calibration Method, Device, Electronic Equipment and Storage Medium

Abstract

The present application provides a method, apparatus, electronic device and storage medium for calibrating an in-vehicle surround-view camera. The method includes: acquiring a calibration image collected by a target camera for a preset pattern on the ground; and determining an initial internal parameter and an initial external parameter of the target camera; adjusting the current internal parameter and the current external parameter of the target camera according to the initial internal parameter and the initial external parameter to obtain an adjusted internal parameter and an adjusted external parameter; performing reprojection on the preset pattern based on the adjusted internal parameter and the adjusted external parameter to obtain a reprojection image; determining a reprojection error based on the calibration image and the reprojection image, where the reprojection error characterizes the image similarity between the preset pattern displayed in the calibration image and the preset image displayed in the reprojection image, and the reprojection error is negatively correlated with the image similarity; if it is determined that the reprojection error meets a preset condition, it is determined that the calibration of the target camera is completed. The method of the present application improves the accuracy of camera calibration and reduces the calibration difficulty.

Description

Technical Field

[0001] The present application relates to camera calibration technology, and in particular, to a calibration method, device, electronic device and storage medium for vehicle surround-view cameras. Background Art

[0002] To enable drivers to understand the vehicle's surrounding environment in real time for safer driving, the vehicle-mounted AVM surround-view system has become a standard feature in intelligent driving. The AVM surround-view system stitches and fuses the image data of the surround-view cameras in the front, rear, left, and right directions of the vehicle body to obtain a 360-degree panoramic view around the driving vehicle body, so as to facilitate the driver to understand the surrounding situation, eliminate the driver's vision blind area, and improve driving and riding safety.

[0003] Therefore, to ensure the accuracy of the panoramic view provided by the AVM surround-view system, it is usually necessary to calibrate the AVM surround-view cameras of the vehicle before leaving the factory, that is, to calibrate the external parameters of the cameras in the front, rear, left, and right directions of the vehicle body.

[0004] However, the calibration of the camera's external parameters depends on the camera's internal parameters, and camera suppliers often cannot provide the internal parameters of the cameras. When calibrating the external parameters of the cameras, it is necessary to calibrate the internal parameters of each camera separately again, which greatly increases the workload of the camera calibration work. In addition, the calibration of the camera's internal parameters has certain requirements for the operator's calibration experience and has a certain calibration difficulty. Moreover, even if the camera supplier provides the internal parameters, there are often certain deviations, which affect the calibration results of the external parameters. It can be seen that there are problems of difficult calibration and poor calibration accuracy in the current calibration of the external parameters of the surround-view cameras. Summary of the Invention

[0005] The present application provides a calibration method, device, electronic device and storage medium for vehicle surround-view cameras to solve the problems of difficult calibration of the external parameters of the current vehicle surround-view cameras and poor calibration accuracy.

[0006] In a first aspect, the present application provides a calibration method for vehicle surround-view cameras, including:

[0007] Obtain a calibration image collected by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view; and determine the initial internal parameters and initial external parameters of the target camera, where the initial internal parameters represent the initial values of the internal parameters of the target camera, and the initial external parameters represent the initial values of the external parameters of the target camera.

[0008] Adjust the current internal parameters and current external parameters of the target camera according to the initial internal parameters and the initial external parameters to obtain the adjusted internal parameters and adjusted external parameters.

[0009] Reproject the preset pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image.

[0010] Based on the calibration image and the reprojected image, determine the reprojection error, where the reprojection error characterizes the image similarity between the preset pattern shown in the calibration image and the preset image shown in the reprojected image, and the reprojection error is negatively correlated with the image similarity.

[0011] If it is determined that the reprojection error meets a preset condition, it is determined that the calibration of the target camera is completed.

[0012] In a second aspect, the present application provides an in-vehicle surround-view camera calibration device, including:

[0013] A calibration image and initial parameter determination module, configured to obtain a calibration image collected by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view, and determine the initial internal parameters and initial external parameters of the target camera, where the initial internal parameters represent the initial values of the internal parameters of the target camera, and the initial external parameters represent the initial values of the external parameters of the target camera.

[0014] A parameter adjustment module, configured to adjust the current internal parameters and current external parameters of the target camera according to the initial internal parameters and the initial external parameters to obtain adjusted internal parameters and adjusted external parameters.

[0015] A reprojection module, configured to reproject the preset pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image.

[0016] A reprojection error determination module, configured to determine the reprojection error based on the calibration image and the reprojected image, where the reprojection error characterizes the image similarity between the preset pattern shown in the calibration image and the preset image shown in the reprojected image, and the reprojection error is negatively correlated with the image similarity.

[0017] A calibration determination module, configured to determine that the calibration of the target camera is completed if it is determined that the reprojection error meets a preset condition.

[0018] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0019] Fourthly, the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the vehicle-mounted surround-view camera calibration method described in the first aspect when executed by a processor.

[0020] Fifthly, the present application provides a computer program product including a computer program, which implements the method described in the first aspect when executed by a processor.

[0021] The vehicle-mounted surround-view camera calibration method provided by the present application includes: obtaining a calibration image collected by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view; and determining the initial internal parameters and initial external parameters of the target camera, where the initial internal parameters represent the initial values of the internal parameters of the target camera, and the initial external parameters represent the initial values of the external parameters of the target camera. Then, according to the initial internal parameters and initial external parameters, adjust the current internal parameters and current external parameters of the target camera to obtain the adjusted internal parameters and adjusted external parameters; then, based on the adjusted internal parameters and adjusted external parameters, reproject the preset pattern to obtain a reprojection image; then, based on the calibration image and the reprojection image, determine the reprojection error, where the reprojection error represents the image similarity between the preset pattern shown in the calibration image and the preset image shown in the reprojection image, and the reprojection error is negatively correlated with the image similarity; finally, if it is determined that the reprojection error meets the preset condition, it is determined that the target camera is calibrated. That is to say, the calibration image can be regarded as the preset pattern collected by the target camera through the calibrated internal parameters and calibrated external parameters. Then, by simultaneously adjusting the current internal parameters and current external parameters of the target camera and using the adjusted internal parameters and external parameters of the target camera to reproject the preset pattern, the obtained projection image, if the projection image is more similar to the calibration image, it indicates that the adjusted internal parameters and external parameters of the target camera are closer to the calibrated internal parameters and calibrated external parameters. Therefore, after continuously adjusting the current internal parameters and current external parameters of the target camera until the reprojection error between the projection image and the calibration image meets the preset condition, it can be determined that the target camera is calibrated, thus ensuring the accuracy of the calibration of the internal parameters and external parameters of the camera. In addition, since the calibration of the internal parameters and external parameters of the target camera is completed simultaneously throughout the calibration process, the operation of separately calibrating the internal parameters is avoided, the problem of difficult calibration of the camera internal parameters is solved, and the calibration efficiency is effectively improved. Description of the Drawings

[0022] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0023] Figure 1 is a schematic diagram of a vehicle-mounted surround-view system camera calibration scenario shown according to an exemplary embodiment;

[0024] Figure 2 is a flowchart of a method for calibrating an in-vehicle surround-view camera shown according to an exemplary embodiment;

[0025] Figure 3 is a flowchart of a method for calibrating an in-vehicle surround-view camera shown according to another exemplary embodiment;

[0026] Figure 4A is according to Figure 3 an exemplary embodiment shows a schematic diagram of surround-view stitching effect;

[0027] Figure 4B is according to Figure 3 another exemplary embodiment shows a schematic diagram of surround-view stitching effect;

[0028] Figure 5A is Figure 4A a partial enlarged view of;

[0029] Figure 5B is Figure 4B a partial enlarged view of;

[0030] Figure 6 is a flowchart of a method for calibrating an in-vehicle surround-view camera shown according to yet another exemplary embodiment;

[0031] Figure 7 is a block diagram of a device for calibrating a camera of an in-vehicle surround-view system shown according to an exemplary embodiment;

[0032] Figure 8 is a schematic structural diagram of an electronic device shown according to an exemplary embodiment.

[0033] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Specific Embodiments

[0034] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] First, the terms involved in the present application are explained:

[0036] AVM: It refers to the Around View Monitor system, with the full English name: Around View Monitor

[0037] The specific application scenario of this application can be the camera calibration scenario of the vehicle surround view system. In the camera calibration scenario of the vehicle surround view system in the related art, there are usually the following problems when calibrating the AVM surround view camera:

[0038] The calibration of the camera extrinsic parameters depends on the camera intrinsic parameters. Often, the camera supplier cannot provide the camera intrinsic parameters. Even if the intrinsic parameters are provided, there are still certain deviations, which affect the result of the extrinsic parameter calibration, resulting in inaccurate calibration parameters of the surround view camera. Due to the inaccurate calibration parameters of the surround view camera, it will also cause distortion and misalignment problems in the 360-degree panoramic stitching view of the AVM surround view system.

[0039] In addition, if the supplier does not provide the intrinsic parameters, theoretically, each camera needs to be calibrated for the intrinsic parameters separately, which is a large workload and requires certain calibration experience for the operator, making it difficult to implement the camera calibration.

[0040] The vehicle surround view camera calibration method provided by this application aims to solve the above technical problems in the prior art.

[0041] The following uses specific embodiments to detail the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0042] Exemplarily, the specific application scenario of this application can be, for example, Figure 1 The vehicle surround view system camera calibration scenario shown in the figure. In this calibration scenario, it can include a pre-laid calibration site. The middle area of this calibration site is used to park the vehicle, and multiple preset patterns can be set on the ground around the middle area in this calibration site. Optionally, the preset pattern can be a checkerboard pattern, and the size and position of this checkerboard pattern are known. Surround view cameras are respectively arranged in the front, rear, left, and right directions of the vehicle, and the surround view cameras on the vehicle can collect the preset patterns within their camera fields of view. Optionally, the surround view camera can be a fisheye camera with a field of view of about 180 degrees. Among them, the vehicle surround view system camera calibration scenario can also include a terminal device. The above four surround view cameras can be communicatively connected to the terminal device, and the terminal device can have data processing functions. Optionally, the terminal device can be the in-vehicle terminal of the above vehicle.

[0043] Figure 2 It is a vehicle surround view camera calibration method shown according to an exemplary embodiment. For example, Figure 2As shown, the calibration method for on-vehicle surround-view cameras may include:

[0044] 110. Obtain a calibration image captured by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can capture within its camera field of view; and determine the initial internal parameters and initial external parameters of the target camera, where the initial internal parameters represent the initial values of the internal parameters of the target camera, and the initial external parameters represent the initial values of the external parameters of the target camera.

[0045] Exemplarily, the calibration method for on-vehicle surround-view cameras can be applied to the above terminal device. The target camera can be any one of the four surround-view cameras on the vehicle.

[0046] In some embodiments, the terminal device can control the target camera to capture a preset image within its camera field of view, and then the target camera uploads the captured image to the terminal device as a landmark image. Optionally, the target camera can capture multiple frames of images and upload them to the terminal device, and the terminal device can select one frame of image from the multiple frames of images as the landmark image according to preset brightness conditions, preset clarity conditions, etc.

[0047] In some embodiments, the specific implementation of determining the initial internal parameters of the target camera may include: when the camera type of the target camera is known (such as a fish-eye camera with a field of view of about 180 degrees), the terminal device can find a reference camera of the same type as the target camera and with recorded camera internal parameters in the camera product database, and then determine the camera internal parameters of the reference camera as the initial internal parameters of the target camera. Since cameras of the same type have similar internal parameters, the initial internal parameters of the target camera can be made closer to the calibration internal parameters of the target camera through the above method, so that when the target camera is adjusted based on the initial internal parameters, it can be more easily adjusted to the calibration internal parameters.

[0048] In some embodiments, since the external parameters of the camera can indicate the pose of the target in the world coordinate system determined by the calibration site, when the installation position of the target camera on the vehicle is known, the initial external parameters of the camera are calculated according to the installation position of the camera and the world coordinate system determined by the known calibration site. Since the external parameters of the camera can reflect the pose of the camera in the world coordinate system, determining the initial external parameters of the target camera through the installation position of the target camera can make the initial external parameters of the target camera closer to the calibration external parameters of the target camera, so that when the target camera is adjusted based on the initial external parameters, it can be more easily adjusted to the calibration external parameters.

[0049] Optionally, when the installation position of the camera is uncertain, calibration points can be extracted from the preset image. Then, based on the correspondence between the coordinate values (in millimeters) of the calibration points in the ground coordinate system and the image coordinates (in pixels) of the calibration points in the calibration image captured by the target camera, the initial external parameters of the target camera can be calculated.

[0050] 120. Adjust the current internal parameters and current external parameters of the target camera according to the initial internal parameters and initial external parameters to obtain the adjusted internal parameters and adjusted external parameters.

[0051] In some embodiments, the terminal device can first adjust the current internal parameters of the target camera to the initial internal parameters and the current external parameters of the target camera to the initial external parameters, and then adjust the initial internal parameters by a preset internal parameter adjustment amount to obtain the adjusted internal parameters; adjust the initial external parameters by a preset external parameter adjustment amount to obtain the adjusted external parameters. Optionally, the preset internal parameter adjustment amount and the preset external parameter adjustment amount can be determined by a preset algorithm.

[0052] 130. Reproject the preset pattern based on the adjusted internal parameters and adjusted external parameters to obtain a reprojected image.

[0053] Exemplarily, for example, multiple calibration points can be selected from the preset pattern. Since the position and size of the preset pattern on the ground are known, the ground coordinates of the multiple calibration points of the preset pattern in the ground coordinate system can be obtained. When the adjusted internal parameters and adjusted external parameters of the target camera are determined, according to the camera imaging principle, the ground coordinates of the multiple calibration points can be converted into image coordinates in the image coordinate system. Then, based on the image coordinates of the multiple calibration points, the reprojected image can be determined. Among them, the selection rule of the calibration points can be based on convenience of recognition. For example, the intersection points between line segments in the preset pattern, the corner points of the figure, etc. can be selected.

[0054] 140. Determine the reprojection error based on the calibration image and the reprojected image, where the reprojection error characterizes the image similarity between the preset pattern shown in the calibration image and the preset image shown in the reprojected image, and the reprojection error is negatively correlated with the image similarity.

[0055] Continuing with the above example, the terminal device can compare the image coordinates of the calibration points in the preset image in the calibration image with the image coordinates of the calibration points in the reprojected image to determine the image similarity between the calibration image and the reprojected image. For example, the distance between the image coordinates of calibration point A in the preset image in the calibration image and the image coordinates of calibration point A in the calibration image is very close, indicating that the higher the image similarity between the calibration image and the reprojected image, the smaller the reprojection error. By analogy, by traversing each calibration point in the preset image through the above comparison method, a more accurate reprojection error can be obtained.

[0056] 150. If it is determined that the reprojection error meets the preset condition, it is determined that the target camera has completed calibration.

[0057] In some embodiments, the terminal device may compare the reprojection error with a preset error threshold. If the reprojection error is less than or equal to the preset error threshold, it is determined that the target camera has completed calibration, that is, it is determined that the adjusted internal parameter is the calibrated internal parameter of the target camera, and the adjusted external parameter is the calibrated external parameter of the target camera.

[0058] Similarly, the other three surround-view cameras except the target camera can also complete camera calibration through the above method.

[0059] It can be seen that the vehicle-mounted surround-view camera calibration method provided in this embodiment obtains a calibration image collected by the target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view; and determines the initial internal parameter and initial external parameter of the target camera, where the initial internal parameter represents the initial value of the internal parameter of the target camera, and the initial external parameter represents the initial value of the external parameter of the target camera. Then, according to the initial internal parameter and initial external parameter, the current internal parameter and current external parameter of the target camera are adjusted to obtain the adjusted internal parameter and adjusted external parameter; then, based on the adjusted internal parameter and adjusted external parameter, the preset pattern is reprojected to obtain a reprojection image; then, based on the calibration image and the reprojection image, the reprojection error is determined, where the reprojection error represents the image similarity between the preset pattern displayed in the calibration image and the preset image displayed in the reprojection image, and the reprojection error is negatively correlated with the image similarity; finally, if it is determined that the reprojection error meets the preset condition, it is determined that the target camera has completed calibration. That is to say, the calibration image can be regarded as the target camera collecting the preset pattern through the calibrated internal parameter and calibrated external parameter. Then, by simultaneously adjusting the current internal parameter and current external parameter of the target camera, and using the adjusted internal parameter and external parameter of the target camera to reproject the preset pattern, the obtained projection image, if the projection image is more similar to the calibration image, it indicates that the adjusted internal parameter and external parameter of the target camera are closer to the calibrated internal parameter and calibrated external parameter. Therefore, after continuously adjusting the current internal parameter and current external parameter of the target camera until the reprojection error between the projection image and the calibration image meets the preset condition, it can be determined that the target camera has completed calibration, and the smaller the reprojection error, the more accurate the current calibration parameters, thus ensuring the accuracy of the calibration of the internal parameter and external parameter of the camera, and avoiding the distortion and misalignment problems in the 360-degree panoramic stitching view of the AVM surround-view system caused by inaccurate calibration parameters of the inner surround-view camera. In addition, since the calibration of the internal parameter and external parameter of the target camera is completed simultaneously during the entire calibration process, the operation of calibrating the internal parameter alone is avoided, the problem of difficult calibration of the camera internal parameter is solved, and the calibration efficiency is effectively improved.

[0060] Figure 3 Another on-vehicle surround-view camera calibration method shown according to an exemplary embodiment is as follows Figure 3 shown, and the on-vehicle surround-view camera calibration method may include:

[0061] 210. Obtain a calibration image collected by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view; and determine the initial internal parameters and initial external parameters of the target camera, where the initial internal parameters represent the initial values of the internal parameters of the target camera, and the initial external parameters represent the initial values of the external parameters of the target camera.

[0062] Among them, the specific implementation manner of step 210 can refer to step 110, so it will not be elaborated here.

[0063] 220. Adjust the current internal parameters and current external parameters of the target camera according to the initial internal parameters and initial external parameters to obtain the adjusted internal parameters and adjusted external parameters.

[0064] In some embodiments, the internal parameters of the target camera may include the camera optical center, focal length, and distortion parameters, and the external parameters of the target camera may include rotation parameters and translation parameters. The specific implementation manner of adjusting the current internal parameters and current external parameters in step 220 may include: adjusting the camera optical center and focal length in the current internal parameters; adjusting the rotation parameters and translation parameters in the current external parameters.

[0065] Considering that the distortion parameter differences of surround-view cameras of the same type are relatively small, in this embodiment, by only adjusting the parameters other than the distortion parameters in the current internal parameters of the target camera, the adjustment efficiency can be improved, and thus the camera calibration efficiency can be enhanced.

[0066] 230. Reproject the preset pattern based on the adjusted internal parameters and adjusted external parameters to obtain a reprojected image.

[0067] Among them, the specific implementation manner of step 230 can refer to step 130, so it will not be elaborated here.

[0068] 240. Determine a reprojection error based on the calibration image and the reprojected image, where the reprojection error represents the image similarity between the preset pattern shown in the calibration image and the preset image shown in the reprojected image, and the reprojection error is negatively correlated with the image similarity.

[0069] In some embodiments, the above preset pattern is a checkerboard pattern, and the specific implementation manner of step 240 may include:

[0070] 241. Obtain the first image coordinates of each corner point of the checkerboard pattern in the calibration image, and obtain the second image coordinates of each corner point of the checkerboard pattern in the reprojection image.

[0071] As an example, the preset pattern can refer to Figure 1 the checkerboard pattern shown in, which is formed by arranging rectangles of different colors at intervals, so that it can be regarded as multiple grids of different colors. Among them, the corner points of each grid are the corner points in the checkerboard pattern. Among them, the number of grids of the checkerboard, the size of each grid, and the physical distance between each checkerboard are all known. Among them, the corner points can be equivalent to the calibration points in the above embodiments.

[0072] Among them, since the calibration image is directly collected by the target camera for the checkerboard pattern, the coordinates of the pixels corresponding to each corner point are directly determined in the figure of the calibration image, and the first image coordinates of each corner point of the checkerboard pattern in the calibration image can be obtained.

[0073] Please refer to Figure 1 again. Multiple checkerboard patterns can enclose a calibration field in a roughly rectangular shape. A ground coordinate system can be established based on the position of any one corner (such as the upper left corner) of the rectangle as the origin. Since the position and size of the preset pattern in the middle of the calibration field are known, the ground coordinates of each corner point in the ground coordinate system can be calculated. When the ground coordinates of each corner point are known, based on the imaging principle of the camera and the adjusted internal parameters and adjusted external parameters of the target camera, the ground coordinates of each corner point in the ground coordinate system can be converted into image coordinates in the image coordinate system, so as to obtain the second image coordinates in the reprojection image.

[0074] 242. For each corner point of the checkerboard pattern, determine the Euclidean distance between the first image coordinates and the second image coordinates corresponding to the corner point as the error value corresponding to the corner point. The Euclidean distance is negatively correlated with the image similarity.

[0075] Exemplarily, for example, for corner point A in the checkerboard pattern, the terminal device can calculate the Euclidean distance between the first image coordinates corresponding to corner point A and the second image coordinates corresponding to corner point A, and then determine this Euclidean distance as the error value corresponding to corner point A. And so on, by traversing each corner point in the above manner, the error value corresponding to each corner point can be obtained.

[0076] 243. Determine the reprojection error based on the error value corresponding to each corner point in the checkerboard pattern.

[0077] In some embodiments, the specific implementation of step 243 may include: calculating the average value of the error values corresponding to all corner points in the checkerboard pattern, and determining the average value as the reprojection error.

[0078] As an example, the terminal device may calculate the sum of the error values corresponding to multiple corner points, and then calculate the quotient of the sum of the error values corresponding to the multiple corner points and the number of the multiple corner points, and determine the quotient as the reprojection error.

[0079] In some embodiments, the specific implementation of step 243 may include:

[0080] 2431. Determine the co-viewing points in the checkerboard pattern. The co-viewing points are the corner points that can be collected by the surround-view cameras other than the target camera on the same vehicle in the checkerboard pattern.

[0081] Exemplarily, please refer to Figure 1 , in Figure 1 , the calibration site may be divided into 9 regions, including region 1 to region 9. Among them, the vehicle is parked in region 5. Taking the surround-view camera located in front of the vehicle as the target camera as an example, assume that the preset patterns that the target camera can collect are the preset patterns of region 1, region 2, and region 3; while the surround-view camera located on the left side of the vehicle can collect the preset patterns of region 1, region 4, and region 5; the surround-view camera located on the left side of the vehicle can collect the preset patterns of region 3, region 6, and region 9. Then, it can be determined that the co-viewing points corresponding to the target camera include the corner points of the checkerboard pattern in region 1 and the corner points of the checkerboard pattern in region 3, and the corner points of the checkerboard pattern in region 2 are the non-co-viewing points corresponding to the target camera.

[0082] 2432. Assign a first weight value to the error value corresponding to the co-viewing point, and assign a second weight value to the error value corresponding to the non-co-viewing point other than the co-viewing point in the checkerboard pattern. The first weight is greater than the second weight.

[0083] 2433. Determine the reprojection error based on the error value corresponding to the co-viewing point, the first weight value, the error value corresponding to the non-co-viewing point, and the second weight value.

[0084] Exemplarily, for example, the co-viewing points include corner point 1, the non-co-viewing points include corner point 2, the error corresponding to corner point 1 is a1, the error corresponding to corner point 2 is a2, the first weight value is k1, and the second weight value is k2. Then the reprojection error at this time = (k1 * a1 + k2 * a2) / 2, where k1 > k2.

[0085] Considering that the number of common perspective points in the preset pattern that can be captured by the target camera is small, and they are mainly distributed at the edge of the camera's field of view of the target camera, while the number of non-perspective points in the preset pattern is large and mainly distributed in the center of the camera's field of view of the target camera. In this embodiment, weights are set according to the distribution of corner points to reduce the influence of uneven point distribution on the accuracy of calibration parameters and reduce the reprojection error of common perspective points.

[0086] 250. If it is determined that the reprojection error meets the preset conditions, it is determined that the calibration of the target camera is completed.

[0087] Among them, the specific implementation of step 250 can refer to step 150, so it will not be elaborated here.

[0088] 260. If it is determined that the reprojection error does not meet the preset conditions, the current internal parameters and current external parameters of the target camera are readjusted to obtain the readjusted internal parameters and readjusted external parameters.

[0089] Exemplarily, when the reprojection error is greater than the preset threshold, it is determined that the reprojection error does not meet the preset conditions.

[0090] 270. Based on the readjusted internal parameters and readjusted external parameters, return to execute the step of reprojecting the checkerboard pattern based on the adjusted internal parameters and adjusted external parameters to obtain a reprojection image until the reprojection error meets the preset conditions.

[0091] Exemplarily, a preset algorithm is pre-stored in the terminal device. For example, the preset algorithm is a non-linear numerical analysis algorithm. When it is determined that the reprojection error does not meet the preset conditions, the terminal device can use the non-linear numerical analysis algorithm for iterative operations to gradually adjust the current internal parameters and current external parameters of the target camera until the reprojection error meets the preset conditions. Optionally, the non-linear numerical analysis algorithm can be the Levenberg-Marquardt (LM) algorithm, where the parameter adjustment amount in this non-linear numerical analysis algorithm includes the internal parameters and external parameters of the target camera.

[0092] Optionally, during the iteration of the internal parameters and external parameters of the target camera using the preset algorithm, optimization constraint conditions can be set. For example, the optimization constraint conditions can be to limit the parameters to be iterated within a suitable range.

[0093] As an example, the reprojection error evaluation after calibrating the camera by the vehicle surround-view camera calibration method of this embodiment is shown in Table 1:

[0094] Table 1

[0095]

[0096] As can be seen from Table 1, the error result of only calibrating the extrinsic parameters of the camera in the related art is much larger than the error result of calibrating both the intrinsic and extrinsic parameters of the corresponding camera in this solution. Therefore, the on-vehicle surround-view camera calibration method of this embodiment can reduce the reprojection error to below sub-pixel, which can effectively improve the accuracy of camera calibration.

[0097] As another example, after calibrating the camera by the on-vehicle surround-view camera calibration method of this embodiment, the surround-view stitching effect completed by the surround-view camera is as Figure 4A shown. And after calibrating the camera by the method of only calibrating the extrinsic parameters of the camera, the surround-view stitching effect completed by the surround-view camera is as Figure 4B shown.

[0098] Comparing Figure 4A and Figure 4B it can be known that after calibrating the camera by the on-vehicle surround-view camera calibration method of this embodiment, the linearity in the surround-view stitching map completed by the surround-view camera is better and there is no obvious dislocation.

[0099] As yet another example, magnifying the upper right corner of Figure 4A can obtain a stitching effect diagram as Figure 5A shown. Magnifying the upper right corner of Figure 4B can obtain a stitching effect diagram as Figure 5B shown.

[0100] Comparing Figure 5A and Figure 5B it can be known that after calibrating the camera by the on-vehicle surround-view camera calibration method of this embodiment, the rectangular frame area in the upper right corner of the surround-view stitching map completed by the surround-view camera, that is, the bending of the wall edge, is improved.

[0101] It can be seen that adopting the on-vehicle surround-view camera calibration method of this embodiment can effectively improve the calibration accuracy of the camera, thereby improving the stitching effect.

[0102] Figure 6 is yet another on-vehicle surround-view camera calibration method shown according to an exemplary embodiment. As Figure 6 shown, the on-vehicle surround-view camera calibration method may include:

[0103] 310. Obtain a calibration image collected by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view; and determine the initial intrinsic parameters and initial extrinsic parameters of the target camera, where the initial intrinsic parameters represent the initial values of the intrinsic parameters of the target camera, and the initial extrinsic parameters represent the initial values of the extrinsic parameters of the target camera.

[0104] 320. Adjust the current intrinsic parameters and current extrinsic parameters of the target camera according to the initial intrinsic parameters and initial extrinsic parameters to obtain the adjusted intrinsic parameters and adjusted extrinsic parameters.

[0105] 330. Reproject the preset pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image.

[0106] 340. Determine a reprojection error based on the calibration image and the reprojected image, where the reprojection error characterizes the image similarity between the preset pattern shown in the calibration image and the preset image shown in the reprojected image, and the reprojection error is negatively correlated with the image similarity.

[0107] 350. If it is determined that the reprojection error meets a preset condition, it is determined that the target camera is calibrated.

[0108] Among them, the specific implementation manners of steps 310 to 350 can refer to steps 110 to 150, so they will not be elaborated here.

[0109] 360. If it is determined that the reprojection error does not meet the preset condition, readjust the current internal parameters and the current external parameters of the target camera to obtain the readjusted internal parameters and the readjusted external parameters.

[0110] 370. Based on the readjusted internal parameters and the readjusted external parameters, return to execute the step of reprojecting the checkerboard pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image, and record the number of times of adjusting the current internal parameters and the current external parameters of the target camera.

[0111] 380. If it is determined that the number of times reaches a preset number of times, stop returning to execute the step of readjusting the current internal parameters and the current external parameters of the target camera, and determine the internal parameters and external parameters obtained by the last adjustment of the current internal parameters and the current external parameters of the target camera as the calibrated internal parameters and calibrated external parameters of the target camera.

[0112] Exemplarily, for example, when it is determined that the reprojection error does not meet the preset condition, the terminal device can use the internal parameters and external parameters of the target camera as input variables of a preset algorithm, and perform iteration using the preset algorithm, where the goal of the preset algorithm is for the reprojection error to meet the preset condition. The above-mentioned number of times of adjusting the current internal parameters and the current external parameters of the target camera is equivalent to the number of iterations. Assume that the preset number of times is N. When the number of iterations reaches N, the internal parameters and external parameters of the target camera obtained by the Nth iteration can be directly determined as the calibrated internal parameters and calibrated external parameters of the target camera, where N is an integer.

[0113] Figure 7 A vehicle surround view camera calibration device shown according to an exemplary embodiment is as Figure 7 shown, and the device may include:

[0114] The calibration image and initial parameter determination module is used to obtain a calibration image captured by the target camera of a preset pattern on the ground, wherein the preset pattern is the largest pattern that the target camera can capture within its camera field of view, and determine the initial intrinsic parameters and initial extrinsic parameters of the target camera, wherein the initial intrinsic parameters represent the initial values of the intrinsic parameters of the target camera, and the initial extrinsic parameters represent the initial values of the extrinsic parameters of the target camera.

[0115] The parameter adjustment module is used to adjust the current internal parameters and the current external parameters of the target camera according to the initial internal parameters and the initial external parameters to obtain the adjusted internal parameters and the adjusted external parameters.

[0116] The reprojection module is used to reproject the preset pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image.

[0117] A reprojection error determination module is used to determine a reprojection error based on the calibration image and the reprojection image, wherein the reprojection error represents the image similarity between the preset pattern displayed in the calibration image and the preset image displayed in the reprojection image, and the reprojection error is negatively correlated with the image similarity.

[0118] The calibration determination module is used to determine whether the target camera has completed calibration if it is determined that the reprojection error meets a preset condition.

[0119] In some embodiments, the preset pattern is a checkerboard pattern, and the reprojection error determination module is specifically configured to:

[0120] Obtain the first image coordinates of each corner point of the checkerboard pattern in the calibration image, and obtain the second image coordinates of each corner point of the checkerboard pattern in the reprojected image. For each corner point in the checkerboard pattern, determine the Euclidean distance between the first image coordinate and the second image coordinate corresponding to the corner point as the error value corresponding to the corner point. The Euclidean distance is negatively correlated with the image similarity. Determine the reprojection error based on the error value corresponding to each corner point in the checkerboard pattern.

[0121] In some embodiments, the reprojection error determination module is further configured to calculate an average value of error values corresponding to all corner points in the checkerboard pattern, and determine the average value as the reprojection error.

[0122] In some embodiments, the reprojection error determination module is further used to: determine the common viewing angle points in the checkerboard pattern, the common viewing angle points being the corner points in the checkerboard pattern that can be captured by the surround view cameras other than the target camera on the same vehicle; assigning a first weight value to the error value corresponding to the common viewing angle points, and assigning a second weight value to the error value corresponding to the non-common viewing angle points in the checkerboard pattern other than the common viewing angle points, wherein the first weight is greater than the second weight value; and determine the reprojection error based on the error value corresponding to the common viewing angle points, the first weight value, the error value corresponding to the non-common viewing angle points, and the second weight value.

[0123] In some embodiments, the apparatus further includes:

[0124] An adjustment module, configured to, if it is determined that the reprojection error does not meet the preset condition, re-adjust the current internal parameters and the current external parameters of the target camera to obtain the re-adjusted internal parameters and the re-adjusted external parameters.

[0125] A return execution module, configured to, based on the re-adjusted internal parameters and the re-adjusted external parameters, return to execute the step of reprojection of the checkerboard pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojection image, until the reprojection error meets the preset condition.

[0126] In some embodiments, the apparatus further includes:

[0127] An adjustment module, configured to, if it is determined that the reprojection error does not meet the preset condition, re-adjust the current internal parameters and the current external parameters of the target camera to obtain the re-adjusted internal parameters and the re-adjusted external parameters.

[0128] A times recording module, configured to, based on the re-adjusted internal parameters and the re-adjusted external parameters, return to execute the step of reprojection of the checkerboard pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojection image, and record the number of times of adjusting the current internal parameters and the current external parameters of the target camera.

[0129] A calibration parameter determination module, configured to, if it is determined that the number of times reaches the preset number of times, stop returning to execute the step of re-adjusting the current internal parameters and the current external parameters of the target camera, and determine the internal parameters and the external parameters obtained by the last adjustment of the current internal parameters and the current external parameters of the target camera as the calibration internal parameters and the calibration external parameters of the target camera.

[0130] In some embodiments, the method further includes: a parameter adjustment module, including:

[0131] A first adjustment sub-module, configured to adjust the camera optical center and focal length in the current internal parameters.

[0132] A second adjustment sub-module, configured to adjust the rotation parameter and the translation parameter in the current external parameters.

[0133] Figure 8 It is a block diagram of an electronic device shown according to an exemplary embodiment. The electronic device may be a computer, a vehicle-mounted terminal, a server, etc. Among them, the electronic device may be equivalent to the terminal device in the above embodiment.

[0134] The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0135] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0136] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0137] The power component 806 provides power to various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0138] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0139] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0140] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0141] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0142] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a wireless network based on communication standards, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0143] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0144] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by a processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0145] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal device, enables the terminal device to execute the above-mentioned on-vehicle surround view camera calibration method of the electronic device.

[0146] A computer program product includes a computer program, and when the computer program is executed by a processor, it is the on-vehicle surround view camera calibration method in the above embodiment.

[0147] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only to be regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0148] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for calibrating an in-vehicle surround-view camera, characterized in that, include: Obtain a calibration image captured by a target camera of a preset pattern on the ground, wherein the preset pattern is the largest pattern that the target camera can capture within its camera field of view; and determine initial intrinsic parameters and initial extrinsic parameters of the target camera, wherein the initial intrinsic parameters represent initial values of the intrinsic parameters of the target camera, and the initial extrinsic parameters represent initial values of the extrinsic parameters of the target camera; Adjusting the current internal parameters and the current external parameters of the target camera according to the initial internal parameters and the initial external parameters to obtain adjusted internal parameters and adjusted external parameters; Reprojecting the preset pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image; determining a reprojection error based on the calibration image and the reprojected image, wherein the reprojection error represents an image similarity between a predetermined pattern displayed in the calibration image and a predetermined image displayed in the reprojected image, and the reprojection error is negatively correlated with the image similarity; If it is determined that the reprojection error meets a preset condition, determining that the target camera has completed calibration; The preset pattern is a checkerboard pattern, and determining the reprojection error based on the calibration image and the reprojection image includes: Obtaining first image coordinates of each corner point of the checkerboard pattern in the calibration image, and obtaining second image coordinates of each corner point of the checkerboard pattern in the reprojected image; For each corner point of the checkerboard pattern, determining the Euclidean distance between the first image coordinate and the second image coordinate corresponding to the corner point as the error value corresponding to the corner point, wherein the Euclidean distance is negatively correlated with the image similarity; Determine common viewing angles in the checkerboard pattern, where the common viewing angles are corner points in the checkerboard pattern that can be captured by surround view cameras other than the target camera on the same vehicle; assign a first weight value to an error value corresponding to the common viewing angle points, and assign a second weight value to error values corresponding to non-common viewing angle points in the checkerboard pattern other than the common viewing angle points, wherein the first weight is greater than the second weight value; and determine the reprojection error based on the error value corresponding to the common viewing angle points, the first weight value, the error value corresponding to the non-common viewing angle points, and the second weight value.

2. The method according to claim 1, wherein The method further comprises: If it is determined that the reprojection error does not meet the preset condition, re-adjusting the current intrinsic parameter and the current extrinsic parameter of the target camera to obtain a re-adjusted intrinsic parameter and a re-adjusted extrinsic parameter; Based on the readjusted internal parameters and the readjusted external parameters, return to the step of reprojecting the checkerboard pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojected image until the reprojection error meets a preset condition.

3. The method according to claim 1, wherein The method further comprises: If it is determined that the reprojection error does not meet the preset condition, re-adjusting the current intrinsic parameter and the current extrinsic parameter of the target camera to obtain a re-adjusted intrinsic parameter and a re-adjusted extrinsic parameter; Based on the re-adjusted internal parameters and the re-adjusted external parameters, return and execute the step of performing reprojection on the checkerboard pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojection image, and record the number of times of adjusting the current internal parameters and the current external parameters of the target camera. If it is determined that the number of times reaches the preset number of times, stop returning and executing the step of re-adjusting the current internal parameters and the current external parameters of the target camera, and determine the internal parameters and external parameters obtained by the last adjustment of the current internal parameters and the current external parameters of the target camera as the calibrated internal parameters and calibrated external parameters of the target camera.

4. The method according to claim 1, characterized in that, The method further includes: The adjustment of the current internal parameters and the current external parameters of the target camera includes: Adjust the camera optical center and focal length in the current internal parameters. Adjust the rotation parameters and translation parameters in the current external parameters.

5. A vehicle surround-view camera calibration device, including: A calibration image and initial parameter determination module, configured to obtain a calibration image collected by a target camera for a preset pattern on the ground, where the preset pattern is the largest pattern that the target camera can collect within its camera field of view, and determine the initial internal parameters and initial external parameters of the target camera, where the initial internal parameters represent the initial values of the internal parameters of the target camera, and the initial external parameters represent the initial values of the external parameters of the target camera; A parameter adjustment module, configured to adjust the current internal parameters and the current external parameters of the target camera according to the initial internal parameters and the initial external parameters to obtain adjusted internal parameters and adjusted external parameters; A reprojection module, configured to perform reprojection on the preset pattern based on the adjusted internal parameters and the adjusted external parameters to obtain a reprojection image; A reprojection error determination module, configured to determine a reprojection error based on the calibration image and the reprojection image, where the reprojection error represents the image similarity between the preset pattern displayed in the calibration image and the preset image displayed in the reprojection image, and the reprojection error is negatively correlated with the image similarity; A calibration determination module, configured to determine that the target camera is calibrated if it is determined that the reprojection error meets a preset condition; The reprojection error determination module is specifically configured to: Obtain the first image coordinates of each corner point of the checkerboard pattern in the calibration image, and obtain the second image coordinates of each corner point of the checkerboard pattern in the reprojection image; For each corner point of the checkerboard pattern, determine the Euclidean distance between the corresponding first image coordinates and the second image coordinates of the corner point as the error value corresponding to the corner point, and the Euclidean distance is negatively correlated with the image similarity. Determine the co-viewpoint points in the checkerboard pattern, where the co-viewpoint points are the corner points that can be captured in the checkerboard pattern by the surround cameras on the same vehicle other than the target camera; assign a first weight value to the error value corresponding to the co-viewpoint points, and assign a second weight value to the error value corresponding to the non-co-viewpoint points other than the co-viewpoint points in the checkerboard pattern, where the first weight is greater than the second weight; determine the reprojection error based on the error value corresponding to the co-viewpoint points, the first weight value, the error value corresponding to the non-co-viewpoint points, and the second weight value.

6. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by the processor, they are used to implement the vehicle surround camera calibration method according to any one of claims 1 to 4.

8. A computer program product, characterized in that, It includes a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1 to 4.

Images