A kind of calibration method and device of vehicle-mounted surround view camera, storage medium and vehicle

Abstract

The application discloses a kind of calibration method and device of vehicle-mounted surround view camera, storage medium and vehicle.The method comprises: in response to calibration trigger information, the original external parameter of the vehicle-mounted surround view camera is acquired;Determine the homonymy point on the original image corresponding to adjacent two cameras in the vehicle-mounted surround view camera respectively;Calibration processing is carried out based on the original external parameter and the homonymy point on the original image, and the external parameter matrix of the camera in the vehicle-mounted surround view camera is obtained;Based on the external parameter matrix, the original image collected in the vehicle-mounted surround view camera is spliced, so that the change of surround view fisheye camera external parameter caused by load, tire pressure change can be effectively solved, and the visual difference of splicing misplacement can be solved positively.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a calibration method and apparatus for an in-vehicle surround view camera, a storage medium, and a vehicle. Background Technology

[0002] With the continuous development of panoramic video (image) technology, in-vehicle surround view systems are becoming increasingly popular. These systems use cameras installed around the vehicle to capture images of its surroundings, which are then stitched together to create a panoramic video view of the vehicle and its surroundings.

[0003] However, the surround view system's stitched-together view relies on calibrated surround view cameras. In existing technology, the extrinsic parameters of the surround view cameras are calibrated using a fixed reference pattern at the vehicle's factory, ensuring that the images captured by the various cameras installed around the vehicle are accurately stitched together. However, during vehicle use, various changing factors inevitably cause changes in the images captured by the surround view cameras, leading to noticeable misalignments in the stitched surround view. Summary of the Invention

[0004] This application provides a calibration method and apparatus for a vehicle-mounted surround-view camera, a storage medium, and a vehicle, to address the shortcomings of existing technologies where changes in the external parameters of the surround-view fisheye camera cause variations in some vehicle parameters.

[0005] To achieve the above objectives, this application provides a calibration method for vehicle-mounted surround-view cameras, wherein the number of vehicle-mounted surround-view cameras is at least four, respectively installed in four directions of the vehicle body, and the method includes:

[0006] In response to calibration trigger information, the raw extrinsic parameters of the vehicle-mounted surround view camera are acquired;

[0007] Determine the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system;

[0008] Based on the original extrinsic parameters, corresponding points on the original image, and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera relative to the world coordinate system, calibration processing is performed to determine the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system.

[0009] Optionally, determining the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system includes:

[0010] When the vehicle's driving status and the surrounding environment meet the calibration requirements, the original image is acquired through the vehicle-mounted surround-view camera.

[0011] The original image is subjected to distortion correction processing to obtain a corresponding distortion-corrected image;

[0012] Determine the overlapping region of the distortion-free images corresponding to two adjacent cameras;

[0013] Based on the overlapping region, the corresponding points representing the same object point in adjacent distortion-free images are determined.

[0014] Optionally, before obtaining the raw extrinsic parameters of the vehicle-mounted surround-view camera, the method further includes:

[0015] If a change in vehicle load or tire pressure is detected that exceeds a threshold, the calibration trigger information is generated to recalibrate the vehicle surround view camera.

[0016] Optionally, the original extrinsic parameters include: the initial translation matrix of the vehicle-mounted surround-view camera relative to the world coordinate system, calibrated at the factory;

[0017] The method further includes:

[0018] Based on the initial translation matrix of the vehicle-mounted surround view camera relative to the world coordinate system, determine the initial translation matrix of the adjacent vehicle-mounted surround view cameras;

[0019] Based on the initial translation matrix of the adjacent vehicle-mounted surround view cameras, determine the optical center distance and corresponding scale factor of the adjacent vehicle-mounted surround view cameras.

[0020] Optionally, after determining the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system, the method further includes:

[0021] Determine the coordinate values ​​corresponding to the points with the same name;

[0022] Based on the coordinate values ​​of the corresponding points, generate the basic matrix of the two adjacent cameras;

[0023] Based on the aforementioned fundamental matrix, the extrinsic parameter matrices of two adjacent cameras are determined.

[0024] Optionally, the two adjacent cameras include a first vehicle-mounted surround-view camera and a camera to be calibrated;

[0025] The methods for determining the extrinsic parameter matrix include:

[0026] Determine the reference original extrinsic parameters of the first vehicle-mounted surround-view camera among the adjacent cameras in the world coordinate system, and the relative extrinsic parameter matrix between the first vehicle-mounted surround-view camera and the camera to be calibrated;

[0027] Based on the original reference extrinsic parameters, the relative extrinsic parameter matrix, and the scale factor, the extrinsic parameter matrix of the camera to be calibrated in the world coordinate system is determined.

[0028] Optionally, the method further includes:

[0029] Obtain the first candidate extrinsic parameter matrix of the first vehicle-mounted surround view camera based on the left adjacent camera to be calibrated and the second candidate extrinsic parameter matrix based on the right adjacent camera to be calibrated;

[0030] The Euclidean distance is determined based on the first candidate extrinsic matrix and the second candidate extrinsic matrix.

[0031] If the Euclidean distance is less than a preset threshold, the extrinsic matrix is ​​determined to be valid so that the image stitching task can be performed based on the extrinsic matrix.

[0032] Optionally, the original images acquired by the vehicle-mounted surround-view camera are stitched together based on the extrinsic parameter matrix.

[0033] This application embodiment also provides an external parameter calibration device for vehicle-mounted surround-view cameras, wherein the number of vehicle-mounted surround-view cameras is at least four, respectively arranged in four directions of the vehicle body, and the device includes:

[0034] The acquisition module is used to acquire the raw external parameters of the vehicle-mounted surround view camera in response to calibration trigger information;

[0035] The determination module is used to determine the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera;

[0036] The calibration module is used to perform calibration processing based on the original extrinsic parameters, corresponding points on the original image, and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera among the multiple vehicle-mounted surround-view cameras relative to the world coordinate system, and to determine the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system.

[0037] This application also provides a computer-readable storage medium storing a computer program executable by a processor, wherein the program, when executed by the processor, implements the calibration method for a vehicle surround-view camera as provided in this application.

[0038] This application also provides a vehicle, including: a controller, multiple electrical devices, and a power supply system, wherein the power supply system includes a battery, and the controller is used to execute the calibration method for the vehicle-mounted surround-view camera provided in this application embodiment.

[0039] The calibration method, apparatus, storage medium, and vehicle for a vehicle-mounted surround-view camera provided in this application embodiment, in response to calibration trigger information, acquire the original extrinsic parameters of the vehicle-mounted surround-view camera; determine corresponding points on the original images of two adjacent cameras in the vehicle-mounted surround-view camera; perform calibration processing based on the original extrinsic parameters and the corresponding points on the original images to obtain the extrinsic parameter matrix of the cameras in the vehicle-mounted surround-view camera; and stitch the original images acquired by the vehicle-mounted surround-view camera based on the extrinsic parameter matrix. Therefore, it can effectively solve the problem of changes in the extrinsic parameters of the surround-view fisheye camera caused by changes in load and tire pressure, and can positively resolve visual differences caused by stitching misalignment.

[0040] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 A schematic diagram of a scenario for the extrinsic parameter calibration scheme of an in-vehicle surround-view camera provided in an embodiment of this application;

[0043] Figure 2 A flowchart of one embodiment of the calibration method for a vehicle-mounted surround-view camera provided in this application;

[0044] Figure 3 A schematic diagram of the external parameter calibration device for a vehicle-mounted surround-view camera provided in this application;

[0045] Figure 4 A schematic diagram of the structure of an embodiment of the vehicle provided in this application;

[0046] Figure 5 A schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0047] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0048] The solution provided in this application can be applied to any system with an in-vehicle camera, such as electric vehicles. Figure 1 This is a schematic diagram illustrating an application scenario for the extrinsic parameter calibration scheme for the vehicle-mounted surround-view camera provided in this application embodiment. Figure 1 The scenario shown is merely one example of the scenarios in which the technical solution of this application can be applied.

[0049] With the widespread use of vehicles, more and more users are using vehicles to travel and make life more convenient. The technology of using cameras to capture images to assist driving and parking safety is also widely used in vehicles. In particular, panoramic video (image) technology is increasingly being applied to vehicles to realize in-vehicle surround view systems.

[0050] Specifically, a surround-view system uses fisheye lenses mounted around the vehicle to capture images of the surroundings, which are then stitched together to allow viewers to see the vehicle's surroundings through a panoramic video. However, because this system uses images from multiple cameras located at different positions on the vehicle (e.g., four at the front, rear, left, and right) to stitch together a complete image, the resulting surround view relies on the extrinsic parameters of the fisheye cameras to ensure the matching of boundaries between the images captured by the cameras at different locations. In existing technologies, the extrinsic parameters of the surround-view fisheye cameras are typically calibrated at the vehicle's manufacturing stage using a fixed reference pattern to ensure that the images captured by the various lenses mounted around the vehicle are accurately stitched together.

[0051] However, after a vehicle leaves the factory, its actual height often differs from the calibrated surround-view camera's height due to changes in load or tire pressure during actual use. This causes changes in the external parameters of the surround-view fisheye camera mounted on the vehicle, resulting in significant misalignment between images captured by adjacent cameras when using images from surround-view cameras at different locations, thus affecting visual perception.

[0052] For example, Figure 1 This is a schematic diagram illustrating an application scenario of the extrinsic parameter calibration scheme for a vehicle-mounted surround-view camera according to an embodiment of this application. For example... Figure 1 As shown, fisheye cameras 0, 1, 2, and 3 are respectively installed at the front, rear, left, and right sides of the vehicle body. Each camera can capture the scene in front of it to form an image. Based on the intrinsic and extrinsic parameters of the fisheye cameras determined during vehicle manufacturing calibration, as well as the distortion table, the images captured by each camera are distorted and mapped onto a planar diagram to obtain four corresponding top views I0, I1, I2, and I3. Therefore, by stitching the four top views I0-I3 according to the corresponding camera positions on the vehicle body, a model can be obtained, for example... Figure 1 The stitched ring view shown.

[0053] However, as Figure 1 As shown, in such a stitched ring view, adjacent top views need to be stitched together precisely at their respective boundaries. For example, when stitching together the front view taken by fisheye camera 0 at the front of the vehicle and the right view taken by fisheye camera 3 on the right side of the vehicle as a top view, the right boundary of I0 needs to be exactly aligned with the left boundary of I3, without overlapping or separating. This ensures that a highly visible stitched ring view is provided to the user.

[0054] However, in actual use, vehicles often cannot maintain the same condition as before they left the factory. Especially during actual use, changes in vehicle load due to carrying passengers or cargo, or fluctuations in tire pressure due to daily use, can alter the vehicle's posture. Consequently, the extrinsic parameters of the fisheye camera mounted on the vehicle also change. This results in noticeable misalignment in the surround view stitching, affecting visual perception.

[0055] Therefore, in existing technologies, the only solution to this problem of camera extrinsic parameter variation is to have the external parameters recalibrated at the manufacturer or a manufacturer-designated repair shop. However, such calibration usually only addresses the issue of camera extrinsic parameter variations caused by tire pressure changes, and cannot resolve splicing misalignments caused by load changes due to passenger or cargo carrying during vehicle use. This solution requires a specific scenario for extrinsic parameter calibration, which necessitates a specific reference frame (such as a calibration board) and a precise positional relationship between the vehicle and the calibration object. Setting up such an environment after the vehicle leaves the factory is cumbersome, error-prone, and may fail to achieve successful calibration due to ambient lighting conditions.

[0056] This application proposes a scheme for extrinsic parameter calibration of vehicle-mounted surround-view cameras. This scheme can automatically perform extrinsic parameter calibration using at least four vehicle-mounted surround-view cameras positioned in four directions of the vehicle body. For example, in... Figure 1In the scenario shown, distortion correction processing can be performed on the original images captured by the vehicle surround-view cameras 0-1 to obtain the corresponding distorted images. Then, the overlapping region of the distorted images from two adjacent vehicle surround-view cameras can be calculated. In other words, as shown above, because changes in load or tire pressure during use after the vehicle leaves the factory have caused changes in the vehicle's posture compared to when it left the factory, the extrinsic parameters of the cameras on the vehicle remain at the values ​​calibrated at the factory, inevitably leading to overlapping regions between the images they capture. Therefore, in this embodiment, the overlapping region of the distorted images captured by adjacent cameras can be calculated. Specifically, the corresponding points in the overlapping region, i.e., the coordinates of the same object point in adjacent distorted images, can be calculated. Then, the extrinsic parameter matrices of adjacent vehicle surround-view cameras can be obtained based on the corresponding points.

[0057] Typically, the extrinsic parameter matrix of an automotive camera can include rotation and translation matrices, which can be used to describe the transformation relationship between the camera coordinate system and the world coordinate system. Further, the scale factor of the translation matrix can be determined based on the optical center distance between adjacent automotive surround-view cameras. For example, the fundamental matrix of two adjacent cameras can be solved by calculating the corresponding points in the overlapping region of images captured by adjacent cameras. The rotation matrix R and translation matrix t are decomposed using the fundamental matrix and camera intrinsic parameters. However, since the t matrix is ​​not the true t matrix at this point, but lacks a scale factor, in this embodiment, the scale factor of the t matrix can be calculated using the condition that the optical center distance between the two cameras remains constant. This allows for the calibration of the extrinsic parameters of adjacent cameras. Specifically, in this embodiment, it can be assumed that the extrinsic parameters of the cameras at the front of the vehicle in the world coordinate system do not change with vehicle load and tire pressure. Therefore, the extrinsic parameter matrix of each automotive surround-view camera relative to the world coordinate system can be determined based on the extrinsic parameter matrices of adjacent automotive surround-view cameras and the extrinsic parameter matrix of the first automotive surround-view camera located at the front of the vehicle relative to the world coordinate system.

[0058] Therefore, by obtaining the distorted images of the original images captured by the vehicle-mounted surround-view cameras in four directions on the vehicle body, the overlapping areas of the distorted images corresponding to the adjacent vehicle-mounted surround-view cameras are obtained, and the corresponding points of the overlapping areas are acquired. Based on the corresponding points, the extrinsic parameter matrices of the adjacent vehicle-mounted surround-view cameras are obtained. Based on the extrinsic parameter matrices of the adjacent vehicle-mounted surround-view cameras and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera located at the front of the vehicle body relative to the world coordinate system, the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system is determined. Therefore, it can effectively solve the problem of changes in the extrinsic parameters of the surround-view fisheye cameras caused by load and tire pressure changes, and can positively solve the visual differences caused by splicing misalignment.

[0059] like Figure 2 A flowchart illustrating the calibration method for a vehicle-mounted surround-view camera provided in this application embodiment. From... Figure 2 The calibration method for vehicle-mounted surround-view cameras includes the following steps:

[0060] Step 201: In response to the calibration trigger information, obtain the original extrinsic parameters of the vehicle surround view camera.

[0061] Step 202: Determine the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system.

[0062] Step 203: Based on the original extrinsic parameters, the corresponding points on the original image, and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera relative to the world coordinate system, perform calibration processing to determine the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system.

[0063] After obtaining the extrinsic parameter matrix through the above calibration steps, the original images acquired by the vehicle-mounted surround-view camera are stitched together based on the extrinsic parameter matrix.

[0064] It should be noted that the calibration trigger information mentioned here can be automatically triggered (for example, when the detection finds that the image stitching is incorrect, or when the load or tire pressure changes, etc.) or manually triggered by the user.

[0065] The original extrinsic parameters mentioned here include the extrinsic parameter matrix of the vehicle's surround-view camera relative to the world coordinate system, the optical center distance, etc., as specified by the vehicle's factory calibration. If the vehicle load or tire pressure changes, although it will cause changes in image stitching, some parameters in the original extrinsic parameters, such as the optical center distance, will remain unchanged.

[0066] When a change in vehicle load or tire pressure is detected to exceed a threshold, the calibration trigger information is generated to recalibrate the vehicle surround view camera.

[0067] Changes in vehicle load or tire pressure can alter the vehicle's extrinsic parameter matrix, making it impossible to stitch images seamlessly using the original parameters. Therefore, sensors can monitor vehicle load and tire pressure. When changes exceed a threshold, indicating a problem with seamless stitching, calibration trigger information is automatically generated. This allows for real-time adjustment of the extrinsic parameter matrix without user intervention, significantly improving efficiency.

[0068] During image acquisition, the original images captured by adjacent cameras have a certain overlap. Within the overlap area, there are corresponding points representing the same features, that is, the coordinates of the same object point in the adjacent distortion-corrected images.

[0069] Therefore, in this embodiment, the overlapping areas of the distortion-free images corresponding to adjacent vehicle surround-view cameras are obtained by using the distortion-free images of the original images captured by the vehicle surround-view cameras located in four directions of the vehicle body. Corresponding points of the overlapping areas are obtained, and the factory extrinsic parameter matrices of adjacent vehicle surround-view cameras are obtained based on the corresponding points. Based on the factory extrinsic parameter matrices of adjacent vehicle surround-view cameras and the factory extrinsic parameter matrix of the first vehicle surround-view camera located in front of the vehicle body relative to the world coordinate system, the extrinsic parameter matrix of each vehicle surround-view camera relative to the world coordinate system is determined. Therefore, it can effectively solve the problem of changes in the extrinsic parameters of the surround-view fisheye camera caused by load and tire pressure changes, and can positively solve the visual differences caused by splicing misalignment.

[0070] In this embodiment, at least four vehicle-mounted surround-view cameras positioned in four directions on the vehicle body can be used to automatically perform camera extrinsic parameter calibration. Of course, this embodiment does not limit the number of vehicle-mounted surround-view cameras used; for example, five or more cameras can be used, as long as these cameras are arranged in four directions on the vehicle body. For example, one camera can be positioned at the front of the vehicle, two cameras at the rear, and cameras on the left and right sides of the vehicle, thus using a total of five cameras to perform the stitching of the vehicle-mounted surround-view images described in this application.

[0071] In one or more embodiments of this application, determining corresponding points on the original images of two adjacent cameras in the vehicle surround view camera includes: acquiring original images through the vehicle surround view camera when the vehicle driving state and the surrounding environment meet the calibration requirements; performing distortion correction processing on the original images to obtain corresponding distorted images; determining the overlapping region of the distorted images of two adjacent cameras; and determining the corresponding points representing the same object point in the adjacent distorted images based on the overlapping region.

[0072] These vehicle-mounted surround-view cameras, deployed on the vehicle body, acquire images from different directions as raw images. Due to limitations in field of view, fisheye cameras are typically used, resulting in image distortion. Therefore, distortion correction processing can be performed on these raw images to obtain distorted images. Furthermore, the overlapping areas of these distorted images from adjacent vehicle-mounted surround-view cameras can be calculated. Since changes in load or tire pressure during vehicle use after manufacturing alter the vehicle's posture compared to its factory settings, the extrinsic parameters of the cameras remain at their factory calibration values. This results in overlapping areas where the captured images are no longer accurately stitched together. Therefore, in this embodiment, the overlapping areas of the distorted images captured by adjacent cameras can be calculated. The corresponding points in the overlapping areas are then obtained, representing the coordinates of the same point in adjacent distorted images.

[0073] It should be noted that meeting the calibration requirements here can be understood as being able to clearly obtain images of the vehicle's surroundings when the vehicle is stationary or traveling at low speed.

[0074] In one or more embodiments of this application, after obtaining the original extrinsic parameters of the vehicle-mounted surround-view camera, the method further includes: determining the scale factors of two adjacent cameras in the vehicle-mounted surround-view camera based on the extrinsic parameter matrix and optical center distance corresponding to the original extrinsic parameters in the world coordinate system.

[0075] In this embodiment, since the camera is fixed to the vehicle body, the distance between the camera's optical center and the extrinsic parameters of the world coordinate system can be considered constant even when changes in tire pressure and load cause changes in these parameters. Therefore, the scale factor of the translation matrix can be determined based on the optical center distances of adjacent vehicle-mounted surround-view cameras.

[0076] In one or more embodiments of this application, after determining the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera, the method further includes: determining the coordinate values ​​corresponding to the corresponding points; generating the basic matrix of the two adjacent cameras based on the coordinate values ​​of the corresponding points; and determining the extrinsic parameter matrix of the two adjacent cameras based on the basic matrix.

[0077] In practical applications, the extrinsic parameter matrix of a vehicle-mounted surround-view camera can include rotation and translation matrices, which can be used to describe the transformation relationship between the camera coordinate system and the world coordinate system. Therefore, the fundamental matrix of two cameras can be calculated based on the acquired corresponding points, and then the extrinsic parameter matrix of two adjacent cameras can be calculated based on this fundamental matrix.

[0078] The extrinsic parameter matrix of each vehicle surround view camera relative to the world coordinate system can be determined based on the extrinsic parameter matrices of adjacent vehicle surround view cameras and the extrinsic parameter matrix of the first vehicle surround view camera located at the front of the vehicle relative to the world coordinate system.

[0079] For example, the corresponding points in the overlapping region are obtained, i.e., the coordinates of the same object point in adjacent distortion-corrected images. Then, the extrinsic parameter matrices of adjacent vehicle-mounted surround-view cameras can be obtained based on these corresponding points. In this embodiment, the extrinsic parameter matrix of the vehicle-mounted camera may include rotation and translation matrices, which can be used to describe the transformation relationship between the camera coordinate system and the world coordinate system. Then, based on this fundamental matrix, the extrinsic parameter matrices of two adjacent cameras are solved using, for example, the `recoverPose` function from OpenEV.

[0080] In one or more embodiments of this application, the determination of the optical center distance includes: the original extrinsic parameters include: the initial translation matrix of the vehicle-mounted surround-view camera relative to the world coordinate system, calibrated at the factory. The method further includes: determining the initial translation matrix of adjacent vehicle-mounted surround-view cameras based on the initial translation matrix of the vehicle-mounted surround-view camera relative to the world coordinate system; and determining the optical center distance and corresponding scale factor of adjacent vehicle-mounted surround-view cameras based on the initial translation matrices of adjacent vehicle-mounted surround-view cameras.

[0081] In this embodiment, the extrinsic parameter matrices of two adjacent cameras calculated from the obtained corresponding points lack a scale factor and therefore cannot be used directly. Therefore, the initial translation matrices of each vehicle-mounted surround-view camera relative to the world coordinate system can be obtained. In this embodiment, the initial translation matrix of each camera can be the translation matrix determined after camera calibration at the vehicle's factory. The initial translation matrix between two adjacent vehicle-mounted cameras can then be determined based on this initial translation matrix relative to the world coordinate system. For example, in... Figure 1 In the scenario shown, based on the extrinsic parameter matrix twi of the surround-view fisheye camera relative to the world coordinate system (i = 0, 1, 2, 3 representing the front, rear, left, and right cameras respectively), the translation matrices t03, t02, t21, and t31 between two adjacent fisheye cameras can be calculated. Here, t03 represents the translation matrix between camera 0 (located at the front of the vehicle) and camera 3 (located on the right side of the vehicle); t02 represents the translation matrix between camera 0 (located at the front of the vehicle) and camera 2 (located on the left side of the vehicle); t21 represents the translation matrix between camera 2 (located on the left side of the vehicle) and camera 1 (located at the rear of the vehicle); and t31 represents the translation matrix between camera 3 (located on the right side of the vehicle) and camera 1 (located at the rear of the vehicle). Since the cameras are fixed to the vehicle body, it can be assumed that although changes in tire pressure and load cause changes in the extrinsic parameters of the cameras relative to the world coordinate system, the distance between the optical centers of the cameras remains constant. Therefore, in this embodiment of the application, the coordinates of the optical center of the camera 0 located at the front of the vehicle in the coordinate system of the camera 2 located on the left side of the vehicle can be represented as t02, and t02 can be represented as a three-dimensional coordinate point (ttx, tty, ttz). Therefore, the distance between the optical centers of the camera 0 and the camera 2 can be obtained as L = sqrt(ttx*ttx+tty*tty+ttz*ttz).

[0082] Based on the premise that the extrinsic parameters of the front camera 0 of the vehicle body remain unchanged in the world coordinate system, the calculated distances between the optical centers of the front camera 0 and the cameras 3 on the right side of the vehicle body and 2 on the left side of the vehicle body can be regarded as the scale factors of the translation matrix between the front camera 0 and the cameras 3 on the right side of the vehicle body and 2 on the left side of the vehicle body. For example, in Figure 1In the scene shown, the scale factor between camera 0 and camera 2 is L. Therefore, the translation matrix t between camera 0 and camera 2 calculated in step S304 can be finally expressed as (L*tx2, L*ty2, L*tz2). Similarly, the translation matrix t between camera 0 and camera 3 can be finally expressed as (L*tx3, L*ty3, L*tz3).

[0083] The two adjacent cameras include a first vehicle-mounted surround view camera and a camera to be calibrated;

[0084] The method for determining the extrinsic parameter matrix includes: determining the reference original extrinsic parameters of the first vehicle-mounted surround-view camera among the adjacent cameras in the world coordinate system, and the relative extrinsic parameter matrix between the first vehicle-mounted surround-view camera and the camera to be calibrated; and determining the extrinsic parameter matrix of the camera to be calibrated in the world coordinate system based on the reference original extrinsic parameters, the relative extrinsic parameter matrix, and the scale factor.

[0085] Based on the extrinsic parameter matrices of adjacent vehicle surround-view cameras and the extrinsic parameter matrix of the first vehicle surround-view camera located at the front of the vehicle relative to the world coordinate system, determine the extrinsic parameter matrix of each vehicle surround-view camera relative to the world coordinate system.

[0086] After determining the scale factor of the translation matrix between adjacent cameras based on the above embodiments, the extrinsic parameter matrix of each vehicle surround view camera relative to the world coordinate system can be determined according to the extrinsic parameter matrices of adjacent vehicle surround view cameras and the extrinsic parameter matrix of the first vehicle surround view camera located in front of the vehicle relative to the world coordinate system.

[0087] For example, the extrinsic parameter matrix of the third vehicle-mounted surround-view camera relative to the world coordinate system can be determined based on the extrinsic parameter matrix of the first vehicle-mounted surround-view camera relative to the world coordinate system and the extrinsic parameter matrices of the first vehicle-mounted surround-view camera and the third vehicle-mounted surround-view camera. The extrinsic parameter matrix of the fourth vehicle-mounted surround-view camera relative to the world coordinate system can be determined based on the extrinsic parameter matrix of the first vehicle-mounted surround-view camera relative to the world coordinate system and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera and the fourth vehicle-mounted surround-view camera. The extrinsic parameter matrix of the second vehicle-mounted surround-view camera relative to the world coordinate system can be determined based on the extrinsic parameter matrix of the third vehicle-mounted surround-view camera relative to the world coordinate system and the extrinsic parameter matrix of the third vehicle-mounted surround-view camera and the second vehicle-mounted surround-view camera, or based on the extrinsic parameter matrix of the fourth vehicle-mounted surround-view camera relative to the world coordinate system and the extrinsic parameter matrix of the fourth vehicle-mounted surround-view camera and the second vehicle-mounted surround-view camera. For example, in... Figure 1In the scenario shown, the extrinsic parameter matrices between camera 0 at the front of the vehicle and camera 2 on the left side of the vehicle are R02 and t02. Based on the extrinsic parameter matrices Rw0 and tw0 between camera 0 and the world coordinate system, and according to the formulas: Tw0*T2w=T20 and Tij*Tji=1, the extrinsic parameter matrix Tw2 of camera 2 relative to the world coordinate system can be calculated given Tw0 and T02. Similarly, according to the formulas: Tw0*T3w=T30 and Tij*Tji=1, the extrinsic parameter matrix Tw3 of camera 3 on the right side of the vehicle relative to the world coordinate system can be calculated given Tw0 and T03.

[0088] In one or more embodiments of this application, the first vehicle-mounted surround view camera obtains a first candidate extrinsic parameter matrix based on the left adjacent camera to be calibrated and a second candidate extrinsic parameter matrix based on the right adjacent camera to be calibrated; based on the first candidate extrinsic parameter matrix and the second candidate extrinsic parameter matrix, the Euclidean distance is determined; if the Euclidean distance is less than a preset threshold, the extrinsic parameter matrix is ​​determined to be valid so that an image stitching task can be performed based on the extrinsic parameter matrix.

[0089] To facilitate understanding, a specific example is given below. Assume the first vehicle-mounted surround-view camera is a front-facing camera, and the second and third vehicle-mounted surround-view cameras are two cameras adjacent to the first. The fourth vehicle-mounted surround-view camera is a rear-facing camera, and is adjacent to both the second and third cameras. Based on the extrinsic parameter matrix of the third vehicle-mounted surround-view camera relative to the world coordinate system, and the extrinsic parameter matrices of the third and second cameras, a first candidate extrinsic parameter matrix for the second vehicle-mounted surround-view camera relative to the world coordinate system is determined. Based on the extrinsic parameter matrix of the fourth vehicle-mounted surround-view camera relative to the world coordinate system, and the extrinsic parameter matrix of the fourth and second cameras, a second candidate extrinsic parameter matrix for the second vehicle-mounted surround-view camera relative to the world coordinate system is determined. The Euclidean distance between the six degrees of freedom parameters of the first and second candidate extrinsic parameter matrices is calculated, and the extrinsic parameter calibration is considered valid when the Euclidean distance is less than a preset threshold.

[0090] In this embodiment, after obtaining the extrinsic parameter matrices of cameras 2 and 3 on the left and right sides of the vehicle relative to the world coordinate system using, for example, camera 0 located at the front of the vehicle, the extrinsic parameter matrix Tw11 of camera 1 located at the rear of the vehicle relative to the world coordinate system can be further calculated using, for example, the extrinsic parameter matrix Tw2 of camera 2 on the left side of the vehicle relative to the world coordinate system. For example, according to the relation: Tw1*T2w=T21, Tij*Tji=1, Tw11 can be calculated when T21 and Tw2 are already known. Similarly, the extrinsic parameter matrix Tw3 of camera 3 on the right side of the vehicle relative to the world coordinate system can be obtained to further calculate the extrinsic parameter matrix Tw12 of camera 1 located at the rear of the vehicle relative to the world coordinate system. For example, according to the relation: Tw1*T3w=T31, Tij*Tji=1, Tw12 can be calculated when T31 and Tw3 are already known. Therefore, the calibrated extrinsic parameter matrix of camera 1 can be obtained from the calibrated extrinsic parameter matrices of two cameras at different adjacent positions. The accuracy of this extrinsic parameter matrix calculation can then be determined by comparing the two calculated results. For example, extrinsic parameter matrices TW11 and TW12 can be replaced with 6-DOF parameters: three Euler angles and three translations. The Euclidean distance between these six parameters is calculated. If this distance is less than a threshold, the confidence level is considered high, the calibration is valid, and either TW11 or TW12 can be used as the calibrated extrinsic parameter matrix of camera 1. The four camera extrinsic parameters are then updated based on the obtained extrinsic parameters; otherwise, recalibration is performed.

[0091] Furthermore, in this embodiment of the application, when it is determined that the calibration is valid, the calibrated extrinsic parameter matrices of cameras 2 and 3 on the left and right sides, as well as the obtained extrinsic parameter matrix of camera 1, can be calculated to stitch together the four images and videos captured by cameras 0-3 according to the new extrinsic parameters to eliminate the misalignment phenomenon.

[0092] Therefore, in this embodiment, the overlapping areas of the distortion-free images corresponding to adjacent vehicle surround-view cameras are obtained by using the distortion-free images of the original images captured by the vehicle surround-view cameras located in four directions of the vehicle body. Corresponding points of the overlapping areas are obtained, and the extrinsic parameter matrices of adjacent vehicle surround-view cameras are obtained based on the corresponding points. Based on the extrinsic parameter matrices of adjacent vehicle surround-view cameras and the extrinsic parameter matrix of the first vehicle surround-view camera located in front of the vehicle body relative to the world coordinate system, the extrinsic parameter matrix of each vehicle surround-view camera relative to the world coordinate system is determined. Therefore, it can effectively solve the problem of changes in the extrinsic parameters of the surround-view fisheye cameras caused by load and tire pressure changes, and can positively solve the visual differences caused by splicing misalignment.

[0093] Figure 3This is a schematic diagram of a structural embodiment of the extrinsic parameter calibration device for a vehicle-mounted surround-view camera provided in this application, which can be used to perform, for example... Figure 2 The method steps are shown. (As shown) Figure 3 As shown, the extrinsic parameter calibration device for the vehicle-mounted surround-view camera may include:

[0094] The acquisition module 31 is used to acquire the original external parameters of the vehicle-mounted surround view camera in response to the calibration trigger information.

[0095] The determination module 32 is used to determine the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera.

[0096] The calibration module 33 is used to perform calibration processing based on the original extrinsic parameters, corresponding points on the original image, and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera among the multiple vehicle-mounted surround-view cameras relative to the world coordinate system, and to determine the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system.

[0097] Optionally, the determination module 32 is used to acquire the original image through the vehicle surround view camera when the vehicle driving status and the surrounding environment of the vehicle meet the calibration requirements;

[0098] The original image is subjected to distortion correction processing to obtain a corresponding distortion-corrected image;

[0099] Determine the overlapping region of the distortion-free images corresponding to two adjacent cameras;

[0100] Based on the overlapping region, the corresponding points representing the same object point in adjacent distortion-free images are determined.

[0101] Optionally, it also includes a generation module 34, which generates the calibration trigger information to recalibrate the vehicle surround view camera if a change in vehicle load or tire pressure is detected to exceed a threshold.

[0102] Optionally, the original extrinsic parameters include: the initial translation matrix of the vehicle-mounted surround-view camera relative to the world coordinate system, calibrated at the factory.

[0103] The determining module 32 is further configured to determine the initial translation matrix of adjacent vehicle-mounted surround-view cameras based on the initial translation matrix of the vehicle-mounted surround-view camera relative to the world coordinate system; and to determine the optical center distance and corresponding scale factor of adjacent vehicle-mounted surround-view cameras based on the initial translation matrix of adjacent vehicle-mounted surround-view cameras.

[0104] Optionally, the determining module 32 is further configured to determine the coordinate values ​​corresponding to the corresponding points;

[0105] Based on the coordinate values ​​of the corresponding points, generate the basic matrix of the two adjacent cameras;

[0106] Based on the aforementioned fundamental matrix, the extrinsic parameter matrices of two adjacent cameras are determined.

[0107] Optionally, the two adjacent cameras include a first vehicle-mounted surround-view camera and a camera to be calibrated;

[0108] The determining module 32 is further configured to determine the reference original extrinsic parameters of the first vehicle-mounted surround-view camera among the adjacent cameras in the world coordinate system, and the relative extrinsic parameter matrix between the first vehicle-mounted surround-view camera and the camera to be calibrated; and to determine the extrinsic parameter matrix of the camera to be calibrated in the world coordinate system based on the reference original extrinsic parameters, the relative extrinsic parameter matrix and the scale factor.

[0109] Optionally, the acquisition module 31 is used to acquire the first candidate extrinsic parameter matrix of the first vehicle-mounted surround view camera based on the left adjacent camera to be calibrated and the second candidate extrinsic parameter matrix based on the right adjacent camera to be calibrated;

[0110] The Euclidean distance is determined based on the first candidate extrinsic matrix and the second candidate extrinsic matrix.

[0111] If the Euclidean distance is less than a preset threshold, the extrinsic matrix is ​​determined to be valid so that the image stitching task can be performed based on the extrinsic matrix.

[0112] Optionally, it also includes a stitching module 35, used to stitch together the original images acquired by the vehicle-mounted surround-view camera based on the extrinsic parameter matrix.

[0113] Therefore, in this embodiment, the overlapping areas of the distortion-free images corresponding to adjacent vehicle surround-view cameras are obtained by using the distortion-free images of the original images captured by the vehicle surround-view cameras located in four directions of the vehicle body. Corresponding points of the overlapping areas are obtained, and the extrinsic parameter matrices of adjacent vehicle surround-view cameras are obtained based on the corresponding points. Based on the extrinsic parameter matrices of adjacent vehicle surround-view cameras and the extrinsic parameter matrix of the first vehicle surround-view camera located in front of the vehicle body relative to the world coordinate system, the extrinsic parameter matrix of each vehicle surround-view camera relative to the world coordinate system is determined. Therefore, it can effectively solve the problem of changes in the extrinsic parameters of the surround-view fisheye cameras caused by load and tire pressure changes, and can positively solve the visual differences caused by splicing misalignment.

[0114] In one possible design, the above Figure 3 The structure of the extrinsic parameter calibration device for the vehicle-mounted surround-view camera shown can be implemented in a vehicle, such as... Figure 4 As shown, the vehicle may include: a controller, at least four cameras arranged in different directions on the vehicle body, multiple devices, and a power supply system. The power supply system includes a battery; and the controller can at least implement the calibration method for the vehicle-mounted surround-view cameras provided in the foregoing embodiments.

[0115] The above describes the internal functions and structure of the extrinsic parameter calibration device for a vehicle-mounted surround-view camera, which can be implemented as an electronic device. Figure 5 A schematic diagram illustrating the structure of an embodiment of the electronic device provided in this application. (See attached diagram.) Figure 5 As shown, the electronic device includes a memory 51 and a processor 52.

[0116] Memory 51 is used to store programs. In addition to the programs described above, memory 51 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0117] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0118] Processor 52 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural network processor (NPU), or an artificial intelligence (AI) chip. Processor 52 is coupled to memory 51 and executes the program stored in memory 51. When the program runs, it executes the calibration method of the vehicle surround view camera in the above embodiment.

[0119] Furthermore, such as Figure 5 As shown, the electronic device may also include other components such as a communication component 53, a power supply component 54, an audio component 55, and a display 55. Figure 5 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 5 The components shown.

[0120] Communication component 53 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 53 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 53 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0121] Power supply component 54 provides power to various components of the electronic device. Power supply component 54 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0122] Audio component 55 is configured to output and / or input audio signals. For example, audio component 55 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 51 or transmitted via communication component 53. In some embodiments, audio component 55 also includes a speaker for outputting audio signals.

[0123] Display 56 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0124] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for a vehicle-mounted surround-view camera, characterized in that, The number of vehicle-mounted surround-view cameras is at least four, respectively installed in four directions of the vehicle body, and the method includes: In response to calibration trigger information, the raw extrinsic parameters of the vehicle-mounted surround view camera are acquired; Determine the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system; Based on the original extrinsic parameters, the corresponding points on the original image, and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera among the multiple vehicle-mounted surround-view cameras relative to the world coordinate system, calibration processing is performed to determine the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system. The original external parameters include: the initial translation matrix of the vehicle-mounted surround view camera relative to the world coordinate system, which was calibrated at the factory. The calibration process includes: Based on the initial translation matrix, the optical center distance between two adjacent vehicle-mounted surround-view cameras is determined, and the scale factor of the translation matrix is ​​determined based on the optical center distance. Based on the corresponding points on the original image, calculate the fundamental matrix between two adjacent cameras, and determine the relative extrinsic matrix between two adjacent cameras according to the fundamental matrix and the camera intrinsic parameters. Based on the original extrinsic parameters of the first vehicle-mounted surround-view camera relative to the world coordinate system, the relative extrinsic parameter matrix, and the scale factor, the extrinsic parameter matrix of the camera to be calibrated relative to the world coordinate system is determined. The first vehicle-mounted surround view camera is a camera located at the front of the vehicle body, and its extrinsic parameter matrix relative to the world coordinate system is considered to remain unchanged when the vehicle load or tire pressure changes.

2. The method according to claim 1, characterized in that, The step of determining corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system includes: When the vehicle's driving status and the surrounding environment meet the calibration requirements, the original image is acquired through the vehicle-mounted surround-view camera. The original image is subjected to distortion correction processing to obtain a corresponding distortion-corrected image; the overlapping region of the distortion-corrected images corresponding to two adjacent cameras is determined. Based on the overlapping region, the corresponding points representing the same object point in adjacent distortion-free images are determined.

3. The method according to claim 1 or 2, characterized in that, The original extrinsic parameters include: the initial translation matrix of the vehicle-mounted surround-view camera relative to the world coordinate system, calibrated at the factory; The method further includes: Based on the initial translation matrix of the vehicle-mounted surround view camera relative to the world coordinate system, determine the initial translation matrix of the adjacent vehicle-mounted surround view cameras; Based on the initial translation matrix of the adjacent vehicle-mounted surround view cameras, determine the optical center distance and corresponding scale factor of the adjacent vehicle-mounted surround view cameras.

4. The method according to claim 3, characterized in that, After determining the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera system, the method further includes: Determine the coordinate values ​​corresponding to the points with the same name; Based on the coordinates of the corresponding points, the fundamental matrix of the two adjacent cameras is generated; based on the fundamental matrix, the extrinsic matrix of the two adjacent cameras is determined.

5. The method according to claim 3, characterized in that, The two adjacent cameras include a first vehicle-mounted surround view camera and a camera to be calibrated; The methods for determining the extrinsic parameter matrix include: Determine the reference original extrinsic parameters of the first vehicle-mounted surround-view camera among the adjacent cameras in the world coordinate system, and the relative extrinsic parameter matrix between the first vehicle-mounted surround-view camera and the camera to be calibrated; Based on the original reference extrinsic parameters, the relative extrinsic parameter matrix, and the scale factor, the extrinsic parameter matrix of the camera to be calibrated in the world coordinate system is determined.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the first candidate extrinsic parameter matrix of the first vehicle-mounted surround view camera based on the left adjacent camera to be calibrated and the second candidate extrinsic parameter matrix based on the right adjacent camera to be calibrated; The Euclidean distance is determined based on the first candidate extrinsic matrix and the second candidate extrinsic matrix. If the Euclidean distance is less than a preset threshold, the extrinsic matrix is ​​determined to be valid so that the image stitching task can be performed based on the extrinsic matrix.

7. The method according to claim 1, characterized in that, Also includes: The original images captured by the vehicle-mounted surround-view camera are stitched together based on the extrinsic parameter matrix.

8. An external parameter calibration device for a vehicle-mounted surround-view camera, characterized in that, The number of vehicle-mounted surround-view cameras is at least four, respectively installed in four directions of the vehicle body, and the device includes: The acquisition module is used to acquire the raw external parameters of the vehicle-mounted surround view camera in response to calibration trigger information; The determination module is used to determine the corresponding points on the original images of two adjacent cameras in the vehicle surround view camera; The calibration module is used to perform calibration processing based on the original extrinsic parameters, corresponding points on the original image, and the extrinsic parameter matrix of the first vehicle-mounted surround-view camera among the multiple vehicle-mounted surround-view cameras relative to the world coordinate system, and to determine the extrinsic parameter matrix of each vehicle-mounted surround-view camera relative to the world coordinate system. The original external parameters include: the initial translation matrix of the vehicle-mounted surround view camera relative to the world coordinate system, which was calibrated at the factory. The calibration process includes: Based on the initial translation matrix, the optical center distance between two adjacent vehicle-mounted surround-view cameras is determined, and the scale factor of the translation matrix is ​​determined based on the optical center distance. Based on the corresponding points on the original image, calculate the fundamental matrix between two adjacent cameras, and determine the relative extrinsic matrix between two adjacent cameras according to the fundamental matrix and the camera intrinsic parameters. Based on the original extrinsic parameters of the first vehicle-mounted surround-view camera relative to the world coordinate system, the relative extrinsic parameter matrix, and the scale factor, the extrinsic parameter matrix of the camera to be calibrated relative to the world coordinate system is determined. The first vehicle-mounted surround view camera is a camera located at the front of the vehicle body, and its extrinsic parameter matrix relative to the world coordinate system is considered to remain unchanged when the vehicle load or tire pressure changes.

9. A computer-readable storage medium having a computer program stored thereon that can be executed by a processor, wherein, When executed by the processor, the program implements the calibration method for the vehicle surround view camera as described in any one of claims 1 to 7.

10. A vehicle, characterized in that, include: The system includes a controller, multiple electrical devices, and a power supply system, the power supply system including a battery, wherein the controller is configured to perform a calibration method for a vehicle-mounted surround-view camera as described in any one of claims 1 to 7.

Images