Camera external parameter calibration method, device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a camera external parameter calibration method, device and equipment, a medium and a program product. Comprising the following steps: acquiring an image frame set obtained by performing image acquisition on a camera calibration site by each to-be-calibrated camera; extracting calibration angular point information corresponding to each to-be-calibrated camera from each image frame set, and determining a first camera external parameter of each to-be-calibrated camera according to each calibration angular point information and the camera internal parameter of each to-be-calibrated camera; determining a second camera extrinsic parameter of the first to-be-calibrated camera according to the calibration angular point information corresponding to the first to-be-calibrated camera; and performing joint calibration on each second to-be-calibrated camera according to the calibration corner point information corresponding to the first to-be-calibrated camera, the first camera external parameter, the second camera external parameter and the calibration corner point information corresponding to each second to-be-calibrated camera, and determining the second camera external parameter of each second to-be-calibrated camera. The manual participation degree in the calibration process is reduced, the error introduction possibility is reduced, and the overall calibration precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of driving assistance technology, and in particular to a camera extrinsic parameter calibration method, device, equipment, medium and program product. Background Art

[0002] In the Driving Assistance System (DAS), visual perception equipment plays an important role. Currently, visual perception systems mostly use multiple cameras arranged on the vehicle body to collect images and videos around the vehicle, and perceive surrounding vehicles, pedestrians, traffic signs and other information by processing the collected images and videos. In order to accurately perceive the traffic environment around the vehicle, the calibration of the on-board camera becomes particularly important.

[0003] The current external parameter calibration methods for vehicle-mounted cameras include calibration methods based on fixed calibration sites and calibration methods based on road information during driving. However, the calibration method based on fixed calibration sites often uses calibration blocks fixed around the vehicle body for calibration, which requires manual measurement of the position relationship between the calibration block and the vehicle body coordinate system, which is time-consuming and prone to manual measurement errors. The calibration method based on road information such as lane lines and zebra crossings obtained during driving is highly dependent on the calibration scene, and its calibration success rate is easily affected by the degree of lane line wear.

[0004] There are also some methods that calibrate the extrinsic parameters of vehicle-mounted cameras by randomly arranging calibration blocks around the vehicle. However, such methods either cannot truly realize the calibration of camera extrinsic parameters and can only realize the calibration of the relative posture relationship between different vehicle-mounted cameras, or they require map construction to assist in calibration. The calibration method is complicated and cumbersome, introduces too many errors, and leads to insufficient calibration accuracy. Summary of the invention

[0005] The present invention provides a camera extrinsic parameter calibration method, device, equipment, medium and program product, which automatically calibrate each camera to be calibrated in a vehicle and the camera extrinsic parameters required for calibration of each camera to be calibrated according to multiple frames of images collected by each camera to be calibrated, thereby reducing the degree of manual participation in the calibration process and reducing the influence of other parameters on the parameters to be calibrated. At the same time, the optimized calibrated extrinsic parameters can be applied to the joint calibration of subsequent extrinsic parameters, thereby improving the overall calibration accuracy and robustness.

[0006] In a first aspect, an embodiment of the present invention provides a camera extrinsic calibration method, comprising:

[0007] Obtain a set of image frames obtained by each camera to be calibrated through image acquisition of the camera calibration site;

[0008] Extracting calibration corner point information corresponding to each camera to be calibrated from each image frame set, and determining the first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated;

[0009] A camera to be calibrated that simultaneously collects fixed calibration blocks and semi-fixed calibration blocks in a camera calibration site is determined as a first camera to be calibrated, and a second camera extrinsic parameter of the first camera to be calibrated is determined according to calibration corner point information corresponding to the first camera to be calibrated;

[0010] Each camera to be calibrated except the first camera to be calibrated is determined as a second camera to be calibrated, and each second camera to be calibrated is jointly calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, and the second camera extrinsic parameter of each second camera to be calibrated is determined.

[0011] In a second aspect, an embodiment of the present invention further provides a camera extrinsic calibration device, comprising:

[0012] An image frame set acquisition module is used to acquire an image frame set obtained by each camera to be calibrated through image acquisition of a camera calibration site;

[0013] A first calibration module is used to extract calibration corner point information corresponding to each camera to be calibrated from each image frame set, and determine a first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated;

[0014] A second calibration module is used to determine the camera to be calibrated that simultaneously collects the fixed calibration block and the semi-fixed calibration block in the camera calibration site as the first camera to be calibrated, and determine the second camera extrinsic parameters of the first camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated;

[0015] The joint calibration module is used to determine each camera to be calibrated except the first camera to be calibrated as a second camera to be calibrated, and jointly calibrate each second camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, so as to determine the second camera extrinsic parameter of each second camera to be calibrated.

[0016] In a third aspect, an embodiment of the present invention further provides a camera extrinsic calibration device, the camera extrinsic calibration device comprising:

[0017] at least one processor; and a memory communicatively coupled to the at least one processor;

[0018] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the camera extrinsic parameter calibration method of any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, enables the computer processor to execute the camera extrinsic parameter calibration method of any embodiment of the present invention.

[0020] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which, when executed by a processor, implements the camera extrinsic parameter calibration method of any embodiment of the present invention.

[0021] The embodiments of the present invention provide a camera extrinsic calibration method, device, equipment, medium and program product, which obtains a set of image frames obtained by each camera to be calibrated performing image acquisition on a camera calibration site; extracts calibration corner point information corresponding to each camera to be calibrated from each set of image frames, and determines the first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated; determines the camera to be calibrated that simultaneously acquires a fixed calibration block and a semi-fixed calibration block in the camera calibration site as the first camera to be calibrated, and determines the second camera extrinsic parameter of the first camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated; determines each camera to be calibrated except the first camera to be calibrated as the second camera to be calibrated, and jointly calibrates each second camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, and determines the second camera extrinsic parameter of each second camera to be calibrated. By adopting the above technical solution, when it is necessary to calibrate the on-board camera, all cameras that need to be calibrated can be used as cameras to be calibrated, and each camera to be calibrated can collect multiple frames of images in the camera calibration site, and then realize the camera extrinsic parameter calibration based on the image frame set collected by different cameras to be calibrated. In the calibration process, each camera to be calibrated will be calibrated step by step and then jointly according to the calibration blocks that can be collected by each camera to be calibrated in the camera calibration site and the different types of camera extrinsic parameters required for calibration. In the step-by-step calibration, the camera extrinsic parameters of each part can be calibrated in a targeted manner, so as to avoid the influence of other parameters on the parameters currently being calibrated, and improve the calibration accuracy; and in the joint calibration, the camera extrinsic parameters that have been calibrated are introduced into the joint calibration process, and the calibration accuracy of the remaining camera extrinsic parameters is optimized. And because the multiple image frames obtained by repeated shooting of each camera to be calibrated are used for comprehensive processing in the process of calibration of each external parameter, the robustness of the calibration process is improved. The entire calibration process can be automatically performed based on the image frame sets captured by the cameras to be calibrated and the pre-arranged camera calibration site. Human participation is only required for the determination of fixed calibration blocks and semi-fixed calibration blocks in the camera calibration site, which reduces the degree of human participation in the calibration process, reduces the possibility of error introduction, and improves the overall calibration accuracy.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of a flow chart of a camera extrinsic calibration method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is an example diagram of a camera calibration site provided in the first embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a flow chart of a camera extrinsic calibration method provided in Embodiment 2 of the present invention;

[0027] Figure 4 An example of a bird's-eye view of a forward-looking camera provided in Embodiment 2 of the present invention;

[0028] Figure 5 Another example of a bird's-eye view of a front-view camera provided in the second embodiment of the present invention;

[0029] Figure 6 An example bird's-eye view diagram of line segment matching in a camera overlap area provided in the second embodiment of the present invention;

[0030] Figure 7 An example bird's-eye view diagram of corner point matching in a camera overlap area provided in the second embodiment of the present invention;

[0031] Figure 8 A schematic diagram of the structure of a camera extrinsic calibration device provided in Embodiment 3 of the present invention;

[0032] Fig. 9 A schematic diagram of the structure of a camera extrinsic calibration device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1 A flowchart of a camera extrinsic parameter calibration method provided in the first embodiment of the present invention is provided. The embodiment of the present invention can be applied to the situation where the extrinsic parameters of the vehicle-mounted camera adapted to the driving assistance system on the vehicle are calibrated when the vehicle is off the production line. The method can be executed by a camera extrinsic parameter calibration device, which can be implemented by software and / or hardware, and the camera extrinsic parameter calibration device can be configured in a camera extrinsic parameter calibration device. Optionally, the camera extrinsic parameter calibration device can be an electronic device, which can be a notebook, a desktop computer, a smart tablet, a vehicle, etc., and the embodiment of the present invention does not limit this.

[0037] like Figure 1 As shown, a camera extrinsic calibration method provided by an embodiment of the present invention specifically includes the following steps:

[0038] S101, obtaining a set of image frames obtained by each camera to be calibrated through image acquisition of a camera calibration site.

[0039] In this embodiment, the camera to be calibrated can be specifically understood as a device that is set around the vehicle and used as a visual perception device in the driving assistance system to collect information such as images and videos around the vehicle. The camera calibration site can be specifically understood as a specific area used to calibrate the on-board camera installed on the vehicle, which may include multiple calibration blocks that are used to calibrate the on-board camera and have certain specific geometric features that can be used as calibration references. The image frame set can be specifically understood as a set of image frames obtained by the camera to be calibrated by performing multiple image acquisitions within the range in which it can acquire images within the camera calibration site.

[0040] It is understandable that the camera calibration site can be adaptively set according to the distribution of the on-board cameras on the vehicle to be calibrated.

[0041] Optionally, the camera calibration site includes a fixed calibration block, a semi-fixed calibration block, and at least two random calibration blocks; wherein the fixed calibration block and the semi-fixed calibration block are calibration blocks arranged in parallel with any side of the vehicle in the camera calibration site; wherein the relative position of the fixed calibration block and the vehicle is fixed and known; wherein the random calibration block is a calibration block placed in an arbitrary posture; wherein each calibration block in the camera calibration site can be completely captured by the adjacent camera to be calibrated on the vehicle.

[0042] In this embodiment, the fixed calibration block can be specifically understood as a calibration block set in the camera calibration site, whose relative position with the vehicle is fixed and known, and the specific geometric shape contained therein has a parallel relationship with any side of the vehicle. The semi-fixed calibration block can be specifically understood as a calibration block set in the camera calibration site, whose relative position with the vehicle is not fixed and unknown, but the specific geometric shape contained therein corresponds to the fixed calibration block, and the vehicle sides with a parallel relationship also have a parallel relationship. The random calibration block can be specifically understood as a calibration block set in the camera calibration site, whose relative position with the vehicle is not fixed and unknown, and which only needs to ensure that it can be fully captured by two adjacent cameras to be calibrated on the vehicle.

[0043] For example, Figure 2 This is an example diagram of a camera calibration site provided in the first embodiment of the present invention, such as Figure 2 As shown, in the embodiment of the present invention, a vehicle is provided with four surround view cameras, each camera is respectively arranged on four sides of the vehicle body, and four calibration blocks are arranged around the vehicle body, each calibration block is a square with four black circles, and the thickness of each calibration block can be ignored. Figure 2 The calibration block shown in the upper left corner can be understood as a fixed calibration block that is parallel to the front side of the vehicle and whose relative position to the vehicle body coordinate system is fixed and known; Figure 2 The calibration block shown in the upper right corner can be understood as a semi-fixed calibration block that is parallel to the front side of the vehicle but has a random position; Figure 2 The calibration blocks shown in the lower left and lower right corners can be understood as randomly placed calibration blocks. Figure 2 Each calibration block in can be completely captured by the adjacent cameras in the vehicle, such as Figure 2 The fixed calibration block shown must be guaranteed to be fully captured by the front-view camera and the left-view camera in the vehicle.

[0044] Specifically, when it is necessary to perform external parameter calibration on each camera to be calibrated in the vehicle, the vehicle must first be controlled to enter the corresponding camera calibration site. The camera calibration site must be pre-arranged according to the distribution of the cameras to be calibrated in the vehicle to be calibrated before the camera external parameter calibration is performed. After the vehicle enters the camera calibration site corresponding to the calibrable position, all the on-board cameras that need to be calibrated on the vehicle are used as cameras to be calibrated, and each camera to be calibrated collects multiple frames of images within its capture range to obtain a set of image frames collected by each camera to be calibrated. The method of collecting images is: parallel movement of the semi-fixed calibration block, and random movement of the lower left and lower right corners of the random calibration blocks, repeated multiple times, to complete the collection of multiple frames of images.

[0045] S102: extracting calibration corner point information corresponding to each camera to be calibrated from each image frame set, and determining a first camera extrinsic parameter of each camera to be calibrated according to each calibration corner point information and a camera intrinsic parameter of each camera to be calibrated.

[0046] In this embodiment, the calibration corner point can be specifically understood as a specific point on the calibration block with certain geometric features. The calibration corner point information can be specifically understood as comprehensive information used to indicate the position of the calibration corner point in the image, the corresponding relationship between the calibration corner point and the calibration block, and the corresponding relationship between the calibration corner point and the captured image frame. The camera intrinsic parameter can be specifically understood as the camera's inherent properties required when the camera maps points in the three-dimensional world to the two-dimensional image plane. Exemplarily, the camera intrinsic parameter may include focal length, principal point, distortion coefficient, pixel size, etc., which is not limited in the embodiment of the present invention. The camera extrinsic parameter can be specifically understood as a parameter used to indicate the position and posture of the camera relative to a certain reference coordinate system. In the embodiment of the present invention, the certain reference coordinate system is the vehicle body coordinate system. Exemplarily, the camera extrinsic parameter may include three angles (pitch angle, yaw angle and roll angle) and three translations (x, y and z) used to describe the posture relationship between the camera coordinate system and the vehicle body coordinate system. The first camera extrinsic parameter can be specifically understood as part of the camera extrinsic parameters. For example, the first camera extrinsic parameter in the embodiment of the present invention may include a pitch angle, a yaw angle, and a translation in the z direction.

[0047] Specifically, image processing analysis is performed on each image frame in the image frame set collected by each camera to be calibrated, and the calibration corner point positions extracted from each image frame are combined with each image frame to obtain the calibration corner point information of the camera to be calibrated for each calibration corner point within its collection range. Then, for each camera to be calibrated, the coordinates of each calibration corner point information corresponding to each camera can be transformed according to the camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters as the parameters to be optimized in the objective function are solved according to the mutual relationship between the converted coordinate points and the objective function pre-constructed with part of the camera extrinsic parameters as the parameters to be optimized, and the solution result is determined as the first camera extrinsic parameter corresponding to the camera to be calibrated.

[0048] S103: Determine a camera to be calibrated that simultaneously collects fixed calibration blocks and semi-fixed calibration blocks in a camera calibration site as a first camera to be calibrated, and determine a second camera extrinsic parameter of the first camera to be calibrated according to calibration corner point information corresponding to the first camera to be calibrated.

[0049] In this embodiment, the second camera extrinsic parameter can be specifically understood as the remaining extrinsic parameters in the camera extrinsic parameter except the first extrinsic parameter. On the basis of determining the first camera extrinsic parameter as the pitch angle, yaw angle and translation in the z direction in the embodiment of the present invention, the second camera extrinsic parameter can be the roll angle, translation in the x direction and translation in the y direction.

[0050] Specifically, in order to improve the accuracy of external parameter calibration for vehicle-mounted cameras as much as possible, since the positional relationship between the fixed calibration block and the vehicle body coordinate system is known, the accuracy of the calibration based on it can be guaranteed as much as possible, so the camera to be calibrated based on the fixed calibration block can be taken as the object that needs to be calibrated first. And because the semi-fixed calibration block has the same characteristic as the fixed calibration block that is parallel to any side of the vehicle, it can also play a certain limiting role in the external parameter calibration of the camera to be calibrated. Therefore, in the process of camera external parameter calibration, the camera to be calibrated that can simultaneously collect the fixed calibration block and the semi-fixed calibration block in the camera calibration site can be taken as the first camera to be calibrated that needs to be calibrated first. Then, the calibration corner point information corresponding to the first camera to be calibrated can be converted to the vehicle body coordinate system to obtain the measurement value of the first camera to be calibrated for the fixed calibration block and the semi-fixed calibration block. Based on the parallel relationship between the fixed calibration block and the semi-fixed calibration block and the vehicle body, and the fixed and known relative position relationship between the fixed calibration block and the vehicle, the true value corresponding to the measured value is determined, and the second camera extrinsic parameter in the first camera to be calibrated is solved according to the error relationship between the measured value and the true value, and the solution result is determined as the second camera extrinsic parameter of the first camera to be calibrated.

[0051] S104, determining each camera to be calibrated except the first camera to be calibrated as a second camera to be calibrated, and jointly calibrating each second camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, and determining the second camera extrinsic parameter of each second camera to be calibrated.

[0052] Specifically, since the first camera to be calibrated has been calibrated, the first camera extrinsic parameters and the second camera extrinsic parameters corresponding to it are accurate camera extrinsic parameters that have been calibrated. Therefore, each camera to be calibrated in the vehicle except the first camera to be calibrated can be determined as the second camera to be calibrated, and since each calibration block in the camera calibration site can be completely collected by the adjacent camera to be calibrated on the vehicle, there must be a second camera to be calibrated that collects the same calibration block as the first camera to be calibrated. Therefore, the calibration corner point information corresponding to each second camera to be calibrated can be converted to the vehicle body coordinate system, and the objective function for calibrating the second camera extrinsic parameters of each second camera to be calibrated is constructed based on the position presentation of the calibration corner points in the overlapping area captured by the second camera to be calibrated and the first camera to be calibrated. The objective function is solved based on the calibration basis of the first camera extrinsic parameters and the second camera extrinsic parameters that have been completed by the first camera to be calibrated, and the joint calibration of the second camera extrinsic parameters of each second camera to be calibrated can be completed based on the solution result.

[0053] The technical solution of this embodiment is to obtain a set of image frames obtained by acquiring images of a camera calibration site by each camera to be calibrated; extract calibration corner point information corresponding to each camera to be calibrated from each set of image frames, and determine the first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated; determine the camera to be calibrated that simultaneously collects fixed calibration blocks and semi-fixed calibration blocks in the camera calibration site as the first camera to be calibrated, and determine the second camera extrinsic parameter of the first camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated; determine each camera to be calibrated except the first camera to be calibrated as the second camera to be calibrated, and jointly calibrate each second camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, and determine the second camera extrinsic parameter of each second camera to be calibrated. By adopting the above technical solution, when it is necessary to calibrate the on-board camera, all cameras that need to be calibrated can be used as cameras to be calibrated, and each camera to be calibrated can collect multiple frames of images in the camera calibration site, and then realize the camera extrinsic parameter calibration based on the image frame set collected by different cameras to be calibrated. In the calibration process, each camera to be calibrated will be calibrated step by step and then jointly according to the calibration blocks that can be collected by each camera to be calibrated in the camera calibration site and the different types of camera extrinsic parameters required for calibration. In the step-by-step calibration, the camera extrinsic parameters of each part can be calibrated in a targeted manner, so as to avoid the influence of other parameters on the parameters currently being calibrated, and improve the calibration accuracy; and in the joint calibration, the camera extrinsic parameters that have been calibrated are introduced into the joint calibration process, and the calibration accuracy of the remaining camera extrinsic parameters is optimized. And because the multiple image frames obtained by repeated shooting of each camera to be calibrated are used for comprehensive processing in the process of calibration of each external parameter, the robustness of the calibration process is improved. The entire calibration process can be automatically performed based on the image frame sets captured by the cameras to be calibrated and the pre-arranged camera calibration site. Human participation is only required for the determination of fixed calibration blocks and semi-fixed calibration blocks in the camera calibration site, which reduces the degree of human participation in the calibration process, reduces the possibility of error introduction, and improves the overall calibration accuracy.

[0054] Embodiment 2

[0055] Figure 3A flow chart of a camera extrinsic calibration method provided in Embodiment 2 of the present invention is provided. The embodiment of the present invention is further optimized on the basis of the above-mentioned optional technical solutions. For each image frame in the image frame set, a calibration block is first extracted, and then the calibration points contained in each extracted calibration block are further extracted and identified. Alternatively, the calibration points contained in the calibration block in each image frame in the image frame set can be directly extracted and identified, and then the coordinate position of each identified calibration corner point in the image frame, the association relationship with the calibration block to which it belongs in the image frame, and the correspondence relationship between the image frames are taken together as the calibration corner point information of a calibration corner point extracted from an image frame. According to the extracted information of each calibration corner point and the camera intrinsic parameters, each calibration corner point can be converted from the image coordinate system to the vehicle body coordinate system. When calibrating the first camera extrinsic parameters of each camera to be calibrated, the relative relationship between the length of the line between adjacent corner points belonging to the same calibration block in the same image frame and the actual length will be determined; and when calibrating the second camera extrinsic parameters of the first camera to be calibrated, the body coordinates of the calibration corner points converted to the body coordinate system and the slope of the side formed by the body coordinates of the calibration corner points that should be parallel to the vehicle will be calibrated and determined; and then the second cameras to be calibrated are jointly calibrated using the calibrated first camera to be calibrated and the body coordinates of the calibration corner points of each second camera to be calibrated converted to the body coordinate system to obtain the second camera extrinsic parameters corresponding to each second camera to be calibrated. By combining step-by-step calibration and joint calibration, the calibration accuracy in the step-by-step calibration process is improved based on the calibration block whose relative relationship with the vehicle body coordinate system has been clearly defined. Then, based on the calibrated camera to be calibrated, other uncalibrated cameras to be calibrated are jointly calibrated, and the calibrated camera extrinsics are introduced into the joint calibration process. The calibration accuracy of the remaining camera extrinsics is optimized, so that the camera to be calibrated in the joint calibration does not need to rely on manual participation in the measurement of the calibration block position, which reduces the degree of manual participation in the calibration process, reduces the possibility of error introduction, and improves the overall calibration accuracy.

[0056] like Figure 3 As shown, a camera extrinsic calibration method provided by an embodiment of the present invention specifically includes the following steps:

[0057] S201, obtaining a set of image frames obtained by each camera to be calibrated through image acquisition of a camera calibration site.

[0058] S202 . For each image frame in the image frame set, extract the image coordinates of calibration corner points corresponding to the calibration blocks from each calibration block collected by the image frame.

[0059] Wherein, a calibration block includes at least four calibration corner points.

[0060] Specifically, for each image frame in the image frame set, the calibration blocks contained therein are first identified, and then the calibration corner points are identified based on the shapes with special geometric features in different calibration blocks, and the position of each identified calibration corner point in the image frame is determined as the calibration corner point image coordinates.

[0061] It can be understood that since the embodiment of the present invention needs to use the parallel relationship between the calibration block and the vehicle to complete the camera extrinsic calibration, two line segments parallel to the vehicle need to be identified in the same calibration block, so the calibration block used in the embodiment of the present invention needs to include at least four calibration corner points.

[0062] For example, as above Figure 2 In the calibration block shown, the special geometric shape is a circle, that is, each calibration block has four black circular areas, and the center of each circular area on the calibration block can be determined as the calibration corner point of the calibration block. In some other examples, if the calibration block is a solid color square calibration block, the four corners of the top of the square calibration block can be used as the four calibration corner points of the calibration block. The embodiment of the present invention does not specifically limit the style of the calibration block.

[0063] S203, taking the image coordinates of each calibration corner point, the correspondence between the image coordinates of each calibration corner point and the calibration block, and the correspondence between the image coordinates of each calibration corner point and the image frame in the image frame set as calibration corner point information corresponding to the camera to be calibrated.

[0064] Specifically, to facilitate the subsequent calibration calculation of camera extrinsic parameters, after clarifying the image coordinates of the calibration corner points of each calibration corner point, the corresponding relationship between the image coordinates of each calibration corner point and the calibration block in the image frame, as well as the corresponding relationship between the image coordinates of each calibration corner point and the image frame in the image frame set can also be clarified, and the above three can be combined into the calibration corner point information corresponding to a calibration corner point. That is, for a camera to be calibrated, the calibration corner point information extracted from an image frame acquired once can include the position coordinates of the calibration corner point in the image frame, which calibration block in the image frame, the geometric relationship between the calibration corner point and other calibration corner points in the calibration block, and the association relationship with the image frame.

[0065] S204 . For each camera to be calibrated, convert each calibration corner point information corresponding to the camera to be calibrated into a vehicle body coordinate system according to the camera intrinsic parameters of the camera to be calibrated, and determine the vehicle body coordinates of the calibration corner points corresponding to each calibration corner point information.

[0066] Specifically, since each camera to be calibrated has different camera intrinsic parameters and the existing camera extrinsic parameters at the current moment are also different, the coordinate conversion from the image coordinate system to the vehicle body coordinate system must rely on the camera intrinsic and extrinsic parameters. Therefore, the coordinate system conversion of the corresponding calibration corner point information will be performed for each camera to be calibrated, and the calibration corner point vehicle body coordinates corresponding to each calibration corner point information will be obtained.

[0067] S205 , connecting adjacent corner points belonging to the same calibration block in the same image frame to determine a plurality of first edges.

[0068] Specifically, since the same image frame may contain multiple calibration blocks, and each calibration block contains multiple calibration corner points, after determining the calibration corner points contained in the image frame, the calibration corner points belonging to the same calibration block in the image frame may be connected according to the adjacent relationship, and each connection line may be determined as a first edge.

[0069] For example, Figure 2 Take an image frame captured by a vehicle's front-view camera in the camera calibration site as an example. Figure 4 This is a bird's-eye view example of a forward-looking camera provided in Embodiment 2 of the present invention, such as Figure 4 As shown in FIG. 1 , the image frame captured by the camera to be calibrated as the front-view camera contains two calibration blocks, each of which contains four calibration corner points. The calibration corner points in each calibration block are connected in sequence according to the adjacent relationship to obtain the following: Figure 4 The eight first edges shown are represented by l 1 ,l 2 ,l 3 .l 4 .l 5 ,l 6 ,l 7 ,l 8 .

[0070] It can be understood that eight first edges can be determined in each image frame captured by the camera to be calibrated.

[0071] S206, determining the first side calibrated length of each first side according to the vehicle body coordinates of each calibrated corner point.

[0072] Specifically, after the calibration corner point body coordinates of each calibration corner point collected by the camera to be calibrated are known, the calibration corner point body coordinates of the two end points of the first edge can be determined for each first edge, and then the calibration corner point body coordinates of the two end points are substituted into the Euclidean distance formula to obtain the length of the first edge, and the length is determined as the first edge calibration side length of the first edge.

[0073] S207, determining the pitch angle, yaw angle, and first translation of the camera to be calibrated according to the calibrated side lengths of each first side, the average calibrated side lengths of the first sides corresponding to each calibrated side length, and the actual side lengths of the first sides corresponding to each calibrated side length, and determining the pitch angle, yaw angle, and first translation as the first camera extrinsic parameters of the camera to be calibrated.

[0074] In this embodiment, the average calibrated length of the first side can be specifically understood as the length value obtained by summing and averaging all the calibrated lengths of the first sides corresponding to the camera to be calibrated. The actual length of the first side can be specifically understood as the actual length of the first side in the calibration block corresponding to the calibrated length of the first side. The first translation amount can be specifically understood as the translation amount in the z direction.

[0075] Specifically, according to the correspondence between the calibrated side lengths of each first side and the image frame, the calibrated side lengths of each first side and the average calibrated side length of the first side are substituted into the predefined objective function with the pitch angle and the yaw angle as the parameters to be optimized, and the solution is performed with the goal of minimizing the variance of the side lengths, and the solution results are determined as the pitch angle and yaw angle corresponding to the camera to be calibrated. At the same time, the calibrated side lengths of each first side and the actual side lengths of each first side can be substituted into the predefined objective function with the z-direction translation as the parameter to be optimized, and the solution is performed with the goal of minimizing the sum of the absolute difference between the actual value and the calibrated value, and the solution results are determined as the first translation corresponding to the camera to be calibrated. Then, the pitch angle, yaw angle and first translation completed by the above calibration can be used together as the first camera extrinsic parameter of the camera to be calibrated.

[0076] It is understandable that each camera to be calibrated will perform the above processing and complete the corresponding first camera extrinsic parameter calibration.

[0077] Exemplarily, the objective function using the pitch angle and the yaw angle as parameters to be optimized can be expressed as:

[0078]

[0079] Among them, α is the pitch angle to be optimized, β is the yaw angle to be optimized, is the variance of the length of the line connecting adjacent corner points, Specifically, it can be expressed as:

[0080]

[0081] Where n is the number of first edges in an image frame, m is the total number of frames in the image frame set, and d ave is the average nominal length of the first side, d j，i Calibrate the length of the first edge of the i-th first edge of the j-th frame.

[0082] Continuing with the above example, the objective function taking the z-direction translation as the parameter to be optimized can be expressed as:

[0083]

[0084] Among them, z is the first translation to be optimized, E d is the sum of the absolute differences between the actual value and the calibrated value of each first side, E d Specifically, it can be expressed as:

[0085]

[0086] Among them, d true The actual length of the first side of the i-th first side in the j-th frame.

[0087] S208. Determine the camera to be calibrated that simultaneously collects information about the fixed calibration block and the semi-fixed calibration block in the camera calibration site as the first camera to be calibrated, convert the calibration corner point information corresponding to the first camera to be calibrated into the vehicle body coordinate system, and determine the vehicle body coordinates of the calibration corner points corresponding to each calibration corner point information.

[0088] Specifically, the camera to be calibrated that can simultaneously capture the fixed calibration block and the semi-fixed calibration block in the camera calibration site is determined as the first camera to be calibrated, and the calibration corner point information corresponding to the first camera to be calibrated is converted to the vehicle body coordinate system according to the camera intrinsic parameters of the first camera to be calibrated, the extrinsic parameters of the first camera that has been calibrated, and the extrinsic parameters of the remaining cameras that have not been calibrated, to obtain the calibration corner point vehicle body coordinates corresponding to each calibration corner point information.

[0089] S209, determining the calibrated corner point body coordinates corresponding to the fixed calibration block in each calibrated corner point body coordinates as the fixed corner point body coordinates, and determining the calibrated corner point body coordinates corresponding to the semi-fixed calibration block as the semi-fixed corner point body coordinates.

[0090] Specifically, according to the correspondence between the body coordinates of each calibrated corner point and the calibration block to which it belongs in the image frame, the body coordinates of the calibrated corner point belonging to the fixed calibration block are determined as the fixed corner point body coordinates, and the body coordinates of the calibrated corner point belonging to the semi-fixed calibration block are determined as the semi-fixed corner point body coordinates.

[0091] S210: Determine a line formed by the vehicle body coordinates of each fixed corner point and the vehicle body coordinates of each semi-fixed corner point and parallel to any side of the vehicle as a second side.

[0092] Specifically, since the fixed calibration block and the semi-fixed calibration block are parallel to an edge of the vehicle, there should also be calibration corner point lines in the fixed calibration block and the semi-fixed calibration block that are parallel to the edge of the vehicle. At this time, for the body coordinates of each fixed corner point, they can be connected in pairs, and the line obtained after the connection is parallel to any edge of the vehicle is determined as the second edge. The same process can be performed for the body coordinates of each semi-fixed corner point.

[0093] Continuing with the above example, Figure 2 In the camera calibration site shown, the fixed calibration block is the upper left corner calibration block, and the semi-fixed calibration block is the upper right corner calibration block. For example, the fixed calibration block and the semi-fixed calibration block are parallel to the front side of the vehicle. Figure 5 Another example of a bird's-eye view of a front-view camera provided in the second embodiment of the present invention is shown in FIG. Figure 5 As shown, according to the fixed corner body coordinates and semi-fixed corner body coordinates corresponding to the front camera, four sets of corner body coordinate combinations parallel to the front side of the vehicle can be determined, and after connecting the corner body coordinate combinations, the following can be obtained: Figure 5 The four second sides shown are represented by l 1 , l 2 , l 3 , l 4 .in, Figure 5 The calibration block on the left side is the image captured by the front-view camera for the fixed calibration block. The converted coordinates of the four calibration corner points a, b, c, and d are the body coordinates of the fixed corner points.

[0094] S211 . Determine a roll angle of the first camera to be calibrated according to the calibrated slopes of the second sides and the actual slopes corresponding to the second sides.

[0095] In this embodiment, the calibration slope can be specifically understood as the slope of the second side currently determined based on the body coordinates of each fixed corner point and each semi-fixed corner point. Since some camera external parameters have not been calibrated, there may be a certain deviation in the slope. The real slope can be specifically understood as the slope that the second side should actually have on the corresponding calibration block.

[0096] Specifically, the calibrated slope of each second edge can be determined based on the vehicle body coordinates of the endpoints of each second edge. Since it is affected by some uncalibrated external parameters, there will be a difference between the corresponding true slope. The calibrated slopes of the second edges in different image frames can be converted into angles, and the average of the calibration angles can be calculated; the true slopes of each second edge are also converted into angles to obtain the corresponding true angle values, and the difference between the average of the calibration angles and the true angle value is determined as the roll angle of the first camera to be calibrated.

[0097] For example, assuming that the calibration slope of a second edge can be expressed as k, the calibration slope can be converted into an angle θ by an inverse tangent function, and then the average calibration angle obtained by processing the calibration slopes of all second edges in different image frames can be expressed as:

[0098]

[0099] Where m is the number of frames; n is the number of second edges in each image frame. Figure 4Take the middle scenario as an example in the embodiment of the present invention, n=4; is the mean value of the calibration angle; θ j，i is the angle corresponding to the calibrated slope of the i-th second edge in the j-th frame.

[0100] S212: Determine a second translation amount of the first camera to be calibrated according to the roll angle of the first camera to be calibrated and the coordinates of each fixed corner point of the vehicle body.

[0101] Specifically, in order to improve the stability of parameter determination through multiple shooting, the fixed corner point body coordinates for the same calibration corner point in multiple frames can be averaged as the average fixed corner point body coordinates used for the camera external parameters. Since the change of the roll angle will cause the coordinates obtained after the conversion on the left to change, after completing the roll angle calibration of the first camera to be calibrated, the average fixed corner point body coordinates corresponding to each calibration corner point will be transformed according to the calibrated roll angle, which is equivalent to only rotating the OXY two-dimensional plane. In this process, the translation of the first camera to be calibrated in the x and y directions will be introduced. At the same time, since the relative position relationship between the fixed calibration block and the vehicle body coordinate system is fixed and known, the true values ​​of the corner point coordinates corresponding to the different fixed corner point vehicle body coordinates in the vehicle body coordinate system can be clearly known. The difference between the fixed corner point vehicle body coordinates after the calibration roll angle processing and the corresponding true values ​​of the corner point coordinates can be used to obtain the translation vector of the first camera to be calibrated in the x and y directions. Adding it to the translation obtained when the roll angle processing is introduced can obtain the final translation required for calibration of the first camera to be calibrated in the x and y directions, that is, the second translation of the first camera to be calibrated is obtained.

[0102] For example, Figure 5 Taking the fixed calibration block shown in FIG. 1 as an example, which includes four calibration corner points a, b, c, and d, the average fixed corner point vehicle body coordinates corresponding to each calibration corner point can be expressed as:

[0103]

[0104] Among them, m is the number of frames, P d，j is the fixed corner point vehicle body coordinate of the calibration corner point d in the jth frame, and the same is true for the other calibration corner points.

[0105] After introducing the roll angle to transform the body coordinates of each average fixed corner point, we can get:

[0106]

[0107] Among them, P a ′ is the calibration corner point a after rotation; is the rotation matrix; t xy，0 =(t x，0 , t y，0) is the translation vector. The processing method of the remaining calibrated corner points is consistent with the above processing method, and the embodiment of the present invention will not be described in detail.

[0108] Then, the difference between each calibrated corner point of the rotation transformation and the true value of the corresponding corner point coordinates is calculated to obtain four groups of translations. The translation vector is calculated by averaging them. The second translation of the first camera to be calibrated can be obtained by combining the translation vector with the translation vector introduced during the rotation process. This can be achieved in the following way:

[0109]

[0110] Where n is the number of calibration corner points in the fixed calibration block; P true，* is the true value of the corner point coordinates of the calibrated corner point*; ΔP=(Δt x , Δt y ).

[0111] The final second translation of the first camera to be calibrated can be expressed as: t x =t x，0 +Δt x , t y =t y，0 +Δt y .

[0112] S213: Determine the roll angle and the second translation amount as the second camera extrinsic parameters of the first camera to be calibrated.

[0113] S214, determining each camera to be calibrated except the first camera to be calibrated as a second camera to be calibrated, and converting the calibration corner point information corresponding to the first camera to be calibrated to the vehicle body coordinate system according to the first camera extrinsic parameters and the second camera extrinsic parameters corresponding to the first camera to be calibrated, and determining them as the first calibration corner point vehicle body coordinates.

[0114] Specifically, after the first camera to be calibrated has completed the calibration of all camera extrinsics, the remaining cameras to be calibrated need to be calibrated. At this time, all cameras to be calibrated except the first camera to be calibrated can be determined as the second camera to be calibrated. Furthermore, based on the first camera extrinsics and the second camera extrinsics that have been calibrated for the first camera to be calibrated, coordinate conversion can be performed on the calibration corner point information corresponding to the first camera to be calibrated, and the calibration corner point coordinates obtained after the conversion are determined as the body coordinates of the first calibration corner point.

[0115] S215. Convert the calibration corner point information corresponding to each second camera to be calibrated to the vehicle body coordinate system according to the first camera extrinsic parameter corresponding to each second camera to be calibrated, and determine the vehicle body coordinates of the second calibration corner point corresponding to each second camera to be calibrated.

[0116] Specifically, since the first camera extrinsic parameters of each second camera to be calibrated have been calibrated, the calibration corner point information corresponding to each second camera to be calibrated can be converted into a coordinate system based on the calibrated first camera extrinsic parameters and the camera intrinsic parameters of each second camera to be calibrated, and the calibration corner points converted to the vehicle body coordinate system can be determined as the second calibration corner point vehicle body coordinates corresponding to each second camera to be calibrated.

[0117] S216: Determine an overlapping area according to the acquisition range of the first camera to be calibrated and each of the second cameras to be calibrated.

[0118] Specifically, since each calibration block in the camera calibration site can be completely collected by the adjacent cameras to be calibrated on the vehicle, that is, the adjacent first camera to be calibrated and the second camera to be calibrated, as well as the adjacent second cameras to be calibrated, can be collected for the same calibration block, the collection range corresponding to the calibration block can be understood as the overlapping area between the adjacent first camera to be calibrated and the second camera to be calibrated, as well as the adjacent second cameras to be calibrated.

[0119] S217 , determining an overlapping region matching vector and an overlapping region matching corner point pair corresponding to each overlapping region according to the vehicle body coordinates of each first calibrated corner point and the vehicle body coordinates of each second calibrated corner point.

[0120] Specifically, since each overlapping area corresponds to a calibration block in the camera calibration site, and a calibration block includes at least four calibration corner points, for an overlapping area, a vector can be formed by randomly connecting lines among the calibration corner points on the calibration block corresponding to the overlapping area, and the vector corresponding to the same random connecting line vector in the image frames taken by different cameras to be calibrated is determined as the overlapping area matching vector, and the calibration corner points in the calibration block of the overlapping area in the image frames taken by different cameras to be calibrated are determined as the overlapping area matching corner point pairs.

[0121] For example, the front-view camera and the right-view camera are Figure 2 Take the camera calibration site shown in the figure as an example. Figure 6 This is a bird's-eye view example of line segment matching in a camera overlap area provided in the second embodiment of the present invention. Figure 7 This is a bird's-eye view example of corner point matching in a camera overlap area provided in the second embodiment of the present invention. Figure 6 As shown, the front camera and the right camera will be Figure 2 The calibration block in the upper right corner is used to shoot, and the vector in the hypotenuse direction of the calibration block is used as the vector for matching the front camera and the right camera. Figure 6 It can be considered and is the matching vector, and is the matching vector. Figure 7As shown in FIG. 1 , a set of points collected for the same calibration corner point in the calibration block can be determined as an overlapping area matching corner point pair. Figure 7 It can be considered that (P 1 ,Q 1 )、(P 2 ,Q 2 )、(P 3 ,Q 3 ) and (P 4 ,Q 4 ) is the matching corner point pair in the overlapping area.

[0122] S218. Determine the roll angle of each second camera to be calibrated according to the matching vectors of each overlapping area.

[0123] Specifically, each overlapping area matching vector is substituted into a predefined objective function that takes the roll angle of each second camera to be calibrated as a parameter to be optimized, a solution is performed with the goal of minimizing the sum of squared deviations, and the solution result is determined as the roll angle of each second camera to be calibrated.

[0124] Continuing with the above example, the objective function using the roll angle of each second camera to be calibrated as the parameter to be optimized can be expressed as:

[0125]

[0126] Among them, ε is the parameter to be optimized, including the roll angles of the left, right and rear cameras, E a Specifically, it can be expressed as:

[0127]

[0128] Wherein, n is the number of overlapping regions. In the embodiment of the present invention, n=4, including four overlapping regions of front left, front right, rear left and rear right. m is the number of frames.

[0129] S219: Determine the second translation amount of each second camera to be calibrated according to the matching corner point pairs in each overlapping area.

[0130] Specifically, the matching corner points of each overlapping area are substituted into the predefined objective function that takes the translation amount of each second camera to be calibrated in the x and y directions as the parameters to be optimized, and the solution is performed with the goal of minimizing the second norm of the deviation between the two points, and the solution result is determined as the second translation amount of each second camera to be calibrated.

[0131] Continuing with the above example, the objective function using the translation amounts of each second camera to be calibrated in the x and y directions as parameters to be optimized can be expressed as:

[0132]

[0133] Among them, τ is the parameter to be optimized, including the translation of the left, right and rear view cameras in the x and y directions, E t Specifically, it can be expressed as:

[0134]

[0135] Among them, s is the number of overlapping area matching corner point pairs in each frame overlapping area, n is the number of overlapping areas, in the embodiment of the present invention, n=4, including four overlapping areas of front left, front right, rear left and rear right, and m is the number of frames.

[0136] S220: Determine each roll angle and each second translation amount as the second camera extrinsic parameters of each second camera to be calibrated.

[0137] The technical solution of this embodiment is to first extract the calibration blocks from each image frame in the image frame set, and then further extract and identify the calibration points contained in each extracted calibration block, and use the coordinate position of each identified calibration corner point in the image frame, the association relationship with the calibration block to which it belongs in the image frame, and the correspondence between the image frames as the calibration corner point information of a calibration corner point extracted from an image frame. According to the extracted information of each calibration corner point and the camera intrinsic parameters, each calibration corner point can be converted from the image coordinate system to the vehicle body coordinate system. When calibrating the first camera extrinsic parameters of each camera to be calibrated, the relative relationship between the length of the line between adjacent corner points belonging to the same coordinate block in the same image frame and the actual length will be determined; and when calibrating the second camera extrinsic parameters of the first camera to be calibrated, the body coordinates of the calibration corner points converted to the body coordinate system and the slope of the side formed by the body coordinates of the calibration corner points that should be parallel to the vehicle will be calibrated and determined; and then the second cameras to be calibrated are jointly calibrated using the calibrated first camera to be calibrated and the body coordinates of the calibration corner points of each second camera to be calibrated converted to the body coordinate system to obtain the second camera extrinsic parameters corresponding to each second camera to be calibrated. By combining step-by-step calibration and joint calibration, the calibration accuracy in the step-by-step calibration process is improved based on the calibration block whose relative relationship with the vehicle body coordinate system has been clearly defined. Then, based on the calibrated camera to be calibrated, other uncalibrated cameras to be calibrated are jointly calibrated, and the calibrated camera extrinsics are introduced into the joint calibration process. The calibration accuracy of the remaining camera extrinsics is optimized, so that the camera to be calibrated in the joint calibration does not need to rely on manual participation in the measurement of the calibration block position, which reduces the degree of manual participation in the calibration process, reduces the possibility of error introduction, and improves the overall calibration accuracy.

[0138] Embodiment 3

[0139] Figure 8 A schematic diagram of the structure of a camera extrinsic calibration device provided in Embodiment 3 of the present invention is shown in FIG. Figure 8As shown, the camera extrinsic parameter calibration device includes an image frame set acquisition module 31, a first calibration module 32, a second calibration module 33 and a joint calibration module 34.

[0140] Among them, the image frame set acquisition module 31 is used to obtain the image frame set obtained by each camera to be calibrated performing image acquisition on the camera calibration site; the first calibration module 32 is used to extract the calibration corner point information corresponding to each camera to be calibrated from each image frame set, and determine the first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated; the second calibration module 33 is used to determine the camera to be calibrated that simultaneously acquires the fixed calibration block and the semi-fixed calibration block in the camera calibration site as the first camera to be calibrated, and determine the second camera extrinsic parameter of the first camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated; the joint calibration module 34 is used to determine each camera to be calibrated except the first camera to be calibrated as the second camera to be calibrated, and jointly calibrate each second camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, and determine the second camera extrinsic parameter of each second camera to be calibrated.

[0141] The technical solution of the embodiment of the present invention, when it is necessary to calibrate the vehicle-mounted camera, all the cameras that need to be calibrated can be used as the cameras to be calibrated, and each camera to be calibrated can collect multiple frames of images in the camera calibration site, and then realize the camera extrinsic parameter calibration based on the image frame sets collected by different cameras to be calibrated. In the calibration process, each camera to be calibrated will be calibrated step by step and then jointly according to the calibration blocks that can be collected by each camera to be calibrated in the camera calibration site and the different types of camera extrinsic parameters required for calibration. In the step-by-step calibration, the camera extrinsic parameters of each part can be calibrated in a targeted manner, so as to avoid the influence of other parameters on the parameters currently being calibrated, and improve the calibration accuracy; and in the joint calibration, the camera extrinsic parameters that have been calibrated are introduced into the joint calibration process, and the calibration accuracy of the remaining camera extrinsic parameters is optimized. And because the multiple image frames obtained by repeated shooting of each camera to be calibrated are used for comprehensive processing in the process of each external parameter calibration, the robustness of the calibration process is improved. The entire calibration process can be automatically performed based on the image frame sets captured by the cameras to be calibrated and the pre-arranged camera calibration site. Human participation is only required for the determination of fixed calibration blocks and semi-fixed calibration blocks in the camera calibration site, which reduces the degree of human participation in the calibration process, reduces the possibility of error introduction, and improves the overall calibration accuracy.

[0142] Optionally, the camera calibration site includes a fixed calibration block, a semi-fixed calibration block, and at least two random calibration blocks;

[0143] Among them, the fixed calibration block and the semi-fixed calibration block are calibration blocks set parallel to any side of the vehicle in the camera calibration site;

[0144] Wherein, the relative position between the fixed calibration block and the vehicle is fixed and known;

[0145] Among them, each calibration block in the camera calibration site can be completely captured by the adjacent to-be-calibrated camera on the vehicle.

[0146] Optionally, the first calibration module 32 is specifically used for:

[0147] For each image frame in the image frame set, the image coordinates of the calibration corner points corresponding to the calibration blocks are extracted from each calibration block collected by the image frame; wherein one calibration block includes at least four calibration corner points;

[0148] The image coordinates of each calibration corner point, the corresponding relationship between the image coordinates of each calibration corner point and the calibration block, and the corresponding relationship between the image coordinates of each calibration corner point and the image frame in the image frame set are collectively used as the calibration corner point information corresponding to the camera to be calibrated;

[0149] For each camera to be calibrated, the calibration corner point information corresponding to the camera to be calibrated is converted to the vehicle body coordinate system according to the camera intrinsic parameters of the camera to be calibrated, and the vehicle body coordinates of the calibration corner points corresponding to the calibration corner point information are determined;

[0150] Connect adjacent corner points belonging to the same calibration block in the same image frame to determine a plurality of first edges;

[0151] Determine the first side calibration length of each first side according to the vehicle body coordinates of each calibration corner point;

[0152] According to the calibrated side lengths of each first side, the average calibrated side lengths of the first sides corresponding to each calibrated side length, and the actual side lengths of the first sides corresponding to each calibrated side length, the pitch angle, yaw angle, and first translation of the camera to be calibrated are determined, and the pitch angle, yaw angle, and first translation are determined as the first camera extrinsic parameters of the camera to be calibrated.

[0153] Optionally, the second calibration module 33 is specifically used for:

[0154] The camera to be calibrated that simultaneously collects the fixed calibration block and the semi-fixed calibration block in the camera calibration site is determined as the first camera to be calibrated, the calibration corner point information corresponding to the first camera to be calibrated is converted to the vehicle body coordinate system, and the vehicle body coordinates of the calibration corner points corresponding to each calibration corner point information are determined;

[0155] The body coordinates of the calibrated corner points corresponding to the fixed calibration blocks are determined as the body coordinates of the fixed corner points, and the body coordinates of the calibrated corner points corresponding to the semi-fixed calibration blocks are determined as the body coordinates of the semi-fixed corner points;

[0156] The connecting line formed by the body coordinates of each fixed corner point and the body coordinates of each semi-fixed corner point and parallel to any side of the vehicle is determined as the second side;

[0157] Determine the roll angle of the first camera to be calibrated according to the calibrated slopes of the second sides and the actual slopes corresponding to the second sides;

[0158] Determine a second translation amount of the first camera to be calibrated according to the roll angle of the first camera to be calibrated and the coordinates of each fixed corner point of the vehicle body;

[0159] The roll angle and the second translation amount are determined as the second camera extrinsic parameters of the first camera to be calibrated.

[0160] Optionally, the joint calibration module 34 is specifically used for:

[0161] Determine each camera to be calibrated except the first camera to be calibrated as a second camera to be calibrated, and convert the calibration corner point information corresponding to the first camera to be calibrated into the vehicle body coordinate system according to the first camera extrinsic parameter and the second camera extrinsic parameter corresponding to the first camera to be calibrated, and determine it as the vehicle body coordinate of the first calibration corner point;

[0162] Convert the calibration corner point information corresponding to each second camera to be calibrated to the vehicle body coordinate system according to the first camera extrinsic parameter corresponding to each second camera to be calibrated, and determine the vehicle body coordinates of the second calibration corner point corresponding to each second camera to be calibrated;

[0163] Determine an overlapping area according to the acquisition range of the first camera to be calibrated and each of the second cameras to be calibrated;

[0164] Determine an overlapping region matching vector and an overlapping region matching corner point pair corresponding to each overlapping region according to the vehicle body coordinates of each first calibrated corner point and the vehicle body coordinates of each second calibrated corner point;

[0165] Determine the roll angle of each second camera to be calibrated according to the matching vectors of each overlapping area;

[0166] Determine the second translation amount of each second camera to be calibrated according to the matching corner point pairs in each overlapping area;

[0167] The roll angles and the second translation amounts are determined as second camera extrinsic parameters of the second cameras to be calibrated.

[0168] The calibration accuracy verification device of the embodiment of the present invention can execute the calibration accuracy verification method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0169] Embodiment 4

[0170] Fig. 9A schematic diagram of the structure of a camera extrinsic calibration device provided in Embodiment 4 of the present invention. The camera extrinsic calibration device 40 may be intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The camera extrinsic calibration device 40 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0171] like Fig. 9 As shown, the camera extrinsic calibration device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In RAM 43, various programs and data required for the operation of the camera extrinsic calibration device 40 can also be stored. The processor 41, ROM 42 and RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0172] Multiple components in the camera extrinsic calibration device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the camera extrinsic calibration device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0173] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as a camera extrinsic calibration method.

[0174] In some embodiments, the camera extrinsic calibration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the camera extrinsic calibration device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the camera extrinsic calibration method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to execute the camera extrinsic calibration method in any other appropriate manner (e.g., by means of firmware).

[0175] Optionally, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the camera extrinsic parameter calibration method provided by any embodiment of the present invention.

[0176] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0178] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein may be implemented on a camera extrinsic calibration device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or trackball), through which a user may provide input to the camera extrinsic calibration device. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0180] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0181] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0182] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A camera extrinsic calibration method, characterized in that: include: Obtain a set of image frames obtained by each camera to be calibrated through image acquisition of the camera calibration site; Extracting calibration corner point information corresponding to each camera to be calibrated from each of the image frame sets, and determining a first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated; Determine a camera to be calibrated that simultaneously collects the fixed calibration block and the semi-fixed calibration block in the camera calibration site as a first camera to be calibrated, and determine a second camera extrinsic parameter of the first camera to be calibrated according to calibration corner point information corresponding to the first camera to be calibrated; Each camera to be calibrated except the first camera to be calibrated is determined as a second camera to be calibrated, and each second camera to be calibrated is jointly calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated to determine the second camera extrinsic parameter of each second camera to be calibrated.

2. The camera extrinsic calibration method according to claim 1, characterized in that: The camera calibration site includes a fixed calibration block, a semi-fixed calibration block, and at least two random calibration blocks; Wherein, the fixed calibration block and the semi-fixed calibration block are calibration blocks arranged parallel to any side of the vehicle in the camera calibration site; Wherein, the relative position between the fixed calibration block and the vehicle is fixed and known; Wherein, each calibration block in the camera calibration site can be completely captured by the adjacent to-be-calibrated camera on the vehicle.

3. The camera extrinsic calibration method according to claim 2, characterized in that: Extracting the calibration corner point information corresponding to each camera to be calibrated from each of the image frame sets includes: For each image frame in the image frame set, extracting the calibration corner point image coordinates corresponding to the calibration block from each calibration block collected by the image frame; The image coordinates of each calibration corner point, the corresponding relationship between each calibration corner point image coordinate and the calibration block, and the corresponding relationship between each calibration corner point image coordinate and the image frame in the image frame set are collectively used as the calibration corner point information corresponding to the camera to be calibrated; Wherein, a calibration block includes at least four calibration corner points.

4. The camera extrinsic calibration method according to claim 3, characterized in that: The determining the first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated includes: For each of the cameras to be calibrated, convert the calibration corner point information corresponding to the camera to be calibrated into a vehicle body coordinate system according to the camera intrinsic parameters of the camera to be calibrated, and determine the vehicle body coordinates of the calibration corner points corresponding to the calibration corner point information; Connect adjacent corner points belonging to the same calibration block in the same image frame to determine a plurality of first edges; Determining the first side calibrated length of each of the first sides according to the vehicle body coordinates of each of the calibrated corner points; According to the calibrated side lengths of each first side, the average calibrated side lengths of the first sides corresponding to each calibrated side length of the first sides, and the actual side lengths of the first sides corresponding to each calibrated side length of the first sides, the pitch angle, the yaw angle, and the first translation of the camera to be calibrated are determined, and the pitch angle, the yaw angle, and the first translation are determined as the first camera extrinsic parameters of the camera to be calibrated.

5. The camera extrinsic calibration method according to claim 3, characterized in that: The determining the second camera extrinsic parameter of the first camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated includes: Converting the calibration corner point information corresponding to the first camera to be calibrated into a vehicle body coordinate system, and determining the vehicle body coordinates of the calibration corner points corresponding to each of the calibration corner point information; Determine the calibrated corner point body coordinates corresponding to the fixed calibration block in the calibrated corner point body coordinates as the fixed corner point body coordinates, and determine the calibrated corner point body coordinates corresponding to the semi-fixed calibration block as the semi-fixed corner point body coordinates; Determine a line formed by the body coordinates of each of the fixed corner points and the body coordinates of each of the semi-fixed corner points and parallel to any side of the vehicle as a second side; Determining the roll angle of the first camera to be calibrated according to the calibrated slopes of the second sides and the actual slopes corresponding to the second sides; Determine a second translation amount of the first camera to be calibrated according to the roll angle of the first camera to be calibrated and the vehicle body coordinates of each of the fixed corner points; The roll angle and the second translation amount are determined as second camera extrinsic parameters of the first camera to be calibrated.

6. The camera extrinsic calibration method according to claim 3, characterized in that: The method of jointly calibrating each of the second cameras to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each of the second cameras to be calibrated to determine the second camera extrinsic parameter of each of the second cameras to be calibrated includes: According to the first camera extrinsic parameter and the second camera extrinsic parameter corresponding to the first camera to be calibrated, the calibration corner point information corresponding to the first camera to be calibrated is converted into the vehicle body coordinate system to determine the first calibration corner point vehicle body coordinate; Converting the calibration corner point information corresponding to each of the second cameras to be calibrated to a vehicle body coordinate system according to the first camera extrinsic parameters corresponding to each of the second cameras to be calibrated, and determining the vehicle body coordinates of the second calibration corner points corresponding to each of the second cameras to be calibrated; Determining an overlapping area according to the acquisition range of the first camera to be calibrated and each of the second cameras to be calibrated; Determine an overlapping region matching vector and an overlapping region matching corner point pair corresponding to each overlapping region according to the vehicle body coordinates of each of the first calibrated corner points and the vehicle body coordinates of each of the second calibrated corner points; Determining the roll angle of each of the second cameras to be calibrated according to the matching vectors of each of the overlapping areas; Determining a second translation amount of each of the second cameras to be calibrated according to the matching corner point pairs of each of the overlapping areas; The roll angles and the second translation amounts are determined as second camera extrinsic parameters of the second cameras to be calibrated.

7. A camera extrinsic calibration device, characterized in that: include: An image frame set acquisition module is used to acquire an image frame set obtained by each camera to be calibrated through image acquisition of a camera calibration site; A first calibration module, configured to extract calibration corner point information corresponding to each camera to be calibrated from each of the image frame sets, and determine a first camera extrinsic parameter of each camera to be calibrated according to the calibration corner point information and the camera intrinsic parameter of each camera to be calibrated; a second calibration module, configured to determine a camera to be calibrated that simultaneously collects the fixed calibration block and the semi-fixed calibration block in the camera calibration site as a first camera to be calibrated, and determine a second camera extrinsic parameter of the first camera to be calibrated according to calibration corner point information corresponding to the first camera to be calibrated; The joint calibration module is used to determine each camera to be calibrated except the first camera to be calibrated as a second camera to be calibrated, and jointly calibrate each second camera to be calibrated according to the calibration corner point information corresponding to the first camera to be calibrated, the first camera extrinsic parameter and the second camera extrinsic parameter, and the calibration corner point information corresponding to each second camera to be calibrated, so as to determine the second camera extrinsic parameter of each second camera to be calibrated.

8. A camera extrinsic calibration device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the camera extrinsic parameter calibration method according to any one of claims 1 to 6.

9. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the camera extrinsic parameter calibration method according to any one of claims 1 to 6 when executed by a computer processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the camera extrinsic parameter calibration method according to any one of claims 1 to 6 is implemented.

Images