An external parameter calibration method, device and equipment

Abstract

This application provides an extrinsic parameter calibration method, apparatus, and device. The method includes: acquiring a first reference image of an auxiliary target using a vehicle-mounted camera; determining a first pose of the auxiliary target relative to the vehicle-mounted camera based on the first reference image; acquiring a second reference image of a target to be detected using the auxiliary camera; determining a second pose of the target plane relative to the auxiliary camera based on the second reference image; determining a fourth pose of the auxiliary camera relative to the vehicle-mounted camera based on the first pose and a previously acquired third pose of the auxiliary target relative to the auxiliary camera; and determining the target extrinsic parameters of the target plane in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose. The technical solution of this application only requires the fabrication of a calibration tool with fixed positions for the auxiliary camera and the auxiliary target, offering advantages such as flexible implementation and higher calibration accuracy.

Description

Technical Field

[0001] This application relates to the field of external parameter calibration technology, and in particular to an external parameter calibration method, apparatus and equipment. Background Technology

[0002] Vehicle-mounted cameras (such as camcorders) can be installed in vehicles to enable functions such as driver assistance and fatigue detection. For example, the vehicle-mounted camera can capture images of the driver within its field of view. Based on these images, the positional relationship between the driver and the camera can be determined, and functions such as driver assistance and fatigue detection can be implemented based on this positional relationship.

[0003] To achieve functions such as driver assistance and fatigue detection, it is also necessary to obtain the positional relationship between a specified plane (such as the plane where the left rearview mirror is located, the plane where the right rearview mirror is located, etc.) and the driver. However, there is no reasonable way to obtain the positional relationship between the specified plane and the driver.

[0004] For example, an onboard camera captures an image of a designated plane. Based on this image, the positional relationship between the designated plane and the onboard camera is determined. Then, based on the positional relationships between the driver and the onboard camera, and between the designated plane and the onboard camera, the positional relationship between the designated plane and the driver is determined. However, if the designated plane is not within the field of view of the onboard camera, an image of the designated plane cannot be captured, and therefore the positional relationship between the designated plane and the onboard camera cannot be determined. Consequently, the positional relationship between the designated plane and the driver cannot be obtained, and functions such as driver assistance and fatigue detection cannot be implemented. Summary of the Invention

[0005] This application provides an external parameter calibration method, wherein an auxiliary camera and an auxiliary target are deployed externally on the vehicle to be tested, the vehicle to be tested includes an on-board camera, and the target plane of the vehicle to be tested includes the target to be tested. The method includes:

[0006] The vehicle-mounted camera acquires a first reference image of the auxiliary target, and the first pose of the auxiliary target to the vehicle-mounted camera is determined based on the first reference image.

[0007] The auxiliary camera acquires a second reference image of the target to be detected, and the second pose of the plane to be detected to the auxiliary camera is determined based on the second reference image.

[0008] Based on the first pose and the acquired third pose of the auxiliary target to the auxiliary camera, the fourth pose of the auxiliary camera to the vehicle camera is determined.

[0009] The target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera are determined based on the fourth pose and the second pose.

[0010] This application provides an external parameter calibration device. An auxiliary camera and an auxiliary target are deployed externally on the vehicle under test. The vehicle under test includes an onboard camera, and the target plane of the vehicle under test includes the target. The device includes:

[0011] The acquisition module is used to acquire a first reference image of the auxiliary target through the vehicle-mounted camera, and to acquire a second reference image of the target to be detected through the auxiliary camera;

[0012] The determining module is configured to determine, based on the first reference image, a first pose of the auxiliary target rotating to the vehicle-mounted camera; based on the second reference image, a second pose of the plane to be detected rotating to the auxiliary camera; and based on the first pose and the acquired third pose of the auxiliary target rotating to the auxiliary camera, a fourth pose of the auxiliary camera rotating to the vehicle-mounted camera.

[0013] The processing module is used to determine the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose.

[0014] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the external parameter calibration method of the above example of this application.

[0015] This application provides a vehicle to be tested, which has an auxiliary camera and an auxiliary target deployed on its exterior. The vehicle to be tested includes an onboard camera, and the plane to be tested of the vehicle includes the target. The onboard camera is calibrated using the extrinsic calibration method described in the above example of this application.

[0016] As can be seen from the above technical solutions, in this embodiment, an auxiliary camera and an auxiliary target can be deployed in the test scene. The extrinsic parameters of the vehicle-mounted camera are calibrated based on the auxiliary camera and the auxiliary target, that is, the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera are calibrated. Based on these target extrinsic parameters, the positional relationship between the plane to be detected and the vehicle-mounted camera can be determined. Then, based on the positional relationship between the driver and the vehicle-mounted camera, and the positional relationship between the plane to be detected and the vehicle-mounted camera, the positional relationship between the plane to be detected and the driver can be determined, thereby realizing functions such as assisted driving and fatigue detection. The relative pose between the vehicle-mounted camera and the auxiliary camera is determined by the auxiliary camera and the auxiliary target. By photographing the plane to be detected by the auxiliary camera, the relative poses between the vehicle-mounted camera, the auxiliary camera, and the plane to be detected can be obtained. The extrinsic parameters of the vehicle-mounted camera are calibrated based on these relative poses. Only a calibration tool with fixed positions for the auxiliary camera and the auxiliary target needs to be made; the position between the auxiliary camera and the vehicle-mounted camera does not need to be fixed, which has the advantages of flexible implementation and higher calibration accuracy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0018] Figure 1 This is a flowchart illustrating an external parameter calibration method in one embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of the in-cabin calibration device in one embodiment of this application;

[0020] Figure 3 This is a schematic diagram showing the positional relationship between the auxiliary camera and the vehicle-mounted camera in one embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating an external parameter calibration method in one embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the external parameter calibration device in one embodiment of this application;

[0023] Figure 6 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0024] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0025] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0026] This application proposes an extrinsic parameter calibration method. The test scene is equipped with an auxiliary camera and an auxiliary target. The vehicle to be tested (each vehicle whose extrinsic parameters need calibration can be referred to as the test vehicle) is moved to the test scene (i.e., the external environment of the test vehicle is equipped with an auxiliary camera and an auxiliary target). The test vehicle includes an onboard camera, and the test plane of the test vehicle includes the test target. See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of the external parameter calibration method, which may include:

[0027] Step 101: Acquire a first reference image of the auxiliary target using an onboard camera, and determine the first pose of the auxiliary target as it rotates to the onboard camera based on the first reference image. The first pose may include a first rotation matrix and a first translation matrix. The first rotation matrix reflects the attitude shift of the auxiliary target as it rotates to the onboard camera, and the first translation matrix reflects the position shift of the auxiliary target as it rotates to the onboard camera.

[0028] Step 102: Acquire a second reference image of the target to be detected using an auxiliary camera, and determine the second pose of the plane to be detected from the auxiliary camera based on the second reference image. The second pose may include a second rotation matrix and a second translation matrix. The second rotation matrix reflects the attitude shift of the plane to be detected from the auxiliary camera, and the second translation matrix reflects the position shift of the plane to be detected from the auxiliary camera.

[0029] Step 103: Based on the first pose and the acquired third pose of the auxiliary target turning to the auxiliary camera, determine the fourth pose of the auxiliary camera turning to the vehicle-mounted camera. The third pose may include a third rotation matrix and a third translation matrix. The third rotation matrix reflects the attitude shift of the auxiliary target turning to the auxiliary camera, and the third translation matrix reflects the position shift of the auxiliary target turning to the auxiliary camera. Similarly, the fourth pose may include a fourth rotation matrix and a fourth translation matrix. The fourth rotation matrix reflects the attitude shift of the auxiliary camera turning to the vehicle-mounted camera, and the fourth translation matrix reflects the position shift of the auxiliary camera turning to the vehicle-mounted camera.

[0030] Step 104: Determine the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle camera based on the fourth pose and the second pose. After obtaining the target extrinsic parameters, the target extrinsic parameters can be stored, that is, the target extrinsic parameters are calibrated for the vehicle camera. The target extrinsic parameters represent the extrinsic parameters of the plane to be detected in the camera coordinate system.

[0031] For example, determining the fourth pose of the auxiliary camera from the vehicle-mounted camera based on the first pose and the acquired third pose of the auxiliary target to the auxiliary camera may include, but is not limited to: determining the rotation matrix of the auxiliary camera from the auxiliary target based on the third rotation matrix, and determining the translation matrix of the auxiliary camera from the auxiliary target based on the third rotation matrix and the third translation matrix. The fourth rotation matrix is ​​determined based on the rotation matrix of the auxiliary camera from the auxiliary target and the first rotation matrix, and the fourth translation matrix is ​​determined based on the first rotation matrix, the translation matrix of the auxiliary camera from the auxiliary target, and the first translation matrix.

[0032] For example, determining the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose may include, but is not limited to: for a target position point on the plane to be detected, obtaining the first three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected; converting the first three-dimensional coordinates into the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera based on the second pose; converting the second three-dimensional coordinates into the target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle-mounted camera based on the fourth pose; and determining the target extrinsic parameters based on the target three-dimensional coordinates.

[0033] For example, the plane to be detected may include, but is not limited to, at least one of the following: the plane where the display screen is located, the plane where the left rearview mirror is located, the plane where the right rearview mirror is located, the plane where the dashboard is located, and the plane where the windshield is located. The target position point may include, but is not limited to, at least one of the following: the upper left corner position point, the upper right corner position point, the lower right corner position point, the lower left corner position point, and the center position point.

[0034] For example, the vehicle-mounted camera may include, but is not limited to, a near-infrared camera, and the auxiliary camera may include, but is not limited to, a visible light camera. The auxiliary camera may be fixed on an auxiliary bracket, and the auxiliary target may be fixed on the auxiliary bracket. The auxiliary target is rotated to a third pose fixed to the auxiliary camera, the auxiliary camera does not obstruct the auxiliary target, and the distance between the auxiliary camera and the auxiliary target does not exceed the field of view of the vehicle-mounted camera.

[0035] For example, the vehicle-mounted camera may include, but is not limited to, a near-infrared camera, and the auxiliary camera may include, but is not limited to, a visible light camera. The auxiliary camera may be fixed on an auxiliary bracket, and the auxiliary target may be fixed on the auxiliary bracket. The auxiliary target is fixed in a third pose of the auxiliary camera, the auxiliary camera does not obstruct the auxiliary target, and the auxiliary target does not exceed the field of view coverage of the vehicle-mounted camera.

[0036] For example, in the above process, the pose from A to B refers to the pose when the coordinate system of A is transformed into the coordinate system of B. For instance, the first pose of the auxiliary target transforming into the vehicle camera refers to the pose when the coordinate system of the auxiliary target is transformed into the coordinate system of the vehicle camera. The second pose of the plane to be detected transforming into the auxiliary camera refers to the pose when the coordinate system of the plane to be detected is transformed into the coordinate system of the auxiliary camera. The third pose of the auxiliary target transforming into the auxiliary camera refers to the pose when the coordinate system of the auxiliary target is transformed into the coordinate system of the auxiliary camera. The fourth pose of the auxiliary camera transforming into the vehicle camera refers to the pose when the coordinate system of the auxiliary camera is transformed into the coordinate system of the vehicle camera.

[0037] As can be seen from the above technical solutions, in this embodiment, an auxiliary camera and an auxiliary target can be deployed in the test scene. The extrinsic parameters of the vehicle-mounted camera are calibrated based on the auxiliary camera and the auxiliary target, that is, the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera are calibrated. Based on these target extrinsic parameters, the positional relationship between the plane to be detected and the vehicle-mounted camera can be determined. Then, based on the positional relationship between the driver and the vehicle-mounted camera, and the positional relationship between the plane to be detected and the vehicle-mounted camera, the positional relationship between the plane to be detected and the driver can be determined, thereby realizing functions such as assisted driving and fatigue detection. The relative pose between the vehicle-mounted camera and the auxiliary camera is determined by the auxiliary camera and the auxiliary target. By photographing the plane to be detected by the auxiliary camera, the relative poses between the vehicle-mounted camera, the auxiliary camera, and the plane to be detected can be obtained. The extrinsic parameters of the vehicle-mounted camera are calibrated based on these relative poses. Only a calibration tool with fixed positions for the auxiliary camera and the auxiliary target needs to be made; the position between the auxiliary camera and the vehicle-mounted camera does not need to be fixed, which has the advantages of flexible implementation and higher calibration accuracy.

[0038] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0039] Vehicle-mounted cameras (such as camcorders) can be installed on vehicles, for example, inside the vehicle (or outside the vehicle), that is, vehicle-mounted cameras are deployed in the cabin. Images are collected through the vehicle-mounted cameras, and intelligent cabin applications are carried out based on the images, realizing intelligent cabin applications based on machine vision.

[0040] For example, images of the driver can be captured by an in-vehicle camera, meaning the driver is within the field of view of the in-vehicle camera. The in-vehicle camera captures images within the field of view, and the positional relationship between the driver and the in-vehicle camera is determined based on these images. Based on this positional relationship, functions such as assisted driving and fatigue detection can be implemented.

[0041] It can also obtain the positional relationship between a specified plane (such as the plane where the left rearview mirror is located, the plane where the right rearview mirror is located, etc.) and the driver. For example, by acquiring an image of the specified plane through an in-vehicle camera, the positional relationship between the specified plane and the in-vehicle camera can be determined based on the image. Based on the positional relationship between the driver and the in-vehicle camera, and the positional relationship between the specified plane and the in-vehicle camera, the positional relationship between the specified plane and the driver can be determined.

[0042] However, when a vehicle-mounted camera is installed inside the vehicle, if the designated plane is not within the field of view of the vehicle-mounted camera, it is impossible to capture an image of the designated plane through the vehicle-mounted camera. Therefore, it is impossible to know the positional relationship between the designated plane and the vehicle-mounted camera, resulting in the inability to obtain the positional relationship between the designated plane and the driver.

[0043] In response to the above findings, this application proposes an extrinsic parameter calibration method that can calibrate the extrinsic parameters of the vehicle-mounted camera before the vehicle leaves the factory. Specifically, it calibrates the target extrinsic parameters of a specified plane (for ease of distinction, this specified plane is referred to as the plane to be detected) in the camera coordinate system of the vehicle-mounted camera.

[0044] For example, camera parameters may include camera intrinsic parameters, distortion coefficients, and camera extrinsic parameters. The camera intrinsic parameters consist of (cx, cy, fx, fy), where (cx, cy) are the principal point coordinates, (fx, fy) are the camera focal length, and (s0, s1, s2, s3) are the camera distortion coefficients. These intrinsic parameters are related to the camera's inherent characteristics. The calibration process for the camera intrinsic parameters is not limited in this embodiment. The camera extrinsic parameters consist of (ω, δ, θ, Tx, Ty, Tz), where (ω, δ, θ) are the rotation parameters of the three axes, and (Tx, Ty, Tz) are the translation parameters of the three axes. The camera extrinsic parameters are used to represent the transformation relationship between the world coordinate system and the camera coordinate system. In order to calibrate the camera extrinsic parameters, this embodiment proposes an extrinsic parameter calibration method to calibrate the camera extrinsic parameters of the vehicle-mounted camera deployed on the vehicle before the vehicle leaves the factory. For ease of distinction, the vehicle whose camera extrinsic parameters need to be calibrated is referred to as the vehicle to be tested.

[0045] To calibrate camera extrinsics, a calibration site (i.e., a test scenario) needs to be constructed. The test scenario deploys an auxiliary camera and an auxiliary target. The auxiliary camera can be fixed to an auxiliary support, and the auxiliary target can also be fixed to the auxiliary support. Furthermore, the pose of the auxiliary target when it is rotated to the position of the auxiliary camera is fixed; that is, the relative pose relationship between the auxiliary camera and the auxiliary target is fixed. See also... Figure 2 The diagram shows the structural relationship between the auxiliary camera, auxiliary target, and auxiliary support. The auxiliary camera, auxiliary target, and auxiliary support can be fixed into an in-vehicle calibration device, which is used to calibrate the external parameters of the vehicle-mounted camera.

[0046] See Figure 2 As shown, the auxiliary camera needs to be fixed on the auxiliary support. The positional relationship between the auxiliary camera and the auxiliary support will not change. This embodiment does not restrict the method of fixing the auxiliary camera, as long as it can be fixed on the auxiliary support. The auxiliary target also needs to be fixed on the auxiliary support. The positional relationship between the auxiliary target and the auxiliary support will not change. This embodiment does not restrict the method of fixing the auxiliary target, as long as it can be fixed on the auxiliary support. Since both the auxiliary camera and the auxiliary target are fixed on the auxiliary support, their relative pose relationship is fixed and will not change.

[0047] Regarding the positional relationship between the auxiliary camera and the auxiliary target, it is necessary to ensure that the auxiliary camera does not obstruct the auxiliary target. That is, when acquiring images of the auxiliary target using the vehicle-mounted camera, the auxiliary camera should not obstruct the target and should not affect the image acquisition process. For example, the auxiliary camera can be fixed to the upper side of the auxiliary target, such as directly above it, or at a position of p centimeters (e.g., 2cm, 3cm) above it. Of course, the auxiliary camera can also be fixed to the left, right, or lower side of the auxiliary target (e.g., directly below it); there are no restrictions on this.

[0048] Regarding the positional relationship between the auxiliary camera and the auxiliary target, in order for the auxiliary camera to acquire an image of the target to be detected, and for the vehicle-mounted camera to acquire an image of the auxiliary target, the distance between the auxiliary camera and the auxiliary target can not exceed the field of view of the vehicle-mounted camera, and there is no restriction on this distance.

[0049] For example, the vehicle-mounted camera may include, but is not limited to, a near-infrared camera, and the auxiliary camera may include, but is not limited to, a visible light camera. The auxiliary camera may be fixed on an auxiliary bracket, and the auxiliary target may be fixed on the auxiliary bracket. The auxiliary target is fixed in a third pose of the auxiliary camera, the auxiliary camera does not obstruct the auxiliary target, and the auxiliary target does not exceed the field of view coverage of the vehicle-mounted camera.

[0050] Provided that the auxiliary camera does not obstruct the auxiliary target, and for ease of implementation, the auxiliary camera can be placed as close to the auxiliary target as possible, thereby reducing the size of the calibration device inside the vehicle cabin.

[0051] For example, the vehicle-mounted camera includes, but is not limited to, a near-infrared camera, and the auxiliary camera includes, but is not limited to, a visible light camera. Taking a near-infrared camera as the vehicle-mounted camera and a visible light camera as the auxiliary camera as an example, since the near-infrared camera (i.e., the vehicle-mounted camera) requires a supplementary light source, while the visible light camera is not sensitive to near-infrared light sources and will not produce overexposure, the visible light camera can be used as an auxiliary camera. In this way, when the image is acquired by the auxiliary camera, even if there is a supplementary light source, the image will not produce overexposure, and the image quality is high. When performing related detection based on the image, more accurate detection results can be obtained.

[0052] By using a visible light camera as an auxiliary camera, and when the vehicle-mounted camera is equipped with a supplementary light for the near-infrared camera, the auxiliary camera uses a visible light camera to avoid the influence of the infrared supplementary light. This method is applicable to near-infrared vehicle-mounted cameras, and the external parameter calibration method can be applied to the visible light-near-infrared band range, making it more universal.

[0053] For example, the auxiliary target can be a checkerboard pattern, or it can be other patterns, such as a circular dot array. There are no restrictions on this, as long as it can distinguish different positions in the image. For instance, in addition to checkerboard patterns and circular dot arrays, the auxiliary target can also be a feature shape with a known relative positional relationship, or a pattern with a known relative positional relationship.

[0054] The size of the auxiliary target needs to be moderate. The target should not be too large, affecting ease of implementation, nor too small, affecting calibration accuracy. For example, an M*N checkerboard pattern can be used, with each square measuring w centimeters. The values ​​of M and N can be configured based on experience, as can the value of w. w can be a positive integer or a decimal, with no restrictions. For instance, an 8*4 checkerboard pattern could be used, with each square measuring 2cm. Of course, the above is just an example of auxiliary target size, and no specific restrictions are imposed on its dimensions.

[0055] The auxiliary camera can be a visible light camera. The optical lens of the visible light camera can be selected according to the size of the auxiliary target. For example, when the auxiliary target is an 8*4 checkerboard pattern, an optical lens with a focal length of about 3.6-16mm can be selected, and a lens with low distortion should be chosen as much as possible to suppress the influence of distortion on calibration. Of course, the above is just an example of selecting an optical lens, and there are no restrictions on the method of selecting an optical lens.

[0056] See Figure 3 As shown, the vehicle to be tested can be moved to the test scene. The vehicle to be tested may include an onboard camera, and the plane to be tested on the vehicle may include a target to be tested. When moving the vehicle to be tested to the test scene, it is necessary to ensure that the auxiliary target is within the field of view of the onboard camera so that the onboard camera can acquire an image of the auxiliary target. Similarly, it is necessary to ensure that the target to be tested is within the field of view of the auxiliary camera so that the auxiliary camera can acquire an image of the target to be tested.

[0057] For example, the vehicle-mounted camera can be deployed inside or outside the vehicle to be inspected; the following example focuses on deployment inside the vehicle. For instance, possible installation locations include, but are not limited to: the steering column, around the rearview mirror, around the center console screen, and the left A-pillar. These are just a few examples of possible installation locations, and no restrictions are placed on the placement. For example, the vehicle-mounted camera can be deployed on the steering column to capture images of the driver's face from the front, with the camera's field of view covering the area around the driver's seat.

[0058] For example, the plane to be detected includes, but is not limited to, at least one of the following: the plane where the display screen is located, the plane where the left rearview mirror is located, the plane where the right rearview mirror is located, the plane where the dashboard is located, and the plane where the windshield is located. Of course, the above are just examples of the planes to be detected and are not intended to be limiting. For instance, the plane where the display screen is located can be used as the plane to be detected, and the target extrinsic parameters of the plane where the display screen is located in the camera coordinate system of the vehicle-mounted camera need to be calibrated. The plane where the left rearview mirror is located can be used as the plane to be detected, and the target extrinsic parameters of the plane where the left rearview mirror is located in the camera coordinate system of the vehicle-mounted camera need to be calibrated. The plane where the right rearview mirror is located can be used as the plane to be detected, and the target extrinsic parameters of the plane where the right rearview mirror is located in the camera coordinate system of the vehicle-mounted camera need to be calibrated. The plane where the dashboard is located can be used as the plane to be detected, and the target extrinsic parameters of the plane where the dashboard is located in the camera coordinate system of the vehicle-mounted camera need to be calibrated. The plane where the windshield is located can be used as the plane to be detected, and the target extrinsic parameters of the plane where the windshield is located in the camera coordinate system of the vehicle-mounted camera need to be calibrated.

[0059] When deploying an automotive camera on the steering column, the relative poses of the display screen, left rearview mirror, right rearview mirror, dashboard, and windshield are all outside the camera's field of view, making direct calibration difficult. Therefore, an auxiliary camera and an auxiliary target are introduced to calibrate the target extrinsic parameters of the plane to be detected in the camera coordinate system of the automotive camera. See [link to relevant documentation] Figure 3 As shown, the positional relationship between the auxiliary camera and the vehicle-mounted camera does not need to be fixed, that is, the positional relationship is not restricted.

[0060] For example, to calibrate the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera, it is also necessary to place a target on the plane to be detected. The target can completely cover the plane to be detected or partially cover the plane to be detected. Figure 3 The diagram shows three targets to be detected on three planes. These targets can be of a checkerboard pattern, or other patterns such as a circular dot array; there are no restrictions, as long as they can distinguish different positions within the image. For example, besides checkerboard patterns and circular dot arrays, the targets can also be feature shapes with known relative positions, or patterns with known relative positions.

[0061] In the above application scenarios, this application proposes an external parameter calibration method. This method can be applied to calibration devices (i.e., electronic devices), such as PCs (Personal Computers), terminal devices, mobile terminals, laptops, etc. There is no limitation on the type of calibration device. See [link to relevant documentation]. Figure 4 The diagram shown is a flowchart of the external parameter calibration method, which may include:

[0062] Step 401: Obtain the third pose of the auxiliary target as it rotates to the auxiliary camera. The third pose may include a third rotation matrix and a third translation matrix. The third rotation matrix reflects the attitude shift of the auxiliary target as it rotates to the auxiliary camera, and the third translation matrix reflects the position shift of the auxiliary target as it rotates to the auxiliary camera.

[0063] For example, the auxiliary camera is fixed on the auxiliary support, the auxiliary target is fixed on the auxiliary support, and the positional relationship between the auxiliary target and the auxiliary support does not change. Therefore, the third pose of the auxiliary target when it is rotated to the auxiliary camera does not change. The calibration device can store the third pose of the auxiliary target when it is rotated to the auxiliary camera in a specified storage medium. In this way, when calibrating the camera extrinsic parameters of the vehicle camera of each vehicle to be tested, the third pose of the auxiliary target when it is rotated to the auxiliary camera can be obtained from the specified storage medium.

[0064] For example, the third pose of the auxiliary target as it rotates to the auxiliary camera can be accurately obtained through the mechanical structure limiting of the calibration device in the vehicle cabin, and this third pose can be stored in a designated storage medium. Specifically, during the production of the calibration device in the vehicle cabin, the poses of the auxiliary target and the auxiliary camera can be controlled through the mechanical structure according to a specified pose relationship, thus directly obtaining the third pose of the auxiliary target as it rotates to the auxiliary camera.

[0065] For example, a precise indoor calibration method can be used to obtain the third pose of the auxiliary target when it is transferred to the auxiliary camera, and this third pose can be stored in a designated storage medium. For instance, a target X can be placed indoors, and the pose relationship between target X and the auxiliary target is known (by moving target X and the auxiliary target, their pose relationship is controlled). An image of target X is acquired using the auxiliary camera, and the pose relationship between the auxiliary camera and target X is determined based on this image. Based on the pose relationships between the auxiliary camera and target X, and between target X and the auxiliary target, the third pose of the auxiliary target when it is transferred to the auxiliary camera can be determined. The above is just an example of a precise indoor calibration method, and no limitations are imposed on this method.

[0066] For example, since the calibration of the calibration device in the cabin is not limited by space, and the auxiliary camera (i.e., the visible light camera) does not have the problem of overexposure of the light source, the third pose of the auxiliary target to the auxiliary camera can also be obtained by means of mirror calibration, auxiliary target, etc., and there are no restrictions on this calibration method.

[0067] Of course, the above are just a few examples, and there are no restrictions on how the third pose is obtained, as long as the third pose of the auxiliary target transformed into the auxiliary camera can be obtained. After obtaining the third pose of the auxiliary target transformed into the auxiliary camera, the third pose is stored in the designated storage medium. In this way, when calibrating the camera extrinsic parameters of the on-board camera of each vehicle to be detected, the third pose can be obtained from the designated storage medium.

[0068] Step 402: Acquire a first reference image of the auxiliary target using the vehicle-mounted camera, and determine the first pose of the auxiliary target when it turns to the vehicle-mounted camera based on the first reference image. The first pose may include a first rotation matrix and a first translation matrix. The first rotation matrix reflects the attitude shift of the auxiliary target when it turns to the vehicle-mounted camera, and the first translation matrix reflects the position shift of the auxiliary target when it turns to the vehicle-mounted camera.

[0069] For example, the calibration device can send an image acquisition command to the vehicle-mounted camera. The image acquisition command is used to instruct the vehicle-mounted camera to acquire images within its field of view. Since the auxiliary target is located within the field of view of the vehicle-mounted camera, the vehicle-mounted camera can acquire an image of the auxiliary target. For ease of distinction, this image is referred to as the first reference image. After the vehicle-mounted camera acquires the first reference image of the auxiliary target, it sends the first reference image to the calibration device. In this way, the calibration device can obtain the first reference image.

[0070] For example, multiple key points can be selected from an auxiliary target. If the auxiliary target is a checkerboard pattern, the intersection of the black and white areas can be used as the region of interest (ROI), and specific locations within the ROI (such as the top left, top right, bottom left, bottom right, or center point) can be used as key points. When the checkerboard pattern includes multiple ROIs, multiple key points can be selected from the auxiliary target. Of course, other locations can also be used as key points; there are no restrictions on this.

[0071] For each keypoint, its physical coordinates in the coordinate system of the auxiliary target can be determined. Based on the first reference image, the pixel coordinates of the keypoint in the first reference image can be determined. The physical coordinates and the pixel coordinates can form a pixel pair, which includes both the physical coordinates and the pixel coordinates. Obviously, multiple keypoints can correspond to multiple pixel pairs. Based on multiple pixel pairs, the first pose of the auxiliary target when it turns to the vehicle-mounted camera can be determined. There are no restrictions on the method of determining this first pose.

[0072] In one possible implementation, the transformation relationship between the coordinate system of the auxiliary target and the camera coordinate system of the vehicle-mounted camera can be found in formula (1). Of course, formula (1) is just an example.

[0073]

[0074] In formula (1), R represents the 3*3 rotation matrix, which can be converted from (ω, δ, θ), i.e., the rotation matrix between the coordinate system of the auxiliary target and the camera coordinate system of the vehicle-mounted camera. This rotation matrix is ​​also the first rotation matrix for the auxiliary target to rotate to the vehicle-mounted camera. T represents the translation parameter, which consists of (Tx, Ty, Tz), i.e., the translation matrix between the coordinate system of the auxiliary target and the camera coordinate system of the vehicle-mounted camera. This translation matrix is ​​also the first translation matrix for the auxiliary target to rotate to the vehicle-mounted camera. Obviously, the first rotation matrix R and the first translation matrix T can form the first pose of the auxiliary target rotating to the vehicle-mounted camera. Wherein, ω represents the rotation angle around the X-axis of the camera coordinate system, δ represents the rotation angle around the Y-axis of the camera coordinate system, θ represents the rotation angle around the Z-axis of the camera coordinate system, Tx represents the translation in the X-axis direction of the camera coordinate system, Ty represents the translation in the Y-axis direction of the camera coordinate system, and Tz represents the translation in the Z-axis direction of the camera coordinate system.

[0075] In formula (1), P w P represents the physical coordinates in the coordinate system of the auxiliary target. cThe first reference image represents the pixel coordinates, which are the pixel coordinates in the camera coordinate system of the vehicle-mounted camera. Based on the physical coordinates of multiple key points in the coordinate system of the auxiliary target and the pixel coordinates of multiple key points in the first reference image, the first rotation matrix R and the first translation matrix T can be solved by formula (1). The first rotation matrix R and the first translation matrix T can form the first pose of the auxiliary target rotating to the vehicle-mounted camera.

[0076] For example, the process of solving the first pose of the auxiliary target to the vehicle-mounted camera can be a pose solution within the field of view, which can be obtained from the first reference image of the auxiliary target (such as a checkerboard pattern, circular dot array, etc.). The acquisition method is the same as described above and will not be repeated here. For instance, when obtaining the first pose of the auxiliary target to the vehicle-mounted camera, it can be solved using the solvePNP function of OpenCV, and there are no restrictions on this solution process. After obtaining the first pose of the auxiliary target to the vehicle-mounted camera, the accuracy of the first pose can be verified by the reprojection error. The reprojection error can be required to be better than 0.3 pixels. If the accuracy of the first pose meets the requirements, the first pose is output. If the accuracy of the first pose does not meet the requirements, the first pose can be re-obtained. The method for obtaining the first pose is described in step 402.

[0077] Step 403: Acquire a second reference image of the target to be detected using an auxiliary camera, and determine the second pose of the plane to be detected from the auxiliary camera based on the second reference image. The second pose may include a second rotation matrix and a second translation matrix. The second rotation matrix reflects the attitude shift of the plane to be detected from the auxiliary camera, and the second translation matrix reflects the position shift of the plane to be detected from the auxiliary camera.

[0078] For example, the calibration device can send an image acquisition command to the auxiliary camera. The image acquisition command is used to instruct the auxiliary camera to acquire images within its field of view. Since the target to be detected is located within the field of view of the auxiliary camera, the auxiliary camera can acquire an image of the target to be detected. For ease of distinction, this image is referred to as the second reference image. After the auxiliary camera acquires the second reference image of the target to be detected, it sends the second reference image to the calibration device. In this way, the calibration device can obtain the second reference image.

[0079] Multiple key points can be selected from the target to be detected. For example, when the target is a checkerboard pattern, the intersection of the black and white areas can be taken as the region of interest (ROI), and specific positions within the ROI (such as the top left corner, top right corner, bottom left corner, bottom right corner, or center point) can be taken as key points. When the checkerboard pattern includes multiple ROIs, multiple key points can be selected from the target. Of course, other positions can also be taken as key points; there are no restrictions on this.

[0080] For each keypoint, its physical coordinates in the coordinate system of the target (i.e., the coordinate system of the plane to be detected) can be determined. Based on the second reference image, the pixel coordinates of the keypoint in the second reference image can be determined. These physical coordinates and pixel coordinates constitute a pixel pair, which includes both the physical coordinates and the pixel coordinates. Clearly, multiple keypoints can correspond to multiple pixel pairs. Based on these multiple pixel pairs, the second pose of the plane to be detected, transformed to the auxiliary camera, can be determined.

[0081] In one possible implementation, the transformation relationship between the coordinate system of the plane to be detected and the camera coordinate system of the auxiliary camera can be found in Equation (1), although Equation (1) is merely an example. In Equation (1), R represents the rotation matrix between the coordinate system of the plane to be detected and the camera coordinate system of the auxiliary camera, i.e., the second rotation matrix from the plane to the auxiliary camera, consisting of (ω, δ, θ). T represents the translation matrix between the coordinate system of the plane to be detected and the camera coordinate system of the auxiliary camera, i.e., the second translation matrix from the plane to the auxiliary camera, consisting of (Tx, Ty, Tz). Obviously, the second rotation matrix R and the second translation matrix T can form the second pose of the plane to be detected from the auxiliary camera.

[0082] In formula (1), P w P represents the physical coordinates of the plane to be detected in the coordinate system. c The pixel coordinates in the second reference image are represented by the pixel coordinates in the camera coordinate system of the auxiliary camera. Based on the physical coordinates of multiple key points in the coordinate system of the plane to be detected and the pixel coordinates of multiple key points in the second reference image, the second rotation matrix R and the second translation matrix T can be solved by formula (1). The second rotation matrix R and the second translation matrix T can form the second pose of the plane to be detected to the auxiliary camera.

[0083] For example, the process of solving for the second pose of the plane to be detected from the auxiliary camera can be done by solving for the pose within the field of view, which can be obtained from a second reference image of the target to be detected (such as a checkerboard pattern, a circular dot array, etc.). When obtaining the second pose of the plane to be detected from the auxiliary camera, the solvePNP function of OpenCV can be used. After obtaining the second pose of the plane to be detected from the auxiliary camera, the accuracy of the second pose can be verified by the reprojection error. It can also be required that the reprojection error is better than 0.3 pixels (0.3 pixels is just an example, and this threshold is not limited). If the accuracy of the second pose meets the requirements, the second pose is output; if the accuracy of the second pose does not meet the requirements, the second pose can be obtained again.

[0084] Step 404: Determine the fourth pose of the auxiliary camera from the vehicle-mounted camera based on the first pose of the auxiliary target from the vehicle-mounted camera and the third pose of the auxiliary target from the vehicle-mounted camera. The fourth pose includes a fourth rotation matrix and a fourth translation matrix. The fourth rotation matrix reflects the attitude shift of the auxiliary camera from the vehicle-mounted camera, and the fourth translation matrix reflects the position shift of the auxiliary camera from the vehicle-mounted camera.

[0085] For example, since the first pose represents the pose of the auxiliary target turning to the vehicle camera, and the third pose represents the pose of the auxiliary target turning to the auxiliary camera, that is, both the first pose and the third pose are poses related to the auxiliary target. In addition to the auxiliary target, the remaining poses are the vehicle camera and the auxiliary camera. Therefore, the fourth pose of the auxiliary camera turning to the vehicle camera can be determined based on the first pose and the third pose. There is no restriction on the method of determination, as long as the fourth pose can be determined based on the first pose and the third pose.

[0086] In one possible implementation, the fourth pose can be determined using the following steps:

[0087] Step 4041: Determine the rotation matrix of the auxiliary camera to the auxiliary target based on the third rotation matrix, and determine the translation matrix of the auxiliary camera to the auxiliary target based on the third rotation matrix and the third translation matrix.

[0088] For example, the third rotation matrix represents the rotation matrix from the auxiliary target to the auxiliary camera, and the third translation matrix represents the translation matrix from the auxiliary target to the auxiliary camera. The rotation matrix from the auxiliary camera to the auxiliary target can be determined based on the third rotation matrix, and the translation matrix from the auxiliary camera to the auxiliary target can be determined based on the third rotation matrix and the third translation matrix. For example, the rotation matrix and translation matrix from the auxiliary camera to the auxiliary target can be determined using formula (2). Of course, formula (2) is just an example and is not a limitation.

[0089]

[0090] In formula (2), R ACS2Cam_A R can be used to represent the third rotation matrix for the auxiliary target to be rotated to the auxiliary camera. Cam_A2ACS It can be used to represent the rotation matrix T from the auxiliary camera to the auxiliary target. ACS2Cam_A This can be used to represent the third translation matrix, T, from the auxiliary target to the auxiliary camera. Cam_A2ACS It can be used to represent the translation matrix when an auxiliary camera rotates to an auxiliary target. R ACS2Cam_A T It can be used to represent the transpose operation of the third rotation matrix. In formula (2), Cam_A represents the auxiliary camera, ACS represents the auxiliary target, and 2 represents the rotation to.

[0091] Step 4042: Determine the fourth rotation matrix based on the rotation matrix of the auxiliary camera to the auxiliary target and the first rotation matrix of the auxiliary target to the vehicle camera; determine the fourth translation matrix based on the first rotation matrix, the translation matrix of the auxiliary camera to the auxiliary target and the first translation matrix of the auxiliary target to the vehicle camera.

[0092] For example, the first rotation matrix represents the rotation matrix from the auxiliary target to the vehicle-mounted camera, the fourth rotation matrix represents the rotation matrix from the auxiliary camera to the vehicle-mounted camera, the first translation matrix represents the translation matrix from the auxiliary target to the vehicle-mounted camera, and the fourth translation matrix represents the translation matrix from the auxiliary camera to the vehicle-mounted camera. The fourth rotation matrix can be determined based on the rotation matrix from the auxiliary camera to the auxiliary target and the first rotation matrix, and the fourth translation matrix can be determined based on the first rotation matrix, the translation matrix from the auxiliary camera to the auxiliary target, and the first translation matrix. For example, the fourth rotation matrix and the fourth translation matrix can be determined using formula (3). Of course, formula (3) is just an example and is not a limitation.

[0093]

[0094] In formula (3), R ACS2Cam_C R is used to represent the first rotation matrix for the auxiliary target to rotate to the vehicle-mounted camera. Cam_A2ACS R is the rotation matrix used to represent the rotation of the auxiliary camera to the auxiliary target. Cam_A2Cam_C The fourth rotation matrix, T, is used to represent the rotation from the auxiliary camera to the vehicle-mounted camera. Cam_A2ACS T is used to represent the translation matrix used to rotate the auxiliary camera to the auxiliary target. ACS2Cam_C T is used to represent the first translation matrix for the auxiliary target to be transferred to the vehicle-mounted camera. Cam_A2Cam_C The fourth translation matrix is ​​used to represent the transition from the auxiliary camera to the vehicle-mounted camera. In formula (3), Cam_A represents the auxiliary camera, ACS represents the auxiliary target, Cam_C represents the vehicle-mounted camera, and 2 represents the transition to.

[0095] At this point, step 404 is complete, and the fourth pose of the auxiliary camera being transferred to the vehicle-mounted camera is obtained.

[0096] Step 405: Determine the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle camera based on the fourth pose of the auxiliary camera and the second pose of the plane to be detected.

[0097] For example, since the fourth pose represents the pose of the auxiliary camera turning to the vehicle camera, and the second pose represents the pose of the plane to be detected turning to the auxiliary camera, that is, both the fourth pose and the second pose are poses related to the auxiliary camera. In addition to the auxiliary camera, the remaining poses are the vehicle camera and the plane to be detected. Therefore, the pose relationship between the plane to be detected and the vehicle camera can be determined based on the fourth pose and the second pose. That is, the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle camera can be determined based on the fourth pose and the second pose. There is no restriction on the method of determination, as long as the target extrinsic parameters can be determined based on the fourth pose and the second pose.

[0098] In one possible implementation, the target extrinsic parameters can be determined using the following steps:

[0099] Step 4051: For the target location point on the plane to be detected, obtain the first three-dimensional coordinates of the target location point in the coordinate system of the plane to be detected. The target location point may include, but is not limited to, at least one of the following: top-left corner, top-right corner, bottom-right corner, bottom-left corner, and center point. For example, the target location point may simultaneously include the top-left corner, top-right corner, bottom-right corner, and bottom-left corner points on the plane to be detected. Of course, the above are just a few examples of target location points, and there is no restriction on the target location point; it can be any location point on the plane to be detected.

[0100] For example, a coordinate system can be established on the plane to be detected. This coordinate system is the coordinate system of the plane to be detected. Since the target position point is a position point on the plane to be detected, the position of the target position point represents the three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected. For ease of distinction, this three-dimensional coordinate is called the first three-dimensional coordinate. The first three-dimensional coordinate of the target position point in the coordinate system of the plane to be detected can be pre-configured or user-inputted, without restriction.

[0101] Step 4052: Based on the second pose, convert the first three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected into the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera.

[0102] For example, the second pose is the pose of the plane to be detected to the auxiliary camera. That is, the second pose represents the mapping relationship between the coordinate system of the plane to be detected and the camera coordinate system of the auxiliary camera. Given the first three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected, the first three-dimensional coordinates can be converted into the second three-dimensional coordinates in the camera coordinate system of the auxiliary camera based on the second pose.

[0103] For example, the mapping relationship between the coordinate system of the plane to be detected and the camera coordinate system of the auxiliary camera can be seen in formula (1), where R represents the second rotation matrix and T represents the second translation matrix. Based on the second pose, the second rotation matrix R and the second translation matrix T can be obtained, that is, the second rotation matrix R and the second translation matrix T are known values. Based on this, after substituting the first three-dimensional coordinates into formula (1), the second three-dimensional coordinates can be obtained. Of course, formula (1) is just an example and does not limit this mapping relationship.

[0104] For example, based on the second pose and the first three-dimensional coordinates, the PNP method can be used to solve for the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera, and there are no restrictions on this process.

[0105] Step 4053: Based on the fourth pose of the auxiliary camera turning to the vehicle camera, convert the second three-dimensional coordinates into the target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle camera.

[0106] For example, the fourth pose is the pose of the auxiliary camera when it is turned to the vehicle camera. That is, the fourth pose represents the mapping relationship between the camera coordinate system of the auxiliary camera and the camera coordinate system of the vehicle camera. Given the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera, the second three-dimensional coordinates can be converted into the target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle camera based on the fourth pose.

[0107] For example, the mapping relationship between the camera coordinate system of the auxiliary camera and the camera coordinate system of the vehicle camera can be seen in formula (4). Of course, formula (4) is just an example and does not limit this mapping relationship.

[0108]

[0109] In formula (4), R Cam_A2Cam_C The fourth rotation matrix, T, is used to represent the rotation from the auxiliary camera to the vehicle-mounted camera. Cam_A2Cam_C The fourth translation matrix is ​​used to represent the transition from the auxiliary camera to the vehicle camera. The fourth rotation matrix and the fourth translation matrix can be obtained based on the fourth pose, meaning that the fourth rotation matrix and the fourth translation matrix are known values.

[0110] In formula (4), (x0,y0,z0) represents the second three-dimensional coordinates, and (x1,y1,z1) represents the target three-dimensional coordinates. Obviously, after substituting the second three-dimensional coordinates (x0,y0,z0) into formula (4), the target three-dimensional coordinates (x1,y1,z1) can be obtained. That is, the second three-dimensional coordinates are converted into the target three-dimensional coordinates through formula (4).

[0111] Step 4054: Determine the target extrinsic parameters based on the target's three-dimensional coordinates. For example, the target's three-dimensional coordinates can be used as the target extrinsic parameters, meaning the target extrinsic parameters include the target's three-dimensional coordinates in the camera coordinate system of the vehicle-mounted camera. For instance, the target extrinsic parameters might include the target's three-dimensional coordinates in the camera coordinate system of the upper left corner of the plane to be detected, the target's three-dimensional coordinates in the camera coordinate system of the upper right corner of the plane to be detected, the target's three-dimensional coordinates in the camera coordinate system of the lower right corner of the plane to be detected, and the target's three-dimensional coordinates in the camera coordinate system of the lower left corner of the plane to be detected. Of course, the above are just examples of target extrinsic parameters, and there are no limitations on these parameters.

[0112] Step 406: Store the target extrinsic parameters, i.e. calibrate the target extrinsic parameters. For example, the calibration device can send the target extrinsic parameters to the vehicle under test, which then stores them to obtain the precise coordinates of the target position point on the plane under test in the camera coordinate system of the vehicle-mounted camera. These precise coordinates also reflect the relative pose relationship between the plane under test and the camera coordinate system of the vehicle-mounted camera.

[0113] As can be seen from the above technical solutions, in this embodiment, an auxiliary camera and an auxiliary target can be deployed in the test scene. The extrinsic parameters of the vehicle-mounted camera are calibrated based on the auxiliary camera and the auxiliary target, that is, the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera are calibrated. Based on these target extrinsic parameters, the positional relationship between the plane to be detected and the vehicle-mounted camera can be known, thereby realizing functions such as assisted driving and fatigue detection. The relative pose between the vehicle-mounted camera and the auxiliary camera is determined by the auxiliary camera and the auxiliary target. By photographing the plane to be detected by the auxiliary camera, the relative poses between the vehicle-mounted camera, the auxiliary camera, and the plane to be detected can be obtained. The extrinsic parameters of the vehicle-mounted camera are calibrated based on these relative poses. Only a calibration tool with fixed positions for the auxiliary camera and the auxiliary target needs to be made; the position between the auxiliary camera and the vehicle-mounted camera does not need to be fixed, offering advantages such as flexible implementation and higher calibration accuracy. For scenarios where near-infrared cameras require supplementary lighting, there will be no overexposure of the light source during imaging. It is applicable to the visible light-near-infrared band and has wide applicability.

[0114] Based on the same application concept as the above method, this application proposes an external parameter calibration device. An auxiliary camera and an auxiliary target are deployed externally on the vehicle to be tested. The vehicle to be tested includes an onboard camera, and the plane to be tested of the vehicle includes the target. For example, an auxiliary camera and an auxiliary target are deployed in a test scene. The vehicle to be tested is moved into the test scene. The vehicle to be tested includes an onboard camera, and the plane to be tested of the vehicle includes the target. See [link to relevant documentation]. Figure 5 The diagram shown is a structural schematic of the external parameter calibration device, which may include:

[0115] The acquisition module 51 is used to acquire a first reference image of the auxiliary target through the vehicle-mounted camera, and to acquire a second reference image of the target to be detected through the auxiliary camera.

[0116] The determining module 52 is used to determine the first pose of the auxiliary target to the vehicle-mounted camera based on the first reference image; to determine the second pose of the plane to be detected to the auxiliary camera based on the second reference image; and to determine the fourth pose of the auxiliary camera to the vehicle-mounted camera based on the first pose and the acquired third pose of the auxiliary target to the auxiliary camera.

[0117] Processing module 53 is used to determine the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose.

[0118] For example, the first pose includes a first rotation matrix and a first translation matrix, the third pose includes a third rotation matrix and a third translation matrix, and the fourth pose includes a fourth rotation matrix and a fourth translation matrix; when the determining module 52 determines the fourth pose of the auxiliary camera from the vehicle camera based on the first pose and the acquired third pose of the auxiliary target to the auxiliary camera, it is specifically used to: determine the rotation matrix of the auxiliary camera from the auxiliary target based on the third rotation matrix; determine the translation matrix of the auxiliary camera from the auxiliary target based on the third rotation matrix and the third translation matrix; determine the fourth rotation matrix based on the rotation matrix of the auxiliary camera from the auxiliary target and the first rotation matrix; and determine the fourth translation matrix based on the first rotation matrix, the translation matrix of the auxiliary camera from the auxiliary target and the first translation matrix.

[0119] For example, the determining module 52 uses the following formula to determine the rotation matrix and translation matrix of the auxiliary camera turning to the auxiliary target: The fourth rotation matrix and the fourth translation matrix are determined using the following formulas: Among them, R ACS2Cam_A Let R represent the third rotation matrix. Cam_A2ACS T represents the rotation matrix from the auxiliary camera to the auxiliary target. ACS2Cam_A T represents the third translation matrix. Cam_A2ACS R represents the translation matrix from the auxiliary camera to the auxiliary target. ACS2Cam_C Let R represent the first rotation matrix. Cam_A2Cam_C Let T represent the fourth rotation matrix. ACS2Cam_C Let T represent the first translation matrix. Cam_A2Cam_C This represents the fourth translation matrix.

[0120] For example, when the processing module 53 determines the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose, it is specifically used to: for a target position point on the plane to be detected, obtain the first three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected, and convert the first three-dimensional coordinates into the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera based on the second pose; convert the second three-dimensional coordinates into the target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle-mounted camera based on the fourth pose, and determine the target extrinsic parameters based on the target three-dimensional coordinates.

[0121] For example, the fourth pose includes a fourth rotation matrix and a fourth translation matrix. When the processing module 53 converts the second three-dimensional coordinates into target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle-mounted camera based on the fourth pose, it specifically uses the following formula to convert the second three-dimensional coordinates into the target three-dimensional coordinates: Among them, R Cam_A2Cam_ T is used to represent the fourth rotation matrix. Cam_A2Cam_C The fourth translation matrix is ​​used to represent the second three-dimensional coordinates, and (x0, y0, z0) is used to represent the target three-dimensional coordinates.

[0122] For example, the plane to be detected includes at least one of the following: the plane where the display screen is located, the plane where the left rearview mirror is located, the plane where the right rearview mirror is located, the plane where the dashboard is located, and the plane where the windshield is located.

[0123] For example, the target location point includes at least one of the following: the upper left corner location point, the upper right corner location point, the lower right corner location point, the lower left corner location point, and the center location point.

[0124] For example, the vehicle-mounted camera is a near-infrared camera, and the auxiliary camera is a visible light camera; the auxiliary camera is fixed on an auxiliary bracket, the auxiliary target is fixed on the auxiliary bracket, the auxiliary target is rotated to the third pose of the auxiliary camera and fixed, the auxiliary camera does not obstruct the auxiliary target, and the distance between the auxiliary camera and the auxiliary target does not exceed the field of view of the vehicle-mounted camera.

[0125] Based on the same application concept as the above method, this application proposes an electronic device (such as a calibration device), see [link to relevant documentation]. Figure 6 As shown, the electronic device includes a processor 61 and a machine-readable storage medium 62, the machine-readable storage medium 62 storing machine-executable instructions that can be executed by the processor 61; the processor 61 is used to execute the machine-executable instructions to implement the external parameter calibration method disclosed in the above example of this application.

[0126] Based on the same application concept as the above method, this application proposes a vehicle to be tested, wherein an auxiliary camera and an auxiliary target are deployed on the outside of the vehicle to be tested, the vehicle to be tested includes an on-board camera, and the plane to be tested of the vehicle to be tested includes the target to be tested; wherein the on-board camera is calibrated using the external parameter calibration method disclosed in the above example of this application.

[0127] For example, the vehicle-mounted camera is a near-infrared camera, and the auxiliary camera is a visible light camera;

[0128] The auxiliary camera is fixed on the auxiliary bracket, and the auxiliary target is fixed on the auxiliary bracket. The auxiliary target is rotated to the third pose of the auxiliary camera and fixed. The auxiliary camera does not obstruct the auxiliary target. The distance between the auxiliary camera and the auxiliary target does not exceed the field of view of the vehicle camera.

[0129] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the external parameter calibration method disclosed in the above examples of this application.

[0130] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0131] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0132] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for calibrating external parameters, characterized in that, An auxiliary camera and an auxiliary target are deployed externally on the vehicle to be inspected. The vehicle to be inspected includes an onboard camera, and the target plane of the vehicle to be inspected includes the target. The method includes: The vehicle-mounted camera acquires a first reference image of the auxiliary target, and the first pose of the auxiliary target to the vehicle-mounted camera is determined based on the first reference image. The auxiliary camera acquires a second reference image of the target to be detected, and the second pose of the plane to be detected to the auxiliary camera is determined based on the second reference image. Based on the first pose and the acquired third pose of the auxiliary target to the auxiliary camera, the fourth pose of the auxiliary camera to the vehicle camera is determined. Based on the fourth pose and the second pose, the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera are determined; wherein, for a target position point on the plane to be detected, the first three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected are obtained, and the first three-dimensional coordinates are converted into the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera based on the second pose; the second three-dimensional coordinates are converted into the target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle-mounted camera based on the fourth pose, and the target three-dimensional coordinates are used as the target extrinsic parameters.

2. The method according to claim 1, characterized in that, The first pose includes a first rotation matrix and a first translation matrix, the third pose includes a third rotation matrix and a third translation matrix, and the fourth pose includes a fourth rotation matrix and a fourth translation matrix; The step of determining the fourth pose of the auxiliary camera from the vehicle-mounted camera based on the first pose and the acquired third pose of the auxiliary target to the auxiliary camera includes: The rotation matrix for the auxiliary camera to rotate to the auxiliary target is determined based on the third rotation matrix, and the translation matrix for the auxiliary camera to rotate to the auxiliary target is determined based on the third rotation matrix and the third translation matrix. The fourth rotation matrix is ​​determined based on the rotation matrix of the auxiliary camera to the auxiliary target and the first rotation matrix, and the fourth translation matrix is ​​determined based on the first rotation matrix, the translation matrix of the auxiliary camera to the auxiliary target and the first translation matrix.

3. The method according to claim 2, characterized in that, The rotation and translation matrices for the auxiliary camera to rotate to the auxiliary target are determined using the following formulas: ； The fourth rotation matrix and the fourth translation matrix are determined using the following formulas: ； in, Represents the third rotation matrix. This represents the rotation matrix from which the auxiliary camera rotates to the auxiliary target. This represents the third translation matrix. This represents the translation matrix by which the auxiliary camera rotates to the auxiliary target. Denotes the first rotation matrix. This represents the fourth rotation matrix. Denotes the first translation matrix. This represents the fourth translation matrix.

4. The method according to claim 1, characterized in that, The fourth pose includes a fourth rotation matrix and a fourth translation matrix. The step of converting the second three-dimensional coordinates based on the fourth pose into target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle-mounted camera includes: The second three-dimensional coordinates are converted to the target three-dimensional coordinates using the following formula: ； in, This represents the fourth rotation matrix. This represents the fourth translation matrix. Indicates the second three-dimensional coordinates. This represents the three-dimensional coordinates of the target.

5. The method according to claim 1 or 4, characterized in that, The plane to be detected includes at least one of the following: the plane where the display screen is located, the plane where the left rearview mirror is located, the plane where the right rearview mirror is located, the plane where the dashboard is located, and the plane where the windshield is located; the target position point includes at least one of the following: the upper left corner position point, the upper right corner position point, the lower right corner position point, the lower left corner position point, and the center position point.

6. An external parameter calibration device, characterized in that, An auxiliary camera and an auxiliary target are deployed on the exterior of the vehicle to be inspected. The vehicle to be inspected includes an onboard camera, and the target surface of the vehicle to be inspected includes the target. The device includes: The acquisition module is used to acquire a first reference image of the auxiliary target through the vehicle-mounted camera, and to acquire a second reference image of the target to be detected through the auxiliary camera; The determining module is configured to determine, based on the first reference image, a first pose of the auxiliary target rotating to the vehicle-mounted camera; based on the second reference image, a second pose of the plane to be detected rotating to the auxiliary camera; and based on the first pose and the acquired third pose of the auxiliary target rotating to the auxiliary camera, a fourth pose of the auxiliary camera rotating to the vehicle-mounted camera. The processing module is configured to determine the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose; wherein, when the processing module determines the target extrinsic parameters of the plane to be detected in the camera coordinate system of the vehicle-mounted camera based on the fourth pose and the second pose, it specifically performs the following steps: for a target position point on the plane to be detected, obtain the first three-dimensional coordinates of the target position point in the coordinate system of the plane to be detected; convert the first three-dimensional coordinates into the second three-dimensional coordinates of the target position point in the camera coordinate system of the auxiliary camera based on the second pose; convert the second three-dimensional coordinates into the target three-dimensional coordinates of the target position point in the camera coordinate system of the vehicle-mounted camera based on the fourth pose; and use the target three-dimensional coordinates as the target extrinsic parameters.

7. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-5.

8. A vehicle to be inspected, characterized in that, An auxiliary camera and an auxiliary target are deployed on the exterior of the vehicle to be tested. The vehicle to be tested includes an onboard camera, and the plane to be tested of the vehicle to be tested includes the target to be tested. The onboard camera is calibrated using the method described in any one of claims 1-5.

9. The vehicle to be inspected according to claim 8, characterized in that, The vehicle-mounted camera is a near-infrared camera, and the auxiliary camera is a visible light camera; The auxiliary camera is fixed on the auxiliary bracket, and the auxiliary target is fixed on the auxiliary bracket. The auxiliary target is rotated to the third pose of the auxiliary camera and fixed. The auxiliary camera does not obstruct the auxiliary target. The distance between the auxiliary camera and the auxiliary target does not exceed the field of view of the vehicle camera.

Images