Camera external parameter calibration method, vehicle-mounted equipment, readable medium and program product

Abstract

The present application relates to the field of intelligent assisted driving technology, and specifically to a camera extrinsic parameter calibration method, vehicle-mounted equipment, readable media, and program products. During vehicle driving, the vehicle-mounted camera can capture multiple frames of images. The predicted image coordinates of the feature points on the image, the preset extrinsic parameters of the camera, and the change in the vehicle's posture during the interval between the camera capturing multiple frames of images are used to calculate the predicted image coordinates of the feature points on the next frame of image. Based on the error between the predicted image coordinates of the feature points on the next frame of image and the image coordinates of the feature points on the next frame of image, the preset extrinsic parameters can be optimized to obtain the calibration results of the extrinsic parameters. In this way, the calibration of the camera extrinsic parameters can be completed during vehicle driving, thereby improving the accuracy of vehicle positioning and environmental perception.

Description

Technical Field

[0001] The present application relates to the field of intelligent assisted driving technology, and specifically to a camera extrinsic parameter calibration method, system, vehicle-mounted equipment, readable medium, and program product. Background Art

[0002] In assisted and autonomous driving systems, cameras serve as core sensors, using image recognition technology to perceive the vehicle's surroundings. To accurately locate these environmental elements, the camera's extrinsic parameters (i.e., the external parameters of the camera coordinate system relative to the vehicle coordinate system (VCS)) play a key role. These extrinsic parameters enable assisted and autonomous driving systems to precisely map environmental elements captured by the camera to the vehicle coordinate system, providing reliable spatial reference information for path planning, obstacle avoidance, and decision-making.

[0003] However, during actual vehicle use, the camera's external parameters will change dynamically due to factors such as long-term vehicle use, vibration, and mechanical wear. The calibration results will gradually deviate from the true values, resulting in inaccurate vehicle positioning and environmental perception. Summary of the Invention

[0004] In order to solve the problem that the camera extrinsic parameters cannot be calibrated during vehicle driving, resulting in inaccurate vehicle positioning and environmental perception, the embodiments of the present application provide a camera extrinsic parameter calibration method, a vehicle-mounted device, a readable medium and a program product.

[0005] In a first aspect, an embodiment of the present application provides a camera extrinsic parameter calibration method, which is applied to an on-board device, comprising: obtaining M frames of images captured by the vehicle's camera during vehicle driving, where the images are environmental images of the vehicle, and M is a positive integer greater than 2; determining feature points on each frame of the image, wherein the feature points of the first frame of the image are determined by a feature point detection algorithm, and the feature points of the i+1th frame of the image are obtained by performing optical flow tracking on the feature points of the i-th frame of the image, where i is a positive integer taking values ​​sequentially between 1 and (M-1); calculating the predicted image coordinates of the feature point on the i+1th frame of the image based on the image coordinates of the feature point on the i-th frame of the image, the preset extrinsic parameters of the camera, and the i-th distance parameter; wherein the i-th distance parameter is the change in the vehicle's posture during the acquisition interval between the i-th frame of the image and the i+1th frame of the image; optimizing the preset extrinsic parameters based on the error between the predicted image coordinates of the feature point on the i+1th frame of the image and the image coordinates of the feature point on the i+1th frame of the image, to obtain a first calibration result of the camera's extrinsic parameters.

[0006] In this way, by collecting multiple frames of images with the camera and monitoring the changes in the vehicle's posture during the intervals between the multiple frames, the preset extrinsic parameters can be optimized and calibrated. That is, the calibration of the camera's extrinsic parameters can be completed without relying on lane lines and vanishing points, thereby improving the accuracy of vehicle positioning and environmental perception.

[0007] In a possible implementation of the first aspect above, determining the feature points of the first frame image includes: identifying the environmental scene on the first frame image, the environmental scene including a dynamic scene and a static scene; selecting feature points on the static scene through a feature point detection algorithm to determine the feature points of the first frame image.

[0008] In this way, by selecting static scenes on the image as feature points, the change of feature points of dynamic scenes during vehicle driving can be avoided to affect the calibration of extrinsic parameters, thus ensuring the accuracy of extrinsic parameter calibration.

[0009] In a possible implementation of the first aspect above, feature points of the i+1th frame image are obtained by performing optical flow tracking on the feature points of the i-th frame image, including: if the optical flow tracking of the feature points of the i-th frame image is successful, the feature points obtained by the optical flow tracking are the feature points of the i+1th frame image; if the optical flow tracking of the feature points of the i-th frame image fails, the feature points whose optical flow tracking fails are supplemented by corner point extraction to obtain the feature points of the i+1th frame image.

[0010] In this way, by tracking feature points in multiple frames using optical flow, we can reduce feature point mismatches and improve the accuracy of extrinsic calibration. At the same time, we extract corner points for feature points that fail optical flow tracking, ensuring the number of feature points in multiple frames, thereby improving the accuracy of extrinsic calibration.

[0011] In a possible implementation of the first aspect above, the predicted image coordinates of the feature point on the i+1 frame image are calculated based on the image coordinates of the feature point on the i-th frame image, the preset extrinsic parameters of the camera, and the i-th distance parameter, and the method also includes: according to the image coordinates of the feature point on the 1st frame image to the i-th frame image, calculating the depth value of the feature point of the i-th frame image captured by the camera based on the triangulated model, wherein the depth value is the distance from the feature point to the camera; according to the depth value of the feature point of the i-th frame image captured by the camera, the image coordinates of the feature point on the i-th frame image, the preset extrinsic parameters of the camera, and the i-th distance parameter, calculating the predicted image coordinates of the feature point on the i+1 frame image.

[0012] In a possible implementation of the first aspect above, the M frames of images are images captured by the camera in the first time period; obtaining the M frames of images captured by the vehicle's camera includes: obtaining N frames of images captured by the camera in the first time period, where N is greater than M; and selecting M frames of images from the N frames of images, where the distance between the feature points in the M frames of images between the i-th frame image and the i+1-th frame image meets a preset condition.

[0013] In this way, by obtaining M frames of images from N consecutive frames of images captured by the camera, the consumption of computing resources can be significantly reduced when performing external parameter calibration, while effectively improving the efficiency of external parameter optimization.

[0014] In a possible implementation of the first aspect above, K frames of images captured by a vehicle's camera are obtained, where the K frames of images are images captured after the M frames of images; K+1 calibration results of the camera's extrinsic parameters are obtained based on the M frames of images and the K frames of images; and a final calibration result of the camera's extrinsic parameters is obtained based on the K+1 calibration results; wherein, obtaining K+1 calibration results of the camera's extrinsic parameters based on the M frames of images and the K frames of images includes: obtaining the jth calibration result of the camera's extrinsic parameters based on the jth to Mth images in the M frames of images and the 1st to j-1th images in the K frames of images, where j is a positive integer taking values ​​sequentially from 2 to K+1; and forming a K+1 calibration result together with the 1st calibration result and the 2nd to K+1 calibration results.

[0015] In this way, by optimizing the external parameters through the feature points of the M-frame image and the K-frame image, the calibration of the camera external parameters can be completed without relying on lane lines and vanishing points, thereby improving the accuracy of vehicle positioning and environmental perception.

[0016] In a possible implementation of the first aspect above, a final calibration result of the camera's extrinsic parameters is obtained based on K+1 calibration results, including: performing histogram statistics on the K+1 calibration results to obtain statistical results of the extrinsic parameters; if the number and variance of the statistical results of the extrinsic parameters meet a specified threshold, obtaining the final calibration result of the camera's extrinsic parameters based on the K+1 calibration results.

[0017] In this way, by performing histogram statistics on K+1 calibration results and confirming that the number and variance of the statistical results of the external parameters meet the specified threshold, the error of the external parameter calibration is reduced and the accuracy of the external parameter is improved.

[0018] In a second aspect, an embodiment of the present application provides a vehicle-mounted device comprising one or more processors; one or more memories; and one or more memories storing one or more programs. When one or more programs are executed by one or more processors, the device executes the camera extrinsic parameter calibration method of any one of the first aspects.

[0019] In a third aspect, an embodiment of the present application provides a computer-readable medium having instructions stored thereon, which, when executed on a vehicle-mounted device, causes the vehicle-mounted device to execute the camera extrinsic parameter calibration method of any one of the first aspects.

[0020] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When executed by a vehicle-mounted device, the vehicle-mounted device executes the camera extrinsic parameter calibration method of any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the accompanying drawings are described below.

[0022] Figure 1 According to some embodiments of the present application, a schematic flow chart of a camera extrinsic parameter calibration method is shown;

[0023] Figure 2 According to some embodiments of the present application, a schematic flow chart of a camera extrinsic parameter calibration method is shown;

[0024] Figure 3 According to some embodiments of the present application, a schematic diagram of obtaining a final calibration result of an extrinsic parameter of a camera is shown;

[0025] Figure 4 According to some embodiments of the present application, a schematic diagram of a scenario of obtaining feature points of multiple frames of images is shown;

[0026] Figure 5 According to some embodiments of the present application, a schematic diagram of a scene for extracting static scene feature points from an environmental scene is shown;

[0027] Figure 6 According to some embodiments of the present application, a schematic structural diagram of a vehicle-mounted device 100 is shown. DETAILED DESCRIPTION

[0028] In order to facilitate those skilled in the art to understand the solutions in the embodiments of the present application, some concepts and terms involved in the embodiments of the present application are explained below.

[0029] 1. Camera external parameters

[0030] Camera extrinsics describe the position and orientation of the camera in three-dimensional space and are used to transform points in the camera coordinate system into the vehicle coordinate system (VCS). Camera extrinsics consist of a rotation matrix (R) and a translation vector (t). The rotation matrix (R) is a 3x3 matrix that describes the orientation of the camera, that is, the rotation of the camera coordinate system relative to the vehicle coordinate system. The translation vector (t) is a 3x1 vector that describes the position of the camera, that is, the offset of the origin of the camera coordinate system relative to the origin of the vehicle coordinate system. Camera extrinsics can be expressed using the following formula:

[0031] ;

[0032] Where R represents the rotation matrix of the camera coordinate system relative to the vehicle coordinate system, and t represents the translation vector of the camera coordinate system relative to the vehicle coordinate system.

[0033] For example, suppose a point in the camera coordinate system is located , the position of this point transformed into the vehicle coordinate system through the camera external parameter is , can be expressed by the following formula:

[0034] ;

[0035] in, represents a point in the vehicle coordinate system, Represents the position of the point in the camera coordinate system, R represents the rotation matrix, and t represents the translation vector.

[0036] In some scenarios, when a vehicle with intelligent assisted driving is in motion, its camera captures and identifies environmental elements surrounding the vehicle and maps them to the vehicle's coordinate system using camera extrinsic parameters. This information then provides decision-making information for autonomous or assisted driving, such as path planning and obstacle avoidance. However, because existing camera extrinsic parameter calibration methods are typically performed at the factory, these parameters can dynamically change during actual vehicle use due to factors such as vehicle vibration, prolonged use, and mechanical wear. Calibration results can gradually deviate from their true values, leading to inaccurate vehicle positioning and environmental perception.

[0037] In some embodiments, the extrinsic parameters of the camera can be calibrated by the lane lines and the vanishing points of the lane lines. It is understandable that the lane lines are parallel in the real world, but under the perspective projection of the camera, the lane lines will intersect at a point, namely the vanishing point. The position of the vanishing point is directly related to the posture of the camera (pitch angle θ, yaw angle ψ). For example, when the camera is facing forward and horizontal, the vanishing point will be at the center of the image, and the camera will not have a pitch angle θ and a yaw angle ψ; when the camera is tilted up or down, the vanishing point will be above or below the image, and the camera will have a pitch angle θ; if the camera is tilted to the left or right, the vanishing point will be on the left or right side of the image, and the camera will have a yaw angle ψ. In this way, the pitch and yaw angles of the camera can be determined by the change in the position of the vanishing point. It is understandable that the rotation matrix R of the extrinsic parameter can be expressed by the following formula of the pitch angle θ and yaw angle Ψ of the camera:

[0038] ;

[0039] in, represents the rotation matrix of the camera coordinate system relative to the vehicle coordinate system, θ represents the pitch angle of the camera, and ψ represents the yaw angle of the camera.

[0040] It can be understood that during vehicle driving, the camera's translation vector relative to the vehicle body and the camera's roll angle roll around the vehicle's front-rear axis (i.e., the vehicle's forward direction) change little. Therefore, when calibrating the camera's extrinsic parameters, usually only the pitch angle θ and yaw angle ψ are calibrated, and the translation vector t and roll angle roll are not calculated. That is, the calibration of the camera's extrinsic parameters can be expressed by the pitch angle and yaw angle.

[0041] In this way, based on the vehicle's odometer and the lane lines and vanishing points in multiple frames of images, the theoretical position of the vanishing point projected onto the image can be predicted. Based on the error results between the theoretical positions of multiple predicted vanishing points projected onto the image and the actual positions of the vanishing points on the image, the error results are minimized through an iterative optimization algorithm, thereby achieving calibration of the camera extrinsic parameters.

[0042] However, the aforementioned extrinsic parameter calibration method requires the presence of lane lines and has certain requirements for their quality. However, in some scenarios (such as unclear lane lines, missing lane lines on rural roads, curved lane lines, etc.), the vanishing point prediction is inaccurate, which can lead to errors in the camera extrinsic parameter calibration, limiting the application of extrinsic calibration.

[0043] Therefore, to address the above-mentioned issues, this application proposes a camera extrinsic parameter calibration method. During vehicle driving, the on-board camera captures multiple frames of images (e.g., M frames). Feature points of static scenes in the first frame are obtained through a feature point detection algorithm, and the positions of the feature points in subsequent images (e.g., the second to the Mth frames) are obtained through optical flow tracking. Based on the image coordinates of the feature points of the static scene (e.g., the i-th frame), the camera's preset extrinsic parameters, and the vehicle's posture changes during the interval between the camera capturing multiple frames of images (e.g., the i-th frame and the i+1-th frame), the predicted image coordinates of the feature points on the image (the i+1-th frame) are obtained. Based on the error between the predicted image coordinates of the feature points on the image and the image coordinates of the feature points on the image, the preset extrinsic parameters can be optimized, thereby obtaining the calibration results of the camera's extrinsic parameters.

[0044] In this way, during the vehicle's driving process, the preset external parameters can be optimized and calibrated by collecting feature points of multiple frames of images through the camera and the changes in the vehicle's posture during the intervals between collecting multiple frames of images. That is, the calibration of the camera's external parameters can be completed without relying on lane lines and vanishing points, thereby improving the accuracy of vehicle positioning and environmental perception.

[0045] In some embodiments, the camera extrinsic parameters may be calibrated in multiple time periods to obtain multiple extrinsic parameter calibration results, and statistical analysis may be performed based on the multiple extrinsic parameter calibration results to determine a final calibration result of the extrinsic parameters.

[0046] The following uses the M frames of images captured by the camera in the first period as an example to illustrate the first calibration result of the camera's extrinsic parameters.

[0047] Figure 1 Taking the first calibration result of the camera's extrinsic parameters as an example, a flow chart of the camera's extrinsic parameter calibration method is shown. Figure 1 The execution subjects of each step of the process shown are all on-board equipment. For the convenience of description, the following Figure 1 The execution entities of each step will not be repeatedly described in the steps of the process shown.

[0048] like Figure 1 As shown, according to some embodiments, the process of determining the camera extrinsic parameter calibration method includes but is not limited to the following steps:

[0049] S101: Obtain M frames of images captured by a vehicle's camera.

[0050] As you can understand, as a core sensor, a camera needs to be able to perceive the surrounding environment while a vehicle is in motion. For example, the camera captures images of the surrounding environment while the vehicle is in motion. These images represent the surrounding environment, including dynamic objects (such as pedestrians, vehicles, and bicycles) and static objects (such as road signs and trees).

[0051] In some embodiments, during the first period of driving of the vehicle, the camera may collect N frames of continuous images, where N is greater than M. It is understood that when performing external parameter calibration, it is not necessary to process all N frames of continuous images, and M frames of key frame images can be selected from the N frames of continuous images. , based on M key frame images Complete the external parameter calibration. Among them, select M key frame images from N continuous frames When the scene changes between adjacent key frame images must meet the preset conditions.

[0052] For example, if the scene displacement change between the previous key frame image and the current frame (continuous frame) image exceeds a certain threshold (e.g., 0.5 cm), it indicates that the scene has changed significantly and the current frame (continuous frame) image needs to be inserted as a key frame.

[0053] For example, if optical flow tracking finds that the scene overlap area between the current frame (continuous frame) image and the previous key frame image is less than a certain threshold (for example, less than 70%), it indicates that the scene has changed significantly and the current frame (continuous frame) image needs to be inserted as a key frame.

[0054] S102: Determine feature points on each frame of image.

[0055] It can be understood that feature points are points with significant characteristics in an image (such as corner points where the edges of two objects intersect, edge points of objects), and are usually used to represent key information in an image.

[0056] It can be understood that for the M key frame images captured by the camera, the feature points of the first frame are determined by the feature point detection algorithm, and the feature points of the (i+1)th frame are obtained by optical flow tracking the feature points of the i-th frame. For example, the feature points of the first frame are determined by the feature point detection algorithm, and the feature points of the second frame are obtained by optical flow tracking the positions of the feature points of the first frame in the second frame.

[0057] It can be understood that when determining the feature points on the first frame image, the environmental scenes on the first frame image include static scenes (such as road signs, trees, etc.) and dynamic scenes (such as pedestrians, vehicles, bicycles, etc.). During the driving process of the vehicle, the position of the dynamic scenes in the environmental scenes changes, which may affect the calibration of the external parameters. Therefore, in some embodiments, when determining the feature points of the first frame image, the first frame image is pixel-wise classified (such as classified into vehicles, pedestrians, roads, etc.) through perceptual semantic segmentation, and the dynamic scenes (such as vehicles, pedestrians, etc.) and static scenes (such as roads, etc.) classified at the pixel level are identified, and only the feature points of the static scenes are extracted as the feature points on the first frame image; the feature points of the dynamic scenes can also be filtered to obtain the feature points of the static scenes as the feature points of the first frame image, which is not specifically limited here.

[0058] In some embodiments, due to occlusion, unclear line of sight, or other factors, optical flow tracking may fail to capture the position of a feature point in the previous frame in the next frame, resulting in loss of tracking of the feature point in the next frame, thus affecting subsequent motion estimation. Therefore, in some embodiments, feature points lost by optical flow tracking can be supplemented with additional feature points (e.g., using corner point extraction). Alternatively, such feature points can be directly removed and not included in the camera extrinsic calibration. This is not specifically limited here.

[0059] S103: Calculate the predicted image coordinates of the feature point on the (i+1)th frame image according to the image coordinates of the feature point on the (i)th frame image, the preset external parameters of the camera, and the (i)th distance parameter.

[0060] It can be understood that the image coordinates of the feature points can be determined by the images captured by the camera. The preset external parameters of the camera are the initial external parameters of the camera coordinate system relative to the vehicle coordinate system. The i-th distance parameter represents the change in the vehicle's posture during the acquisition interval between the i-th frame image and the i+1-th frame image, wherein the vehicle's posture change includes the vehicle's position change and the vehicle's attitude change. For example, the vehicle's position change can be the distance traveled by the vehicle during the acquisition interval between two frames of image; the vehicle's attitude change can be the change in the vehicle's orientation during the acquisition interval between two frames of image.

[0061] Taking the feature point P as an example, the predicted image coordinates of the feature point on the i+1th frame image are calculated. The process is described below.

[0062] First, obtain the coordinates of the feature point P on the i-th frame image in the camera coordinate system Provide explanation.

[0063] In some embodiments, the image coordinates on the image can be determined when the camera captures the image. For example, when the camera captures the i-th frame image, the image coordinates of the feature point P on the i-th frame image are .

[0064] In some embodiments, in order to express the relationship between the coordinates of the feature points in the camera coordinate system and the image coordinates, the normalized coordinates in the camera coordinate system (hereinafter referred to as normalized coordinates) are introduced. The normalized coordinates are the plane coordinates converted from the image coordinates in the camera coordinate system, which can be expressed by the following formula:

[0065] ;

[0066] in, is the normalized coordinate of the feature point P when capturing the i-th frame image, Represents the image coordinates of the feature point P on the i-th frame image, Indicates the conversion coefficient from image coordinates to normalized coordinates.

[0067] In some embodiments, the depth value of the feature point (i.e., the distance from the feature point to the camera in the camera coordinate system) can be obtained through the triangulated model based on the corresponding relationship between the image coordinates of the feature point in multiple frames. For example, the depth value of the feature point P when capturing the i-th frame image can be obtained through the triangulated model. , the depth value of the feature point P when collecting the i+1 frame image can be obtained through the triangulated model .

[0068] The coordinates of the feature points in the camera coordinate system can be obtained based on the depth value of the feature points and the normalized coordinates of the feature points, which can be expressed by the following formula:

[0069] ;

[0070] in, Indicates the coordinates of the feature point P in the camera coordinate system when capturing the i-th frame image, Indicates the depth value of the feature point P when capturing the i-th frame image, is the normalized coordinate of the feature point P when capturing the i-th frame image.

[0071] It is understandable that the distribution of depth values ​​is usually non-Gaussian. Especially at long distances, the distribution range of depth values ​​is large, and the depth value of distant objects may have a greater impact on the external parameter calibration results. Therefore, in order to make the collected depth value more stable and conform to the Gaussian distribution, the depth value of the feature point P when collecting the i-th frame image can be Expressed as an inverse depth value that conforms to a Gaussian distribution , the coordinates of the feature points in the camera coordinate system can also be expressed by the following formula:

[0072] ;

[0073] in, Indicates the coordinates of the feature point P in the camera coordinate system when capturing the i-th frame image, is the normalized coordinate of the feature point P when capturing the i-th frame image, It represents the inverse depth value of the feature point P when capturing the i-th frame image.

[0074] Next, we obtain the predicted coordinates of the feature point P in the camera coordinate system when capturing the i+1 frame image. Provide explanation.

[0075] It can be understood that according to the coordinates of the feature point P in the camera coordinate system when the i-th frame image is collected With the i-th distance parameter and the preset external parameters of the camera, we can get the predicted coordinates of the feature point P in the camera coordinate system when capturing the i+1-th frame image, which can be expressed by the following formula:

[0076] ;

[0077] in, It represents the predicted coordinates of the feature point P in the camera coordinate system when the i+1 frame image is collected. It represents the motion displacement of the feature point P in the camera coordinate system between the acquisition of the i-th frame image and the i+1-th frame image, which can be obtained by express, is the i-th distance parameter, which represents the change in the vehicle's posture during the acquisition interval between the i-th frame image and the i+1-th frame image. Indicates the parameters of the camera coordinate system relative to the vehicle coordinate system (i.e. the preset external parameters of the camera), Represents the parameters of the vehicle coordinate system relative to the camera coordinate system (i.e., the inverse transformation of the camera's preset external parameters), Represents the coordinates of the feature point P in the camera coordinate system when capturing the i-th frame image.

[0078] Next, according to the predicted coordinates of the feature point P in the camera coordinate system when the i+1 frame image is collected , to obtain the predicted image coordinates of the feature point P i+1 frame image Provide explanation.

[0079] According to the predicted coordinates of the feature point P in the camera coordinate system when the i+1 frame image is collected , the inverse depth value of the feature point P when collecting the i+1 frame image , we can get the predicted normalized coordinates of the feature point P on the i+1th frame image, which can be expressed by the following formula:

[0080] ;

[0081] in, represents the predicted normalized coordinates of the feature point P when the i+1th frame image is collected, It represents the predicted coordinates of the feature point P in the camera coordinate system when the i+1 frame image is collected. It represents the inverse depth value of the feature point P when the i+1th frame image is collected.

[0082] Project the predicted normalized coordinates of the feature point P when the i+1th frame image is collected to obtain the predicted image coordinates of the feature point P in the i+1th frame image, which can be expressed by the following formula:

[0083] ;

[0084] in, Indicates the predicted image coordinates of the feature point P when the i+1th frame image is collected, represents the predicted normalized coordinates of the feature point P when the i+1th frame image is collected, Indicates the conversion coefficient from normalized coordinates to image coordinates.

[0085] In this way, based on the image coordinates of the feature point P on the i-th frame image, the preset external parameters of the camera, the i-th distance parameter, and the depth values ​​of the feature point P on the i-th frame image and the i+1-th frame image, the predicted image coordinates of the feature point P on the i+1-th frame image can be obtained.

[0086] S104: Optimize the preset extrinsic parameters according to the error between the predicted image coordinates of the feature point on the (i+1)th frame image and the image coordinates of the feature point on the (i+1)th frame image to obtain a first calibration result of the extrinsic parameters of the camera.

[0087] In some embodiments, the image coordinates of the feature point P when the i+1th frame image is captured can be obtained by optical flow tracking. , according to the predicted image coordinates of the feature point P on the i+1 frame image and image coordinates , we can get the error between the predicted image coordinates and the image coordinates of the feature point P on the i+1th frame image, which can be expressed by the following formula:

[0088] ;

[0089] in, represents the error between the predicted image coordinates and the image coordinates of the feature point, Indicates the predicted image coordinates of the feature point P when the i+1th frame image is collected, Represents the image coordinates of the feature point P on the i+1th frame image.

[0090] It can be understood that multiple predicted image coordinates and image coordinate errors are obtained through w feature points on M key frame images. By minimizing the errors between multiple predicted image coordinates and image coordinates, the first calibration result of the camera's extrinsic parameters and the pseudo-depth values ​​of the w feature points can be obtained, which can be expressed by the following formula:

[0091] ;

[0092] in, Represents M frame images, pw represents w feature points on M frame images, Represents the error of w feature points on M frame images, Indicates minimizing the error between the predicted image coordinates and the image coordinates of w feature points on the M frame image. Represents the inverse depth value of w feature points, Represents the first calibration result of the camera coordinate system relative to the vehicle coordinate system parameters (i.e. the external parameters of the camera) after minimization ).

[0093] In this way, the first calibration result of the camera's external parameters can be obtained by calculating the error between the predicted image coordinates of w feature points on the M frame image and the image coordinates on the feature point image. .

[0094] It can be understood that when calibrating the external parameters of the camera, in order to improve the accuracy of the camera external parameters, it is necessary to track the feature points multiple times to obtain the calibration results of multiple external parameters, and then perform statistics based on the calibration results of multiple external parameters to obtain the final calibration results of the external parameters.

[0095] The following example uses the process of acquiring K key-frame images after acquiring M key-frame images. Based on the M and K frame images, K+1 extrinsic parameter calibration results are obtained. By statistically analyzing the K+1 calibration results, the final calibration result of the camera's extrinsic parameters is obtained.

[0096] Figure 2 Taking obtaining K+1 calibration results of the camera's extrinsic parameters as an example, a flow chart of the camera extrinsic parameter calibration method is shown. Figure 2 The execution subjects of each step of the process shown are all on-board equipment. For the convenience of description, the following Figure 2 The execution entities of each step will not be repeatedly described in the steps of the process shown.

[0097] like Figure 2 As shown, the method includes:

[0098] S201: Acquire an image captured by a vehicle's camera.

[0099] It can be understood that when a vehicle is driving, the camera, as one of the core sensors, can perceive the environmental factors around the vehicle.

[0100] Figure 3 This is a schematic diagram of obtaining the final calibration result of the camera's external parameters. Figure 3 As shown in the figure, during a certain period of driving, the camera can collect N frames of continuous images and the posture changes of the vehicle when collecting N frames of images, which are used as the input of external parameter calibration.

[0101] S202: Determine feature points on each frame of image.

[0102] For N consecutive frames of images captured by the camera, the feature points on each frame can be obtained. Figure 3 As shown, feature points on multiple frames can be obtained by optical flow tracking. For example, the feature points of the first frame are determined by a feature point detection algorithm, and the feature points of the second frame are obtained by optical flow tracking the positions of the feature points of the first frame in the second frame.

[0103] Figure 4 A schematic diagram of the scene for obtaining feature points of multiple frames of images. Figure 4 As shown in the figure, when the vehicle is driving on the road, the camera can collect multiple frames of images and obtain the feature points on each frame of the image (the area occupied by the black points in the figure) through optical flow tracking.

[0104] In some embodiments, when acquiring feature points on an image, the feature points on the image may be pre-processed. Figure 3 As shown, when the feature points on the image are preprocessed, dynamic scenes (such as vehicles, pedestrians, etc.) and static scenes (such as roads, etc.) on the image are identified through perceptual semantic segmentation, and the feature points of the static scenes are extracted as the feature points on the image; the feature points of the dynamic scenes can also be filtered to obtain the feature points of the static scenes as the feature points of the image, which is not specifically limited here.

[0105] Figure 5 It is a scene diagram for extracting static scene feature points from the environment. Figure 5 As shown in the figure, the environmental scenes in the image include static scenes such as road signs, telephone poles, and trees, as well as dynamic scenes such as vehicles. When preprocessing the feature points on the image, the feature points of the dynamic scenes of the vehicles are filtered out, and the feature points of the static scenes such as road signs, telephone poles, and trees on the image are obtained, and the feature points containing only the static scenes (such as Figure 5 white feature points in the middle).

[0106] S203: storing feature points of each frame image, and obtaining depth values ​​of the feature points on each frame image.

[0107] It can be understood that in order to analyze the motion position of feature points in multiple frames, the same feature points on each frame can be stored uniformly so that the feature points are associated in multiple frames. For example, the coordinates of feature point P on the first frame are , the coordinates on the second frame image ...coordinates on the Nth frame image Uniformly stored in feature point set The coordinates of feature point Q on the first frame image , the coordinates on the second frame image ...coordinates on the Nth frame image Uniformly stored in feature point set .

[0108] It can be understood that according to the image coordinates of the camera capturing multiple frames of images, the depth value of the feature point P when the camera captures the i-th frame image is obtained through the triangulated model. (i.e. the distance from the feature point to the camera in the camera coordinate system) and the inverse depth value The specific method is the same as step S103 and will not be repeated here.

[0109] S204: Select M key frame images.

[0110] It can be understood that while the vehicle is driving, the camera can capture N consecutive frames of images. When performing extrinsic calibration on the feature points on the image, M keyframe images can be selected from the N consecutive frames for extrinsic calibration. When selecting M keyframe images from N consecutive frames, the first frame image is initialized as the keyframe image. The method for selecting the current i-th frame image (2≤i≤N) as the keyframe is as follows:

[0111] When the running distance between the feature points on the previous key frame image and the feature points on the current i-th frame (continuous frame) image is greater than a certain threshold, it means that the scene has changed significantly and the current i-th frame (continuous frame) image needs to be inserted as a key frame.

[0112] When the feature points on the previous key frame can be tracked to the current i-th frame (continuous frame) image according to the optical flow, and the ratio of the feature points on the current i-th frame (continuous frame) to the feature points on the previous key frame is less than a certain threshold (for example, less than 70%), it means that the scene has changed significantly and the current i-th frame (continuous frame) needs to be inserted as a key frame.

[0113] S205: Determine whether the number of key frame images reaches M frames.

[0114] If the current key frame image does not meet the quantity requirement (reaches M frames), step S204 is executed to select a key frame image from the collected N consecutive frame images. If the current key frame image meets the quantity requirement (reaches M frames), step S206 is executed.

[0115] In some embodiments, the selected multiple key frame images can be placed in a sliding window. It can be understood that the sliding window is a collection of a certain number of key frame images. Figure 3 As shown, the selected key frame images can be placed in a sliding window, and whether the key frame images meet the quantity requirement can be determined based on the sliding window.

[0116] It can be understood that when the number of key frame images in the sliding window reaches a certain size (for example, 5 key frame images), it means that the image has enough feature points to calculate the inverse depth value. and external parameters, the key frame image in the sliding window can be sent to step S206.

[0117] When the number of key frame images in the sliding window does not reach a certain size (for example, 5 key frame images), it means that there are fewer feature points in the image, and the inverse depth value is calculated. If the error between the two parameters is large, the process returns to step S204.

[0118] When M key frame images are obtained, the corresponding processed continuous frame images are recorded as G, satisfying N>G>M.

[0119] S206: Obtain the first calibration result of the camera's external parameters.

[0120] According to step S101 to step S104, the first calibration result of the camera's external parameters is obtained And the inverse depth values ​​of w feature points in M ​​frame images .

[0121] S207: Obtain K key frame images captured by the vehicle's camera.

[0122] Repeat steps S204-S205 from NG frame continuous frame images until K key frames are obtained, which are recorded as .

[0123] For example, in the first period, the camera can collect M key frame images ,After the first period, the camera continues to collect K key frame images .

[0124] S208: Obtain K+1 calibration results of the camera extrinsic parameters.

[0125] After collecting K key frame images, combined with the M key frame images obtained in the first period, a total of M+K key frame images are obtained, which are recorded as .

[0126] Repeat step S206 based on the jth to Mth images in the M key frame images and the 1st to j-1th images in the K key frame images to optimize the preset external parameters and obtain the jth calibration result of the external parameters of the camera. , where j is a positive integer ranging from 2 to K+1.

[0127] Take the value of j in turn and repeat the above operation to obtain the 2nd to K+1th calibration results of the camera external parameters .

[0128] According to the first calibration result and the 2nd to K+1st calibration results , get K+1 calibration results .

[0129] S209: Determine whether the statistical results of the K+1 calibration results meet a specified threshold.

[0130] The first calibration result of the camera's external parameters according to steps S101 to S104 , the 2nd to K+1th calibration results of the camera external parameters in step S208 , determine K+1 calibration results .

[0131] Continue as Figure 3 As shown, the K+1 calibration results are subjected to histogram statistics to obtain the statistical results of the K+1 calibration results.

[0132] If the number and variance of the statistical results of the K+1 calibration results meet the specified threshold, execute step S210;

[0133] If the number and variance of the statistical results of the K+1 calibration results do not meet the specified threshold, the process returns to step S201.

[0134] In some embodiments, the K+1 calibration results are subjected to histogram statistical processing to obtain a statistical result of the K+1 calibration results, which can be expressed by the following formula:

[0135] ;

[0136] in, Indicates the final calibration result of the camera's extrinsic parameters when the number and variance of the statistical results meet the specified threshold (determined in step S210), Indicates the K+1 external parameter calibration results, Indicates that histogram statistical processing is performed on the K+1 external parameter calibration results.

[0137] It can be understood that by adjusting K+1 external parameters ( ) The calibration results are statistically processed using a histogram (e.g., determining whether the number of K+1 extrinsic parameter calibration results meets the requirement and whether the calculated variance meets the threshold), and determining whether the statistical result of the K+1 calibration results meets the specified threshold. If the statistical result of the K+1 calibration results meets the specified threshold, step S210 is executed. If the statistical result of the K+1 calibration results does not meet the specified threshold, it indicates that the extrinsic parameter calibration is not accurate enough and more frames of image data need to be recaptured, and the process returns to step S201.

[0138] S210: Obtain the final calibration result of the camera's external parameters.

[0139] Through Perform histogram statistics on the K+1 external parameter calibration results and determine if the statistical results of the K+1 calibration results meet the specified threshold, indicating that the camera external parameter calibration is completed. The external parameter with the highest frequency of occurrence can be selected from the histogram as the final calibration result of the camera's external parameter. , thus achieving the calibration of external parameters.

[0140] In this way, the preset extrinsic parameters are optimized based on the difference between the predicted position of the feature points on the image and the actual position of the feature points on the image, so that the calibration of the camera extrinsic parameters can be completed without the need for lane lines while the vehicle is moving.

[0141] For example, Figure 6 According to some embodiments of the present application, a schematic structural diagram of a vehicle-mounted device 100 is shown. Figure 6 As shown, the in-vehicle device 100 includes one or more processors 101, a system memory 102, a non-volatile memory (NVM) 103, a communication interface 104, an input / output (I / O) device 105, and a system control logic unit 106 for coupling the processor 101, the system memory 102, the non-volatile memory 103, the communication interface 104, and the I / O device 105. Among them:

[0142] The processor 101 can be used to control the vehicle-mounted device 100 to execute the camera extrinsic parameter calibration method of the present application. The processor 101 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit of a programmable logic device (FPGA). The processor 101 may include one or more single-core or multi-core processors.

[0143] The processor 101 can be used to control the vehicle-mounted device 100 to execute the camera extrinsic parameter calibration method of the present application. The processor 101 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit of a programmable logic device (FPGA). The processor 101 may include one or more single-core or multi-core processors. In some embodiments, the processor 101 may be used to execute the camera extrinsic parameter calibration method of the embodiments of the present invention.

[0144] The system memory 102 is a volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. The system memory 102 is used to temporarily store data and / or instructions.

[0145] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card.

[0146] In particular, the system memory 102 and the non-volatile memory 103 may respectively include a temporary copy and a permanent copy of the instruction 107. The instruction 107 may include: when executed by the processor 101, enabling the vehicle-mounted device 100 to implement the camera extrinsic parameter calibration method provided in various embodiments of the present application.

[0147] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the vehicle-mounted device 100, thereby communicating with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the vehicle-mounted device 100, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the vehicle-mounted device 100 can communicate with other devices via the communication interface 104.

[0148] The input / output device 105 may include input devices such as a keyboard, a mouse, etc., and output devices such as a display, etc. A user may interact with the in-vehicle device 100 through the input / output device 105 .

[0149] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface with other modules of the vehicle-mounted device 100. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide interfaces to the system memory 102 and the non-volatile memory 103.

[0150] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may be integrated with the logic of one or more controllers for the system control logic unit 106 on the same chip to form a SoC.

[0151] I understand. Figure 6 The structure of the vehicle-mounted device 100 shown is merely an example. In other embodiments, the vehicle-mounted device 100 may include more or fewer components than shown, or may combine or separate certain components, or may have different component arrangements. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0152] In some embodiments, an embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer program instruction, at least one program, code set or instruction set. The at least one computer program instruction, at least one program, code set or instruction set is loaded and executed by a vehicle-mounted device to implement the camera extrinsic parameter calibration method provided by the above-mentioned method embodiments.

[0153] In some embodiments, the embodiments of the present application further provide a computer program product, which includes computer instructions. When executed by a vehicle-mounted device, the vehicle-mounted device executes the camera extrinsic parameter calibration method provided by the above-mentioned various method embodiments.

[0154] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of this application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0155] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0156] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0157] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions may be distributed over a network or via other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to floppy disks, optical disks, optical discs, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage for transmitting information via the Internet using electrical, optical, acoustic, or other propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0158] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0159] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0160] It should be noted that in the examples and description of this patent, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a" does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0161] Although the present application has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.

Claims

1. A camera extrinsic parameter calibration method, applied to vehicle-mounted equipment, characterized by: include: During the driving of the vehicle, M frames of images captured by a camera of the vehicle are obtained, where the images are images of the environment of the vehicle, and M is a positive integer greater than or equal to 2; Determine feature points on each frame of the image, wherein the feature points of the first frame of the image are determined by a feature point detection algorithm, and the feature points of the (i+1)th frame of the image are obtained by performing optical flow tracking on the feature points of the i-th frame of the image, where i is a positive integer ranging from 1 to (M-1); Calculating the predicted image coordinates of the feature point on the (i+1)th frame image based on the image coordinates of the feature point on the (i)th frame image, the preset extrinsic parameters of the camera, the (i)th distance parameter, and the depth value of the feature point P on the (i)th frame image and the (i+1)th frame image; wherein the (i)th distance parameter is the change in the vehicle's posture during the acquisition interval between the (i)th frame image and the (i+1)th frame image; According to the error between the predicted image coordinates of the feature point on the i+1th frame image and the image coordinates of the feature point on the i+1th frame image, the preset extrinsic parameters are optimized to obtain a first calibration result of the extrinsic parameters of the camera.

2. The method according to claim 1, characterized in that Determining the feature points of the first frame of image includes: Identifying environmental scenes on the first frame of image, wherein the environmental scenes include dynamic scenes and static scenes; Feature points are selected on the static scene using the feature point detection algorithm to determine the feature points of the first frame image.

3. The method according to claim 1, characterized in that The step of obtaining the feature points of the (i+1)th frame image by performing optical flow tracking on the feature points of the (i)th frame image comprises: If the optical flow tracking of the feature points of the i-th frame image is successful, the feature points obtained by the optical flow tracking are the feature points of the i+1-th frame image; If optical flow tracking of the feature points of the i-th frame image fails, the feature points for which optical flow tracking fails are supplemented by corner point extraction to obtain feature points of the i+1-th frame image.

4. The method according to claim 1, wherein The step of calculating the predicted image coordinates of the feature point on the (i+1)th frame image according to the image coordinates of the feature point on the (i)th frame image, the preset external parameters of the camera, and the (i)th distance parameter further includes: According to the image coordinates of the feature point on the first to i-th frames of image, a depth value of the feature point in the i-th frame of image captured by the camera is calculated based on a triangulated model, wherein the depth value is the distance from the feature point to the camera; According to the depth value of the feature point in the i-th frame image captured by the camera, the image coordinates of the feature point on the i-th frame image, the preset external parameters of the camera, and the i-th distance parameter, the predicted image coordinates of the feature point on the i+1-th frame image are calculated.

5. The method according to any one of claims 1 to 4, characterized in that The M frames of images are images collected by the camera in the first period; The acquiring of M frames of images captured by a camera of the vehicle includes: Acquire N frames of images captured by the camera during the first time period, where N is greater than M; The M frames of images are selected from the N frames of images, wherein the distance between the feature points in the M frames of images and the i-th frame of image and the i+1-th frame of image meets a preset condition.

6. The method according to claim 1, characterized in that , the method further includes: Acquire K frames of images captured by a camera of the vehicle, where the K frames of images are images captured after the M frames of images; Obtaining K+1 calibration results of the camera extrinsic parameters according to the M frames of images and the K frames of images; Obtaining a final calibration result of the camera's extrinsic parameters based on the K+1 calibration results; Wherein, obtaining K+1 calibration results of the camera extrinsic parameters according to the M frames of images and the K frames of images includes: Obtain a j-th calibration result of the camera's extrinsic parameters based on the j-th to M-th images in the M frames and the 1-j-1-th images in the K frames, where j is a positive integer ranging from 2 to K+1. The K+1 calibration results are formed together with the first calibration result and the second to K+1 calibration results.

7. The method according to claim 6, characterized in that Obtaining a final calibration result of the camera's extrinsic parameters based on the K+1 calibration results includes: Perform histogram statistics on the K+1 calibration results to obtain the statistical results of the external parameters; If the number and variance of the statistical results of the extrinsic parameters meet a specified threshold, a final calibration result of the extrinsic parameters of the camera is obtained according to the K+1 calibration results.

8. A vehicle-mounted device, characterized in that: The device comprises one or more processors; one or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the device executes the camera extrinsic parameter calibration method according to any one of claims 1 to 7.

9. A computer-readable medium, characterized in that The readable medium stores instructions, which, when executed on the vehicle-mounted device, enable the vehicle-mounted device to execute the camera extrinsic parameter calibration method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The program product includes computer instructions, and when executed by an onboard device, the onboard device executes the camera extrinsic parameter calibration method according to any one of claims 1 to 7.

Images