Camera external parameter calibration method, vehicle-mounted equipment, readable medium and program product

Abstract

The invention relates to the technical field of intelligent auxiliary driving, in particular to a camera external parameter calibration method, vehicle-mounted equipment, a readable medium and a program product. In the vehicle driving process, a vehicle-mounted camera can collect multiple frames of images, and through image coordinates of feature points on the images, preset external parameters of the camera and pose changes of the vehicle within the interval time when the camera collects the multiple frames of images, predicted image coordinates of the feature points on the next frame of image are calculated; and according to the error between the predicted image coordinate of the feature point on the next frame of image and the image coordinate of the feature point on the next frame of image, the preset external parameter can be optimized to obtain the calibration result of the external parameter. In this way, calibration of the external parameters of the camera can be completed in the vehicle driving process, and the accuracy of vehicle positioning and environment sensing is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent assisted driving technology, and particularly to a method and system for calibrating the extrinsic parameters of a camera, a vehicle-mounted device, a readable medium, and a program product. Background Art

[0002] In assisted driving systems and autonomous driving systems, as one of the core sensors, a camera can perceive environmental elements around the vehicle based on image recognition technology. To achieve the accurate positioning of environmental elements, the extrinsic parameters of the camera (i.e., the external parameters of the camera coordinate system relative to the vehicle body coordinate system (Vehicle Coordinate System, VCS)) play a crucial role. Through the extrinsic parameters, the assisted driving system and the autonomous driving system can accurately map the environmental elements captured by the camera into the vehicle body coordinate system, thereby providing reliable spatial reference information for path planning, obstacle avoidance, and decision-making.

[0003] However, during the actual use of the vehicle, the extrinsic parameters of the camera will change dynamically due to factors such as long-term use of the vehicle, vibration, and mechanical wear, and the calibration result gradually deviates from the true value, resulting in inaccurate vehicle positioning and environmental perception. Summary of the Invention

[0004] To solve the problem that the extrinsic parameters of the camera cannot be calibrated during vehicle driving, resulting in inaccurate vehicle positioning and environmental perception, embodiments of this application provide a method for calibrating the extrinsic parameters of a camera, a vehicle-mounted device, a readable medium, and a program product.

[0005] In a first aspect, an embodiment of this application provides a method for calibrating the extrinsic parameters of a camera, which is applied to a vehicle-mounted device and includes: during vehicle driving, acquiring M frames of images collected by the vehicle's camera, where the images are environmental images of the vehicle, and M is a positive integer greater than or equal to 2; determining the feature points on each frame of the image, where the feature points of the first frame of the image are determined by a feature point detection algorithm, and the feature points of the (i + 1)-th frame of the image are obtained by optical flow tracking of the feature points of the i-th frame of the image, and i is a positive integer that takes values in sequence between 1 and (M - 1); calculating the predicted image coordinates of the feature points on the (i + 1)-th frame of the image according to the image coordinates of the feature points on the i-th frame of the image, the preset extrinsic parameters of the camera, and the i-th distance parameter; where the i-th distance parameter is the pose change of the vehicle during the acquisition interval between the i-th frame of the image and the (i + 1)-th frame of the image; optimizing the preset extrinsic parameters according to the error between the predicted image coordinates of the feature points on the (i + 1)-th frame of the image and the image coordinates of the feature points on the (i + 1)-th frame of the image to obtain the first calibration result of the extrinsic parameters of the camera.

[0006] In this way, by collecting multiple frames of images through a camera and the pose changes of the vehicle during the time interval between collecting multiple frames of images, the preset external parameters can be optimized and calibrated. That is, the calibration of the camera's external parameters can be completed without relying on lane lines and vanishing points, improving the accuracy of vehicle positioning and environmental perception.

[0007] In a possible implementation of the first aspect above, determining the feature points of the first frame of image includes: identifying the environmental scenes on the first frame of image, where the environmental scenes include dynamic scenes and static scenes; selecting feature points on the static scenes through a feature point detection algorithm to determine the feature points of the first frame of image.

[0008] In this way, by selecting the static scenes on the image as feature points, it is avoided that the change of feature points of dynamic scenes during vehicle driving affects the calibration of external parameters, ensuring the accuracy of external parameter calibration.

[0009] In a possible implementation of the first aspect above, obtaining the feature points of the (i + 1)-th frame of image by performing optical flow tracking on the feature points of the i-th frame of image includes: if the optical flow tracking of the feature points of the i-th frame of image is successful, the feature points obtained by optical flow tracking are the feature points of the (i + 1)-th frame of image; if the optical flow tracking of the feature points of the i-th frame of image fails, corner points are extracted to supplement the feature points where optical flow tracking fails, obtaining the feature points of the (i + 1)-th frame of image.

[0010] In this way, by performing optical flow tracking on the feature points in multiple frames of images, the incorrect matching of feature points is reduced, improving the accuracy of external parameter calibration. At the same time, corner points are extracted to supplement the feature points where optical flow tracking fails, ensuring the number of feature points on multiple frames of images, thereby improving the accuracy of external parameter calibration.

[0011] In a possible implementation of the first aspect above, calculating the predicted image coordinates of the feature points on the (i + 1)-th frame of image according to the image coordinates of the feature points on the i-th frame of image, the preset external parameters of the camera, and the i-th distance parameter further includes: calculating the depth value of the feature points of the i-th frame of image collected by the camera based on the triangulation model according to the image coordinates of the feature points from the first frame of image to the i-th frame of image, where the depth value is the distance from the feature points to the camera; calculating the predicted image coordinates of the feature points on the (i + 1)-th frame of image according to the depth value of the feature points of the i-th frame of image collected by the camera, the image coordinates of the feature points on the i-th frame of image, the preset external parameters of the camera, and the i-th distance parameter.

[0012] In a possible implementation of the first aspect above, the M frames of images are the images collected by the camera in the first time period; obtaining the M frames of images collected by the vehicle's camera includes: obtaining N frames of images collected by the camera in the first time period, where N is greater than M; selecting M frames of images from the N frames of images, where the distance between the feature points in the M frames of images on the i-th frame of image and the (i + 1)-th frame of image satisfies a preset condition.

[0013] In this way, by obtaining M frames of images from N consecutive frames of images collected by the camera, the consumption of computing resources is significantly reduced during the external parameter calibration, and at the same time, the efficiency of external parameter optimization is effectively improved.

[0014] In a possible implementation of the first aspect above, K frames of images collected by the vehicle's camera are obtained, and the K frames of images are the images collected after the M frames of images; K + 1 calibration results of the external parameters of the camera are obtained according to the M frames of images and the K frames of images; the final calibration result of the external parameters of the camera is obtained according to the K + 1 calibration results; wherein, obtaining the K + 1 calibration results of the external parameters of the camera according to the M frames of images and the K frames of images includes: obtaining the jth calibration result of the external parameters of the camera according to the jth to Mth images in the M frames of images and the 1st to j - 1th images in the K frames of images, where j is a positive integer that takes values in sequence from 2 to K + 1; the 1st calibration result and the 2nd to K + 1th calibration results are jointly used to form the K + 1 calibration results.

[0015] In this way, by optimizing the external parameters through the feature points of the M frames of images and the K frames of images, it is possible to complete the calibration of the external parameters of the camera without relying on lane lines and vanishing points, improving the accuracy of vehicle positioning and environmental perception.

[0016] In a possible implementation of the first aspect above, obtaining the final calibration result of the external parameters of the camera according to the K + 1 calibration results includes: performing a histogram statistics on the K + 1 calibration results to obtain the statistical result of the external parameters; if the quantity and variance of the statistical result of the external parameters meet the specified threshold, the final calibration result of the external parameters of the camera is obtained according to the K + 1 calibration results.

[0017] In this way, by performing a histogram statistics on the K + 1 calibration results and confirming that the quantity and variance of the statistical result of the external parameters meet the specified threshold, the error of external parameter calibration is reduced, and the accuracy of the external parameters is improved.

[0018] In a second aspect, an embodiment of the present application provides an in-vehicle device, including one or more processors; one or more memories; one or more programs are stored in the one or more memories, and when the one or more programs are executed by the one or more processors, the device executes the calibration method for the external parameters of the camera according to any one of the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a computer-readable medium, on which instructions are stored, and when the instructions are executed on the in-vehicle device, the in-vehicle device executes the calibration method for the external parameters of the camera according to any one of the first aspect.

[0020] Fourthly, an embodiment of the present application provides a computer program product, which includes computer instructions. When executed by a vehicle-mounted device, the vehicle-mounted device executes the calibration method for the external parameters of the camera according to any one of the first aspects. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following provides an illustration of the drawings.

[0022] Figure 1 According to some embodiments of the present application, a flowchart of the calibration method for the external parameters of the camera is shown; Figure 2 According to some embodiments of the present application, a flowchart of the calibration method for the external parameters of the camera is shown; Figure 3 According to some embodiments of the present application, a schematic diagram of obtaining the final calibration result of the external parameters of the camera is shown; Figure 4 According to some embodiments of the present application, a schematic diagram of the scene of obtaining feature points of multiple frames of images is shown; Figure 5 According to some embodiments of the present application, a schematic diagram of the scene of extracting feature points of static scenes in the environmental scenery is shown; Figure 6 According to some embodiments of the present application, a schematic diagram of the structure of a vehicle-mounted device 100 is shown. Detailed Embodiments

[0023] To facilitate those skilled in the art to understand the solutions in the embodiments of the present application, some concepts and terms related to the embodiments of the present application will be explained below.

[0024] 1. External parameters of the camera The external parameters of the camera describe the position and orientation of the camera in three-dimensional space and are used to transform points in the camera coordinate system into the vehicle body coordinate system (Vehicle Coordinate System, VCS). The composition of the external parameters of the camera includes a rotation matrix (R) and a translation vector (t). Among them, the rotation matrix (R) is a 3x3 matrix that describes the orientation of the camera, that is, the rotation of the camera coordinate system relative to the vehicle body coordinate system. The translation vector (t) is a 3x1 vector that describes the position of the camera, that is, the offset of the origin of the camera coordinate system relative to the origin of the vehicle body coordinate system. The external parameters of the camera can be expressed by the following formula: ; Among them, R represents the rotation matrix of the camera coordinate system relative to the vehicle body coordinate system, and t represents the translation vector of the camera coordinate system relative to the vehicle body coordinate system.

[0025] For example, assume the position of a point in the camera coordinate system , the position of this point transformed into the vehicle body coordinate system through the extrinsic parameters of the camera is , which can be expressed by the following formula: ; Among them, represents a point in the vehicle body coordinate system, represents the position of this point in the camera coordinate system, R represents the rotation matrix, and t represents the translation vector.

[0026] In some scenarios, when an intelligent assisted driving vehicle is driving, the camera captures and identifies the environmental elements around the vehicle and maps them into the vehicle body coordinate system through the extrinsic parameters of the camera, and provides decision-making information such as path planning and obstacle avoidance for autonomous driving or assisted driving. However, since some existing methods for calibrating the extrinsic parameters of the camera are generally carried out when the vehicle leaves the factory, the extrinsic parameters of the camera will change dynamically due to factors such as vehicle vibration, long-term use, and mechanical wear during actual vehicle use, and the calibration results gradually deviate from the true values, resulting in inaccurate vehicle positioning and environmental perception.

[0027] In some embodiments, the extrinsic parameters of the camera can be calibrated by the lane lines and the vanishing point of the lane lines. It can be understood that the lane lines are parallel in the real world, but the lane lines will intersect at a point, that is, the vanishing point, under the perspective projection of the camera. The position of the vanishing point is directly related to the pose of the camera (pitch angle θ, yaw angle ψ). For example, when the camera is facing directly forward and horizontal, the vanishing point will be located at the center of the image, and the camera has no pitch angle θ and yaw angle ψ; when the camera tilts up or down, the vanishing point is above or below the image, and the camera has a pitch angle θ; if the camera deflects left or right, the vanishing point is on the left or right side of the image, and the camera has a yaw angle ψ. In this way, the pitch angle and yaw angle of the camera can be determined by the change in the position of the vanishing point. It can be understood that the rotation matrix R of the extrinsic parameters can be expressed by the following formula of the pitch angle θ and yaw angle Ψ of the camera: ; Among them, represents the rotation matrix of the camera coordinate system relative to the vehicle body coordinate system, θ represents the pitch angle of the camera, and ψ represents the yaw angle of the camera.

[0028] It can be understood that during the vehicle driving process, the translation vector of the camera relative to the vehicle body and the roll angle roll of the camera around the front and rear axes of the vehicle body (i.e., the forward direction of the vehicle body) change little. Therefore, when calibrating the extrinsic parameters of the camera, usually only the pitch angle θ and yaw angle ψ are calibrated, and the translation vector t and roll angle roll are not calculated. That is, the calibration of the extrinsic parameters of the camera can be represented by the pitch angle and yaw angle.

[0029] Thus, based on the odometer of the vehicle and the lane lines and vanishing points in multiple frames of images, the theoretical position of the vanishing point projected onto the image can be predicted. According to the error results between the theoretical positions of multiple predicted vanishing points projected onto the image and the actual positions of the vanishing points on the image, the error results are minimized through an iterative optimization algorithm, and thus the calibration of the external parameters of the camera can be achieved.

[0030] However, the above external parameter calibration method must have lane lines, and there are certain requirements for the quality of the lane lines. However, in some scenarios (such as unclear lane lines, lack of lane lines on rural roads, curved lane lines, etc.), the prediction of the vanishing point is inaccurate, which may lead to errors in the calibration of the external parameters of the camera, restricting the application of the calibrated external parameters.

[0031] Therefore, to solve the above problems, the present application proposes a method for calibrating the external parameters of a camera. During the driving process of the vehicle, the on-vehicle camera collects multiple frames of images (for example, M frames). The feature points of the static scene in the first frame of the image are obtained through a feature point detection algorithm, and the positions of the feature points in the subsequent images (for example, the second frame of the image to the Mth frame of the image) are obtained through optical flow tracking. According to the image coordinates of the feature points of the static scene (for example, the ith frame) on the image, the preset external parameters of the camera, and the pose change of the vehicle during the interval time when the camera collects multiple frames of images (for example, the ith frame of the image and the (i + 1)th frame of the image), the predicted image coordinates of the feature points on the image (the (i + 1)th frame of the image) are obtained. According to the error between the predicted image coordinates of the feature points on the image and the image coordinates of the feature points on the image, the preset external parameters can be optimized, and thus the calibration result of the external parameters of the camera can be obtained.

[0032] Thus, during the driving process of the vehicle, the preset external parameters can be optimized and calibrated by collecting the feature points of multiple frames of images by the camera and the pose change of the vehicle during the interval time when multiple frames of images are collected, that is, the calibration of the external parameters of the camera can be completed without relying on lane lines and vanishing points, improving the accuracy of vehicle positioning and environmental perception.

[0033] In some embodiments, the external parameters of the camera can be calibrated in multiple time periods to obtain multiple external parameter calibration results, and statistical analysis is performed based on the multiple external parameter calibration results to determine the final calibration result of the external parameters.

[0034] Taking the M frames of images collected by the camera in the first time period as an example, the first calibration result of the external parameters of the camera is obtained for illustration.

[0035] Figure 1 Taking obtaining the first calibration result of the external parameters of the camera as an example, a schematic flowchart of the method for calibrating the external parameters of the camera is shown. Figure 1 The execution subject of each step of the shown process is the on-vehicle device. For the sake of convenience of description, the following is introduced Figure 1The execution entity of each step in the shown process will not be repeatedly described for each step.

[0036] As Figure 1 shown, according to some embodiments, the process of determining the external camera calibration method includes but is not limited to the following steps: S101: Obtain M frames of images collected by the vehicle's camera.

[0037] It can be understood that during the driving process of the vehicle, as one of the core sensors, the camera needs to be able to perceive the environmental elements around the vehicle. For example, during the driving process of the vehicle, environmental images are collected through the camera, where the environmental images represent the environmental elements around the vehicle, including dynamic scenes (such as pedestrians, vehicles, bicycles, etc.) and static scenes (such as road signs, trees, etc.).

[0038] In some embodiments, during the first period of the vehicle's driving, the camera can collect N frames of consecutive images, where N is greater than M. It can be understood that when performing external calibration, it is not necessary to process all N frames of consecutive images, and M key frame images can be selected from the N frames of consecutive images , and based on the M key frame images complete the external calibration. Among them, when selecting M key frame images from the N frames of consecutive images , the scene change between adjacent key frame images needs to meet a preset condition.

[0039] For example, if the scene displacement change between the previous key frame image and the current (consecutive) frame image exceeds a certain threshold (for example, 0.5 cm), it indicates that the scene has changed significantly, and the current (consecutive) frame image needs to be inserted as a key frame.

[0040] Another example is that through optical flow tracking, it is found that the proportion of the scene overlap area between the current (consecutive) frame image and the previous key frame image is less than a certain threshold (for example, less than 70%), which indicates that the scene has changed significantly, and the current (consecutive) frame image needs to be inserted as a key frame.

[0041] S102: Determine the feature points on each frame of the image.

[0042] It can be understood that the feature points are points with significant characteristics in the image (such as the corner points where the edges of two objects intersect, the edge points of the object), and are usually used to represent the key information in the image.

[0043] It can be understood that for the M key-frame images collected by the camera, the feature points of the first frame image are determined by the feature point detection algorithm, and the feature points of the (i + 1)-th frame image are obtained by optical flow tracking of the feature points of the i-th frame image. For example, the feature points of the first frame image are determined by the feature point detection algorithm, and the feature points of the second frame image are obtained by optical flow tracking of the positions of the feature points of the first frame image in the second frame image.

[0044] It can be understood that when determining the feature points on the first frame image, the environmental scenes on the first frame image include static scenes (such as road signs, trees, etc.) and also dynamic scenes (such as pedestrians, vehicles, bicycles, etc.). During the driving of the vehicle, the change in the position of the dynamic scenes in the environmental scenes may affect the calibration of the external parameters. Therefore, in some embodiments, when determining the feature points of the first frame image, pixel-level classification of the first frame image is performed through semantic segmentation perception (such as classifying into vehicles, pedestrians, roads, etc.), and the dynamic scenes (such as vehicles, pedestrians, etc.) and static scenes (such as roads, etc.) of the pixel-level classification are identified, and only the feature points of the static scenes are extracted as the feature points on the first frame image; alternatively, the feature points of the dynamic scenes can be filtered to obtain the feature points of the static scenes as the feature points on the first frame image, and no specific limitation is made here.

[0045] In some embodiments, due to reasons such as object occlusion and unclear vision, optical flow tracking may not capture the positions of the feature points in the previous frame image in the next frame image, resulting in the loss of tracking of the feature points in the next frame image, thereby affecting subsequent motion estimation. Therefore, in some embodiments, for the feature points with lost optical flow tracking, additional feature points can be supplemented (for example, the method of corner extraction). Alternatively, the feature point can be directly removed, and this feature point is not included in the calibration of the camera external parameters, and no specific limitation is made here.

[0046] S103: Calculate the predicted image coordinates of the feature points on the (i + 1)-th frame image according to the image coordinates of the feature points on the i-th frame image, the preset external parameters of the camera, and the i-th distance parameter.

[0047] It can be understood that the image coordinates of the feature points can be determined from the images collected by the camera. The preset external parameters of the camera are the initial external parameters of the camera coordinate system relative to the vehicle body coordinate system. The i-th distance parameter characterizes the pose change of the vehicle during the acquisition interval between the i-th frame image and the (i + 1)-th frame image, where the pose change of the vehicle includes the position change of the vehicle and the attitude change of the vehicle. For example, the position change of the vehicle can be the distance traveled by the vehicle during the acquisition interval between two frames of images; the attitude change of the vehicle can be the orientation change of the vehicle during the acquisition interval between two frames of images.

[0048] Taking the feature point P as an example below, the predicted image coordinates of the feature points on the (i + 1)-th frame image The process will be described.

[0049] First, the coordinates of the feature point P on the i-th frame image in the camera coordinate system will be described. will be described.

[0050] In some embodiments, when the camera captures an image, the image coordinates on the image can be determined. For example, when the camera captures the i-th frame image, the image coordinates of the feature point P on the i-th frame image are .

[0051] In some embodiments, in order to represent the relationship between the coordinates of the feature point in the camera coordinate system and the image coordinates, the normalized coordinates in the camera coordinate system (hereinafter referred to as normalized coordinates) are introduced. Among them, the normalized coordinates are the plane coordinates in the camera coordinate system obtained by converting the image coordinates, and can be expressed by the following formula: ; Among them, is the normalized coordinate of the feature point P when the i-th frame image is captured, represents the image coordinates of the feature point P on the i-th frame image, represents the conversion coefficient from the image coordinates to the normalized coordinates.

[0052] In some embodiments, according to the correspondence relationship of the image coordinates of the feature point in multiple frames of images, the depth value of the feature point (i.e., the distance from the feature point to the camera in the camera coordinate system) can be obtained through the triangulation model. For example, the depth value of the feature point P when the i-th frame image is captured can be obtained through the triangulation model , and the depth value of the feature point P when the (i + 1)-th frame image is captured can be obtained through the triangulation model .

[0053] According to the depth value of the feature point and the normalized coordinates of the feature point, the coordinates of the feature point in the camera coordinate system can be obtained, and can be expressed by the following formula: ; Among them, represents the coordinates of the feature point P in the camera coordinate system when the i-th frame image is captured, represents the depth value of the feature point P when the i-th frame image is captured, is the normalized coordinate of the feature point P when the i-th frame image is captured.

[0054] It can be understood that the distribution of the depth values is usually non-Gaussian. Especially at long distances, the distribution range of the depth values is large, and the depth values of distant objects may have a greater impact on the result of the external parameter calibration. Therefore, in order to make the collected depth values more stable and conform to the Gaussian distribution, the depth value of the feature point P when the i-th frame image is captured The inverse depth value represented as conforming to the Gaussian distribution , the coordinates of the feature point in the camera coordinate system can also be expressed by the following formula: ; Wherein, represents the coordinates of the feature point P in the camera coordinate system when the i-th frame of image is acquired, is the normalized coordinates of the feature point P when the i-th frame of image is acquired, represents the inverse depth value of the feature point P when the i-th frame of image is acquired.

[0055] Next, the predicted coordinates of the feature point P in the camera coordinate system when the (i + 1)-th frame of image is acquired will be described.

[0056] It can be understood that according to the coordinates of the feature point P in the camera coordinate system when the i-th frame of image is acquired and the i-th distance parameter and the preset external parameters of the camera, the predicted coordinates of the feature point P in the camera coordinate system when the (i + 1)-th frame of image is acquired can be obtained and can be expressed by the following formula: ; Wherein, represents the predicted coordinates of the feature point P in the camera coordinate system when the (i + 1)-th frame of image is acquired, represents the displacement of the feature point P in the camera coordinate system between the acquisition of the i-th frame of image and the (i + 1)-th frame of image, which can be represented by represented as, is the i-th distance parameter, representing the pose change of the vehicle during the acquisition interval between the i-th frame of image and the (i + 1)-th frame of image, represents the parameters of the camera coordinate system relative to the vehicle body coordinate system (i.e., the preset external parameters of the camera), represents the parameters of the vehicle body coordinate system relative to the camera coordinate system (i.e., the inverse transformation of the preset external parameters of the camera), represents the coordinates of the feature point P in the camera coordinate system when the i-th frame of image is acquired.

[0057] Next, based on the predicted coordinates of the feature point P in the camera coordinate system when the (i + 1)-th frame of image is acquired , the predicted image coordinates of the feature point P in the (i + 1)-th frame of image will be described.

[0058] Based on the predicted coordinates of the feature point P in the camera coordinate system when the (i + 1)-th frame of image is acquired and the inverse depth value of the feature point P when the (i + 1)-th frame of image is acquired , the predicted normalized coordinates of the feature point P on the (i + 1)-th frame image can be obtained, which can be expressed by the following formula: ; Wherein, represents the predicted normalized coordinates of the feature point P when collecting the (i + 1)-th frame image, represents the predicted coordinates of the feature point P in the camera coordinate system when collecting the (i + 1)-th frame image, represents the inverse depth value of the feature point P when collecting the (i + 1)-th frame image.

[0059] Project the predicted normalized coordinates of the feature point P when collecting the (i + 1)-th frame image to obtain the predicted image coordinates of the feature point P on the (i + 1)-th frame image, which can be expressed by the following formula: ; Wherein, represents the predicted image coordinates of the feature point P when collecting the (i + 1)-th frame image, represents the predicted normalized coordinates of the feature point P when collecting the (i + 1)-th frame image, represents the conversion coefficient from normalized coordinates to image coordinates.

[0060] In this way, according to the image coordinates of the feature point P on the i-th frame image, the preset external parameters of the camera, the i-th distance parameter, and the depth values of the feature point P on the i-th frame image and the (i + 1)-th frame image, the predicted image coordinates of the feature point P on the (i + 1)-th frame image can be obtained.

[0061] S104: Optimize the preset external parameters according to the error between the predicted image coordinates of the feature point on the (i + 1)-th frame image and the image coordinates of the feature point on the (i + 1)-th frame image to obtain the first calibration result of the external parameters of the camera.

[0062] In some embodiments, the image coordinates of the feature point P when collecting the (i + 1)-th frame image can be obtained by optical flow tracking , according to the predicted image coordinates of the feature point P on the (i + 1)-th frame image and the image coordinates ; Wherein, represents the error between the predicted image coordinates and the image coordinates of the feature point, represents the predicted image coordinates of the feature point P when collecting the (i + 1)-th frame image, represents the image coordinates of the feature point P on the (i + 1)-th frame image.

[0063] It can be understood that multiple predicted image coordinates and the errors of the image coordinates are obtained from w feature points on the M-frame key-frame image. By minimizing the multiple predicted image coordinates and the errors of the image coordinates, the first calibration result of the external parameters of the camera and the pseudo-depth values of the w feature points can be obtained, which can be expressed by the following formula: ; Among them, represents the M-frame image, pw represents the w feature points on the M-frame image, represents the error of the w feature points on the M-frame image, represents minimizing the predicted image coordinates and the errors of the image coordinates of the w feature points on the M-frame image, represents the inverse depth values of the w feature points, represents the parameters of the camera coordinate system relative to the vehicle body coordinate system after minimization processing (i.e., the first calibration result of the external parameters of the camera ).

[0064] In this way, the first calibration result of the external parameters of the camera can be obtained by the error between the predicted image coordinates of the w feature points on the M-frame image and the image coordinates on the feature point image .

[0065] It can be understood that when calibrating the external parameters of the camera, in order to improve the accuracy of the external parameters of the camera, it is necessary to track the feature points multiple times to obtain multiple calibration results of the external parameters, and perform statistics based on the multiple calibration results of the external parameters to obtain the final calibration result of the external parameters.

[0066] Taking the example of collecting K key-frame images after collecting M key-frame images, based on the M-frame image and the K-frame image, K + 1 calibration results of the external parameters are obtained, and the final calibration result of the external parameters of the camera is obtained through statistical analysis of the K + 1 calibration results for explanation.

[0067] Figure 2 Taking the example of obtaining K + 1 calibration results of the external parameters of the camera, a schematic flow diagram of the camera external parameter calibration method is shown. Figure 2 The execution subject of each step of the shown process is the vehicle-mounted device. For the convenience of description, the execution subject of each step will not be repeatedly described below when introducing Figure 2 each step of the shown process.

[0068] As Figure 2 shown, the method includes: S201: Obtain the images collected by the camera of the vehicle.

[0069] It can be understood that during the driving process of a vehicle, as one of the core sensors, a camera can sense the environmental elements around the vehicle.

[0070] Figure 3 It is a schematic diagram for obtaining the final calibration result of the extrinsic parameters of the camera. As Figure 3 shown, within a certain period of the vehicle's driving, the camera can collect N consecutive frames of images, and the pose changes of the vehicle when collecting N frames of images, which are used as the input for extrinsic parameter calibration.

[0071] S202: Determine the feature points on each frame of the image.

[0072] For the N consecutive frames of images collected by the camera, the feature points on each frame of the image can be obtained. As Figure 3 shown, the feature points on multiple frames of images can be obtained according to optical flow tracking. For example, the feature points of the first frame of the image are determined by a feature point detection algorithm, and the feature points of the second frame of the image are obtained by tracking the position of the feature points of the first frame of the image in the second frame of the image through optical flow.

[0073] Figure 4 It is a schematic diagram of the scene for obtaining the feature points of multiple frames of images. As Figure 4 shown, during the process of the vehicle driving on the road, the camera can collect multiple frames of images, and obtain the feature points on each frame of the image (the area occupied by the black dots in the figure) through optical flow tracking.

[0074] In some embodiments, when obtaining the feature points on the image, the feature points on the image can be preprocessed. As Figure 3 shown, when preprocessing the feature points on the image, the dynamic scenes (such as vehicles, pedestrians, etc.) and static scenes (such as roads, etc.) on the image are recognized through semantic segmentation perception, and the feature points of the static scenes are extracted as the feature points on the image; alternatively, the feature points of the dynamic scenes can be filtered to obtain the feature points of the static scenes as the feature points of the image, which is not specifically limited here.

[0075] Figure 5 It is a schematic diagram of the scene for extracting the feature points of static scenes in the environmental scene. As Figure 5 shown, the environmental scene on the image includes static scenes such as road signs, telegraph poles, trees, etc., and also includes dynamic scenes such as vehicles. When preprocessing the feature points on the image, the feature points of the dynamic scenes of the vehicles are filtered, and the feature points of the static scenes such as road signs, telegraph poles, trees, etc. on the image are obtained, resulting in the feature points that only contain static scenes (such as Figure 5 the white feature points in the figure).

[0076] S203: Store the feature points of each frame of the image, and obtain the depth values of the feature points on each frame of the image.

[0077] It can be understood that, in order to analyze the moving positions of feature points in multiple frames of images, the same feature points on each frame of image can be uniformly stored so that the feature points are associated in multiple frames of images. For example, the coordinates of feature point P on the first frame of image , the coordinates on the second frame of image ... the coordinates on the Nth frame of image are uniformly stored in the feature point set . The coordinates of feature point Q on the first frame of image , the coordinates on the second frame of image ... the coordinates on the Nth frame of image are uniformly stored in the feature point set .

[0078] It can be understood that, according to the image coordinates of multiple frames of images collected by the camera, the depth value (i.e., the distance from the feature point to the camera in the camera coordinate system) and the inverse depth value of feature point P when the camera collects the ith frame of image are obtained through the triangulation model. The specific method is the same as that in step S103 and will not be elaborated here.

[0079] S204: Select M key-frame images.

[0080] It can be understood that during the driving process of the vehicle, the camera can collect N consecutive frames of images. When performing external parameter calibration on the feature points on the images, M key-frame images can be selected from the N consecutive frames of images for external parameter calibration. When selecting M key-frame images from the N consecutive frames of images, the first frame of image is initialized as a key-frame image. For whether to select the current ith frame of image (2 ≤ i ≤ N) as a key-frame, the method is as follows: When the running distance between the feature points on the previous key-frame image and the feature points on the current ith frame (consecutive frame) image is greater than a certain threshold, it indicates that the scene has changed significantly, and it is necessary to insert the current ith frame (consecutive frame) image as a key-frame.

[0081] When the feature points on the previous key-frame can be tracked to the current ith frame (consecutive frame) image according to optical flow, and the proportion of the feature points on the current ith frame (consecutive frame) image occupying the feature points on the previous key-frame is less than a certain threshold (for example, less than 70%), it indicates that the scene has changed significantly, and it is necessary to insert the current ith frame (consecutive frame) image as a key-frame.

[0082] S205: Determine whether the number of key-frame images reaches M frames.

[0083] If the current key-frame images do not meet the quantity requirement (reach M frames), execute step S204 to select key-frame images from the N consecutive frames of images collected. If the current key-frame images meet the quantity requirement (reach M frames), execute step S206.

[0084] In some embodiments, a plurality of selected key-frame images may be placed into a sliding window. It can be understood that the sliding window is a set for storing a certain number of key-frame images. Continuing as Figure 3 shown, the selected key-frame images may be placed into the sliding window, and it is determined whether the key-frame images meet the quantity requirement according to the sliding window.

[0085] It can be understood that when the number of key-frame images in the sliding window reaches a certain size (for example, 5 key-frame images), it indicates that the image has sufficient feature points to calculate the inverse depth value and the external parameters, and the key-frame images in the sliding window may be sent to step S206.

[0086] When the number of key-frame images in the sliding window does not reach a certain size (for example, 5 key-frame images), it indicates that there are fewer feature points in the image, and the error is relatively large when calculating the inverse depth value and the external parameters, and step S204 is returned for execution.

[0087] When M key-frame images are obtained, the corresponding processed consecutive-frame images are denoted as G, and N > G > M is satisfied.

[0088] S206: Obtain the first calibration result of the external parameters of the camera.

[0089] According to steps S101 to S104, the first calibration result of the external parameters of the camera and the inverse depth values of w feature points in M frames of images are obtained .

[0090] S207: Obtain K key-frame images collected by the vehicle's camera.

[0091] Repeat steps S204 - S205 from the N - G consecutive-frame images until K key frames are obtained, denoted as .

[0092] For example, within the first time period, the camera may collect M key-frame images , and after the first time period, the camera continues to collect K key-frame images .

[0093] S208: Obtain the K + 1 calibration results of the camera external parameters.

[0094] After K key-frame images are collected, combined with the M key-frame images obtained within the first time period, a total of M + K key-frame images are obtained, denoted as .

[0095] Repeat step S206 according to the j-th to M-th images among the M key-frame images and the 1st to (j - 1)-th images among the K key-frame images to optimize the preset external parameters, and obtain the j-th calibration result of the external parameters of the camera. , where j is a positive integer that takes values in sequence from 2 to K + 1.

[0096] Take values for j in sequence and repeat the above operations respectively to obtain the 2nd to (K + 1)-th calibration results of the external parameters of the camera. .

[0097] According to the 1st calibration result and the 2nd to (K + 1)-th calibration results , obtain K + 1 calibration results .

[0098] S209: Determine whether the statistical result of the K + 1 calibration results meets the specified threshold.

[0099] According to the 1st calibration result of the external parameters of the camera in steps S101 to S104 and the 2nd to (K + 1)-th calibration results of the external parameters of the camera in step S208 , determine the K + 1 calibration results .

[0100] Continue as Figure 3 shown, perform histogram statistics on the K + 1 calibration results to obtain the statistical result of the K + 1 calibration results.

[0101] If the quantity and variance of the statistical result of the K + 1 calibration results meet the specified threshold, execute step S210; if the quantity and variance of the statistical result of the K + 1 calibration results do not meet the specified threshold, return to execute step S201.

[0102] In some embodiments, performing histogram statistical processing on the K + 1 calibration results to obtain the statistical result of the K + 1 calibration results can be expressed by the following formula: ; where represents the final calibration result of the external parameters of the camera when the quantity and variance of the statistical result meet the specified threshold (determined in step S210), represents the K + 1 external parameter calibration results, represents performing histogram statistical processing on the K + 1 external parameter calibration results.

[0103] It can be understood that by performing histogram statistical processing on the K + 1 external parameters ( Perform histogram statistical processing on the calibration results (for example, determine whether the number of K+1 external parameter calibration results meets the requirements and calculate whether the variance meets the threshold), and determine whether the statistical results of the K+1 calibration results meet the specified threshold. If the statistical results of the K+1 calibration results meet the specified threshold, execute step S210; if the statistical results of the K+1 calibration results do not meet the specified threshold, it indicates that the external parameter calibration is not accurate enough, and more frame image data needs to be collected again, and return to step S201.

[0104] S210: Obtain the final calibration result of the external parameters of the camera.

[0105] By performing histogram statistical processing on K+1 external parameter calibration results and determining that the statistical results of the K+1 calibration results meet the specified threshold, it indicates that the external parameter calibration of the camera is completed, and the external parameter with the highest frequency of occurrence can be selected from the histogram as the final calibration result of the external parameters of the camera , thus realizing the calibration of the external parameters.

[0106] In this way, according to the difference between the position of the predicted feature point on the image and the position of the actual feature point on the image, the preset external parameters are optimized, so that the calibration of the external parameters of the camera can be completed without lane lines during vehicle driving.

[0107] Exemplarily, Figure 6 According to some embodiments of the present application, a schematic structural diagram of an in-vehicle device 100 is shown. As Figure 6 shown, the in-vehicle device 100 includes one or more processors 101, a system memory 102, a non-volatile memory (NVM) 103, a communication interface 104, an input / output (I / O) device 105, and a system control logic unit 106 for coupling the processor 101, the system memory 102, the non-volatile memory 103, the communication interface 104, and the input / output device 105. Among them: The processor 101 can be used to control the vehicle-mounted device 100 to execute the calibration method of the external parameters of the camera in this application. Among them, the processor 101 can include one or more processing units. For example, it can include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro-programmed control unit (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit of a field programmable gate array (FPGA). The processor 101 can include one or more single-core or multi-core processors.

[0108] The processor 101 can be used to control the vehicle-mounted device 100 to execute the calibration method of the external parameters of the camera in this application. Among them, the processor 101 can include one or more processing units. For example, it can include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro-programmed control unit (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit of a field programmable gate array (FPGA). The processor 101 can include one or more single-core or multi-core processors. In some embodiments, the processor 101 can be used to execute the calibration method of the external parameters of the camera in the embodiments of the present invention.

[0109] The system memory 102 is a volatile memory, such as a random-access memory (RAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), etc. The system memory 102 is used to temporarily store data and / or instructions.

[0110] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card, etc.

[0111] Specifically, the system memory 102 and the non-volatile memory 103 may respectively include: a temporary copy and a permanent copy of the instruction 107. The instruction 107 may include: when executed by the processor 101, enabling the vehicle-mounted device 100 to implement the calibration method of the external parameters of the camera provided in the embodiments of the present application.

[0112] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the vehicle-mounted device 100, and then communicating with any other suitable device through one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the vehicle-mounted device 100. For example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the vehicle-mounted device 100 may communicate with other devices through the communication interface 104.

[0113] The input / output device 105 may include input devices such as a keyboard, a mouse, etc., and output devices such as a display, etc. The user may interact with the vehicle-mounted device 100 through the input / output device 105.

[0114] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface for other modules of the vehicle-mounted device 100. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide an interface connected to the system memory 102 and the non-volatile memory 103.

[0115] In some embodiments, at least one of the processors 101 may be logically packaged with one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated with the logic of one or more controllers for the system control logic unit 106 on the same chip to form a system on chip (SoC).

[0116] It can be understood that Figure 6 The structure of the vehicle-mounted device 100 shown is only an example. In some other embodiments, the vehicle-mounted device 100 may include more or fewer components than those shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0117] In some embodiments, the embodiments of the present application also provide a computer-readable storage medium. At least one computer program instruction, at least one program segment, a code set, or an instruction set is stored in the computer-readable storage medium. The at least one computer program instruction, at least one program segment, a code set, or an instruction set is loaded and executed by the vehicle-mounted device to implement the calibration method for the external parameters of the camera provided in each of the above method embodiments.

[0118] In some embodiments, the embodiments of the present application also provide a computer program product. The program product includes computer instructions. When executed by the vehicle-mounted device, the vehicle-mounted device executes the calibration method for the external parameters of the camera provided in each of the above method embodiments.

[0119] The embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system. The programmable system includes at least one processor, a storage system (including volatile and non-volatile memories and / or storage elements), at least one input device, and at least one output device.

[0120] The program code can be applied to the input instructions to execute the various functions described in the present application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of the present application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0121] The program code can be implemented in a high-level procedural language or an object-oriented programming language to communicate with the processing system. When necessary, the program code can also be implemented in assembly language or machine language. In fact, the mechanism described in the present application is not limited to the scope of any specific programming language. In any case, the language can be a compiled language or an interpreted language.

[0122] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer-readable media. Thus, machine-readable media can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to, floppy disks, optical disks, optical discs, CD-ROMs, magneto-optical discs, read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) in electrical, optical, acoustic, or other forms using the Internet. Thus, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0123] In the drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or ordering may not be required. Rather, in some embodiments, these features may be arranged in a different manner and / or order than shown in the illustrative drawings. Additionally, the inclusion of a structural or method feature in a particular figure does not imply that such a feature is required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0124] It should be noted that each unit / module mentioned in the device embodiments of this application is a logical unit / module. Physically, a logical unit / module may be a physical unit / module, a part of a physical unit / module, or may be implemented as a combination of multiple physical units / module. The physical implementation manner of these logical units / modules themselves is not the most important. The combination of the functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. In addition, to highlight the innovative part of this application, the above device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems proposed in this application, which does not mean that there are no other units / modules in the above device embodiments.

[0125] It should be noted that in the examples and the description of this patent, the terms "comprising", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one" does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0126] Although this application has been illustrated and described by reference to certain embodiments thereof, those of ordinary skill in the art should understand that various changes may be made thereto in form and detail without departing from the spirit and scope of this application.

Claims

1. A camera extrinsic parameter calibration method, applied to vehicle-mounted equipment, characterized in that: include: During the driving of the vehicle, M frames of images captured by the camera of the vehicle are obtained, where the images are environment images of the vehicle, and M is a positive integer greater than 2; Determine feature points on each frame of the image, wherein the feature points of the first frame of the image are determined by a feature point detection algorithm, and the feature points of the i+1th frame of the image are obtained by performing optical flow tracking on the feature points of the i-th frame of the image, where i is a positive integer ranging from 1 to (M-1); Calculate the predicted image coordinates of the feature point on the i+1th frame image according to the image coordinates of the feature point on the i-th frame image, the preset external parameters of the camera, and the i-th distance parameter; wherein the i-th distance parameter is the change in the posture of the vehicle within the acquisition interval between the i-th frame image and the i+1-th frame image; According to the error between the predicted image coordinates of the feature point on the i+1th frame image and the image coordinates of the feature point on the i+1th frame image, the preset extrinsic parameters are optimized to obtain a first calibration result of the extrinsic parameters of the camera.

2. The method according to claim 1, characterized in that Determining the feature points of the first frame of image includes: Identifying environmental scenes on the first frame of image, wherein the environmental scenes include dynamic scenes and static scenes; Feature points are selected on the static scene using the feature point detection algorithm to determine the feature points of the first frame image.

3. The method according to claim 1, characterized in that: The step of obtaining the feature points of the i+1th frame image by performing optical flow tracking on the feature points of the i-th frame image comprises: If the optical flow tracking of the feature points of the i-th frame image is successful, the feature points obtained by the optical flow tracking are the feature points of the i+1-th frame image; If the optical flow tracking of the feature points of the i-th frame image fails, the feature points for which the optical flow tracking fails are supplemented by corner point extraction to obtain the feature points of the i+1-th frame image.

4. The method according to claim 1, characterized in that The step of calculating the predicted image coordinates of the feature point on the (i+1)th frame image according to the image coordinates of the feature point on the (i)th frame image, the preset external parameters of the camera, and the (i)th distance parameter, further includes: According to the image coordinates of the feature point on the first frame image to the i-th frame image, a depth value of the feature point in the i-th frame image captured by the camera is calculated based on a triangulated model, wherein the depth value is the distance from the feature point to the camera; According to the depth value of the feature point in the i-th frame image captured by the camera, the image coordinates of the feature point on the i-th frame image, the preset external parameters of the camera, and the i-th distance parameter, the predicted image coordinates of the feature point on the i+1-th frame image are calculated.

5. The method according to any one of claims 1 to 4, characterized in that: The M frames of images are images collected by the camera in the first period of time; The step of obtaining M frames of images collected by a camera of the vehicle includes: Acquire N frames of images captured by the camera during the first period of time, where N is greater than M; The M frames of images are selected from the N frames of images, wherein the distance between the feature points in the M frames of images and the i-th frame of image and the i+1-th frame of image meets a preset condition.

6. The method according to claim 1, characterized in that , the method further comprises: Acquire K frames of images captured by the camera of the vehicle, where the K frames of images are images captured after the M frames of images; Obtaining K+1 calibration results of the camera extrinsic parameters according to the M frame images and the K frame images; Obtaining a final calibration result of the external parameters of the camera according to the K+1 calibration results; Wherein, obtaining K+1 calibration results of the camera extrinsic parameters according to the M frame images and the K frame images includes: According to the jth to Mth images in the M frames and the 1st to j-1th images in the K frames, a jth calibration result of the external parameters of the camera is obtained, where j is a positive integer having values ​​in the range of 2 to K+1; The K+1 calibration results are formed together with the first calibration result and the second to K+1 calibration results.

7. The method according to claim 6, characterized in that The step of obtaining a final calibration result of the external parameters of the camera according to the K+1 calibration results includes: Perform histogram statistics on the K+1 calibration results to obtain the statistical results of the external parameters; If the number and variance of the statistical results of the external parameters meet a specified threshold, a final calibration result of the external parameters of the camera is obtained according to the K+1 calibration results.

8. A vehicle-mounted device, characterized in that: The device comprises one or more processors; one or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the device executes the camera extrinsic parameter calibration method described in any one of claims 1 to 7.

9. A computer-readable medium, characterized in that The readable medium stores instructions, which, when executed on the vehicle-mounted device, enable the vehicle-mounted device to execute the camera extrinsic parameter calibration method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The program product includes computer instructions, and when executed by a vehicle-mounted device, the vehicle-mounted device executes the camera extrinsic parameter calibration method according to any one of claims 1 to 7.

Images