Vehicle camera calibration method and device, vehicle and medium

Abstract

The application discloses a kind of vehicle camera calibration method, device, vehicle and medium.The method comprises: when satisfying calibration condition, the image information of current frame photographed by vehicle camera is acquired, and the current vehicle motion information of itself vehicle;According to image information and the preset field of view angle boundary condition, the ground feature point set of image information is determined;According to ground feature point set, determined historical ground feature point information and current vehicle motion information, vehicle camera is calibrated.Through vehicle driving and photographing image information under any scene, according to image information and the preset field of view angle boundary condition, ground attribute feature is screened out in combination with historical ground feature point information and current vehicle motion information, and the external parameter of vehicle camera is determined to complete calibration.It realizes automatic calibration of vehicle camera under any scene, improves the calibration accuracy of vehicle camera, and reduces the requirement of calibration scene.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus, vehicle, and medium for calibrating vehicle-mounted cameras. Background Technology

[0002] Research on autonomous driving has been booming in recent years. Currently, achieving a certain level of safety for autonomous driving systems requires breakthroughs in fundamental research and key technologies. Vision-based environmental perception aims to detect and track targets such as roads, static objects, and dynamic objects, and to make predictions based on the tracking results. Accurate target localization is crucial in this process. Camera extrinsic calibration calculates the relative coordinate transformation between the camera position and the vehicle, which is the foundation for accurate target localization.

[0003] Currently, lane-line-based methods are commonly used for online calibration of some parameters.

[0004] However, this method often uses lane line parallelism, fixed width, and geometric features as constraints. It generally requires clear lane line information (two or three lane lines) in the calibration environment and restricts vehicles to travel slowly and only in straight lines. There are many constraints, and when these constraints are not met, the calibration accuracy will decrease. This method has poor application flexibility. Summary of the Invention

[0005] This invention provides a method, apparatus, vehicle, and medium for calibrating vehicle-mounted cameras, enabling calibration of vehicle-mounted cameras in any scenario.

[0006] According to a first aspect of the present invention, a method for calibrating an in-vehicle camera is provided, comprising:

[0007] When the calibration conditions are met, the image information of the current frame captured by the vehicle camera, as well as the current vehicle motion information of the vehicle itself, are obtained.

[0008] Based on the image information and the preset field of view boundary conditions, determine the set of ground feature points of the image information;

[0009] The vehicle-mounted camera is calibrated based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle movement information.

[0010] According to a second aspect of the present invention, a vehicle-mounted camera calibration device is provided, comprising:

[0011] The information acquisition module is used to acquire the image information of the current frame captured by the vehicle-mounted camera, as well as the current vehicle motion information of the vehicle itself, when the calibration conditions are met.

[0012] The set determination module is used to determine the set of ground feature points of the image information based on the image information and preset field of view boundary conditions;

[0013] The camera calibration module is used to calibrate the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle motion information.

[0014] According to a third aspect of the present invention, a vehicle is provided, the vehicle comprising:

[0015] At least one controller;

[0016] At least one camera communicatively connected to the at least one controller, and

[0017] A memory communicatively connected to the at least one controller; wherein,

[0018] The memory stores a computer program that can be executed by the at least one controller, which enables the at least one controller to perform the vehicle camera calibration method according to any embodiment of the present invention.

[0019] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a controller to execute and implement the vehicle camera calibration method according to any embodiment of the present invention.

[0020] The technical solution of this invention acquires image information of the current frame captured by the vehicle-mounted camera and the current vehicle motion information when calibration conditions are met; determines the set of ground feature points of the image information based on the image information and preset field-of-view boundary conditions; and calibrates the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle motion information. By having the vehicle drive and capture image information in any scenario, and by selecting ground attribute features based on the image information and preset field-of-view boundary conditions, combined with historical ground feature point information and current vehicle motion information, the extrinsic parameters of the vehicle-mounted camera are determined to complete the calibration. This achieves automatic calibration of the vehicle-mounted camera in any scenario, improves the calibration accuracy of the vehicle-mounted camera, and reduces the requirements of the calibration scenario.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a vehicle-mounted camera calibration method according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a vehicle-mounted camera calibration method according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a vehicle-mounted camera calibration device according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a structural schematic diagram of a vehicle that implements an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1This is a flowchart illustrating a vehicle-mounted camera calibration method according to Embodiment 1 of the present invention. This embodiment is applicable to calibrating vehicle-mounted cameras during driving. The method can be executed by a vehicle-mounted camera calibration device, which can be implemented in hardware and / or software and can be configured in a vehicle. Figure 1 As shown, the method includes:

[0031] S110. When the calibration conditions are met, acquire the image information of the current frame captured by the vehicle-mounted camera, as well as the current vehicle motion information of the vehicle itself.

[0032] In this embodiment, calibration conditions can be understood as the failure to obtain optimal calibration parameters, requiring further calibration. The vehicle-mounted camera can be understood as a device used for taking pictures within the vehicle. Image information can be understood as an image captured relative to the front of the vehicle. Current vehicle motion information can be understood as information representing changes in the vehicle's position.

[0033] Specifically, when calibration conditions are met, the controller can first obtain initialization parameters. These parameters can be obtained through pre-set intrinsic and distortion parameters, or through calibration using the Zhang Zhengyou calibration algorithm. Alternatively, it can select a suitable distortion correction model based on the field of view (FOV) of the vehicle-mounted camera to obtain intrinsic and distortion parameters. Theoretical extrinsic parameters are calculated based on the vehicle-mounted camera's mounting position on the vehicle body, and these intrinsic, distortion, and extrinsic parameters are used as initialization parameters. The controller then configures the vehicle-mounted camera and can acquire image information from the current frame captured by the camera through appropriate transmission methods. Furthermore, the controller can obtain the vehicle's current motion information from its own chassis.

[0034] S120. Based on the image information and the preset field of view boundary conditions, determine the set of ground feature points of the image information.

[0035] In this embodiment, the field-of-view boundary condition can be understood as a condition used to filter the ground within the field of view of the vehicle-mounted camera. The set of ground feature points can be understood as a set of multiple relatively obvious and stable feature points in the image.

[0036] Specifically, the controller can extract features from the image information using the corresponding feature extraction algorithm. Since the image information is two-dimensional, each extracted feature point is a feature in two-dimensional coordinates. Since the field of view boundary condition is the position in three-dimensional coordinates, the controller can first perform distortion correction calculation and inverse perspective projection transformation on the feature set to convert the two-dimensional coordinates into three-dimensional coordinates. Then, the controller determines the ground feature points that fall within the field of view boundary condition and uses each ground feature point as the ground feature point set.

[0037] S130. Based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle movement information, calibrate the vehicle-mounted camera.

[0038] In this embodiment, historical ground feature point information can be understood as ground feature points that meet the requirements and the corresponding vehicle positions obtained before the current frame during this calibration process.

[0039] Specifically, the controller can match and track feature points from two adjacent frames in the ground feature set and historical ground feature point information, record the tracking trajectory and the corresponding vehicle motion information, and when the continuous tracking trajectory reaches a set number of frames and the number of feature points included meets the requirements, the controller transforms the coordinates of the ground feature points using the inverse perspective projection transformation formula based on the continuous trajectory information and the current vehicle motion information, calculates the world point coordinates relative to the vehicle position based on the vehicle motion information, calculates the optimal extrinsic parameters using the least squares method, updates the current extrinsic parameters, and records the average reprojection deviation. If the deviation is less than the threshold, the accuracy requirement is met, the optimized extrinsic parameters are output, and the online calibration ends; otherwise, the acquisition of image information continues.

[0040] The technical solution of this invention acquires image information of the current frame captured by the vehicle-mounted camera and the current vehicle motion information when calibration conditions are met; determines the set of ground feature points of the image information based on the image information and preset field-of-view boundary conditions; and calibrates the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle motion information. By having the vehicle drive and capture image information in any scenario, and by selecting ground attribute features based on the image information and preset field-of-view boundary conditions, combined with historical ground feature point information and current vehicle motion information, the extrinsic parameters of the vehicle-mounted camera are determined to complete the calibration. This achieves automatic calibration of the vehicle-mounted camera in any scenario, improves the calibration accuracy of the vehicle-mounted camera, and reduces the requirements of the calibration scenario.

[0041] Example 2

[0042] Figure 2 This is a flowchart of a vehicle-mounted camera calibration method provided in Embodiment 2 of the present invention. This embodiment is a further refinement based on the above embodiments. Figure 2 As shown, the method includes:

[0043] S201. Obtain the image information of the current frame captured by the vehicle-mounted camera, as well as the current vehicle motion information of the vehicle itself.

[0044] S202. Extract features from the image information to obtain the feature information of the image information.

[0045] In this embodiment, feature information can be understood as feature information in two-dimensional coordinates.

[0046] Specifically, the controller can use a set algorithm to extract features from the image features to obtain the feature information of the image information.

[0047] For example, the SIFT algorithm can be selected for feature extraction. The SIFT algorithm is invariant to image scale and rotation, and also has a certain degree of stability to viewpoint transformation. In addition, SIFT features are rich in information and have good robustness. Therefore, SIFT feature information can be obtained by extracting features from image information through the SIFT algorithm.

[0048] S203. Perform projection transformation on the key point coordinates included in the feature information to obtain the projected ground position coordinates.

[0049] In this embodiment, keypoint coordinates can be understood as representing the position of feature points in the image in coordinate form. Ground position coordinates can be understood as three-dimensional coordinates.

[0050] Specifically, distortion correction calculations and inverse perspective projection transformations are performed on the key point coordinates included in the feature information. The controller can use the pinhole imaging model commonly used by camera sensors to describe the geometric relationship between the coordinate system relative to the vehicle body and the camera mounting position, and perform inverse perspective projection calculations to obtain the ground position coordinates.

[0051] For example, the controller can perform inverse perspective projection calculations using the imaging relationship described by the following formula to obtain the three-dimensional ground position coordinates.

[0052]

[0053] Where X, Y, and Z represent coordinates in the world coordinate system, A is the intrinsic parameter matrix, R is the rotation matrix, t is the translation vector, M is the perspective projection matrix, m is the element included in the perspective projection matrix, and x, y, and Zc are coordinates in the camera coordinate system (i.e., keypoint coordinates).

[0054] S204. Determine the target position coordinates that satisfy the field of view boundary conditions from the ground position coordinates.

[0055] In this embodiment, the target location coordinates can be understood as the location coordinates of the ground within the viewing angle range.

[0056] It's important to know that vehicle cameras typically have a field of view (FOV), which is the range of the camera's field of view; it can only capture images within this angular range. Therefore, the boundary conditions within the field of view, i.e., the boundary coordinates of the points within the FOV, can be calculated using the FOV.

[0057] Specifically, the controller can compare the ground position coordinates with the boundary coordinates in the field of view boundary conditions to determine whether the ground position coordinates fall within the boundary of the field of view. If so, the ground position coordinates are considered to meet the field of view boundary conditions and are used as the target position coordinates.

[0058] S205. Determine the set of ground feature points for the image information based on the coordinates of each target location.

[0059] Specifically, the controller can integrate the coordinates of each target location as a set of ground feature points for the image information.

[0060] S206. Extract the historical tracking trajectory information of each adjacent frame from the historical ground feature point information.

[0061] In this embodiment, historical tracking trajectory information can be understood as information formed by the tracking trajectory determined by adjacent frames and the corresponding vehicle motion information.

[0062] Specifically, the controller can extract historical tracking trajectory information from each adjacent frame in the historical ground feature point information.

[0063] S207. Perform feature point matching and tracking on the previous frame's feature point set and the ground feature point set in the historical ground feature point information to obtain the current tracking trajectory information.

[0064] In this embodiment, the previous feature point set can be understood as the set of ground feature points calculated in the previous frame using the method described above. The current tracking trajectory information can be understood as the trajectory of the same point for two consecutive frames.

[0065] Specifically, the controller can extract the previous set of feature points from the previous frame in the historical ground feature point information according to the order corresponding to the frame number, and perform feature point matching and tracking on the ground feature points included in the previous set of feature points and the ground feature points included in the ground feature point set of this frame, that is, determine the motion trajectory of the same feature points in the current frame and obtain the current tracking trajectory information.

[0066] S208. Based on the current tracking trajectory information and the historical tracking trajectory information, determine the continuous trajectory information of the tracking trajectory.

[0067] In this embodiment, continuous trajectory information can be understood as trajectory information when the same point appears in different frames.

[0068] Specifically, the controller can determine continuous trajectory information in which the same feature points appear consecutively in these historical tracking trajectory information based on the current tracking trajectory information and other historical tracking trajectory information.

[0069] S209. When the continuous trajectory information meets the optimization extrinsic parameter conditions, determine the extrinsic parameter information of the vehicle camera and calibrate the vehicle camera based on the continuous trajectory information and the current vehicle motion information.

[0070] The optimization of external parameters includes: the number of consecutive frames included in the continuous trajectory information reaches a preset frame number threshold, and the number of feature points included in the continuous trajectory information reaches a preset number threshold.

[0071] Specifically, when the number of consecutive frames and the number of feature points in the continuous trajectory information both meet the conditions for optimizing extrinsic parameters, the coordinates of the ground feature points are transformed using the inverse perspective projection transformation formula based on the continuous trajectory information and the current vehicle motion information. The world point coordinates relative to the vehicle position are calculated based on the vehicle motion information. The optimal extrinsic parameters are calculated using the least squares method and updated to the current extrinsic parameters. The average reprojection deviation is recorded. If the deviation is less than the threshold, the accuracy requirement is met. The optimized extrinsic parameter information is output and the vehicle camera is calibrated.

[0072] a1. Extract the set of continuous feature points and the set of vehicle motion information from the continuous trajectory information.

[0073] In this embodiment, the continuous feature point set can be understood as the set of feature points for each frame of the continuous trajectory information. The vehicle motion information set can be understood as the set of vehicle motion information for each frame included in the continuous trajectory information.

[0074] Specifically, the controller can extract the set of continuous feature points and the set of vehicle motion information from the continuous trajectory information.

[0075] b1. Based on the coordinates of the continuous feature points included in the continuous feature point set, determine the target vehicle motion information in the vehicle motion information set corresponding to the coordinates of the continuous feature points.

[0076] In this embodiment, the coordinates of continuous feature points can be understood as the image coordinates of continuous and stable feature points.

[0077] Specifically, the controller can determine the corresponding target vehicle motion information from the vehicle motion information set based on the feature identifiers of the feature point coordinates in the continuous feature point set.

[0078] c1. Based on the target vehicle's motion information and the coordinates of continuous feature points, determine the world point coordinates of the continuous feature points relative to the vehicle's position.

[0079] In this embodiment, vehicle position can be understood as the vehicle's location when the image is captured, which can be extracted from the target vehicle's motion information. World coordinates can be understood as coordinates in the world point coordinate system.

[0080] Specifically, the controller can perform coordinate transformation on the coordinates of continuous feature points using the inverse perspective projection transformation formula to obtain coordinates in the world coordinate system, and calculate the world point coordinates of the relative vehicle position based on the target vehicle's motion information.

[0081] d1. Determine the extrinsic parameters of the vehicle-mounted camera based on the coordinates of each continuous feature point and the corresponding world point coordinates, and calibrate the vehicle-mounted camera.

[0082] Specifically, the controller can calculate the optimal extrinsic parameters using the least squares method based on the relationship between continuous feature point coordinates and world point coordinates, combined with the relationship between image coordinates and world point coordinates. The relationship between image coordinates and world point coordinates can be expressed as the continuous feature point coordinates being equal to the product of the intrinsic parameter matrix, the rotation matrix, and the translation vector, which forms the extrinsic parameter matrix, multiplied by the world point coordinates. This product is then updated to the current extrinsic parameters. The controller calculates the average reprojection deviation based on the continuous feature point coordinates under the current extrinsic parameters and the theoretical feature point coordinates under the previous extrinsic parameters. Based on the deviation, it determines whether the optimal extrinsic parameters are the best extrinsic parameter information. If so, the vehicle camera is calibrated based on the optimal extrinsic parameters; otherwise, step S201 is continued.

[0083] Furthermore, the steps of determining the extrinsic parameters of the vehicle-mounted camera and calibrating the camera based on the coordinates of each continuous feature point and its corresponding world point coordinates can be further optimized as follows:

[0084] d11. Based on the coordinates of each continuous feature point and the corresponding world point coordinates, determine the current extrinsic parameter information and the coordinates of each optimized feature point under the current extrinsic parameter information.

[0085] In this embodiment, optimizing feature point coordinates can be understood as optimizing feature point coordinates using the current extrinsic parameter information.

[0086] Specifically, the controller can calculate the current extrinsic parameters using the least squares method based on the relationship between continuous feature point coordinates and world point coordinates, combined with the relationship between image coordinates and world point coordinates. Then, it determines the corresponding optimized feature point coordinates using the current extrinsic parameters, the world point coordinates, and the relationship between image coordinates and each world point coordinate.

[0087] d12. Determine the average projection deviation value based on the coordinates of each continuous feature point and the corresponding optimized feature point coordinates.

[0088] In this embodiment, the average projection deviation value can be understood as the average of multiple projection deviation values.

[0089] Specifically, the controller can obtain multiple projection deviation values ​​by subtracting the coordinates of continuous feature points from the coordinates of the corresponding optimized feature points, and then calculate the average projection deviation value among the multiple projection deviation values.

[0090] d13. When the average projection deviation value is less than the preset deviation threshold, the current extrinsic information is used as the extrinsic information of the vehicle camera and the vehicle camera is calibrated.

[0091] In this embodiment, the deviation threshold can be understood as a value set to determine whether the current external parameter information meets the calibration requirements.

[0092] Specifically, the controller can compare the average projection deviation value with the deviation threshold. When the average projection deviation value is less than the preset deviation threshold, the current external parameter information is used as the external parameter information of the vehicle camera and the vehicle camera is calibrated.

[0093] Furthermore, based on the above embodiments, it also includes:

[0094] When the continuous trajectory information does not meet the optimization extrinsic conditions, or when the average projection deviation value is greater than or equal to the preset deviation threshold, the current frame is taken as the previous frame, and the steps of obtaining the image information of the current frame and the current vehicle motion information are re-executed.

[0095] Specifically, when the continuous trajectory information does not meet the optimized extrinsic parameters, or when the average projection deviation value is greater than or equal to the preset deviation threshold, the controller can use the current frame as the previous frame and return to re-execute the steps of acquiring the image information of the current frame and the current vehicle motion information.

[0096] The technical solution of this invention, when calibration conditions are met, acquires the image information of the current frame captured by the vehicle-mounted camera and the current vehicle motion information of the vehicle itself. By extracting feature information from the image, transforming the features through distortion correction and inverse perspective projection, and filtering out a set of ground feature points based on the field of view boundary conditions, and performing feature point matching and tracking on the feature point sets of adjacent frames based on historical ground feature information, when the continuous trajectory information meets the optimized extrinsic parameter conditions, a stable and reliable set of continuous feature points is determined based on the current vehicle motion information, thereby determining the optimized current extrinsic parameter information, and judging whether the accuracy of the current extrinsic parameter information meets the requirements. Through multiple optimizations, extrinsic parameters that meet the requirements are optimized, realizing automatic calibration of the vehicle-mounted camera in any scenario, improving the calibration accuracy of the vehicle-mounted camera, and reducing the requirements of the calibration scenario.

[0097] Example 3

[0098] Figure 3 This is a schematic diagram of the structure of a vehicle-mounted camera calibration device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an information acquisition module 31, a set determination module 32, and a camera calibration module 33. Among them,

[0099] The information acquisition module 31 is used to acquire the image information of the current frame captured by the vehicle-mounted camera and the current vehicle motion information of its own vehicle when the calibration conditions are met.

[0100] The set determination module 32 is used to determine the set of ground feature points of the image information based on the image information and the preset field of view boundary conditions;

[0101] The camera calibration module 33 is used to calibrate the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle motion information.

[0102] The technical solution of this invention acquires image information of the current frame captured by the vehicle-mounted camera and the current vehicle motion information when calibration conditions are met; determines the set of ground feature points of the image information based on the image information and preset field-of-view boundary conditions; and calibrates the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle motion information. By having the vehicle drive and capture image information in any scenario, and by selecting ground attribute features based on the image information and preset field-of-view boundary conditions, combined with historical ground feature point information and current vehicle motion information, the extrinsic parameters of the vehicle-mounted camera are determined to complete the calibration. This achieves automatic calibration of the vehicle-mounted camera in any scenario, improves the calibration accuracy of the vehicle-mounted camera, and reduces the requirements of the calibration scenario.

[0103] Optionally, the set determination module 32 includes:

[0104] Feature extraction is performed on the image information to obtain the feature information of the image information;

[0105] The key point coordinates included in the feature information are subjected to projection transformation to obtain the projected ground position coordinates.

[0106] Determine the target position coordinates that satisfy the field of view boundary conditions from the ground position coordinates;

[0107] Based on the coordinates of each target location, a set of ground feature points for the image information is determined.

[0108] Optionally, the camera calibration module 33 includes:

[0109] The information extraction unit is used to extract the historical tracking trajectory information of each adjacent frame from the historical ground feature point information;

[0110] The first determining unit is used to perform feature point matching and tracking on the previous feature point set of the previous frame in the historical ground feature point information and the ground feature point set to obtain the current tracking trajectory information;

[0111] The second determining unit is used to determine continuous trajectory information of the tracking trajectory based on the current tracking trajectory information and each of the historical tracking trajectory information;

[0112] The camera calibration unit is used to determine the extrinsic information of the vehicle-mounted camera and calibrate the vehicle-mounted camera based on the continuous trajectory information, the initialization parameters, and the current vehicle motion information when the continuous trajectory information meets the optimized extrinsic parameter conditions.

[0113] Wherein, the number of consecutive frames included in the continuous trajectory information reaches a preset frame number threshold, and the number of feature points included in the continuous trajectory information reaches a preset number threshold.

[0114] Furthermore, the camera calibration unit includes:

[0115] The set extraction subunit is used to extract the initial acquisition time, initial vehicle position, continuous feature point set, and vehicle motion information set from the continuous trajectory information;

[0116] The first determining subunit is used to determine the target vehicle motion information in the vehicle motion information set corresponding to the coordinates of the continuous feature points included in the continuous feature point set.

[0117] The second determining subunit is used to determine the world point coordinates of the continuous feature points relative to the vehicle position based on the target vehicle motion information and the coordinates of the continuous feature points.

[0118] The third determining subunit is used to determine the extrinsic parameter information of the vehicle-mounted camera and calibrate the vehicle-mounted camera based on the coordinates of each continuous feature point and the corresponding world point coordinates.

[0119] Specifically, the third determining subunit is used for:

[0120] Based on the coordinates of each continuous feature point and the corresponding world point coordinates, determine the current extrinsic parameter information and the coordinates of each optimized feature point under the current extrinsic parameter information;

[0121] The average projection deviation value is determined based on the coordinates of each continuous feature point and the corresponding optimized feature point coordinates.

[0122] When the average projection deviation value is less than the preset deviation threshold, the current extrinsic information is used as the extrinsic information of the vehicle camera and the vehicle camera is calibrated.

[0123] The vehicle-mounted camera calibration device provided in this embodiment of the invention can execute the vehicle-mounted camera calibration method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0124] Example 4

[0125] Figure 4 This is a structural schematic diagram of a vehicle provided in Embodiment 4 of the present invention, as shown below. Figure 4 As shown, the vehicle includes a controller 41, a memory 42, an input device 43, an output device 44, and a camera 45. The number of controllers 41, memory 42, and cameras 45 can be one or more. Figure 4 Taking a controller 41 and a memory 42 as an example; the controller 41, memory 42 and camera 45 in the vehicle can be connected via a bus or other means. Figure 4 Taking a bus connection as an example, the controller refers to the controller of the execution entity in this embodiment of the invention.

[0126] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the vehicle-mounted camera calibration method in this embodiment of the invention (e.g., the information acquisition module 31, the set determination module 32, and the camera calibration module 33 in the vehicle-mounted camera calibration device). The controller 41 executes various functional applications and data processing of the vehicle by running the software programs, instructions, and modules stored in the memory 42, thereby realizing the above-described vehicle-mounted camera calibration method.

[0127] The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 42 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include memory remotely configured relative to the controller 41, which can be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The input device 43 can be used to receive digital or character information, and to generate key signal inputs related to vehicle user settings and function control.

[0129] Output device 44 may include a display device.

[0130] Example 5

[0131] Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer controller, are used for a vehicle-mounted camera calibration method, the method comprising:

[0132] It acquires the vehicle's own attitude information and multimedia data collected by the multimedia data acquisition device.

[0133] When the calibration conditions are met, the image information of the current frame captured by the vehicle camera, as well as the current vehicle motion information of the vehicle itself, are obtained.

[0134] Based on the image information and the preset field of view boundary conditions, determine the set of ground feature points of the image information;

[0135] The vehicle-mounted camera is calibrated based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle movement information.

[0136] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the vehicle camera calibration method provided in any embodiment of the present invention.

[0137] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0138] It is worth noting that in the embodiments of the above-mentioned vehicle camera calibration device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0139] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for calibrating a vehicle-mounted camera, characterized in that, include: When the calibration conditions are met, the image information of the current frame captured by the vehicle camera, as well as the current vehicle motion information of the vehicle itself, are obtained. Based on the image information and the preset field of view boundary conditions, determine the set of ground feature points of the image information; The vehicle-mounted camera is calibrated based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle movement information; The step of determining the set of ground feature points of the image information based on the image information and preset field-of-view boundary conditions includes: Feature extraction is performed on the image information to obtain the feature information of the image information; The key point coordinates included in the feature information are subjected to projection transformation to obtain the projected ground position coordinates. Determine the target position coordinates that satisfy the field of view boundary conditions from the ground position coordinates; Based on the coordinates of each target location, a set of ground feature points for the image information is determined.

2. The method according to claim 1, characterized in that, The calibration of the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle movement information includes: Extract the historical tracking trajectory information of each adjacent frame from the historical ground feature point information; Feature point matching and tracking are performed on the previous feature point set of the previous frame in the historical ground feature point information and the ground feature point set to obtain the current tracking trajectory information; Based on the current tracking trajectory information and each of the historical tracking trajectory information, continuous trajectory information is determined to be continuous. When the continuous trajectory information meets the optimized extrinsic parameter conditions, the extrinsic parameter information of the vehicle-mounted camera is determined and the vehicle-mounted camera is calibrated based on the continuous trajectory information and the current vehicle motion information.

3. The method according to claim 2, characterized in that, The optimized extrinsic conditions include: The number of consecutive frames included in the continuous trajectory information reaches a preset frame number threshold, and the number of feature points included in the continuous trajectory information reaches a preset number threshold.

4. The method according to claim 2, characterized in that, The step of determining the extrinsic parameters of the vehicle-mounted camera and calibrating the vehicle-mounted camera based on the continuous trajectory information and the current vehicle motion information includes: Extract the set of continuous feature points and the set of vehicle motion information from the continuous trajectory information; For the coordinates of the continuous feature points included in the set of continuous feature points, determine the target vehicle motion information in the set of vehicle motion information corresponding to the coordinates of the continuous feature points; Based on the target vehicle motion information and the coordinates of the continuous feature points, determine the world point coordinates of the continuous feature points relative to the vehicle position; Based on the coordinates of each continuous feature point and the corresponding world point coordinates, the extrinsic parameters of the vehicle-mounted camera are determined and the vehicle-mounted camera is calibrated.

5. The method according to claim 4, characterized in that, The step of determining the extrinsic parameter information of the vehicle-mounted camera and calibrating the vehicle-mounted camera based on the coordinates of each of the continuous feature points and the corresponding world point coordinates includes: Based on the coordinates of each continuous feature point and the corresponding world point coordinates, determine the current extrinsic parameter information and the coordinates of each optimized feature point under the current extrinsic parameter information; The average projection deviation value is determined based on the coordinates of each continuous feature point and the corresponding optimized feature point coordinates. When the average projection deviation value is less than the preset deviation threshold, the current extrinsic information is used as the extrinsic information of the vehicle camera and the vehicle camera is calibrated.

6. The method according to claim 5, characterized in that, Also includes: When the continuous trajectory information does not meet the optimized extrinsic parameter conditions, or when the average projection deviation value is greater than or equal to the preset deviation threshold, the current frame is taken as the previous frame, and the steps of obtaining the image information of the current frame and the current vehicle motion information are re-executed.

7. A vehicle-mounted camera calibration device, characterized in that, include: The information acquisition module is used to acquire the image information of the current frame captured by the vehicle-mounted camera, as well as the current vehicle motion information of the vehicle itself, when the calibration conditions are met. The set determination module is used to determine the set of ground feature points of the image information based on the image information and preset field of view boundary conditions; The camera calibration module is used to calibrate the vehicle-mounted camera based on the set of ground feature points, the determined historical ground feature point information, and the current vehicle motion information. Specifically, the set determination module is used for: Feature extraction is performed on the image information to obtain the feature information of the image information; The key point coordinates included in the feature information are subjected to projection transformation to obtain the projected ground position coordinates. Determine the target position coordinates that satisfy the field of view boundary conditions from the ground position coordinates; Based on the coordinates of each target location, a set of ground feature points for the image information is determined.

8. A vehicle, characterized in that, The vehicles include: At least one controller; At least one camera communicatively connected to the at least one controller, and A memory communicatively connected to the at least one controller; wherein, The memory stores a computer program that can be executed by the at least one controller, the computer program being executed by the at least one controller to enable the at least one controller to perform the vehicle camera calibration method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the controller to execute the vehicle camera calibration method according to any one of claims 1-6.

Images