Camera parameter calibration method and device, equipment, medium and product

By obtaining the motion trajectory data of the vehicle and camera for segment optimization, the problems of strict external parameter calibration environment requirements and poor stability of vehicle-mounted cameras in the prior art are solved, and higher accuracy and flexibility are achieved.

CN118941641APending Publication Date: 2024-11-12XUANCHENG LUXSHARE PRECISION IND CO LTD

Patent Information

Application Number
CN202410939878.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing vehicle-mounted camera external parameter calibration method has strict environmental requirements, the calibration method is not flexible enough, and the stability is poor, especially when the lane line is damaged, affecting the accuracy of the subsequent algorithm.

Method used

By acquiring the vehicle driving data and image frame sets, the motion trajectory data of the vehicle and the camera are determined, segmented processing and optimization are performed, and the final external camera parameters are obtained.

Benefits of technology

It improves the accuracy and stability of camera calibration, has strong adaptability, reduces dependence on the environment, and enhances the calibration effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941641A_ABST
    Figure CN118941641A_ABST
Patent Text Reader

Abstract

The invention discloses a camera parameter calibration method and device, equipment, a medium and a product. The method comprises the following steps: acquiring vehicle driving data and an image frame set of a vehicle to which a to-be-calibrated camera belongs; determining vehicle movement track data under the first coordinate system according to the vehicle driving data; according to the image frame set, determining camera motion track data under the second coordinate system; determining a segmented track data set after segmentation according to the vehicle motion track data and the camera motion track data; and optimizing the current external parameters of the to-be-calibrated camera based on the segmented trajectory data in the segmented trajectory data set to obtain calibrated final external parameters. According to the invention, segmentation is carried out by fusing the vehicle motion track and the camera motion track, the precision of the motion track is improved, external parameters are optimized based on segmented track data with finer precision, and final external parameters after calibration are obtained. The calibration environment is flexible, the calibration stability is improved, and the calibration effect is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of camera calibration technology, and in particular to a camera parameter calibration method, device, equipment, medium and product. Background Art

[0002] In vehicle autonomous driving and intelligent assisted driving systems, vehicle cameras play an important role in perceiving and understanding the surrounding environment. In order to ensure that the camera accurately obtains scene information, it is necessary to accurately calibrate the camera's external parameters, that is, the position and posture of the camera relative to the vehicle coordinate system.

[0003] At present, most of the extrinsic parameter calibration solutions for vehicle-mounted cameras are factory calibration, which uses known calibration plates or feature points to calibrate the camera's extrinsic parameters. This method requires the acquisition of images containing calibration plates or feature points, and the use of image processing algorithms (such as OpenCV) to extract feature points or corner points. Then, the extrinsic parameters of the camera are calculated using the calibration algorithm. There is also a multi-view calibration method, which uses multiple cameras or images from multiple viewpoints of the camera to calibrate the extrinsic parameters. This method requires matching and aligning multiple cameras or images from multiple viewpoints, and then calculating the camera's extrinsic parameters through three-dimensional reconstruction or stereo vision algorithms. Existing online calibration solutions generally use lane lines or arrange specific calibration plates on the driving section to assist in calibration.

[0004] However, the above methods have strict requirements on the calibration environment, and have accuracy requirements on the layout of the calibration plate and the distance between corner points. In addition, the calibration method is not flexible enough and can only be calibrated in a specific venue. Finally, the calibration stability of the method is poor. When the lane line is damaged, it will affect the accuracy of the subsequent algorithm. Summary of the invention

[0005] The present invention provides a camera parameter calibration method, device, equipment, medium and product to achieve accurate and stable calibration of camera parameters.

[0006] According to a first aspect of the present invention, a method for calibrating camera parameters is provided, comprising:

[0007] Acquire vehicle driving data and an image frame set of a vehicle to which the camera to be calibrated belongs, wherein the image frame set is obtained by shooting according to current camera parameters of the camera to be calibrated;

[0008] Determining vehicle motion trajectory data in a first coordinate system according to the vehicle driving data;

[0009] Determining camera motion trajectory data in a second coordinate system according to the image frame set;

[0010] Determining a segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data;

[0011] The current camera parameters are optimized based on each segmented trajectory data in the segmented trajectory data set to obtain calibrated final camera parameters.

[0012] According to a second aspect of the present invention, there is provided a camera parameter calibration device, comprising:

[0013] A data acquisition module, used to acquire vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs;

[0014] A first determination module, used to determine vehicle motion trajectory data in a first coordinate system according to the vehicle driving data;

[0015] A second determination module, used to determine the camera motion trajectory data in a second coordinate system according to the image frame set;

[0016] A data segmentation module, used to determine a segmented segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data;

[0017] The parameter calibration module is used to optimize the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration.

[0018] According to a third aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the camera parameter calibration method described in any embodiment of the present invention.

[0022] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the camera parameter calibration method described in any embodiment of the present invention when executed.

[0023] According to a fifth aspect of the present invention, an embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the camera parameter calibration method of any embodiment of the present invention is implemented.

[0024] The technical solution of the embodiment of the present invention is to obtain the vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs; determine the vehicle motion trajectory data in the first coordinate system based on the vehicle driving data; determine the camera motion trajectory data in the second coordinate system based on the image frame set; determine the segmented segmented trajectory data set based on the vehicle motion trajectory data and the camera motion trajectory data; optimize the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration. By segmenting the fusion of the vehicle motion trajectory and the camera motion trajectory, the accuracy of the motion trajectory is improved, and the external parameters are optimized based on the segmented trajectory data with finer accuracy to obtain the final external parameters after calibration. The calibration environment is flexible, which improves the stability of the calibration and thus improves the calibration effect.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flow chart of a camera parameter calibration method provided according to Embodiment 1 of the present invention;

[0028] Figure 2 is a flow chart of a camera parameter calibration method provided according to Embodiment 2 of the present invention;

[0029] Figure 3 is a segmented example diagram of a camera parameter calibration method provided in accordance with the second embodiment of the present invention.

[0030] Figure 4 is a structural schematic diagram of a camera parameter calibration device provided according to Embodiment 3 of the present invention;

[0031] Figure 5 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 A flowchart of a camera parameter calibration method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of calibrating camera external parameters. The method can be executed by a camera parameter calibration device. The camera parameter calibration device can be implemented in the form of hardware and / or software. The camera parameter calibration device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0036] S110, obtaining vehicle driving data and a set of image frames of a vehicle to which the camera to be calibrated belongs.

[0037] In this embodiment, the camera to be calibrated can be understood as a camera that needs to calibrate external parameters, such as a vehicle-mounted camera. The vehicle can be understood as the vehicle on which the camera to be calibrated is installed. The vehicle driving data can be understood as the driving information corresponding to the movement of the vehicle, such as speed and turning angle. The image frame set can be understood as a set of multiple frames of images collected by the camera to be calibrated.

[0038] Specifically, a relatively flat road surface (such as a city road, etc.) can be selected in advance, and the vehicle to which the camera to be calibrated belongs can be controlled to go straight and turn several times. The mobile state signal is received in real time by the controller and other devices in the vehicle, and the mobile state data is processed and sent to the processor. The processor can obtain the mobile state data of the vehicle and record it in the memory. A time interval ΔT can be set, and a frame of image data can be collected and recorded every ΔT as a key frame image, and the original image collected by the camera to be calibrated can be obtained as a key frame image set. Ensure that the posture of the device and the camera to be calibrated changes greatly during the data collection process to obtain a more accurate calibration result.

[0039] Exemplarily, a flat road surface (such as a city road) can be selected in advance, the vehicle can be started, and the calibration device can be powered on through the vehicle. At this time, the MCU controller and the SOC controller of the vehicle start working. The MCU controller receives and processes the wheel pulse signal and the steering wheel angle signal data in real time to obtain the speed and front wheel angle of the vehicle. By integrating the speed and front wheel angle of the vehicle, the real-time position and yaw angle of the vehicle are obtained, and the obtained data are transmitted to the processor as the mobile state data and saved and recorded in the memory. The SOC controller in the vehicle can receive the original image data collected by the camera to be calibrated in real time, and record the original image data as a key frame image set in the memory. The specific control method can be to control the vehicle to go straight for 5-15 meters, then turn left (steering wheel angle 360° to the left) and drive 5-10 meters, and then turn right (steering wheel angle 360° to the right) and drive 5-10 meters. The actual driving distance and steering wheel angle can be adjusted appropriately according to the road conditions. Try to ensure smoothness when the steering wheel changes to achieve the purpose of smooth steering.

[0040] S120: Determine vehicle motion trajectory data in a first coordinate system according to the vehicle driving data.

[0041] In this embodiment, the first coordinate system can be understood as a coordinate system established with the vehicle as the center, for example, the vehicle's forward direction is the positive direction of the X axis, and the left direction is the positive direction of the Y axis. The vehicle motion trajectory data can be understood as a complete driving trajectory and a collection of motion states at each moment.

[0042] Specifically, the processor can take the vehicle start-up moment as the initial moment, that is, the starting point of the motion trajectory, and input the movement state data into a preset vehicle motion model, such as the Ackerman set motion model, to determine the vehicle state in the first coordinate system at each moment, and then determine the vehicle motion trajectory information in the first coordinate system.

[0043] For example, the rear axle center (RAC) position at the time of vehicle startup is used as the origin of the world coordinate system (Cartesian coordinate system), the vehicle forward direction is the positive direction of the X axis, the left direction of the vehicle is the positive direction of the Y axis, and the vehicle motion model uses the Ackerman geometric motion model, which is described as follows:

[0044] X=v*cos(ψ)

[0045] Y=v*sin(ψ)

[0046] ψ=v*tan(δ) / L

[0047] Among them, X is the x-axis coordinate value of RAC, unit is m, Y is the y-axis coordinate value of RAC, unit is m, ψ is the yaw angle (also called heading angle) of the vehicle, unit is rad, v is the vehicle speed, unit is m / s, L is the wheelbase, unit is m, and δ is the front wheel steering angle, unit is rad.

[0048] According to the motion model, the state of the vehicle is described by (X, Y, ψ), and the vehicle state (0, 0, 0) is recorded when the vehicle starts. During the driving process of the vehicle, the MCU controller records the wheel pulses in real time and calculates the vehicle's speed v and front wheel angle δ. Given the vehicle state at the current time k (X(k), Y(k), ψ(k)), the speed v and front wheel angle δ are substituted into the motion model, and the vehicle state at time k+1 (X(k+1), Y(k+1), ψ(k+1)) can be calculated. According to the initial state of the vehicle (0, 0, 0), the speed v and front wheel angle δ at each moment are substituted into the model, and the vehicle state (X(N), Y(N), ψ(N)) is obtained at all times of the vehicle's driving by iterative calculation. The MCU controller transmits the vehicle state obtained each time to the SOC controller for storage, thereby obtaining the complete motion trajectory of the vehicle, and the vehicle posture change at any time interval, that is, the vehicle state change, can also be obtained.

[0049] S130 . Determine camera motion trajectory data in a second coordinate system according to the image frame set.

[0050] In this embodiment, the second coordinate system can be understood as a coordinate system established with the camera to be calibrated as the center, for example, it can be a Cartesian coordinate system, and the camera coordinate system can use a right-hand system, with the horizontal right direction being the positive direction of the x-axis, the optical axis direction being the positive direction of the y-axis, and the vertical upward direction being the positive direction of the z-axis. The camera motion trajectory data can be understood as the trajectory of the camera to be calibrated and the collection of the state of the camera to be calibrated at each moment.

[0051] Specifically, the processor can extract feature points from the key frame image, take the posture of the camera to be calibrated in the first key frame image as the initial posture, determine the camera state in each frame image in the camera coordinate system through the change relationship between the same feature points in adjacent frames, and then determine the camera trajectory information.

[0052] S140: Determine a segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data.

[0053] In this embodiment, the segmented trajectory data set can be understood as all trajectory information after being divided into small distances.

[0054] Specifically, the processor may perform data fusion processing according to the motion trajectory information and the camera trajectory information, segment the trajectory after the fusion processing, and obtain a set consisting of all segmented trajectory data after the segmentation.

[0055] S150 , optimizing current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain final external parameters after calibration.

[0056] In this embodiment, the current external parameters can be understood as the external parameters of the currently used camera, and the final external parameters can be understood as the calibrated external parameters.

[0057] Specifically, the processor can determine the vehicle change and camera change at different times in each segmented trajectory data based on the segmented trajectory data in the segmented trajectory data set, and then determine the conversion relationship between the changes in the two coordinate systems. The current external parameters of the camera to be calibrated are optimized through the conversion relationship to obtain the final external parameters after calibration.

[0058] The technical solution of the embodiment of the present invention is to obtain the vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs; determine the vehicle motion trajectory data in the first coordinate system based on the vehicle driving data; determine the camera motion trajectory data in the second coordinate system based on the image frame set; determine the segmented segmented trajectory data set based on the vehicle motion trajectory data and the camera motion trajectory data; optimize the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration. By segmenting the fusion of the vehicle motion trajectory and the camera motion trajectory, the accuracy of the motion trajectory is improved, and the external parameters are optimized based on the segmented trajectory data with finer accuracy to obtain the final external parameters after calibration. The calibration environment is flexible, which improves the stability of the calibration and thus improves the calibration effect.

[0059] Embodiment 2

[0060] Figure 2This is a flow chart of a camera parameter calibration method provided in the second embodiment of the present invention. This embodiment is a further refinement of the above embodiment. Figure 2 As shown, the method includes:

[0061] S201, obtaining vehicle driving data and a set of image frames of a vehicle to which the camera to be calibrated belongs.

[0062] S202: Determine vehicle motion trajectory data in a first coordinate system according to the vehicle driving data.

[0063] S203: Perform background filtering on each key image frame included in the image frame set to obtain an image of a region of interest.

[0064] In this embodiment, the key image frame can be understood as each frame image included in the image frame set. The background can be understood as an area that does not contain feature points. The region of interest image can be understood as a divided image set that needs to be subjected to subsequent feature point extraction.

[0065] Specifically, the region of interest ROI (Region of Interest) of the image may be preset, and the processor may filter the region of interest for each key image frame included in the image frame set, filter out regions without feature points such as the ground and the sky, and obtain an image of the region of interest.

[0066] S204 , extracting feature points from each region of interest image to obtain a feature point image containing the feature points.

[0067] In this embodiment, the feature image can be understood as a set of images divided into feature points.

[0068] Specifically, the processor may extract ORB (Oriented FAST and Rotated BRIEF) feature points within the image ROI, obtain a feature image including the feature points, and form a feature image set.

[0069] S205 . Determine camera motion trajectory data in a second coordinate system according to each feature point image.

[0070] Specifically, the processor can set the camera motion coordinates at the initial moment in the second coordinate system to zero, determine the movement information of the camera between adjacent frame images through the feature points included in the feature point image, and determine the camera motion coordinates of the next frame by adding the movement information to the camera motion coordinates of the previous frame, and so on, to determine the camera motion trajectory data in the second coordinate system.

[0071] Further, based on the above embodiment, the step of determining the camera motion trajectory data in the second coordinate system according to each feature point image can be refined as follows:

[0072] Obtain the initial camera coordinates of the camera to be calibrated in the second coordinate system; match the feature points of the feature point images of adjacent frames to determine the matching relationship of the feature points; determine the motion relationship between the feature point images of adjacent frames based on the matching relationship of each feature point; determine the camera posture of the feature point image of each frame in the second coordinate system according to each motion relationship and the initial camera coordinates; determine the camera motion trajectory data in the second coordinate system according to each camera posture.

[0073] In this embodiment, the initial camera coordinates can be understood as the position of the camera in the second coordinate system at the start time. The feature point matching relationship can be understood as the relationship used to determine the same feature points in the previous and next frames of images. The motion relationship can be understood as the change relationship between the previous and next frames of the camera. The camera posture can be understood as the position coordinates of the camera in the second coordinate system.

[0074] Specifically, the processor can set the camera coordinates of the camera to be calibrated at the initial moment in the second coordinate system to zero to obtain the initial camera coordinates. The processor can match the feature points of the feature point images of adjacent frames, such as by using a Brute Force (BF) algorithm to match the feature points, and determine the feature point matching relationship between the feature points. The processor can determine the moving distance between the same feature points in adjacent frames based on the matching relationship between each feature point, and obtain the motion relationship between the feature point images of adjacent frames. According to each motion relationship and the initial camera coordinates, the camera posture of each frame of the feature point image in the second coordinate system is determined, and according to each camera posture, the camera motion trajectory data in the second coordinate system is determined.

[0075] Exemplarily, the initial camera coordinates can be represented by a 6-dimensional vector, that is, (0,0,0,0,0,0). Taking a feature point matching relationship as an example, assuming that p1 is a point on the T-th frame image, p2 is a point on the T+1-th image, and the preset optimization parameter ξ describes the transformation relationship between the two frames of images. The optimization parameter ξ is used to map the p2 point on the T+1-th image to the T-th frame image, denoted as p3. Then the reprojection error is ||p1-p3||, that is, the distance between the p1 point and the p3 point.

[0076] S206: Filter the vehicle motion trajectory data to obtain filtered intermediate vehicle trajectory data.

[0077] In this embodiment, the intermediate vehicle trajectory data can be understood as the data after filtering out points with large fluctuations.

[0078] Specifically, the processor may first use a filter (such as a low-pass filter, etc.) to filter the vehicle motion trajectory information, remove data with large errors, reduce the error of the data with large errors on subsequent parameter determination, and obtain intermediate vehicle trajectory data.

[0079] S207 , filtering the camera motion trajectory data to obtain filtered intermediate camera trajectory data.

[0080] Specifically, the processor may first use a filter (such as a low-pass filter, etc.) to filter the camera motion trajectory data, eliminate data with large errors, reduce the error of the data with large errors on subsequent parameter determination, and obtain intermediate camera trajectory data.

[0081] S208 , aligning the timestamp of the intermediate vehicle trajectory data with the timestamp of the intermediate camera trajectory data.

[0082] In this embodiment, the timestamp can be understood as being used to indicate the time corresponding to when the data is collected.

[0083] Specifically, the processor may perform time stamp alignment processing on the two track data after filtering. When the clock sources of the two data are inconsistent, the clock sources need to be unified.

[0084] S209 , matching the timestamp of the aligned intermediate camera trajectory data with the timestamp of the aligned intermediate vehicle trajectory data to obtain fused trajectory data.

[0085] In this embodiment, the fused trajectory data can be understood as the result of matching and fusion of the intermediate camera trajectory data and the intermediate vehicle trajectory data at each moment.

[0086] Specifically, the processor may match the timestamp of each key frame with the timestamp of the motion trajectory information to obtain the fused trajectory information.

[0087] S210 , segmenting the fused trajectory data according to the set segmentation conditions to obtain a segmented trajectory data set.

[0088] In this embodiment, setting the segmentation condition can be understood as a condition for segmenting the fused trajectory data, for example, equal distance segmentation can be performed according to the distance l, or equal time segmentation can be performed according to the time ps.

[0089] Specifically, the processor may segment the fused trajectory data according to a set segmentation condition, for example, segmenting according to equal time or equal distance to obtain a segmented trajectory data set.

[0090] S211 . For the segmented trajectory data included in the segmented trajectory data set, take the start time of the segmented trajectory data as the initial time.

[0091] Specifically, for the segmented trajectory data included in the segmented trajectory data set, the start time of each segmented trajectory data is used as the initial time.

[0092] S212: Update the trajectory data included in the segmented trajectory data based on the initial time to obtain target trajectory data.

[0093] In this embodiment, the trajectory data includes the intermediate vehicle trajectory data and the intermediate camera trajectory data in each frame. The target trajectory data can be understood as the updated trajectory data.

[0094] Specifically, since each segmented trajectory data starts from an initial moment, that is, each segmented trajectory data starts from initial vehicle trajectory data and initial camera trajectory data, the processor can update the trajectory data included in the segmented trajectory data based on the initial vehicle trajectory data and initial camera trajectory data corresponding to the initial moment to obtain the target trajectory data.

[0095] Exemplarily, the processor may initialize the vehicle body state corresponding to the initial moment to obtain the initial vehicle body state, and initialize the camera state at the initial moment to obtain the initial camera state. Based on the initial vehicle body state, the vehicle body state at other moments in the segmented trajectory data is updated, and the change amount of the vehicle body state at other moments relative to the original vehicle body state at the initial moment is calculated, and then the target vehicle body state at each moment is obtained according to the change amount and the initial vehicle body state, and the target vehicle body trajectory data is formed. The processor may update the camera state at other moments in the segmented trajectory data based on the initial camera state, calculate the change amount of the camera state at other moments relative to the original camera state at the initial moment, and then the target camera state at each moment is obtained according to the change amount and the initial camera state, and the target camera trajectory data is formed; the target vehicle body trajectory data and the target camera trajectory data are used as the target trajectory data.

[0096] S213, determining, according to each target trajectory data, a vehicle body change relationship in the first coordinate system and a camera change relationship in the second coordinate system at each moment in the segmented trajectory data.

[0097] In this embodiment, the body change relationship can be understood as the coordinate change of the body at any two moments in the segmented trajectory data, including rotational change and translational change. The camera change relationship can be understood as the coordinate change of the camera at any two moments in the segmented trajectory data, including rotational change and translational change.

[0098] Specifically, the processor can calculate the coordinate change of the body state at the latter moment relative to the body state at the previous moment between any two moments according to each target trajectory data, thereby calculating the body change relationship in the first coordinate system. The processor can calculate the coordinate change of the camera state at the latter moment relative to the camera state at the previous moment between any two moments according to each target trajectory data, thereby calculating the camera change relationship in the second coordinate system.

[0099] S214: Determine a fusion conversion relationship set of the segmented trajectory data according to the change relationship of each vehicle body and the change relationship of each camera.

[0100] In this embodiment, the fused transformation relationship set can be understood as a transformation relationship between two coordinate systems.

[0101] Specifically, the processor can determine the fusion transformation relationship between the two coordinate systems at any two moments in the segmented trajectory data based on the vehicle body change relationship and the camera change relationship between any two moments in the segmented trajectory data, combined with unknown external parameters, and obtain the fusion transformation relationship set of the segmented trajectory data by calculating the fusion transformation relationship between any two moments of a set number.

[0102] For example, in order to facilitate understanding of the determination of the fusion conversion relationship in the present invention, a specific example is taken as an example. Figure 3 This is a segmented example diagram of a camera parameter calibration method provided in the second embodiment of the present invention, such as Figure 3 As shown, for any two moments in the i-th segmented trajectory data, the second coordinate system is from moment C1 to moment C2. Based on the same timestamp, the first coordinate system corresponds to moment W1 to moment W2. The camera change relationship is R1 and t1, where R1 represents the rotation vector in the second coordinate system, and t1 represents the translation vector in the second coordinate system. The body change relationship is R2 and t2, where R2 represents the rotation vector in the first coordinate system, and t2 represents the translation vector in the first coordinate system. There is a rotation matrix relationship from coordinate system C2 to coordinate system W1: R1R=RR2, which is equivalent to another form Among them, the unknown external parameters include the rotation vector Φ∈so(3), which is a 3D vector that describes the rotation relationship between the second coordinate system and the first coordinate system on the vehicle. Each rotation vector has a corresponding rotation matrix R, and the calculation relationship between the two is given by the formula R=exp(Φ^). The unknown external parameters also include the translation vector t∈R 3 , is a 3D vector that describes the rotation relationship between the second coordinate system and the first coordinate system on the vehicle. The k scale factors s, denoted as s[k], are k-dimensional vectors, s_i in the figure, i.e., the i-th scale factor. There is also a coordinate transformation relationship: R1t+t1=Rt2+t, which is equivalent to another form (1-R1)t+s kRt2-t1=0. The fusion conversion relationship cost can be expressed as:

[0103]

[0104] Among them, d1 is the rotation constraint weight and d2 is the translation constraint weight.

[0105] S215 , optimizing the current external parameters of the camera to be calibrated according to each fusion transformation relationship set to obtain the final external parameters after calibration.

[0106] Specifically, the processor can accumulate the fusion transformation relationship sets corresponding to each segmented trajectory data, and then obtain the total fusion transformation relationship including the unknown external parameters, determine the unknown external parameters through the total fusion transformation relationship, and optimize the current external parameters of the camera to be calibrated through the determined external parameters to obtain the final external parameters after calibration.

[0107] Further, based on the above embodiment, the steps of optimizing the current external parameters of the camera to be calibrated according to each fusion transformation relationship set to obtain the final external parameters after calibration can be refined as follows:

[0108] According to the sum of each fusion transformation relationship set, the total transformation relationship is determined; according to the total transformation relationship, the next external parameter value is determined; the current external parameter is optimized by the next external parameter value to obtain the final camera parameter.

[0109] In this embodiment, the total conversion relationship can be understood as the sum of all fused conversion relationships. The next external parameter value can be understood as the calculated optimized result.

[0110] Specifically, the processor can add up all the fused transformation relationships in the fused transformation relationship set to obtain the total transformation relationship, and calculate the unknown parameters in the total transformation relationship through the set optimization algorithm (such as a nonlinear optimization algorithm). For example, the nonlinear optimization algorithm determines the unknown parameters by calculating the minimum value of the total transformation relationship, and uses the determined unknown parameters as the next external parameter value. The change between the current external parameter and the next external parameter is used to determine whether the optimization process meets the set convergence condition. If not, the external parameters will continue to be optimized until they are met and the next external parameter value will be determined as the final camera parameter.

[0111] The technical solution of the embodiment of the present invention obtains the vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs, and determines the vehicle motion trajectory data in the first coordinate system according to the vehicle driving data. By filtering the image of the region of interest and extracting feature points on the image frame set, an image containing feature points is obtained, and then feature point matching is performed to obtain a feature point matching relationship, and the motion relationship between images is determined according to the feature point matching relationship, and the camera posture under each frame is determined according to the motion relationship, and the camera motion trajectory data in the second coordinate system is obtained. The vehicle motion trajectory data and the camera motion trajectory data are filtered to eliminate the influence of abnormal data with large fluctuations, thereby providing a guarantee for the subsequent accurate determination of the final external parameters of the camera. The filtered intermediate vehicle trajectory data and the intermediate camera trajectory data are time stamped and matched to obtain the fused trajectory data that integrates the trajectory data in the first coordinate system and the second coordinate system. The fused trajectory data is segmented by setting segmentation conditions to divide the fused trajectory data into segmented trajectory data sets with higher precision. The data in each segmented trajectory data is initialized and updated. Each segmented trajectory data starts from the initial moment to ensure the consistency of each segmented trajectory data. Then, based on the updated target trajectory data, the body change relationship in the first coordinate system and the camera change relationship in the second coordinate system are determined. Based on the body change relationship and camera change relationship of each segmented trajectory data in the segmented trajectory data set, the current external parameters of the camera to be calibrated are optimized to obtain the final external parameters after calibration. The accuracy of the motion trajectory is improved, and the external parameters are optimized based on the segmented trajectory data with finer precision to obtain the final external parameters after calibration, which improves the accuracy of the external parameter determination. The calibration environment is flexible, which improves the stability of the calibration and thus improves the calibration effect.

[0112] Embodiment 3

[0113] Figure 4 This is a schematic diagram of the structure of a camera parameter calibration device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0114] The data acquisition module 31 is used to acquire the vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs;

[0115] A first determination module 32, configured to determine vehicle motion trajectory data in a first coordinate system according to the vehicle driving data;

[0116] A second determination module 33, configured to determine camera motion trajectory data in a second coordinate system according to the image frame set;

[0117] A data segmentation module 34, used to determine a segmented segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data;

[0118] The parameter calibration module 35 is used to optimize the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration.

[0119] The technical solution of the embodiment of the present invention is to obtain the vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs; determine the vehicle motion trajectory data in the first coordinate system based on the vehicle driving data; determine the camera motion trajectory data in the second coordinate system based on the image frame set; determine the segmented segmented trajectory data set based on the vehicle motion trajectory data and the camera motion trajectory data; optimize the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration. By segmenting the fusion of the vehicle motion trajectory and the camera motion trajectory, the accuracy of the motion trajectory is improved, and the external parameters are optimized based on the segmented trajectory data with finer accuracy to obtain the final external parameters after calibration. The calibration environment is flexible, which improves the stability of the calibration and thus improves the calibration effect.

[0120] Further, the second determining module 33 includes:

[0121] A first determining unit, configured to perform background filtering on each key image frame included in the image frame set to obtain an image of a region of interest;

[0122] A second determining unit is used to extract feature points from each of the images of the region of interest to obtain a feature point image containing the feature points;

[0123] The third determining unit is used to determine the camera motion trajectory data in the second coordinate system according to each of the feature point images.

[0124] The third determining unit is specifically used for:

[0125] Obtaining the initial camera coordinates of the camera to be calibrated in the second coordinate system;

[0126] Matching feature points of the feature point images of adjacent frames to determine a feature point matching relationship;

[0127] Based on the matching relationship of each feature point, determining the motion relationship between the feature point images of adjacent frames;

[0128] Determine the camera posture of each frame feature point image in the second coordinate system according to each of the motion relationships and the initial camera coordinates;

[0129] According to each of the camera postures, the camera motion trajectory data in the second coordinate system is determined.

[0130] Furthermore, the data segmentation module 34 is specifically used for:

[0131] Filtering the vehicle motion trajectory data to obtain filtered intermediate vehicle trajectory data;

[0132] Filtering the camera motion trajectory data to obtain filtered intermediate camera trajectory data;

[0133] Aligning a timestamp of the intermediate vehicle trajectory data with a timestamp of the intermediate camera trajectory data;

[0134] Matching the timestamp of the aligned intermediate camera trajectory data with the timestamp of the aligned intermediate vehicle trajectory data to obtain fused trajectory data;

[0135] The fused trajectory data is segmented according to the set segmentation conditions to obtain a segmented trajectory data set.

[0136] Furthermore, the parameter calibration module 35 includes:

[0137] a fourth determining unit, configured to use, for the segmented trajectory data included in the segmented trajectory data set, a start time of the segmented trajectory data as an initial time;

[0138] a fifth determining unit, configured to update the trajectory data included in the segmented trajectory data based on the initial moment to obtain target trajectory data;

[0139] a sixth determining unit, configured to determine, according to each of the target trajectory data, a vehicle body change relationship in the first coordinate system and a camera change relationship in the second coordinate system at each moment in the segmented trajectory data;

[0140] a seventh determining unit, configured to determine a fusion transformation relationship set of the segmented trajectory data according to each of the vehicle body change relationships and each of the camera change relationships;

[0141] The eighth determination unit is used to optimize the current external parameters of the camera to be calibrated according to each of the fusion transformation relationship sets to obtain the final external parameters after calibration.

[0142] The fifth determining unit is specifically used for:

[0143] Initializing the first vehicle body state at the initial moment to obtain an initial vehicle body state;

[0144] Initializing the first camera state at the initial moment to obtain an initial camera state;

[0145] Based on the initial vehicle body state, updating the second vehicle body state at other times in the segmented trajectory data to obtain a target vehicle body state and form target vehicle body trajectory data;

[0146] Based on the initial camera state, updating the second camera state at other times in the segmented trajectory data to obtain the target camera state and form the target camera trajectory data;

[0147] The target vehicle body trajectory data and the target camera trajectory data are used as the target trajectory data.

[0148] The eighth determining unit is specifically used for:

[0149] Determine the total conversion relationship according to the sum of the fused conversion relationship sets;

[0150] Determining the next external parameter value according to the total conversion relationship;

[0151] The current external parameters are optimized by using the next external parameter value to obtain final camera parameters.

[0152] The camera parameter calibration device provided in the embodiment of the present invention can execute the camera parameter calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0153] Embodiment 4

[0154] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0155] like Figure 4As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0156] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0157] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as a calibration method for camera parameters.

[0158] In some embodiments, the calibration method of the camera parameters may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the calibration method of the camera parameters described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to execute the calibration method of the camera parameters in any other appropriate manner (e.g., by means of firmware).

[0159] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0161] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0163] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0164] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0165] In one embodiment, the embodiment of the present invention further includes a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the camera parameter calibration method of any embodiment of the present invention.

[0166] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A camera parameter calibration method, characterized in that: include: Obtain vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs; Determining vehicle motion trajectory data in a first coordinate system according to the vehicle driving data; Determining camera motion trajectory data in a second coordinate system according to the image frame set; Determining a segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data; Based on each segmented trajectory data in the segmented trajectory data set, the current external parameters of the camera to be calibrated are optimized to obtain the final external parameters after calibration.

2. The method according to claim 1, characterized in that: Determining the camera motion trajectory data in the second coordinate system according to the image frame set includes: Performing background filtering on each key image frame included in the image frame set to obtain an image of a region of interest; Extracting feature points from each of the region of interest images to obtain a feature point image containing the feature points; According to each of the feature point images, the camera motion trajectory data in the second coordinate system is determined.

3. The method according to claim 2, characterized in that Determining the camera motion trajectory data in the second coordinate system according to each of the feature point images includes: Obtaining the initial camera coordinates of the camera to be calibrated in the second coordinate system; Matching feature points of the feature point images of adjacent frames to determine a feature point matching relationship; Based on the matching relationship of each feature point, determining the motion relationship between the feature point images of adjacent frames; Determine the camera posture of each frame feature point image in the second coordinate system according to each of the motion relationships and the initial camera coordinates; According to each of the camera postures, the camera motion trajectory data in the second coordinate system is determined.

4. The method according to claim 1, characterized in that The step of determining a segmented segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data includes: Filtering the vehicle motion trajectory data to obtain filtered intermediate vehicle trajectory data; Filtering the camera motion trajectory data to obtain filtered intermediate camera trajectory data; Aligning a timestamp of the intermediate vehicle trajectory data with a timestamp of the intermediate camera trajectory data; Matching the timestamp of the aligned intermediate camera trajectory data with the timestamp of the aligned intermediate vehicle trajectory data to obtain fused trajectory data; The fused trajectory data is segmented according to the set segmentation conditions to obtain a segmented trajectory data set.

5. The method according to claim 1, characterized in that: The optimizing the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration includes: For the segmented trajectory data included in the segmented trajectory data set, taking the start time of the segmented trajectory data as the initial time; Based on the initial time, the trajectory data included in the segmented trajectory data is updated to obtain target trajectory data; Determine, according to each of the target trajectory data, a vehicle body change relationship in the first coordinate system and a camera change relationship in the second coordinate system at each moment in the segmented trajectory data; Determining a fusion transformation relationship set of the segmented trajectory data according to each of the vehicle body change relationships and each of the camera change relationships; The current external parameters of the camera to be calibrated are optimized according to each of the fusion transformation relationship sets to obtain the final external parameters after calibration.

6. The method according to claim 5, characterized in that The optimizing the current external parameters of the camera to be calibrated according to each of the fusion transformation relationship sets to obtain the final external parameters after calibration includes: Determine the total conversion relationship according to the sum of the fused conversion relationship sets; Determining the next external parameter value according to the total conversion relationship; The current external parameters are optimized by using the next external parameter value to obtain final camera parameters.

7. A camera parameter calibration device, characterized in that: include: A data acquisition module, used to acquire vehicle driving data and image frame set of the vehicle to which the camera to be calibrated belongs; A first determination module, used to determine vehicle motion trajectory data in a first coordinate system according to the vehicle driving data; A second determination module, used to determine the camera motion trajectory data in a second coordinate system according to the image frame set; A data segmentation module, used to determine a segmented segmented trajectory data set according to the vehicle motion trajectory data and the camera motion trajectory data; The parameter calibration module is used to optimize the current external parameters of the camera to be calibrated based on each segmented trajectory data in the segmented trajectory data set to obtain the final external parameters after calibration.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the camera parameter calibration method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the camera parameter calibration method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the camera parameter calibration method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Automatic calibration method and device for installation angle of collision early warning radar

    CN112147616A

  • Automobile data recorder external parameter calibration method and device and electronic equipment

    CN112348903A

  • Radar parameter parameter determination method and device, equipment and storage medium

    CN112630736A

  • Test method, device, equipment and system of pose tracking module and medium

    CN114442808A

  • Space-time joint calibration method of laser radar and IMU system

    CN116338652A

Cited By

  • Method of calibrating camera of trailer attached to vehicle, and ECU thereof

    GB2705675A