Vehicle-mounted surround view camera calibration method, device, equipment and medium

Abstract

The application provides a kind of vehicle-mounted surround view camera calibration method, device, equipment and medium, comprising: ground feature point detection and matching are carried out to current frame image and adjacent frame image of image data, determine the feature point set obtained by matching two frame images;Determine the driving direction vector coordinate of vehicle driving direction vector in camera coordinate system and the ground normal vector coordinate of ground normal vector in camera coordinate system based on the feature point set;Based on driving direction vector coordinate and ground normal vector coordinate, obtain camera pose matrix, camera pose matrix is used to represent the pose transformation relationship between camera coordinate system and vehicle-mounted coordinate system;Determine the position calibration result of each camera based on camera internal parameter, installation position parameter and camera pose matrix.The application is used to solve the defect that it is difficult to guarantee the calibration accuracy in the prior art due to the need for specific calibration object to assist calibration, to improve the accuracy of camera calibration.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted camera technology, and in particular to a method, apparatus, device, and medium for calibrating a vehicle-mounted surround-view camera. Background Technology

[0002] Currently, with the development of imaging technology, camera devices are widely used in the automotive field as important sensors for driver assistance. Especially for large vehicles, there are many blind spots around them, which can easily lead to scrapes, collisions, and running over pedestrians / non-motorized vehicles when handling situations such as starting, turning, parking, passing oncoming traffic in narrow lanes, and avoiding obstacles, frequently causing serious traffic accidents. Installing surround-view devices on vehicles, using multiple cameras to simultaneously capture images of the vehicle's surroundings, and processing them to create a panoramic overhead view of the vehicle's location and surrounding environment, can effectively reduce accidents. Among these, the calibration of the surround-view cameras using image processing technology is crucial to the quality of the panoramic overhead view stitching.

[0003] Traditional surround-view camera calibration requires specific calibration objects. In OEM scenarios, a dedicated calibration area is typically created by printing specific shapes of markers (squares, dots, or parallel lines) on the ground. The vehicle is driven to the center of the area, and the surround-view camera is calibrated by recognizing specific features and the pre-set size of the marker patterns. In aftermarket scenarios, a certain number of specific markers need to be manually laid around the vehicle. The distances between these patterns or the dimensions of the patterns themselves need to be measured, and after recognition and processing, the relative relationships between the cameras are calculated to complete the final surround-view calibration.

[0004] Traditional circumferential calibration methods using specific calibration objects rely on the measurement or printing accuracy of the calibration object's dimensions. Furthermore, repeated handling of the calibration object and repeated pressing of the printed pattern can lead to damage or blurring, making it difficult to guarantee calibration accuracy. This adds extra production, transportation, and maintenance costs. In addition, different manufacturers have different solutions with varying requirements and dimensions for the patterns, making them incompatible and unusable.

[0005] Therefore, there is an urgent need for a method for calibrating surround-view cameras that does not require specific calibration objects. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for calibrating a vehicle-mounted surround-view camera, which solves the problem that the calibration accuracy is difficult to guarantee due to the need for specific calibration objects in the prior art, thereby improving the accuracy of camera calibration.

[0007] This invention provides a method for calibrating a vehicle-mounted surround-view camera, comprising:

[0008] Ground feature point detection and matching are performed on the current frame image and adjacent frame images to determine the feature point set obtained by matching the two frames;

[0009] Based on the feature point set, determine the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system.

[0010] Based on the driving direction vector coordinates and the ground normal vector coordinates, a camera attitude matrix is ​​obtained, which is used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system.

[0011] Based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix, the position calibration results of each camera are determined.

[0012] According to the present invention, a method for calibrating a vehicle-mounted surround-view camera determines the position calibration results of each camera based on camera intrinsic parameters, installation position parameters, and the camera attitude matrix, including:

[0013] Based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix, a top view of each camera is determined;

[0014] Feature detection and matching are performed within the overlapping area of ​​the top view to determine the set of matching points. Based on the set of matching points, camera positions are optimized to determine the position calibration results of each camera.

[0015] According to a method for calibrating a vehicle-mounted surround-view camera provided by the present invention, determining the feature point set includes:

[0016] Based on the image data from the calibrated camera, the current frame image and the adjacent frame images are determined;

[0017] Image processing, feature detection, and matching are performed sequentially on the current frame image and the adjacent frame images to obtain a common set of feature points.

[0018] According to the present invention, a method for calibrating a vehicle-mounted surround-view camera, based on the feature point set, determines the coordinates of the vehicle's driving direction vector in the camera coordinate system, including:

[0019] Based on the feature point set and the camera intrinsic parameters, determine the fundamental matrix corresponding to the camera intrinsic parameters;

[0020] Based on the fundamental matrix, the pose transformation matrix of the camera from the current frame image to the adjacent frame image is determined. The pose transformation matrix includes a rotation matrix and a translation vector, and the translation vector is the vehicle driving direction vector of the vehicle coordinate system.

[0021] 5. The vehicle-mounted surround-view camera calibration method according to claim 4, characterized in that, based on the feature point set, determining the ground normal vector coordinates in the camera coordinate system includes:

[0022] Based on the pose transformation matrix, the camera intrinsic parameters, and the feature point set, the 3D coordinates of the feature points in the camera coordinate system are determined;

[0023] The ground plane equation is fitted based on the 3D coordinates, and the ground normal vector coordinates in the camera coordinate system are determined.

[0024] A method for calibrating a vehicle-mounted surround-view camera according to the present invention includes:

[0025] Multiple sets of vehicle translation vectors and ground normal vectors are determined using multi-frame image data, and the camera pose matrix is ​​optimized based on the multiple sets of vehicle translation vectors and ground normal vectors.

[0026] The optimization of the camera pose matrix refers to optimizing the camera pose matrix using nonlinear constraints, including forward direction constraints and vertical direction constraints.

[0027] The present invention also provides a vehicle-mounted surround-view camera calibration device, comprising:

[0028] The ground feature point detection module is used to detect and match ground feature points in the current frame image and adjacent frame images to determine the feature point set obtained by matching the two frames;

[0029] The coordinate calculation module is used to determine the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system based on the feature point set.

[0030] The attitude matrix calculation module is used to obtain the camera attitude matrix based on the driving direction vector coordinates and the ground normal vector coordinates. The camera attitude matrix is ​​used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system.

[0031] The position calibration calculation module is used to determine the position calibration results of each camera based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle surround view camera calibration method described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vehicle surround-view camera calibration method as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the vehicle surround view camera calibration method as described above.

[0035] The present invention provides a method, apparatus, device, and medium for calibrating vehicle-mounted surround-view cameras. Considering that the ground surface of roads where vehicles travel typically has certain texture features and various traffic markings (such as lane lines, zebra crossings, and turn signs), the method utilizes these ground features to perform attitude calibration on each camera separately, obtaining an attitude matrix. Based on this matrix, combined with camera intrinsic parameters and camera installation positions, and leveraging the overlapping fields of view of adjacent cameras, the method optimizes and calibrates the position of each camera, improving the accuracy of camera calibration. This invention eliminates the need for calibrators with specific shapes during camera calibration; only a flat surface with a certain texture is required. This reduces the cost of transporting and maintaining calibrators and minimizes problems caused by measurement errors, making the process more convenient and efficient. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts illustrating the vehicle surround view camera calibration method provided by the present invention;

[0038] Figure 2 This is an explanatory diagram of the camera coordinate system and vehicle coordinate system described in this invention;

[0039] Figure 3 This is the second flowchart illustrating the vehicle surround view camera calibration method provided by the present invention;

[0040] Figure 4 This is the third flowchart illustrating the vehicle surround view camera calibration method provided by the present invention;

[0041] Figure 5 This is a schematic diagram of the coordinate system positions between adjacent frame data as described in this invention;

[0042] Figure 6 This is the fourth flowchart illustrating the vehicle surround view camera calibration method provided by the present invention;

[0043] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] The following is combined Figures 1-6 The present invention describes the vehicle-mounted surround-view camera calibration method.

[0046] Please refer to Figure 1 The vehicle-mounted surround-view camera calibration method proposed in this invention includes:

[0047] Step 10: Detect and match ground feature points in the current frame image and adjacent frame images to determine the feature point set obtained by matching the two frames.

[0048] In this embodiment, image data from the currently calibrated camera is acquired, and the current frame image data F1 and the previous frame image data F0 are sent to the image processing module for processing. The current frame image data F1 and the previous frame image data F0 can be adjacent frames or data from multiple adjacent frames, and it is necessary to ensure that the two frames have overlapping parts. The image processing also includes performing image processing, feature detection, and matching on the two frames of image data F1 and F0 to obtain the image coordinates of a common set of feature points.

[0049] Step 20: Based on the feature point set, determine the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system.

[0050] In this embodiment, the vehicle is started, a flat ground with certain texture features is selected, and an automatic calibration program is entered. The vehicle is kept moving slowly in a straight line. During the movement, the attitude of each camera installed on the vehicle is calibrated separately.

[0051] During the slow straight-line movement of the vehicle, the attitude of each camera is calibrated. The specific process includes: First, ground feature point detection and matching are performed using the current frame image and the previous frame image. The pose change of the vehicle from the position of the previous frame to the position of the current frame is calculated. The direction vector coordinates of the vehicle's driving direction in the camera coordinate system are obtained. The direction vector coordinates are obtained. And the coordinates of the corresponding ground normal vector in the camera coordinate system are obtained. The ground normal vector coordinates are obtained.

[0052] The calculation process of the ground normal vector coordinates is as follows: based on the vehicle's pose change between two frames and the matching feature point pairs between the two frames, the three-dimensional coordinates of the corresponding ground feature points in the camera coordinate system can be obtained. After calculation and fitting, the plane normal vector is obtained, and the coordinates of the ground normal vector in the camera coordinate system are obtained, i.e., the ground normal vector coordinates.

[0053] Step 30: Based on the driving direction vector coordinates and the ground normal vector coordinates, obtain the camera attitude matrix, which is used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system;

[0054] Then, using the vehicle's driving direction vector and the ground normal vector in the camera coordinate system, the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system is obtained, i.e., the camera's attitude matrix. The vehicle coordinate system is the world coordinate system on the vehicle.

[0055] Furthermore, this step can be repeated after optimizing the camera pose matrix. The optimization of the camera pose matrix includes optimizing the pose matrix based on the vehicle driving direction vector and ground normal vector from multiple frames of data.

[0056] Step 40: Determine the position calibration results of each camera based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix.

[0057] It should be noted that each camera with pre-calibrated intrinsic parameters is pre-installed and fixed, and the installation position of each camera on the vehicle body is measured to obtain installation position parameters, including lateral distance, front-rear distance, and height. Specifically, the camera intrinsic parameters are pre-calibrated, including focal length, principal point, lens distortion coefficient, and other intrinsic camera parameters. The calibration method can be selected according to requirements and will not be elaborated here. Cameras are pre-installed and fixed around the vehicle body, and the installation position of each camera on the vehicle body is measured to obtain installation position data, including lateral distance, front-rear distance, and height from the vehicle center. Before position calibration, according to... Figure 2 The indicated vehicle coordinate system direction converts the left-right distance, front-back distance, and height into coordinates under the vehicle coordinate system to obtain the installation position data.

[0058] Based on the camera intrinsic parameters, installation position parameters, and the camera pose matrix obtained above, the top view of each camera is calculated; feature detection and matching are performed in the overlapping area of ​​the top views of adjacent cameras, and the camera position is optimized based on the matching point pairs to obtain the calibration result with the best stitching effect.

[0059] The vehicle-mounted surround-view camera calibration method provided by this invention takes into account the texture features and various traffic markings (such as lane lines, zebra crossings, and turn signs) typically present on the road surface where vehicles travel. It utilizes these ground features to perform attitude calibration on each camera separately, obtaining an attitude matrix. Based on this matrix, and combining the camera's intrinsic parameters and installation position, the position calibration result for each camera is calculated. This invention eliminates the need for calibrators with specific shapes during camera calibration; only a flat, textured surface is required. This reduces the cost of transporting and maintaining calibrators and minimizes measurement errors, making the method more convenient and efficient.

[0060] In one possible embodiment, please refer to Figure 3 Step 40: Based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix, determine the position calibration results of each camera, including:

[0061] Step 41: Determine the top view of each camera based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix;

[0062] Step 42: Perform feature detection and matching within the overlapping area of ​​the top view to determine the matching point set, and optimize the camera position based on the matching point set to determine the position calibration result of each camera.

[0063] In this embodiment, based on the camera intrinsic parameters, installation position, and the optimized camera pose matrix obtained above, the top view of each camera is calculated; feature detection and matching are performed in the overlapping area of ​​the top views of adjacent cameras, and the camera position is optimized based on the matching point pairs to obtain the calibration result with the best stitching effect.

[0064] This embodiment calculates the top view of each camera by combining camera intrinsic parameters and camera installation position, performs feature detection and matching in the overlapping area of ​​the top views of adjacent cameras, and optimizes the camera position based on multiple sets of matched feature point pairs to obtain smaller stitching errors.

[0065] In one possible embodiment, step 10, determining the feature point set, includes:

[0066] Step 11: Based on the image data from the calibrated camera, determine the current frame image and the adjacent frame images;

[0067] Step 12: Perform image processing, feature detection, and matching on the current frame image and the adjacent frame images in sequence to obtain a common set of feature points.

[0068] In this embodiment, image data from the currently calibrated camera is acquired, and the current frame image data F1 and the previous frame image data F0 are sent to the image processing module for processing. The current frame image data F1 and the previous frame image data F0 can be adjacent frames or data from multiple adjacent frames, and it is necessary to ensure that the two frames have overlapping parts. The image processing also includes performing image processing, feature detection, and matching on the two frames of image data F1 and F0 to obtain the image coordinates of a common set of feature points.

[0069] In this embodiment, image processing, feature detection, and matching can be performed using the current frame image and the adjacent frame images to obtain a common set of feature points, and ground feature points can be extracted, thereby improving the accuracy of feature point extraction and further improving the accuracy of camera calibration.

[0070] In one possible embodiment, please refer to Figure 4 Step 20: Based on the feature point set, determine the coordinates of the vehicle's driving direction vector in the camera coordinate system, including:

[0071] Step 201: Based on the feature point set and the camera intrinsic parameters, determine the fundamental matrix corresponding to the camera intrinsic parameters;

[0072] Step 202: Based on the fundamental matrix, determine the pose transformation matrix of the camera from the current frame image to the adjacent frame image. The pose transformation matrix includes a rotation matrix and a translation vector, and the translation vector is the vehicle driving direction vector in the vehicle coordinate system.

[0073] Please refer to Figure 5 The vehicle's driving direction vector is the coordinate O of the vehicle coordinate system origin at the position indicated by the data in the adjacent frame F0. v0 The coordinates O of the vehicle coordinate system origin are at the current position indicated by the F1 frame data in the current frame image. v1 .

[0074] In this embodiment, the vehicle driving direction vector in the camera coordinate system is calculated using the feature point set shared by the current frame image F1 and the adjacent frame image F0. Specifically:

[0075] To calculate the vehicle's direction vector We need to use epipolar geometry principles, combined with camera intrinsic parameters, to calculate the fundamental or essential matrix, and then recover the camera's pose transformation from the position in the adjacent frame F0 to the position in the current frame F1, thus obtaining the rotation matrix R. 01 The translation vector t01, here is rigid body motion, therefore The resulting translation vector is the vehicle's direction of travel vector.

[0076] In this embodiment, a method for calculating the vehicle driving direction vector is provided. That is, the pose transformation of the camera from the current frame image to the adjacent frame image can be obtained by calculating the camera's fundamental matrix, and the vehicle driving direction vector in the vehicle coordinate system can be obtained.

[0077] In one possible embodiment, please refer to Figure 6 Step 20: Based on the feature point set, determine the ground normal vector coordinates in the camera coordinate system, including:

[0078] Step 211: Based on the pose transformation matrix, the camera intrinsic parameters, and the feature point set, determine the 3D coordinates of the feature points in the camera coordinate system;

[0079] Step 212: Fit the ground plane equation based on the 3D coordinates and determine the ground normal vector coordinates in the camera coordinate system.

[0080] This embodiment proposes a method for calculating the ground normal vector. Using camera intrinsic parameters and rotation matrix R 01 Translation vector t 01 The image coordinates of the feature point set are used to obtain the 3D coordinates of the feature points in the camera coordinate system through a triangulation algorithm. Then, the ground plane equation is fitted to obtain the ground normal vector coordinates in the camera coordinate system.

[0081] In one possible embodiment, step 30, obtaining the camera attitude matrix based on the driving direction vector coordinates and the ground normal vector coordinates, includes:

[0082] Step 31: Based on the driving direction vector coordinates and the ground normal vector coordinates, calculate the initial value of the attitude matrix of the camera coordinate system relative to the vehicle coordinate system.

[0083] Step 32: Based on the initial value of the attitude matrix, optimize the attitude matrix using multiple frames of data;

[0084] Among them, use and Calculate the initial pose of the camera coordinate system relative to the vehicle coordinate system, i.e., the rotation matrix R. c2v :

[0085] In the forward straight-ahead state, the vehicle's translation vector Along the y-axis of the vehicle coordinate system, i.e. The direction vector of the y-axis in the vehicle coordinate system in the camera coordinate system; on a flat road surface, the ground normal vector. Parallel to the z-axis of the vehicle coordinate system, i.e. Let z be the direction vector of the vehicle's z-axis in the camera's coordinate system; in an ideal situation... and The x-axis vector of the vehicle coordinate system is perpendicular to each other, and its coordinates are given in the camera coordinate system. It can be represented as:

[0086]

[0087] According to the Euclidean criterion, the following relationship holds:

[0088]

[0089] Among them, [e vx e vy e vz [e] is the unit orthogonal basis of the vehicle coordinate system. cx e cy e cz [ ] is the unit orthonormal basis of the camera coordinate system. These are the direction vectors of the x, y, and z axes of the vehicle coordinate system, and are actually an identity matrix. It represents the coordinates of the corresponding direction vector in the camera coordinate system.

[0090] By definition, a rotation matrix consists of the inner product of two bases, i.e., R0 c2v =[e vx T e vy T e vz T ] T ·[e cx e cy e cz Therefore, the above formula can be expressed as

[0091]

[0092] Therefore, we can obtain According to this formula, we can... and The rotation matrix R is calculated. c2v An initial value.

[0093] In one possible embodiment, the vehicle-mounted surround-view camera calibration method proposed in this invention further includes a method for optimizing the camera attitude matrix, comprising:

[0094] Multiple sets of vehicle translation vectors and ground normal vectors are determined using multi-frame image data, and the camera pose matrix is ​​optimized based on these multiple sets of vehicle translation vectors and ground normal vectors.

[0095] The optimization of the camera pose matrix refers to optimizing the camera pose matrix using nonlinear constraints, including forward direction constraints and vertical direction constraints.

[0096] Calculate multiple sets of vehicle translation vectors using multi-frame data. and ground normal vector For the camera pose matrix R c2v Optimizations will be made, specifically including:

[0097] The method uses multi-frame data to calculate multiple sets of vehicle translation vectors. and ground normal vector This refers to repeatedly executing steps 10 and 20 while the vehicle is moving slowly straight ahead, resulting in multiple sets of... and

[0098] The camera pose matrix R c2v Optimization refers to optimizing the attitude using nonlinear constraints, including forward direction constraints and vertical direction constraints.

[0099] The forward direction constraint is a constraint imposed when the vehicle travels in a straight line on a plane, and the displacement change of the camera is the same as the displacement change of the vehicle's center. This constraint can be derived as follows: With R c2 The second column of v is parallel, and the following uses... This represents the x-th column of the matrix, where constraints exist:

[0100]

[0101] Vertical constraints are derived by utilizing the fact that the vehicle is always positioned on the ground plane, allowing us to obtain the ground plane normal vector. With R c2v The third column parallel to R c2v There are constraints

[0102]

[0103] The rotation matrix is ​​optimized by combining these optimization objectives.

[0104] This embodiment optimizes the camera pose matrix based on multiple sets of matched feature points to obtain a more accurate camera pose matrix.

[0105] Furthermore, after obtaining the optimized position calibration results for all cameras, image data from all cameras in the vehicle surround view system are collected while the vehicle is stationary. Combined with camera intrinsic parameters, initial camera position values, and the optimized camera rotation matrix, a top-down view is generated, and position calibration is performed again to further improve camera calibration accuracy. Specific steps include:

[0106] Based on the intrinsic parameters K of each camera i Initial value of camera position t i And the camera's rotation matrix R ci2v Generate top-down view mappings for each camera.

[0107] The relationship between the pixel coordinate system and the world coordinate system in the top view is as follows:

[0108]

[0109] Relationship between the world coordinate system and individual cameras

[0110]

[0111] The relationship between the top view and the coordinate systems of each camera image is obtained as follows:

[0112]

[0113] After obtaining the top-view, based on the overlapping field of view, a ROI region is determined in the top-view of each adjacent camera. Feature point detection and matching are performed within this region to obtain the coordinates of the feature points in the top-view, which are then converted into the corresponding camera image coordinates. Using the camera image coordinates of the feature point sets in the adjacent top-views, combined with camera intrinsic parameters, position, and pose, a triangulation algorithm is used to obtain the 3D coordinates of the feature points. Based on the 3D coordinates of the feature points in each overlapping region, the camera image coordinates, combined with camera intrinsic parameters, rotation matrix, and initial position values, the BundleAdjustment algorithm is used with reprojection error as the objective function to perform nonlinear optimization of the camera position. Thus, the pose and position calibration results for each camera are obtained.

[0114] The vehicle surround view camera calibration device provided by the present invention is described below. The vehicle surround view camera calibration device described below and the vehicle surround view camera calibration method described above can be referred to in correspondence.

[0115] The vehicle-mounted surround-view camera calibration device provided by the present invention includes:

[0116] The ground feature point detection module is used to detect and match ground feature points in the current frame image and adjacent frame images to determine the feature point set obtained by matching the two frames;

[0117] The coordinate calculation module is used to determine the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system based on the feature point set.

[0118] The attitude matrix calculation module is used to obtain the camera attitude matrix based on the driving direction vector coordinates and the ground normal vector coordinates. The camera attitude matrix is ​​used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system.

[0119] The position calibration calculation module is used to determine the position calibration results of each camera based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix.

[0120] Furthermore, the position calibration calculation module is also used for:

[0121] Based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix, a top view of each camera is determined;

[0122] Feature detection and matching are performed within the overlapping area of ​​the top view to determine the set of matching points. Based on the set of matching points, camera positions are optimized to determine the position calibration results of each camera.

[0123] Furthermore, the ground feature point detection module is also used for:

[0124] Based on the image data from the calibrated camera, the current frame image and the adjacent frame images are determined;

[0125] Image processing, feature detection, and matching are performed sequentially on the current frame image and the adjacent frame images to obtain a common set of feature points.

[0126] Furthermore, the coordinate calculation module is also used for:

[0127] Based on the feature point set and the camera intrinsic parameters, determine the fundamental matrix corresponding to the camera intrinsic parameters;

[0128] Based on the fundamental matrix, the pose transformation matrix of the camera from the current frame image to the adjacent frame image is determined. The pose transformation matrix includes a rotation matrix and a translation vector, and the translation vector is the vehicle driving direction vector of the vehicle coordinate system.

[0129] Furthermore, the coordinate calculation module is also used for:

[0130] Based on the pose transformation matrix, the camera intrinsic parameters, and the feature point set, the 3D coordinates of the feature points in the camera coordinate system are determined;

[0131] The ground plane equation is fitted based on the 3D coordinates, and the ground normal vector coordinates in the camera coordinate system are determined.

[0132] Furthermore, the vehicle-mounted surround-view camera calibration device also includes a camera attitude matrix optimization module, which is further used for:

[0133] Multiple sets of vehicle translation vectors and ground normal vectors are determined using multi-frame image data, and the camera pose matrix is ​​optimized based on the multiple sets of vehicle translation vectors and ground normal vectors.

[0134] The optimization of the camera pose matrix refers to optimizing the camera pose matrix using nonlinear constraints, including forward direction constraints and vertical direction constraints.

[0135] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a vehicle-mounted surround-view camera calibration method. This method includes: detecting and matching ground feature points on the current frame image and adjacent frame images to determine a feature point set obtained by matching the two frames; based on the feature point set, determining the vehicle driving direction vector coordinates in the camera coordinate system and the ground normal vector coordinates in the camera coordinate system; based on the driving direction vector coordinates and the ground normal vector coordinates, obtaining a camera attitude matrix, which represents the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system; and determining the position calibration results of each camera based on camera intrinsic parameters, installation position parameters, and the camera attitude matrix.

[0136] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vehicle surround-view camera calibration method provided by the above methods. The method includes: detecting and matching ground feature points on the current frame image and adjacent frame images to determine a feature point set obtained by matching the two frames; determining the vehicle driving direction vector coordinates in the camera coordinate system and the ground normal vector coordinates in the camera coordinate system based on the feature point set; obtaining a camera attitude matrix based on the driving direction vector coordinates and the ground normal vector coordinates, the camera attitude matrix being used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system; and determining the position calibration result of each camera based on camera intrinsic parameters, installation position parameters, and the camera attitude matrix.

[0138] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the vehicle-mounted surround-view camera calibration method provided by the above methods. The method includes: detecting and matching ground feature points in the current frame image and adjacent frame images to determine a feature point set obtained by matching the two frames; determining the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system based on the feature point set; obtaining a camera attitude matrix based on the driving direction vector coordinates and the ground normal vector coordinates, the camera attitude matrix being used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system; and determining the position calibration result of each camera based on camera intrinsic parameters, installation position parameters, and the camera attitude matrix.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for a vehicle-mounted surround-view camera, characterized in that, include: Ground feature point detection and matching are performed on the current frame image and adjacent frame images to determine the feature point set obtained by matching the two frames; Based on the feature point set, determine the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system. Based on the driving direction vector coordinates and the ground normal vector coordinates, a camera attitude matrix is ​​obtained, which is used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system. Based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix, the position calibration results of each camera are determined; Based on the feature point set, the coordinates of the vehicle's driving direction vector in the camera coordinate system are determined, including: Based on the feature point set and the camera intrinsic parameters, determine the fundamental matrix corresponding to the camera intrinsic parameters; Based on the fundamental matrix, the pose transformation matrix of the camera from the current frame image to the adjacent frame image is determined. The pose transformation matrix includes a rotation matrix and a translation vector. The translation vector is the vehicle driving direction vector in the vehicle coordinate system. In the forward straight-line state, the vehicle translation vector is the direction vector of the y-axis of the vehicle coordinate system in the camera coordinate system.

2. The vehicle-mounted surround-view camera calibration method according to claim 1, characterized in that, Based on camera intrinsic parameters, installation position parameters, and the camera attitude matrix, the position calibration results for each camera are determined, including: Based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix, a top view of each camera is determined; Feature detection and matching are performed within the overlapping area of ​​the top view to determine the set of matching points. Based on the set of matching points, camera positions are optimized to determine the position calibration results of each camera.

3. The vehicle-mounted surround-view camera calibration method according to claim 1, characterized in that, Determining the feature point set includes: Based on the image data from the calibrated camera, the current frame image and the adjacent frame images are determined; Image processing, feature detection, and matching are performed sequentially on the current frame image and the adjacent frame images to obtain a common set of feature points.

4. The vehicle-mounted surround-view camera calibration method according to claim 1, characterized in that, Based on the feature point set, determining the ground normal vector coordinates in the camera coordinate system includes: Based on the pose transformation matrix, the camera intrinsic parameters, and the feature point set, the 3D coordinates of the feature points in the camera coordinate system are determined; The ground plane equation is fitted based on the 3D coordinates, and the ground normal vector coordinates in the camera coordinate system are determined.

5. The vehicle-mounted surround-view camera calibration method according to claim 1, characterized in that, include: Multiple sets of vehicle translation vectors and ground normal vectors are determined using multi-frame image data, and the camera pose matrix is ​​optimized based on the multiple sets of vehicle translation vectors and ground normal vectors. The optimization of the camera pose matrix refers to optimizing the camera pose matrix using nonlinear constraints, including forward direction constraints and vertical direction constraints.

6. A vehicle-mounted surround-view camera calibration device, characterized in that, include: The ground feature point detection module is used to detect and match ground feature points in the current frame image and adjacent frame images to determine the feature point set obtained by matching the two frames; The coordinate calculation module is used to determine the coordinates of the vehicle driving direction vector in the camera coordinate system and the coordinates of the ground normal vector in the camera coordinate system based on the feature point set. The attitude matrix calculation module is used to obtain the camera attitude matrix based on the driving direction vector coordinates and the ground normal vector coordinates. The camera attitude matrix is ​​used to represent the attitude transformation relationship between the camera coordinate system and the vehicle coordinate system. The position calibration calculation module is used to determine the position calibration results of each camera based on the camera intrinsic parameters, installation position parameters, and the camera attitude matrix. Based on the feature point set, the coordinates of the vehicle's driving direction vector in the camera coordinate system are determined, including: Based on the feature point set and the camera intrinsic parameters, determine the fundamental matrix corresponding to the camera intrinsic parameters; Based on the fundamental matrix, the pose transformation matrix of the camera from the current frame image to the adjacent frame image is determined. The pose transformation matrix includes a rotation matrix and a translation vector. The translation vector is the vehicle driving direction vector in the vehicle coordinate system. In the forward straight-line state, the vehicle translation vector is the direction vector of the y-axis of the vehicle coordinate system in the camera coordinate system.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the vehicle surround view camera calibration method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the vehicle surround view camera calibration method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the vehicle surround view camera calibration method as described in any one of claims 1 to 5.

Images