Method and device for calibrating 360-degree vehicle-mounted looking-around camera based on ground texture features

Abstract

The invention discloses a 360 vehicle-mounted around-view camera calibration method and device based on ground texture features, and the method comprises the steps: enabling a camera with calibrated internal parameters to be installed and fixed on a vehicle, measuring the installation position of the camera, controlling the vehicle to go straight at a low speed on a flat and wide ground with certain textures, detecting and matching ground feature point sets of adjacent frames, and carrying out the calibration of a 360-degree vehicle-mounted around-view camera. Calculating the position and attitude change of the vehicle by using the internal reference and the feature points; further calculating 3d coordinates of the ground feature points, and fitting to obtain a ground normal vector; calculating a rotation matrix of a camera coordinate system and a world coordinate system, and calculating multiple groups of vectors by using multiple frames of data to optimize the rotation matrix; and obtaining a top view of each camera by using the rotation matrix and the camera position initial value, collecting feature points of adjacent overlapping areas, and performing position optimization on all cameras of the surround view system. According to the method, calibration auxiliary tools are not needed, a site does not need to be printed in advance, calibration can be achieved only through a flat ground with certain textures, and the possibility of errors caused by personnel operation and measurement is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of 360-degree vehicle-mounted surround view system calibration, and in particular to a 360-degree vehicle-mounted surround view camera calibration method and device based on ground texture features. Background Art

[0002] With the development of image technology, computer vision is widely used in the automotive field. Surround view cameras are installed on vehicles. Images around the vehicle are collected by multiple cameras at the same time. After image processing, a panoramic bird's-eye view projection image around the vehicle is finally synthesized to show the location and surrounding conditions of the vehicle, which can effectively reduce accidents. Among them, the calibration of the surround view camera system using image processing technology is the key supporting technology for the stitching of panoramic bird's-eye views.

[0003] However, the traditional surround-view camera calibration requires the use of specific calibration objects. In the pre-installation scenario, a special calibration workshop is usually built by printing specific shapes of marks (chessboard, dots or parallel lines) on the ground, and the vehicle is parked in the middle of the site. The surround-view camera is calibrated by identifying the ground marks and the pre-set mark pattern size; in the after-installation scenario, a certain number of specific markers need to be manually laid around the vehicle body, and the distance between these patterns or the size of the patterns themselves need to be measured. After the recognition process, the relative relationship between the cameras is calculated and calibrated to complete the final surround-view device calibration.

[0004] This design has the following disadvantages: for the traditional surround calibration method using specific calibration objects, the calibration accuracy depends largely on the measurement accuracy or printing accuracy of the calibration object's dimensional data; the calibration object may be transported back and forth, and the printed pattern may be repeatedly rolled, which may cause defects or blurs, adding additional production, transportation and maintenance costs; different manufacturers have different solutions, and the requirements and sizes of the patterns are different, making them incompatible and reusable. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a 360-degree surround camera calibration method and device without the assistance of specific calibration objects.

[0006] Considering that the ground of the road where the vehicle is traveling usually has certain texture features and various traffic signs (such as lane lines, zebra crossings, arrow marks, etc.), these road textures can be used for feature extraction, and the camera calibration can be performed in conjunction with the constraints of the vehicle body movement. Therefore, the present invention provides a method for calibrating a vehicle-mounted surround view camera based on ground texture features. There is no need to specify a calibration pattern. Only a wide, flat ground with certain textures is needed to achieve calibration. This not only reduces the cost of transporting and maintaining the calibration object, but also reduces the error caused by measurement, making it more convenient and efficient.

[0007] In a first aspect, in order to achieve the above technical objectives, the present invention provides a 360-degree vehicle-mounted surround view camera calibration method based on ground texture features, comprising:

[0008] 1. Install and fix four fisheye cameras, calibrate the camera internal parameters K and distortion parameters D, and measure the installation positions of each camera on the vehicle body, including the three-dimensional coordinates P of the four cameras in the world coordinate system with the center of the vehicle as the origin. cam (xyz);

[0009] 2. Detect and match ground feature points of the camera's current frame image F1 and the adjacent frame image F0, filter the matching point set, and obtain the feature point set E obtained by matching the two frames of images;

[0010] 3. Based on the feature point set E, calculate the direction vector of the vehicle in the camera coordinate system and the ground normal vector The coordinates of

[0011] 4. Based on the vehicle driving direction vector and the ground normal vector And the cross product vector of the first two vectors After normalization, we get the three coordinate axis vectors of the world coordinate system in the camera coordinate system and solve the camera attitude matrix R c2w , the camera posture matrix R c2w Used to represent the transformation relationship between the camera coordinate system and the world coordinate system;

[0012] 5. Based on the camera intrinsic parameter K, distortion parameter D, and installation position parameter P cam (xyz) and the camera posture matrix R c2w , perform initial extrinsic calibration on the four cameras of the surround view system and fit the extrinsic matrix of each camera;

[0013] 6. Based on the camera intrinsic parameter K, distortion parameter D and the extrinsic parameter matrix of each camera, determine the calibration result of each camera, and obtain the top view projection of each camera through pixel mapping;

[0014] 7. Based on the results of the initial calibration, the image is grayed, eroded and expanded, and other morphological processing and feature point extraction are performed in the overlapping area of ​​the field of view of two adjacent cameras in the resulting bird's-eye view. The initial extrinsic parameters are optimized using the construction error evaluation algorithm and the point cloud registration algorithm to obtain a bird's-eye view of the four-camera screen. Based on the optimized extrinsic parameter results, the bird's-eye view projection images of the four surround-view cameras are merged into one image, and the image fusion is performed on the overlapping area of ​​the four cameras to complete the camera attitude calibration and generate a 360-degree bird's-eye view projection image.

[0015] Wherein, step S1 comprises:

[0016] Install 4 fisheye cameras on the vehicle body, adjust the camera installation angle to ensure that the road surface occupies most of the camera screen, and calibrate the intrinsic parameters and distortion parameters of all cameras;

[0017] Choose a flat road with a lot of textures, drive the vehicle steadily, avoid bumpy roads, control the vehicle speed within 15km / h, and try to maintain a constant speed to avoid standing still for long periods of time.

[0018] Wherein, step S2 comprises:

[0019] Determine the current frame image and the adjacent frame image based on the original image data of the surround view camera;

[0020] Performing image processing, feature detection and matching on the current frame image and the adjacent frame image in sequence to obtain a feature point set shared by the two frame images;

[0021] Use feature point set screening algorithm to screen out point pairs with large matching errors;

[0022] Wherein, step S3 comprises:

[0023] Based on the feature point set and the camera's intrinsic parameters and distortion parameters, feature point matching is performed between two frames of images, and a basic matrix corresponding to the camera's intrinsic parameters is solved through epipolar geometry constraints;

[0024] Based on the basic matrix, solving the camera pose change from the previous frame image to the current frame image;

[0025] By changing the camera pose between two frames, the direction vector of the vehicle's driving direction in the world coordinate system is calculated;

[0026] Based on the feature point set in the current frame image and the calculated inter-frame pose change, the 3D coordinates of the feature points in the current frame image are restored through the triangulation principle;

[0027] The ground plane equation is fitted based on the 3D coordinate point set, and the normal vector of the ground plane in the camera coordinate system is determined.

[0028] Wherein, step S4 comprises:

[0029] Using multiple frames of image data, determine the vehicle driving direction vector and ground normal vector in multiple camera coordinate systems, and filter the two vectors according to the vehicle driving conditions to obtain a more accurate result;

[0030] Based on the vehicle driving direction vector and the ground normal vector, the right vector is solved, and after normalization, the approximate vectors of the three coordinate axes of the world coordinate system in the camera coordinate system are obtained, and the camera rotation matrix is ​​fitted;

[0031] Use subsequent multi-frame image data to optimize the fitted camera posture matrix;

[0032] The optimization refers to optimizing the camera posture matrix using nonlinear constraints including a forward direction constraint, a vertical direction constraint, and a vehicle rotation direction constraint.

[0033] Wherein, step S5 comprises:

[0034] Based on the measured camera installation position coordinates, the relationship between each camera position and the world coordinate system is solved to obtain the translation vector of each camera;

[0035] Concatenate the above rotation matrix and translation vector to obtain the extrinsic matrix of each camera.

[0036] Wherein, step S6 comprises:

[0037] Perform internal and external parameter mapping on the image of each camera to achieve camera distortion removal and world coordinate system projection, and obtain a bird's-eye view of each camera;

[0038] A scale factor is added to the mapping relationship of each camera to determine the size of the mapped image and unify the world coordinate system scales of the four cameras.

[0039] Wherein, step S7 comprises:

[0040] Based on the top view of the above four cameras, feature point detection and matching are performed in the overlapping area of ​​the camera images, and the point cloud is registered using the matched point set to further optimize the external parameter matrix of the four cameras;

[0041] Based on the obtained top-view images of the four cameras and the registered external parameters, the image fusion algorithm is used to project the top-view images of the four cameras into the same image, and the fusion transition is performed in the overlapping area of ​​the cameras.

[0042] A second aspect of the present invention relates to a vehicle-mounted surround view camera calibration device, comprising:

[0043] The fisheye camera video acquisition module is used to acquire the images of the fisheye camera and cache the current frame image data and the adjacent frame image data;

[0044] A ground feature point detection module is used to detect and match ground feature points on the current frame image and the adjacent frame image acquired by the video acquisition module, and determine a set of matched feature points in the two frame images;

[0045] A coordinate vector calculation module, used to determine the driving direction vector of the vehicle and the normal vector of the ground plane fitted by the ground feature points in the world coordinate system based on the feature point set;

[0046] A camera attitude matrix calculation module, used to obtain a camera attitude matrix based on the vehicle driving direction vector and the ground normal vector, wherein the camera attitude matrix is ​​used to represent the change relationship between the camera coordinate system and the world coordinate system;

[0047] The camera attitude calibration calculation module is used to determine the position calibration results of each camera based on the camera intrinsic parameters, installation position parameters and the camera attitude matrix.

[0048] A third aspect of the present invention provides a calibration device for a vehicle-mounted camera, comprising:

[0049] The power management module is used to supply power to the entire hardware platform and ensure the stable operation of each module;

[0050] The video signal decoding module is used to collect and process the video images of the four cameras installed around the vehicle, and obtain the image data of the camera while the vehicle is moving and the image data of the previous frame of the vehicle moving;

[0051] The core processing module processes the image data transmitted by the video signal decoding module, calculates the relative posture change of the camera in the two frames according to the current frame image data and the previous frame image data during the vehicle's driving process, calibrates the ground according to the obtained posture change, obtains the calibration result of the ground, and solves and optimizes the external parameters of the corresponding camera according to the vehicle's motion direction vector and the ground normal vector, calibrates each camera one by one, and synthesizes a 360-degree bird's-eye view projection image;

[0052] The video output module obtains the 360-degree bird's-eye view projection image according to the core processing module, outputs the image data, and displays the image on the vehicle-mounted display.

[0053] The present invention overcomes the defect in the prior art that it is difficult to ensure the calibration accuracy due to the need for specific calibration objects to assist in calibration, improves the accuracy and efficiency of camera calibration, reduces the cost of transportation and maintenance of calibration objects, and reduces the impact of measurement errors, making it more convenient and efficient.

[0054] The advantages of the present invention are: no calibration auxiliary tools are required, no pre-printed site is required, and calibration can be achieved on a flat ground with a certain texture, thereby reducing the possibility of errors caused by human operation and measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention.

[0056] Figure 1A flow chart of a vehicle-mounted surround view camera calibration method based on road texture features disclosed in the present invention;

[0057] Figure 2 An illustration of the camera coordinate system described in the present invention and the world coordinate system defined on the vehicle;

[0058] Figure 3 A schematic diagram of the coordinate system positions between adjacent frames described in the present invention;

[0059] Figure 4 It is a module diagram of the hardware platform described in the present invention. DETAILED DESCRIPTION

[0060] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0061] Example 1

[0062] This embodiment proposes a 360-degree vehicle surround view system calibration method based on ground texture features. Figure 1 FIG. 1 is a flow chart of a method for calibrating a vehicle-mounted surround view camera based on ground features disclosed in the present invention. The method may include the following steps:

[0063] 1. Install 4 fisheye wide-angle cameras on the front, back, left and right sides of the vehicle body, drive steadily on the road, and collect camera image data, including:

[0064] 1.1 Calibrate the camera internal parameters to obtain the focal length, principal point, lens distortion coefficient and other camera internal parameters. The internal parameter calibration method is a mature existing technology and will not be described here;

[0065] 1.2 Fix the fisheye camera around the vehicle body and adjust the camera installation angle to ensure that the road area occupies most of the fisheye camera image;

[0066] 1.3 Measure the installation position of each camera on the vehicle, including the left and right distance, front and rear distance and horizontal height from the center point of the vehicle. Figure 2 The direction of the vehicle coordinate system shown converts the distance into 3D coordinates;

[0067] 1.4 Start the vehicle, select a flat surface with certain texture characteristics, enter the automatic calibration program, keep the vehicle moving slowly and straight, control the driving speed within 15km / h, and collect the camera images.

[0068] 2. Detect and match ground feature points of the current frame image and the adjacent frame image of the image data to determine the feature point set obtained by matching the two frames of image data, including:

[0069] 2.1 Collect the image data of the current calibration camera, and send the current frame image data F1 and the previous frame image data F0 to the image processing module for processing;

[0070] The current frame image data F1 and the previous frame image data F0 may be adjacent frame data or data with a certain frame interval, but it is necessary to ensure that the two frame data have an overlapping part;

[0071] 2.2 The image processing also includes performing image processing, feature detection, and matching on the two frames of image data F1 and F0 to obtain the image coordinates of the common feature point set in the two frames of image;

[0072] 2.3 Use the feature point set screening algorithm to screen out point pairs with large matching errors;

[0073] 3. Based on the feature point set, determine the coordinates of the direction vector of the vehicle's driving direction and the ground normal vector in the camera coordinate system, including:

[0074] 3.1 Use the feature point set shared by the two frames F1 and F0 to calculate the vehicle translation vector in the camera coordinate system and the ground normal vector

[0075] 3.2 The vehicle translation vector is the coordinate P of the origin of the world coordinate system at the previous position indicated by the F0 frame data. w0 Points to the coordinate P of the world coordinate system origin at the current position indicated by the F1 frame data w1 , the ground normal vector is the normal vector of the plane where the above feature point set is located in the camera coordinate system, that is, the normal vector of the road plane;

[0076] 3.3 In order to calculate It is necessary to use the principle of epipolar geometry and the camera internal parameters to calculate the basic matrix or essential matrix, and then recover the camera's posture change from the F0 frame position to the F1 frame position to obtain the rotation matrix R 01 and the translation vector t 01 , here is a rigid body motion, so

[0077] 3.4 In order to calculate Using camera internal parameters and rotation matrix R 01 , translation vector t 01 The image coordinates of the feature point set are used to obtain the 3D coordinates of the feature points in the camera coordinate system through the triangulation algorithm. The plane equation of the ground is fitted to obtain the normal vector of the ground in the camera coordinate system.

[0078] 4. Based on the vehicle driving direction vector and the ground normal vector, a camera rotation matrix is ​​obtained, where the camera rotation matrix is ​​used to represent the posture transformation relationship between the camera coordinate system and the world coordinate system, including:

[0079] 4.1 Use and Calculate the initial value of the camera coordinate system relative to the vehicle coordinate system: rotation matrix R c2w ;

[0080] 4.2 Here, in the forward straight state, the vehicle translation vector Along the y-axis direction of the vehicle coordinate system, that is, is the direction vector of the y-axis of the world coordinate system in the camera coordinate system; on a flat road, the ground normal vector Parallel to the z-axis of the vehicle coordinate system, that is is the direction vector of the z-axis of the world coordinate system in the camera coordinate system; under ideal conditions, and Perpendicular to each other, the coordinates of the x-axis vector of the vehicle coordinate system in the camera coordinate system It can be expressed as:

[0081]

[0082] 4.3 According to the European criterion, the following relationship holds:

[0083]

[0084] Among them, [e wx e wy e wz ] is the standard orthogonal basis of the vehicle coordinate system, [e cx e cy e cz ] is the standard orthogonal basis of the camera coordinate system, is the direction vector of the xyz axis of the vehicle coordinate system, which is actually a unit matrix. is the coordinate value of the corresponding direction vector in the camera coordinate system;

[0085] 4.4 By definition, a rotation matrix consists of the inner product of two bases, namely:

[0086]

[0087] Therefore, the above formula (3) can be expressed as:

[0088]

[0089] So we can get:

[0090]

[0091] According to this formula, we can get and Calculate the rotation matrix R c2w An initial value of .

[0092] 4.5 Using multiple frames of data to calculate multiple sets of vehicle translation vectors and the ground normal vector The rotation matrix R c2w Perform iterative optimization;

[0093] The method uses multiple frames of data to calculate multiple sets of vehicle translation vectors and the ground normal vector It means that step 3 is repeated while the vehicle is driving slowly and straight, and multiple groups of and

[0094] 4.6 Rotation Matrix R c2w Optimize the camera posture using nonlinear constraints including forward direction constraints and vertical direction constraints:

[0095] The forward direction constraint is the constraint that the vehicle moves slowly in a straight line on a plane, and the displacement change of the camera is the same as the displacement change of the center of the vehicle body. It can be deduced that With R c2w The second column is parallel, and the following is represents the x-th column of the matrix, and there is a constraint:

[0096]

[0097] The vertical constraint is to use the fact that the vehicle is always located on the ground plane, and the ground plane normal vector can be derived With R c2w The third column of is parallel, with the constraint:

[0098]

[0099] The rotation matrix is ​​optimized based on these optimization objectives;

[0100] 5. Based on the camera attitude matrix and the camera installation position parameters, fit the external parameter matrix of each camera, including:

[0101] Based on the camera installation position coordinates measured in step 1, solve the relationship between each camera position and the world coordinate system to obtain the translation vector of each camera;

[0102] Concatenate the rotation matrix obtained in step 4 and the above translation vector to obtain the extrinsic matrix of each camera;

[0103] 6. After obtaining the optimized attitude calibration results of all cameras, collect the image data of all cameras in the vehicle surround view system when the vehicle is stationary, combine the camera intrinsic parameters, the initial value of the camera position and the optimized camera rotation matrix to generate a top view, and then perform position calibration, including:

[0104] 6.1 Intrinsic Parameters K Based on Each Camera i 、The initial value of the camera position t i And the camera's rotation matrix R c2w , generate the top view mapping of each camera;

[0105] The relationship between the pixel coordinate system and the world coordinate system in the top view is

[0106]

[0107] The relationship between the world coordinate system and each camera

[0108]

[0109] The relationship between the top view and each camera image coordinate system is obtained as follows:

[0110]

[0111] The above relationship is used to map the four-camera images to obtain a bird's-eye view image of the four-camera images.

[0112] 7. Based on the top-view projection images of the four-view cameras, image fusion and registration are performed to synthesize the overall top-view image, including:

[0113] 7.1 After the images of the four cameras are projected from top view, they are cropped and spliced ​​to unify the scale. After obtaining the top view, the overlapping area of ​​the field of view of adjacent cameras is determined;

[0114] 7.2 Detect and extract texture feature points in the area within the viewing angles of the front and rear cameras to obtain the coordinates of the feature points on the top view. Use the intrinsic and extrinsic parameters of the left and right cameras to inversely map the coordinates of the texture feature points in the area of ​​the front and rear cameras back to the fisheye imaging distortion maps of the left and right cameras to obtain their coordinates.

[0115] 7.3 Using a bilinear interpolation algorithm, extract the pixels of the grayscale texture points in the region to obtain the pixel values;

[0116] 7.4 Construct a pixel projection error evaluation algorithm to calculate the difference in grayscale pixel values ​​in the fisheye imaging distortion map of the left and right cameras that are mapped back to the texture feature points in the front and rear camera areas. In order to balance the brightness error between the two cameras, the average brightness of the two camera areas is calculated, and the ratio of the average brightness of the two images, ncoef, is obtained. The loss is the sum of the mapping errors of all extracted texture feature points:

[0117]

[0118] 7.5 Establish a random search optimization strategy. The external parameter matrix is ​​composed of a rotation matrix and a translation vector, and its minimum degree of freedom is 6. To ensure the effectiveness of the random search, the search upper and lower limits are established for the six degrees of freedom. The rotation angle is set to a random search upper and lower limit of ±4 degrees, and the translation vector is set to a random search upper and lower limit of ±0.2. Multiple random searches are performed within this range, and the error sum loss of all texture feature points is continuously calculated during the search process. If the loss is reduced, the group of random parameters is used to optimize the external parameter matrix;

[0119] 7.6 To ensure the accuracy of the optimized results, a second round of random search optimization external parameter strategy is added on the basis of the first random search, and the threshold of random search is appropriately reduced to rotation angle ±1 and translation amount ±0.1, and a second round of fine optimization is carried out;

[0120] 7.7 Optimize the camera images of the four viewing angles one by one to obtain four optimized camera extrinsic parameter matrices;

[0121] 7.8 After the optimization is completed, use the optimal external parameter matrix to perform a top-down projection of the fisheye image, and stitch the top-down projection images of the four cameras;

[0122] 7.9 Perform image fusion on the overlapping areas of the four cameras in the synthesized overhead view, and transition the pixels according to specific weights to make the overlapping areas of the camera images transition smoothly, thus obtaining the final calibration result.

[0123] At this point, the attitude and position calibration results of each camera are obtained, as well as a complete 360-degree bird's-eye view.

[0124] Example 2

[0125] This embodiment proposes a 360-degree vehicle-mounted surround view system calibration device based on ground texture features, which is used to implement a 360-degree vehicle-mounted surround view system calibration method based on ground texture features in Example 1.

[0126] See also Figure 4 The module diagram of the hardware platform of the 360-degree vehicle surround view system calibration device shown in FIG includes:

[0127] 1. The power management module is used to supply power to the entire hardware platform, effectively distribute power to different components of the system, and ensure the stable operation of each module;

[0128] 2. Video signal decoding module, used to collect and decode image data from four cameras installed around the vehicle, obtain the current frame image data of the camera during the vehicle's driving process and the image data of the previous frame during the vehicle's driving process, and transmit the data to the core processing module;

[0129] 3. The core processing module processes the image data transmitted by the video signal decoding module, identifies and matches the feature points in the image according to the current frame image data and the previous frame image data during the vehicle's driving process, calculates the relative posture change of the camera in these two frames through the matched feature point set, and solves the direction vector of the camera movement according to the obtained relative posture change, fits the ground plane, obtains the calibration result of the ground, and solves and optimizes the external parameters of the corresponding camera according to the vehicle's motion direction vector, the ground normal vector and the cross product vector of the first two vectors, calibrates each camera one by one and optimizes the calibration result, and synthesizes a 360-degree bird's-eye view projection image;

[0130] 4. The video output module obtains the 360-degree overhead projection image according to the core processing module, outputs the image data, and displays the image on the vehicle display.

[0131] Example 3

[0132] This embodiment relates to a calibration device for a vehicle-mounted camera, which is used to implement a calibration method for a 360-degree vehicle-mounted surround view system based on ground texture features in Embodiment 1, including:

[0133] The power management module is used to supply power to the entire hardware platform and ensure the stable operation of each module;

[0134] The video signal decoding module is used to collect and process the video images of the four cameras installed around the vehicle, and obtain the image data of the camera while the vehicle is moving and the image data of the previous frame of the vehicle moving;

[0135] The core processing module processes the image data transmitted by the video signal decoding module, calculates the relative posture change of the camera in the two frames according to the current frame image data and the previous frame image data during the vehicle's driving process, calibrates the ground according to the obtained posture change, obtains the calibration result of the ground, and solves and optimizes the external parameters of the corresponding camera according to the vehicle's motion direction vector and the ground normal vector, calibrates each camera one by one, and synthesizes a 360-degree bird's-eye view projection image;

[0136] The video output module obtains the 360-degree bird's-eye view projection image according to the core processing module, outputs the image data, and displays the image on the vehicle-mounted display.

[0137] It should be noted that the special terms used in describing certain features or schemes of the present invention should not be used to indicate that the term is redefined here to limit certain specific features, characteristics or schemes of the present invention associated with the term. In general, the terms used in the appended claims should not be interpreted as limiting the present invention to the specific embodiments disclosed in the specification unless the above detailed description section explicitly defines these terms. Therefore, the actual scope of the present invention includes not only the disclosed embodiments, but also all equivalent schemes for implementing or performing the present invention under the claims.

Claims

1. A 360-degree vehicle-mounted surround view camera calibration method based on ground texture features, characterized in that: The method comprises the following steps: S1. Install 4 fisheye wide-angle cameras on the front, back, left and right sides of the vehicle body, drive steadily on the road, and collect camera image data; S2. Detect and match ground feature points of the current frame image and the adjacent frame image of the image data to determine the feature point set obtained by matching the two frames; S3. Based on the feature point set, the coordinates of the direction vector of the vehicle's travel direction and the ground normal vector in the camera coordinate system are calculated; S4. Based on the vehicle driving direction vector and the ground normal vector, a camera rotation matrix is ​​calculated, wherein the camera rotation matrix is ​​used to represent the posture transformation relationship between the camera coordinate system and the world coordinate system; S5. Fitting the external parameter matrix of each camera based on the camera posture matrix and the camera installation position parameters; S6. Based on the camera intrinsic parameters, distortion parameters and the extrinsic parameter matrix of each camera, the calibration results of each camera are fitted to obtain the top-view projection image of each camera. S7. Based on the top-view projection images of the four view cameras, image fusion and registration are performed to synthesize the overall top-view image.

2. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S1 includes: Install 4 fisheye cameras on the vehicle body, adjust the camera installation angle to ensure that the road surface occupies most of the camera screen, and calibrate the intrinsic parameters and distortion parameters of all cameras; Choose a flat road with a lot of textures, drive the vehicle steadily, avoid bumpy roads, control the vehicle speed within 15km / h, and try to maintain a constant speed to avoid standing still for long periods of time.

3. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S2 includes: Determine the current frame image and the adjacent frame image based on the original image data of the surround view camera; Performing image processing, feature detection and matching on the current frame image and the adjacent frame image in sequence to obtain a feature point set shared by the two frame images; The feature point set screening algorithm is used to screen out point pairs with large matching errors.

4. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S3 includes: Based on the feature point set and the camera's intrinsic parameters and distortion parameters, feature point matching is performed between the two frames of images, and the basic matrix corresponding to the camera's intrinsic parameters is solved through epipolar geometry constraints; Based on the basic matrix, solving the camera pose change from the previous frame image to the current frame image; By changing the camera's position between two frames of images, the direction vector of the vehicle's driving direction in the world coordinate system is calculated; Based on the feature point set in the current frame image and the calculated inter-frame pose change, the 3D coordinates of the feature points in the current frame image are restored through the triangulation principle; The ground plane equation is fitted based on the 3D coordinate point set, and the normal vector of the ground plane in the camera coordinate system is determined.

5. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S4 includes: Using multiple frames of image data, determine the vehicle driving direction vector and ground normal vector in multiple camera coordinate systems, and filter the two vectors according to the vehicle driving conditions to obtain a more accurate result; Based on the vehicle driving direction vector and the ground normal vector, the right vector is solved, and after normalization, the approximate vectors of the three coordinate axes of the world coordinate system in the camera coordinate system are obtained, and the camera rotation matrix is ​​fitted; Use subsequent multi-frame image data to optimize the fitted camera posture matrix; The optimization refers to optimizing the camera posture matrix using nonlinear constraints including a forward direction constraint, a vertical direction constraint, and a vehicle rotation direction constraint.

6. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S5 includes: Based on the measured camera installation position coordinates, the relationship between each camera position and the world coordinate system is solved to obtain the translation vector of each camera; Concatenate the above rotation matrix and translation vector to obtain the extrinsic parameter matrix of each camera.

7. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S6 includes: Perform internal and external parameter mapping on the image of each camera to achieve camera distortion removal and world coordinate system projection, and obtain the top view of each camera; A scale factor is added to the mapping relationship of each camera to determine the size of the mapped image and unify the world coordinate system scales of the four cameras.

8. The 360-degree vehicle-mounted surround view camera calibration method based on ground texture features according to claim 1, characterized in that: Step S7 includes: Based on the top view of the above four cameras, feature point detection and matching are performed in the overlapping area of ​​the camera images, and the matched point set is used for point cloud registration to further optimize the external parameter matrix of the four cameras; Based on the obtained top view of the four cameras and the registered external parameters, the image fusion algorithm is used to project the top view of the four cameras into the same image, and the fusion transition is performed in the overlapping area of ​​the cameras.

9. A vehicle-mounted surround view camera calibration device for implementing the method of claim 1, characterized in that: include: The fisheye camera video acquisition module is used to acquire the images of the fisheye camera and cache the current frame image data and the adjacent frame image data; A ground feature point detection module is used to detect and match ground feature points on the current frame image and the adjacent frame image acquired by the video acquisition module, and determine a set of matched feature points in the two frame images; A coordinate vector calculation module, used to determine the driving direction vector of the vehicle and the normal vector of the ground plane fitted by the ground feature points in the world coordinate system based on the feature point set; A camera attitude matrix calculation module, used to obtain a camera attitude matrix based on the vehicle driving direction vector and the ground normal vector, wherein the camera attitude matrix is ​​used to represent the change relationship between the camera coordinate system and the world coordinate system; The camera attitude calibration calculation module is used to determine the position calibration results of each camera based on the camera intrinsic parameters, installation position parameters and the camera attitude matrix.

10. A calibration device for a vehicle-mounted camera, comprising: The power management module is used to supply power to the entire hardware platform and ensure the stable operation of each module; The video signal decoding module is used to collect and process the video images of the four cameras installed around the vehicle, and obtain the image data of the camera while the vehicle is moving and the image data of the previous frame of the vehicle moving; The core processing module processes the image data transmitted by the video signal decoding module, calculates the relative posture change of the camera in the two frames according to the current frame image data and the previous frame image data during the vehicle's driving process, calibrates the ground according to the obtained posture change, obtains the calibration result of the ground, and solves and optimizes the external parameters of the corresponding camera according to the vehicle's motion direction vector and the ground normal vector, calibrates each camera one by one, and synthesizes a 360-degree bird's-eye view projection image; The video output module obtains the 360-degree bird's-eye view projection image according to the core processing module, outputs the image data, and displays the image on the vehicle-mounted display.

Images