Engineering vehicle multi-camera external parameter calibration method based on multiple constraints and adaptive optimization

Abstract

In the application of a multi-camera visual system of an engineering vehicle, multi-camera external parameter calibration precision is crucial to system performance, but a traditional calibration method often faces the problems of insufficient precision, large error, obvious visual angle fusion deviation and the like. Therefore, the invention discloses an engineering vehicle multi-camera external parameter calibration method based on multiple constraints and adaptive optimization, and the method comprises the following steps: employing a ChArUco calibration board, employing a Zhang Zhengyou calibration method to obtain the internal parameters of fisheye cameras, and combining with an L-M algorithm to complete the initial external parameter calibration; by extracting an accurate mask area in a public view field, computing resources are concentrated in an overlapping area for optimization processing; texture points are screened under triple constraints of geometric boundaries, regions of interest and gray gradients, and coarse calibration and fine calibration optimization of external parameters are sequentially carried out by taking re-projection errors of the texture points as a loss function; through the adaptive iterative optimization mechanism, the system error is effectively reduced, and the geometric accuracy and view consistency of image splicing are remarkably improved.

Description

Technical Field

[0001] The present invention relates to fields such as computer vision and multi-camera system calibration, and in particular to a multi-camera extrinsic parameter calibration method for engineering vehicles based on multiple constraints and adaptive optimization. Background Art

[0002] With the development of intelligent engineering machinery, multi-camera vision systems are increasingly used in engineering vehicles such as excavators, bulldozers, cranes, and self-driving mining trucks. This type of vision system is mainly used for tasks such as environmental perception, blind spot monitoring, precise positioning, and intelligent assisted driving to improve vehicle operating efficiency and safety. In a multi-camera vision system, the accuracy of the camera's external parameters (abbreviated as "extrinsic parameters"), namely the camera's position and attitude information, is crucial to the overall performance of the system. Errors in extrinsic parameters may lead to a decrease in the accuracy of multi-camera perspective fusion, which in turn affects the quality of image stitching and the accuracy of target detection. Therefore, ensuring the accuracy of extrinsic parameters is a key step in optimizing multi-camera vision systems.

[0003] Currently, the commonly used multi-camera extrinsic calibration method usually relies on a calibration cloth as a reference object, which is placed around the vehicle to complete the camera calibration. However, this method has the following shortcomings in the application scenario of engineering vehicles:

[0004] (1) The generalization ability of the calibration scheme is weak

[0005] Due to the wide variety of engineering vehicles and their significant size differences, the traditional calibration cloth solution has poor versatility. Different types of engineering vehicles require calibration cloths of different sizes and shapes to adapt to their specific field of view requirements, which complicates the calibration process and increases costs. At the same time, even if a dedicated calibration cloth is designed for a specific vehicle, the solution is still difficult to apply to other vehicles and lacks good adaptability and reusability. In addition, the working environment of engineering vehicles is changeable, and external factors (such as uneven ground, changes in lighting, etc.) may affect the stability and calibration accuracy of the calibration cloth, which limits the application of external parameter calculation methods based on calibration cloth in engineering vehicles.

[0006] (2) Cumulative camera attitude offset

[0007] In practical applications, engineering vehicles operate under complex conditions for long periods of time and are inevitably affected by vehicle vibrations and environmental interference. For example, when driving on rough roads or performing tasks such as excavation, lifting, and steering, the vehicle's mechanical components generate irregular impacts and vibrations, which are transmitted to the camera installation location. These interference factors may cause the physical position of the camera to shift, which in turn leads to the accumulation of extrinsic parameter errors, affecting the accuracy of image stitching and multi-camera fusion. Because engineering vehicles typically need to operate stably for long periods of time, their vision systems need to have adaptive capabilities to dynamically adjust the camera's extrinsic parameters to compensate for the accumulated deviations caused by vibration and environmental changes.

[0008] Given the above issues, there is an urgent need to develop a high-precision multi-camera extrinsic parameter calibration method suitable for the actual environment of engineering vehicles. To this end, this paper proposes a multi-camera extrinsic parameter calibration method for engineering vehicles based on multiple constraints and adaptive optimization. This method uses the LM algorithm to obtain initial extrinsic parameters and combines it with an adaptive iterative optimization mechanism to achieve dynamic adjustment and optimization of the extrinsic parameters of the multi-camera system. This effectively reduces the stitching error caused by camera perspective offset and improves the accuracy and reliability of the engineering vehicle vision system. Summary of the Invention

[0009] The present invention proposes a multi-camera extrinsic parameter calibration method for engineering vehicles based on multiple constraints and adaptive optimization, which is used to solve problems such as weak generalization ability of engineering vehicle calibration schemes and physical position offset caused by vehicle operation.

[0010] The technical solution of the present invention to solve the above problems is: a method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization, comprising the following steps:

[0011] S1: Using the ChArUco calibration board, the Zhang Zhengyou calibration method is used to obtain the fisheye camera intrinsic parameters and the initial extrinsic parameters are calibrated using the LM algorithm;

[0012] S2: By extracting accurate mask areas within the common field of view, computing resources are concentrated in the overlapping areas for optimization processing;

[0013] S3: Screening texture points based on the triple constraint mechanism of geometric boundary constraint, region of interest constraint and grayscale gradient constraint;

[0014] S4: iterative optimization of external parameter rough calibration;

[0015] S5: Iterative optimization of external parameter fine calibration.

[0016] Furthermore, the specific method of step S1 is as follows:

[0017] S1.1: Move the ChArUco calibration plate multiple times to collect images, and use the Zhang Zhengyou calibration method to calculate the intrinsic parameter matrix K and distortion coefficient D of the multi-camera group;

[0018] S1.2: Collect the 3D coordinate information of the feature points on the ChArUco calibration plate, and extract the 2D coordinates of the feature points in the pixel coordinate system by collecting the ChArUco calibration plate image, and establish the correspondence between the 2D and 3D coordinates of the feature points;

[0019] S1.3: Using the relationship between the two-dimensional and three-dimensional coordinates of feature points and the camera internal parameters as the objective function to minimize the reprojection error, the L-M least squares method is used to iteratively optimize the camera pose parameters, and finally the pose parameters of the camera relative to the world coordinate system, that is, the external parameters of the camera, are accurately solved.

[0020] Furthermore, the specific method of step S2 is as follows:

[0021] S2.1: Use the camera internal parameter matrix K and distortion coefficient D obtained in S1.1 to undistort the original image of the fisheye camera;

[0022] S2.2: Convert the undistorted image into a grayscale image and binarize the grayscale image using a fixed threshold algorithm to obtain a binary image. The binarization method is as follows: take the threshold as τ1:

[0023]

[0024] In the formula, I represents the pixel grayscale value;

[0025] S2.3: Create erosion kernels and dilation kernels, and perform dilation and erosion on the image to smooth and eliminate noise;

[0026] S2.4: Implement the extraction process of the common vision mask based on the following formula:

[0027]

[0028] In the formula, I1(i, j) and I2(i, j) respectively represent the pixel matrices of the input images for extracting the common mask, where 0 ≤ i < rows and 0 ≤ j < cols, and rows and cols are the length and width of the input image; D(i, j) represents the pixel matrix of the target mask image.

[0029] Furthermore, the specific method of step S3 is as follows:

[0030] S3.1: Perform geometric boundary constraints. Let the size of the image be M×N. The boundary constraints ignore all points less than 10 pixels away from the image edge. Specifically, for any pixel point (x, y) in the image, if it satisfies the following conditions, the point will be excluded:

[0031] min(x, y, M - x - 1, N - y - 1) < 10

[0032] S3.2: Perform region of interest constraints. Let the camera center of the image be C = (C y , x , C y), and define a circular region of interest with the camera center as the center and a radius of size×4.5 / 5. For each pixel point (x, y), if the point is within the region of interest, it is retained, otherwise it is ignored. Specifically, for any pixel point (x, y) in the image, if the following conditions are not met, the point will be excluded:

[0033]

[0034] Where size is the maximum circle radius determined based on the overlapping field of view size;

[0035] S3.3: Perform grayscale gradient constraint. Let the grayscale value of the pixel (x, y) be I(x, y). Calculate the grayscale difference between the current pixel (x, y) and the pixel to its left (x-2, y) and the pixel above (x, y-2). Obtain a two-dimensional vector δ, as shown below:

[0036] δ=(|I(x,y)-I(x-2,y)|,|I(x,y)-I(x,y-2)|)

[0037] Then calculate the size of the grayscale difference |δ|, that is:

[0038]

[0039] If |δ| is less than 15, the texture of the feature point is considered insignificant and is ignored.

[0040] Furthermore, the specific method of step S4 is as follows:

[0041] S4.1: Initialize a random integer generator R and randomly generate a set of external parameter candidates within the specified search range according to the preset resolution Δr = 1 / 100:

[0042] P i ={R i ,T i},i=1,2,...,N

[0043] Where, P i is a randomly generated external parameter; R i is the Euler angle; T i is the translation vector; N is the random external parameter generated in this iteration;

[0044] S4.2: For each set of randomly generated camera extrinsic parameters P i , calculate the corresponding texture point reprojection error E(P i ), the reprojection error is defined as:

[0045]

[0046] Where E(P) is the reprojection error; M is the total number of projection points; (x j ,y j ) is the coordinate of the texture point on the camera projection bird's-eye view coordinate system I1; (x' j ,y' j ) is the coordinate of the camera projection bird's-eye view coordinate system I2; I1(x j ,y j ) and I2(x′ j ,y′ j ) are the grayscale values of I1 and I2 at the corresponding coordinates;

[0047] S4.3: In each iteration, the reprojection error E(P i ) and the reprojection error E(P init ), select the external parameter P that minimizes the error i Denoted as P best , as follows:

[0048]

[0049] S4.4: After multiple iterations, record the external parameter P that minimizes the reprojection error in the current iteration. best , select P best As the optimization result of the coarse calibration stage.

[0050] Furthermore, the specific method of step S5 is as follows:

[0051] S5.1: Initialize a random floating point real number generator R and randomly generate a candidate set of external parameters:

[0052] P i ={R i ,T i},i=1,2,...,N

[0053] Where, P i is a randomly generated external parameter; R i is the Euler angle; T i is the translation vector; N is the random external parameter generated in this iteration;

[0054] S5.2: For each set of randomly generated external parameters P i , calculate the corresponding texture point reprojection error E(P i ), the reprojection error is defined as:

[0055]

[0056] Where E(P) is the reprojection error; M is the total number of projection points; (x j,y j ) is the coordinate of the texture point on the camera projection bird's-eye view coordinate system I1; (x′ j ,y′ j ) is the coordinate of the camera projection bird's-eye view coordinate system I2; I1(x j ,y j ) and I2(x′ j ,y′ j ) are the grayscale values of I1 and I2 at the corresponding coordinates;

[0057] S5.3: In each iteration, the reprojection error E(P i ) and the reprojection error E(P init ), select the external parameter P that minimizes the error i Denoted as P best , as follows:

[0058]

[0059] S5.4: After multiple iterations, record the external parameter P that minimizes the reprojection error in the current iteration. best , select P best As the optimization result of the fine calibration stage.

[0060] The beneficial effects of the present invention are:

[0061] The present invention adopts the ChArUco calibration plate, which combines the advantages of the checkerboard and ArUco codes, effectively solves the errors caused by the confusion of point sequences in traditional calibration plates, significantly improves the calibration accuracy and reliability, and enhances the generalization ability of the calibration scheme; based on the LM iterative optimization algorithm to solve the external parameters of the multi-camera group, combined with the camera internal parameters, the three-dimensional coordinate information of the feature points and other data, continuously iteratively updates the pose estimation, while ensuring stability and minimizing the reprojection error; by extracting the precise mask area in the common field of view area, the computing resources are concentrated in the overlapping area for optimization processing, which not only ensures the accuracy of the external parameter correction, but also reduces the computational complexity and improves the efficiency of the algorithm; adopts a multi-constraint collaborative screening strategy to significantly improve the robustness and reliability of feature point extraction, and effectively reduce environmental interference and system errors; by adopting an external parameter optimization method combining coarse calibration and fine calibration, the geometric accuracy and visual quality of image stitching are significantly improved. Figure 1 The consistency enables the multi-camera system to achieve a stable and clear bird's-eye view stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of a method in an embodiment of the present invention;

[0063] Figure 2Schematic diagram of a fisheye camera in an embodiment of the present invention;

[0064] Figure 3 The input fisheye camera original image in the embodiment of the present invention;

[0065] Figure 4 Schematic diagram of the geometry of the PNP problem in an embodiment of the present invention;

[0066] Figure 5 The ChArCuo calibration plate (left) and its corner detection effect (right) in an embodiment of the present invention;

[0067] Figure 6 This is a diagram showing the effect of fisheye image dedistortion in an embodiment of the present invention;

[0068] Figure 7 Schematic diagram of the common field of view of the front camera and the left camera in an embodiment of the present invention;

[0069] Figure 8 is the extracted common view mask image in the embodiment of the present invention;

[0070] Figure 9 A public field of view texture point extraction map in an embodiment of the present invention;

[0071] Figure 10 This is an enlarged view of the details of the public field of view texture point in an embodiment of the present invention;

[0072] Figure 11 This is a bird's-eye view image after rough calibration in an embodiment of the present invention;

[0073] Figure 12 This is a bird's-eye view after fine calibration in an embodiment of the present invention;

[0074] Figure 13 This is a comparison chart of the splicing and fusion effects before and after rough calibration in an embodiment of the present invention. Specific implementation plan

[0075] The present invention will be further described below with reference to the accompanying drawings and specific implementation examples.

[0076] like Figure 1 As shown, the embodiment of the present invention discloses a method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization, which mainly includes the following steps:

[0077] (1) Fisheye camera original image input. The embodiment of the present invention adopts Figure 2 The fisheye camera shown in the figure is used to collect images. The specific technical parameters of the camera are shown in Table 1. The original image samples collected by the fisheye camera system are as follows: Figure 3 shown.

[0078] Table 1: Specific technical parameters of fisheye cameras

[0079]

[0080] (2) Calibration of fisheye camera intrinsic parameters. The PNP (Perspective-N-Point) problem is the "perspective n-point problem", which involves estimating the camera's pose from a set of 3D points and their corresponding 2D image points. Its geometric diagram is shown in the figure below. Figure 4 The geometric structure of this problem can be described as: given the coordinates of a 3D point, the coordinates of its corresponding 2D point, and the intrinsic parameter matrix K, the camera's pose, i.e., the extrinsic parameters, can be solved. The embodiments of the present invention obtain the initial extrinsic parameters of the camera by solving the PNP problem.

[0081] Therefore, the effective implementation of the embodiments of the present invention relies on accurate matching of 3D spatial feature points and their corresponding 2D image plane points. The accuracy of this matching process is closely related to the accuracy of intrinsic calibration, as the camera's intrinsic parameters directly determine the spatial positioning accuracy of image feature point extraction. Large intrinsic parameter errors can lead to deviations in the calculation of feature point correspondences, significantly reducing the accuracy of pose estimation.

[0082] To ensure the accuracy of the internal reference calibration, the embodiment of the present invention uses a fisheye camera to collect ChArCuo calibration plate images from multiple perspectives, and uses Zhang Zhengyou calibration method to perform internal reference calibration. Figure 5 As shown in Figure 2, it combines the advantages of the checkerboard grid commonly used in traditional methods with the ArUco code: on the one hand, it retains the regular structure of the checkerboard grid, facilitating rapid detection of corner points; on the other hand, it fully utilizes the characteristics of ArUco code detection speed, strong robustness, and unique encoding to achieve automatic numbering of corner points. This automated process reduces calibration time and labor costs, ensures that each corner point position can be mapped one-to-one to the corresponding ArUco code, effectively solves the errors caused by point sequence confusion in traditional calibration plates, and significantly improves calibration accuracy and reliability. This method obtains accurate intrinsic parameter matrix K and distortion coefficient D. The specific calibration results are shown in Table 2.

[0083] Table 2: Internal reference calibration results

[0084]

[0085] (3) LM algorithm calibrates the initial extrinsic parameters of the multi-camera group. In the process of calibrating the extrinsic parameters of the multi-camera group, the embodiment of the present invention adopts an algorithm based on LM (Levenberg-Marquardt) iterative optimization to solve the extrinsic parameters of the multi-camera group. The LM iterative method has obvious advantages in solving the PNP problem, especially in cases with large noise, small number of points (n≤5), or high precision. It minimizes the reprojection error through iterative optimization, can maintain high accuracy and robustness in complex environments, and is suitable for a variety of practical application scenarios.

[0086] The core idea of LM iterative optimization method to solve PNP problem is to optimize the camera posture by minimizing the reprojection error. i , whose reprojection error vector e i is defined as:

[0087] e i =p i -f(K(RP i +t),D)

[0088] Where p i is the actual observed coordinate of the two-dimensional image point; P i is the corresponding three-dimensional world point coordinate; f is the projection function, which projects the three-dimensional point to the two-dimensional image plane while considering the distortion; R is the rotation matrix; t is the translation vector; K is the intrinsic parameter matrix; D is the distortion coefficient.

[0089] The goal of iterative optimization is to find the rotation matrix R and translation vector t that minimize the reprojection error of all points, as shown below:

[0090]

[0091] Where N is the number of three-dimensional points; ||e i || 2 is the L2 norm of the error vector.

[0092] The specific iterative optimization method is to use the LM algorithm to solve the above nonlinear least squares problem. First, assume that the camera is located near the origin of the world coordinate system, so as to initialize R and t; secondly, use the current R and t to project the 3D point onto the 2D image plane and calculate the reprojection error e i ; Then, calculate the Jacobian matrix J of the error function for R and t, which is used to describe the gradient of the error with the change of parameters; finally, use the LM algorithm to update R and t. The specific formula is as follows:

[0093] Δx=(J T J+λI) -1 J T e i

[0094] Where Δx is the parameter update amount (including ΔR and Δt); I is the identity matrix; λ is the damping factor introduced by the LM algorithm compared to the Gauss-Newton method, which is used to control the update step size.

[0095] When λ is small, the LM algorithm approaches the Gauss-Newton method, with a larger update step size. When λ is large, the LM algorithm approaches the gradient descent method, with a smaller update step size. This adaptive mechanism allows the LM algorithm to maintain convergence speed while effectively avoiding the numerical instability of the Gauss-Newton method when the Hessian matrix approaches singularity or irreversibility, thereby improving the robustness of the algorithm.

[0096] The updated R, t parameters are:

[0097] R←R+ΔR

[0098] t←t+Δt

[0099] The above steps are repeated until the reprojection error converges (i.e., the error change is less than a certain threshold) or the maximum number of iterations is reached, at which point the iterative optimization ends. The LM algorithm effectively balances convergence speed and stability by introducing a damping factor λ based on the Gauss-Newton method, significantly improving the algorithm's robustness and enabling it to better handle nonlinear optimization problems. Furthermore, iterative optimization can avoid local optimal solutions to a certain extent, especially when the initial guess is good.

[0100] In an embodiment of the present invention, a high-precision laser rangefinder is used to obtain the three-dimensional coordinate information of feature points on the ChArUco calibration plate, and the two-dimensional coordinates of the corresponding feature points in the pixel coordinate system are extracted in combination with the collected calibration image, thereby establishing a correspondence between the feature points in two-dimensional and three-dimensional space. Based on this correspondence and the camera intrinsic parameters, the LM algorithm is used to iteratively optimize the camera pose parameters with the objective function of minimizing the reprojection error. This effectively reduces the reprojection error while ensuring the optimization stability. After multiple rounds of iteration, the Euler angles and translation vectors of each camera relative to the unified world coordinate system are finally obtained, achieving high-precision extrinsic parameter estimation. The initial extrinsic parameter results of each camera are shown in Table 3.

[0101] Table 3: Initial external parameters

[0102]

[0103] (4) Fisheye camera dedistortion. The original fisheye camera image is dedistorted using the obtained camera internal parameter matrix and distortion coefficient. The obtained effect is shown in the figure below. Figure 6 shown.

[0104] (5) Texture point extraction preprocessing. In the camera group of the vehicle-mounted panoramic view system, there is a common field of view between the front camera's field of view and the left and right side cameras' field of view. In addition, there is also a certain common field of view between the rear camera's field of view and the left and right side cameras' field of view, such as Figure 7 As shown in the figure, the common field of view of the front camera and the left camera is displayed. The camera's extrinsic parameter correction is mainly achieved by calculating the minimum reprojection error of the texture points in the common field of view. After the input image is converted into a grayscale image and subjected to binarization and erosion and dilation processing, the precise mask area is extracted in the common field of view, and the computing resources are concentrated in the overlapping area for optimization processing. The extraction effect is shown in the figure. Figure 8 As shown in the figure, the white area represents the effective overlapping field of view, and the black area is the non-overlapping part. This strategy not only ensures the accuracy of extrinsic parameter correction, but also reduces the computational complexity and improves the efficiency of the algorithm.

[0105] (6) Triple constraint mechanism to screen texture points. The extraction of common field of view texture points is an important step for calculating the reprojection error. In order to improve the effectiveness of texture point screening, the embodiment of the present invention sequentially subjects the image to triple constraints of geometric edge constraints, region of interest constraints, and grayscale gradient constraints. These constraints are intended to optimize the selection of texture points and ensure the accuracy and robustness of the subsequent calculation process. In the region of interest constraint process, the size in the radius of the circular region of interest is the maximum circle radius determined based on the overlapping field of view size. If the overlapping field of view size is 340×390, the size is 170. At the same time, in order to improve the computational efficiency of the reprojection error, the embodiment of the present invention adopts a random sampling strategy to extract 500 representative samples from all candidate texture points. Figure 9 The spatial distribution of feature points after sampling is shown, and the corresponding local detail enlargement diagram is as follows Figure 10 shown.

[0106] (7) Rough calibration and fine calibration optimization of extrinsic parameters. After the texture point screening is completed, the embodiment of the present invention divides the subsequent random iterative search extrinsic parameter optimization process driven by reprojection error into two stages: rough calibration and fine calibration. The specific process of each round of calibration is as follows: First, the field of view of the front camera is fixed and kept unchanged. Based on this, the reprojection error of the texture point is used as the loss function to iteratively optimize and adjust the extrinsic parameters of the left and right cameras; after the calibration of the left and right cameras is completed, the field of view of the left and right cameras is fixed, and the calibration optimization of the rear camera is further performed. Through this gradual fixation and optimization strategy, the accuracy of the extrinsic parameters is gradually improved.

[0107] Taking the right camera as an example, the specific optimization process is as follows: initialize a random number generator to randomly generate a set of extrinsic parameter candidates; for each set of randomly generated camera extrinsic parameters, calculate the corresponding texture point reprojection error, where I1 and I2 are the bird's-eye view coordinate system diagrams of the front camera projection and the right camera projection, respectively; in each iteration, by comparing the reprojection error of the candidate extrinsic parameters with the reprojection error of the initial extrinsic parameters, select the extrinsic parameter with the minimum error and record it; after multiple iterations, select the extrinsic parameter parameter with the minimum reprojection error as the final optimization result.

[0108] The main difference between coarse calibration and fine calibration is that the range of external parameters is different during the iterative random search process, as shown in Table 4.

[0109] Table 4: External parameter ranges for coarse and fine calibration

[0110]

[0111] Fine calibration has higher processing accuracy than coarse calibration, and its optimization process can be performed at a finer scale. In addition, the parameter ranges of coarse and fine calibration can be dynamically adjusted according to actual needs. This feature reflects the flexibility of automatic external parameter calibration optimization and its robustness in different situations. Figure 11 As shown in the figure, the bird's-eye view after fine calibration is as follows Figure 12 shown.

[0112] After the extrinsic parameter optimization process based on the iterative search algorithm is completed, the embodiment of the present invention obtains the optimized extrinsic parameter data after the first round of rough calibration and the second round of fine calibration (where the initial extrinsic parameter is obtained in step (3), wherein the Euler angle optimization data is shown in Table 5, and the translation vector optimization data is shown in Table 6. Figure 13 As shown in the figure, the left side is the stitching fusion image before coarse calibration, and the right side is the stitching fusion image after coarse calibration. By comparison, it can be found that after coarse calibration optimization, the field of view images of the two cameras are accurately aligned at the pixel level, which significantly improves the image stitching quality.

[0113] Table 5: Euler angle optimization data by stage (unit: °)

[0114]

[0115] Table 6: Translation vector optimization data by stage (unit: mm)

[0116]

[0117] (8) In summary, the multi-camera extrinsic parameter calibration method for engineering vehicles based on multiple constraints and adaptive optimization proposed in this invention shows significant advantages in terms of automation and calibration accuracy. This method effectively reduces the system error introduced by manual calibration through an adaptive iterative optimization mechanism, significantly improves the geometric accuracy and visual quality of image stitching, and improves the accuracy of image stitching. Figure 1 Experimental results show that this method can enable the multi-camera system to achieve stable and clear bird's-eye view stitching effect.

Claims

1. A multi-camera extrinsic parameter calibration method for engineering vehicles based on multiple constraints and adaptive optimization, characterized by: It includes the following steps: S1: Use a ChArUco calibration board, and obtain the internal parameters of the fisheye camera by using Zhang's calibration method and calibrate the initial external parameters through the L-M algorithm; S2: By extracting an accurate mask area in the common vision area, concentrate the computing resources on the overlapping area for optimization processing; S3: Screen texture points according to the triple constraint mechanism of geometric boundary constraint, region of interest constraint and gray gradient constraint; S4: Iteratively optimize the rough calibration of the external parameters; S5: Iteratively optimize the fine calibration of the external parameters.

2. The method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization according to claim 1, characterized in that: The specific method of step S1 is as follows: S1.1: Move the ChArUco calibration board multiple times to collect images, and calculate the internal parameter matrix K and distortion coefficient D of the multi-camera group by using Zhang's calibration method; S1.2: Collect the three-dimensional coordinate information of the feature points on the ChArUco calibration board, and extract the two-dimensional coordinates in the pixel coordinate system corresponding to the feature points by collecting the ChArUco calibration board images, and establish the corresponding relationship between the two-dimensional and three-dimensional coordinates of the feature points; S1.3: Taking the relationship between the two-dimensional and three-dimensional coordinates of the feature points and the internal parameters of the camera as the objective function to minimize the reprojection error, use the L-M least squares method to iteratively optimize the camera pose parameters, and finally accurately solve the pose parameters of the camera relative to the world coordinate system, that is, the external parameters of the camera.

3. The method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization according to claim 1, characterized in that: The specific method of step S2 is as follows: S2.1: Use the internal parameter matrix K and distortion coefficient D obtained in S1.1 to perform undistortion processing on the original images of the fisheye camera; S2.2: Convert the undistorted image into a grayscale image and binarize the grayscale image by using a fixed threshold algorithm to obtain a binary image. The binarization method is as follows. Take the threshold as τ1: In the formula, I represents the pixel grayscale value; S2.3: Create an erosion kernel and a dilation kernel, and perform dilation and erosion on the image to smooth and eliminate noise; S2.4: Implement the extraction process of the common vision mask based on the following formula: In the formula, I1(i,j) and I2(i,j) respectively represent the pixel matrices of the input images for extracting the common mask, where 0 ≤ i < rows and 0 ≤ j < cols, and rows and cols are the length and width of the input image; D(i,j) represents the pixel matrix of the target mask image.

4. The method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization according to claim 1, characterized in that: The specific method of step S3 is as follows: S3.1: Perform geometric boundary constraint. Let the size of the image be M×N. The boundary constraint ignores all points less than 10 pixels away from the image edge. Specifically, for any pixel point (x,y) in the image, if it satisfies the following conditions, this point will be excluded: min(x,y,M - x - 1,N - y - 1) < 10 S3.2: Perform region of interest constraints, assuming the camera center of the image is C = (C x ,C y ), and define a circular region of interest with the camera center as the center and a radius of size×4.5 / 5. For each pixel point (x, y), if the point is within the region of interest, it is retained, otherwise it is ignored. Specifically, for any pixel point (x, y) in the image, if the following conditions are not met, the point will be excluded: In the formula, size is the maximum circle radius determined based on the overlapping vision size; S3.3: Perform gray gradient constraint. Let the grayscale value of the pixel point (x,y) be I(x,y), and calculate the gray difference between the current pixel point (x,y) and its left pixel point (x - 2,y) and upper pixel point (x,y - 2) to obtain a two-dimensional vector δ, as follows: δ = (|I(x,y) - I(x - 2,y)|,|I(x,y) - I(x,y - 2)|) Then calculate the magnitude |δ| of the gray difference, that is: If |δ| is less than 15, the texture of the feature point is considered insignificant and is ignored.

5. The method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization according to claim 1, characterized in that: The specific method of step S4 is as follows: S4.1: Initialize a random integer generator R and randomly generate a set of external parameter candidates within the specified search range according to the preset resolution Δr = 1 / 100: P i ={R i ,T i },i=1,2,...,N Where, P i is a randomly generated external parameter; R i is the Euler angle; T i is the translation vector; N is the random external parameter generated in this iteration; S4.2: For each set of randomly generated camera extrinsic parameters P i , calculate the corresponding texture point reprojection error E(P i ), the reprojection error is defined as: Where E(P) is the reprojection error; M is the total number of projection points; (x j ,y j ) is the coordinate of the texture point on the camera projection bird's-eye view coordinate system I1; (x' j ,y′ j ) is the coordinate of the camera projection bird's-eye view coordinate system I2; I1(x j ,y j ) and I2(x′ j ,y′ j ) are the grayscale values of I1 and I2 at the corresponding coordinates; S4.3: In each iteration, the reprojection error E(P i ) and the reprojection error E(P init ), select the external parameter P that minimizes the error i Denoted as P best , as follows: S4.4: After multiple iterations, record the external parameter P that minimizes the reprojection error in the current iteration. best , select P best As the optimization result of the coarse calibration stage.

6. The method for calibrating multi-camera extrinsic parameters of an engineering vehicle based on multiple constraints and adaptive optimization according to claim 1, characterized in that: The specific method of step S5 is as follows: S5.1: Initialize a random floating point real number generator R and randomly generate a candidate set of external parameters: P i ={R i ,T i },i=1,2,...,N Where, P i is a randomly generated external parameter; R i is the Euler angle; T i is the translation vector; N is the random external parameter generated in this iteration; S5.2: For each set of randomly generated external parameters P i , calculate the corresponding texture point reprojection error E(P i ), the reprojection error is defined as: Where E(P) is the reprojection error; M is the total number of projection points; (x j ,y j ) is the coordinate of the texture point on the camera projection bird's-eye view coordinate system I1; (x' j ,y′ j ) is the coordinate of the camera projection bird's-eye view coordinate system I2; I1(x j ,y j ) and I2(x′ j ,y' j ) are the grayscale values of I1 and I2 at the corresponding coordinates; S5.3: In each iteration, the reprojection error E(P i ) and the reprojection error E(P init ), select the external parameter P that minimizes the error i Denoted as P best , as follows: S5.4: After multiple iterations, record the external parameter P that minimizes the reprojection error in the current iteration. best , select P best As the optimization result of the fine calibration stage.

Images