Camera external parameter self-checking method and system for unmanned mining truck and unmanned mining truck

Abstract

The present invention proposes a camera extrinsic parameter self-checking method, system, and unmanned mining truck for an unmanned mining truck. The method comprises: using a camera on the mining truck to capture a road image, fitting the road boundary line in the road image, extracting several boundary sampling points on the boundary line, and obtaining the parameters of the road boundary line; obtaining boundary point cloud data corresponding to the road in a 3D point cloud map of the road, converting the data to a vehicle body coordinate system, fitting the boundary line of the boundary points in the vehicle body coordinate system, extracting several interior points as boundary interior points in the vehicle body coordinate system, projecting the boundary interior points into a pixel coordinate system, and obtaining boundary interior point projection points; fitting the boundary interior point projection points to the boundary line on which the boundary interior point projection points lie, and obtaining the parameters of the boundary line on which the boundary interior point projection points lie; and determining whether the boundary line on which the boundary interior point projection points lie is aligned with the road boundary line, thereby completing the camera extrinsic parameter self-checking process. This method can be executed in real time without delaying the normal operation of the mining truck.

Description

Technical Field

[0001] The present invention relates to the field of camera parameter self-testing, and in particular to a camera external parameter self-testing method and system for an unmanned mining truck, and the unmanned mining truck. Background Art

[0002] The camera system is an essential subsystem for unmanned mining trucks (hereinafter referred to as "mining trucks"). The forward-facing camera is connected to the vehicle's exterior canopy via a specific flange, a rigid connection. Bolts secure the connection, securing the camera's position relative to the vehicle. As specialized vehicles used in specialized scenarios, mining trucks often operate on bumpy mountain roads, where vehicle vibrations are unavoidable. Therefore, loosening of the bolts securing the forward-facing camera is inevitable. Once loosened, the camera's mounting position will change, directly impacting the safety and efficiency of engineering operations. Therefore, mining trucks must be inspected before use, specifically to determine whether the bolts securing the forward-facing camera are loose and whether the camera's pitch and heading angles have changed. Currently, this inspection is mostly performed manually. However, due to the limited number of skilled operators on-site, the large number of mining trucks, and the shift-based operation, daily routine inspections can be lax, directly impacting the efficiency and safety of engineering operations. Summary of the Invention

[0003] In order to overcome the defects in the above-mentioned prior art, the purpose of the present invention is to provide a camera external parameter self-checking method and system for unmanned mining trucks.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides a camera external parameter self-test method for an unmanned mining truck, comprising the following steps:

[0005] Make the mining truck run or stay on a road;

[0006] Using a camera on a mining truck to take a road picture, fitting a boundary line of the road on the road picture, extracting a number of boundary sampling points on the boundary line, obtaining the coordinates of each boundary sampling point in a pixel coordinate system, and obtaining the parameters of the boundary line of the road on the picture;

[0007] Obtain boundary point cloud data corresponding to the road in a 3D point cloud map of the road, convert the boundary point cloud of the road in the 3D point cloud map into a vehicle body coordinate system, perform boundary line fitting on the boundary points in the vehicle body coordinate system, extract several inner points on the fitted boundary line as boundary inner points in the vehicle body coordinate system, project the boundary inner points in the vehicle body coordinate system onto a pixel coordinate system in combination with the camera coordinate system to obtain boundary inner point projection points; fit the boundary inner point projection points to the boundary line where the boundary inner point projection points are located, and obtain parameters of the boundary line where the boundary inner point projection points are located;

[0008] It is determined whether the boundary line where the projection point of the boundary point is located is aligned with the boundary line of the road in the pixel coordinate system, that is, the camera external parameter self-checking process is completed.

[0009] This self-test method for camera extrinsic parameters in unmanned mining trucks does not require a specialized calibration workshop. Instead, it requires only a straight road with clear boundaries, which serves as a reference for determining whether camera extrinsic parameters have changed. This method uses a projected image of map data to perform judgments within the image data, allowing for real-time operation without disrupting the truck's normal operation.

[0010] Preferably, the road is a straight road or a curved road. When the road is a curved road, the absolute value of the curvature of the road is not greater than λ, and λ is a positive number.

[0011] A preferred solution of the camera extrinsic parameter self-checking method of the unmanned mining truck: when fitting the boundary line of the road on the road image, a second-order polynomial model is fitted to the pixel points on the road in the image to obtain the boundary line of the road on the road image, and an inlier point on the boundary line is obtained as a boundary sampling point;

[0012] When fitting the boundary lines of the boundary points in the vehicle body coordinate system, the boundary points in the vehicle body coordinate system are fitted with a second-order polynomial model in the XOY plane, and a number of inner points are extracted from the model according to a predetermined threshold as the boundary inner points in the vehicle body coordinate system;

[0013] When fitting the boundary line where the boundary inner point projection point is located, a second-order polynomial model is fitted to the boundary inner point projection point to obtain the parameters of the boundary line where the boundary inner point projection point is located.

[0014] This preferred solution is not only applicable to the fitting of straight road boundary lines, but also to the fitting of curved road boundary lines.

[0015] A preferred solution of the camera external parameter self-checking method of the unmanned mining truck is as follows: the boundary point cloud of the road in the 3D point cloud map is converted to the vehicle body coordinate system according to the following formula:

[0016] Among them, P L_car Represents the left boundary point of the road in the vehicle body coordinate system, P R_car Represents the right boundary point of the road in the vehicle body coordinate system, P L_map Represents the left boundary point of the road in the 3D point cloud map, P R_map Represents the right boundary point of the road in the 3D point cloud map, and It represents the rotation and translation matrix of the point cloud data in the 3D point cloud map toward the vehicle body coordinate system.

[0017] A preferred solution of the camera extrinsic parameter self-checking method of the unmanned mining truck is as follows: the steps of projecting the boundary inner point in the vehicle body coordinate system to the pixel coordinate system to obtain the projection point of the boundary inner point are as follows:

[0018] The left inner point P in the vehicle body coordinate system L_car and point P on the right boundary R_car Convert to the camera coordinate system, the conversion parameters are Represents the initialization conversion parameters from the body coordinate system to the camera coordinate system, Represents the translation parameter from the body coordinate system to the camera coordinate system, Represents the rotation parameter from the body coordinate system to the camera coordinate system;

[0019] Point P inside the left boundary of the vehicle body coordinate system L_car and point P on the right boundary R_car Projection point of the left boundary point projected to the pixel coordinate system C P L_car_inliner and the projection point inside the right boundary C P R_car_inliner The coordinate transformation formula is: K represents the camera intrinsic parameter.

[0020] described Expand to get:

[0021] Among them, (u, v) represents the coordinates of the point inside the left boundary or the right boundary projected into the pixel coordinate system, that is, the coordinates of the projection point of the left boundary point C P L_car_inliner Or the projection coordinates of the point inside the right boundary C P R_car_inliner , (X, Y, Z, 1) represents the homogeneous coordinates of the left or right boundary points in the vehicle body coordinate system, f x and f y Represents the scale factor of the camera in the u-axis and v-axis directions in the pixel coordinate system, C x 、C y Represents the zero point offset of the pixel coordinate system in the x and y directions of the camera coordinate system.

[0022] This preferred solution can quickly and accurately realize the projection of the boundary inner point in the vehicle body coordinate system to the pixel coordinate system.

[0023] A preferred solution of the camera extrinsic parameter self-checking method of the unmanned mining truck: the step of determining whether the projection point of the boundary point is aligned with the boundary sampling point in the pixel coordinate system is:

[0024] Calculate a loss function based on the coordinates of the boundary sampling point, the parameters of the boundary line, the coordinates of the projection point of the boundary point, and the parameters of the boundary line where the projection point of the boundary point is located; the loss function aims to calculate the loss value between the boundary line where the projection point of the boundary point is located and the boundary line of the road in the image;

[0025] When the value of the loss function is greater than the set threshold, the projection point of the boundary point is not aligned with the boundary sampling point in the pixel coordinate system, and the camera extrinsic parameters are abnormal;

[0026] When the value of the loss function is not greater than the set threshold, the projection point of the boundary point is aligned with the boundary sampling point in the pixel coordinate system, and the camera extrinsic parameters are normal.

[0027] A preferred solution of the camera external parameter self-checking method of the unmanned mining truck: the loss function is:

[0028] Among them, k1, k2, k3 are regularization parameters, N is the number of boundary sampling points, a left boundary sampling point and a right boundary sampling point are a pair, M is the number of boundary inner point projection points, a left boundary inner point projection point and a right boundary inner point projection point are a pair, c P L_i [1] represents the left boundary sampling point on the road image c P L _ i The value of the horizontal coordinate u, c P R_i [1] represents the right boundary sampling point on the road image c P R_i The value of the horizontal coordinate u, C P L_car_inliner_j [1] represents the projection point inside the left boundary C P L_car_inliner_j The value of the horizontal coordinate u, C P R_car_inliner_j [1] represents the projection point inside the right boundary C P R_car_inliner_j The value of the horizontal coordinate u; A L 、B L 、C L is the parameter of the left boundary line on the road image, A R 、B R 、C R is the parameter of the right boundary line on the road image, A L_map 、B L_map 、C L_map A is the parameter of the boundary line where the projection point of the left boundary point is located, R_map 、B R_map 、C R_mapis the parameter of the boundary line where the projection point of the point inside the right boundary is located. This preferred solution reduces the error and improves the judgment effect.

[0029] The present application also provides a camera external parameter self-test system for an unmanned mining truck, comprising a processor, a memory, and a camera mounted on the mining truck. The memory and camera are each communicatively connected to the processor. The camera captures an image of a straight road and sends the image to the processor. The memory is configured to store at least one executable instruction that causes the processor to perform operations corresponding to the above-described camera external parameter self-test method for an unmanned mining truck based on the image of the straight road. This camera external parameter self-test system for an unmanned mining truck possesses all the advantages of the above-described camera external parameter self-test method for an unmanned mining truck.

[0030] The present application also provides an unmanned mining truck, comprising a mining truck body, on which the camera external parameter self-checking system of the unmanned mining truck as described above is provided.

[0031] The beneficial effects of the present invention are as follows: the implementation of the present invention does not require a special calibration workshop, but only requires a straight road with clear boundaries, and the boundaries of the road are used as a reference to determine whether the camera extrinsic parameters have changed; the workload of manual detection of camera extrinsic parameters is reduced, and the working efficiency of mining trucks is improved.

[0032] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0034] Figure 1 This is a flowchart of the camera external parameter self-test method for unmanned mining trucks. DETAILED DESCRIPTION

[0035] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0036] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0037] Example 1

[0038] like Figure 1 As shown, the present application provides an embodiment of a camera external parameter self-checking method for an unmanned mining truck. In this embodiment, the required road is a straight road, which is preferably but not limited to 10-20 meters.

[0039] The specific steps of the camera external parameter self-test method of the unmanned mining truck are as follows:

[0040] Make the mining truck drive or stay on a straight road with clear boundaries.

[0041] Initialize the camera extrinsic parameters and use the camera on the mining truck to capture a road image. Then, fit the road boundary line in the image. Extract several boundary sampling points along the boundary line, obtain the coordinates of each boundary sampling point in the pixel coordinate system, and obtain the parameters of the road boundary line in the image. The road boundary line can be a clear lane line, a clear road boundary, or a boundary guardrail.

[0042] In this embodiment, the left and right boundary lines of the road on the road image are obtained by default fitting. The boundary line formula of the road is: Among them, A L 、B L 、C L is the parameter of the left boundary line of the road, A R 、B R 、C R is the parameter of the right boundary line of the road. The fitting method involved here can use the existing straight line fitting method. The extracted boundary sampling points are divided into left boundary sampling points c P L _ i and right boundary sampling point c P R _ i , where i=1...N, N is the number of boundary sampling point pairs, and a left boundary sampling point and a right boundary sampling point constitute a pair.

[0043] Obtain the boundary point cloud data corresponding to the road in the 3D point cloud map of the road. The 3D point cloud map here can be collected and established by the sensors of the mining truck or other map acquisition equipment. It is a prior art and will not be described here in detail. Convert the boundary point cloud of the road in the 3D point cloud map to the vehicle body coordinate system, and then perform boundary line fitting on the boundary points of the road in the vehicle body coordinate system. The fitting method involved here can adopt the existing straight line fitting method, and extract several points on the fitted boundary line as the boundary inner points in the vehicle body coordinate system. Combined with the camera coordinate system, the boundary inner points in the vehicle body coordinate system are projected to the pixel coordinate system to obtain the boundary inner point projection points. Fit the boundary inner point projection points to the boundary line where the boundary inner point projection points are located, and obtain the parameters of the boundary line where the boundary inner point projection points are located. The vehicle body coordinate system and camera coordinate system here have been determined when the unmanned driving system determines the requirements. The vehicle body coordinate system and camera coordinate system remain fixed in the development and operation links.

[0044] In this embodiment, the above operations are performed on the left and right boundary points of the road in the 3D point cloud map by default.

[0045] Specifically, in this embodiment, the boundary point cloud of the road in the 3D point cloud map is converted to the vehicle body coordinate system according to the following formula:

[0046] Among them, P L_car Represents the left boundary point of the road in the vehicle body coordinate system, P R_car Represents the right boundary point of the road in the vehicle body coordinate system, P L_map Represents the left boundary point of the road in the 3D point cloud map, P R_map Represents the right boundary point of the road in the 3D point cloud map, P L_map and P R_map Both can be directly obtained from the 3D point cloud map of the road and the road boundary point cloud data. and It represents the rotation and translation matrix of the point cloud data in the 3D point cloud map toward the vehicle body coordinate system, which is a known quantity.

[0047] Then the left boundary point P of the road in the vehicle body coordinate system is L_car and the right boundary point P of the road in the vehicle body coordinate system R_car Perform fitting, with the fitting threshold preferably but not limited to 2-3 cm, to obtain the left and right boundary lines of the fitted road, and extract several points on the left and right boundary lines as the left boundary inner points in the vehicle body coordinate system. L P L_car_inlier_j and the inner point on the right boundary L P R_car_inliner_j , where j = 1…M, M is the number of pairs of points in the boundary, and a left boundary point and a right boundary point constitute a pair.

[0048] In this embodiment, the body coordinate system adopts the imu coordinate system, that is, the front left sky coordinate system is adopted. Therefore, according to the following steps, the boundary inner point under the body coordinate system is projected to the pixel coordinate system to obtain the boundary inner point projection point:

[0049] The left inner point P in the vehicle body coordinate system L_car and point P on the right boundary R_car Convert to the camera coordinate system, the conversion parameters are Represents the initialization conversion parameters from the body coordinate system to the camera coordinate system, Represents the translation parameter from the body coordinate system to the camera coordinate system, Represents the rotation parameter from the body coordinate system to the camera coordinate system; and All are known initial values.

[0050] Point P inside the left boundary of the vehicle body coordinate system L_car and point P on the right boundary R_car Projection point of the left boundary point projected to the pixel coordinate system C P L_car_inliner and the projection point inside the right boundary C P R_car_inliner The coordinate transformation formula is: K represents the camera intrinsic parameter.

[0051] Will Expand to get:

[0052] Among them, (u, v) represents the coordinates of the point inside the left boundary or the right boundary projected into the pixel coordinate system, that is, the coordinates of the projection point of the left boundary point C P L_car_inliner Or the projection coordinates of the point inside the right boundary C P R_car_inliner , (X, Y, Z, 1) represents the homogeneous coordinates of the left or right boundary points in the vehicle body coordinate system, f x and f y Represents the scale factor of the camera in the u-axis and v-axis directions in the pixel coordinate system, which is a known initial value, C x 、C y Represents the zero point offset of the pixel coordinate system in the x and y directions of the camera coordinate system. The zero point offset refers to the difference between the zero points of (u, v) and (x, y), that is, the difference between the origin coordinates of the pixel coordinate system and the camera coordinate system, which is a known initial value.

[0053] Fit the projection point of the inner boundary point to the boundary line where the projection point of the inner boundary point is located. Here, the existing straight line fitting method can be used to obtain the parameters of the boundary line where the projection point of the inner boundary point is located: the parameter A of the boundary line where the projection point of the left boundary point is located L_map 、B L_map 、C L_map and the parameter A of the boundary line where the projection point of the right boundary point is located R_map 、B R_map 、C R_map .

[0054] Finally, it is determined whether the projection point of the boundary point is aligned with the boundary sampling point in the pixel coordinate system. If aligned, the camera extrinsic parameters are normal; if not, the camera extrinsic parameters are abnormal.

[0055] In this embodiment, the step of determining whether the projection point of the boundary inner point is aligned with the boundary sampling point of the road boundary line under the camera standard external parameter in the pixel coordinate system is:

[0056] The loss function is calculated based on the coordinates of the boundary sampling points, the parameters of the boundary line, the coordinates of the projection points of the boundary points, and the parameters of the boundary line where the projection points of the boundary points are located.

[0057] The loss function calculates the loss between the boundary line where the projection of the point inside the boundary lies and the boundary line of the road in the image. If the loss function value is greater than the set threshold, the camera extrinsic parameters are abnormal; if the loss function value is less than the set threshold, the camera extrinsic parameters are normal.

[0058] The loss function in this embodiment is specifically:

[0059] Among them, k1, k2, k3 are regularization parameters, N is the number of boundary sampling points, a left boundary sampling point and a right boundary sampling point are a pair, M is the number of boundary inner point projection points, a left boundary inner point projection point and a right boundary inner point projection point are a pair, c P L_i [1] represents the left boundary sampling point c P L _ i The value of the horizontal coordinate u, c P R_i [1] indicates the right boundary sampling point c P R_i The value of the horizontal coordinate u, C P L_car_inliner_j [1] represents the projection point inside the left boundary C P L_car_inliner_j The value of the horizontal coordinate u, C P R_car_inliner_j [1] represents the projection point inside the right boundary C P R_car_inliner_jThe value of the horizontal coordinate u; A L 、B L 、C L is the parameter of the left boundary line, A R 、B R 、C R is the parameter of the right boundary line, A L_map 、B L_map 、C L_map A is the parameter of the boundary line where the projection point of the left boundary point is located, R_map 、B R_map 、C R_map The parameter of the boundary line where the projection point of the point inside the right boundary lies.

[0060] When RegularizedCostFunction is greater than a set threshold, the camera extrinsic parameters are abnormal. When RegularizedCostFunction is not greater than the set threshold, the camera extrinsic parameters are normal. In this embodiment, the set threshold is preferably, but not limited to, 3.

[0061] Example 2

[0062] like Figure 1 As shown, the present application provides another embodiment of a camera extrinsic parameter self-checking method for an unmanned mining truck. In this embodiment, the required road is a curved road, which is preferably but not limited to 10-20 meters. The absolute value of the curvature of the road is not greater than λ, where λ is a positive number. In this embodiment, λ is preferably but not limited to 0.02, that is, the road turning radius is not greater than 50 meters.

[0063] The specific steps of the camera external parameter self-test method of the unmanned mining truck are as follows:

[0064] Make the mining truck run or stay on the above-mentioned curved road;

[0065] Initialize the camera extrinsic parameters, make the camera on the mining truck take a road picture, fit the boundary line of the road in the road picture, extract several boundary sampling points on the boundary line, obtain the coordinates of each boundary sampling point in the pixel coordinate system, and obtain the parameters of the boundary line of the road in the picture.

[0066] Here, it is preferred but not limited to using the RANSAC algorithm to fit the second-order polynomial model. The default fitting method is to obtain the left and right boundary lines of the road on the road image. Among them, A L 、B L 、C L is the parameter of the left boundary line of the road, A R 、B R 、C R is the parameter of the right boundary line of the road, and the extracted boundary sampling points are divided into left boundary sampling pointsc P L _ i and right boundary sampling point c P R _ i , where i=1...N, N is the number of boundary sampling point pairs, and a left boundary sampling point and a right boundary sampling point constitute a pair.

[0067] Obtain the boundary point cloud data corresponding to the road in the 3D point cloud map of the road, convert the boundary point cloud of the road in the 3D point cloud map to the vehicle body coordinate system, fit the boundary lines of the boundary points in the vehicle body coordinate system, and extract several inner points on the fitted boundary lines as the boundary inner points in the vehicle body coordinate system. Combined with the camera coordinate system, the boundary inner points in the vehicle body coordinate system are projected into the pixel coordinate system to obtain the boundary inner point projection points. The boundary inner point projection points are fitted to the boundary line where the boundary inner point projection points lie, and the parameters of the boundary line where the boundary inner point projection points lie are obtained. The vehicle body coordinate system and camera coordinate system here are determined when the requirements of the unmanned driving system are determined. The vehicle body coordinate system and camera coordinate system remain fixed during development and operation.

[0068] In this embodiment, the above operation is performed on the left and right boundary points of the road in the 3D point cloud map by default. The specific steps are basically the same as those in the first embodiment, with the following differences:

[0069] The left boundary point P of the road in the vehicle body coordinate system L_car and the right boundary point P of the road in the vehicle body coordinate system R_car When fitting the boundary line, the left boundary point P of the road in the vehicle body coordinate system is L_car and the right boundary point P of the road in the vehicle body coordinate system R_car Perform curve fitting respectively. In this embodiment, it is preferred but not limited to perform XOY plane second-order polynomial model fitting, and extract several inner points on the model according to a predetermined threshold as the left boundary inner points in the vehicle body coordinate system. L P L_car_inlier_j and the inner point on the right boundary L P R_car_inliner_j , where j = 1…M, M is the number of pairs of points inside the boundary, and a left boundary point and a right boundary point constitute a pair.

[0070] When fitting the projection point of the inner point to the boundary line where the projection point of the inner point is located and obtaining the parameters of the boundary line where the projection point of the inner point is located, a curve fitting method needs to be adopted. In this embodiment, it is preferred but not limited to fitting the projection point of the inner point to a second-order polynomial model to obtain the model parameters of the projection point of the inner point.

[0071] Then, the same steps as those in Example 1 are used to determine whether the boundary line where the projection point of the boundary point is located is aligned with the boundary line of the road in the pixel coordinate system, thereby completing the camera extrinsic parameter self-checking process.

[0072] Example 3

[0073] The present application also provides an embodiment of a camera external parameter self-test system for an unmanned mining truck. In this embodiment, the camera external parameter self-test system for an unmanned mining truck includes a processor, a memory, and a camera disposed on the mining truck. The memory and the camera are respectively in communication with the processor. The camera captures an image of a straight road and sends the image of the straight road to the processor. The memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the above-mentioned unmanned mining truck camera external parameter self-test method based on the straight road image.

[0074] Example 4

[0075] The present application also provides an unmanned mining truck, which includes a mining truck body, on which is provided a camera external parameter self-checking system of the unmanned mining truck as described above, and the unmanned mining truck can realize self-checking of the camera external parameters.

[0076] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0077] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A camera external parameter self-checking method for an unmanned mining truck, characterized in that: The following steps are involved: Make the mining truck run or stay on a road; Using a camera on a mining truck to take a road picture, fitting a boundary line of the road on the road picture, extracting a number of boundary sampling points on the boundary line, obtaining the coordinates of each boundary sampling point in a pixel coordinate system, and obtaining the parameters of the boundary line of the road on the picture; Obtain boundary point cloud data corresponding to the road in a 3D point cloud map of the road, convert the boundary point cloud of the road in the 3D point cloud map into a vehicle body coordinate system, perform boundary line fitting on the boundary points in the vehicle body coordinate system, extract several inner points on the fitted boundary line as boundary inner points in the vehicle body coordinate system, project the boundary inner points in the vehicle body coordinate system onto a pixel coordinate system in combination with the camera coordinate system to obtain boundary inner point projection points; fit the boundary inner point projection points to the boundary line where the boundary inner point projection points are located, and obtain parameters of the boundary line where the boundary inner point projection points are located; It is determined whether the boundary line where the projection point of the boundary point is located is aligned with the boundary line of the road in the pixel coordinate system, that is, the camera external parameter self-checking process is completed.

2. The camera external parameter self-test method for an unmanned mining truck according to claim 1 is characterized in that: The road is a straight road or a curved road. When the road is a curved road, the absolute value of the curvature of the road is not greater than λ, and λ is a positive number.

3. The camera external parameter self-checking method for an unmanned mining truck according to claim 1 is characterized in that: When fitting the boundary line of the road on the road image, a second-order polynomial model is fitted to the pixel points on the road in the image to obtain the boundary line of the road on the road image, and the inliers on the boundary line are obtained as boundary sampling points; When fitting the boundary lines of the boundary points in the vehicle body coordinate system, the boundary points in the vehicle body coordinate system are fitted with a second-order polynomial model in the XOY plane, and a number of inner points are extracted from the model according to a predetermined threshold as the boundary inner points in the vehicle body coordinate system; When fitting the boundary line where the boundary inner point projection point is located, a second-order polynomial model is fitted to the boundary inner point projection point to obtain the parameters of the boundary line where the boundary inner point projection point is located.

4. The camera external parameter self-test method for an unmanned mining truck according to claim 1, characterized in that: The boundary point cloud of the road in the 3D point cloud map is converted to the vehicle body coordinate system according to the following formula: Among them, P L_car Represents the left boundary point of the road in the vehicle body coordinate system, P R_car Represents the right boundary point of the road in the vehicle body coordinate system, P L_map Represents the left boundary point of the road in the 3D point cloud map, P R_map Represents the right boundary point of the road in the 3D point cloud map, and It represents the rotation and translation matrix of the point cloud data in the 3D point cloud map toward the vehicle body coordinate system.

5. The camera external parameter self-checking method for an unmanned mining truck according to claim 1 is characterized in that: The steps to project the inner boundary point in the vehicle body coordinate system to the pixel coordinate system to obtain the projection point of the inner boundary point are as follows: The left boundary point P in the vehicle body coordinate system L_car and point P on the right boundary R_car Convert to the camera coordinate system, the conversion parameters are Represents the initialization conversion parameters from the body coordinate system to the camera coordinate system, Represents the translation parameter from the body coordinate system to the camera coordinate system, Represents the rotation parameter from the body coordinate system to the camera coordinate system; Point P inside the left boundary of the vehicle body coordinate system L_car and point P on the right boundary R_car Projection point of the left boundary point projected to the pixel coordinate system C P L_car_inliner and the projection point inside the right boundary C P R_car_inliner The coordinate transformation formula is: K represents the camera intrinsic parameter.

6. The camera external parameter self-checking method for an unmanned mining truck according to claim 5 is characterized in that: described Expand to get: Among them, (u, v) represents the coordinates of the point inside the left boundary or the right boundary projected into the pixel coordinate system, that is, the coordinates of the projection point of the left boundary point C P L_car_inliner Or the projection coordinates of the point inside the right boundary C P R_car_inliner , (X, Y, Z, 1) represents the homogeneous coordinates of the left or right boundary points in the vehicle body coordinate system, f x and f y Represents the scale factor of the camera in the u-axis and v-axis directions in the pixel coordinate system, C x 、C y Represents the zero point offset of the pixel coordinate system in the x and y directions of the camera coordinate system.

7. The camera external parameter self-checking method for an unmanned mining truck according to claim 1 is characterized in that: The steps of determining whether the projection point of the boundary point is aligned with the boundary sampling point in the pixel coordinate system are: Calculate a loss function based on the coordinates of the boundary sampling point, the parameters of the boundary line, the coordinates of the projection point of the boundary point, and the parameters of the boundary line where the projection point of the boundary point is located; the loss function aims to calculate the loss value between the boundary line where the projection point of the boundary point is located and the boundary line of the road in the image; When the value of the loss function is greater than the set threshold, the projection point of the boundary point is not aligned with the boundary sampling point in the pixel coordinate system, and the camera extrinsic parameters are abnormal; When the value of the loss function is not greater than the set threshold, the projection point of the boundary point is aligned with the boundary sampling point in the pixel coordinate system, and the camera extrinsic parameters are normal.

8. The camera external parameter self-checking method for an unmanned mining truck according to claim 7 is characterized in that: The loss function is: Among them, k1, k2, k3 are regularization parameters, N is the number of boundary sampling points, a left boundary sampling point and a right boundary sampling point are a pair, M is the number of boundary inner point projection points, a left boundary inner point projection point and a right boundary inner point projection point are a pair, c P L_i [1] represents the left boundary sampling point on the road image c P L _ i The value of the horizontal coordinate u, c P R_i [1] represents the right boundary sampling point on the road image c P R_i The value of the horizontal coordinate u, C P L_car_inliner_j [1] represents the projection point inside the left boundary C P L_car_inliner_j The value of the horizontal coordinate u, C P R_car_inliner_j [1] represents the projection point inside the right boundary C P R_car_inliner_j The value of the horizontal coordinate u; A L 、B L 、C L is the parameter of the left boundary line on the road image, A R 、B R 、C R is the parameter of the right boundary line on the road image, A L_map 、B L_map 、C L_map A is the parameter of the boundary line where the projection point of the left boundary point is located, R_map 、B R_map 、C R_map The parameter of the boundary line where the projection point of the point inside the right boundary lies.

9. A camera external parameter self-checking system for an unmanned mining truck, characterized in that: The invention comprises a processor, a memory, and a camera provided on a mining truck, wherein the memory and the camera are respectively connected to the processor for communication, the camera takes a road picture and sends the road picture to the processor, the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the camera external parameter self-test method of the unmanned mining truck according to any one of claims 1 to 8 according to the road picture.

10. An unmanned mining truck, characterized in that: It comprises a mining truck body, on which the camera external parameter self-checking system for the unmanned mining truck as claimed in claim 9 is provided.

Images