A ranging method, device, system and medium based on camera and radar fusion

Abstract

The application discloses a ranging method, device and system based on camera and radar fusion and a medium. In order to solve the problem that when a shovel remotely controls operation through a remote control console outside the visual range, the remote control end often does not have a good operation visual angle, the perception of the excavated material depth information is not intuitive, the position perception of the excavated material area is not intuitive, and the shovel cannot be accurately lowered, a method for fusing a front-view camera of a shovel and laser radar point clouds is provided. Through the new radar and visual fusion technology, the method can realize video monitoring of the material area in front of the shovel, and video pictures are superimposed to display point clouds of the excavated material, target material point clouds and operation information.

Description

Technical Field

[0001] This invention belongs to the field of engineering technology, specifically relating to a ranging method, device, system, and medium based on camera and radar fusion. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent driving is gradually becoming a development trend in the automotive industry. Simultaneously, heavy industries such as excavators are also slowly moving towards intelligentization. The field of intelligent driving mainly includes three technologies: perception, decision-making, and planning. Perception is the key component, aiming to identify the environment surrounding the vehicle. Using a single sensor on an excavator in complex environments is difficult to meet current project requirements. Furthermore, it is limited by the inherent characteristics of the sensors themselves. A single camera is limited in its inability to accurately obtain target position information, and point cloud radar is limited in its inability to obtain target texture information. Therefore, relying on a single camera or radar cannot adequately meet the environmental perception and accurate ranging capabilities required for excavators. To address the problems of remote control terminals often lacking a good working view when operating excavators beyond visual range, resulting in unclear perception of the depth and location of excavated materials and thus inaccurate digging, further research is needed. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention provides a ranging method, device, system, and medium based on camera and radar fusion. This method combines radar-visual fusion and target segmentation technologies, enabling video monitoring of the material area in front of the excavator and obtaining target material location and distance information.

[0004] To achieve the above objectives / to solve the above technical problems, the present invention is implemented using the following technical solution:

[0005] On one hand, the present invention provides a ranging method based on camera and radar fusion, comprising:

[0006] Point cloud data and image data of the material area to be excavated are acquired by radar and cameras, respectively.

[0007] The image data of the material area to be excavated is input into the pre-trained target segmentation model, which outputs image regions containing categories and filters out the image regions of the target material.

[0008] The point cloud data of the material area to be excavated is input into the pre-calibrated excavator radar-visual fusion imaging model to obtain the image data corresponding to the point cloud data of the material area to be excavated. The point cloud data and the corresponding image data are combined with the segmentation detection box of the image region where the target material is located in the target segmentation model to obtain point cloud data containing only the target material. Finally, the distance information of the target material is extracted based on the point cloud data of the target material.

[0009] Furthermore, the target segmentation model is the UNET model.

[0010] The output includes image regions containing categories, and image regions for target materials are filtered out, including:

[0011] Image data of the material area to be excavated is input into the UNET model network for feature learning and extraction. The Softmax function is used to convert the output vector into a probability distribution, and the category with the highest probability is selected as the target region A. ij , where i is the image region index and j represents the target category.

[0012] Furthermore, the calibration method for the pre-calibrated excavator radar-visual fusion imaging model includes: primary calibration and secondary calibration;

[0013] The first calibration includes:

[0014] Point cloud data and image data of the calibration board are acquired using radar and cameras, respectively.

[0015] The radar and camera are initially calibrated in different poses using a calibration board to obtain the rotation and translation matrices from the radar to the camera.

[0016] Based on the camera intrinsic parameter matrix and the rotation and translation matrices of the radar to the camera, the point cloud data of the calibration board is projected into the image data to obtain the two-dimensional point cloud data of the calibration board;

[0017] The secondary calibration includes:

[0018] Based on the corresponding positions of the calibration board in the two-dimensional point cloud data and image data, a correction matrix is ​​generated through affine transformation to correct the radar data mapped to the camera.

[0019] Furthermore, the radar and camera are calibrated at different poses using a calibration board to obtain the rotation and translation matrices from the radar to the camera.

[0020] Based on the camera intrinsic parameter matrix and the rotation and translation matrices of the radar to the camera, the point cloud data of the calibration board is projected onto the image data to obtain the two-dimensional point cloud data of the calibration board, including:

[0021] The system acquires point cloud data obtained by the radar scanning the calibration board, and image data obtained by the camera capturing the calibration board; it obtains the feature points corresponding to the calibration board in the radar and camera under multiple sets of angles and ranges, and finally solves the transformation matrix from the radar coordinate system to the camera coordinate system.

[0022] The camera intrinsic parameter matrix was obtained using Zhang Zhengyou's chessboard calibration method.

[0023] Z C [ u v 1 ] = [ f x 0 u 0 0 f y v 0 0 0 1 ] [ X C Y C Z C ] = K [ X C Y C Z C ]

[0024] Among them, f x and f y For the camera's equivalent focal length,

[0025] u0 is the pixel value of the horizontal translation between the center point of the image coordinate system and the pixel coordinate system;

[0026] v0 is the pixel value of the vertical translation between the center point of the image coordinate system and the pixel coordinate system;

[0027] X C Y C Z C is the camera coordinate system; u and v are the pixel coordinate systems, and K is the camera intrinsic parameter matrix;

[0028] The rectangular shape of the calibration board is fitted from the point cloud data of the calibration board using a fitting method, and the four three-dimensional points corresponding to the calibration board are obtained.

[0029] Based on the position of the calibration board, the pixel coordinates of the four two-dimensional points on the calibration board are obtained from the image data of the calibration board.

[0030] By acquiring multiple sets of corresponding data from two-dimensional points to three-dimensional points, the transformation matrix from the radar coordinate system to the camera coordinate system can be solved:

[0031] The formula for the radar-camera transformation matrix is: Z C [ u v 1 ] = K [ R LC T LC 0 1 ] [ X L Y L Z L 1 ]

[0032] Among them, (X) L ,Y L Z L () represents the radar coordinate system, (u,v) represents the pixel coordinate system; Z C Let K be the camera coordinate system, K be the camera intrinsic parameter matrix, and R be the camera coordinate system. LC For the rotation matrix of the radar to the camera, T LC This is the translation matrix from the radar to the camera.

[0033] Furthermore, the step of generating a correction matrix through affine transformation based on the corresponding positions of the calibration board in the two-dimensional point cloud data and image data includes:

[0034] Given the point cloud data of a 2D calibration board, the coordinates of the four points of the fitted quadrilateral are (x1, y1), (x2, y2), (x3, y3), and (x4, y4). The coordinates of the four points of the calibration board in the image data are (x... ’ 1,y ’ 1), (x) ’ 2,y ’ 2), (x) ’ 3,y’ 3), (x) ’ 4,y ’ 4) Substitute the coordinates into the following correction formula to generate the correction matrix:

[0035]

[0036] Where (x, y) represents the coordinates of the calibration board in the point cloud data, (x... ’ ,y ’ M represents the coordinates of the calibration plate in the image data. a Let represent the correction matrix, 'a' represent the scaling parameter, 'd' represent the rotation parameter, and 'b' and 'c' represent the shearing parameters. x and t y It is represented as a translation parameter.

[0037] Furthermore, after acquiring the distance information of the target material, the coordinate system of the radar location of the target material is transformed to the coordinate system of the excavator location, including:

[0038] Establish the excavator coordinate system with the operator inside the excavator cab as the origin, and connect the radar coordinate system (X... L ,Y L Z L Transform to the excavator coordinate system (x) b y b , z b );

[0039] [ x b y b z b ] = R [ X L Y L Z L ] + T

[0040] Where: R is the rotation matrix of the radar to the excavator, and T is the translation matrix of the radar to the excavator;

[0041] T = [ x t y t z t ]

[0042] Where: x t y t z t It is the difference in three dimensions between the excavator's coordinate origin and the radar's installation position;

[0043] The data acquired by the radar are sequentially rotated around the X coordinate system of the radar. L Y L Z L After rotating the axis, we get R:

[0044]

[0045] in:

[0046] R yaw = [ cos i − sin i 0 sin i cos i 0 0 0 1 ]

[0047] R pitch = [ cos β 0 sin β 0 1 0 − sin β 0 cos β ]

[0048] R roll = [ 1 0 0 0 cos c − sin c 0 sin c cos c ]

[0049] Wherein: rotation angle It is the heading angle (yaw) in Euler angles;

[0050] This is the rotation matrix for rotating about the heading angle yaw;

[0051] Rotation angle It is the pitch angle (pictch) in Euler angles;

[0052] This is the rotation matrix for rotating about the pitch angle pictch;

[0053] Rotation angle It is the roll angle in Euler angles;

[0054] This is the rotation matrix for rotating around the roll angle.

[0055] Furthermore, the distance information for extracting the target material includes:

[0056] Point cloud data of the target material is transmitted to target area A. ij Mapping is performed to generate multidimensional data containing image RGB information and point cloud XYZ three-dimensional coordinate information. The excavator coordinate system (x...) is selected. b y b z b The minimum or average value in the multidimensional data is used as the final distance information from the target material to the excavator;

[0057] [ x b y b z b ] = R [ X L Y L Z L ] + T

[0058]

[0059] Where, x dis y dis z dis Indicates the target material to the excavator (x) b y b z b The final distance information in dimension x; p y p z p This indicates a transformation to the excavator coordinate system (x). b y b z bInformation about points in dimension ), min means finding the minimum value, and avg means finding the average value.

[0060] Secondly, the present invention provides a ranging device based on camera and radar fusion, comprising:

[0061] Acquisition module: used to acquire point cloud data and image data of the material area to be excavated through radar and camera, respectively;

[0062] The filtering module is used to input the image data of the material area to be excavated into the pre-trained target segmentation model, output the image regions containing the categories, and filter out the image regions of the target materials.

[0063] Extraction module: This module is used to input the point cloud data of the material area to be excavated into the pre-calibrated excavator radar-visual fusion imaging model to obtain the image data corresponding to the point cloud data of the material area to be excavated. The point cloud data and the corresponding image data are combined with the segmentation detection box of the image region where the target material is located in the target segmentation model to obtain point cloud data containing only the target material. Finally, the distance information of the target material is extracted based on the point cloud data of the target material.

[0064] Thirdly, the present invention provides a ranging system based on camera and radar fusion, characterized in that,

[0065] One or more processors;

[0066] Memory, used to store one or more programs.

[0067] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described ranging method based on camera and radar fusion.

[0068] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the above-described ranging method based on camera and radar fusion.

[0069] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This embodiment first performs preliminary calibration based on the various poses of the calibration board acquired by the radar and camera. By comparing the two-dimensional calibration board data mapped to the image with the calibration board data in the image, the rotation matrix and translation matrix are further adjusted. The rotation matrix and translation matrix of the radar to the camera obtained by the two calibration methods are more accurate, and the fusion of laser and radar on the excavator can be realized more accurately.

[0070] The method of this invention adopts a multi-sensor fusion approach, which effectively avoids the poor measurement stability of a single sensor. By combining image segmentation algorithm and Ray vision fusion technology, the distance information of the target material of the excavator can be accurately obtained. Attached Figure Description

[0071] Figure 1 This is a flowchart of the calibration method provided in the embodiments of the present invention;

[0072] Figure 2 This illustrates the installation positions of the lidar and camera on the excavator in an embodiment of the present invention.

[0073] Figure 3 This is a diagram of a non-contact ranging model for an excavator according to an embodiment of the present invention. Detailed Implementation

[0074] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0075] Example 1:

[0076] like Figure 1~3 One embodiment of the above describes a ranging method based on camera and radar fusion, comprising:

[0077] Point cloud data and image data of the material area to be excavated are acquired by radar and cameras, respectively.

[0078] The image data of the material area to be excavated is input into the pre-trained target segmentation model, which outputs image regions containing categories and filters out the image regions of the target material.

[0079] The point cloud data of the material area to be excavated is input into the pre-calibrated excavator radar-visual fusion imaging model to obtain the image data corresponding to the point cloud data of the material area to be excavated. The point cloud data and the corresponding image data are combined with the segmentation detection box of the image region where the target material is located in the target segmentation model to obtain point cloud data containing only the target material. Finally, the distance information of the target material is extracted based on the point cloud data of the target material.

[0080] The target segmentation model is the UNET model.

[0081] The output includes image regions containing categories, and image regions for target materials are filtered out, including:

[0082] Image data A of the material area to be excavated is input into the UNET model network for feature learning and extraction. The Softmax function is used to convert the output vector into a probability distribution, and the category with the highest probability is selected as the target region A. ij , where i is the image region index and j represents the target category.

[0083] The calibration method for the pre-calibrated excavator radar-visual fusion imaging model includes: primary calibration and secondary calibration;

[0084] The first calibration includes:

[0085] like Figure 2 As shown, a lidar and a camera are mounted on the top of an excavator. The lidar and camera acquire point cloud data and image data of the calibration board, respectively. The calibration board is placed in a position that can be detected by both the camera and the lidar, and the surrounding environment is kept open and free of obstacles. The point cloud data obtained by the lidar scanning the calibration board and the image obtained by the camera in the same pose are used to obtain the point cloud data and image data of the calibration board in the same pose.

[0086] Preliminary calibration of the radar and camera at different poses is performed using a calibration board. Multiple sets of corresponding 2D point-to-3D point data are obtained at different positions and with varying rotation angles of the calibration board. For example, data are obtained at positions to the left and right of the camera and radar, and at positions closer and farther from the camera and radar, with the calibration board rotated by 30, 60, and 90 degrees respectively. These sets of corresponding 3D and 2D points are then paired to solve for the rotation and translation matrices from the radar to the camera.

[0087] Based on the camera intrinsic parameter matrix and the rotation and translation matrices of the radar to the camera, the point cloud data of the calibration board is projected into the image data to obtain the two-dimensional point cloud data of the calibration board;

[0088] The secondary calibration includes:

[0089] Based on the corresponding positions of the calibration board in the two-dimensional point cloud data and image data, a correction matrix is ​​generated through affine transformation to correct the radar data mapped to the camera.

[0090] Furthermore, the radar and camera are calibrated at different poses using a calibration board to obtain the rotation and translation matrices from the radar to the camera, including:

[0091] The system acquires point cloud data obtained by the radar scanning the calibration board, and image data obtained by the camera capturing the calibration board; it obtains the feature points corresponding to the calibration board in the radar and camera under multiple sets of angles and ranges, and finally solves the transformation matrix from the radar coordinate system to the camera coordinate system.

[0092] The camera intrinsic parameter matrix was obtained using Zhang Zhengyou's chessboard calibration method.

[0093] Z C [ u v 1 ] = [ f x 0 u 0 0 f y v 0 0 0 1 ] [ X C Y C Z C ] = K [ X C Y C Z C ]

[0094] Among them, f x and f yFor the camera's equivalent focal length,

[0095] u0 is the pixel value of the horizontal translation between the center point of the image coordinate system and the pixel coordinate system;

[0096] v0 is the pixel value of the vertical translation between the center point of the image coordinate system and the pixel coordinate system;

[0097] X C Y C Z C is the camera coordinate system; u and v are the pixel coordinate systems, and K is the camera intrinsic parameter matrix;

[0098] In the point cloud data of the calibration board, the rectangular shape of the calibration board is fitted by fitting method, and four three-dimensional points corresponding to the calibration board are obtained. Each point is composed of three-dimensional coordinates (x, y, z).

[0099] In the image data of the calibration board, the pixel coordinates of four two-dimensional points of the calibration board are obtained according to the position of the calibration board. Each point is composed of (x, y) two dimensions.

[0100] By acquiring multiple sets of corresponding 2D point-to-3D point data, such as data obtained from the left and right sides of the camera and radar, and from positions close to and far from the camera and radar, with the calibration plate rotated by 30, 60, and 90 degrees respectively, and pairing these multiple sets of corresponding 3D and 2D points, the rotation and translation matrices from the radar to the camera are solved. Finally, the transformation matrix from the radar coordinate system to the camera coordinate system is solved.

[0101] The formula for the radar-camera transformation matrix is: Z C [ u v 1 ] = K [ R LC T LC 0 1 ] [ X L Y L Z L 1 ]

[0102] Among them, (X) L ,Y L Z L () represents the radar coordinate system, (u,v) represents the pixel coordinate system; Z C Let K be the camera coordinate system, K be the camera intrinsic parameter matrix, and R be the camera coordinate system. LC For the rotation matrix of the radar to the camera, T LC This is the translation matrix from the radar to the camera.

[0103] The step of generating a correction matrix through affine transformation based on the corresponding positions of the calibration board in the two-dimensional point cloud data and image data of the calibration board includes:

[0104] Based on the camera intrinsic parameter matrix and the rotation and translation matrices obtained from the first calibration of the radar to the camera, the radar calibration board data is projected onto the image coordinate system to obtain two-dimensional calibration board point cloud data. Using the coordinates of four points on the rectangular outline of the image calibration board and the coordinates of four points on the rectangular outline fitted from the two-dimensional calibration board point cloud data, a correction matrix M is obtained through affine transformation.a ;

[0105] Assume the four points of the rectangle fitted from the calibration board point cloud data are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the four points of the calibration board in the image are (x1, y1), (x2, y2), (x3, y3), and (x4, y4). ’ 1,y ’ 1), (x) ’ 2,y ’ 2), (x) ’ 3,y ’ 3), (x) ’ 4,y ’ 4) The corrected formula is as follows:

[0106]

[0107] Where (x, y) represents the coordinates of the calibration board in the point cloud data, (x... ’ ,y ’ M represents the coordinates of the calibration plate in the image data. a Let represent the correction matrix, 'a' represent the scaling parameter, 'd' represent the rotation parameter, and 'b' and 'c' represent the shearing parameters. x and t y Represented as translation parameters;

[0108] Through M a Correct the error between the position of the two-dimensional calibration board point cloud data and the position of the calibration board in the image, so that they completely coincide with the calibration board in the image.

[0109] Based on the target material region information A obtained from the image by the UET target segmentation network ij Only the point cloud data of the target material is sent to the target area A. ij Mapping is performed to generate multidimensional data containing image RGB information and point cloud XYZ three-dimensional coordinate information. The excavator coordinate system (x...) is selected. b y b z b The minimum or average value in the multidimensional data is used as the final distance information from the target material to the excavator;

[0110] [ x b y b z b ] = R [ X L Y L Z L ] + T

[0111]

[0112] Among them, x dis y dis z dis Indicates the target material to the excavator (x) b y b z b The final distance information in dimension x;p y p z p This indicates a transformation to the excavator coordinate system (x). b y b z b Information about points in dimension ), min means finding the minimum value, and avg means finding the average value.

[0113] After obtaining the distance information of the target material, the coordinate system of the radar location of the target material is transformed to the coordinate system of the excavator location, including:

[0114] Establish the excavator coordinate system with the operator inside the excavator cab as the origin, and connect the radar coordinate system (X... L ,Y L Z L Transform to the excavator coordinate system (x) b y b , z b );

[0115] [ x b y b z b ] = R [ X L Y L Z L ] + T

[0116] Where: R is the rotation matrix of the radar to the excavator, and T is the translation matrix of the radar to the excavator;

[0117] T = [ x t y t z t ]

[0118] Where: x t y t z t It is the difference in three dimensions between the excavator's coordinate origin and the radar's installation position;

[0119] The data acquired by the radar are sequentially rotated around the X coordinate system of the radar. L Y L Z L After rotating the axis, we get R:

[0120]

[0121] in:

[0122] R yaw = [ cos i − sin i 0 sin i cos i 0 0 0 1 ]

[0123] R pitch = [ cos β 0 sin β 0 1 0 − sin β 0 cos β ]

[0124] R roll = [ 1 0 0 0 cos c − sin c 0 sin c cos c ]

[0125] Wherein: rotation angle It is the heading angle (yaw) in Euler angles;

[0126] This is the rotation matrix for rotating about the heading angle yaw;

[0127] Rotation angle It is the pitch angle (pictch) in Euler angles;

[0128] This is the rotation matrix for rotating about the pitch angle pictch;

[0129] Rotation angle It is the roll angle in Euler angles;

[0130] This is the rotation matrix for rotating around the roll angle.

[0131] Example 2

[0132] This invention provides a ranging device based on camera and radar fusion, comprising:

[0133] Acquisition module: used to acquire point cloud data and image data of the material area to be excavated through radar and camera, respectively;

[0134] The filtering module is used to input the image data of the material area to be excavated into the pre-trained target segmentation model, output the image regions containing the categories, and filter out the image regions of the target materials.

[0135] Extraction module: Used to input the point cloud data of the material area to be excavated into the pre-calibrated excavator radar-visual fusion imaging model, and to map the point cloud data of the material area to be excavated onto the image area of ​​the target material to extract the distance information of the target material.

[0136] Example 3:

[0137] This invention provides a ranging system based on camera and radar fusion.

[0138] One or more processors;

[0139] Memory, used to store one or more programs.

[0140] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described ranging method based on camera and radar fusion.

[0141] Example 4:

[0142] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described ranging method based on camera and radar fusion.

[0143] This embodiment first performs preliminary calibration based on the various poses of the calibration board acquired by the radar and camera. By comparing the two-dimensional calibration board data mapped to the image with the calibration board data in the image, the rotation matrix and translation matrix are further adjusted. The rotation and translation matrices of the radar and camera obtained by the two calibration methods are more accurate, and the fusion of laser and radar on the excavator can be realized more accurately.

[0144] The method of this invention adopts a multi-sensor fusion approach, which effectively avoids the poor measurement stability of a single sensor. By combining image segmentation algorithm and Ray vision fusion technology, the distance information of the target material of the excavator can be accurately obtained.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure one One or more processes and / or boxes Figure one A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure one One or more processes and / or boxes Figure one The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure one One or more processes and / or boxes Figure one The steps of the function specified in one or more boxes.

[0149] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A camera and radar fusion based ranging method, characterized in that, The method comprises the steps of: acquiring point cloud data and image data of the area to be excavated by radar and camera respectively; inputting the image data of the area to be excavated into a pre-trained target segmentation model to output image regions containing categories, and screening out image regions of target materials; inputting the point cloud data of the area to be excavated into a pre-calibrated radar and camera fusion imaging model to obtain image data corresponding to the point cloud data of the area to be excavated, combining the point cloud data and the image data corresponding to the segmentation bounding box of the target material in the target segmentation model, obtaining point cloud data containing only the target material, and finally extracting distance information of the target material according to the point cloud data of the target material, the calibration method of the pre-calibrated radar and camera fusion imaging model comprises primary calibration and secondary calibration; the primary calibration comprises the steps of: acquiring point cloud data and image data of a calibration board by radar and camera respectively, preliminarily calibrating different poses of the radar and the camera by the calibration board to obtain a rotation matrix and a translation matrix of the radar to the camera, projecting the point cloud data of the calibration board into the image data according to a camera intrinsic matrix and the rotation matrix and the translation matrix of the radar to the camera to obtain two-dimensional point cloud data of the calibration board, the secondary calibration comprises the steps of: generating a correction matrix by affine transformation according to the corresponding positions of the calibration board in the two-dimensional point cloud data and the image data of the calibration board to correct the radar data mapped to the camera, the calibration of different poses of the radar and the camera by the calibration board to obtain the rotation matrix and the translation matrix of the radar to the camera, projecting the point cloud data of the calibration board into the image data according to the camera intrinsic matrix and the rotation matrix and the translation matrix of the radar to the camera to obtain two-dimensional point cloud data of the calibration board, comprising the steps of: acquiring point cloud data obtained by scanning the calibration board by radar and image data obtained by photographing the calibration board by a camera; obtaining feature points of the calibration board in the radar and the camera under multiple angles and ranges to solve a conversion matrix from a radar coordinate system to a camera coordinate system; obtaining a camera intrinsic matrix by Zhang Zhengyou chessboard calibration method: ； where f x and f y are the camera equivalent focal lengths, u0 is a horizontal translation pixel value between a center point of an image coordinate system and a pixel coordinate system; v0 is a vertical translation pixel value between the center point of the image coordinate system and the pixel coordinate system; X C , Y C , Z C are camera coordinate system; u, v are pixel coordinate system, K is camera intrinsic matrix; fitting a rectangular shape of the calibration board in the point cloud data of the calibration board by a fitting method to obtain four three-dimensional points corresponding to the calibration board; obtaining pixel coordinates of four two-dimensional points corresponding to the calibration board in the image data of the calibration board according to the position of the calibration board; solving the conversion matrix from the radar coordinate system to the camera coordinate system by acquiring corresponding data of a plurality of groups of two-dimensional points to three-dimensional points: The camera conversion matrix formula is: ; where (X L ,Y L ,Z L ) represents the radar coordinate system, (u,v) represents the pixel coordinate system; Z C is the camera coordinate system, R LC is the rotation matrix of radar to camera, T LC is the translation matrix of radar to camera.

2. The camera and radar fusion based ranging method of claim 1, wherein, the target segmentation model is an unet model, the outputting of image regions containing categories and the screening out of image regions of target materials comprise the steps of: Image data of the area to be excavated is input into the unet model network for feature learning and feature extraction, and the output vector is converted into a probability distribution using a Softmax function, and the class with the highest probability is selected as the target area A ij where i is the image region index, and j represents the target class.

3. The camera and radar fusion based ranging method of claim 1, wherein, the generating of the correction matrix by affine transformation according to the corresponding positions of the calibration board in the two-dimensional point cloud data and the image data of the calibration board comprises the steps of: Given the four points of the quadrilateral fitted from the point cloud data of the two-dimensional calibration board as (x1, y1), (x2, y2), (x3, y3), (x4, y4), and the four points of the calibration board in the image data as (x ’ 1,y ’ 1), (x ’ 2,y ’ 2), (x ’ 3,y ’ 3), (x ’ 4,y ’ 4), the method comprises the following steps: (1) calculating the coordinates of the four points of the quadrilateral fitted from the point cloud data of the two-dimensional calibration board in the image data according to the following formula: x1' = x1 + (x2 - x1) * (x4 - x1) / (x2 - x1 + x4 - x3) y1' = y1 + (y2 - y1) * (y4 - y1) / (y2 - y1 + y4 - y3) x2' = x2 + (x3 - x2) * (x1 - x2) / (x3 - x2 + x1 - substituting coordinates into the following correction formula to generate the correction matrix: ； Where (x, y) represents the coordinates of the calibration board in the point cloud data, (x... ’ ,y ’ M represents the coordinates of the calibration plate in the image data. a Let represent the correction matrix, 'a' represent the scaling parameter, 'd' represent the rotation parameter, and 'b' and 'c' represent the shearing parameters. x and t y It is represented as a translation parameter.

4. The camera and radar fusion based ranging method of claim 2, wherein, after acquiring the distance information of the target material, converting a coordinate system of a radar at a position of the target material to a coordinate system of the excavator, comprising the steps of: Taking the driver in the driver's cabin of the excavator as the coordinate origin, an excavator coordinate system is established, and the radar coordinate system (X L ,Y L ,Z L ) is converted into the excavator coordinate system (x b , y b , z b ). ； Wherein: R is the rotation matrix of the radar to the excavator, T is the translation matrix of the radar to the excavator; ； wherein: x t , y t , z t are the differences in three-dimensional directions of the excavator coordinate origin and the radar installation location; The data obtained by the radar are rotated in turn around the X L , Y L , Z L axes of the radar coordinate system to obtain R: ； Wherein: ； ； ； where: rotation angle is the yaw angle in Euler angles; Rotation matrix for rotation around the yaw angle yaw; Rotation angle is the pitch angle in Euler angles pictch; rotation matrix for rotation about pitch angle pictch; Rotation angle is the roll angle in Euler angles; is the rotation matrix for rotation about the roll angle roll.

5. The camera and radar fusion based ranging method of claim 1, wherein, The distance information of the target material includes: The point cloud data of the area to be excavated is mapped to the target area A ij Mapping is performed to generate multi-dimensional data containing image RGB information and point cloud XYZ three-coordinate information, and the minimum value or average value in the multi-dimensional data of the excavator coordinate system (x b , y b , z b ) is selected as the final distance information of the target material to the excavator. ； ； where x dis , y dis , z dis represent the final distance information of the target material to the dimensions of the excavator (x b , y b , z b ); x p , y p , z p respectively represent the information of the point converted to the dimensions of the excavator coordinate system (x b , y b , z b ), and min represents the minimum value, and avg represents the average value.

6. A camera and radar fusion based ranging device according to the method of any one of claims 1 to 5, characterized in that, Including: The acquisition module is configured to acquire point cloud data and image data of the to-be-excavated material area by the radar and the camera respectively. The screening module is configured to input the image data of the to-be-excavated material area into a pre-trained target segmentation model, output an image region containing a category, and screen out an image region of the target material. The extraction module is configured to input the point cloud data of the to-be-excavated material area into a pre-calibrated radar and vision fusion imaging model of the excavator, obtain image data corresponding to the point cloud data of the to-be-excavated material area, combine the point cloud data and the corresponding image data with a segmentation bounding box of the target material in the target segmentation model, obtain point cloud data containing only the target material, and finally extract distance information of the target material according to the point cloud data of the target material.

7. A camera and radar fusion-based ranging system, characterized in that, one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the camera and radar fusion-based ranging method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the camera and radar fusion-based ranging method of any one of claims 1 to 5.

Images