Obstacle fusion detection method, chip and terminal for mining ramps

Abstract

This application proposes a fusion obstacle detection method, chip, and terminal for use on mining ramps. The method includes: in response to detecting a mining vehicle entering a ramp, acquiring 3D point cloud data from a lidar, a camera image, and millimeter-wave radar detection data to respectively determine first obstacle detection information, second obstacle detection information, and third obstacle detection information; projecting the 3D point cloud data onto the camera image; determining the 3D position information of the obstacle on the captured image as reflected by the 3D point cloud data based on the relative positional relationship between the obtained 2D projection point and the 2D detection frame of the captured image; extracting first fusion information from the first and second obstacle detection information; and determining second fusion information based on the first and third obstacle detection information. The technical solution of this application helps improve the driving safety of mining vehicles on mining ramps.

Description

Technical field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an obstacle fusion detection method, chip, and terminal for mining slopes. [Background Technology]

[0002] Several different types of sensors currently available have different strengths and weaknesses in their ability to perceive obstacles in autonomous driving. For example, LiDAR offers high positioning accuracy and accurate ranging, but the resulting point cloud is relatively sparse, making it difficult to classify and detect targets. Compared to LiDAR, cameras can effectively identify targets, but their positioning accuracy is relatively low and their ranging capabilities are limited. Millimeter-wave radar offers high accuracy in speed measurement, but the resulting point cloud is relatively sparse and lacks altitude information. In actual autonomous driving scenarios, complex road conditions make it impossible to obtain accurate measurement results using these traditional obstacle ranging methods, which are used on flat roads, thus impacting the safe operation of autonomous driving.

[0003] Therefore, how to accurately detect the location information of obstacles in complex road conditions during unmanned driving has become a technical problem that needs to be solved urgently. [Summary of the invention]

[0004] The embodiments of the present application provide an obstacle fusion detection method, chip and terminal for mining slopes, aiming to solve the technical problem in related technologies that it is difficult to accurately detect obstacle location information when unmanned driving is carried out in complex road conditions.

[0005] In a first aspect, an embodiment of the present application provides an obstacle fusion detection method for a mining slope, comprising: in response to detecting a mining vehicle entering a slope, obtaining 3D point cloud data from a lidar, a captured image from a camera, and detection data from a millimeter-wave radar; determining first obstacle detection information, second obstacle detection information, and third obstacle detection information based on the 3D point cloud data, the captured image, and the detection data, respectively; projecting the 3D point cloud data in the first obstacle detection information onto the captured image of the camera to obtain 2D projection points corresponding to the 3D point cloud data; determining the 3D position information of the obstacle reflected by the 3D point cloud data on the captured image based on the relative position relationship between the 2D projection points and the 2D detection frame of the captured image; extracting first fusion information from the first obstacle detection information and the second obstacle detection information based on the degree of overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame; and determining second fusion information of the effective obstacle based on the first fusion information and the third obstacle detection information.

[0006] In the second aspect, an embodiment of the present application provides an obstacle fusion detection device for a mining slope, comprising: a detection data acquisition unit for acquiring 3D point cloud data from a lidar, a captured image from a camera, and detection data from a millimeter-wave radar in response to detecting that a mining vehicle has entered a slope; an obstacle detection information determination unit for determining first obstacle detection information, second obstacle detection information, and third obstacle detection information based on the 3D point cloud data, the captured image, and the detection data; a projection processing unit for projecting the 3D point cloud data in the first obstacle detection information onto the captured image of the camera to obtain the first obstacle detection information, the second obstacle detection information, and the third obstacle detection information. a 2D projection point corresponding to the 3D point cloud data; a 3D position information acquisition unit, configured to determine, based on the relative positional relationship between the 2D projection point and the 2D detection frame of the captured image, the 3D position information of the obstacle reflected by the 3D point cloud data on the captured image; a first fusion information generation unit, configured to extract first fusion information from the first obstacle detection information and the second obstacle detection information based on a degree of overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame; and a second fusion information generation unit, configured to determine second fusion information of the valid obstacle based on the first fusion information and the third obstacle detection information.

[0007] In a third aspect, an embodiment of the present application provides a computer device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute any of the methods described in the first aspect above.

[0008] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method flow described in any one of the above-mentioned first aspects.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which includes at least one processor and a communication interface, wherein the communication interface is coupled to the at least one processor, and the at least one processor is used to run a computer program or instruction to implement any of the methods described in the first aspect above.

[0010] In a sixth aspect, an embodiment of the present application provides a terminal, which includes the obstacle fusion detection device for mining ramps as described in the second aspect above.

[0011] The beneficial effects of the present invention are as follows: To address the technical problem in related technologies that it is difficult to accurately detect obstacle location information when unmanned driving on complex road conditions, first, in response to detecting that a mining vehicle has entered a ramp, a laser radar, a camera, and a millimeter-wave radar are used to respectively obtain 3D point cloud data, captured images, and detection data on the ramp in front of the mining vehicle;

[0012] Next, based on the 3D point cloud data, captured images, and detection data, first, second, and third obstacle detection information are determined. The first, second, and third obstacle detection information reflect obstacles detected by the lidar, camera, and millimeter-wave radar, respectively.

[0013] Next, projecting the 3D point cloud data in the first obstacle detection information onto the image captured by the camera to obtain a 2D projection point corresponding to the 3D point cloud data, where the 2D projection point reflects the projection position of the obstacle corresponding to the 3D point cloud data on the captured image;

[0014] The 2D detection frame of the captured image reflects the 2D position information of the obstacle detected in the captured image, while the 2D projection point of the 3D point cloud data on the captured image reflects the projection position of the obstacle corresponding to the 3D point cloud data on the captured image. The difference between these two positions reflects, to a certain extent, the difference between the 3D detection results of the obstacle by the lidar and the 2D detection results of the obstacle by the camera. Based on this difference, the two positions can be fused and corrected to obtain the 3D position information of the obstacle within the 2D image of the captured image.

[0015] Secondly, the 2D detection frame is used to identify the position of the obstacle in the captured image, while the 2D projection frame is used to identify the projected position of the obstacle detected by the LiDAR in the captured image. Therefore, the degree of overlap between the 2D detection frame and the 2D projection frame reflects the degree of match between the obstacle in the captured image represented by the 2D detection frame and the obstacle detected by the LiDAR identified by the 2D projection frame. Furthermore, the degree of match between the two determines the fusion of the first obstacle detection information and the second obstacle detection information corresponding to the two.

[0016] At this point, the 3D point cloud data and the obstacle recognition results from the captured image can be fused based on the overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame. LiDAR has the advantages of high positioning accuracy and accurate ranging, while camera images have the advantage of easy target identification. In other words, the first obstacle detection information contains accurate obstacle position data, while the second obstacle detection information contains accurate obstacle type data. The first fused information obtained by fusing the first and second obstacle detection information can accurately reflect the location and type of the obstacle. Compared with the separate detection results of LiDAR and camera, both have higher detection accuracy, helping to improve the safety of autonomous driving.

[0017] Furthermore, millimeter-wave radar has a high accuracy in speed measurement. In other words, the third obstacle detection information includes valid obstacle speed information. The second fused information obtained by fusing the first fused information and the third obstacle detection information can accurately reflect the location, type, and speed of the obstacle. Compared with the individual detection results of lidar, camera, and millimeter-wave radar, it is more comprehensive and accurate, which helps to improve the reliability and safety of autonomous driving.

[0018] In summary, for scenarios where vehicles enter ramps in mining areas, the 3D position information of obstacles within the 2D image captured can be obtained by projecting the lidar's 3D point cloud data onto the captured image. The detection results of the lidar and camera are then fused to obtain first fused information. The effective obstacle speed information detected by the millimeter-wave radar is then fused with the first fused information to obtain second fused information. This second fused information combines the advantages of the lidar's high positioning accuracy and accurate ranging, the camera's accurate target recognition, and the millimeter-wave radar's accurate speed measurement. In other words, by fusing the advantageous parts of the detection results from the lidar, camera, and millimeter-wave radar, the resulting fused result can accurately reflect multi-dimensional information such as the obstacle's location, type, and speed. This is more comprehensive and accurate than the individual detection results of the lidar, camera, and millimeter-wave radar, helping to improve the reliability and safety of autonomous driving.

Brief Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flow chart of an obstacle fusion detection method for a mining ramp according to an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of monocular ranging according to an embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram of map model ranging according to an embodiment of the present application is shown;

[0023] Figure 4 A schematic diagram of obstacle height correction according to an embodiment of the present application is shown;

[0024] Figure 5 A schematic diagram of an obstacle fusion detection device for a mining ramp according to an embodiment of the present application is shown;

[0025] Figure 6 A schematic diagram of an obstacle fusion detection device for a mining ramp according to another embodiment of the present application is shown;

[0026] Figure 7 A schematic diagram of the structure of a storage medium provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0028] Figure 9 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0029] Figure 10 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. [Specific implementation method]

[0030] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A flow chart of an obstacle fusion detection method for a mine ramp according to an embodiment of the present application is shown.

[0032] like Figure 1 As shown, the obstacle fusion detection method for a mining ramp according to one embodiment of the present application includes:

[0033] Step 102 : In response to detecting that a mining vehicle has entered a ramp, 3D point cloud data from a laser radar, images from a camera, and detection data from a millimeter-wave radar are acquired.

[0034] There are many ways to detect whether a mining vehicle has entered a ramp. In one possible design, the location information of the mining vehicle can be periodically obtained, and based on a known map model, it is determined whether the location information of the mining vehicle is located at a ramp position. When it is detected that the location information of the mining vehicle is located at a ramp position, it is determined that the mining vehicle has entered a ramp, triggering the technical solution of obstacle fusion detection described below in this application. In another possible design, the heading information of the mining vehicle can be periodically obtained, and the angle between the heading information of the mining vehicle and the horizontal plane is calculated. When the angle exceeds the angle range corresponding to plane travel, it is determined that the mining vehicle has entered a ramp, triggering the technical solution of obstacle fusion detection described below in this application. Of course, the method of detecting whether a mining vehicle has entered a ramp is not limited to the above example, and can also be any other method that meets the actual obstacle detection needs.

[0035] Furthermore, when a mining vehicle enters a ramp, information on the ramp ahead can be acquired through LiDAR, cameras, and millimeter-wave radar. In one possible design, the vehicle's heading angle can be acquired through combined inertial navigation, with the direction corresponding to that heading angle serving as the detection direction for the LiDAR, cameras, and millimeter-wave radar.

[0036] Step 104 : Determine first obstacle detection information, second obstacle detection information, and third obstacle detection information based on the 3D point cloud data, the captured image, and the detection data.

[0037] Obstacles can be identified using a preset target recognition algorithm on the 3D point cloud data, captured images, and detection data, respectively, to obtain corresponding first, second, and third obstacle detection information. The first, second, and third obstacle detection information reflect obstacles detected by the lidar, camera, and millimeter-wave radar, respectively. The preset target recognition algorithm includes, but is not limited to, any method capable of target recognition, such as a deep learning algorithm and a neural network model.

[0038] Step 106 : Projecting the 3D point cloud data in the first obstacle detection information onto the image captured by the camera to obtain 2D projection points corresponding to the 3D point cloud data.

[0039] The 2D projection point corresponding to the 3D point cloud data on the captured image reflects the projection position of the obstacle corresponding to the 3D point cloud data on the captured image.

[0040] Step 108 : Determine 3D position information of the obstacle on the captured image reflected by the 3D point cloud data based on the relative position relationship between the 2D projection point and the 2D detection frame of the captured image.

[0041] The 2D detection frame of the captured image reflects the 2D position information of the obstacle detected in the captured image, while the 2D projection point corresponding to the 3D point cloud data on the captured image reflects the projection position of the obstacle corresponding to the 3D point cloud data on the captured image. The difference between these two positions reflects, to a certain extent, the difference between the 3D detection results of the obstacle by the lidar and the 2D detection results of the obstacle by the camera. Based on this difference, the two positions can be fused and corrected to obtain the 3D position information of the obstacle within the 2D image of the captured image.

[0042] Step 110 : extracting first fusion information from the first obstacle detection information and the second obstacle detection information according to the degree of overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame.

[0043] The 2D detection frame is used to identify the position of the obstacle in the captured image, while the 2D projection frame is used to identify the projected position of the obstacle detected by the lidar in the captured image. Therefore, the degree of overlap between the 2D detection frame and the 2D projection frame can reflect the degree of match between the obstacle in the captured image represented by the 2D detection frame and the obstacle detected by the lidar identified by the 2D projection frame. Furthermore, the degree of match between the two determines the fusion of the first obstacle detection information and the second obstacle detection information corresponding to the two.

[0044] Among them, when the overlap between the 2D detection frame and the 2D projection frame is large enough, it can be considered that the obstacle in the captured image represented by the 2D detection frame and the obstacle detected by the laser radar identified by the 2D projection frame match and are the same valid obstacle.

[0045] At this point, the 3D point cloud data and the obstacle recognition results of the captured image can be fused based on the overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame. LiDAR has the advantages of high positioning accuracy and accurate ranging, while the camera's captured image has the advantage of easy target identification. In other words, the first obstacle detection information contains accurate obstacle position data, and the second obstacle detection information contains accurate obstacle type data. The first fused information obtained by fusing the first and second obstacle detection information can accurately reflect the location and type of the obstacle. Compared with the individual detection results of LiDAR and camera, the first fused information has higher detection accuracy, which helps to improve the safety of autonomous driving.

[0046] Step 112: Determine second fusion information of the valid obstacle based on the first fusion information and the third obstacle detection information.

[0047] Millimeter-wave radar has high speed measurement accuracy. In other words, the third obstacle detection information includes valid obstacle speed information. The second fused information obtained by fusing the first and third obstacle detection information can accurately reflect the location, type, and speed of the obstacle. Compared with the individual detection results of lidar, camera, and millimeter-wave radar, the second fused information is more comprehensive and accurate, helping to improve the reliability and safety of autonomous driving.

[0048] The above technical solution, for scenarios where vehicles enter ramps in mining areas, can project the 3D point cloud data from the lidar onto the captured image to obtain the 3D position information of the obstacle within the 2D image captured. The detection results from the lidar and camera are then fused to obtain first fused information. The effective obstacle speed information detected by the millimeter-wave radar is then fused with the first fused information to obtain second fused information. This second fused information combines the advantages of the lidar's high positioning accuracy and accurate ranging, the camera's accurate target recognition, and the millimeter-wave radar's accurate speed measurement. In other words, by fusing the advantageous parts of the detection results from the lidar, camera, and millimeter-wave radar, the resulting fused result can accurately reflect multi-dimensional information such as the obstacle's location, type, and speed. This is more comprehensive and accurate than the individual detection results from the lidar, camera, and millimeter-wave radar, helping to improve the reliability and safety of autonomous driving.

[0049] The timestamps of the 3D point cloud data, the captured image, the real-time position data of the mining vehicle, the first obstacle detection information, the second obstacle detection information, and the third obstacle detection information can be aligned with the timestamp of the detection data from the millimeter-wave radar. This synchronizes the timelines of these pieces of information, ensuring that during information fusion, the fused set of first obstacle detection information, second obstacle detection information, and third obstacle detection information is data from the same time. This allows the fusion result to more accurately reflect the obstacle situation at that time, thereby improving the accuracy of obstacle detection.

[0050] In one possible design, the specific method of determining the 3D position information of the obstacle on the captured image reflected by the 3D point cloud data based on the relative position relationship between the 2D projection point and the 2D detection frame of the captured image includes: for the 2D detection frame where each obstacle is located on the captured image of the camera, if there are several 2D projection points in the 2D detection frame, the center point of the several 2D projection points is determined as the 3D position information of the obstacle on the captured image.

[0051] When the camera captures an image, it uses the target detection function to identify obstacles in the captured image with a 2D detection frame. The 2D projection point is a mapping of the obstacle in the 3D point cloud data of the LiDAR onto the captured image, reflecting the location of the obstacle in the 3D point cloud data. Therefore, if there are several 2D projection points within the 2D detection frame, it means that the obstacle identified by the 2D detection frame and the obstacles identified by several 2D projection points overlap, and there is a possibility that they are the same obstacle. Therefore, the center point of the several 2D projection points can be determined as the location of the obstacle in the 3D point cloud data. Accordingly, this center point becomes the 3D position information of the obstacle in the captured image.

[0052] If the 2D projection point does not exist within the 2D detection frame, detect whether there are multiple 2D projection points within a specified distance range outside the 2D detection frame; when multiple 2D projection points exist within the specified distance range, determine the center point of the multiple 2D projection points as the 3D position information of the obstacle on the captured image; otherwise, obtain the 3D position information of the obstacle on the captured image according to a predetermined measurement method.

[0053] When the 2D projection point does not exist in the 2D detection frame, it is possible to continue to detect whether there are several 2D projection points near the 2D detection frame. If so, it can still be explained that the obstacle identified by the 2D detection frame and the obstacles identified by the several 2D projection points have a certain possibility of overlap, and there is a possibility that they are the same obstacle. Therefore, the center point of the several 2D projection points can be determined as the 3D position information of the obstacle on the captured image.

[0054] The specified distance range is positively correlated with the size of the 2D detection frame. The specified distance range outside the 2D detection frame can be considered to be near the 2D detection frame. The specified distance range can be adaptively set based on the size of the 2D detection frame. A larger 2D detection frame corresponds to a larger specified distance range.

[0055] In one possible design, the specified distance range is 1 / 3 of the length of the bottom side of the 2D detection box.

[0056] In another possible design, the specified distance range is 1 / 2 of the diagonal length of the 2D detection box.

[0057] Of course, the specified distance range can be any size that is positively correlated with the size of the 2D detection frame and meets actual obstacle detection requirements, and is not limited to the examples given in this application.

[0058] If there are no 2D projection points near the 2D detection frame, it means that the obstacle identified by the 2D detection frame and the obstacle identified by the 2D projection point do not have the possibility of overlapping. At this time, a predetermined measurement method can be used to further detect the 3D position information of the obstacle on the captured image.

[0059] Through the above technical solution, the 3D position information obtained by projecting the 3D point cloud data of the lidar onto the 2D image of the captured image can be obtained, providing reliable position information for the subsequent fusion of the detection results of the lidar and camera, and improving the accuracy of obstacle fusion detection.

[0060] In some embodiments of the present invention, obtaining the target position information according to a predetermined measurement method includes: obtaining the first relative position information between the mining vehicle and the obstacle by a monocular ranging method; and determining the obstacle height based on the category of the obstacle; obtaining the intersection of a specified ray and a specified section of a predetermined map road model as the bottom center point of the obstacle, wherein the specified ray is a ray with a camera optical center as the origin, a line connecting the camera optical center and the bottom midpoint of the 2D detection frame of the obstacle, and the specified section is a section of the predetermined map road model on a vertical plane in the current driving direction of the mining vehicle; determining the position of the bottom center point after raising it by 1 / 2 of the obstacle height as the second relative position information of the mining vehicle and the obstacle; performing weighted sum processing on the first relative position information and the second relative position information to obtain the 3D position information of the obstacle on the captured image.

[0061] Among them, the principle of monocular ranging method is as follows Figure 2 As shown in the figure, the target direction vector v is determined based on the camera's optical center and the center of the 2D detection frame. Then, the obstacle's pixel height h and true height H, as well as the camera's focal length f, are used to determine the true distance D from the obstacle to the camera based on the principle of similar triangles. Finally, the true distance D is corrected using the target direction vector v, and v*D is used as the relative position between the obstacle and the camera, which is then used as the first relative position information.

[0062] like Figure 3As shown, the specific method of obtaining the second relative position information can be described as constructing a ray x with the optical center of the camera and the midpoint of the bottom edge of the 2D detection frame of the obstacle as two points on the ray. Ray x can represent the light of the bottom of the obstacle during the imaging process. Then, the intersection of ray x and the preset map road model is obtained, and the intersection is the actual bottom center point of the obstacle. Next, based on the category of the obstacle obtained by target recognition by the camera, the actual height of the obstacle is queried in the preset obstacle category and corresponding height table. Finally, the position after the actual bottom center point is raised by 1 / 2 of the actual height is used as the actual position of the obstacle after the corrected height, and the actual position is used as the second relative position information.

[0063] Next, a weighted summation process is performed on the first relative position information and the second relative position information. Optionally, the weights of the first relative position information and the second relative position information can be set to 0.5 respectively. The coordinates of the first relative position information are multiplied by the weight 0.5 to obtain a first product, and then the coordinates of the second relative position information are multiplied by the weight 0.5 to obtain a second product. The sum of the first product and the second product is used as the final 3D position information of the obstacle on the captured image.

[0064] The above technical solution determines the bottom center point of an obstacle using a preset map road model. This is then adjusted based on the height corresponding to the obstacle type, thereby correcting the obstacle's position information and improving the accuracy of obstacle detection results. Furthermore, this corrected obstacle position information can be further refined using monocular ranging, resulting in even more accurate obstacle location information and further improving the accuracy of obstacle detection results.

[0065] In some embodiments of the present invention, after obtaining the 3D position information of the obstacle in the 2D image of the captured image, and before using it for fusion, that is, before extracting the first fusion information from the first obstacle detection information and the second obstacle detection information, it also includes: determining the vehicle coordinate system based on the real-time posture information of the mining vehicle; using the calibration information in the 3D point cloud data and the calibration information in the captured image as reference information, converting the 3D position information into converted position information in the vehicle coordinate system, so as to extract the first fusion information based on the overlap between the converted position information and the 2D detection frame.

[0066] The real-time position information of mining vehicles includes their position and heading angle. The calibration information for the 3D point cloud data and the captured images is obtained from a pre-defined calibration file, which includes sample calibration information collected from sample 3D point cloud data and sample captured images. Using the calibration information in the 3D point cloud data and the captured images as reference information, the 3D position information is converted to converted position information in the vehicle coordinate system. This allows subsequent data processing to be performed in the vehicle coordinate system, facilitating calculations during unmanned driving and improving computational efficiency.

[0067] In some embodiments of the present invention, the specific step of extracting first fusion information from the first obstacle detection information and the second obstacle detection information includes: determining whether the obstacle is a valid obstacle based on the 3D position information and a predetermined fusionable obstacle screening rule; if the obstacle is a valid obstacle, when the degree of overlap between a 2D detection frame and a 2D projection frame corresponding to the 3D position information is greater than or equal to a specified overlap threshold, determining that the valid obstacle matches the obstacle within the 2D detection frame, and using the obstacle position information in the first obstacle detection information and the obstacle category information in the second obstacle detection information as the first fusion information of the valid obstacle.

[0068] Among them, the predetermined fusionable obstacle screening rules are used to screen out effective obstacles that require obstacle detection information fusion. Through this screening, the practicality of obstacle detection information fusion can be improved, providing a reliable foundation for the accuracy of unmanned driving.

[0069] Specifically, the predetermined fusionable obstacle screening rule is: if the position indicated by the 3D position information is within the field of view of the camera, and the obstacle is a foreground target of the camera, the obstacle is determined to be a valid obstacle.

[0070] Because the LiDAR's field of view is larger than the camera's, common obstacles detected by both the LiDAR and the camera exist only within their intersection, that is, within the camera's field of view. Furthermore, since cameras can recognize a wide range of obstacles and distinguish between foreground and background, the background can include obstacles farther ahead on the road, known as background targets. However, the obstacles that need to be detected and addressed during autonomous driving are often those closer to the camera or to vehicles in the mining area, known as foreground targets. Therefore, obstacles that need to be detected and addressed during autonomous driving can be considered foreground targets.

[0071] Based on the above, the common obstacles detected by the lidar and camera can be screened by taking the conditions that must be met simultaneously, namely, being within the camera's field of view and belonging to the camera's foreground. This achieves effective identification of the common obstacles detected by the lidar and camera, and contributes to further obstacle detection information fusion.

[0072] Specifically, a specified overlap threshold can be set to represent the minimum overlap required for a valid obstacle in the captured image represented by the 2D detection frame to match a valid obstacle detected by the LiDAR and identified by the 2D projection frame. Therefore, when the overlap between the 2D detection frame and the 2D projection frame is greater than or equal to the specified overlap threshold, the valid obstacle is determined to match the obstacle within the 2D detection frame.

[0073] At this time, since the laser radar has the advantages of high positioning accuracy and accurate ranging, and the camera's captured images have the advantage of easy target identification, the advantages of both can be combined, and the obstacle position information in the first obstacle detection information of the laser radar and the obstacle category information in the second obstacle detection information detected by the camera are used as the first fusion information of the effective obstacle.

[0074] It should be supplemented that, based on actual detection requirements, the overlap between the 2D detection frame and the 2D projection frame can be set to a first ratio of the intersection area of the 2D detection frame and the 2D projection frame to the 2D detection frame; or a second ratio of the intersection area of the 2D detection frame and the 2D projection frame to the 2D projection frame; or a third ratio of the intersection area of the 2D detection frame and the 2D projection frame to the smaller frame in the 2D detection frame and the 2D projection frame; or a fourth ratio of the intersection area of the 2D detection frame and the 2D projection frame to the larger frame in the 2D detection frame and the 2D projection frame; or the maximum value or the average value of any multiple of the first ratio, the second ratio, the third ratio and the fourth ratio. This application does not impose any further restrictions on the overlap between the 2D detection frame and the 2D projection frame corresponding to the 3D position information. Any method of setting the overlap is within the scope of protection of this application.

[0075] The aforementioned solution mainly adopts screening measures to integrate common obstacles in the foreground targets of the lidar and the camera. On the other hand, the background targets of the camera can include obstacles at a longer distance ahead on the road. On this basis, in order to increase the comprehensiveness of obstacle detection during the autonomous driving process and detect as much real-time road conditions as possible, the background targets of the camera can be directly identified as valid obstacles.

[0076] Specifically, when the overlap between the 2D detection frame and the 2D projection frame is less than a specified overlap threshold, the obstacle within the 2D detection frame is set as a valid obstacle; and the obstacle category and location information of the obstacle within the 2D detection frame are set as the first fusion information of the valid obstacle. In other words, because the camera detects objects more accurately, the remaining obstacles in the camera image that are not matched with the obstacles detected by the lidar after the above screening scheme are set as valid obstacles. These remaining obstacles include at least all background objects in the camera image.

[0077] The above technical solutions improve the comprehensiveness of obstacle detection for mining vehicles driving on slopes, helping to improve driving safety.

[0078] In one possible design, a specific method for determining the second fused information of the valid obstacle based on the first fused information and the third obstacle detection information includes: determining whether the relative distance between the obstacle in the third obstacle detection information and the closest valid obstacle is less than a specified distance threshold based on the position information of the obstacle after height information correction in the third obstacle detection information and the position information of the closest valid obstacle to the obstacle in the third obstacle detection information; if the relative distance is determined to be less than the specified distance threshold, determining that the obstacle in the third obstacle detection information matches the closest valid obstacle, where the specified distance threshold is a specified percentage of the length of the closest valid obstacle to the obstacle in the third obstacle detection information; and determining the second fused information of the valid obstacle using the velocity information of the obstacle in the third obstacle detection information and the first fused information of the closest valid obstacle as the second fused information of the valid obstacle.

[0079] That is, for each valid obstacle determined in the aforementioned scheme, a corresponding obstacle is matched in the third obstacle detection information of the millimeter-wave radar. If the match is successful, the speed information of the matched obstacle in the third obstacle detection information is assigned to the first fusion information, thereby obtaining the second fusion information with accurate speed information.

[0080] Therefore, we can take advantage of the high accuracy of millimeter-wave radar in measuring obstacle speed and merge its measured speed information with the obstacle position information measured by lidar and the obstacle type information measured by camera, so that the second fusion information obtained is more comprehensive and accurate, thereby providing an accurate and reliable driving control data foundation for unmanned driving.

[0081] Specifically, if the relative distance between the obstacle in the third obstacle detection information and the nearest valid obstacle is sufficiently small, the two are considered to match and are the same obstacle. In other words, when the relative distance between the obstacle in the third obstacle detection information and the nearest valid obstacle is less than a specified distance threshold, the obstacle in the third obstacle detection information is determined to match the nearest valid obstacle.

[0082] The specified distance threshold is a specified percentage of the length of the closest valid obstacle to the obstacle in the third obstacle detection information. In one possible design, the specified distance threshold is 1 / 2 of the length of the closest valid obstacle to the obstacle in the third obstacle detection information. This application does not further limit the specified percentage; any percentage that is consistent with practical application scenarios is within the scope of protection of this application.

[0083] Furthermore, millimeter-wave radar can only detect obstacle distances in planar terms and cannot account for factors such as road slope. Consequently, the obstacle height information in the third obstacle detection signal from the millimeter-wave radar may differ from the actual obstacle height. In other words, due to long, uneven road slopes, the obstacle position output by the millimeter-wave radar is lower than the actual road surface. Therefore, to improve the accuracy of matching obstacles detected by the millimeter-wave radar with valid obstacles, the obstacle height information in the third obstacle detection signal can be corrected.

[0084] Specifically, if Figure 4 As shown, an arc can be generated with the position of the millimeter-wave radar as the origin and the effective detection range of the millimeter-wave radar as the radius r; and the height information in the position information of the obstacle in the third obstacle detection information is corrected according to the intersection of the arc and the road in the predetermined map model and the individual height of the obstacle in the third obstacle detection information.

[0085] The above technical solution corrects the obstacle height information detected by the millimeter-wave radar, thereby correcting the obstacle position information detected by the millimeter-wave radar. This allows matching with valid obstacles based on the corrected position information, improving the accuracy of matching the millimeter-wave radar detection results with valid obstacles. Furthermore, if the obstacle in the third obstacle detection information is determined to match a valid obstacle based on the corrected position information of the millimeter-wave radar, the obstacle speed information detected by the millimeter-wave radar can be fused with the first fused information of the valid obstacle to obtain the second fused information. Therefore, correcting the obstacle height information detected by the millimeter-wave radar indirectly improves the accuracy of the second fused information, which is the final obstacle fusion detection result.

[0086] The individual height of an obstacle is calculated by searching a preset table of obstacle categories and corresponding heights for the actual height of the obstacle based on the category of the obstacle obtained by target recognition by the camera.

[0087] Figure 5 A schematic diagram of an obstacle fusion detection device for a mine ramp according to an embodiment of the present application is shown.

[0088] In combination with the above technical solutions, the obstacle fusion detection device for mining slopes can be installed externally on the vehicle or integrated into the vehicle as a part of the vehicle. Figure 5 As shown, an obstacle fusion detection device for mining slopes in one embodiment of the present application includes at least an input module, a detection module, and a fusion module. The input module can collect 3D point cloud data through a lidar, collect 2D image data (i.e., capture images) through a camera, collect vehicle posture data through GPS (Global Positioning System) / IMU (Inertial Measurement Unit), and obtain point cloud data through a millimeter-wave radar. At the same time, the input module can also obtain calibration data from sensor calibration parameters. The calibration data is used to convert the various data in the above technical solutions into vehicle coordinates for calculation to reduce computational complexity. In addition, the input module can also obtain a road model from a high-precision map provided by a third party.

[0089] The detection module includes three sub-modules: 3D point cloud detection, 2D image detection and 2.5D detection. 3D point cloud detection is used to identify obstacles in the 3D point cloud data of the lidar, 2D image detection is used to identify obstacles in the images taken by the camera, and 2.5D detection is used to identify obstacles in the detection results of the millimeter wave radar.

[0090] At this point, the input module transmits all the collected data to the fusion module, and each sub-module of the detection module also transmits the recognition results to the fusion module.

[0091] After receiving this data, the fusion module first fuses the detection results of the lidar and camera. Specifically, it obtains the first fused information through various processes such as point cloud information projection, image target ranging, coordinate conversion, data screening, and detection fusion. Next, the fusion module fuses this first fused information with the detection results of the millimeter-wave radar. Specifically, it obtains the second fused information through altitude correction, target matching, and velocity assignment, forming the final fused detection result. The specific fusion process is described in the previous embodiments and will not be repeated here.

[0092] In response to the problem in related technologies that the presence of ramps on mining roads leads to inaccurate obstacle detection by unmanned mining vehicles, the above technical solution first realizes the detection of the 3D position of obstacles in the 2D image detection results by fusing the detection results of the lidar and the camera. In the process of fusing the detection results of the millimeter-wave radar, the height information in the millimeter-wave radar detection results is corrected, thereby improving the accuracy and reliability of the detection results, which is conducive to the safe implementation of unmanned driving.

[0093] Figure 6 A schematic diagram of an obstacle fusion detection device for a mine ramp according to another embodiment of the present application is shown.

[0094] like Figure 6 As shown, according to another embodiment of the present application, an obstacle fusion detection device 600 for a mining slope includes: a detection data acquisition unit 602, for acquiring 3D point cloud data from a lidar, a captured image from a camera, and detection data from a millimeter-wave radar in response to detecting that a mining vehicle has entered a slope; an obstacle detection information determination unit 604, for determining first obstacle detection information, second obstacle detection information, and third obstacle detection information based on the 3D point cloud data, the captured image, and the detection data; a projection processing unit 606, for projecting the 3D point cloud data in the first obstacle detection information onto the captured image of the camera to obtain the first obstacle detection information, the second obstacle detection information, and the third obstacle detection information. a 2D projection point corresponding to the 3D point cloud data; a 3D position information acquiring unit 608, for determining the 3D position information of the obstacle reflected by the 3D point cloud data on the captured image based on the relative positional relationship between the 2D projection point and the 2D detection frame of the captured image; a first fusion information generating unit 610, for extracting first fusion information from the first obstacle detection information and the second obstacle detection information based on the degree of overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame; and a second fusion information generating unit 612, for determining second fusion information of the valid obstacle based on the first fusion information and the third obstacle detection information.

[0095] In one embodiment of the present invention, optionally, the obstacle fusion detection device 600 for mining slopes further includes: a time alignment unit for aligning the respective timestamps of the 3D point cloud data, the captured image, the real-time posture data of the mining vehicle, the first obstacle detection information, the second obstacle detection information and the third obstacle detection information with the timestamp of the detection data of the millimeter wave radar.

[0096] In one embodiment of the present invention, optionally, the 3D position information acquisition unit 608 is used to: for the 2D detection frame where each obstacle is located on the captured image of the camera, if there are several 2D projection points within the 2D detection frame, determine the center point of the several 2D projection points as the 3D position information of the obstacle on the captured image; if the 2D projection point does not exist within the 2D detection frame, detect whether there are several 2D projection points within a specified distance range outside the 2D detection frame; when there are several 2D projection points within the specified distance range, determine the center point of the several 2D projection points as the 3D position information of the obstacle on the captured image, wherein the specified distance range is positively correlated with the size of the 2D detection frame; otherwise, obtain the 3D position information of the obstacle on the captured image according to a predetermined measurement method.

[0097] In one embodiment of the present invention, optionally, the 3D position information acquisition unit 608 is used to: obtain the first relative position information between the mining vehicle and the obstacle by monocular ranging; and determine the obstacle height based on the category of the obstacle; obtain the intersection of a specified ray and a specified section of a predetermined map road model as the bottom center point of the obstacle, wherein the specified ray is a ray with a camera optical center as the origin, a line connecting the camera optical center and the bottom midpoint of the 2D detection frame of the obstacle, and the specified section is a section of the predetermined map road model on a vertical plane in the current driving direction of the mining vehicle; determine the position of the bottom center point after raising it by 1 / 2 of the obstacle height as the second relative position information of the mining vehicle and the obstacle; perform weighted sum processing on the first relative position information and the second relative position information to obtain the 3D position information of the obstacle on the captured image.

[0098] In one embodiment of the present invention, optionally, it also includes: a coordinate system conversion unit, which is used to determine the vehicle coordinate system based on the real-time posture information of the mining vehicle before the first fusion information generation unit 610 extracts the first fusion information; using the calibration information in the 3D point cloud data and the calibration information in the captured image as reference information, converting the 3D position information into converted position information in the vehicle coordinate system, so as to extract the first fusion information according to the overlap between the converted position information and the 2D detection frame.

[0099] In one embodiment of the present invention, optionally, the first fusion information generating unit 610 is configured to determine whether the obstacle is a valid obstacle based on the 3D position information and a predetermined fusionable obstacle screening rule, wherein the predetermined fusionable obstacle screening rule is: if the position indicated by the 3D position information is within the field of view of the camera and the obstacle is a foreground target of the camera, the obstacle is determined to be a valid obstacle; if the obstacle is a valid obstacle, when the degree of overlap between a 2D detection frame and a 2D projection frame corresponding to the 3D position information is greater than or equal to a specified overlap threshold, the valid obstacle is determined to match the obstacle within the 2D detection frame, and the obstacle position information in the first obstacle detection information and the obstacle category information in the second obstacle detection information are used as the first fusion information of the valid obstacle.

[0100] In one embodiment of the present invention, optionally, the obstacle fusion detection device 600 for a mining slope further includes: an obstacle supplement unit, configured to set the obstacle within the 2D detection frame as the valid obstacle when the degree of overlap between the 2D detection frame and the 2D projection frame is less than a specified overlap threshold; and the first fusion information generation unit 610 is further configured to set the obstacle category information and obstacle position information of the obstacle within the 2D detection frame as the first fusion information of the valid obstacle.

[0101] In one embodiment of the present invention, optionally, the degree of overlap between the 2D detection frame and the 2D projection frame is: a first ratio of the intersection area of the 2D detection frame and the 2D projection frame to the 2D detection frame; or a second ratio of the intersection area of the 2D detection frame and the 2D projection frame to the 2D projection frame; or a third ratio of the intersection area of the 2D detection frame and the 2D projection frame to the smaller frame between the 2D detection frame and the 2D projection frame; or a fourth ratio of the intersection area of the 2D detection frame and the 2D projection frame to the larger frame between the 2D detection frame and the 2D projection frame; or the maximum value or the average of any multiple of the first ratio, the second ratio, the third ratio and the fourth ratio.

[0102] In one embodiment of the present invention, optionally, the second fusion information generating unit 612 is used to: generate an arc with the position of the millimeter-wave radar as the origin and the effective detection range of the millimeter-wave radar as the radius; and correct the height information in the position information of the obstacle in the third obstacle detection information based on the intersection of the arc and the road in the predetermined map model and the individual height of the obstacle in the third obstacle detection information.

[0103] In one embodiment of the present invention, optionally, the second fusion information generating unit 612 is configured to: determine, based on the position information of the obstacle after height information correction in the third obstacle detection information and the position information of the valid obstacle closest to the obstacle in the third obstacle detection information, whether a relative distance between the obstacle in the third obstacle detection information and the valid obstacle closest to the obstacle is less than a specified distance threshold; if the relative distance is determined to be less than the specified distance threshold, determine that the obstacle in the third obstacle detection information matches the valid obstacle closest to the obstacle, where the specified distance threshold is a specified percentage of the length of the valid obstacle closest to the obstacle in the third obstacle detection information; and determine the speed information of the obstacle in the third obstacle detection information and the first fusion information of the valid obstacle closest to the obstacle as the second fusion information of the valid obstacle.

[0104] The obstacle fusion detection device 600 for mining slopes uses any one of the solutions in the above embodiments, and therefore has all the above technical effects, which will not be repeated here.

[0105] Figure 7 A schematic diagram of the structure of a storage medium provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, a computer-readable storage medium 700 stores a computer program 710. When executed by a processor, the computer program 710 is used to implement the obstacle fusion detection method for mining ramps as described in any of the above embodiments. The obstacle fusion detection method for mining ramps has been described in detail above and will not be repeated here.

[0106] The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Storage medium 700 can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0107] As one possible design, storage medium 700 may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage computer devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0108] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, computer device 800 includes memory 820, processor 810, and a computer program stored on memory 820 and executable by the processor. When processor 810 executes computer program 840, it performs the steps of the method described herein, enabling fusion obstacle detection for vehicles traveling on slopes in mining areas. It should be noted that computer program 840 in this embodiment is identical to computer program 710 described above.

[0109] The memory 820 may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. The memory 820 has a storage space 830 for storing a computer program 840 for executing any of the method steps of the above-described method. The computer program 840 may be read from or written to one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disk (CD), a memory card, or a floppy disk. Such computer program products are typically, for example, Figure 7 The computer-readable storage medium. A computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0110] Figure 9A schematic diagram of the structure of a chip provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the chip 900 includes one or more (including two) processors 910 and a communication interface 930. The communication interface 930 is coupled to the at least one processor 910, and the at least one processor 910 is configured to run computer programs or instructions to execute the steps of the method of the present application, thereby enabling obstacle fusion detection for vehicles traveling on slopes in mining areas.

[0111] Preferably, the memory 940 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0112] In the embodiment of the present application, the memory 940 may include a read-only memory and a random access memory, and provides instructions and data to the processor 910. A portion of the memory 940 may also include a non-volatile random access memory (NVRAM).

[0113] In the embodiment of the present application, the memory 940, the communication interface 930 and the memory 940 are coupled together through the bus system 920. In addition to the data bus, the bus system 920 may also include a power bus, a control bus and a status signal bus. Figure 9 Various buses are labeled as bus system 920 .

[0114] The method described in the above embodiment of the present application can be applied to the processor 910, or implemented by the processor 910. The processor 910 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the steps of the above method can be completed by the hardware integrated logic circuit in the processor 910 or the instructions in the form of software. The above-mentioned processor 910 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 910 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present invention.

[0115] Figure 10 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention is shown in FIG. Figure 10As shown, the terminal 1000 includes the obstacle fusion detection device 600 for mining slopes described in the above technical solution of this application.

[0116] The terminal 1000 can execute the steps of the method of the present application through the obstacle fusion detection device 600 for mining ramps, thereby realizing obstacle fusion detection for vehicles traveling on mining ramps. It is understood that the implementation method of the terminal 1000 controlling the obstacle fusion detection device 600 for mining ramps can be set according to the actual application scenario and is not specifically limited in the embodiments of the present application.

[0117] The terminal 1000 includes but is not limited to: vehicles, vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, vehicle-mounted units, vehicle-mounted radars or vehicle-mounted cameras and other sensors. The vehicle can implement the method provided in this application through the vehicle-mounted terminal, vehicle-mounted controller, vehicle-mounted module, vehicle-mounted module, vehicle-mounted components, vehicle-mounted chips, vehicle-mounted units, vehicle-mounted radars or cameras. The vehicles in this application include passenger cars and commercial vehicles. Common models of commercial vehicles include but are not limited to: pickup trucks, micro trucks, light trucks, micro buses, dump trucks, trucks, tractors, trailers, special vehicles and mining vehicles. Mining vehicles include but are not limited to mining trucks, wide-body trucks, articulated trucks, excavators, electric shovels, bulldozers, etc. This application does not further limit the type of smart cars, and any type of vehicle is within the scope of protection of this application.

[0118] The technical solution of the present application is described in detail above in conjunction with the accompanying drawings. Through the technical solution of the present application, for the scenario where a vehicle in a mining area enters a slope, the 3D position information of the obstacle in the 2D image of the captured image can be obtained by projecting the 3D point cloud data of the lidar onto the captured image. The detection results of the lidar and the camera are fused to obtain first fusion information, and then the effective obstacle speed information detected by the millimeter-wave radar is fused with the first fusion information to obtain second fusion information. This allows the second fusion information to have the advantages of high positioning accuracy and accurate ranging of the lidar, accurate target recognition of the camera, and accurate speed measurement of the millimeter-wave radar. In other words, the advantageous parts of the detection results of the lidar, camera, and millimeter-wave radar are fused, and the resulting fusion result can accurately reflect multi-dimensional information such as the position, type, and speed of the obstacle. Compared with the individual detection results of the lidar, camera, and millimeter-wave radar, it is more comprehensive and accurate, which helps to improve the reliability and safety of unmanned driving.

[0119] It should be understood that although the terms "first," "second," and so on may be used in embodiments of this application to describe obstacle detection information, such obstacle detection information should not be limited to these terms. These terms are merely used to distinguish one type of obstacle detection information from another. For example, without departing from the scope of the embodiments of this application, the first obstacle detection information could also be referred to as the second obstacle detection information, and similarly, the second obstacle detection information could also be referred to as the first obstacle detection information.

[0120] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interface, indirect coupling or communication connection of the device or unit, which may be electrical, mechanical or other forms.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0123] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0124] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0125] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An obstacle fusion detection method for mining slopes, characterized in that: include: In response to detecting that a mining vehicle has entered a ramp, acquiring 3D point cloud data from a lidar, images from a camera, and detection data from a millimeter-wave radar; determining first obstacle detection information, second obstacle detection information, and third obstacle detection information based on the 3D point cloud data, the captured image, and the detection data; Projecting the 3D point cloud data in the first obstacle detection information onto the image captured by the camera to obtain 2D projection points corresponding to the 3D point cloud data; Determining 3D position information of the obstacle on the captured image reflected by the 3D point cloud data based on a relative positional relationship between the 2D projection point and the 2D detection frame of the captured image; extracting first fusion information from the first obstacle detection information and the second obstacle detection information based on a degree of overlap between a 2D projection frame corresponding to the 3D position information and the 2D detection frame; determining second fusion information of a valid obstacle based on the first fusion information and the third obstacle detection information; The determining, based on the relative positional relationship between the 2D projection point and the 2D detection frame of the captured image, 3D position information of the obstacle reflected by the 3D point cloud data on the captured image includes: For the 2D detection frame of each obstacle in the image captured by the camera, If there are multiple 2D projection points within the 2D detection frame, determine the center point of the multiple 2D projection points as the 3D position information of the obstacle on the captured image; If the 2D projection point does not exist in the 2D detection frame, detecting whether there are multiple 2D projection points within a specified distance range outside the 2D detection frame; When there are multiple 2D projection points within the specified distance range, determining the center point of the multiple 2D projection points as the 3D position information of the obstacle on the captured image, wherein the specified distance range is positively correlated with the size of the 2D detection frame; otherwise, obtaining the 3D position information of the obstacle on the captured image according to a predetermined measurement method; The obtaining of the 3D position information of the obstacle on the captured image according to a predetermined measurement method includes: Obtaining first relative position information between the mining vehicle and the obstacle by monocular ranging; Determining the height of the obstacle based on the category of the obstacle; Obtaining the intersection of a specified ray and a specified cross section of a predetermined map road model as the bottom center point of the obstacle, wherein the specified ray is a ray with the camera optical center as its origin and a line connecting the camera optical center and the midpoint of the bottom edge of the 2D detection frame of the obstacle, and the specified cross section is a cross section of the predetermined map road model on a vertical plane in the current driving direction of the mining vehicle; The position of the bottom center point after being raised by 1 / 2 of the height of the obstacle is determined as the second relative position information between the mining vehicle and the obstacle; A weighted summation process is performed on the first relative position information and the second relative position information to obtain 3D position information of the obstacle on the captured image.

2. The obstacle fusion detection method for mining slopes according to claim 1, characterized in that: The determining, based on the first fusion information and the third obstacle detection information, second fusion information of the valid obstacle includes: Generate an arc with the position of the millimeter-wave radar as the origin and the effective detection range of the millimeter-wave radar as the radius; The height information in the position information of the obstacle in the third obstacle detection information is corrected according to the intersection point of the arc and the road in the predetermined map model and the individual height of the obstacle in the third obstacle detection information.

3. The obstacle fusion detection method for mining slopes according to claim 2, characterized in that: The determining, based on the first fusion information and the third obstacle detection information, second fusion information of a valid obstacle includes: determining, based on the position information of the obstacle after the height information is corrected in the third obstacle detection information and the position information of the valid obstacle closest to the obstacle in the third obstacle detection information, whether the relative distance between the obstacle in the third obstacle detection information and the valid obstacle closest to the obstacle is less than a specified distance threshold; If it is determined that the relative distance is less than the specified distance threshold, it is determined that the obstacle in the third obstacle detection information matches the valid obstacle closest to itself, The specified distance threshold is a specified percentage of the length of the valid obstacle closest to the obstacle in the third obstacle detection information; The speed information of the obstacle in the third obstacle detection information and the first fusion information of the valid obstacle closest to the obstacle are determined as the second fusion information of the valid obstacle.

4. The obstacle fusion detection method for mining slopes according to claim 1, characterized in that: Before extracting first fusion information from the first obstacle detection information and the second obstacle detection information, the method further includes: Determining a vehicle coordinate system based on the real-time position information of the mining vehicle; Using the calibration information in the 3D point cloud data and the calibration information in the captured image as reference information, the 3D position information is converted into converted position information in the vehicle coordinate system, so as to extract the first fusion information based on the overlap between the converted position information and the 2D detection frame.

5. The obstacle fusion detection method for mining slopes according to claim 1, characterized in that: The extracting first fusion information from the first obstacle detection information and the second obstacle detection information according to the overlap between the 2D projection frame corresponding to the 3D position information and the 2D detection frame includes: Determining whether the obstacle is a valid obstacle based on the 3D position information and a predetermined fusionable obstacle screening rule, wherein the predetermined fusionable obstacle screening rule is: if the position indicated by the 3D position information is within the field of view of the camera and the obstacle is a foreground object of the camera, determining that the obstacle is a valid obstacle; If the obstacle is the valid obstacle, when the degree of overlap between the 2D detection frame and the 2D projection frame corresponding to the 3D position information is greater than or equal to a specified overlap threshold, the valid obstacle is determined to match the obstacle within the 2D detection frame, and the obstacle position information in the first obstacle detection information and the obstacle category information in the second obstacle detection information are used as the first fusion information of the valid obstacle.

6. The obstacle fusion detection method for mining slopes according to claim 5, characterized in that: The degree of overlap between the 2D detection frame and the 2D projection frame corresponding to the 3D position information is: a first ratio of an intersection area of the 2D detection frame and the 2D projection frame to the 2D detection frame; or a second ratio of an intersection area of the 2D detection frame and the 2D projection frame to the 2D projection frame; or a third ratio of an intersection area of the 2D detection frame and the 2D projection frame to a smaller area of the 2D detection frame and the 2D projection frame; or a fourth ratio of an intersection area of the 2D detection frame and the 2D projection frame to a larger area of the 2D detection frame and the 2D projection frame; or The maximum value or the average value of any of the first ratio, the second ratio, the third ratio and the fourth ratio.

7. The obstacle fusion detection method for mining slopes according to claim 5, characterized in that: Also includes: When the overlap between the 2D detection frame and the 2D projection frame is less than a specified overlap threshold, setting the obstacle in the 2D detection frame as the valid obstacle; The obstacle category information and obstacle position information of the obstacle in the 2D detection frame are set as the first fusion information of the valid obstacle.

8. The obstacle fusion detection method for mining slopes according to any one of claims 1 to 7, characterized in that: Before projecting the 3D point cloud data in the first obstacle detection information onto the image captured by the camera to obtain 2D projection points corresponding to the 3D point cloud data, the method further includes: Align the respective timestamps of the 3D point cloud data, the captured image, the real-time posture data of the mining vehicle, the first obstacle detection information, the second obstacle detection information, and the third obstacle detection information with the timestamp of the detection data of the millimeter wave radar.

9. An obstacle fusion detection device for mining slopes, characterized in that: include: A detection data acquisition unit is used to acquire 3D point cloud data from a laser radar, images from a camera, and detection data from a millimeter-wave radar in response to detecting that a vehicle in the mining area has entered a ramp; an obstacle detection information determining unit, configured to respectively determine first obstacle detection information, second obstacle detection information, and third obstacle detection information based on the 3D point cloud data, the captured image, and the detection data; a projection processing unit, configured to project the 3D point cloud data in the first obstacle detection information onto the image captured by the camera to obtain 2D projection points corresponding to the 3D point cloud data; a 3D position information acquiring unit, configured to determine 3D position information of the obstacle reflected by the 3D point cloud data on the captured image based on a relative positional relationship between the 2D projection point and the 2D detection frame of the captured image; a first fusion information generating unit, configured to extract first fusion information from the first obstacle detection information and the second obstacle detection information according to a degree of overlap between a 2D projection frame corresponding to the 3D position information and the 2D detection frame; a second fusion information generating unit, configured to determine second fusion information of a valid obstacle based on the first fusion information and the third obstacle detection information; The 3D position information acquisition unit is configured to: for a 2D detection frame in which each obstacle is located on an image captured by the camera, if there are multiple 2D projection points within the 2D detection frame, determine the center point of the multiple 2D projection points as the 3D position information of the obstacle on the captured image; if the 2D projection point does not exist within the 2D detection frame, detect whether there are multiple 2D projection points within a specified distance range outside the 2D detection frame; when multiple 2D projection points exist within the specified distance range, determine the center point of the multiple 2D projection points as the 3D position information of the obstacle on the captured image, wherein the specified distance range is positively correlated with the size of the 2D detection frame; otherwise, obtain the 3D position information of the obstacle on the captured image according to a predetermined measurement method; The 3D position information acquisition unit is used to: obtain the first relative position information between the mining vehicle and the obstacle through monocular ranging; and determine the height of the obstacle based on the category of the obstacle; obtain the intersection of a specified ray and a specified section of a predetermined map road model as the bottom center point of the obstacle, wherein the specified ray is a ray with a camera optical center as the origin, a line connecting the camera optical center and the midpoint of the bottom edge of the 2D detection frame of the obstacle, and the specified section is a section of the predetermined map road model on a vertical plane in the current driving direction of the mining vehicle; determine the position of the bottom center point after raising it by 1 / 2 of the obstacle height as the second relative position information of the mining vehicle and the obstacle; perform weighted sum processing on the first relative position information and the second relative position information to obtain the 3D position information of the obstacle on the captured image.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A chip, characterized in that: The chip includes at least one processor and a communication interface, the communication interface is coupled to the at least one processor, and the at least one processor is used to run a computer program or instruction to implement the obstacle fusion detection method for a mining slope as described in any one of claims 1-8.

13. A terminal, characterized in that: The terminal includes the obstacle fusion detection method for mining ramps according to any one of claims 1-8.

Images