Mine car obstacle recognition and distance measurement method and device and storage medium

Abstract

The invention discloses a mine car obstacle recognition and distance measurement method and device and a storage medium. The method comprises the steps of constructing a segmentation template, marking image feature points, extracting an obstacle contour line, performing coarse and fine segmentation on a target obstacle based on the intersection point proportion of a background segmentation surface and the obstacle contour line, and determining a contour segmentation area of the target obstacle; the method comprises the following steps: marking point cloud feature points, constructing an obstacle template, carrying out matching mapping on a target obstacle and the obstacle template by adopting a non-rigid iterative nearest point algorithm, correcting feature point offset in a matching process in combination with a KD-Tree method, and calculating to obtain point cloud parameters of the target obstacle. And identifying the type of the target obstacle and measuring the distance according to the contour segmentation region and the point cloud parameters. According to the method, the target obstacle image and the point cloud information are combined, the target obstacle is segmented according to the intersection point proportion, the fuzzy area is smoothly segmented, the point cloud parameters are measured while the target obstacle is segmented, and the purpose of identifying the type and the distance of the target obstacle is achieved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a mine car obstacle recognition and distance measurement method, device and storage medium based on feature fusion. Background Art

[0002] In the application scenario of unmanned mining vehicles in open-pit mines, obstacle recognition method is one of the key technologies to achieve unmanned driving.

[0003] Currently, existing perception systems struggle to stably complete perception tasks, meaning they are unable to consistently and correctly output accurate obstacle information to the path planning and vehicle control modules of autonomous driving systems. This situation is primarily due to two factors: software and hardware. In terms of hardware, sensors such as lidar, millimeter-wave radar, and cameras, which are important components of the perception system, are at risk of data loss due to damage to themselves or their wiring harnesses. Once such a situation occurs, it will directly affect the normal operation of the perception system and interfere with the unmanned mining vehicle's recognition and judgment of obstacles.

[0004] On the software side, current perception system software suffers from serious issues with missed detections and false detections. For example, when an obstacle is actually in the vehicle's direction of travel, the perception system fails to correctly identify and promptly issue obstacle information, preventing the vehicle from properly stopping and posing a safety hazard. Furthermore, even when there is no obstacle in the vehicle's direction of travel, the perception system mistakenly issues obstacle information, causing the vehicle to incorrectly stop, significantly impacting normal mine operations.

[0005] In addition, although the current obstacle feature point parameter measurement method based on sensor fusion has improved the obstacle perception capability to a certain extent, the perception range of this method for multi-target obstacles is largely limited by the computing power of the on-board computing platform.

[0006] It can be seen that it is crucial to study an unmanned driving perception method that can identify obstacles and measure their distance while reducing computing resource constraints. Summary of the Invention

[0007] In a first aspect of the present invention, in order to solve the above technical problems, a method for identifying and measuring obstacles in a mine car is provided, the method comprising: A segmentation template is constructed by averaging multi-sensor perception templates, image feature points are annotated according to the segmentation template, and obstacle contours are extracted. The target obstacle is coarsely and finely segmented based on the contour ratio of the intersection of the background segmentation surface and the obstacle contour, and the contour segmentation area of the target obstacle is determined; Based on the image feature points, point cloud feature points are annotated on the segmentation template, and an obstacle template containing point cloud parameter information is constructed. A non-rigid iterative closest point algorithm is used to match and map the target obstacle to the obstacle template. The KD-Tree method is used to correct feature point offsets during the matching process, and the point cloud parameters of the target obstacle are calculated. The type of the target obstacle is identified and the distance is measured based on the contour segmentation area and the point cloud parameters.

[0008] Furthermore, the multi-sensor includes at least two of a laser radar, a millimeter-wave radar and a camera.

[0009] Furthermore, the coarse segmentation and fine segmentation of the target obstacle specifically include: Acquire the contour line segmentation points of the target obstacle according to the contour line ratio, for use in segmenting the inner contour line of the target obstacle; According to the contour line segmentation points, a number of segmentation areas are divided to determine the outer contour segmentation boundary of the target obstacle.

[0010] Furthermore, the step of marking feature points of the point cloud on the segmentation template includes: Select feature point location , make rough annotation; Then mark the point As the root node, use the KD-Tree algorithm to search for the optimal point within the threshold , for detailed annotation.

[0011] Furthermore, the calculation to obtain the point cloud parameters of the target obstacle includes: Mapping the point cloud feature points on the obstacle template to the target obstacle; The point cloud feature points of the target obstacle are corrected according to the geometric positions of the point cloud feature points on the obstacle template, and the point cloud parameters of the target obstacle are calculated according to the mutual mapping relationship between the point cloud feature points.

[0012] Furthermore, the step of calculating the point cloud parameters of the target obstacle according to the mutual mapping relationship between the point cloud feature points specifically includes: The point cloud parameters are passed through a multi-tuple Indicates that, Indicates the obstacle template and target obstacle; Represents image feature points; Represents point cloud parameters; Indicates a mapping relationship.

[0013] Furthermore, the multi-tuple middle: ,in, is the obstacle template, is the target obstacle; ,in, is the feature point of the obstacle template image, is the target obstacle image feature point; ,in, is the obstacle template point cloud parameter, is the point cloud parameter of the target obstacle; Used to describe and The mapping relationship; The obstacle template image feature points are transformed into Mapped onto the target obstacle, recorded as ; The KD-Tree method is used to correct the position offset of the feature points that occur during the matching process to obtain the point cloud feature points. .

[0014] Furthermore, the method is applied to an obstacle perception system of an unmanned mining vehicle in an open-pit mine.

[0015] A second aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the mine car obstacle identification and distance measurement method when executing the computer program.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the mine car obstacle identification and distance measurement method.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: The present invention combines the target obstacle image and point cloud information, uses the obstacle morphological contour segmentation ratio to perform internal segmentation on the target obstacle, and combines the open-pit mine environment to smoothly segment the fuzzy area. While segmenting the target obstacle image information, the point cloud parameters are measured to achieve the purpose of identifying the target obstacle type and distance, and output the target obstacle information or drivable area information to meet the needs of unmanned mines to realize full-scene unmanned transportation operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a flowchart of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] This paper aims to provide a method for identifying and measuring obstacles in mine carts based on feature fusion. This method is applied to the obstacle perception system of unmanned mine carts in open-pit mines. It can identify obstacles and measure their distances while reducing computing resource constraints, meeting the needs of mines requiring unmanned mine carts to achieve full-scale unmanned transport operations.

[0022] See also Figure 1 ,The mine car obstacle recognition and distance measurement method based on ,feature fusion mainly includes the following steps: Step 1: Construct a segmentation template by averaging the multi-sensor perception template, annotate the image feature points according to the segmentation template, and extract the obstacle contour line; perform coarse and fine segmentation on the target obstacle based on the contour line ratio of the intersection of the background segmentation surface and the obstacle contour line, and determine the contour segmentation area of the target obstacle.

[0023] Those skilled in the art further explain that "segmentation through averaging multi-sensor perception templates" is a key step in constructing the segmentation template. This is because open-pit mine environments are complex, and data acquired by a single sensor may have limitations or errors. Averaging multi-sensor perception templates involves generating separate perception templates based on data collected by multiple sensors, such as lidar and cameras, on obstacles and the surrounding environment. These perception templates from different sources are then averaged, leveraging the strengths of each sensor to produce a segmentation template that more comprehensively and accurately reflects the actual scene. This effectively reduces the impact of single sensor failures or data deviations on segmentation results, providing a more reliable foundation for subsequent operations such as extracting obstacle outlines and determining outline ratios based on image feature points, thereby improving the stability and accuracy of the entire obstacle detection process.

[0024] Preferably, the multi-sensor includes at least two of a laser radar, a millimeter-wave radar and a camera.

[0025] In a further solution of this embodiment, the target obstacle is coarsely segmented and finely segmented based on the contour ratio of the intersection of the background segmentation surface and the obstacle contour line, specifically including: First, the contour segmentation points of the target obstacle are obtained according to the contour ratio, which are used to segment the internal contour of the target obstacle and realize the coarse segmentation of the target obstacle.

[0026] Then, according to the segmentation points of the target obstacle's contour, several segmentation areas are divided to determine the outer contour segmentation boundary of the target obstacle, thereby realizing the fine segmentation of the target obstacle's contour.

[0027] Step 2: Based on the image feature points, point cloud feature points are annotated on the segmentation template to construct an obstacle template containing point cloud parameter information. The target obstacle is matched and mapped with the obstacle template using a non-rigid iterative closest point algorithm. The KD-Tree method is used to correct the feature point offset during the matching process. The point cloud parameters of the target obstacle are calculated. The type of the target obstacle is identified and the distance is measured based on the contour segmentation area and the point cloud parameters.

[0028] In a further solution of this embodiment, there is an inclusion relationship between the point cloud feature point set and the image feature point set, and the point cloud feature points are annotated in the segmentation template to construct an obstacle template containing point cloud parameter information, specifically including the following steps: First, select the feature point position , for rough marking.

[0029] Then, mark the points As the root node, use the KD-Tree algorithm to search for the optimal point within the threshold , to achieve fine marking.

[0030] Next, the point cloud parameters of the target obstacle are calculated using the spatial distance and angle measurement formulas. The point cloud feature point constraints and point cloud parameter methods are recorded to construct an obstacle template containing point cloud parameter information.

[0031] In a further embodiment of the present invention, calculating the point cloud parameters of the target obstacle further includes the following steps: The point cloud feature points on the obstacle template are mapped to the target obstacle; the point cloud feature points of the target obstacle are corrected according to the geometric positions of the point cloud feature points on the obstacle template, and the point cloud parameters of the target obstacle are calculated based on the mutual mapping relationship between the point cloud feature points.

[0032] Among them, according to the mutual mapping relationship of point cloud feature points, the point cloud parameters of the target obstacle are calculated, specifically including: Defining a tuple ,in: , used to describe the obstacle template, is the obstacle template, The target obstacle.

[0033] , used to describe image feature points, is the feature point of the obstacle template image, is the target obstacle image feature point.

[0034] , used to describe point cloud parameters, is the obstacle template point cloud parameter, is the target obstacle point cloud parameter.

[0035] Used to describe and The mapping relationship.

[0036] The non-rigid iterative closest point algorithm is used to match and map the obstacle template to the target obstacle so that the two are close in shape and the feature points of the obstacle template image are mapped to the target obstacle. Mapped to the target obstacle, recorded as Due to the existence of local differences, feature point position offsets will occur during the matching process. The KD-Tree method is used to correct the feature point position offsets that occur during the matching process to obtain point cloud feature points. , calculate the target obstacle point cloud parameters.

[0037] The target obstacle contour segmentation area and point cloud parameters provide information for the subsequent perception system to identify the obstacle type and distance.

[0038] Those skilled in the art further explained that the open-pit mine environment is complex, and it is impossible to effectively realize obstacle recognition and background segmentation at the same time using only a single sensor and recognition method. Therefore, the present invention combines obstacle images and point cloud information to construct an original obstacle template containing lidar point cloud parameter information. By defining the feature points of the lidar point cloud, constraint rules are set to measure the point cloud parameters of the target obstacle. In addition, since the obstacle information obtained by the sensor contains both contour lines and external environment representations, the same obstacle at the same distance has the same contour lines and different external representations. Therefore, the present invention uses the obstacle morphology contour segmentation ratio to perform internal segmentation on the target obstacle, and combines the open-pit mine environment to smoothly segment the fuzzy area, and measures the point cloud parameters while segmenting the target obstacle image information, so as to achieve the purpose of identifying the obstacle type and distance, and output obstacle information or mine car drivable area information, to meet the needs of unmanned mines to achieve full-scene unmanned transportation operations.

[0039] In addition, the present invention also protects an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned mine car obstacle identification and distance measurement method are implemented.

[0040] The present invention also protects a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned mine car obstacle identification and distance measurement method when the computer program is executed by a processor.

[0041] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A mine car obstacle identification and distance measurement method, characterized in that: The method comprises: A segmentation template is constructed by averaging multi-sensor perception templates, image feature points are annotated according to the segmentation template, and obstacle contours are extracted. The target obstacle is coarsely and finely segmented based on the contour ratio of the intersection of the background segmentation surface and the obstacle contour, and the contour segmentation area of the target obstacle is determined; Based on the image feature points, point cloud feature points are annotated on the segmentation template, and an obstacle template containing point cloud parameter information is constructed. A non-rigid iterative closest point algorithm is used to match and map the target obstacle to the obstacle template. The KD-Tree method is used to correct feature point offsets during the matching process, and the point cloud parameters of the target obstacle are calculated. The type of the target obstacle is identified and the distance is measured based on the contour segmentation area and the point cloud parameters.

2. The mine car obstacle identification and distance measurement method according to claim 1, characterized in that: The multi-sensor includes at least two of a laser radar, a millimeter-wave radar and a camera.

3. The mine car obstacle identification and distance measurement method according to claim 1, characterized in that: The coarse segmentation and fine segmentation of the target obstacle specifically include: Acquire the contour line segmentation points of the target obstacle according to the contour line ratio, for use in segmenting the inner contour line of the target obstacle; According to the contour line segmentation points, a number of segmentation areas are divided to determine the outer contour segmentation boundary of the target obstacle.

4. The mine car obstacle identification and distance measurement method according to claim 1, characterized in that: The step of marking feature points of the point cloud in the segmentation template includes: Select feature point location , make rough annotation; Then mark the point As the root node, use the KD-Tree algorithm to search for the optimal point within the threshold , for detailed annotation.

5. The mine car obstacle identification and distance measurement method according to claim 1, characterized in that: The calculating and obtaining the point cloud parameters of the target obstacle includes: Mapping the point cloud feature points on the obstacle template to the target obstacle; The point cloud feature points of the target obstacle are corrected according to the geometric positions of the point cloud feature points on the obstacle template, and the point cloud parameters of the target obstacle are calculated according to the mutual mapping relationship between the point cloud feature points.

6. The mine car obstacle identification and distance measurement method according to claim 5, characterized in that: Calculating the point cloud parameters of the target obstacle according to the mutual mapping relationship of the point cloud feature points specifically includes: The point cloud parameters are passed through a multi-tuple Indicates that, Represents the obstacle template and target obstacle; Represents image feature points; Represents point cloud parameters; Indicates a mapping relationship.

7. The mine car obstacle identification and distance measurement method according to claim 6, characterized in that: The multi-tuple middle: ,in, is the obstacle template, is the target obstacle; ,in, is the feature point of the obstacle template image, is the target obstacle image feature point; ,in, is the obstacle template point cloud parameter, is the point cloud parameter of the target obstacle; Used to describe and The mapping relationship; The obstacle template image feature points are transformed into Mapped onto the target obstacle, recorded as ; The KD-Tree method is used to correct the position offset of the feature points that occur during the matching process to obtain the point cloud feature points. .

8. The mine car obstacle identification and distance measurement method according to claim 1, characterized in that: The method is applied to an obstacle perception system of an unmanned mining vehicle in an open-pit mine.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the mine car obstacle identification and distance measurement method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the mine car obstacle identification and distance measurement method according to any one of claims 1 to 7 are implemented.

Images