Machine vision and millimeter wave radar fused roadway environment modeling method

Through the fusion technology of machine vision and millimeter wave radar, a high-precision tunnel environment model is built, which solves the problems of low positioning efficiency and insufficient environmental perception in coal mining excavation, and achieves high-precision and efficient positioning in complex environments .

CN119048701APending Publication Date: 2024-11-29XIAN UNIV OF SCI & TECH +1
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411043980.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, drilling anchor equipment has low positioning efficiency in coal mine tunnel boring, is susceptible to environmental interference, and has a limited field of vision of millimeter-wave radar, making it difficult to monitor anchors in all directions, and cannot meet the needs of high-precision and long-distance detection.

Method used

The fusion technology of machine vision and millimeter wave radar is adopted to achieve high-precision perception and positioning of the tunnel environment through joint calibration, acquisition of image and point cloud data, fusion data, and construction of three-dimensional models.

Benefits of technology

It improves the positioning accuracy and robustness of drilling anchor equipment in complex environments, solves the problem of autonomous precise positioning of downhole drilling anchor equipment, enhances environmental perception capabilities, and is suitable for anchor rod detection in harsh environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048701A_ABST
    Figure CN119048701A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of coal mine intellectualization, in particular to a roadway environment modeling method based on machine vision and millimeter wave radar fusion. Comprising the following steps: step 1, carrying out joint calibration on a visual sensor and a millimeter-wave radar; 2, a visual sensor and a millimeter wave radar are used for roadway environment perception, and image data and point cloud data are obtained respectively; 3, fusing the image data and the point cloud data; 4, forming point cloud data information of the whole roadway by using the fused point cloud data, and constructing a roadway model; and 5, correcting the established three-dimensional roadway model by using the fused point cloud data. By fusing data of the visual sensor and the millimeter wave radar, a high-precision roadway environment model can be established, and the problem of autonomous and accurate positioning of underground drilling and anchoring equipment is solved. And by using a dynamically adjusted clustering algorithm, the sparse point cloud clustering quality of the millimeter wave radar is improved, and the detection effect of a target point is improved. In addition, machine vision and millimeter wave radar fused roadway environment modeling can also be used for correcting a three-dimensional roadway model, and the universality of the modeling method is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent coal mine technology, and in particular to a tunnel environment modeling method integrating machine vision and millimeter wave radar. Background Art

[0002] Coal mine tunnels are important working areas in coal mining. In order to ensure the stability and safety of the tunnels, anchor rods need to be installed on the tunnel walls to reinforce the support. Therefore, anchor support plays a vital role in coal mine excavation production. Improving tunnel excavation efficiency has become one of the important contents of coal mine intelligence.

[0003] At present, one of the main factors affecting the efficiency of tunnel excavation is the intelligent level of drilling and anchoring operations. However, the positioning of drilling and anchoring equipment is mostly completed through manual operation, which has problems such as low efficiency, susceptibility to environmental interference and limited field of view, and cannot ensure the accuracy and efficiency of positioning. Millimeter-wave radar usually has a limited field of view and transmission loss, and cannot monitor the anchors in the tunnel in an all-round way. The resolution is relatively low, and it is difficult to provide detailed information equivalent to the visual system and cannot meet the needs of long-distance detection. Millimeter-wave radar is not suitable for the classification of multiple targets. For visual sensors, it can make up for the shortcomings of millimeter-wave radar in the perception of target size and contour, and can retain the appearance information of the target, so as to effectively classify the target. However, there are also many problems in the processing that relies entirely on vision. Traditional visual methods have some limitations in specific environments, such as insufficient light, dust occlusion and reflection. Relying solely on visual detection may affect the appearance detection due to the harsh environment. Since the pixels corresponding to small targets are often few, background noise is more likely to have an adverse effect on detection. Summary of the invention

[0004] 1. Technical issues

[0005] In view of the above-mentioned prior art, in order to reduce the impact of the complex environment of the tunnel on the environmental perception and positioning of the drilling and anchoring equipment, improve accuracy, enhance robustness, and meet real-time requirements, this application proposes a tunnel environment modeling method that integrates machine vision and millimeter-wave radar. The introduction of vision and millimeter-wave radar fusion technology can effectively overcome these problems, and provide an accurate and efficient tunnel environment perception solution, provide accurate tunnel information, and effectively improve the positioning accuracy of drilling and anchoring equipment in complex environments. The drilling and anchoring equipment can refer to the supported anchor rods to solve the position information, and the visual sensor can provide high-resolution image information to capture the visual features of the anchor rods, such as shape, color, etc. The millimeter-wave radar can penetrate harsh environments such as coal dust and water mist to achieve non-contact detection of anchor rods and tunnel environments. By fusing the data of visual sensors and millimeter-wave radars, a high-precision tunnel environment model can be established to solve the problem of autonomous and precise positioning of underground drilling and anchoring equipment.

[0006] (II) Technical solution

[0007] According to a first aspect of an embodiment of this specification, a method for modeling a lane environment based on the fusion of machine vision and millimeter-wave radar is provided, the method comprising:

[0008] Step 1: Jointly calibrate the visual sensor and millimeter-wave radar;

[0009] Step 2: Use visual sensors and millimeter-wave radars to perceive the roadway environment and obtain image data and point cloud data respectively;

[0010] Step 3, fusing image data and point cloud data;

[0011] Step 4: Use the fused point cloud data to form point cloud data information of the entire tunnel and construct a tunnel model.

[0012] Furthermore, the tunnel environment modeling method also includes the following steps: Step 5, using the fused point cloud data to correct the established three-dimensional tunnel model.

[0013] Furthermore, the joint calibration steps in step 1 are as follows:

[0014] 1.1. Calibrate the visual sensor to obtain its internal parameters;

[0015] 1.2. Synchronize the time between the visual sensor and the millimeter wave radar;

[0016] 1.3. Obtain the spatial conversion relationship between the visual sensor and the millimeter wave radar;

[0017] 1.4. Obtain the depth image, RGB image and point cloud data at the same time, and parse the position, speed and signal strength information of each point through the point cloud data.

[0018] Furthermore, in step 2, the visual sensor is used to perceive the lane environment, and the steps of acquiring image data are as follows:

[0019] 2.1.1. Collect RGB image data of lanes using visual sensors;

[0020] 2.1.2. Build a target detection network, obtain an anchor visual recognition model, and detect the real-time image of the visual sensor to obtain the anchor detection frame and confidence;

[0021] Extract and save the anchor edge features in the detection frame, and build an anchor image local feature database according to all anchor edge features;

[0022] 2.1.3. Align the depth image collected at the same time to the RGB image according to the machine vision internal parameters, extract the target detection frame as the region of interest, process the corresponding region of interest in the depth image, and restore the effective information in the depth image;

[0023] 2.1.4. Extract the center point of the detection frame, obtain the depth value of the point on the depth map, and obtain the three-dimensional coordinates of the anchor target point in the RGB vision sensor coordinate system through coordinate transformation;

[0024] 2.1.5. Use the algorithm to perform straight line fitting for each row of anchor target points and correct the coordinate values.

[0025] Furthermore, in step 2, the millimeter wave radar is used to perceive the roadway environment, and the steps for obtaining the roadway point cloud data are as follows:

[0026] 2.2.1. Using the dynamically adjusted clustering algorithm, the point cloud data extracted from the same frame is clustered to form point cloud subsets to obtain a priori clustering frames;

[0027] 2.2.2. Filter out noise points using the method of inter-frame tracking information;

[0028] 2.2.3. Extract the point cloud intensity at the same depth value, automatically remove the point cloud data within a certain intensity range, and classify the clustered point set according to intensity;

[0029] 2.2.4. After point cloud data processing, the millimeter wave radar target detection results are obtained, the features of each cluster are extracted and the point cloud local feature database is constructed.

[0030] Furthermore, the steps of the dynamically adjusted clustering algorithm are as follows:

[0031] 6.1. Randomly divide the detection area into n sub-areas, each with a different volume. Let the volume of the mth sub-area be V m ;

[0032] 6.2. Traverse the number of point clouds in each sub-area and calculate the point cloud density of each sub-area

[0033]

[0034] Where N m is the number of point clouds in the mth sub-region;

[0035] 6.3. Divide the density intervals according to the calculated point cloud density of each sub-area, and set the first value for each density interval.

[0036] Furthermore, in step 3, after the machine vision and the millimeter wave radar each complete the target detection, the detection results of the two are weighted fused to achieve heterogeneous data feature matching between the two-dimensional image data and the three-dimensional point cloud. The fusion steps are as follows:

[0037] 3.1. Project the processed radar points onto the RGB image with the anchor detection frame according to the coordinate transformation relationship;

[0038] 3.2. The processed radar point cluster generates multiple regions of interest according to the prior clustering frame, extracts the centroid of the region of interest and the center point of the anchor detection frame in the image, and performs feature matching through the overlap of the two regions; the feature matching strategy includes calculating the intersection and union ratio of the radar region of interest and the detection frame to form a feature matching region pair;

[0039] 3.3. After feature matching, the anchor detected by machine vision is edge extracted, the point cloud cluster within the edge contour is densified, a complete anchor point cloud is generated according to the contour, and the extracted anchor position information is assigned to the centroid of the anchor point cloud.

[0040] Furthermore, in step 4, after the point cloud data at different times are spliced ​​together to form the point cloud data information of the entire tunnel, the point cloud is densified and the three-dimensional model of the tunnel is obtained by surface fitting.

[0041] Furthermore, in step 5, the process of correcting the three-dimensional tunnel model is as follows:

[0042] 5.1. After processing each frame of point cloud data, data fusion is performed with the image that has been recognized at the same time;

[0043] 5.2. Extract the complete anchor point cloud cluster after fusion, match the point cloud cluster to the corresponding position of the established three-dimensional tunnel model through spatial transformation, further perform surface fitting, and complete the correction of the three-dimensional tunnel model.

[0044] Furthermore, the anchor rods and the coal wall are distinguished in the three-dimensional tunnel model, and the position information of the anchor rods is extracted, and the anchor rods are used as targets to facilitate relative positioning of underground anchor drilling equipment during operation.

[0045] According to a second aspect of an embodiment of the present specification, a system for implementing any of the above-mentioned methods for modeling a lane environment is provided.

[0046] (III) Beneficial effects

[0047] This application designs a tunnel environment modeling method that integrates machine vision and millimeter-wave radar, which can effectively reduce the impact of the complex tunnel environment on the environmental perception and positioning of drilling and anchoring equipment, improve accuracy, enhance robustness, and meet real-time requirements.

[0048] (1) Tunnel environment modeling that integrates machine vision and millimeter-wave radar provides an accurate and efficient tunnel environment perception solution. By integrating the data of visual sensors and millimeter-wave radar, a high-precision tunnel environment model can be established to solve the problem of autonomous and precise positioning of underground drilling and anchoring equipment.

[0049] (2) The dynamically adjusted clustering algorithm is used to improve the clustering quality of millimeter-wave radar sparse point clouds and enhance the detection effect of target points.

[0050] (3) The tunnel environment modeling based on the fusion of machine vision and millimeter-wave radar can also modify the three-dimensional tunnel model, thus improving the universality of the modeling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the process of the laneway environment modeling method of this application;

[0052] Figure 2 It is a schematic diagram of coordinate transformation relationship;

[0053] Figure 3 This is a schematic diagram of the anchor bolt visual recognition detection and three-dimensional coordinate acquisition process;

[0054] Figure 4 This is a schematic diagram of the millimeter wave radar point cloud data processing process;

[0055] Figure 5 Schematic diagram of feature matching strategy;

[0056] Figure 6 A flowchart of the roadway environment modeling method based on the fusion of vision and millimeter-wave radar; DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0058] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0059] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0060] Next, the embodiments of this specification are described in detail.

[0061] like Figure 1 As shown, Figure 1 This is a flow chart of a tunnel environment modeling method according to an exemplary embodiment of the present specification, comprising the following steps:

[0062] Step 1: Jointly calibrate the visual sensor and the millimeter wave radar. The joint calibration steps are as follows:

[0063] 1.1. Calibrate the visual sensor to obtain its internal parameters. The machine vision here can use the visual sensor;

[0064] 1.2. Synchronize the time of the visual sensor and the millimeter-wave radar. According to the type of vision used and the sampling frequency of the millimeter-wave radar, the Lagrange interpolation method is used to align the time series of the two to ensure that the visual sensor and the millimeter-wave radar can collect data at the same time;

[0065] 1.3. According to the installation positions of the visual sensor and the millimeter-wave radar, obtain the spatial conversion relationship between the visual sensor and the millimeter-wave radar;

[0066] 1.4. Obtain the depth image, RGB image and point cloud data at the same time, and parse the position, speed and signal strength information of each point through the point cloud data.

[0067] Step 2: Use visual sensors and millimeter-wave radars to perceive the lane environment and obtain image data and point cloud data respectively.

[0068] First, the visual recognition and detection of anchor bolts and the acquisition of their three-dimensional coordinates are shown in Figure 3 As shown:

[0069] (1) Using visual sensors to collect RGB image data of the tunnel, the image is enhanced by using contrast-limited adaptive histogram equalization because dust, water mist, and uneven lighting in the tunnel will have a certain impact on the recognition effect.

[0070] (2) Build a YOLOv7+BiFormer+CBAM target detection network. The CBAM attention mechanism improves recognition accuracy by strengthening feature information. The BiFormer attention mechanism is used to improve the detection ability of small targets. The anchor visual recognition model is trained by a self-built anchor sample data set, and the real-time image of the visual sensor is detected to obtain the anchor detection frame and confidence. The anchor edge features in the detection frame are extracted and saved, and the anchor image local feature database is constructed according to all anchor edge features;

[0071] (3) The depth image collected at the same time is aligned with the RGB image according to the intrinsic parameters of the visual sensor, the target detection frame is extracted as the region of interest, and the corresponding region of interest in the depth image is processed by median filtering and bilateral filtering to restore the effective information in the depth image;

[0072] (4) Extract the center point of the detection frame, obtain the depth value of the point on the depth map, and obtain the three-dimensional coordinates of the anchor target point in the RGB vision sensor coordinate system through coordinate transformation. The coordinate transformation model is as follows:

[0073] The conversion relationship from the visual sensor coordinate system to the pixel coordinate system is as follows: Figure 2 shown. Figure 2 Where P is the real target point, P' is the corresponding point of the target point on the image, uv is the pixel coordinate system, xoy is the image coordinate system, and is the origin of the RGB vision sensor coordinate system.

[0074] The image coordinate system uv and the pixel coordinate system xoy are on the same imaging plane. Their measurement units and origin positions are different. The coordinate conversion formula is:

[0075]

[0076] Wherein, (u, v) is the coordinate of the target point in the pixel coordinate system, in pixels; (x, y) is the coordinate of the target point in the image coordinate system, in mm; dx and dy represent the actual size of a single pixel along the x-axis and y-axis, in mm; (u0, v0) is the coordinate value of the origin of the image coordinate system in the pixel coordinate system.

[0077] The conversion formula between the visual sensor coordinate system and the image coordinate system is:

[0078]

[0079] Where f is the focal length of the visual sensor, and (X, Y, Z) is the coordinate of the target point in the visual sensor coordinate system.

[0080] Combining formula (1) and formula (2), we can get the conversion relationship from the visual sensor coordinate system to the pixel coordinate system:

[0081]

[0082] (5) Use the least squares method to perform straight line fitting on the target points of each row of anchor rods and correct the coordinate values.

[0083] Furthermore, the millimeter wave radar point cloud data processing of the tunnel

[0084] (1) Using the DBSCAN clustering algorithm with dynamically adjusted Minpoints, the point cloud data extracted from the same frame is clustered to form point cloud subsets and obtain a priori clustering boxes;

[0085] (2) Although the clustering effect of the DBSCAN clustering algorithm is relatively ideal, it is only applicable to dense point cloud data. For the sparse point cloud of the millimeter wave radar, the clustering quality is poor, resulting in the target points being marked as noise points. The Minpoints parameter in the DESCAN algorithm is improved to dynamic adjustment. The steps are as follows:

[0086] a. Randomly divide the radar detection area into n sub-areas. The volume of each sub-area is uncertain. Let the volume of the mth sub-area be V m ;

[0087] b. Traverse the number of point clouds in each sub-area and calculate the point cloud density of each sub-area

[0088]

[0089] Where N m is the number of point clouds in the mth sub-region;

[0090] C. Divide the density intervals according to the calculated point cloud density of each sub-area, and set the first value for each density interval, recorded as the Minpoints value.

[0091] (2) Use the method of inter-frame tracking information to filter out noise points. Analyze the speed and signal strength value of each frame of point cloud data. For the first frame of data, filter out the data with abnormal speed value, extract the centroid of each prior clustering box and calculate its speed value, use the Kalman filter to predict the speed of each cluster at the next moment, set the speed range according to the predicted speed, and filter out the points outside the speed range at the next moment;

[0092] (3) The point cloud signal strength value is related to the material and distance of the target being measured. The point cloud strength at the same depth value is extracted, and the point cloud data within a certain intensity range is automatically removed. The clustered point set is classified into targets such as anchors, steel strips, and coal walls according to the intensity.

[0093] (4) After processing the point cloud data, the millimeter-wave radar target detection results are obtained, the features of each cluster are extracted, and a point cloud local feature database is constructed;

[0094] Step 3, fusing image data and point cloud data;

[0095] After vision and millimeter-wave radar complete target detection, the detection results of the two are weighted fused to achieve heterogeneous data feature matching between two-dimensional image data and three-dimensional point cloud. The fusion process is as follows:

[0096] (1) Project the processed radar points onto the RGB image with the anchor detection frame according to the coordinate transformation relationship;

[0097] (2) The processed radar point cluster generates multiple regions of interest according to the prior clustering box, extracts the centroid of the region of interest and the center point of the anchor detection box in the image, and performs feature matching based on the overlap between the two regions;

[0098] Furthermore, the feature matching strategy includes:

[0099] Calculate the Euclidean distance between the center of each detection frame of the image and the centroid of each radar region of interest, select the radar region of interest with the smallest Euclidean distance and the detection frame to form a feature matching region pair, and calculate the intersection over union (IOU) between the two;

[0100] a. If IOU>0.5, the millimeter wave radar and vision both detect the anchor target, and the coordinate information extracted by visual detection is used as the anchor positioning information;

[0101] b. If 0.5>IOU>0, assign weights ω1 and ω2 to vision and millimeter-wave radar, fuse the detection results, and output the three-dimensional coordinates of the anchor:

[0102]

[0103] Where K T is the three-dimensional coordinate of the anchor in the center, is the anchor coordinate obtained in step 2, is the 3D coordinate of the centroid of the point cloud prior clustering box;

[0104] c. If IOU = 0, it means that the detection results of the two are not related, and there may be false detection or missed detection, so no fusion processing is performed. The detection frame area in the image is measured for similarity with the anchor image local feature database to determine whether the selected frame is the correct anchor target. If so, the three-dimensional coordinates of the anchor are obtained according to the method in step 2; the point cloud cluster is measured for similarity with the point cloud local feature database to determine whether the point cloud cluster is the correct anchor target. If so, the centroid coordinates of the cluster are extracted as the location information of the anchor;

[0105] (3) After feature matching, the edges of the visually detected anchors are extracted, the point cloud clusters within the edge contours are densified, a complete anchor point cloud is generated according to the contours, and the extracted anchor position information is assigned to the centroid of the anchor point cloud.

[0106] Step 4: Use the fused point cloud data to form point cloud data information of the entire tunnel and construct a tunnel model.

[0107] As the millimeter-wave radar moving time T increases, the position of the scanned lane moves forward, and the point cloud data at different times are spliced ​​together to form the point cloud data information of the entire lane. The point cloud is densified through the PU-GAN network, and the three-dimensional model of the lane is obtained through surface fitting.

[0108] Due to the lack of anchor targets in the tunnel model, the visual sensor and millimeter-wave radar fix their relative positions, move synchronously and at a uniform speed to scan the tunnel, and perform data fusion with the recognized image at the same time after processing each frame of point cloud data; extract the complete anchor point cloud cluster after fusion, match the point cloud cluster to the corresponding position of the established three-dimensional tunnel model through spatial transformation, further perform surface fitting, and complete the correction of the three-dimensional tunnel model; the anchor and coal wall can be distinguished in the model, and the position information of the anchor can be extracted. The anchor is used as a target to facilitate the relative positioning of the underground drilling and anchoring machine equipment during operation.

[0109] This application designs a tunnel environment modeling method that integrates machine vision and millimeter-wave radar, which can effectively reduce the impact of the complex tunnel environment on the environmental perception and positioning of drilling and anchoring equipment, improve accuracy, enhance robustness, and meet real-time requirements.

[0110] The tunnel environment modeling that integrates machine vision and millimeter-wave radar provides an accurate and efficient tunnel environment perception solution. By integrating the data of visual sensors and millimeter-wave radar, a high-precision tunnel environment model can be established to solve the problem of autonomous and precise positioning of underground drilling and anchoring equipment. The dynamically adjusted clustering algorithm is used to improve the clustering quality of millimeter-wave radar sparse point clouds and improve the detection effect of target points.

[0111] The tunnel environment modeling that integrates machine vision and millimeter-wave radar can also correct the three-dimensional tunnel model, improving the universality of the modeling method.

[0112] It should be understood that the present description is not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0113] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A tunnel environment modeling method based on the fusion of machine vision and millimeter wave radar, characterized in that: The method comprises: Step 1: Jointly calibrate the visual sensor and millimeter-wave radar; Step 2: Use visual sensors and millimeter-wave radars to perceive the roadway environment and obtain image data and point cloud data respectively; Step 3, fusing image data and point cloud data; Step 4: Use the fused point cloud data to form point cloud data information of the entire tunnel and construct a tunnel model.

2. A tunnel environment modeling method according to claim 1, characterized in that: The tunnel environment modeling method also includes the following steps: Step 5, using the fused point cloud data to correct the established three-dimensional tunnel model.

3. A tunnel environment modeling method according to claim 1, characterized in that: The joint calibration steps in step 1 are as follows: 1.

1. Calibrate the visual sensor to obtain its internal parameters; 1.

2. Synchronize the time between the visual sensor and the millimeter wave radar; 1.

3. Obtain the spatial conversion relationship between the visual sensor and the millimeter wave radar; 1.

4. Obtain the depth image, RGB image and point cloud data at the same time, and parse the position, speed and signal strength information of each point through the point cloud data.

4. A tunnel environment modeling method according to claim 1, characterized in that: In step 2, the visual sensor is used to perceive the lane environment, and the steps for obtaining image data are as follows: 2.1.

1. Collect RGB image data of lanes using visual sensors; 2.1.

2. Build a target detection network, obtain an anchor visual recognition model, and detect the real-time image of the visual sensor to obtain the anchor detection frame and confidence; Extract and save the anchor edge features in the detection frame, and build an anchor image local feature database according to all anchor edge features; 2.1.

3. Align the depth image collected at the same time to the RGB image according to the machine vision internal parameters, extract the target detection frame as the region of interest, process the corresponding region of interest in the depth image, and restore the effective information in the depth image; 2.1.

4. Extract the center point of the detection frame, obtain the depth value of the point on the depth map, and obtain the three-dimensional coordinates of the anchor target point in the RGB vision sensor coordinate system through coordinate transformation; 2.1.

5. Use the algorithm to perform straight line fitting for each row of anchor target points and correct the coordinate values.

5. A tunnel environment modeling method according to claim 1, characterized in that: In step 2, the millimeter wave radar is used to perceive the roadway environment and obtain the roadway point cloud data in the following steps: 2.2.

1. Using the dynamically adjusted clustering algorithm, the point cloud data extracted from the same frame is clustered to form point cloud subsets to obtain a priori clustering frames; 2.2.

2. Filter out noise points using the method of inter-frame tracking information; 2.2.

3. Extract the point cloud intensity at the same depth value, automatically remove the point cloud data within a certain intensity range, and classify the clustered point set according to intensity; 2.2.

4. After point cloud data processing, the millimeter wave radar target detection results are obtained, the features of each cluster are extracted and the point cloud local feature database is constructed.

6. A tunnel environment modeling method according to claim 5, characterized in that: The steps of the dynamically adjusted clustering algorithm are as follows: 6.

1. Randomly divide the detection area into n sub-areas, each with a different volume. Let the volume of the mth sub-area be V m ; 6.

2. Traverse the number of point clouds in each sub-area and calculate the point cloud density of each sub-area Where N m is the number of point clouds in the mth sub-region; 6.

3. Divide the density intervals according to the calculated point cloud density of each sub-area, and set the first value for each density interval.

7. A tunnel environment modeling method according to claim 1, characterized in that: In step 3, after the machine vision and the millimeter wave radar complete the target detection respectively, the detection results of the two are weighted fused to achieve heterogeneous data feature matching between the two-dimensional image data and the three-dimensional point cloud. The fusion steps are as follows: 3.

1. Project the processed radar points onto the RGB image with the anchor detection frame according to the coordinate transformation relationship; 3.

2. The processed radar point cluster generates multiple regions of interest according to the prior clustering frame, extracts the centroid of the region of interest and the center point of the anchor detection frame in the image, and performs feature matching through the overlap of the two regions; the feature matching strategy includes calculating the intersection and union ratio of the radar region of interest and the detection frame to form a feature matching region pair; 3.

3. After feature matching, the anchor detected by machine vision is edge extracted, the point cloud cluster within the edge contour is densified, a complete anchor point cloud is generated according to the contour, and the extracted anchor position information is assigned to the centroid of the anchor point cloud.

8. A tunnel environment modeling method according to claim 1, characterized in that: In step 4, after the point cloud data at different times are spliced ​​together to form the point cloud data information of the entire lane, the point cloud is densified and the three-dimensional model of the lane is obtained by surface fitting.

9. A tunnel environment modeling method according to claim 1, characterized in that: In step 5, the process of correcting the three-dimensional tunnel model is as follows: 5.

1. After processing each frame of point cloud data, data fusion is performed with the image that has been recognized at the same time; 5.

2. Extract the complete anchor point cloud cluster after fusion, match the point cloud cluster to the corresponding position of the established three-dimensional tunnel model through spatial transformation, further perform surface fitting, and complete the correction of the three-dimensional tunnel model.

10. A tunnel environment modeling method according to claim 1, characterized in that: The anchor rods and coal walls are distinguished in the three-dimensional tunnel model, and the position information of the anchor rods is extracted. The anchor rods are used as targets for relative positioning during the operation of underground anchor drilling equipment.

Citation Information

Cited By

  • Multi-sensor fusion drilling and anchoring robot positioning method

    CN120333466A

  • A Multi-Sensor Fusion Method for Drilling and Anchoring Robot Localization

    CN120333466B

  • Tunnel surrounding rock three-dimensional deformation monitoring system and method integrating monocular vision and millimeter wave radar

    CN120991736A

  • Method for processing two-side measurement data of underground roadway millimeter wave radar and measurement device

    CN122019965A