Method for detecting mouse hole in city through fusion of camera and laser radar based on unmanned vehicle

Abstract

A method for detecting a mouse hole in a city based on fusion of a camera and a laser radar of an unmanned vehicle comprises the steps that in the advancing process of the unmanned vehicle, an image of the ground surface or the wall root of the city is collected through the camera, and a suspected mouse hole area is recognized through a target detection model; after laser radar data and image data are fused, depth information of a suspected mouse hole area is extracted, and geometric features of the suspected mouse hole area are judged; and when the candidate area meets a preset depth condition, finally determining that the candidate area is a mouse hole. In order to adapt to the difference of the distribution positions of the mouse holes, a camera and a radar device are arranged on the roof of the unmanned vehicle and used for detecting the ground holes; and meanwhile, another collection device is arranged at the bottom of the unmanned vehicle and is used for detecting a horizontal hole in the wall root. According to the method, false detection caused by shadow or background factors can be effectively reduced in a complex urban environment, and the accuracy and robustness of mouse hole detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to urban environment intelligent detection and monitoring technology, and particularly relates to a rat hole detection method based on laser radar and image fusion, which is suitable for unmanned vehicle inspection scene in urban roads, wall roots and the like. BACKGROUND

[0002] In urban environment, rat holes are important habitats for rodents and activities, which pose a threat to public health and municipal safety. The existing detection methods mainly rely on manual patrol, which is low in efficiency and high in cost, and it is difficult to accurately find rat holes in complex environments.

[0003] The existing camera-based image detection method is easily disturbed by light, shadow, debris and the like, and the detection result is often not to identify the hole itself, but to misjudge as a shadow or black area. Single visual information lacks depth constraint, resulting in high false detection rate. The existing rat hole detection method mostly relies on camera-based visual detection, which can identify suspected rat holes at the image level, but is easily disturbed by light and shadow in complex environments. For example, in sunny conditions, the shadow area formed by plants or roadside facilities is often misreported as a rat hole by the target detection model, resulting in high false detection rate.

[0004] Therefore, a method of fusing depth information and image information is needed to improve the accuracy and robustness of rat hole identification. SUMMARY

[0005] The purpose of the present application is to provide a method for detecting rat holes in cities based on camera and laser radar fusion of unmanned vehicles. The method first identifies suspected rat holes in the image by a target detection model, and then uses laser radar depth data to make a second determination on the identification result, thereby effectively eliminating false positives caused by shadows and other non-real holes, significantly reducing the false detection rate, improving the detection accuracy, and can be deployed on an unmanned vehicle platform to realize automatic patrol.

[0006] The present application is realized by the above technical solution: a method for detecting rat holes in cities based on camera and laser radar fusion of unmanned vehicles, comprising the following steps: Step 1, using a target detection model to preliminarily identify the collected environment image to obtain a suspected rat hole candidate area; Step 2, acquiring vertical depth information of the ground candidate area by a first laser radar sensor installed on the top of the unmanned vehicle to form depth distribution data; Step 3, acquiring horizontal depth information of the wall root candidate area by a second laser radar sensor installed on the bottom of the unmanned vehicle close to the ground to form depth profile data; Step 4, calculating a depth feature score according to the depth difference between the candidate area and the surrounding background;

[0007] in The average depth around the candidate box for ground depression. The average depth within the region. The contrast threshold; Step 5: Calculate the edge contrast of the candidate region and determine whether the grayscale difference between it and the background meets the preset threshold.

[0008] in, The average intensity of edge pixels, The average intensity of the background pixels. This is the edge contrast threshold; Step 6: Determine whether the candidate region meets the criteria for a mouse hole based on the opening morphology characteristics, using the following formula;

[0009] in, The horizontal width of the candidate opening. Vertical height , The set aspect ratio threshold range; Step 7: Calculate the overall score based on the judgment results from Steps 2 to 5.

[0010] in, Scoring for deep features, Score the edge contrast. Score based on shape and proportion. The comprehensive scoring threshold, For weight parameters, if If the candidate area is identified as a mouse hole, then the candidate area is determined to be a mouse hole.

[0011] The first lidar sensor collects depth The range is from 0.1 m to 3 m, and the acquisition depth of the second lidar sensor is... The range is from 0.05 m to 1 m.

[0012] The edge contrast threshold The value ranges from 0.2 to 0.5.

[0013] The aspect ratio threshold range of the opening morphological features satisfies .

[0014] The comprehensive scoring weight parameters satisfy ,in .

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Combining visual candidates with secondary depth determination significantly suppresses false detections caused by shadows, smudges, etc.

[0016] 2. By using a dual-device layout of "upward downward view + downward near-ground view" to match the actual geometric opening direction of ground-type and wall-root-type openings respectively, the detection rate is improved.

[0017] 3. The feasibility and repeatability of the scheme are guaranteed by using calculable and reproducible experimental indicators such as planar residuals, aspect ratio, and edge contrast.

[0018] 4. Supports online threshold adaptation and result plotting, suitable for long-term, automated inspection deployment on unmanned vehicle platforms. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 False alarms that rely solely on visual detection; Figure 3 This is a schematic diagram of the horizontal hole-like shape of a mouse hole at the base of a wall; Figure 4 A schematic diagram of the morphology of a ground-dwelling rodent burrow; Figure 5 This is a schematic diagram showing the distribution of depth information of the mouse hole; Figure 6 This is a schematic diagram of the secondary determination based on depth information according to the present invention; Figure 7 This is a schematic diagram of the secondary determination based on the shape information of the opening in this invention; Figure 8 This is a schematic diagram of the secondary determination based on the edge information of the opening in this invention. Detailed Implementation

[0020] like Figure 1 The method shown is a camera and lidar fusion detection method for detecting rat holes in cities based on an unmanned vehicle, including the following steps: Step 1: Use the target detection model to perform preliminary identification on the collected environmental images to obtain candidate areas for suspected mouse holes; Step 2: Obtain data using the first lidar sensor mounted on the top of the autonomous vehicle, such as... Figure 5 The vertical depth information of the candidate ground areas shown is used to form depth distribution data for detecting ground hole types such as rat burrows. Figure 4 As shown; Step 3: Obtain horizontal depth information of the candidate area at the base of the wall using a second lidar sensor installed on the bottom of the unmanned vehicle near the ground, forming depth profile data for detecting mouse holes of the type of horizontal opening at the base of the wall, such as... Figure 3 As shown; Step 4: Calculate the depth feature score based on the depth difference between the candidate region and the surrounding background. The judgment result is visualized as follows: Figure 6 As shown;

[0021] in The average depth around the candidate box for ground depression. The average depth within the region. The contrast threshold; Step 5: Calculate the edge contrast of the candidate region and determine whether the grayscale difference between it and the background meets the preset threshold. The result of the hole edge feature determination is visualized as follows: Figure 8 As shown;

[0022] in, The average intensity of edge pixels, The average intensity of the background pixels. This is the edge contrast threshold; Step 6: Determine whether the candidate region meets the criteria for a mouse hole based on the opening morphology characteristics, using the following formula. The results of the hole shape determination are visualized as follows: Figure 7 As shown;

[0023] in, The horizontal width of the candidate opening. Vertical height , The set aspect ratio threshold range; Step 7: Calculate the overall score based on the judgment results from Steps 2 to 5.

[0024] in, Scoring for deep features, Score the edge contrast. Score based on shape and proportion. The comprehensive scoring threshold, For weight parameters, if If the candidate area is identified as a mouse hole, then the candidate area is determined to be a mouse hole.

[0025] The first lidar sensor collects depth The range is from 0.1 m to 3 m, and the acquisition depth of the second lidar sensor is... The range is from 0.05 m to 1 m.

[0026] The edge contrast threshold The value ranges from 0.2 to 0.5.

[0027] The aspect ratio threshold range of the opening morphological features satisfies .

[0028] The comprehensive scoring weight parameters satisfy ,in .

[0029] Example 1: When using existing camera-based image detection methods, such as Figure 2 As shown, the results are easily affected by light, shadows, and debris, which can influence the detection results. This invention addresses these problems using the following solution.

[0030] I. A system for detecting rat holes in cities based on camera and lidar fusion from unmanned vehicles: 1. System Composition: 1.1 Above-mounted acquisition device: Installed on the roof of the unmanned vehicle or on the support rod, it is used to acquire visible light images and corresponding depth data of the ground area, adapting to the downward concave features of the "ground-type" opening.

[0031] 1.2 Lower Acquisition Device: Installed at the bottom of the unmanned vehicle near the ground, it is used to acquire image and depth data of the wall base and near-ground facade area, adapting to the horizontal concave features of the "wall base type" opening.

[0032] 1.3 Computation and storage unit: Performs target detection, feature calculation, comprehensive judgment and result management.

[0033] 1.4 Positioning and Attitude Unit: Obtains the position and pose of the unmanned vehicle for coordinate unification and result mapping.

[0034] 1.5 Timing and Calibration Module: Enables time synchronization between the camera and LiDAR, calibration of intrinsic and extrinsic parameters, and multi-sensor coordinate system integration.

[0035] 2. Data Acquisition and Preprocessing: 2.1 Synchronous Acquisition: Image and depth / point cloud data are acquired by the upper and lower devices respectively; the two types of data are registered to a unified coordinate system based on timestamp alignment and extrinsic parameters.

[0036] 2.2 Depth Data Preprocessing: The acquired depth map is first normalized to unify the data range. Then, morphological opening operations are used to remove isolated noise points, and closing operations are used to fill local holes, thereby enhancing the continuity and recognizability of the mouse hole depression area and providing a more stable input for subsequent feature extraction.

[0037] 2.3 Image Enhancement: Brightness equalization and contrast limiting are applied to scenes with strong light and shadow to reduce the visual candidate's dependence on shadows.

[0038] 3. Candidate Region Detection (Vision-Driven): The image is input into the object detection model to output candidate bounding boxes / mask regions R (suspected mouse holes). To balance real-time performance and robustness, a single-stage detection framework can be adopted; candidates are only used for the first stage of screening, and the final decision depends on depth and geometric features.

[0039] 4. Feature Calculation and Type Determination: For each candidate region Calculate the following constraint characteristics that are strongly correlated with the physical structure: 4.1 Depth difference characteristics (significance of inward concavity) Calculate the average depth of the candidate region Average depth of its surrounding background The difference:

[0040] when At that time, it was considered that there was a significant concavity; among which This is a preset threshold.

[0041] 4.2 Opening morphology characteristics (length-to-short axis ratio)

[0042] Elliptical fitting is performed on the binary mask of the candidate region to obtain the major axis. With short axis ,definition:

[0043] The opening shape is limited to a typical range to avoid misjudgment of thin cracks or irregular shadows; This is the upper threshold value.

[0044] 4.3 Edge contrast features (light / deep boundary consistency)

[0045] The image gradient intensity and depth gradient intensity are calculated in the candidate boundary band to obtain the edge contrast index. (For example, the average grayscale difference / average depth transition magnitude between boundary pixels and the surrounding background). When At that time, it was assumed that there was a clear opening boundary.

[0046]

[0047] in For edge pixel grayscale, This represents the background grayscale of the corresponding neighborhood.

[0048] 4.4 Type Determination (Ground Type / Wall Base Type)

[0049] a. For the "ground type", use the estimated ground normal vector For reference, check whether the negative residual of the candidate region relative to the ground plane is significant (concave downward).

[0050] b. For the "wall-base type", use the wall surface normal vector. For reference, check whether there is a concave residual (horizontal concavity) in the candidate region along the horizontal direction (normal direction).

[0051] 5. Fusion Decision and Threshold Strategy

[0052] Multi-feature comprehensive scoring method:

[0053] in Depend on Normalization yielded Depend on Normalization yielded Depend on Obtained through normalization; weights .when Identify mouse holes and output their category (ground type / wall type). Threshold , , It can adaptively adjust according to ambient light, ground material and driving speed.

[0054] 6. Results Output and Mapping: The confirmed mouse hole locations are transformed from the sensor coordinate system to the unmanned vehicle coordinate system, and then combined with positioning and maps to achieve geographic coordinate mapping; the records include: spatial location, category, confidence level, timestamp and data acquisition device source (above / below), supporting subsequent management and review.

[0055] 7. Operation Mode and Robustness Measures

[0056] a. Dual-channel collaboration: The upper device prioritizes coverage of ground-type devices, while the lower device prioritizes coverage of wall-based devices; the results from both are merged and deduplicated in the computing unit.

[0057] b. Occlusion and frame loss handling: When the data quality of any channel is insufficient, reduce the weight of the corresponding channel or trigger the frame interpolation mechanism.

[0058] c. Calibration and maintenance: Periodically check the consistency between the plane fitting residual and the edge, and trigger a fast recalibration process when an anomaly occurs.

[0059] II. The specific method is as follows: Step 1: Data Collection 1.1 The driverless car is equipped with two types of sensors: a. Camera module: used to acquire visible light images of urban roads and wall base areas, with a preferred resolution of ≥1280×720 and a preferred frame rate of ≥30fps.

[0060] b. LiDAR module: Used to acquire scene depth information, including vertical depth of the ground. horizontal depth of the wall base .

[0061] • Upper lidar unit: Installed at a height of 0.8–1.2 m above the ground on the unmanned vehicle, used to detect ground depressions.

[0062] • Lower lidar unit: Installed at a height of 0.05–0.15 m above the ground, used to detect horizontal openings at the base of the wall.

[0063] Step 2: Initial Visual Screening

[0064] 2.1 Preprocessing the images acquired by the camera: a. Grayscale conversion and histogram equalization; b. Data augmentation methods include flipping, random cropping, and brightness adjustment.

[0065] 2.2 Using a pre-trained object detection model Select region set Each candidate region Corresponding confidence level .

[0066] a. Set threshold ,like If it is, then it will be directly removed.

[0067] Step 3: Secondary judgment by lidar

[0068] 3.1 Obtain the depth profile corresponding to the candidate region: a. Extract vertical depth features from candidate regions corresponding to ground depressions:

[0069] in The average depth around the candidate box for ground depression. This represents the average depth within the region.

[0070] Judgment conditions: ,in .

[0071] b. Extract horizontal depth features from the candidate region corresponding to the wall base opening:

[0072] in The average depth around the candidate frame for the wall base opening.

[0073] Judgment conditions: ,in

[0074] Step 4: Determining the Opening Morphology

[0075] 4.1 Calculate the edge contrast features of the candidate region:

[0076] in, The average grayscale value of the pixels at the region's edge. This represents the average grayscale value of the pixels within the region.

[0077] Judgment conditions: ,in .

[0078] 4.2 Combine morphological opening and closing operations to extract connectivity and remove fragmented shadows.

[0079] Step 5: Comprehensive Scoring and Final Judgment

[0080] 5.1 Setting the comprehensive scoring formula

[0081] in: For the weight parameters, satisfying

[0082] Recommended value range

[0083] 5.2 If ,in If the value is 0.8, it is judged as a mouse hole; otherwise, it is discarded.

[0084] Step 6: Output and Application of Results

[0085] 6.1 Location coordinates of the identified mouse hole The category labels are saved to the database and marked on the autonomous vehicle navigation map.

[0086] 6.2 The output results can be used for urban disease record keeping or to drive unmanned vehicles to issue alarms and conduct automatic patrols.

Claims

1. A method for detecting rat holes in cities based on camera and lidar fusion from an unmanned vehicle, characterized in that, Includes the following steps: Step 1: Use the target detection model to perform preliminary identification on the collected environmental images to obtain candidate areas for suspected mouse holes; Step 2: Obtain vertical depth information of the candidate area on the ground by using the first lidar sensor installed on the top of the unmanned vehicle to form depth distribution data; Step 3: Obtain horizontal depth information of the candidate area at the base of the wall by using a second lidar sensor installed on the bottom of the unmanned vehicle near the ground, and form depth profile data; Step 4: Calculate the depth feature score based on the depth difference between the candidate region and the surrounding background; in The average depth around the candidate box for ground depression. The average depth within the region. The contrast threshold; Step 5: Calculate the edge contrast of the candidate region and determine whether the grayscale difference between it and the background meets the preset threshold. in, The average intensity of edge pixels, The average intensity of the background pixels. This is the edge contrast threshold; Step 6: Determine whether the candidate region meets the criteria for a mouse hole based on the opening morphology characteristics, using the formula... in, The horizontal width of the candidate opening. Vertical height , The set aspect ratio threshold range; Step 7: Calculate the overall score based on the judgment results from Steps 2 to 5. in, Scoring for deep features, Score the edge contrast. Score based on shape and proportion. The comprehensive scoring threshold, For weight parameters, if If the candidate area is identified as a mouse hole, then the candidate area is determined to be a mouse hole.

2. The method for detecting rat holes in cities based on camera and lidar fusion using an unmanned vehicle, as described in claim 1, is characterized in that... The first lidar sensor collects depth The range is from 0.1 m to 3 m, and the acquisition depth of the second lidar sensor is... The range is from 0.05 m to 1 m.

3. The method for detecting rat holes in cities based on camera and lidar fusion using an unmanned vehicle, as described in claim 1, is characterized in that... The edge contrast threshold The value ranges from 0.2 to 0.

5.

4. The method for detecting rat holes in cities based on camera and lidar fusion using an unmanned vehicle, as described in claim 1, is characterized in that... The aspect ratio threshold range of the opening morphological features satisfies .

5. The method for detecting rat holes in cities based on camera and lidar fusion using an unmanned vehicle, as described in claim 1, is characterized in that... The comprehensive scoring weight parameters satisfy ,in .

Images