An irregular hole recognition and crossing method based on deep estimation large model and physical depth fusion

Abstract

The application discloses an irregular hole recognition and crossing method based on deep estimation large model and physical depth fusion, relates to the technical field of unmanned aerial vehicle or robot control, maps the geometric size constraint of an unmanned aerial vehicle / robot into a morphological structure element, realizes structure passability analysis of irregular holes, and establishes a safety score model through multi-distance field fusion, so that optimal crossing point decision in a complex environment is realized.

Description

Technical Field

[0001] This invention relates to the field of drone or robot control technology, specifically to a method for identifying and traversing irregular holes based on the fusion of a large depth estimation model and physical depth. Background Technology

[0002] In the field of autonomous navigation for drones and various mobile robots, obstacle avoidance is a core component. Traditional obstacle avoidance solutions typically rely on LiDAR or binocular stereo vision to generate sparse point clouds. However, when obstacles are extremely irregular in shape (such as damaged walls, fences, rock holes, or complex industrial structures), the point cloud data is often sparse and noisy, making it difficult to accurately extract boundary information of passable areas. On the other hand, monocular depth estimation models can obtain rich semantic information and relative depth distribution from RGB images, but lack absolute scale references, resulting in scale drift in the generated depth maps, making them unsuitable for direct and accurate obstacle avoidance. In the aforementioned scenarios, inaccurate identification or size estimation deviations can easily lead to collisions between drones or robots, posing a significant risk. Therefore, achieving accurate identification and safe passage through various irregular hole obstacles is of great importance for drone exploration in complex environments such as search and rescue operations in ruins and mines. Summary of the Invention

[0003] To overcome the shortcomings of the above technologies, this invention provides a hole identification and crossing method that establishes a safety scoring model through multi-distance field fusion, thereby enabling the optimal crossing point decision for drones or robots in complex environments.

[0004] The technical solution adopted by this invention to overcome its technical problems is:

[0005] An irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth includes:

[0006] S1. Acquire synchronized physical depth images by mounting a forward-looking depth camera on a drone or robot. and RGB images ;

[0007] S2. Transfer the physical depth image Convert to 3D point cloud Using 3D point clouds Obtain the unit normal vector of the plane , projection point set ;

[0008] S3. Using RGB images Obtain a normalized dense depth image ;

[0009] S4. Based on the projection point set and normalized dense depth images get The outline of the first hole, the first The outline of the hole is , ;

[0010] S5. Obtain the projected size in the image based on the physical dimensions of the drone or robot;

[0011] S6. Using the projection size to... The outline of the hole Perform corrosion operation to obtain a map of passable locations. ;

[0012] S7. Utilize the accessible location map Get the first The outline of the hole Contour score ;

[0013] S8. Determine the optimal crossing point based on the outline scores of all holes. ;

[0014] S9. Drones or robots use planar unit normals. After attitude alignment, the drone or robot controller determines the optimal crossing point. Control drones or robots to traverse.

[0015] Furthermore, in step S1, the depth camera is a RealSense D435i depth camera, providing a physical depth image. and RGB images All resolutions are 640×480.

[0016] Furthermore, step S2 includes the following steps:

[0017] S2-1. Transfer the physical depth image The physical depth image is obtained by sampling at intervals of 8 pixels in the width direction and 6 pixels in the height direction. ;

[0018] S2-2. Sampled physical depth image Filtering pixels with depth values ​​between 0 meters and 10 meters yields a filtered physical depth image. ;

[0019] S2-3. Use the intrinsic parameters of the depth camera to filter the physical depth image. Convert to 3D point cloud ;

[0020] S2-4. 3D point cloud A generalized plane model with thickness constraints is obtained using the RANSAC plane detection algorithm from the PCL open-source library. and 3D point sets on a generalized plane ,in , , All are plane equation parameters. For displacement terms, 3D point cloud The Middle The X-axis coordinates of the points 3D point cloud The Middle The Y-axis coordinates of the points 3D point cloud The Middle Z-axis coordinates of the points , 3D point cloud The number of midpoints The permissible perturbation thickness threshold for the plane. The value is 0.5;

[0021] S2-5. Through formula Calculate the unit normal vector of the plane In the formula, It is a plane normal vector;

[0022] S2-6. Using depth camera intrinsics to extract 3D point sets Projecting onto the image coordinate system yields the set of projected points on the image. ;

[0023] S2-7. The unit normal vector of the plane Normal vector to the ground Perform a dot product operation to obtain the similarity between the normal vectors of the two planes. If the similarity is greater than a threshold... The generalized plane is the ground, and the points on this generalized plane are transferred from the 3D point cloud. After removing a value from the list, return to step S2-4. If the similarity is less than or equal to the threshold... Then proceed to step S3, threshold. The value is 0.8.

[0024] Furthermore, step S3 includes the following steps:

[0025] S3-1. Convert the RGB image The input is fed into a pre-trained Distill-Any-Depth-Multi-Teacher-Small model, and the output is a dense depth image;

[0026] S3-2. Normalize the dense depth image to obtain a normalized dense depth image. .

[0027] Furthermore, step S4 includes the following steps:

[0028] S4-1. Calculate the projection point set The average of the X-axis coordinates of all points is used as the X-axis coordinate of the centroid to calculate the projection point set. The average of the Y-axis coordinates of all points is taken as the Y-axis coordinate of the centroid;

[0029] S4-2. Calculate the projection point set The distances from all points to the centroid are calculated, and these distances are sorted in ascending order. The top 90% of these distances are then filtered, and the minimum X-axis coordinate value of all points corresponding to the filtered distances is selected. Select the maximum X-axis coordinate value of all points corresponding to the filtered distance. Select the minimum Y-axis coordinate value of all points corresponding to the filtered distances. Select the maximum Y-axis coordinate value of all points corresponding to the filtered distance. coordinate points As the top left corner point of the rectangle, with coordinates... Use the bottom right corner point to construct the rectangle. ;

[0030] S4-3. Through formula Calculate the segmentation threshold In the formula, For the projection point set medium rectangle Dense depth image within the range The mean of the depth values ​​of all pixels, For the projection point set medium rectangle Dense depth image within the range The variance of the depth values ​​of all pixels, For safety offset, ;

[0031] S4-4. If dense depth image The Middle The depth value of each pixel is greater than the segmentation threshold. If the depth value of that pixel is 1, then the depth value of that pixel is set to 1. (This is for dense depth images.) The Middle The depth value of each pixel is less than or equal to the segmentation threshold. Then set the depth value of that pixel to 0 to obtain the segmented dense depth image. ;

[0032] S4-5. Segmented dense depth image Using the closing and opening operations in OpenCV, we obtain a denoised dense depth image. For the denoised dense depth image Edge detection is performed using the findContours function in OpenCV to obtain... The outline of the first hole, the first The outline of the hole is .

[0033] Furthermore, step S5 includes the following steps:

[0034] S5-1. Obtain the width of the physical dimensions of the drone or robot. and physical dimensions of height ;

[0035] S5-2. and Projecting intrinsic parameters from a depth camera onto a dense depth image The width of the projection dimension in the image is obtained as The height of the projected size is ,in For safety margin, .

[0036] Furthermore, step S6 includes the following steps:

[0037] S6-1 uses the approxPolyDP function in OpenCV to process the... The outline of the hole Perform polygon approximation sampling, calculate the maximum bounding rectangle for points in the sampled contour, and obtain the bounding rectangle. ;

[0038] S6-2. Create an image with an outer rectangle. Images of the same size ,image The pixel value of each pixel in the image is 0. The corresponding number in the middle The outline of the hole The pixel values ​​of each pixel in the region are filled with 255 to obtain a binary image. ;

[0039] S6-3. If the circumscribed rectangle The width is less than Or an external rectangle The height is less than This means that you cannot pass through the hole. The outline score of each hole Set to 0, when the outer rectangle is... Width greater than or equal to Or an external rectangle The height is greater than or equal to This means that the passage can pass through the hole and step S6-4 can be executed;

[0040] S6-4. Using the erode function in OpenCV to process binary images The applied corrosion core size is The corrosion operation yields a map of accessible locations. .

[0041] Furthermore, step S7 includes the following steps:

[0042] S7-1. For images Use the distanceTransform function in OpenCV to obtain the image. No. Distance transformation of pixels ;

[0043] S7-2. Map of accessible locations Use the distanceTransform function in OpenCV to obtain a map of passable locations. No. Distance transformation of pixels ;

[0044] S7-3. Calculate the first using the area-weighted average method. The outline of the hole The centroid of the binary image is calculated. No. The pixel and the The outline of the hole Euclidean distance between the centers of mass ;

[0045] S7-4. Through formula Calculate the first Optimal crossing position score for each pixel In the formula , , All are weights;

[0046] S7-5. Select the maximum value of the score for the optimal crossing position of all pixels as the first... The outline of the hole Contour score .

[0047] Preferably, in step S7-4 The value is 0.5. The value is 0.5. The value is 0.1.

[0048] Furthermore, step S8 includes the following steps:

[0049] S8-1. Select the pixel corresponding to the optimal crossing position score corresponding to the maximum value of the contour score of all holes as the best crossing pixel;

[0050] S8-2. Use the depth camera intrinsics to transform the coordinates of the optimal crossing pixel to the physical coordinate system of the drone or robot to obtain the optimal crossing point. .

[0051] The beneficial effects of this invention are:

[0052] (1) It can accurately identify hole areas in obstacle scenarios with various irregular shapes, damage or complex structures, realize the accessibility modeling of irregular hole structures, and provide a reliable foundation for subsequent crossing planning.

[0053] (2) By combining the fusion method of large depth estimation model and physical depth, the unified scale modeling is used to eliminate the risk of monocular depth drift and significantly improve the obstacle avoidance accuracy and success rate in complex environments.

[0054] (3) The proposed algorithm is applicable to various indoor and outdoor environments and has strong adaptability to different types of UAVs and mobile robots. It can be widely applied to actual autonomous obstacle avoidance tasks.

[0055] (4) Under the premise of meeting the real-time requirements, the present invention improves the efficiency and safety of obstacle recognition and hole crossing, and provides a feasible solution for real-time obstacle avoidance. Attached Figure Description

[0056] Figure 1 This is a flowchart of the method of the present invention;

[0057] Figure 2 Physical depth image;

[0058] Figure 3 Dense depth maps obtained from large model inference;

[0059] Figure 4 This represents the binary segmentation map and the optimal crossing point for the UAV. Detailed Implementation

[0060] The following is in conjunction with the appendix Figure 1To be continued Figure 4 The present invention will be further described below.

[0061] As attached Figure 1 As shown, a method for identifying and traversing irregular holes based on the fusion of a large depth estimation model and physical depth includes:

[0062] S1. Acquire synchronized physical depth images by mounting a forward-looking depth camera on a drone or robot. and RGB images .

[0063] S2. Transfer the physical depth image Convert to 3D point cloud Using 3D point clouds Obtain the unit normal vector of the plane , projection point set .

[0064] S3. Using RGB images Obtain a normalized dense depth image .

[0065] S4. Based on the projection point set and normalized dense depth images get The outline of the first hole, the first The outline of the hole is , .

[0066] S5. Obtain the projected size in the image based on the physical dimensions of the drone or robot.

[0067] S6. Using the projection size to... The outline of the hole Perform corrosion operation to obtain a map of passable locations. .

[0068] S7. Utilize the accessible location map Get the first The outline of the hole Contour score .

[0069] S8. Determine the optimal crossing point based on the outline scores of all holes. .

[0070] S9. Drones or robots use planar unit normals. After attitude alignment, the drone or robot controller determines the optimal crossing point. Control drones or robots to traverse.

[0071] A unified spatial modeling method based on the fusion of physical depth scale benchmark and large depth estimation model is constructed. The geometric constraints of UAV / robot are mapped into morphological structural elements to realize the structural accessibility analysis of irregular holes. A safety scoring model is established through multi-distance field fusion, thereby realizing the optimal crossing point decision in complex environments.

[0072] As attached Figure 2 As shown, in step S1, the depth camera is a RealSense D435i depth camera, and the physical depth image... and RGB images All resolutions are 640×480.

[0073] In one embodiment of the present invention, step S2 includes the following steps:

[0074] S2-1. Transfer the physical depth image The physical depth image is obtained by sampling at intervals of 8 pixels in the width direction and 6 pixels in the height direction. .

[0075] S2-2. Sampled physical depth image Filtering pixels with depth values ​​between 0 meters and 10 meters yields a filtered physical depth image. .

[0076] S2-3. Use the intrinsic parameters of the depth camera to filter the physical depth image. Convert to 3D point cloud .

[0077] S2-4. 3D point cloud A generalized plane model with thickness constraints is obtained using the RANSAC plane detection algorithm from the PCL open-source library. and 3D point sets on a generalized plane ,in , , All are plane equation parameters. For displacement terms, 3D point cloud The Middle The X-axis coordinates of the points 3D point cloud The Middle The Y-axis coordinates of the points 3D point cloud The Middle Z-axis coordinates of the points , 3D point cloud The number of midpoints The permissible perturbation thickness threshold for the plane. The value is set to 0.5 to accommodate uneven mine walls or damaged structures.

[0078] S2-5. Through formula Calculate the unit normal vector of the plane In the formula, It is the plane normal vector.

[0079] S2-6. Using depth camera intrinsics to extract 3D point sets Projecting onto the image coordinate system yields the set of projected points on the image. .

[0080] S2-7. The unit normal vector of the plane Normal vector to the ground Perform a dot product operation to obtain the similarity between the normal vectors of the two planes. If the similarity is greater than a threshold... The generalized plane is the ground, and the points on this generalized plane are transferred from the 3D point cloud. After removing a value from the list, return to step S2-4. If the similarity is less than or equal to the threshold... Then proceed to step S3, threshold. The value is 0.8.

[0081] In one embodiment of the present invention, step S3 includes the following steps:

[0082] S3-1. Convert the RGB image The input is fed into the pre-trained Distill-Any-Depth-Multi-Teacher-Small model, and the output is shown in the attached figure. Figure 3 The image shown is a dense depth image. The Distill-Any-Depth-Multi-Teacher-Small model is an existing publicly available model, which can be found at (https: / / github.com / Westlake-AGI-Lab / Distill-Any-Depth) for details.

[0083] S3-2. Normalize the dense depth image to obtain a normalized dense depth image. It compensates for the shortcomings of physical depth sensors in terms of sparse or missing point clouds at the edges of holes and in small structures, providing rich depth details for fine segmentation of holes.

[0084] In one embodiment of the present invention, step S4 includes the following steps:

[0085] S4-1. Calculate the projection point set The average of the X-axis coordinates of all points is used as the X-axis coordinate of the centroid to calculate the projection point set. The average of the Y-axis coordinates of all points is taken as the Y-axis coordinate of the centroid.

[0086] S4-2. Calculate the projection point set The distances from all points to the centroid are calculated, and these distances are sorted in ascending order. The top 90% of these distances are then filtered, and the minimum X-axis coordinate value of all points corresponding to the filtered distances is selected. Select the maximum X-axis coordinate value of all points corresponding to the filtered distance. Select the minimum Y-axis coordinate value of all points corresponding to the filtered distances. Select the maximum Y-axis coordinate value of all points corresponding to the filtered distance. coordinate points As the top left corner point of the rectangle, with coordinates... Use the bottom right corner point to construct the rectangle. .

[0087] S4-3. Through formula Calculate the segmentation threshold In the formula, For the projection point set medium rectangle Dense depth image within the range The mean of the depth values ​​of all pixels, For the projection point set medium rectangle Dense depth image within the range The variance of the depth values ​​of all pixels, For safety offset, .

[0088] S4-4. If dense depth image The Middle The depth value of each pixel is greater than the segmentation threshold. If the depth value of that pixel is 1, then the depth value of that pixel is set to 1. (This is for dense depth images.) The Middle The depth value of each pixel is less than or equal to the segmentation threshold. Then set the depth value of that pixel to 0 to obtain the segmented dense depth image. .

[0089] S4-5. Segmented dense depth image Using the closing and opening operations in OpenCV, we obtain a denoised dense depth image. For the denoised dense depth image Edge detection is performed using the findContours function in OpenCV to obtain... The outline of the first hole, the first The outline of the hole is .

[0090] In one embodiment of the present invention, step S5 includes the following steps:

[0091] S5-1. Obtain the width of the physical dimensions of the drone or robot. and physical dimensions of height .

[0092] S5-2. and Projecting intrinsic parameters from a depth camera onto a dense depth image The width of the projection dimension in the image is obtained as The height of the projected size is ,in For safety margin, .

[0093] In one embodiment of the present invention, step S6 includes the following steps:

[0094] S6-1 uses the approxPolyDP function in OpenCV to process the... The outline of the hole Perform polygon approximation sampling, calculate the maximum bounding rectangle for points in the sampled contour, and obtain the bounding rectangle. .

[0095] S6-2. Create an image with an outer rectangle. Images of the same size ,image The pixel value of each pixel in the image is 0. The corresponding number in the middle The outline of the hole The pixel values ​​of each pixel in the region are filled with 255 to obtain a binary image. .

[0096] S6-3. If the circumscribed rectangle The width is less than Or an external rectangle The height is less than This means that you cannot pass through the hole. The outline score of each hole Set to 0, when the outer rectangle is... Width greater than or equal to Or an external rectangle The height is greater than or equal to This means that the hole can be passed through and step S6-4 can be executed.

[0097] S6-4. Using the erode function in OpenCV to process binary images The applied corrosion core size is The corrosion operation yields a map of accessible locations. .

[0098] This step innovatively transforms the geometrical feasibility determination problem into a spatial feasible region search problem after morphological erosion, and maps spatial structural constraints to image computation.

[0099] In one embodiment of the present invention, step S7 includes the following steps:

[0100] S7-1. For images Use the distanceTransform function in OpenCV to obtain the image. No. Distance transformation of pixels Distance transformation An effective measure of the safety of a hole boundary is a value that indicates the greater the distance from the boundary, and thus the safer the hole.

[0101] S7-2. Map of accessible locations Use the distanceTransform function in OpenCV to obtain a map of passable locations. No. Distance transformation of pixels Distance transformation The stability of passage was measured, taking into account external factors (such as wind) on the stability of the passage while ensuring passage.

[0102] S7-3. Calculate the first using the area-weighted average method. The outline of the hole The centroid of the binary image is calculated. No. The pixel and the The outline of the hole Euclidean distance between the centers of mass .

[0103] S7-4. Through formula Calculate the first Optimal crossing position score for each pixel In the formula , , All are weights.

[0104] S7-5. Select the maximum value of the score for the optimal crossing position of all pixels as the first... The outline of the hole Contour score .

[0105] This step constructs a unified risk assessment model that integrates multiple physical distance fields. It unifies boundary security, traffic stability, and geometric balance into a single evaluation space.

[0106] In this embodiment, preferably, in step S7-4 The value is 0.5. The value is 0.5. The value is 0.1.

[0107] As attached Figure 4 As shown, in one embodiment of the present invention, step S8 includes the following steps:

[0108] S8-1. Select the pixel corresponding to the optimal crossing position score corresponding to the maximum contour score of all holes as the best crossing pixel.

[0109] S8-2. Use the depth camera intrinsics to transform the coordinates of the optimal crossing pixel to the physical coordinate system of the drone or robot to obtain the optimal crossing point. .

[0110] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An irregular hole recognition and crossing method based on depth estimation large model and physical depth fusion, characterized in that, include: S1. Acquire synchronized physical depth images by mounting a front-facing depth camera on a drone or robot and RGB images ; S2. convert the physical depth image to a three-dimensional point cloud , utilize the three-dimensional point cloud to obtain a planar unit normal vector , project the point set ; S3. Utilizing RGB images Obtaining normalized dense depth images ; S4. According to the projection point set and the normalized dense depth image get the contour of the first hole, the contour of the first hole is , , ; S5. Obtain the projected size in the image based on the physical dimensions of the drone or robot; S6. Utilizing the projected size to profile the holes of the first corrosion operation to obtain a passable position map ;​ S7. Utilizing the passable location map Obtaining a profile of the first hole Obtaining a profile of the second hole ; S8. Obtain the best crossing point from the profile scores of all holes ; S9. Drones or robots use planar unit normals. After attitude alignment, the drone or robot controller determines the optimal crossing point. Control drones or robots to traverse; Step S6 includes the following steps: S6-1 uses the approxPolyDP function in OpenCV to process the... The outline of the hole Perform polygon approximation sampling, calculate the maximum bounding rectangle for points in the sampled contour, and obtain the bounding rectangle. ; S6-2. Create an image with an outer rectangle. Images of the same size ,image The pixel value of each pixel in the image is 0. The corresponding number in the middle The outline of the hole The pixel values ​​of each pixel in the region are filled with 255 to obtain a binary image. ; S6-3. If the circumscribed rectangle The width is less than Or an external rectangle The height is less than This means that you cannot pass through the hole. The outline score of each hole Set to 0, when the outer rectangle is... Width greater than or equal to Or an external rectangle The height is greater than or equal to This means that the passage can pass through the hole and step S6-4 can be executed; S6-4. Using the erode function in OpenCV to process binary images The applied corrosion core size is The corrosion operation yields a map of accessible locations. ; Step S7 includes the following steps: S7-1. For images Use the distanceTransform function in OpenCV to obtain the image. No. Distance transformation of pixels ; S7-2. Map of accessible locations Use the distanceTransform function in OpenCV to obtain a map of passable locations. No. Distance transformation of pixels ; S7-3. Calculate the first using the area-weighted average method. The outline of the hole The centroid of the binary image is calculated. No. The pixel and the The outline of the hole Euclidean distance between the centers of mass ; S7-4. Through formula Calculate the first Optimal crossing position score for each pixel In the formula , , All are weights; S7-5. Select the maximum value of the score for the optimal crossing position of all pixels as the first... The outline of the hole Contour score .

2. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that: In step S1, the depth camera is a RealSense D435i depth camera, and the physical depth image is... and RGB images All resolutions are 640×480.

3. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that, Step S2 includes the following steps: S2-1. Transfer the physical depth image The physical depth image is obtained by sampling at intervals of 8 pixels in the width direction and 6 pixels in the height direction. ; S2-2. Sampled physical depth image Filtering pixels with depth values ​​between 0 meters and 10 meters yields a filtered physical depth image. ; S2-3. Use the intrinsic parameters of the depth camera to filter the physical depth image. Convert to 3D point cloud ; S2-4. 3D point cloud A generalized plane model with thickness constraints is obtained using the RANSAC plane detection algorithm from the PCL open-source library. and 3D point sets on a generalized plane ,in , , All are plane equation parameters. For displacement terms, 3D point cloud The Middle The X-axis coordinates of the points 3D point cloud The Middle The Y-axis coordinates of the points 3D point cloud The Middle Z-axis coordinates of the points , 3D point cloud The number of midpoints The permissible perturbation thickness threshold for the plane. The value is 0.5; S2-5. Through formula Calculate the unit normal vector of the plane In the formula, It is a plane normal vector; S2-6. Using depth camera intrinsics to extract 3D point sets Projecting onto the image coordinate system yields the set of projected points on the image. ; S2-7. The unit normal vector of the plane Normal vector to the ground Perform a dot product operation to obtain the similarity between the normal vectors of the two planes. If the similarity is greater than a threshold... The generalized plane is the ground, and the points on this generalized plane are transferred from the 3D point cloud. After removing a value from the list, return to step S2-4. If the similarity is less than or equal to the threshold... Then proceed to step S3, threshold. The value is 0.

8.

4. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that, Step S3 includes the following steps: S3-1. Convert the RGB image The input is fed into a pre-trained Distill-Any-Depth-Multi-Teacher-Small model, and the output is a dense depth image; S3-2. Normalize the dense depth image to obtain a normalized dense depth image. .

5. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that, Step S4 includes the following steps: S4-1. Calculate the projection point set The average of the X-axis coordinates of all points is used as the X-axis coordinate of the centroid to calculate the projection point set. The average of the Y-axis coordinates of all points is taken as the Y-axis coordinate of the centroid; S4-2. Calculate the projection point set The distances from all points to the centroid are calculated, and these distances are sorted in ascending order. The top 90% of these distances are then filtered, and the minimum X-axis coordinate value of all points corresponding to the filtered distances is selected. Select the maximum X-axis coordinate value of all points corresponding to the filtered distance. Select the minimum Y-axis coordinate value of all points corresponding to the filtered distances. Select the maximum Y-axis coordinate value of all points corresponding to the filtered distance. coordinate points As the top left corner point of the rectangle, with coordinates... Use the bottom right corner point to construct the rectangle. ; S4-3. Through formula Calculate the segmentation threshold In the formula, For the projection point set medium rectangle Dense depth image within the range The mean of the depth values ​​of all pixels, For the projection point set medium rectangle Dense depth image within the range The variance of the depth values ​​of all pixels, For safety offset, ; S4-4. If dense depth image The Middle The depth value of each pixel is greater than the segmentation threshold. If the depth value of that pixel is 1, then the depth value of that pixel is set to 1. (This is for dense depth images.) The Middle The depth value of each pixel is less than or equal to the segmentation threshold. Then set the depth value of that pixel to 0 to obtain the segmented dense depth image. ; S4-5. Segmented dense depth image Using the closing and opening operations in OpenCV, we obtain a denoised dense depth image. For the denoised dense depth image Edge detection is performed using the findContours function in OpenCV to obtain... The outline of the first hole, the first The outline of the hole is .

6. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that, Step S5 includes the following steps: S5-1. Obtain the width of the physical dimensions of the drone or robot. and physical dimensions of height ; S5-2. and Projecting intrinsic parameters from a depth camera onto a dense depth image The width of the projection dimension in the image is obtained as The height of the projected size is ,in For safety margin, .

7. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that: In step S7-4 The value is 0.

5. The value is 0.

5. The value is 0.

1.

8. The irregular hole identification and traversal method based on the fusion of a large depth estimation model and physical depth as described in claim 1, characterized in that, Step S8 includes the following steps: S8-1. Select the pixel corresponding to the optimal crossing position score corresponding to the maximum value of the contour score of all holes as the best crossing pixel; S8-2. Use the depth camera intrinsics to transform the coordinates of the optimal crossing pixel to the physical coordinate system of the drone or robot to obtain the optimal crossing point. .

Images