A mapping and path planning method and device based on dynamic removal, equipment, medium and product

Abstract

The application discloses a mapping and path planning method and device based on dynamic removal, equipment, medium and product, relates to the field of computer vision and robot navigation, and the method comprises the following steps: acquiring an RGB image of a target environment where a robot is currently located, identifying the category of objects in the image by using a deep learning algorithm, setting a dynamic degree value of corresponding pixel points, and forming a category-dynamic degree marked image; acquiring a depth image by using a Depth-Anything algorithm; combining the RGB image, the category-dynamic degree marked image and the depth image to construct a 3D map; identifying and removing dynamic points in the 3D map to generate a global static environment map, and planning an optimal path by using a path planning algorithm. The application realizes effective identification and removal of dynamic objects, constructs a high-precision global static environment map, provides an accurate navigation basis for the robot, and improves the efficiency and accuracy of path planning.

Description

Technical Field

[0001] This application relates to the fields of computer vision and robot navigation, and in particular to a method, apparatus, device, medium and product for mapping and path planning based on dynamic removal. Background Technology

[0002] With the rapid development of technology, miniature quadruped robots have shown great application potential in fields such as environmental monitoring, unmanned reconnaissance, and confined space operations. These robots, with their flexible movement capabilities and unique structural features, demonstrate unique advantages in complex environments.

[0003] However, current research on environmental perception and path planning for micro-quadruped robots, both domestically and internationally, is relatively limited. Existing research on micro-quadruped robots focuses more on improving their own motion stability, but their ability to plan motion based on the environment remains weak, making it difficult to complete comprehensive exploration tasks through autonomous path planning. Traditional control methods typically rely heavily on body dynamics modeling, resulting in low system robustness and difficulty adapting to the high-dynamic motion requirements of unknown and complex environments. Furthermore, SLAM (Simultaneous Localization and Mapping) in confined spaces still faces many challenges. Most existing work assumes a static environment, ignoring interference from dynamic objects, which significantly affects localization accuracy in real-world environments, especially when moving objects are present, and can even lead to system failure.

[0004] In existing technologies, micro-sized quadruped robots have significant deficiencies in perception and path planning capabilities in dynamic environments. This problem limits the autonomous navigation capabilities of micro-sized quadruped robots in complex and unstructured environments, and has become a key issue that urgently needs to be addressed. Summary of the Invention

[0005] The purpose of this application is to provide a mapping and path planning method, apparatus, device, medium and product based on dynamic removal, which can remove dynamic objects, construct accurate static environment maps, and achieve optimal path planning in complex unstructured environments.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a mapping and path planning method based on dynamic removal, including:

[0008] Acquire an RGB image of the target environment; the target environment is the environment in which the robot is located at the current moment.

[0009] Based on RGB images, a deep learning algorithm is used to determine the category of each object in the RGB image; and the dynamic level value of the corresponding pixel in the RGB image is set according to the category of each object to obtain the category-dynamic level label image at the current time; the categories of objects include humans, animals, vehicles and everyday items;

[0010] Based on the RGB image, the Depth-Anything algorithm is used to obtain the depth image;

[0011] Based on RGB images, category-dynamic degree labeled images, and depth images, a 3D map is constructed by converting each pixel into point cloud data in three-dimensional space using a visual odometry method.

[0012] Based on the dynamic value of each point cloud data in the 3D map, point cloud data that are dynamic points in the 3D map are identified and removed to obtain a global static environment map.

[0013] Based on a global static environment map, a path planning algorithm is used to obtain the optimal planned path for the robot in the target environment.

[0014] Secondly, this application provides a mapping and path planning apparatus based on dynamic removal, comprising:

[0015] An RGB image acquisition module is used to acquire an RGB image of the target environment, which is the environment in which the robot is located at the current moment.

[0016] The object category recognition and dynamic level labeling module is used to determine the category of each object in an RGB image based on a deep learning algorithm; and to set the dynamic level value of the corresponding pixel in the RGB image according to the category of each object, so as to obtain the category-dynamic level labeling image at the current moment; the object categories include humans, animals, vehicles and everyday items;

[0017] The depth image acquisition module is used to obtain a depth image based on an RGB image using the Depth-Anything algorithm;

[0018] The 3D map building module is used to convert each pixel into point cloud data in three-dimensional space based on RGB images, category-dynamic degree labeled images, and depth images, and to build a 3D map using visual odometry methods.

[0019] The dynamic point removal module is used to identify and remove point cloud data that are dynamic points in the 3D map based on the dynamic degree value of each point cloud data in the 3D map, so as to obtain a global static environment map.

[0020] The path planning module is used to obtain the optimal planned path for the robot in the target environment based on a global static environment map and a path planning algorithm.

[0021] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dynamic removal-based mapping and path planning method described in any one of the above.

[0022] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic removal-based mapping and path planning method described above.

[0023] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the dynamic removal-based mapping and path planning method described above.

[0024] According to the specific embodiments provided in this application, this application has the following technical effects:

[0025] This application provides a method, apparatus, device, medium, and product for mapping and path planning based on dynamic removal. By acquiring RGB images of the target environment and using deep learning algorithms to determine the categories of objects in the images, the robot can accurately identify different objects in the surrounding environment, laying the foundation for subsequent dynamic degree assessment. Simultaneously, dynamic degree values ​​are set for corresponding pixels according to object categories to obtain category-dynamic degree labeled images, achieving preliminary differentiation of dynamic elements in the environment. A depth image is acquired using the Depth-Anything algorithm, and combined with the RGB image, category-dynamic degree labeled image, and depth image, a 3D map is constructed using visual odometry, solving the problem of mapping from two-dimensional images to three-dimensional space. This approach not only provides 3D structural information of the environment but also preserves the dynamic range of objects, enabling subsequent dynamic point removal. By identifying and removing dynamic points based on the dynamic range value of each point cloud in the 3D map, it addresses the impact of dynamic elements on map accuracy, ensuring that the final generated global static environment map contains only static elements, thus improving the map's accuracy and reliability. Based on the global static environment map, a path planning algorithm is employed to obtain the optimal planned path for the robot in the target environment, achieving high efficiency and accuracy in robot navigation. It fully utilizes the information from the static map, providing the robot with a clear and unobstructed navigation path. This effectively solves the challenges posed by dynamic environments to robot mapping and path planning, achieving the construction of a high-precision global static environment map and efficient path planning. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a mapping and path planning method based on dynamic removal, provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram of the functional modules of a mapping and path planning device based on dynamic removal provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] In one exemplary embodiment, such as Figure 1 As shown, a mapping and path planning method based on dynamic removal is provided, including the following steps 101 to 106. Wherein:

[0033] Step 101: Obtain the RGB image of the target environment; the target environment is the environment in which the robot is located at the current moment.

[0034] Step 102: Based on the RGB image, a deep learning algorithm is used to determine the category of each object in the RGB image; and the dynamic level value of the corresponding pixel in the RGB image is set according to the category of each object to obtain the category-dynamic level label image at the current moment; the categories of the objects include humans, animals, vehicles and everyday items.

[0035] Step 103: Based on the RGB image, the Depth-Anything algorithm is used to obtain the depth image.

[0036] Step 104: Based on the RGB image, category-dynamic degree labeled image, and depth image, convert each pixel into point cloud data in three-dimensional space using the visual odometry method to construct a 3D map.

[0037] Step 105: Based on the dynamic value of each point cloud data in the 3D map, identify and remove point cloud data that are dynamic points in the 3D map to obtain a global static environment map.

[0038] Step 106: Based on the global static environment map, a path planning algorithm is used to obtain the optimal planned path for the robot in the target environment.

[0039] By implementing steps 101 to 106 above, this application can significantly improve the accuracy of environmental modeling and the efficiency of path planning, especially in dynamic environments. Through the organic combination of deep learning and visual odometry, it can perceive environmental changes in real time, intelligently remove dynamic interference, and provide the robot with a clear and accurate environmental model, thereby supporting its efficient and reliable navigation and path planning.

[0040] In another exemplary embodiment of this application, step 102 specifically includes:

[0041] An RGB image is input into an object detection and segmentation model to perform object category detection and instance segmentation on the RGB image at the current time, thereby obtaining the object category in the RGB image. The object detection and segmentation model is obtained by iteratively training the YOLOv8 network based on a sample training set. The sample training set includes historical RGB images with object categories labeled.

[0042] As an optional implementation method, the training process of the object detection and segmentation model based on the YOLOv8 network can be divided into the following specific steps:

[0043] 1. Data Preparation: Prepare a labeled training set, typically including images and labels. For object detection, the labels include the category and bounding box of each object. For segmentation tasks, the labels also need to include the segmentation mask for each object, usually a binary image or contour information. Datasets can use standard datasets such as COCO (Common Objects in Context) or VOC (Visual Object Classes), or a custom dataset. This application uses the COCO dataset.

[0044] 2. Environment configuration: To install the environment required for YOLOv8, you first need to install Python. In this application, we choose Python 3.8. Then install the PyTorch and Ultralytics libraries.

[0045] 3. Model Selection and Initialization: The YOLOv8 open-source code provides pre-trained models of different sizes. A suitable model can be selected based on the computational resources available for the task. YOLOv8s is chosen in this application. The pre-trained model is loaded using the following code:

[0046] a)fromultralytics importYOLO.

[0047] b) model=YOLO('yolov8s.pt').

[0048] 4. Data Augmentation and Loading: YOLOv8 automatically handles data loading and augmentation through the ultralytics library. Common data augmentation techniques include random cropping, scaling, flipping, and color adjustment.

[0049] 5. Training: Use the following command to start model training:

[0050] a)yolo task=detect mode=train model=yolov8n.pt data=your_dataset.yaml epochs=50imgsz=640.

[0051] In this configuration, `task=detect` indicates the object detection task, `mode=train` indicates the training mode, `model=yolov8s.pt` specifies the use of a pre-trained YOLOv8 model, `data=your_dataset.yaml` is the dataset configuration file, `epochs=50` indicates 50 training epochs, and `imgsz=640` indicates the size of the input image. During training, the model will automatically be evaluated on the validation set and output accuracy metrics such as MAP (Mean Average Precision), Recall, and Precision.

[0052] 6. Evaluation and Validation: During training, YOLOv8 periodically evaluates the model's performance on the validation set.

[0053] 7. Model Saving and Inference: After training, YOLOv8 will use the following code to save the best model as a .pt file for subsequent inference:

[0054] a)model.save('best_model.pt').

[0055] During inference, use the following code to load the saved model and make predictions:

[0056] b) results=model.predict('path_to_image_or_video').

[0057] 8. Tuning and Optimization: During training, the model's hyperparameters (such as learning rate, batch size, number of training epochs, etc.) can be adjusted based on the training results and validation set performance. Furthermore, for segmentation tasks, the input image resolution, enhancement strategies, or a larger model can be adjusted to further improve performance.

[0058] By following the steps above, YOLOv8 can complete object detection and instance segmentation tasks, and can be further tuned and optimized as needed.

[0059] As an optional implementation, the dynamic range value is specifically represented as a number ranging from 0 to 100. A higher value indicates a higher degree of dynamics. The COCO dataset contains 80 categories covering a wide range of objects, including humans, animals, vehicles, and everyday items. These categories are scored based on their degree of dynamics in real life (i.e., how often the object is in motion), with scores ranging from 0 (almost immobile) to 100 (frequently moving). Table 1 below shows the scores based on the dynamic range of these categories.

[0060] Table 1. COCO Dataset Category Dynamics Scoring Table

[0061] Initial category Initial dynamic level value Initial category Initial dynamic level value Initial category Initial dynamic level value people 100 bike 90 car 90 motorcycle 85 airplane 95 bus 85 train 85 truck 85 Boat 70 Traffic lights 10 fire hydrant 5 Parking sign 5 Parking fee meter 5 bench 0 bird 80 cat 70 dog 80 horse 80 sheep 60 dairy cow 50 elephant 40 Bear 60 zebra 60 giraffe 40 Backpack 10 Umbrella 5 handbag 10 tie 0 suitcase 10 Frisbee 70 sled 40 skis 40 sports ball 80 Kite 90 bat 20 Baseball glove 10 skateboard 80 surfboard 80 tennis racket 20 bottle 5 glass wine glass 5 Measuring cup 5 fork 0 knife 0 spoon 0 bowl 5 banana 0 apple 0 sandwich 0 orange 0 broccoli 0 carrot 0 hot dog 0 pizza 0 Donut 0 cake 0 Chair 0 sofa 0 Potted plants 5 bed 0 dining table 0 flush toilet 0 TV set 0 laptop 0 mouse 0 remote control 0 keyboard 0 cell phone 0 Micro-wave oven 0 oven 0 toaster 0 Dishwasher Sink 0 refrigerator 0 Book 0 Clocks 0 vase 0 Scissors 0 Teddy bear 0 Hair dryer 0 toothbrush 0

[0062] In another exemplary embodiment of this application, step 103 specifically includes:

[0063] The Depth-Anything depth estimation algorithm is used to estimate the depth of the original environment image at the current moment, obtaining the depth information corresponding to each pixel. The specific steps are as follows:

[0064] 1. Prepare input data (RGB image of the target environment)

[0065] First, you need to prepare an RGB image of the target environment, usually a regular RGB image. This image will be used as input to the depth estimation algorithm.

[0066] 2. Loading and preprocessing images

[0067] Before inputting images into the Depth-Anything model, preprocessing is typically required. This includes: resizing the image to 640×480 to meet the input requirements of the Depth-Anything model; and normalizing the image, usually by scaling pixel values ​​to the range of [0, 1] or subtracting the mean and dividing by the standard deviation, to make the image consistent with the input format used when training the Depth-Anything model.

[0068] 3. Load the pre-trained Depth-Anything model

[0069] The Depth-Anything algorithm typically relies on a pre-trained deep neural network, which has already been trained on a large number of labeled images. To obtain depth estimates, this pre-trained model needs to be loaded using the following code.

[0070] from depth_anything importDepthAnything.

[0071] model=DepthAnything.from_pretrained("model_checkpoint_or_url").

[0072] 4. Image Reasoning (Depth Estimation)

[0073] The following code demonstrates how to perform inference on an RGB image of a target environment using a pre-trained model. The depth estimation algorithm predicts the depth information of each pixel in the image through the network's forward propagation. Typically, depth information is represented as floating-point numbers, indicating the distance of that pixel from the camera (usually in meters or centimeters).

[0074] depth_map=model.predict(left_image).

[0075] Where left_image is the RGB image of the target environment, and depth_map is the depth image output by the model, where each pixel value represents the depth of the corresponding pixel in the image (i.e., the distance from the camera).

[0076] In another exemplary embodiment of this application, step 104 specifically includes:

[0077] 2D feature points are extracted from an RGB image using an image processing algorithm and matched with the 2D feature points corresponding to the previous time step to obtain the camera's motion information; the 2D feature points include intersection points and edge points; the motion information includes translation parameters and rotation parameters.

[0078] The camera pose information at the current moment is determined based on the camera's motion information and IMU data; the IMU data includes accelerometer measurements and gyroscope measurements.

[0079] Based on category-dynamic degree labeled images, depth images, and camera pose information, each pixel in the depth image is back-projected into 3D space to obtain the corresponding point cloud data, and a 3D map is constructed.

[0080] The VIO (Visual-IMU Odometry) system combines visual features with data from inertial measurement units (IMUs) to estimate the current camera pose. Combined with depth information, 2D image feature points are accurately converted into 3D point cloud data, resulting in a 3D map.

[0081] In another exemplary embodiment of this application, step 105 specifically includes:

[0082] Calculate the modulus of each point cloud data in the 3D map.

[0083] Based on the modulus of each point cloud data in the 3D map, calculate the average modulus of all point cloud data of the same category in the 3D map, and subtract it from the average modulus of all point cloud data of the corresponding category in the 3D map at the previous time step to obtain the difference in the average modulus of each category.

[0084] The dynamic degree value of the same category is corrected based on the difference in the mean modulus of each category, so as to obtain the dynamic degree value of each category at the current time.

[0085] Point cloud data whose current dynamic level value is greater than the preset dynamic level threshold are marked as dynamic points and removed.

[0086] Based on the point cloud data after dynamic point removal in the 3D map, the ORB-SLAM3 algorithm is used for map construction and optimization to obtain a global static environment map that reflects the static structure of the target environment.

[0087] As an optional implementation, the depth image at the current moment is obtained using the aforementioned Depth-Anything depth estimation algorithm. Assume D(i t j t ) is the (i)th time in the depth image at time t. t j t The depth value of a pixel represents the straight-line distance from that point to the camera. The camera's intrinsic parameter matrix K (usually the camera's focal length and optical center) is known.

[0088] 1. Calculate the 3D coordinates of a pixel.

[0089] First, the 3D spatial coordinates of each pixel need to be calculated from the depth image at the current moment. Assuming the camera intrinsic matrix K is known, it can typically be represented as:

[0090]

[0091] Where: f x f y It is the camera's focal length (unit: pixels); c x c y These are the coordinates of the camera's principal point (in pixels).

[0092] For each pixel (i) at time t in the depth image t j t Its corresponding three-dimensional coordinates can be calculated using the following formula:

[0093]

[0094] Z = D(i t ,j t )

[0095] Where X, Y, Z are the three-dimensional spatial coordinates of pixels in the image at time t; D(i t j t () is the depth value of the pixel in the depth map at the current moment.

[0096] 2. Calculate the modulus

[0097] To calculate the magnitude (i.e., the distance from that point to the camera origin) of each pixel in the depth map at the current time, the magnitude can be calculated using the three-dimensional coordinates described above. For each pixel (i...) at time t... t j t Its corresponding three-dimensional coordinate vector is (X, Y, Z), and its magnitude can be calculated using the Euclidean distance formula.

[0098] As an optional implementation method, the formula for calculating the modulus of each point cloud data in a 3D map is:

[0099]

[0100] Where, ||v(i t ,j t || represents the modulus of the point cloud data in the i-th row and j-th column of the 3D map at time t; fx and f y c represents the camera's horizontal focal length and vertical focal length, respectively. x and c y D(i) represents the x-coordinate and y-coordinate of the principal point of the camera, respectively; t j t) represents the depth information of the point cloud data in the i-th row and j-th column of the 3D map at time t.

[0101] As an optional implementation, the global static environment map is generated using the ORB-SLAM3 system, which generates the environment map and performs real-time positioning. This process involves the following main steps:

[0102] 1. Camera initialization and feature extraction.

[0103] Feature extraction (ORB features): ORB-SLAM3 first extracts ORB features (OrientedFAST and RotatedBRIEF) from the image input from the camera. These feature points have high invariance in the image and can be effectively matched under different viewing angles and lighting conditions.

[0104] Initialization: The system initializes the camera's position and orientation by matching features in consecutive image frames and combining them with camera motion information, thus creating a preliminary environmental map. For stereo or RGB-D cameras, the system can accelerate the initialization process using stereo matching or depth information.

[0105] 2. Keyframes and Map Points.

[0106] Keyframe selection: ORB-SLAM3 automatically selects representative image frames as keyframes based on camera motion trajectory and feature matching. Keyframes are the core of ORB-SLAM3, containing significant information relative to other frames.

[0107] Map points: ORB-SLAM3 adds these feature points to the map whenever the camera detects enough feature points and they have a stable matching relationship across multiple frames. Each map point represents a physical location in space.

[0108] 3. Pose estimation and localization.

[0109] Pose estimation: ORB-SLAM3 estimates the camera's relative pose, i.e., its position and orientation in space, through feature matching between consecutive frames. This process is optimized using bundle adjustment.

[0110] Localization: The system reduces reprojection error by optimizing the camera pose, so that feature points in the image match the corresponding points in the map as closely as possible, thereby achieving accurate localization.

[0111] 4. Map optimization (map point and keyframe optimization).

[0112] Local optimization: ORB-SLAM3 employs a local map optimization method, continuously refining the map by adding new keyframes and map points in each frame. Local optimization is primarily achieved through bundle adjustment, which reduces reprojection errors and improves positioning accuracy.

[0113] Global Optimization: ORB-SLAM3 also supports global optimization, which further optimizes the entire map through global loop closure detection, eliminating drift caused by accumulated errors. Loop closure detection compares the system's trajectory and map with previous trajectories, corrects errors, and optimizes the entire map.

[0114] 5. Loop Closing

[0115] Loop closure detection: In the environment, the camera may return to an area it has previously traversed. ORB-SLAM3 detects this loop and aligns the current map with the previous map using a loop closure detection algorithm. This step is crucial for reducing positioning drift and improving map accuracy.

[0116] Closed-loop optimization: After successful closed-loop detection, the system will adjust the map and pose through global optimization to correct previous accumulated errors.

[0117] 6. Generate an environment map

[0118] Finally, through the above steps, ORB-SLAM3 will generate an accurate environment map. This map consists of the following parts:

[0119] 3D point cloud map: Map points form a dense three-dimensional point cloud. Each point represents a feature point in space, and the position of these points is usually obtained through methods such as triangulation.

[0120] Camera trajectory: By estimating the camera pose, ORB-SLAM3 generates the camera's motion trajectory, i.e., the path the camera takes in the environment.

[0121] Optimized map: After closed-loop optimization, ORB-SLAM3 generates a globally optimized map with higher accuracy and reduced drift and error.

[0122] 7. Output map.

[0123] The generated environment map can be output and visualized in various formats:

[0124] 3D point cloud visualization: Environmental maps can be displayed using point cloud visualization tools (such as PCL, RViz, etc.), which are typically composed of 3D map points and camera trajectories.

[0125] Map saving: ORB-SLAM3 can save the generated map in a specific format (such as PLY, PCD, etc.) for later use or further processing.

[0126] Path planning and navigation: The generated map can be directly used for the robot's path planning and environmental perception, helping the robot understand and navigate its environment.

[0127] 8. Dynamic Environment Handling

[0128] Dynamic Object Detection: In dynamic environments, ORB-SLAM3 may encounter moving objects. To maintain map accuracy, ORB-SLAM3 employs dynamic object detection and separation techniques to minimize the misidentification of dynamic objects as part of the environment map.

[0129] Updates and optimizations: ORB-SLAM3 dynamically updates the map, handles new objects or structures appearing in the scene, and maintains the map's real-time performance.

[0130] In summary, the process of ORB-SLAM3 generating an environment map can be summarized as follows: First, the camera is localized through ORB feature extraction and keyframe selection. Then, a preliminary map is constructed through feature matching and pose estimation of consecutive frames. During the continuous optimization of keyframes and map points, map accuracy is improved through local and global optimization, and loop closure detection is used to further correct map errors. Finally, ORB-SLAM3 generates an accurate environment map, typically presented as a 3D point cloud, which can be used for subsequent navigation, localization, and path planning tasks.

[0131] In another exemplary embodiment of this application, the path planning algorithm in step 106 includes the Fast-Planner path planning algorithm.

[0132] As an alternative implementation, the optimal path is obtained using the Fast Planner algorithm. Fast Planner is an algorithm for robot path planning, particularly suitable for real-time path planning in complex dynamic environments. Based on optimization, Fast Planner calculates an optimal path from the starting point to the target by considering factors such as robot kinematic constraints, environmental obstacles, and target position. This path typically considers factors such as shortest path, collision avoidance, and velocity / acceleration constraints.

[0133] The basic process of FastPlanner obtaining the optimal path:

[0134] 1. Environment Modeling and State Space Definition

[0135] Environmental modeling: First, FastPlanner acquires information about the robot's environment, specifically the global static map obtained via ORB-SLAM3. This map is typically a grid map, indicating whether each spatial location is occupied by obstacles.

[0136] State Space: The space in which the robot resides is discretized into a high-dimensional state space (usually 2D or 3D). Each state includes kinematic information such as the robot's position, velocity, and acceleration.

[0137] 2. Determining the initial and target states

[0138] Starting point: The robot's current pose (position and orientation) is used as the starting point for path planning.

[0139] Target: The target point can be a specific location.

[0140] 3. Cost Function Design

[0141] Fast Planner uses a cost function to evaluate the merits of each path. The cost function includes the following aspects:

[0142] Path length: The total length or distance of a path; the goal is usually to find the shortest path.

[0143] Collision cost: Collision detection between the path and obstacles to prevent the robot path from crossing obstacle areas.

[0144] Speed ​​and acceleration constraints: Robots are subject to physical constraints such as speed and acceleration during movement. Path planning needs to take these dynamic constraints into account to avoid speeding or sudden changes in acceleration.

[0145] Dynamic obstacle avoidance: In dynamic environments, path planning must not only avoid static obstacles, but also consider the dynamic avoidance of other moving objects.

[0146] These costs form a comprehensive objective function, which FastPlanner will try to minimize to obtain the optimal path.

[0147] 4. Path Search and Planning Methods

[0148] Fast Planner employs a sampling-based optimized path search method, among which common search methods include:

[0149] Sampling method: Different path search algorithms are used to sample the state space, and these sampled points are evaluated according to the cost function.

[0150] Optimization methods: After finding a set of feasible paths, Fast Planner further reduces the tortuosity of the paths and optimizes their smoothness through path smoothing and optimization, ensuring the feasibility of the paths under the robot's kinematic constraints. Commonly used optimization algorithms include gradient descent and dynamic programming that minimizes path cost.

[0151] During this process, Fast Planner generates paths incrementally, combining environmental information and cost functions to gradually adjust the paths until an optimal path is found.

[0152] In summary, this application acquires environmental information through sensor data to construct an obstacle map; transforms the robot and its motion constraints into a state space; designs a cost function considering factors such as path length, collision cost, dynamic obstacles, and velocity and acceleration constraints; and searches for the optimal planned path in the state space using sampling and optimization algorithms, such as RRT* (Rapidly-exploring Random TreesStar).

[0153] This application also provides an application scenario in which the above-mentioned mapping and path planning method based on dynamic removal is applied. Specifically, the mapping and path planning method based on dynamic removal provided in this embodiment can be applied to search and rescue operations in complex ruin environments. This scenario is divided into three core stages: environmental perception and dynamic map construction, intelligent path planning and optimization, and search and rescue task execution and feedback. In the environmental perception and dynamic map construction stage, the miniature quadruped robot fully utilizes its onboard vision sensors, inertial measurement units (IMUs), and other multi-sensoring devices to comprehensively collect various types of data in the ruin environment. By fusing visual and IMU information, visual limitations in narrow spaces are effectively overcome. At the same time, with the help of target detection and depth estimation technology, the robot can accurately identify and remove dynamic obstacles, thereby constructing a high-precision static environmental map. In the intelligent path planning and optimization stage, based on the constructed static map, the robot uses advanced path planning algorithms to quickly plan an optimal or near-optimal search and rescue path. During this process, the robot monitors environmental changes in real time and dynamically adjusts the path to avoid newly appearing obstacles. Furthermore, by combining multi-dimensional environmental information and high-precision positioning technology, the robot can ensure accurate navigation in complex rubble environments. Finally, in the search and rescue mission execution and feedback phase, the miniature quadruped robot will autonomously navigate to the target area according to a planned path to perform specific tasks such as searching for and rescuing trapped personnel. During the mission execution, the robot will collect mission execution data in real time, such as search and rescue progress and environmental changes, and feed this data back to the control center. This feedback data not only helps the control center understand the search and rescue progress but also provides important references for subsequent path optimization and mission adjustments.

[0154] The mapping and path planning method based on dynamic removal provided in this embodiment belongs to the intelligent path planning and optimization stage in search and rescue operations in complex rubble environments. This method not only significantly improves the robot's autonomous navigation capabilities in complex environments but also greatly enhances its ability to cope with dynamic challenges, thereby ensuring the smooth execution of search and rescue missions. Through the application of this method, miniature quadruped robots have demonstrated enormous potential and value in high-risk and high-difficulty tasks such as rubble search and rescue.

[0155] This application has the following technical effects:

[0156] 1. Compared with traditional SLAM methods in dynamic environments, this application uses the Depth-Anything algorithm for depth estimation, which requires less computing power and places a lighter load on the robot's computing platform compared to the binocular triangulation method.

[0157] 2. The Faster-Planner algorithm is used for path planning and navigation based on the reconstructed environmental map, which effectively achieves optimal path planning and generates motion control commands, enabling autonomous motion control.

[0158] 3. This application fully considers the interference of objects in dynamic environments. By using dynamic object extraction and high-precision map reconstruction technology, it ensures the robustness and stability of the system in unstructured dynamic environments, thereby improving the adaptability of the robot in practical applications.

[0159] Based on the same inventive concept, this application also provides a dynamic removal-based mapping and path planning apparatus for implementing the dynamic removal-based mapping and path planning method described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the dynamic removal-based mapping and path planning apparatus provided below can be found in the limitations of the dynamic removal-based mapping and path planning method described above, and will not be repeated here.

[0160] In one exemplary embodiment, such as Figure 2 As shown, a mapping and path planning device based on dynamic removal is provided, comprising:

[0161] The RGB image acquisition module 201 is used to acquire the RGB image of the target environment, which is the environment in which the robot is located at the current moment.

[0162] The object category recognition and dynamic degree labeling module 202 is used to determine the category of each object in the RGB image based on the RGB image and using a deep learning algorithm; and to set the dynamic degree value of the corresponding pixel in the RGB image according to the category of each object to obtain the category-dynamic degree labeling image at the current moment; the object categories include humans, animals, vehicles and everyday items.

[0163] The depth image acquisition module 203 is used to obtain a depth image based on an RGB image using the Depth-Anything algorithm.

[0164] The 3D map building module 204 is used to build a 3D map by converting each pixel into point cloud data in three-dimensional space based on RGB images, category-dynamic degree labeled images and depth images through visual odometry.

[0165] The dynamic point removal module 205 is used to determine and remove point cloud data that are dynamic points in the 3D map based on the dynamic degree value of each point cloud data in the 3D map, so as to obtain a global static environment map.

[0166] The path planning module 206 is used to obtain the optimal planned path for the robot in the target environment based on a global static environment map and a path planning algorithm.

[0167] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a dynamic removal-based graph construction and path planning method.

[0168] Those skilled in the art will understand that Figure 3The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0169] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0170] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0173] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A mapping and path planning method based on dynamic removal, characterized in that, The mapping and navigation method based on dynamic removal includes: Acquire an RGB image of the target environment; the target environment is the environment in which the robot is located at the current moment. Based on RGB images, a deep learning algorithm is used to determine the category of each object in the RGB image; and the dynamic level value of the corresponding pixel in the RGB image is set according to the category of each object to obtain the category-dynamic level label image at the current time; the categories of objects include humans, animals, vehicles and everyday items; Based on the RGB image, the Depth-Anything algorithm is used to obtain the depth image; Based on RGB images, category-dynamic degree labeled images, and depth images, a 3D map is constructed by converting each pixel into point cloud data in three-dimensional space using a visual odometry method. Based on the dynamic value of each point cloud data in the 3D map, point cloud data that are dynamic points in the 3D map are identified and removed to obtain a global static environment map. Based on a global static environment map, a path planning algorithm is used to obtain the optimal planned path for the robot in the target environment.

2. The mapping and path planning method based on dynamic removal according to claim 1, characterized in that, Based on RGB images, a deep learning algorithm is used to determine the category of each object in the RGB image, specifically including: An RGB image is input into an object detection and segmentation model to perform object category detection and instance segmentation on the RGB image at the current time, thereby obtaining the object category in the RGB image. The object detection and segmentation model is obtained by iteratively training the YOLOv8 network based on a sample training set. The sample training set includes historical RGB images with object categories labeled.

3. The mapping and path planning method based on dynamic removal according to claim 1, characterized in that, Based on RGB images, category-dynamic degree-labeled images, and depth images, a 3D map is constructed by converting each pixel into point cloud data in three-dimensional space using a visual odometry method. Specifically, this includes: 2D feature points are extracted from an RGB image using an image processing algorithm and matched with the 2D feature points corresponding to the previous time step to obtain the camera's motion information; the 2D feature points include intersection points and edge points; the motion information includes translation parameters and rotation parameters. The camera pose information at the current moment is determined based on the camera's motion information and IMU data; the IMU data includes accelerometer measurements and gyroscope measurements. Based on category-dynamic degree labeled images, depth images, and camera pose information, each pixel in the depth image is back-projected into 3D space to obtain the corresponding point cloud data, and a 3D map is constructed.

4. The mapping and path planning method based on dynamic removal according to claim 1, characterized in that, Based on the dynamic range value of each point cloud data in the 3D map, point cloud data that are dynamic points are identified and removed from the 3D map to obtain a global static environment map, which specifically includes: Calculate the modulus of each point cloud data in the 3D map; Based on the modulus of each point cloud data in the 3D map, calculate the average modulus of all point cloud data of the same category in the 3D map, and subtract it from the average modulus of all point cloud data of the corresponding category in the 3D map at the previous time step to obtain the difference in the average modulus of each category. The dynamic degree value of the same category is corrected according to the difference of the mean modulus of each category, so as to obtain the dynamic degree value of each category at the current time. Point cloud data whose current dynamic level value is greater than the preset dynamic level threshold are marked as dynamic points and removed; Based on the point cloud data after dynamic point removal in the 3D map, the ORB-SLAM3 algorithm is used for map construction and optimization to obtain a global static environment map that reflects the static structure of the target environment.

5. The mapping and path planning method based on dynamic removal according to claim 4, characterized in that, The formula for calculating the modulus of each point cloud data in a 3D map is: Where, ||v(i t ,j t || represents the modulus of the point cloud data in the i-th row and j-th column of the 3D map at time t; fx and f y c represents the camera's horizontal focal length and vertical focal length, respectively. x and c y D(i) represents the x-coordinate and y-coordinate of the principal point of the camera, respectively; t j t ) represents the depth information of the point cloud data in the i-th row and j-th column of the 3D map at time t.

6. The mapping and path planning method based on dynamic removal according to claim 1, characterized in that, Path planning algorithms include: Fast-Planner path planning algorithm.

7. A mapping and path planning device based on dynamic removal, characterized in that, The mapping and path planning apparatus based on dynamic removal applies the mapping and path planning method based on dynamic removal according to any one of claims 1-6, wherein the mapping and path planning apparatus based on dynamic removal comprises: An RGB image acquisition module is used to acquire an RGB image of the target environment, which is the environment in which the robot is located at the current moment. The object category recognition and dynamic level labeling module is used to determine the category of each object in an RGB image based on a deep learning algorithm; and to set the dynamic level value of the corresponding pixel in the RGB image according to the category of each object, so as to obtain the category-dynamic level labeling image at the current moment; the object categories include humans, animals, vehicles and everyday items; The depth image acquisition module is used to obtain a depth image based on an RGB image using the Depth-Anything algorithm; The 3D map building module is used to convert each pixel into point cloud data in three-dimensional space based on RGB images, category-dynamic degree labeled images, and depth images, and to build a 3D map using visual odometry methods. The dynamic point removal module is used to identify and remove point cloud data that are dynamic points in the 3D map based on the dynamic degree value of each point cloud data in the 3D map, so as to obtain a global static environment map. The path planning module is used to obtain the optimal planned path for the robot in the target environment based on a global static environment map and a path planning algorithm.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the mapping and path planning method based on dynamic removal as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the mapping and path planning method based on dynamic removal as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the mapping and path planning method based on dynamic removal as described in any one of claims 1-6.

Images