Quadruped robot path planning method fusing visual semantic information and laser radar

Abstract

The invention relates to a quadruped robot path planning method fusing visual semantic information and a laser radar. The method comprises the following steps: acquiring environment data, wherein the environment data comprises key ground semantic information, a depth image and original laser radar point cloud information; converting the key ground semantic information into laser radar key ground semantic point cloud information based on the depth image, and performing spatial fusion on the laser radar key ground semantic point cloud information and original laser radar point cloud information to obtain multi-source point cloud fusion information under a unified reference system; the multi-source point cloud fusion information is preprocessed, and non-ground interference points are filtered out; constructing a semantic enhanced local path planning model based on a semantic penalty term and a dynamic window path planning algorithm; on the basis of the feasible speed of the quadruped robot, the preprocessed multi-source point cloud fusion information and a semantic enhanced local path planning model, feasible trajectory selection is carried out, and an optimal obstacle avoidance planning path is determined. And the navigation safety and the task execution efficiency of the robot in a high-risk industrial environment are remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of mobile robot path planning, and in particular to a quadruped robot path planning method that integrates visual semantic information and laser radar. Background Art

[0002] Amid the rapid development of industrial digitalization, mobile robots, thanks to their flexible maneuverability and excellent environmental adaptability, are widely used in automated inspection tasks in various workshops. Compared to traditional manual inspections, mobile robots not only significantly reduce labor costs but also significantly improve inspection efficiency and operational safety. Railway maintenance workshops, a typical industrial inspection scenario, feature trenches on the floor for equipment maintenance. While convenient for manual operation, these trenches pose significant safety risks to mobile robots performing autonomous inspections. In particular, quadruped robots, if their perception systems cannot accurately identify the trench location during inspections, can easily stray into it, causing falls, collisions, and other accidents, seriously threatening equipment safety and mission stability.

[0003] Currently, in the field of mobile robot environmental perception, LiDAR (Light Detection and Ranging) is widely used in positioning, mapping, and obstacle avoidance tasks due to its high-precision three-dimensional ranging capabilities. Typical LiDAR-based SLAM (Simultaneous Localization and Mapping) algorithms, such as Cartographer and LIO-SAM, have demonstrated excellent performance in a wide range of complex scenarios. However, due to limitations such as laser emission angle, reflection characteristics, and ground obstruction, LiDAR has blind spots near the ground (especially at the edges of ditches), making it difficult to fully obtain critical ground structure information, resulting in ineffective path planning safety.

[0004] Vision-based SLAM methods, such as LSD-SLAM and the ORB-SLAM series, can capture rich semantic and texture information and offer unique advantages in scene understanding. However, these methods are generally sensitive to lighting conditions, rely on feature extraction, and consume large amounts of computational resources. This makes stable, real-time deployment difficult in industrial workshop environments with complex lighting conditions and limited computing power.

[0005] Therefore, in the relevant technology, there is an urgent need for a method that can improve the navigation safety and operating efficiency of quadruped robots in industrial scenarios such as railway maintenance. Summary of the Invention

[0006] Based on this, it is necessary to address the above technical problems and provide a quadruped robot path planning method that integrates visual semantic information and lidar, which can improve the navigation safety and efficiency of quadruped robots in industrial scenarios such as railway maintenance.

[0007] In a first aspect, the present application provides a quadruped robot path planning method that integrates visual semantic information and laser radar. The method comprises: Acquiring environmental data, including key ground semantic information, depth images, and raw lidar point cloud information; Converting the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fusing it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system; Preprocessing the multi-source point cloud fusion information to filter out non-ground interference points; A semantically enhanced local path planning model is constructed based on semantic penalty terms and dynamic window path planning algorithm; Based on the feasible speed of the quadruped robot, the pre-processed multi-source point cloud fusion information and the semantically enhanced local path planning model, feasible trajectory selection is performed to determine the optimal obstacle avoidance planning path.

[0008] Optionally, in one embodiment of the present application, obtaining environmental data includes: Collect ground images and use a lightweight semantic segmentation network to extract key ground semantic information.

[0009] Optionally, in one embodiment of the present application, converting the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fusing the information with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system includes: Mapping the key ground semantic information into a semantic key point cloud in three-dimensional space by combining the depth image and the depth camera intrinsic parameter matrix; The Opencalib tool is used in combination with the depth camera extrinsic calibration matrix to project the semantic key point cloud of the three-dimensional space into the lidar coordinate system to obtain multi-source point cloud fusion information in a unified reference system.

[0010] Optionally, in one embodiment of the present application, preprocessing the multi-source point cloud fusion information to filter out non-ground interference points includes: downsampling the multi-source point cloud fusion information using voxel grid filtering; The DBSCAN algorithm is used to cluster the downsampled point cloud, and the RANSAC algorithm is used to perform plane fitting on the clustered point clusters to determine the angle between the plane normal vector and the world coordinate system. Based on the angle and the preset angle threshold, non-ground interference points are eliminated to retain the key semantic structure of the ground.

[0011] Optionally, in one embodiment of the present application, constructing a semantically enhanced local path planning model based on a semantic penalty term and a dynamic window path planning algorithm includes: Determine the semantic penalty function based on the distance between the trajectory point and the semantic obstacle point; A comprehensive trajectory cost function is determined based on the semantic penalty function, the target yaw angle cost function, the obstacle distance cost function and the speed cost function.

[0012] Optionally, in one embodiment of the present application, the method further includes: A control group was set up, and the success rate and number of redundant actions were used to evaluate the success rate, safety, and movement efficiency of path planning.

[0013] In a second aspect, the present application also provides a quadruped robot path planning device that integrates visual semantic information and laser radar. The device includes: An environmental data acquisition module is used to acquire environmental data, including key ground semantic information, depth images, and original lidar point cloud information; A multi-source point cloud fusion module is used to convert the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fuse it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system; A multi-source point cloud preprocessing module is used to preprocess the multi-source point cloud fusion information and filter out non-ground interference points; A semantically enhanced local path planning model construction module, which is used to construct a semantically enhanced local path planning model based on semantic penalty terms and a dynamic window path planning algorithm; The path planning module is used to select feasible trajectories based on the feasible speed of the quadruped robot, the pre-processed multi-source point cloud fusion information and the semantically enhanced local path planning model, and determine the optimal obstacle avoidance planning path.

[0014] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program and the processor executes the steps of the method described in each of the above embodiments.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in each of the above embodiments.

[0016] The above-mentioned quadruped robot path planning method that integrates visual semantic information and lidar first obtains environmental data, which includes key ground semantic information, depth image and original lidar point cloud information; then, based on the depth image, the key ground semantic information is converted into lidar key ground semantic point cloud information, and spatially fused with the original lidar point cloud information to obtain multi-source point cloud fusion information under a unified reference system; then, the multi-source point cloud fusion information is preprocessed to filter out non-ground interference points; then, a semantically enhanced local path planning model is constructed based on the semantic penalty term and the dynamic window path planning algorithm; finally, feasible trajectory selection is performed based on the feasible speed of the quadruped robot, the preprocessed multi-source point cloud fusion information and the semantically enhanced local path planning model to determine the optimal obstacle avoidance planning path. In other words, in view of the fact that lidar has blind spots in near-ground areas, a visual semantic segmentation network dedicated to identifying ground warning signs (such as yellow warning lines) is introduced, and combined with depth map generation and point cloud fusion technology, a three-dimensional environmental perception model with semantic understanding capabilities is constructed. By introducing semantic penalty terms in the path planning algorithm to constrain the robot to avoid dangerous areas, a dynamic obstacle avoidance strategy that is more suitable for complex industrial scenarios (such as areas with dense trenches) is implemented, thereby significantly improving the robot's navigation safety and task execution efficiency in high-risk industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A diagram illustrating an application environment of a quadruped robot path planning method that integrates visual semantic information and laser radar in one embodiment; Figure 2 1 is a flow chart of a quadruped robot path planning method that integrates visual semantic information and laser radar in one embodiment; Figure 3 Schematic diagram of the structure of a lightweight semantic segmentation network in one embodiment; Figure 4 A schematic diagram of the principle of generating multi-source point cloud fusion information in one embodiment; Figure 5 A schematic diagram of point cloud ground extraction and interference filtering in one embodiment; Figure 6 A schematic diagram of semantic map generation and path update in one embodiment; Figure 7 1. A block diagram of a quadruped robot path planning device that integrates visual semantic information and laser radar in one embodiment; Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0019] The quadruped robot path planning method provided by the embodiment of the present application that integrates visual semantic information and laser radar can be applied to Figure 1 In the application environment shown, the terminal communicates with the server through the network. The data storage system can store data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0020] In one embodiment, Figure 2 As shown in the figure, a quadruped robot path planning method that integrates visual semantic information and laser radar is provided. Figure 1 The following steps are used as an example to illustrate the server in the example: S201: Acquire environmental data, where the environmental data includes key ground semantic information, depth images, and original lidar point cloud information.

[0021] In this application example, a Unitree GO1 quadruped robot equipped with a Mid-360 solid-state LiDAR and a ZED2 binocular camera was used. The robot's control platform was developed based on the ROS system, with all sensors and control logic synchronized via ROS messaging. The ZED2 camera and LiDAR were used to obtain environmental data about the quadruped robot, including key ground semantic information, depth images, and raw LiDAR point cloud information. Key ground semantic information refers to the semantic information of important area markers, such as yellow warning lines.

[0022] In one embodiment of the present application, obtaining environmental data includes: Collect ground images and use a lightweight semantic segmentation network to extract key ground semantic information.

[0023] In one embodiment of the present application, a ground image frame is acquired through a ZED2 camera, and a DeepLabv3+ network based on the MobileNetV2 backbone is used for semantic segmentation to extract important ground semantic area information such as the yellow warning line and generate a segmentation map. Figure 3As shown in Figure 2, the model employs an encoder-decoder architecture. The encoder uses MobileNetV2 as the backbone network, leveraging depthwise separable convolutions and an inverted residual structure for efficient feature extraction. The decoder incorporates the ASPP module to incorporate multi-scale contextual information. This model, deployed on an edge computing device equipped with an RTX 3060 graphics card, achieves a frame rate of 28 FPS, meeting real-time requirements.

[0024] S203: Convert the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fuse it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system.

[0025] In an embodiment of the present application, the key semantic pixels extracted from the image are combined with the depth map, and their three-dimensional position in the camera coordinate system is calculated through the camera intrinsic parameter matrix; then the semantic point cloud is projected to the lidar coordinate system through the extrinsic parameter calibration matrix to achieve multi-source point cloud fusion in a unified reference system.

[0026] Specifically, in one embodiment of the present application, converting the key ground semantic information based on the depth image into lidar key ground semantic point cloud information, and spatially fusing it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system includes: S301: Mapping the key ground semantic information into a semantic key point cloud in a three-dimensional space by combining a depth image and a depth camera intrinsic parameter matrix.

[0027] S303: Using the Opencalib tool in combination with the depth camera extrinsic calibration matrix, the semantic key point cloud of the three-dimensional space is projected into the lidar coordinate system to obtain multi-source point cloud fusion information in a unified reference system.

[0028] In one embodiment of the present application, Figure 4 As shown in the figure, after extracting the foreground pixels in the semantic segmentation map, each foreground pixel in the image can be Mapped to a three-dimensional space point in the camera coordinate system The conversion process uses the camera's intrinsic parameter matrix , which has the form:

[0029] in, 、 is the horizontal and vertical focal length of the camera, 、 The main point coordinates.

[0030] If the depth value of the corresponding pixel in the depth map is , then the three-dimensional coordinates of the point in the camera coordinate system are obtained by the following formula:

[0031] In order to achieve unified processing with the LiDAR point cloud, these points need to be converted from the camera coordinate system to the LiDAR coordinate system. After completing the external parameter calibration of the camera and LiDAR using the Opencalib tool, the rotation matrix is ​​obtained. and translation vectors , the coordinate system conversion can be completed through the following matrix transformation:

[0032] Ultimately, all foreground semantic pixels will be mapped into semantic point clouds in the lidar coordinate system, spatially fused with the lidar point cloud, and provide unified input for subsequent path planning.

[0033] S205: Pre-process the multi-source point cloud fusion information to filter out non-ground interference points.

[0034] In an embodiment of the present application, the multi-source point cloud fusion information is then clustered and fitted, the normal vector of each plane is estimated, and by calculating the angle between it and the Z-axis direction of the world coordinate system, non-ground interference points are filtered out, and only the ground structure related to the channel edge is retained.

[0035] In one embodiment of the present application, preprocessing the multi-source point cloud fusion information to filter out non-ground interference points includes: S401: down-sampling the multi-source point cloud fusion information using voxel grid filtering.

[0036] S403: Use the DBSCAN algorithm to cluster the downsampled point cloud, use the RANSAC algorithm to perform plane fitting on the clustered point clusters, determine the angle between the plane normal vector and the world coordinate system, and eliminate non-ground interference points based on the angle and a preset angle threshold to retain the key semantic structure of the ground.

[0037] In one embodiment of the present application, after point cloud fusion, in order to reduce processing complexity, ROI (region of interest) is first defined to narrow the processing range, and voxel grid filtering is used for downsampling to retain the main structural features of the point cloud. Figure 5As shown in the figure, the DBSCAN algorithm is used to cluster the point cloud. This algorithm clusters densely contiguous regions (e.g., clusters 1, 2, and 3) based on a set neighborhood radius and minimum number of points, identifying core and non-core points to construct the initial structural relationship of the point cloud. Subsequently, to further filter out non-ground interference points, the RANSAC algorithm is used to perform plane fitting on each point cluster, estimate the direction of its normal vector, and determine the angle with the gravity direction of the world coordinate system. Based on the set angle threshold, all point clusters with inclined or vertical structures are eliminated, retaining only the key semantic structures on the ground that approximate the horizontal plane, providing stable support for subsequent path planning.

[0038] S207: Constructing a semantically enhanced local path planning model based on the semantic penalty term and the dynamic window path planning algorithm.

[0039] To improve the robot's obstacle avoidance capabilities in semantically high-risk areas (such as near warning lines), this embodiment of the application introduces a semantic penalty term based on the traditional Dynamic Window Approach (DWA) method. This term characterizes the minimum distance effect between a trajectory point and a semantic obstacle. This term calculates a penalty based on the distance between the current path point and the semantic obstacle, preventing the path from crossing identified dangerous areas (such as ditches or warning lines).

[0040] Specifically, in one embodiment of the present application, the construction of a semantically enhanced local path planning model based on a semantic penalty term and a dynamic window path planning algorithm includes: S501: Determine a semantic penalty function based on the distance between the trajectory point and the semantic obstacle point.

[0041] S503: Determine a comprehensive trajectory cost function based on the semantic penalty function, the target yaw angle cost function, the obstacle distance cost function, and the speed cost function.

[0042] In one embodiment of the present application, the semantic penalty function is defined as follows:

[0043] in, Represents the linear velocity and angular velocity of the robot at the current moment, Represents a trajectory point in trajectory prediction, represents the nearest obstacle point in the semantic point cloud, represents the Euclidean distance between the trajectory point and the nearest semantic obstacle point, represents the semantic action threshold distance, is the scaling factor of the semantic penalty term.

[0044] It should be noted that although the semantic penalty function is formally defined as ,in Indicates the linear velocity, Indicates angular velocity, but the function itself does not directly use the velocity value for calculation, but through Specifically, the dynamic window algorithm (DWA) samples multiple velocity pairs in each control cycle. , predict the trajectory point sequence within the time window For these trajectory points, the system will calculate their distance to the semantic obstacle points in turn. Euclidean distance , and then substitute it into the semantic penalty function.

[0045] The final comprehensive trajectory cost function is defined as:

[0046] in, represents the target yaw angle cost (indicating the deviation of the trajectory from the target direction), represents the obstacle distance cost, Represents the speed cost, For the semantic penalty cost added above, , , , The weight coefficients corresponding to each item are: It is a normalization function used to compress multiple cost functions to the same scale range for easy comparison.

[0047] The calculation method of each cost function is as follows: (1) Target yaw cost function : Measured by speed command The deviation between the end direction of the generated trajectory and the target direction is defined as follows:

[0048] in is the direction angle of the line from the current point to the target point, is the heading angle of the predicted trajectory end point.

[0049] (2) Obstacle distance cost function : It represents the inverse of the minimum distance between the predicted trajectory point and the obstacle, and is used to penalize paths close to obstacles. It is defined as follows:

[0050] in, Indicates the trajectory prediction points, is the nearest common obstacle point.

[0051] (3) Speed ​​cost function : It is used to encourage the robot to select a higher line speed to improve efficiency, and is defined as:

[0052] in, is the current sampling rate, The maximum line speed allowed by the system.

[0053] (4) Semantic penalty cost function : As defined above.

[0054] The above cost function is calculated separately for each set of speed samples and is calculated by the set weight coefficient , , , Perform weighted accumulation. Finally, the comprehensive cost is input into the normalization function , which is used to compress multiple cost items into a unified numerical range (such as 0–1) to facilitate comprehensive comparison and optimal trajectory selection.

[0055] S209: Based on the feasible speed of the quadruped robot, the pre-processed multi-source point cloud fusion information and the semantically enhanced local path planning model, a feasible trajectory is selected to determine the optimal obstacle avoidance planning path.

[0056] In the embodiment of the present application, the trajectory generated by all feasible speed combinations is evaluated based on the cost function with semantic constraints, the one with the lowest cost is selected as the optimal path, and control instructions are generated to drive the robot to move forward at the optimal speed and avoid obstacles. Specifically, Figure 6 As shown in the figure, during the robot's inspection process, the camera continuously captures ground images and updates the semantic point cloud information in real time through steps such as foreground mask extraction, point cloud generation, clustering, and normal vector estimation, filtering out non-ground interference objects of similar color or shape. Local maps and danger zones are constructed in real time, providing reliable input for path planning. Path planning results are dynamically updated based on the latest semantic information. When the system detects semantically high-risk areas such as newly added warning lines, it can promptly deviate from the current path to effectively avoid the danger zone, enabling local replanning and dynamic obstacle avoidance to ensure safe robot inspections.

[0057] In one embodiment of the present application, the method further includes: A control group was set up, and the success rate and number of redundant actions were used to evaluate the success rate, safety, and movement efficiency of path planning.

[0058] In one embodiment of the present application, a simulated railway maintenance workshop environment was built, yellow tape was pasted on the ground to mark the channel warning area, and target points were set at different starting points for inspection tasks to evaluate the path planning success rate, safety, and movement efficiency. A control group was set up: Option 1: only using the lidar point cloud; Option 2: fusing the unfiltered semantic point cloud; Option 3: this method: fusing the filtered semantic point cloud. The experiment set up 3 task points in total, and each task point was tested 10 times, and the following three performance indicators were recorded: success rate ( ), the number of redundant actions ( ) and the comprehensive score ( ).in, Refers to the proportion of the number of times the robot successfully reaches the target point before entering the semantically marked danger zone (such as the yellow warning line) to the total number of tests; Refers to the number of invalid or redundant actions performed by the robot during navigation due to incorrect path evaluation or misidentification of obstacle avoidance. The final test results are shown in Table 1 below. The specific calculation formula for performance indicators is as follows:

[0059] in, Indicates the Whether the task is successful, success is recorded as 1, failure is recorded as 0, is the total number of tasks, and Respectively represent The actual path length of the robot in this task and the global optimal path length. Weighting coefficient and The execution capability and path redundancy of the navigation system are comprehensively evaluated and set to 0.7 and 0.3 respectively in the experiment.

[0060] Table 1 Navigation performance test results of the method of the present invention and the comparative scheme under three tasks

[0061] The results show that the success rate of this method is 100% at all task points, with the least number of redundant actions and a higher comprehensive score than other schemes, which demonstrates its stability and superiority in semantically enhanced path planning.

[0062] In the above-mentioned quadruped robot path planning method that integrates visual semantic information and lidar, first, environmental data is acquired, and the environmental data includes key ground semantic information, depth image and original lidar point cloud information; then, the key ground semantic information is converted into lidar key ground semantic point cloud information based on the depth image, and spatially fused with the original lidar point cloud information to obtain multi-source point cloud fusion information under a unified reference system; then, the multi-source point cloud fusion information is preprocessed to filter out non-ground interference points; then, a semantically enhanced local path planning model is constructed based on the semantic penalty term and the dynamic window path planning algorithm; finally, feasible trajectory selection is performed based on the feasible speed of the quadruped robot, the preprocessed multi-source point cloud fusion information and the semantically enhanced local path planning model to determine the optimal obstacle avoidance planning path. In other words, in view of the fact that lidar has blind spots in near-ground areas, a visual semantic segmentation network dedicated to identifying ground warning signs (such as yellow warning lines) is introduced, and combined with depth map generation and point cloud fusion technology, a three-dimensional environmental perception model with semantic understanding capabilities is constructed. By introducing semantic penalty terms in the path planning algorithm to constrain the robot to avoid dangerous areas, a dynamic obstacle avoidance strategy that is more suitable for complex industrial scenarios (such as areas with dense trenches) is implemented, thereby significantly improving the robot's navigation safety and task execution efficiency in high-risk industrial environments.

[0063] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0064] Based on the same inventive concept, the embodiments of the present application also provide a quadruped robot path planning device that integrates visual semantic information and laser radar, which is used to implement the aforementioned method for quadruped robot path planning that integrates visual semantic information and laser radar. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the quadruped robot path planning device that integrates visual semantic information and laser radar provided below can be found in the above-mentioned limitations of the quadruped robot path planning method that integrates visual semantic information and laser radar, and will not be repeated here.

[0065] In one embodiment, Figure 7 As shown, a quadruped robot path planning device 700 that integrates visual semantic information and laser radar is provided, comprising: an environmental data acquisition module 701, a multi-source point cloud fusion module 703, a multi-source point cloud preprocessing module 705, a semantically enhanced local path planning model construction module 707, and a path planning module 709, wherein: An environmental data acquisition module is used to acquire environmental data, including key ground semantic information, depth images, and original lidar point cloud information; A multi-source point cloud fusion module is used to convert the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fuse it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system; A multi-source point cloud preprocessing module is used to preprocess the multi-source point cloud fusion information and filter out non-ground interference points; A semantically enhanced local path planning model construction module, which is used to construct a semantically enhanced local path planning model based on semantic penalty terms and a dynamic window path planning algorithm; The path planning module is used to select feasible trajectories based on the feasible speed of the quadruped robot, the pre-processed multi-source point cloud fusion information and the semantically enhanced local path planning model, and determine the optimal obstacle avoidance planning path.

[0066] In one embodiment of the present application, the environmental data acquisition module is further configured to: Collect ground images and use a lightweight semantic segmentation network to extract key ground semantic information.

[0067] In one embodiment of the present application, the multi-source point cloud fusion module is further configured to: Mapping the key ground semantic information into a semantic key point cloud in three-dimensional space by combining the depth image and the depth camera intrinsic parameter matrix; The Opencalib tool is used in combination with the depth camera extrinsic calibration matrix to project the semantic key point cloud of the three-dimensional space into the lidar coordinate system to obtain multi-source point cloud fusion information in a unified reference system.

[0068] In one embodiment of the present application, the multi-source point cloud pre-processing module is further configured to: downsampling the multi-source point cloud fusion information using voxel grid filtering; The DBSCAN algorithm is used to cluster the downsampled point cloud, and the RANSAC algorithm is used to perform plane fitting on the clustered point clusters to determine the angle between the plane normal vector and the world coordinate system. Based on the angle and the preset angle threshold, non-ground interference points are eliminated to retain the key semantic structure of the ground.

[0069] In one embodiment of the present application, the semantically enhanced local path planning model construction module is further used to: Determine the semantic penalty function based on the distance between the trajectory point and the semantic obstacle point; A comprehensive trajectory cost function is determined based on the semantic penalty function, the target yaw angle cost function, the obstacle distance cost function and the speed cost function.

[0070] The quadruped robot path planning device integrating visual semantic information and laser radar further includes an evaluation module. In one embodiment of the present application, the evaluation module is further configured to: A control group was set up, and the success rate and number of redundant actions were used to evaluate the success rate, safety, and movement efficiency of path planning.

[0071] Each module in the aforementioned quadruped robot path planning device that integrates visual semantic information with lidar can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device's memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0072] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a quadruped robot path planning method that integrates visual semantic information with lidar. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0073] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0074] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0075] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0076] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0078] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0079] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A quadruped robot path planning method integrating visual semantic information and laser radar, characterized in that: The method comprises: Acquiring environmental data, including key ground semantic information, depth images, and raw lidar point cloud information; Converting the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fusing it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system; Preprocessing the multi-source point cloud fusion information to filter out non-ground interference points; A semantically enhanced local path planning model is constructed based on semantic penalty terms and dynamic window path planning algorithm; Based on the feasible speed of the quadruped robot, the pre-processed multi-source point cloud fusion information and the semantically enhanced local path planning model, feasible trajectory selection is performed to determine the optimal obstacle avoidance planning path.

2. The quadruped robot path planning method integrating visual semantic information and laser radar according to claim 1 is characterized in that: The obtaining of environmental data includes: Collect ground images and use a lightweight semantic segmentation network to extract key ground semantic information.

3. The quadruped robot path planning method integrating visual semantic information and laser radar according to claim 1, characterized in that: The converting the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fusing the information with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system includes: Mapping the key ground semantic information into a semantic key point cloud in three-dimensional space by combining the depth image and the depth camera intrinsic parameter matrix; The Opencalib tool is used in combination with the depth camera extrinsic calibration matrix to project the semantic key point cloud of the three-dimensional space into the lidar coordinate system to obtain multi-source point cloud fusion information in a unified reference system.

4. The quadruped robot path planning method integrating visual semantic information and laser radar according to claim 1, characterized in that: The pre-processing of the multi-source point cloud fusion information to filter out non-ground interference points includes: downsampling the multi-source point cloud fusion information using voxel grid filtering; The DBSCAN algorithm is used to cluster the downsampled point cloud, and the RANSAC algorithm is used to perform plane fitting on the clustered point clusters to determine the angle between the plane normal vector and the world coordinate system. Based on the angle and the preset angle threshold, non-ground interference points are eliminated to retain the key semantic structure of the ground.

5. The quadruped robot path planning method integrating visual semantic information and laser radar according to claim 1, characterized in that: The semantically enhanced local path planning model based on the semantic penalty term and the dynamic window path planning algorithm comprises: Determine the semantic penalty function based on the distance between the trajectory point and the semantic obstacle point; A comprehensive trajectory cost function is determined based on the semantic penalty function, the target yaw angle cost function, the obstacle distance cost function and the speed cost function.

6. The quadruped robot path planning method integrating visual semantic information and laser radar according to claim 1, characterized in that: The method further comprises: A control group was set up, and the success rate and number of redundant actions were used to evaluate the success rate, safety, and movement efficiency of path planning.

7. A quadruped robot path planning device that integrates visual semantic information and laser radar, characterized in that: The device comprises: An environmental data acquisition module is used to acquire environmental data, including key ground semantic information, depth images, and original lidar point cloud information; A multi-source point cloud fusion module is used to convert the key ground semantic information into lidar key ground semantic point cloud information based on the depth image, and spatially fuse it with the original lidar point cloud information to obtain multi-source point cloud fusion information in a unified reference system; A multi-source point cloud preprocessing module is used to preprocess the multi-source point cloud fusion information and filter out non-ground interference points; A semantically enhanced local path planning model construction module, which is used to construct a semantically enhanced local path planning model based on semantic penalty terms and a dynamic window path planning algorithm; The path planning module is used to select feasible trajectories based on the feasible speed of the quadruped robot, the pre-processed multi-source point cloud fusion information and the semantically enhanced local path planning model, and determine the optimal obstacle avoidance planning path.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images