Quadruped robot efficient semantic perception method and device based on instance point cloud

Abstract

The invention provides a quadruped robot efficient semantic perception method and device based on instance point clouds, and the method comprises the steps: building a visual large model through collecting RGB images and point cloud data, and carrying out the object instance segmentation and semantic feature extraction; combining a point cloud re-projection technology with an object mask, aggregating semantic features of the point cloud, and performing fusion processing on the instance point cloud of the current frame and the global instance point cloud to maintain the global point cloud and the semantic features thereof; point cloud and image data at multiple moments are processed through a time sequence, and global instance representation is constructed and continuously updated. The method has good scene generalization ability, is efficient in data processing, can effectively adapt to a dynamic environment, and keeps global consistency of semantic features.

Description

Technical Field

[0001] This invention relates to the field of efficient semantic perception technology for quadruped robots, specifically to a method and apparatus for acquiring scene geometric information using scene point cloud data, performing object semantic perception and segmentation using reprojection technology and large visual models, efficiently processing point cloud data, and realizing real-time scene semantic perception. Background Technology

[0002] Quadruped mobile robots, with their superior terrain adaptability and mobility, can walk stably on stairs, rugged roads, and complex outdoor environments, thus showing broad application prospects in disaster relief, inspection operations, and service interactions. Existing scene perception methods mainly rely on sensors such as cameras and LiDAR to acquire visual and structural information of the scene for localization, autonomous navigation, and dynamic obstacle avoidance. However, these methods are limited to geometric and physical level scene perception, making it difficult to support robots in performing high-level tasks in diverse and complex environments. Semantic perception technology, on the other hand, introduces human understanding of the environment onto the basis of sensors, enabling robots to identify and distinguish different categories of scenes and targets. For quadruped robots, combining environmental semantic perception with their mobility characteristics not only enables semantic-level path planning and dynamic obstacle avoidance in complex terrain but also allows them to autonomously adjust their gait and posture based on semantic information, thereby traversing obstacles or passing through narrow passages. Furthermore, semantic perception can significantly improve the quadruped robot's ability to perform high-level tasks, such as recognizing steps and climbing stairs under semantic guidance, or recognizing doors and passing through them. This technology bridges the gap between humans and robots in environmental perception, enabling robots to more naturally understand and execute complex human commands, thereby improving intelligence and human-computer interaction. By leveraging image data and combining it with large visual models to acquire environmental semantic information, semantic perception offers advantages such as strong scene generalization capabilities, accurate target recognition, and support for end-to-end processing. This allows quadruped robots to quickly adapt to unfamiliar environments and complete complex tasks, making it an important development direction for enhancing the intelligence and application value of quadruped robots.

[0003] For semantic perception in mobile robots, patent document CN112802204A discloses a target semantic navigation method based on prior knowledge of a 3D spatial scene in an unknown environment. This method performs semantic segmentation of a 3D point cloud using a depth camera and projects point clouds of different semantic layers to construct a 2D obstacle map. It utilizes deep reinforcement learning combined with scene semantic priors to achieve global navigation decisions. Patent document CN120293150A discloses a robot navigation method based on a 3D semantic map and a large language model. This method constructs a 3D semantic map using point clouds and establishes a three-layer collaborative planning architecture with a hierarchical feedback mechanism using a large language model, thereby achieving semantic navigation of the robot in dynamic scenes.

[0004] Despite the broad application prospects of semantic perception, it still faces many challenges in practical robot deployment. First, the semantic features extracted using large visual models are typically high-dimensional, making it difficult to guarantee data processing efficiency and real-time performance. Second, maintaining the consistency of semantic features in multi-view or dynamic scenarios is also a critical issue that urgently needs to be addressed.

[0005] To address the aforementioned problems, this invention provides an environmental semantic perception method that fuses scene point cloud and image information for quadruped robots. This method combines a large visual model for object segmentation and semantic feature extraction. By maintaining instance-level semantic features, it achieves efficient, real-time, and dynamically consistent semantic perception, effectively overcoming the limitations of existing methods in practical applications. Summary of the Invention

[0006] To address the limitations of existing scene semantic perception technologies, such as low data processing efficiency and difficulty in maintaining dynamic consistency, the first aspect of this invention provides an efficient semantic perception method for quadruped robots based on instance point clouds, mainly including the following steps:

[0007] Step 1: Initialize sensor parameters and calibrate robot positioning information;

[0008] Step 2: Initialize the data processing program and build the image processing model, including the Mobile-SAM model and the CLIP model;

[0009] The Mobile-SAM model's network structure starts with an input layer, sequentially connecting an image encoder, a cue encoder, a mask decoder, and an output layer. The image encoder extracts deep feature representations of the input image, employing a lightweight vision transformer (Tiny-ViT) structure to adapt to resource-constrained devices. The cue encoder processes user-provided cues such as points, boxes, or masks, encoding them into embedding vectors aligned with image features. The mask decoder, based on a Transformer architecture, fuses image features with the cue embedding vectors to generate a preliminary prediction of the segmentation mask. The output layer outputs the final image segmentation mask.

[0010] The CLIP model's network structure starts from the input layer and is divided into two parallel branches: an image encoder and a text encoder. The image encoder uses a Vision Transformer structure to extract visual features from the input image and generate corresponding vector representations. The text encoder uses...

[0011] The Transformer structure processes the input text description and converts it into a corresponding feature vector;

[0012] Step 3: Collect scene point cloud and image data at the current moment; use ROS2 as the communication framework to read RGB images and point cloud data from the sensor and publish them;

[0013] Step 4: Use the visual model built in Step 2 to process image data, use the Mobile-SAM model for instance segmentation, and use the CLIP model for semantic feature extraction; obtain point cloud data with semantic information through point cloud reprojection; combine the object mask obtained from segmentation to aggregate semantic point cloud features belonging to the same object.

[0014] Step 5: Fuse the current time frame instance point cloud with the global instance point cloud to maintain the global point cloud and global semantic features;

[0015] Step 6: Based on time series, collect scene point cloud and image data at different times for processing, and construct and maintain a global instance representation;

[0016] A second aspect of the present invention relates to an efficient semantic perception device for a quadruped robot based on instance point clouds, comprising a sensor, a memory, and one or more processors. The sensor is used for scene data required by the method, the memory stores executable code, and the one or more processors execute the executable code to process the scene data and realize the efficient semantic perception method based on instance point clouds of the present invention.

[0017] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the efficient semantic perception method for quadruped robots based on instance point clouds of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] 1. Compared with existing semantic perception methods, the solution of this invention has higher applicability and engineering practicality; the required point cloud data has a wide range of sources, which can be obtained through LiDAR or depth cameras, with low hardware dependence, low computational resource requirements, and good robustness and real-time performance; by maintaining instance-level semantic features and point cloud data, the memory burden caused by high-dimensional semantic features is effectively alleviated, while the semantic perception accuracy is further improved in the process of constructing instance-level scenes; by using pre-trained large visual models to extract semantic information, the system has good generalization ability for new scenes, can adapt to diverse environmental changes, and significantly enhances perception performance in complex and unknown environments;

[0020] 2. By maintaining global object instance information, the present invention can maintain the consistency of semantic features even when objects move or the viewpoint changes, overcoming the discontinuity problem caused by directly using local semantic features in existing solutions; by adopting a pixel-level semantic processing strategy, even after sparsifying the global instance information, it can still maintain high spatial perception accuracy; the degree of sparsification can be adjusted according to specific application requirements, thereby achieving a balance between perception accuracy and system real-time performance.

[0021] 3. The instance-level semantic scene representation constructed by this invention has broad downstream application potential; through rasterization processing, this representation can be used for semantic navigation and obstacle avoidance tasks of mobile robots; combined with multimodal large models, it can be further extended to embodied intelligence environmental understanding and task decision-making scenarios, significantly improving the intelligence level and adaptability of the system. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the present invention.

[0023] Figure 2 This is a hardware structure diagram of Embodiment 1 of the present invention.

[0024] Figure 3 This is a flowchart of the algorithm of the present invention.

[0025] Figure 4 This is a flowchart of the image and point cloud data processing of the present invention.

[0026] Figure 5 This is a flowchart of the global instance point cloud maintenance process of the present invention.

[0027] Figure 6 This is a schematic diagram illustrating the projection principle from the point cloud coordinate system to the pixel coordinate system of the present invention. Detailed Implementation

[0028] To make the objectives, technical methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings;

[0029] Example 1

[0030] like Figure 1 and Figure 3 As shown, this invention relates to an efficient semantic perception method for quadruped robots based on instance point clouds; this embodiment uses a combination of the Unitree Go2 quadruped robot and the Ark Infinite X5 robotic arm as a carrier, and the detailed hardware settings are as follows. Figure 2 As shown; the specific implementation steps of the present invention are as follows:

[0031] Step 1: Initialize sensor parameters and calibrate robot positioning information:

[0032] The sensor parameters mentioned include the device parameters of a monocular camera, such as video stream resolution and frame rate; and the device parameters for acquiring point cloud data, such as depth cameras and LiDAR. In this example, the Unitree L1 radar is used as the source of point cloud data.

[0033] The robot positioning information refers to the robot's own position and attitude information, which is obtained by the odometry calculation method based on the LIVOX MID-360 LiDAR;

[0034] Step 2: Initialize the data processing program and build the image processing model, including the Mobile-SAM model and the CLIP model;

[0035] The Mobile-SAM model's network structure starts with an input layer, sequentially connecting an image encoder, a cue encoder, a mask decoder, and an output layer. The image encoder extracts deep feature representations of the input image, employing a lightweight vision transformer (Tiny-ViT) structure to adapt to resource-constrained devices. The cue encoder processes user-provided cues such as points, boxes, or masks, encoding them into embedding vectors aligned with image features. The mask decoder, based on a Transformer architecture, fuses image features with the cue embedding vectors to generate a preliminary prediction of the segmentation mask. The output layer outputs the final image segmentation mask.

[0036] The CLIP model's network structure starts from the input layer and consists of two parallel branches: an image encoder and a text encoder. The image encoder uses a Vision Transformer structure to extract visual features from the input image and generate corresponding vector representations. The text encoder uses...

[0037] The Transformer structure is used to process the input text description and convert it into the corresponding feature vector;

[0038] Step 3: Collect scene point cloud and image data at the current moment; use ROS2 as the communication framework to read RGB images and point cloud data from the sensor and publish them;

[0039] Step 4: As Figure 4 As shown, the visual model constructed in step 2 is used to process image data, the Mobile-SAM model is used for instance segmentation, and the CLIP model is used for semantic feature extraction; point cloud data with semantic information is obtained through back projection; and the semantic point cloud features belonging to the same object are aggregated by combining the object mask obtained from the segmentation.

[0040] Step 4-1: Subscribe to image messages and process the received current image I using a visual model. t , where I t At the current time; using the Mobile-SAM model, image I is calculated. t object mask Where n is the number of objects in the current frame; using the CLIP model, the semantic feature map I of the image is calculated. f t ;

[0041] Step 4-2: Subscribe to point cloud data and perform projection calculations based on camera pose; the projection principle is as follows. Figure 6 As shown; the calculation is as shown in formula (1):

[0042]

[0043] Among them, (X) r ,Y r Z r (R, t) represents the position of the point cloud in the radar coordinate system; (R, t) is the transformation matrix from the radar coordinate system to the camera coordinate system. Z is the camera intrinsic parameter matrix; c Let (u, v) be the depth value in the camera coordinate system; (u, v) be the image plane coordinates of the projection point; and (u, v) be the depth value in the semantic feature map F. t Perform a retrieval to obtain the semantic features f of the corresponding points. t If (u, v) is inside a pixel, bilinear interpolation is used, as shown in formula (2):

[0044]

[0045]

[0046] Using object masks Point clouds and semantic features of the same object are aggregated to construct object instances; each instance includes a corresponding point cloud set P. t ={p t 1 p t 2 ...} and the semantic features F of the instance t =Avg(f t 1 f t 2 ,…), where Avg is the average calculation;

[0047] Step 5: Fuse the instance objects of the current time frame with the global instance objects to maintain the global instance point cloud and global instance semantic features; the maintenance process of the global instance point cloud is as follows: Figure 5 As shown;

[0048] Step 5-1: Check if it is the initial frame; if it is the initial frame, save the current frame instance as a global instance and return to step 3; current frame instance S t and global instance S g This is represented as shown in formula (3):

[0049]

[0050] If it is not the initial frame, continue with the following steps;

[0051] Step 5-2: Use cosine similarity, as shown in formula (4), to calculate the similarity between the semantic features of the current instance and the semantic features of the global instance;

[0052]

[0053] If the calculated similarity is greater than the threshold λ s This indicates that the two instances represent the same object;

[0054] Step 5-3: After finding two instances that represent the same object, calculate the Euclidean distance between the geometric centers of their point clouds, as shown in Formula (5);

[0055]

[0056] If the calculated distance is greater than the threshold λ d This indicates that the object has moved at the current moment;

[0057] Step 5-4: Traverse all instances in the current frame; if a corresponding instance is found in the global instance and the calculated Euclidean distance is less than the threshold, then fuse the point clouds and calculate the average semantic features of the two as the global instance semantic features; if the distance between the two is greater than the threshold, it indicates that the object has moved, and the point cloud of the current frame instance object is used as the point cloud of the global instance object, and the average semantic features of the two are calculated as the global instance semantic features; if there is an instance in the current frame that does not match any of the global instances, it is considered a new instance, and it is saved as a global instance.

[0058] Step 5-5: Perform point cloud sparsification on the global instance to reduce memory usage;

[0059] Step 6: Based on the time sequence, collect scene point cloud and image data at different times for processing, and construct and maintain a global instance representation.

[0060] Compared to other scene semantic perception methods based on multi-sensor fusion, this invention has the following significant advantages: low dependence on device hardware, strong system robustness, high processing efficiency, and good real-time performance. Furthermore, thanks to the large visual model, this solution exhibits outstanding generalization ability in new scenes and can effectively adapt to complex and changing environments. Addressing the target movement problem in dynamic scenes and the potential inconsistencies in local semantic features under multiple perspectives, this solution maintains a global instance-level semantic representation, achieving consistent alignment and continuous tracking of semantic features, significantly improving the stability and reliability of the perception system. In addition, this solution possesses good scalability and broad downstream application potential: it can be applied to tasks such as 3D map reconstruction, semantic navigation, and autonomous obstacle avoidance through rasterization, and can also be combined with a multimodal large model to extend to environmental understanding and task decision-making in embodied intelligent systems.

[0061] Example 2

[0062] This embodiment relates to an efficient semantic perception device for quadruped robots based on instance point clouds, including a sensor, a memory, and one or more processors. The sensor is used for scene data required by this method, and the memory stores executable code. When the one or more processors execute the executable code, they are used to process the scene data and implement the efficient semantic perception method for quadruped robots based on instance point clouds of Embodiment 1.

[0063] Example 3

[0064] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the efficient semantic perception method for quadruped robots based on instance point clouds of the present invention.

Claims

1. An efficient semantic perception method for a quadruped robot based on instance point cloud, characterized in that, The method comprises the following steps: Step 1: initialize sensor parameters, correct robot positioning information; Step 2: initialize data processing program, build image processing model, including Mobile-SAM model and CLIP model; The network structure of the Mobile-SAM model is that, from the input layer, an image encoder, a prompt encoder, a mask decoder and an output layer are connected in turn; wherein the image encoder is used to extract deep feature representation of the input image, and a Tiny Vision Transformer (Tiny-ViT) structure is adopted to adapt to the deployment of resource-limited devices; the prompt encoder processes the prompt information such as points, boxes or masks provided by the user, and encodes them into embedding vectors aligned with the image features; the mask decoder is based on the Transformer architecture, fuses image features and prompt embedding vectors, and generates a preliminary prediction of the segmentation mask; the output layer outputs the final image segmentation mask; The network structure of the CLIP model is that, from the input layer, an image encoder and a text encoder are connected in parallel; wherein the image encoder adopts a Vision Transformer structure to extract visual features of the input image and generate corresponding vector representation; the text encoder adopts a Transformer structure to process the input text description and convert it into a corresponding feature vector; Step 3: collect the scene point cloud and image data at the current time; use ROS2 as the communication framework to read the RGB image and point cloud data from the sensor and publish them; Step 4: use the visual model built in step 2 to process the image data, use the Mobile-SAM model for instance segmentation, and use the CLIP model for semantic feature extraction; through point cloud reprojection, obtain point cloud data with semantic information; combine the object mask obtained by segmentation to aggregate semantic point cloud features belonging to the same object; Step 5: fuse the instance point cloud of the current time frame with the global instance point cloud, and maintain the global point cloud and global semantic features; Step 6: based on the time sequence, collect scene point cloud and image data at different times for processing, and build and maintain global instance representation.

2. The method for efficient semantic perception based on instance point cloud according to claim 1, characterized in that, The visual large model in step 2 is used for monocular image processing, including image semantic segmentation and semantic feature extraction operations; the model includes but is not limited to Mobile-SAM model, CLIP model, etc.

3. The method for efficient semantic perception based on instance point cloud according to claim 1, characterized in that, Step 4 specifically includes: Step 4-1: Subscribe to image messages, use visual model to process the received current image I t treatment, where t is the current time; using the Mobile-SAM model, the object mask of the image I t is calculated where n is the number of objects in the current frame; using the CLIP model, the semantic feature map of the image I f t is calculated Step 4-2: subscribe to point cloud data, combine camera pose, and perform projection calculation; the projection principle is shown in FIG. 6; the calculation is shown in formula (1): where (X r ,Y r ,Z r ) is the position of the point cloud in the radar coordinate system; (R, t) is the transformation matrix from the radar coordinate system to the camera coordinate system; is the camera intrinsic matrix; Z c is the depth value in the camera coordinate system; (u, v) is the image plane coordinate of the projected point; by searching in the semantic feature map F t , the semantic feature f t of the corresponding point is obtained; if (u, v) is inside the pixel point, the bilinear interpolation method is used, as shown in equation (2): (u2, v2) = (ul + 1, vl + 1), α=u-u1,β=v-v1 Using object masks Point clouds and semantic features of the same object are aggregated to construct object instances; each instance includes a corresponding point cloud set P. t ={p t 1 p t 2 ...} and the semantic features F of the instance t =Avg(f t 1 f t 2 , ...), where Avg is the average calculation.

4. The method for efficient semantic perception based on instance point cloud according to claim 1, characterized in that, Step 5 specifically includes: Step 5-1: Detect whether it is an initial frame; if it is an initial frame, save the instance of the current frame as a global instance, and return to Step 3; the current frame instance S t and the global instance S g represents as shown in formula (3): If not the initial frame, continue with the following steps; Step 5-2: use cosine similarity, as shown in formula (4), to calculate the similarity between the current instance semantic feature and the global instance semantic feature; If the calculated similarity is greater than a threshold λ s then the two instances are said to represent the same object. Step 5-3: after finding two instances representing the same object, calculate the Euclidean distance between the geometric centers of the two instances, as shown in formula (5); Step 6 specifically includes: Step 6-1: use the global instance semantic feature to query the global instance database, and obtain the global instance semantic feature corresponding to the query result; If the calculated distance is greater than a threshold value λ d then it indicates that the object has moved at the current time. Step 5-4: traverse all instances of the current frame; if a corresponding instance is found in the global instance, and the calculated Euclidean distance is less than the threshold, then the point clouds are fused and the average semantic features of the two are calculated as the global instance semantic features; if the distance between the two is greater than the threshold, it indicates that the object has moved, and the point cloud of the current frame instance object is taken as the point cloud of the global instance object, and the average semantic features of the two are calculated as the global instance semantic features; if there is an instance in the current frame that does not match the global instance, it indicates a new instance, which is saved as a global instance; Step 5-5: perform point cloud sparsification operation on the global instance to reduce memory occupation.

5. An efficient semantic perception device based on instance point cloud, characterized in that, The memory stores executable code, and the one or more processors execute the executable code to implement the instance point cloud-based efficient semantic perception method of the quadruped robot in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The program is stored thereon and executed by the processor to implement the instance point cloud-based efficient semantic perception method of the quadruped robot in any one of claims 1-4.

Images