Leg-foot type robot sensing system based on multi-mode visual large model and robot

Through a leg-foot robot perception system based on a multimodal vision model, combined with multi-source sensors and optimized algorithms, the perception challenges in complex environments are solved, higher perception accuracy and adaptability are achieved, and the intelligence of action decisions is improved level of transformation.

CN118907141APending Publication Date: 2024-11-08TONGJI UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411108116.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Leg-foot robots face perception challenges in complex environments, such as accurate obstacle identification and efficient path planning. Existing perception solutions are difficult to achieve comprehensive and accurate environmental mapping under extreme conditions, affecting the accuracy and stability of their output.

Method used

Adopt a leg foot robot perception system based on a multimodal vision model, combining sensor unit, object detection unit, semantic segmentation unit, perception data processing unit and intelligent control unit, and aggregation through YOLO detector, text encoder, and visual language path Components such as network, image encoder, prompt encoder and mask decoder deeply integrate multi-source sensor data and algorithms to achieve complementary and fusion of cross-modal information, and improve perception accuracy and intelligence of action decisions.

Benefits of technology

It significantly enhances the perception accuracy and adaptability of robots in complex environments, improves object recognition and interaction capabilities, improves the intelligence level of action decision-making, and opens up a new path for the development of leg-foot robot technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118907141A_ABST
    Figure CN118907141A_ABST
Patent Text Reader

Abstract

The invention provides a leg-foot type robot sensing system based on a multi-mode visual large model, and the system comprises a sensor unit which is disposed on a leg-foot type robot and is used for obtaining image data and various sensing data in an environment; the target detection unit is connected to the sensor unit and is used for identifying a target object in the environment based on the image data and outputting a target object enhanced feature map; the semantic segmentation unit is connected to the target detection unit and is used for carrying out contour segmentation on the target object enhanced feature map and outputting a segmentation mask with the same size as the image data; the perception data processing unit is connected to the semantic segmentation unit and is used for receiving and processing various perception data and segmentation masks acquired by the sensor unit; and the intelligent control unit is connected to the sensing data processing unit and is used for generating a corresponding control instruction according to a processing result of the sensing data processing unit. According to the invention, the environment perception, object recognition and interaction capabilities of the leg-foot type robot can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot perception and control technology, and in particular to a leg-foot robot perception system and a robot based on a multi-modal visual large model. Background Art

[0002] In today's wave of technological progress, robotics technology has shown unlimited development potential and a wide range of applications. In particular, in the field of robot perception technology, with the rapid development of artificial intelligence, especially computer vision and machine learning technology, breakthrough progress has been made. The improvement of these technologies has greatly enhanced the perception and understanding ability of robots in complex environments, laying a solid foundation for their advanced functions in autonomous navigation and operation.

[0003] Especially in the field of legged robots, these robots are inspired by the walking mechanism of natural organisms and imitate the gait of insects or mammals, giving robots the unique ability to move freely on rugged surfaces and changing environments. Although legged robots have shown great potential in exploring unknown areas and performing difficult tasks, they still face challenges in perception challenges brought by complex environments, such as accurate obstacle recognition and efficient path planning.

[0004] Although the current mainstream perception solutions integrate high-end sensors such as LiDAR and cameras, it is still difficult to achieve comprehensive and accurate environmental mapping under extremely complex conditions. At the same time, object detection and semantic segmentation technologies in the field of computer vision still face challenges in practical applications such as complex and variable environments and signal interference, which affect the accuracy and stability of their output.

[0005] Therefore, how to further improve the environmental perception, object recognition and interaction capabilities of legged robots has become a technical problem that urgently needs to be solved in this field. Summary of the invention

[0006] In order to overcome the defects of the above-mentioned related technologies, the present invention provides a legged robot perception system and a robot based on a multimodal visual large model, which can significantly improve the environmental perception, object recognition and interaction capabilities of the legged robot.

[0007] According to one aspect of the present invention, a legged robot perception system based on a multimodal visual large model is provided, comprising:

[0008] A sensor unit is provided on the legged robot and is used to obtain image data and various perception data in the environment;

[0009] a target detection unit, connected to the sensor unit, for identifying a target object in the environment based on the image data and outputting an enhanced feature map of the target object;

[0010] A semantic segmentation unit, connected to the target detection unit, configured to perform contour segmentation on the target object enhanced feature map and output a segmentation mask having the same size as the image data;

[0011] A perception data processing unit, connected to the semantic segmentation unit, for receiving and processing the various perception data acquired by the sensor unit and the segmentation mask;

[0012] An intelligent control unit is connected to the perception data processing unit and is used to generate corresponding control instructions according to the processing results of the perception data processing unit.

[0013] In some embodiments, the target detection unit includes:

[0014] A YOLO detector is used to extract multi-scale image features of the target object from the image data;

[0015] Text encoder, which is used to encode text into text embeddings;

[0016] The visual language path aggregation network is connected to the YOLO detector and the text encoder respectively, and is used to enhance the representation of text and image by utilizing the cross-modal fusion between the image features and the text embedding, and outputs a target object enhanced feature map that fuses the image features and the text embedding.

[0017] In some embodiments, the YOLO detector includes:

[0018] An image encoder, which uses a lightweight convolutional neural network structure and a Darknet framework as the main architecture of the network structure, is used to receive the image data and convert it into an image feature representation to capture visual information in the image;

[0019] a path aggregation network, connected to the image encoder, for performing cross-scale and cross-modal feature fusion based on the image features extracted by the image encoder, and constructing a multi-scale feature pyramid, thereby enhancing the representation capability of the image features extracted by the image encoder;

[0020] The head is connected to the path aggregation network and is used to further process the image features enhanced by the path aggregation network to generate multi-scale image features, including target location, category and embedding information.

[0021] In some embodiments, the text encoder is used to extract corresponding text embedding from input text data, and the text embedding is used to capture the semantic information and grammatical structure of the text.

[0022] In some embodiments, the visual language path aggregation network adopts a top-down and bottom-up dual path structure and utilizes the cross-modal fusion between the image features and text embedding to enhance the representation of text and images.

[0023] In some embodiments, the semantic segmentation unit includes:

[0024] Image encoder, which takes an image as input, extracts visual features of the image, and outputs image embedding;

[0025] The prompt encoder is used to input the prompt, convert the prompt into a semantic representation, and output the prompt embedding;

[0026] The mask decoder is connected to the image encoder and the prompt encoder respectively, and is used to input the image embedding and the prompt embedding, and combine the two to output the segmentation mask.

[0027] According to another aspect of the present application, a leg-foot robot is provided, comprising:

[0028] A robot mechanical body, used to support the robot;

[0029] And the legged robot perception system based on the multimodal visual large model as mentioned above is connected to the robot mechanical body to control the movement of the robot mechanical body.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] The present invention constructs a legged robot perception system based on a multimodal visual large model. The system deeply integrates advanced mechanical engineering design, multi-source sensor integration technology, and optimized and upgraded target detection and semantic segmentation algorithms, aiming to build an all-round and deep environmental perception framework. This system can not only significantly enhance the perception accuracy and adaptability of robots in complex environments, but also improve the intelligent level of action decision-making through the complementarity and fusion of multimodal information, opening up a new path for the future development of legged robot technology and setting a new benchmark for technological innovation and application expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0033] Figure 1 The module diagram of the legged robot perception system based on the multimodal visual large model of the present invention is shown. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined. Therefore, the actual execution order may be changed according to the actual situation.

[0037] Figure 1 The module diagram of the legged robot perception system based on the multimodal visual large model of the present invention is shown. Figure 1 As shown, the legged robot perception system based on a multimodal visual large model provided in the present application includes: a sensor unit 1, a target detection unit 2, a semantic segmentation unit 3, a perception data processing unit 4 and an intelligent control unit 5.

[0038] The sensor unit 1 is arranged on the legged robot and is used to obtain image data and various perception data in the environment; the various perception data may include distance data and force data. The present application cleverly combines the precise mechanical structure of the legged robot with the diversified sensor unit 1, which not only ensures the stable support and agile movement of the legged robot in various terrains, but also provides a solid basic platform for environmental perception. The sensor unit 1 comprehensively covers the capture of various physical signals such as images, distances, and force feedback, laying a rich and accurate information foundation for subsequent data analysis.

[0039] Specifically, the sensor unit 1 may include an inertial measurement unit, which is arranged at the center of the leg-legged robot or on each leg, and is used to measure the acceleration and angular velocity of the leg-legged robot. A force / torque sensor is arranged at the joint of the leg-legged robot or the end of the leg, and is used to measure the force and torque of the joint or the end effector. A contact sensor is arranged at the end of the leg or the foot of the leg-legged robot, and is used to detect the contact force and contact state between the robot and the ground or other objects. Distance sensors, such as high-precision laser radars, are arranged at different parts of the leg-legged robot to measure the distance between the leg-legged robot and obstacles or the height of the ground. With its accurate distance and height measurement capabilities, the laser radar can scan the terrain in detail to form an accurate three-dimensional map, which can provide the leg-legged robot with an in-depth understanding of the terrain structure and obstacle distribution. Visual sensors, such as high-definition cameras or depth cameras, are arranged on the head of the leg-legged robot or other appropriate positions for visual perception, target recognition and environmental understanding. As the main collector of visual information, the camera can continuously capture rich images of the surrounding environment, covering static backgrounds and dynamic entities, and can provide intuitive visual data for environmental modeling. Encoders are set at the joints or actuators of the legged robot to measure the joint angles or actuator positions, providing precise motion control and positioning feedback. Pressure sensors are set at the ends of the legs or key parts of the legged robot to measure the pressure distribution on the contact surface. As can be understood by those skilled in the art, the specific locations of these sensors can vary depending on the design and application scenarios of the legged robot to maximize its functionality and performance.

[0040] The target detection unit 2 is connected to the sensor unit 1 and is used to identify the target object in the environment based on the image data and output the enhanced feature map of the target object. Specifically, the target detection unit 2 can rely on the YOLOv8 framework, adopt visual language modeling and pre-training methods on a large number of data sets, and is used to detect and locate objects in the environment. The target detection unit 2 is not limited by the predefined object categories and can adapt to target detection tasks in different environments and scenarios. Through the use of visual language modeling and large-scale pre-training data sets, the target detection unit 2 breaks through the limitation of the traditional model's dependence on predefined categories and demonstrates excellent pan-environmental adaptability.

[0041] Furthermore, the target detection unit 2 can perform parallel analysis on the images captured by the visual sensor, quickly identify and mark various potential obstacles, and attach precise position coordinates and bounding boxes, which can improve the instant response capability of the legged robot.

[0042] In some embodiments, the target detection unit 2 includes:

[0043] YOLO detector, used to extract multi-scale image features of target objects from image data;

[0044] In some embodiments, the YOLO detector includes: an image encoder, which selects a lightweight convolutional neural network structure and uses the Darknet framework as the main architecture of the network structure, for receiving image data and converting it into image feature representation to capture visual information in the image; a path aggregation network, connected to the image encoder, for performing cross-scale and cross-modal feature fusion based on the image features extracted by the image encoder, and constructing a multi-scale feature pyramid, thereby enhancing the representation capability of the image features extracted by the image encoder; a head, connected to the path aggregation network, for further processing the image features enhanced by the path aggregation network to generate multi-scale image features, including target location, category, and embedding information.

[0045] Furthermore, the YOLO detector can use Darknet backbone image coding technology and adopt Darknet as its core image coding module. As a lightweight and efficient convolutional neural network (CNN) framework, Darknet not only ensures the effective capture of subtle features in the input image data, but also maintains computational efficiency, laying a solid foundation for subsequent target positioning and recognition. The Darknet backbone structure extracts hierarchical and information-rich feature maps through deep convolution operations, providing strong feature support for object detection tasks.

[0046] A text encoder is used to encode text into text embeddings; in some embodiments, the text encoder is used to extract corresponding text embeddings from input text data, and the text embeddings are used to capture the semantic information and grammatical structure of the text. The text encoder is designed specifically for extracting high-quality semantic embeddings from text data. The encoder goes beyond the boundaries of traditional text processing models and demonstrates enhanced visual semantic integration capabilities. By converting text into an embedded representation in a high-dimensional vector space, it not only retains the deep meaning and structural information of the text, but also promotes the bridging of text and visual data, achieving cross-modal semantic alignment. Furthermore, when processing specific types of input such as titles or quoted text, n-gram analysis can be used. This algorithm is good at extracting key noun phrases from text streams, providing key contextual information for subsequent semantic understanding and matching.

[0047] The visual language path aggregation network is connected to the YOLO detector and the text encoder respectively, and is used to enhance the representation of text and images by cross-modal fusion between image features and text embeddings, and output an enhanced feature map of the target object that fuses image features and text embeddings. In some embodiments, the visual language path aggregation network adopts a top-down and bottom-up dual path structure, and enhances the representation of text and images by cross-modal fusion between image features and text embeddings, constructs a multi-level image feature pyramid structure, and effectively integrates multi-scale image features. The visual language path aggregation network is reparameterizable. The visual language path aggregation network significantly enhances the model's comprehensive understanding of visual and language information by cross-modally fusing image features and text embeddings.

[0048] Specifically, the visual-language path aggregation network can utilize the text-guided context-aware pooling layer and the image pooling attention mechanism to deepen the interaction between image features and text features, enhance the visual-language path aggregation network's visual semantic expressiveness in an open vocabulary environment, and improve the ability to understand and interpret complex scenes.

[0049] The semantic segmentation unit 3 is connected to the target detection unit 2, and is used to perform contour segmentation on the enhanced feature map of the target object and output a segmentation mask with the same size as the image data; in some embodiments, the semantic segmentation unit 3 includes: an image encoder, a prompt encoder, and a mask decoder.

[0050] The image encoder is used to input an image, extract the visual features of the image, and output an image embedding. Specifically, the image encoder can use deep learning technology to efficiently extract a detailed visual feature map from the original image, such as extracting low-level features of the image such as edges and textures, while capturing higher-level structural information, providing a solid visual foundation for subsequent semantic understanding.

[0051] The prompt encoder is used to input the prompt, convert the prompt into a semantic representation, and output the prompt embedding. Furthermore, the prompt encoder can encode the instructions or natural language descriptions provided by the user and convert them into meaningful semantic vectors. Through advanced natural language processing technology, the intention in the prompt is accurately decoded and converted into a machine-understandable form, so that the system can focus on specific parts or objects in the image according to the user's instructions.

[0052] The mask decoder is connected to the image encoder and the prompt encoder respectively, and is used to input the image embedding and the prompt embedding, and combine the two to output the segmentation mask. Furthermore, the mask decoder carries the visual features output by the image encoder and the semantic vector generated by the prompt encoder, and can generate a refined segmentation mask through deep interaction and fusion. This process not only relies on the efficient integration of the two types of information, image embedding and prompt embedding, but also requires the use of complex attention mechanisms and prediction algorithms to ensure the accuracy of the mask and the semantic fit. The final output segmentation mask can accurately mark the required areas in the image, providing a direct basis for subsequent decision-making and operations.

[0053] Furthermore, the semantic segmentation unit 3 can perform in-depth interpretation of the image at the pixel level and classify each pixel into a preset semantic category, covering terrain type, vegetation cover, water body, etc., which can deepen the legged robot's understanding of the environment and optimize its spatial cognition and decision-making capabilities.

[0054] The core capability of the semantic segmentation unit 3 is to accurately distinguish and define the boundaries and categories of each component in the image, giving each area a deep semantic understanding. Especially in the scene application of legged robots, this model enables legged robots to have unprecedented target tracking and recognition accuracy, greatly improving their adaptability and task execution in complex environments. The semantic segmentation unit 3 consists of three core components: image encoder, prompt encoder and mask decoder, forming an efficient information processing pipeline. In the system's reasoning process, this ingenious design shows its unique advantages.

[0055] The perception data processing unit 4 is connected to the semantic segmentation unit 3 and is used to receive and process the various perception data and segmentation masks acquired by the sensor unit 1 .

[0056] As the core of the system, the perception data processing unit 4 aggregates and integrates a large amount of information from sensors such as visual sensors and lidar, and can coordinate the real-time position information and task requirements of the legged robot for comprehensive analysis. This process not only involves obstacle positioning and the use of terrain maps, but also covers the dynamic calculation of path planning and obstacle avoidance strategies to ensure the efficiency and safety of the robot's actions.

[0057] The intelligent control unit 5 is connected to the perception data processing unit 4 and is used to generate corresponding control instructions according to the processing results of the perception data processing unit 4. The intelligent control unit 5 can intelligently generate control instructions based on the processing results of the perception data processing unit 4 to guide the actions of the legged robot, including but not limited to speed adjustment, change of travel direction and selection of obstacle avoidance measures, which reflects a high degree of autonomy and adaptability. The legged robot executes the instructions accordingly, moves along the established optimal path, flexibly avoids obstacles, and accurately locates the target. For example, when a specific target to be inspected is found, the motion parameters can be adjusted autonomously until it reaches the target point and carries out the predetermined task.

[0058] The present application also provides a leg-foot robot, comprising:

[0059] A robot mechanical body, used to support the robot;

[0060] And the legged robot perception system based on the multimodal visual large model as mentioned above is connected to the robot mechanical body to control the movement of the robot mechanical body.

[0061] In summary, this application 1) cleverly combines the precise mechanical structure and diversified sensor units of the legged robot, which not only ensures the stable support and agile movement of the legged robot in various terrains, but also provides a solid basic platform for environmental perception; 2) is equipped with a highly optimized target detection unit and semantic segmentation unit. The former is committed to accurately identifying environmental objects, while the latter focuses on the detailed distinction of object contours. Together, the two enhance the level of environmental understanding of the legged robot.

[0062] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the appended claims.

Claims

1. A legged robot perception system based on a multimodal visual large model, characterized by: include: A sensor unit, which is arranged on the legged robot and is used to obtain image data and various perception data in the environment; a target detection unit, connected to the sensor unit, for identifying a target object in the environment based on the image data and outputting an enhanced feature map of the target object; A semantic segmentation unit, connected to the target detection unit, configured to perform contour segmentation on the target object enhanced feature map and output a segmentation mask having the same size as the image data; A perception data processing unit, connected to the semantic segmentation unit, for receiving and processing the various perception data acquired by the sensor unit and the segmentation mask; An intelligent control unit is connected to the perception data processing unit and is used to generate corresponding control instructions according to the processing results of the perception data processing unit.

2. The legged robot perception system based on a multimodal visual large model as claimed in claim 1, characterized in that: The target detection unit comprises: A YOLO detector is used to extract multi-scale image features of the target object from the image data; Text encoder, which is used to encode text into text embeddings; The visual language path aggregation network is connected to the YOLO detector and the text encoder respectively, and is used to enhance the representation of text and image by utilizing the cross-modal fusion between the image features and the text embedding, and outputs a target object enhanced feature map that fuses the image features and the text embedding.

3. The legged robot perception system based on a multimodal visual large model as claimed in claim 2, characterized in that: The YOLO detector includes: An image encoder, which uses a lightweight convolutional neural network structure and a Darknet framework as the main architecture of the network structure, is used to receive the image data and convert it into an image feature representation to capture visual information in the image; a path aggregation network, connected to the image encoder, for performing cross-scale and cross-modal feature fusion based on the image features extracted by the image encoder, and constructing a multi-scale feature pyramid, thereby enhancing the representation capability of the image features extracted by the image encoder; The head is connected to the path aggregation network and is used to further process the image features enhanced by the path aggregation network to generate multi-scale image features, including target location, category and embedding information.

4. The legged robot perception system based on a multimodal visual large model as claimed in claim 2, characterized in that: The text encoder is used to extract corresponding text embedding from input text data, and the text embedding is used to capture the semantic information and grammatical structure of the text.

5. The legged robot perception system based on a multimodal visual large model as claimed in claim 2, characterized in that: The visual-language path aggregation network adopts a top-down and bottom-up dual-path structure and utilizes the cross-modal fusion between the image features and text embedding to enhance the representation of text and images.

6. The legged robot perception system based on a multimodal visual large model as claimed in claim 1, characterized in that: The semantic segmentation unit comprises: Image encoder, which takes an image as input, extracts visual features of the image, and outputs image embedding; The prompt encoder is used to input the prompt, convert the prompt into a semantic representation, and output the prompt embedding; The mask decoder is connected to the image encoder and the prompt encoder respectively, and is used to input the image embedding and the prompt embedding, and combine the two to output the segmentation mask.

7. A leg-foot robot, characterized in that: include: A robot mechanical body, used to support the robot; And the legged robot perception system based on the multimodal visual large model as described in any one of claims 1 to 6 is connected to the robot mechanical body and is used to control the movement of the robot mechanical body.

Citation Information

Cited By

  • Robot cognitive decision-making method and system

    CN120023809A

  • Obstacle identification method based on multi-sensor fusion

    CN120496019A