Multi-mode sensing and decision-making method, system and equipment for intelligent robot with body and medium

Abstract

The invention discloses a multi-mode sensing and decision-making method, system and device for an intelligent robot with a body and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps that a cloud end receives a user instruction, carries out task analysis and planning, and generates an atomic action sequence; according to the currently executed atomic action, a local lightweight model is called to perform real-time environment perception and multi-modal data fusion, and a precise control instruction is generated; and the robot body executes the generated precise control instruction to complete corresponding actions. According to the method, the powerful reasoning capability of the cloud and the quick response capability of the local marginal small model are combined, so that the intelligent robot can deeply understand the task intention in a complex and dynamic unstructured environment, and the delay, the calculation power consumption and the calculation power demand can be greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal perception and decision-making method, system, device and medium for embodied intelligent robots. BACKGROUND

[0002] An embodied intelligent robot is an artificial intelligence system that interacts with the environment in real time through a physical entity, combining technologies such as machine vision and natural language understanding to achieve a closed loop of perception, cognition, decision-making and action. As a core carrier in future intelligent manufacturing, medical rehabilitation, home service and other fields, breaking through the key technology of multi-modal perception and decision-making of embodied intelligent robots has great practical significance and strategic value.

[0003] Currently, the perception and decision-making of embodied intelligent robots generally use an end-to-end vision-language-action model like Google RT-2 robot, which is trained jointly with massive network text and robot data to achieve direct mapping from instructions to actions (see the paper "PaLM-E: An Embodied Multimodal Language Model").

[0004] However, end-to-end large models (such as PaLM-E with 562 billion parameters) have huge computational requirements. If deployed in the cloud, the decision-making delay is high (usually > 500 ms) due to network transmission, which cannot meet the millisecond-level real-time control requirements of robots. If local deployment is attempted, the requirements for the computing power and power consumption of the robot itself are extremely high, making it difficult to be practical. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal perception and decision-making method, system, device and medium for embodied intelligent robots to solve the problems of computing power allocation and real-time response in the perception and decision-making process of embodied intelligent robots.

[0006] To achieve the above-mentioned purpose, the present application provides the following solutions: In a first aspect, the present application provides a multi-modal perception and decision-making method for embodied intelligent robots, comprising: The cloud receives user instructions and performs task analysis and planning in combination with a visual knowledge graph, generating an atomic action sequence that conforms to physical common sense and logical constraints; According to the currently executed atomic action, a lightweight model deployed locally is called to perform real-time environment perception and multi-modal data fusion, generating precise control instructions; The robot body executes the generated precise control instructions to complete the corresponding actions; The calling of the lightweight model deployed locally to perform real-time environment perception and multi-modal data fusion specifically includes: adopting a local lightweight model to perform full-instance segmentation on the input image to generate masks of all potential targets; According to the requirements of the current atomic action, a specific target is selected from the masks of all potential targets through a multi-modal prompt guidance mechanism; The specific target is fused with the depth camera point cloud data to calculate the 6D pose information of the target object.

[0007] Optionally, the cloud receives user instructions and performs task analysis and planning in combination with a visual knowledge graph to generate an atomic action sequence that conforms to physical common sense and logical constraints, specifically including: The cloud performs semantic understanding on the natural language instructions issued by the user, performs common sense reasoning in combination with the visual knowledge graph, and decomposes the complex natural language instruction task into the atomic action sequence for sequential execution, wherein each atomic action sequence includes action type, target object, and execution parameter; The visual knowledge graph is established by the following methods: Collect and process multi-source visual data, extract target, attribute and relationship features in images and videos through computer vision technology; Use natural language processing technology to clean, classify, label and name entity recognition of the extracted features to generate readable text description; Align visual features and text descriptions across modalities to build a visual knowledge graph containing visual entities, attributes and relationships.

[0008] Optionally, the combination of the visual knowledge graph for common sense reasoning decomposes the complex natural language instruction task into the atomic action sequence for sequential execution, specifically including: Retrieve structured knowledge related to entities and actions in natural language instructions from the visual knowledge graph; Generate knowledge embedding vectors through structured knowledge representation learning in the visual knowledge graph; Fuse the knowledge embedding vectors with the semantic embedding of the natural language instruction, and guide the task planning process through the knowledge-aware attention mechanism; Based on the fused semantic vector, use thought chain reasoning technology to decompose the instruction into an atomic action sequence; The generation of knowledge embedding vectors through structured knowledge representation learning in the visual knowledge graph specifically includes: Encode entities, attributes and relationships in the structured knowledge using graph neural networks; Aggregate the multi-hop neighbor information of the encoded entities through graph attention mechanism to generate knowledge embedding vectors with context awareness.

[0009] Optionally, the knowledge embedding vector is fused with the semantic embedding of the natural language instruction, and a knowledge-aware attention mechanism is used to guide the task planning process, specifically including: The knowledge embedding vector is mapped to the same vector space as the semantic embedding of the natural language instruction through a learnable projection layer; The semantic embedding of the natural language instruction is used as a query vector, and the mapped knowledge embedding vector is used as a key vector and a value vector; for each query vector, the dot product similarity with all key vectors is calculated respectively, and then normalized by a Softmax function to generate a knowledge-aware attention weight distribution; According to the attention weight distribution, the value vectors are weighted and summed to generate a knowledge context vector corresponding to the semantic of the current natural language instruction; The knowledge context vector is fused with the semantic embedding of the natural language instruction to generate a knowledge-enhanced semantic representation; Through a trainable gating mechanism, the contribution strength of the knowledge-enhanced semantic representation to the original semantic representation is dynamically adjusted, and a final fused semantic vector is output.

[0010] Optionally, the multi-modal prompt guiding mechanism specifically includes three interactive modes of point prompt, box prompt and text prompt, wherein: The point prompt fuses the mask of the foreground point or the background point through morphological operation to filter out non-target regions; The box prompt adopts an intersection over union matching strategy to quickly locate the object most matched with the target box; The text prompt extracts text semantic features through a CLIP model, and matches the visual features of the masks of all potential targets through cosine similarity to select the potential target with the highest similarity to achieve specific target selection based on natural language.

[0011] Optionally, the generation of the precise control instruction specifically includes: Based on the 6D pose information of the target object, the joint angles of the robot arm are calculated through an inverse kinematics solver; An RRT path planning algorithm is used to plan a collision-free motion trajectory under the conditions of considering environmental obstacles and joint limits; The collision-free motion trajectory is converted into low-level motor control signals to generate precise control instructions.

[0012] Optionally, after the robot body executes the generated precise control instruction to complete the corresponding action, the embodied intelligent robot multi-modal perception and decision-making method further includes: The execution state is monitored in real time, and if the execution fails, the state information is fed back to the cloud to trigger the cloud to re-plan the task; If the execution fails, state information is fed back to the cloud to trigger the cloud to re-plan the task, and specifically includes: If the execution fails, an execution result status code and a current environment image are fed back to the cloud, the cloud performs dynamic re-planning based on the feedback information, and generates an adjusted atomic action sequence.

[0013] In a second aspect, the present application provides a body-equipped intelligent robot multi-modal perception and decision system, which comprises: A cloud large model module deployed on a cloud server, configured to receive user instructions on the cloud and perform task analysis and planning, and generate an atomic action sequence; An edge computing module deployed on the robot body, comprising a lightweight perception model and a local decision maker, configured to generate accurate control instructions by calling the local lightweight model for real-time environment perception and multi-modal data fusion according to the currently executed atomic action, and configured to monitor the execution state in real time, and feed back state information if the execution fails to trigger the cloud to re-plan the task; An execution module deployed on the robot body, configured to execute the accurate control instructions by the robot body to complete corresponding actions; A communication module configured to perform data transmission and instruction issuing between the cloud large model module and the edge computing module, and between the execution module and the edge computing module; The execution module comprises at least: A six-axis collaborative manipulator, an autonomous navigation chassis, and a multi-modal sensor, wherein the multi-modal sensor comprises at least a vision sensor, a voice sensor, a laser radar, and a depth camera.

[0014] In a third aspect, the present application provides a computer device, comprising a memory, a processor, a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the body-equipped intelligent robot multi-modal perception and decision method of any one of the first aspect.

[0015] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the body-equipped intelligent robot multi-modal perception and decision method of any one of the first aspect.

[0016] In a fifth aspect, the present application provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the steps of the body-equipped intelligent robot multi-modal perception and decision method of any one of the first aspect.

[0017] According to the specific embodiments provided by the present application, the following technical effects are disclosed: The application provides a body-intelligent robot multi-modal perception and decision method, system, device and medium. A user instruction is received by a cloud end and task analysis and planning are performed to understand a user intention, and the user intention is converted into an atomic action sequence. According to a currently executed atomic action, real-time environment perception and multi-modal data fusion are performed by calling a local lightweight model, so that perception and identification of a real complex scene can be realized, thereby generating a precise control instruction. A robot body executes the precise control instruction to realize precise physical interaction. The cloud end, the local lightweight model and the robot body end are closely interacted to build a complete 'cloud-edge-end' system. The powerful reasoning capability of the cloud end is combined with the rapid response capability of the local edge small model (lightweight model), so that the body-intelligent robot can deeply understand the task intention in a complex and dynamic unstructured environment, and can greatly reduce the delay, computing power consumption and computing power requirement. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a body-intelligent robot multi-modal perception and decision method provided by an embodiment of the present application is shown. Figure 2 A functional module diagram of a body-intelligent robot multi-modal perception and decision system provided by an embodiment of the present application is shown. Figure 3 A structural diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] The above-mentioned purposes, features and advantages of the present application can be more obvious and easy to understand. The present application will be further described in detail below with reference to the drawings and specific embodiments.

[0022] In an exemplary embodiment, a somatic intelligent robot multi-modal perception and decision-making method is provided, which is executed by a computer device, specifically by a terminal or a server, or by both the terminal and the server. In the embodiments of the present application, the method comprises the following steps 101 to 103. Wherein: In step 101, the cloud receives user instructions and performs task analysis and planning in combination with a visual knowledge graph to generate an atomic action sequence that conforms to physical common sense and logical constraints.

[0023] As an optional implementation, the specific process of step 101, in which the cloud receives user instructions and performs task analysis and planning in combination with a visual knowledge graph to generate an atomic action sequence that conforms to physical common sense and logical constraints, comprises: The cloud performs semantic understanding on the natural language instructions issued by the user, performs common sense reasoning in combination with a visual knowledge graph, and decomposes complex natural language instruction tasks into sequentially executed atomic action sequences, wherein each atomic action sequence includes an action type, a target object, and an execution parameter.

[0024] In the embodiments of the present application, the generation of the atomic action sequence adopts a Transformer-based thought chain technology to dynamically generate a combined sequence of basic actions such as navigation, recognition, grasping, and placing through a hinting engineering (also known as a context hinting or instruction engineering). For example, after receiving the instruction to “grasp object B”, a series of steps such as [A1: navigate 120 cm to the right to point A, A2: recognize object B, A3: grasp object B] are generated, wherein each step is an atomic action.

[0025] In the above implementation, by unifying multi-granularity visual tasks such as target detection and semantic segmentation into a sequence prediction problem, cross-task feature alignment and perception and recognition of real complex scenes are achieved. By introducing a visual knowledge graph, the controllability and rationality of the decision-making are improved, and factual errors are reduced, by incorporating prior knowledge into the generation process of the atomic action sequence.

[0026] As an optional implementation, the visual knowledge graph is established by the following methods: Collect and process multi-source visual data, and extract target, attribute, and relationship features in images and videos through computer vision technology; Use natural language processing technology to clean, classify, label, and perform named entity recognition on the extracted features to generate readable text descriptions; Align the visual features and the text descriptions across modalities, and construct a visual knowledge graph containing visual entities, attributes, and relationships.

[0027] Wherein, by implementing the above-mentioned embodiments, the visual knowledge graph is combined into the embodiment intelligent robot perception and decision-making method, firstly, the reliability and explainability of reasoning can be enhanced, the traditional pure data-driven large model (such as PaLM-E, RT-2) is essentially a “black box”, and the decision-making process is difficult to understand and control, and it is easy to produce absurd output that does not conform to physical common sense or logic. The visual knowledge graph provides structured prior knowledge, so that the reasoning process can be changed from “statistical association” to “logical association”; secondly, the adaptability of the embodiment intelligent robot in complex unstructured scenes can be improved, because the real world is open, dynamic, full of unknown objects and unexpected situations. The visual knowledge graph can provide common sense about object function, attribute and common use scene, help the robot to make analogical reasoning and generalization; then, the visual knowledge graph can be continuously expanded and updated with the accumulation of robot interaction experience, forming a virtuous cycle of “perception-learning-knowledge construction-reasoning”; in addition, the construction of the visual knowledge graph can use unsupervised or weakly supervised methods to mine knowledge from a large number of network images and videos, reduce the dependence on expensive manual annotation, and at the same time, the data amount required for model training can be reduced.

[0028] As an optional embodiment, the combination of the visual knowledge graph for common sense reasoning divides the complex natural language instruction task into the sequential execution of the atomic action sequence, specifically including: Retrieving structured knowledge related to entities and actions in natural language instructions from the visual knowledge graph; Generating knowledge embedding vectors through structured knowledge representation learning in the visual knowledge graph; Fusing the knowledge embedding vectors with the semantic embedding of the natural language instruction, and guiding the task planning process through the knowledge-aware attention mechanism; Based on the fused semantic vector, using the thought chain reasoning technology to divide the instruction into an atomic action sequence; Wherein, the generating knowledge embedding vectors through structured knowledge representation learning in the visual knowledge graph specifically includes: Using graph neural networks to encode entities, attributes and relationships in the structured knowledge; Aggregating the multi-hop neighbor information of the encoded entities through graph attention mechanism to generate knowledge embedding vectors with context awareness.

[0029] Wherein, by implementing this embodiment, the knowledge embedding vectors generated through knowledge graph representation learning can integrate high-level semantic information (such as “fragile”, “graspable”, “belong to tableware”) into the decision-making process. At the same time, knowledge guidance can reduce the data amount required for model training.

[0030] In the embodiments of the present application, the "structured knowledge" is a discrete and symbolic data organization form, which is usually centered on "triples", for example: (cup, has part, handle), (cup, color, red), (handle, allowed action, grasp).

[0031] In the embodiments of the present application, in order to facilitate the understanding of "multi-hop neighbors", the knowledge graph can be imagined as a huge relationship network, in which each node represents an entity (such as "cup", "table", "hand"), and each edge represents a relationship (such as "placed on", "has part", "can be used for"). Among them, "one-hop neighbors" refer to nodes directly connected to a certain node, for example, for the node "cup", its one-hop neighbors can include: "handle" (relationship: "has part"), "table" (relationship: "placed on"), "water" (relationship: "can contain"); "two-hop neighbors" refer to nodes connected to the target node through an intermediate node, that is, the neighbors of the neighbors, for example, for the node "cup", its two-hop neighbors can be found through "handle", "cup" -- (has part) --> "handle" -- (can grasp) --> "hand", so "hand" is the "two-hop neighbor" of "cup". Therefore, "multi-hop neighbors" generally refer to nodes that can be reached through multiple edges (paths), including two-hop, three-hop and further neighbors, and "multi-hop neighbor information" refers to the feature information of these indirect but related entities, which represents broader and richer context and semantic information.

[0032] In the embodiments of the present application, this step of "aggregating the multi-hop neighbor information of the entity after encoding through the graph attention mechanism" is illustrated by taking the task of "taking the red cup on the table" that the robot needs to understand: Task understanding: the cloud receives the instruction "take the red cup". It locates to the core entity "cup" and the attribute "red".

[0033] Knowledge retrieval and multi-hop reasoning: The model finds the "cup" node from the graph; One-hop information: knows that it is "red", "placed on the table", "has a handle"; Two-hop information (through 'handle'): knows that the handle "allows the grasp" action; Three-hop information (through 'grasp'): knows that the "grasp" action is "performed by the hand".

[0034] Generate a more intelligent decision: By aggregating these multi-hop information, the robot's understanding of "cup" is no longer isolated color and location, but forms a semantic network: "this is a red, placed on a wooden table, with a graspable handle, object suitable for grasping with the hand".

[0035] The above decisions directly affect the subsequent atomic action planning: When planning actions, actions that use the hand to grip the handle are generated first, rather than attempting to adhere to the cup body; When moving and grasping, the stability of the "wooden table" is expected; Knowing that the target is "red", color becomes a strong distinguishing feature when segmenting and identifying.

[0036] As an optional implementation, the knowledge embedding vector is fused with the semantic embedding of the instruction, and the task planning process is guided through a knowledge-aware attention mechanism, specifically including: The knowledge embedding vector is mapped to the same vector space as the semantic embedding of the natural language instruction through a learnable projection layer; The semantic embedding of the natural language instruction is used as the query vector, and the mapped knowledge embedding vector is used as the key vector and the value vector; for each query vector, the dot product similarity with all key vectors is calculated, and then normalized through the Softmax function to generate a knowledge-aware attention weight distribution; According to the attention weight distribution, the value vectors are weighted and summed to generate a knowledge context vector corresponding to the current natural language instruction semantics; The knowledge context vector is fused with the semantic embedding of the natural language instruction to generate a knowledge-enhanced semantic representation; Through a trainable gating mechanism, the contribution strength of the knowledge-enhanced semantic representation to the original semantic representation is dynamically adjusted, and the final fused semantic vector is output.

[0037] In the embodiments of the present application, the cloud performs complex task understanding, decomposition and macro planning by relying on the domestic "Zidongtaichu" billion-parameter cross-modal large model, and finally generates an atomic action sequence that conforms to physical common sense and logical constraints.

[0038] Step 102, according to the currently executed atomic action, a lightweight model deployed locally is called to perform real-time environment perception and multi-modal data fusion, and precise control instructions are generated.

[0039] As an optional implementation, the lightweight model deployed locally is called to perform real-time environment perception and multi-modal data fusion, specifically including: A local lightweight model (such as FastSAM model (Fast Segment Anything Model, Fast Segment Anything Model)) is used to perform full-instance segmentation on the input image (real-time image obtained through a depth camera), generating masks of all potential targets; According to the requirements of the current atomic action, a specific target is selected from the masks of all potential targets through a multi-modal prompt guidance mechanism; The specific target is fused with the depth camera point cloud data to calculate the 6D pose information (3D position + 3D rotation) of the target object.

[0040] In the embodiments of the present application, the multi-modal prompt guidance mechanism includes three interactive modes of points, boxes and texts. The point prompt fuses the masks of foreground points or background points through morphological operations to filter out non-target regions. The box prompt adopts an intersection over union (IoU) matching strategy to quickly locate the object that best matches the target box. The text prompt extracts text semantic features through a CLIP model and matches the visual features of the masks of all potential targets through cosine similarity to select the potential target with the highest similarity to achieve specific target selection based on natural language. The multi-modal prompt guidance mechanism of the present application can reduce the single-image inference time to 40 ms (the traditional SAM model (Segment Anything Model) requires 2099 ms) while maintaining high robustness, meeting the real-time requirements.

[0041] In this embodiment, the FastSAM model decouples the segmentation task into two stages: the first stage uses a CNN backbone network (such as YOLOv8-seg) for full-instance segmentation to generate masks of all potential targets; the second stage uses a prompt guidance selection mechanism (such as text-image feature matching through a CLIP model) to quickly locate the target, avoiding the overhead of running a heavy ViT encoder for each inference in the traditional SAM model, thereby achieving an inference speed of 40 ms / frame in practical applications (50 times faster than the traditional SAM model). In addition, the cloud trillion-parameter model is used for task planning and high-level decision-making, and the local FastSAM model is used for edge real-time response and local control. Compared with existing end-to-end large models, the delay and power consumption can be significantly reduced, and the response speed and reliability of the entire perception and decision-making process can be improved.

[0042] In the zero-shot transfer task, the FastSAM model achieves a bounding box AR@1000 of 63.7 on the COCO dataset, surpassing the traditional SAM model of 62.5. In downstream tasks such as edge detection and salient object segmentation, the FastSAM model exhibits comparable performance to the Transformer model, especially in small target segmentation and complex scene processing. Through optimization of the mask generation strategy, the edge blur and detail loss problems of traditional methods are effectively reduced.

[0043] As an optional embodiment, the generation of the precise control instruction in step 102 includes: Based on the 6D pose information of the target object, the joint angles of the robot arm are calculated by an inverse kinematics solver; An RRT path planning algorithm is used to plan a collision-free motion trajectory under the conditions of considering environmental obstacles and joint limits. The collision-free motion trajectory is converted into low-level motor control signals to generate precise control instructions.

[0044] In step 103, the robot body executes the generated precise control instructions to complete the corresponding action.

[0045] By implementing the above steps 101 to 103, the user instructions are received through the cloud, and the task analysis and planning are performed to understand the user intent and convert the user intent into an atomic action sequence. According to the currently executed atomic action, real-time environment perception and multi-modal data fusion are performed by calling the local lightweight model, which can realize the perception and recognition of real complex scenes, so as to generate precise control instructions. The robot body executes the precise control instructions to realize precise physical interaction. The present application closely interacts the cloud, the local lightweight model and the robot body, and constructs a complete "cloud-edge-end" system. The powerful reasoning ability of the cloud is combined with the fast response ability of the local edge small model (lightweight model), so that the embodied intelligent robot can deeply understand the task intent in a complex and dynamic unstructured environment, and can greatly reduce the delay, computing power consumption and computing power demand.

[0046] In the robot arm grasping task, the success rate of the multi-modal perception and decision method provided by the present application is 93.7%, which is higher than that of the Google RT-2 model of 82.1%.

[0047] As an optional implementation, after the robot body executes the generated precise control instructions to complete the corresponding action in step 103, step 104 of real-time monitoring of the execution state is further included. If the execution fails, the state information is fed back to the cloud to trigger the cloud to re-plan the task.

[0048] In the embodiment of the present application, if the execution fails, the state information is fed back to the cloud to trigger the cloud to re-plan the task. Specifically, if the execution fails, the execution result status code and the current environment image are fed back to the cloud. The cloud performs dynamic re-planning based on the feedback information to generate an adjusted atomic action sequence.

[0049] In this implementation, step 104 monitors the execution state to cooperate with step 101 to establish a feedback channel. If the execution fails, the state information is fed back to the cloud to trigger the re-planning task. By establishing the feedback channel, the final task success rate can be improved.

[0050] Based on the same inventive concept, the embodiment of the present application also provides a somatic intelligent robot multi-modal perception and decision system for implementing the somatic intelligent robot multi-modal perception and decision method described above. The implementation scheme for solving problems provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more somatic intelligent robot multi-modal perception and decision system embodiments provided below can refer to the limitations of the somatic intelligent robot multi-modal perception and decision method described above, which will not be repeated here.

[0051] In one exemplary embodiment, a somatic intelligent robot multi-modal perception and decision system is provided, comprising: a cloud large model module 201 deployed on a cloud server, configured to receive user instructions and perform task analysis and planning in the cloud, and generate an atomic action sequence; an edge computing module 202 deployed on the somatic intelligent robot body, comprising a lightweight perception model and a local decision maker; configured to call the local lightweight model to perform real-time environment perception and multi-modal data fusion according to the currently executed atomic action, and generate accurate control instructions. At the same time, it is used to monitor the execution state in real time, and if the execution fails, it feeds back the state information and triggers the cloud to re-plan the task; an execution module 203 deployed on the robot body, configured to execute the accurate control instructions to complete the corresponding action; a communication module 204 for data transmission and instruction issuance between the cloud large model module and the edge computing module, and between the execution module and the edge computing module.

[0052] In the embodiment of the present application, the cloud large model module 201 is constructed based on the "Zidongtai initial" 100 billion parameter multi-modal large model, which supports joint understanding and reasoning of vision-language-action.

[0053] In the embodiment of the present application, the edge computing module 202 uses an embedded AI computing platform with a computing power of not less than 100 TOPS, and at least integrates a FastSAM model.

[0054] In the embodiment of the present application, the execution module 203 at least includes a six-axis collaborative manipulator, an autonomous navigation chassis, and a multi-modal sensor, wherein the six-axis collaborative manipulator is equipped with a torque sensor at the end for grasp force control, and the autonomous navigation chassis is used to realize the movement of the somatic intelligent robot.

[0055] In the embodiments of the present application, the multi-modal sensor includes but is not limited to a visual sensor, a voice sensor, a laser radar, a depth camera, etc., wherein the depth camera is used to acquire a surrounding real-time environment image and point cloud data, and the laser radar is used to scan the surrounding environment and construct a map. The multi-modal sensor can rely on a cloud large model module (such as “Zidong Taichu”) and an edge computing module to cooperatively infer, so as to realize real-time and high-precision perception and semantic understanding of the environment.

[0056] In the embodiments of the present application, the communication module 204 adopts a gRPC high-performance remote procedure call protocol, supports low-delay transmission of structured data, and has an instruction buffer and a timeout retransmission mechanism.

[0057] Among them, by implementing this implementation, the cloud large model module 201 provides general cognition and common sense, understands the user's intention, the edge computing module 202 is responsible for real-time perception and adaptation of the specific environment, and the execution module 203 realizes precise physical interaction, thereby constructing a complete “cloud-edge-end” system. The three are tightly coupled through an efficient communication module 204, and are optimized through continuous closed-loop feedback, so that the embodied intelligent robot can “think”, “observe” and “act” like a person, and reliably complete various tasks in a real, complex and dynamic unstructured environment. In addition, the embodiments of the present application rely on the cross-modal platform of “Zidong Taichu” of the Chinese Academy of Sciences to realize the cooperative inference of the cloud ten-billion-parameter multi-modal large model and the edge 100TOPS computing small model, thereby breaking through the traditional robot computing bottleneck. Compared with the Google PaLM-E model (562 billion parameters deployed in the cloud alone), the model inference delay of the present system is reduced by 42%, and the edge computing power consumption is reduced by 65%. In the embodiments of the present application, taking a specific task “please take a bottle of water from the dining table for me” as an example, the execution process is described in detail: ① Instruction uploading and cloud analysis (0.5-2 seconds): The user issues an instruction through voice or an App The instruction is uploaded to the cloud “Zidong Taichu” large model module The large model performs multi-modal understanding: identifies “dining table” (scene), “water bottle” (target object), “take” (action), and “give me” (final goal).

[0058] The large model performs task planning and generates an atomic action sequence: [A1: navigate to the dining table, A2: locate the water bottle, A3: grab the water bottle, A4: return to the user's location, and A5: deliver].

[0059] The sequence is issued to the edge computing unit of the robot through the gRPC gateway.

[0060] ② Edge processing and real-time perception (millisecond-level cycle): Execution A1: The edge computing module receives the "navigate to the dining table" instruction. The local navigation algorithm (such as the ROS navigation stack) plans a path based on the existing map and controls the mobile chassis to avoid obstacles and move forward.

[0061] Execution A2: The robot arrives near the dining table, and the edge activates the visual processing flow.

[0062] FastSAM model: Real-time segmentation of camera images to generate masks for all objects (such as cups, plates, water bottles, etc.).

[0063] CLIP model: Match the "water bottle" text feature with all object features to find the most similar water bottle target.

[0064] Multimodal fusion: Combine depth camera data to calculate the precise 6D pose of the water bottle.

[0065] ③ Action execution and closed-loop control (second level): Execution A3: The edge sends the water bottle pose to the robot arm controller.

[0066] The controller performs inverse kinematics solving and motion planning to generate the motion trajectory of each joint of the robot arm.

[0067] The robot arm moves smoothly to the target pose, and the electric gripper executes the grasp with preset force control parameters.

[0068] The force sensor confirms the successful grasp and feeds back the signal to the edge computing module.

[0069] Execution A4 & A5: The edge triggers the return_to_user (i.e., return to user space) and deliver (i.e., broadcast message) instructions. The robot navigates back and announces "water bottle retrieved" through the voice module.

[0070] ④ Exception handling and dynamic re-planning: Scenario: If the robot arm fails to grasp (detects force sensor anomalies or visual feedback finds that the object has fallen).

[0071] Feedback: The edge computing module immediately uploads the failed state code and the current environment image back to the cloud.

[0072] Replanning: The "Zidong Taichu" large model may generate new plans based on the new feedback, such as [A3_retry: re-adjust the pose to grasp] or [A3_new: move the cup next to it first and then grasp], and reissue them. This process reflects the system's strong fault tolerance and environmental adaptability.

[0073] Example 1: Part defect monitoring in an industrial quality inspection scenario.

[0074] Implementation conditions: cloud deployment of "Zidong Taichu" large model; edge computing module is Jetson Orin NX; communication module uses gRPC high-performance remote procedure call protocol; robot system includes six-axis mechanical arm and Intel RealSense D435i depth camera.

[0075] Implementation steps: 1. The user sends the instruction "detect the surface scratch of part A" and uploads it to the cloud. The large model decomposes the task into [navigate to the detection station, take pictures of the part, analyze the image, locate the scratch, and report the results].

[0076] 2. After receiving the "analyze image" instruction, the edge starts the FastSAM model for full instance segmentation, matches the "scratch" text feature through CLIP, and locates the defect area pixel coordinates.

[0077] 3. Fuse the depth camera point cloud data to calculate the 3D coordinates of the defect, and the mechanical arm moves to the specified position marker.

[0078] Test results: Defect detection rate 99.2%, single detection cycle <150ms.

[0079] Example 2: Autonomous grabbing and carrying in logistics warehouse.

[0080] Implementation conditions: The robot platform is a mobile chassis (AGV) equipped with a mechanical arm, the edge computing module has a computing power ≥100TOPS, and the remaining conditions are the same as in Example 1.

[0081] 1. Send the cloud instruction "take the blue box on the third layer of shelf B to the packing area", and the large model generates the atomic action sequence.

[0082] 2. The edge identifies the box through FastSAM, matches "blue box" through CLIP, and calculates the 6D grabbing pose combined with depth information.

[0083] 3. The mechanical arm plans a collision-free path to complete the grabbing; if it fails, it returns the image to trigger the cloud to re-plan.

[0084] Test results: Grabbing success rate 93.7%, task completion time reduced by 35%.

[0085] In an exemplary embodiment, a computer device, which can be a server or a terminal, is provided, and an internal structure diagram thereof can be as follows: Figure 3As shown in the figure. The computer device includes a processor (CPU / GPU), a memory, an input / output interface (I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the multi-modal perception and decision-making data of the embodied intelligent robot. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor, and the multi-modal perception and decision-making method of the embodied intelligent robot can be realized.

[0086] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0087] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0088] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to implement the steps in the above method embodiments.

[0089] In an exemplary embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps in the above method embodiments.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0091] Those skilled in the art can understand and implement all or part of the processes in the above-mentioned embodiment methods, which can be completed by a computer program controlling related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0092] The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0093] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0094] The principles and implementation modes of the present application are described by applying specific examples herein, and the above-mentioned embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A multimodal perception and decision-making method for an embodied intelligent robot, characterized in that, The embodied intelligent robot multimodal perception and decision-making method includes: The cloud receives user instructions and combines them with visual knowledge graphs to perform task analysis and planning, generating atomic action sequences that conform to physical common sense and logical constraints; Based on the currently executing atomic action, a locally deployed lightweight model is invoked to perform real-time environmental perception and multimodal data fusion, generating precise control commands; The robot body executes the generated precise control commands to complete the corresponding actions; Specifically, the invocation of a locally deployed lightweight model for real-time environmental perception and multimodal data fusion includes: A local lightweight model is used to perform full instance segmentation on the input image to generate masks for all potential targets; Based on the needs of the current atomic action, a specific target is selected from the mask of all potential targets through a multimodal cueing guidance mechanism; The specific target is fused with the point cloud data from the depth camera to calculate the 6D pose information of the target object.

2. The embodied intelligent robot multimodal perception and decision-making method according to claim 1, characterized in that, The cloud-based system receives user commands and combines them with a visual knowledge graph to perform task analysis and planning, generating a sequence of atomic actions that conforms to physical common sense and logical constraints, specifically including: The cloud performs semantic understanding on the natural language commands issued by the user, combines visual knowledge graphs to perform common sense reasoning, and decomposes complex natural language command tasks into the sequentially executed atomic action sequence, wherein each atomic action sequence includes action type, target object and execution parameters; The visual knowledge graph is established in the following way: Collect and process multi-source visual data, and extract target, attribute, and relationship features from images and videos using computer vision technology; Natural language processing techniques are used to clean, classify, label, and perform named entity recognition on the extracted features to generate readable text descriptions. By aligning visual features with textual descriptions across modalities, a visual knowledge graph containing visual entities, attributes, and relationships is constructed.

3. The embodied intelligent robot multimodal perception and decision-making method according to claim 2, characterized in that, The method of combining visual knowledge graphs for common-sense reasoning decomposes complex natural language instruction tasks into sequentially executed atomic action sequences, specifically including: Retrieve structured knowledge related to entities and actions in natural language instructions from a visual knowledge graph; Knowledge embedding vectors are generated by learning the structured knowledge representation in the visual knowledge graph. The knowledge embedding vector is fused with the semantic embedding of natural language instructions, and the task planning process is guided by the knowledge-aware attention mechanism. Based on the fused semantic vector, the instructions are decomposed into atomic action sequences using the thought chain reasoning technique. Specifically, the step of generating knowledge embedding vectors through learning the structured knowledge representation in the visual knowledge graph includes: A graph neural network is used to encode the entities, attributes, and relationships in the structured knowledge. The multi-hop neighbor information of the encoded entity is aggregated through a graph attention mechanism to generate a context-aware knowledge embedding vector.

4. The embodied intelligent robot multimodal perception and decision-making method according to claim 3, characterized in that, The process of fusing the knowledge embedding vector with the semantic embedding of natural language instructions, and guiding the task planning process through a knowledge-aware attention mechanism, specifically includes: The knowledge embedding vector is mapped to the same vector space as the semantic embedding of the natural language instruction through a learnable projection layer; The semantic embedding of the natural language instruction is used as the query vector, and the mapped knowledge embedding vector is used as the key vector and value vector. For each query vector, the dot product similarity with all key vectors is calculated, and then normalized by the Softmax function to generate the knowledge-aware attention weight distribution. The value vector is weighted and summed according to the attention weight distribution to generate a knowledge context vector corresponding to the semantics of the current natural language instruction. The knowledge context vector is fused with the semantic embedding of the natural language instruction to generate a knowledge-enhanced semantic representation; By employing a trainable gating mechanism, the contribution strength of the knowledge-enhanced semantic representation to the original semantic representation is dynamically adjusted, and the final fused semantic vector is output.

5. The embodied intelligent robot multimodal perception and decision-making method according to claim 1, characterized in that, The multimodal prompting and guidance mechanism specifically includes three interaction methods: point prompts, box prompts, and text prompts, wherein: The tooltip uses morphological operations to merge the mask containing foreground or background points, filtering out non-target areas. The bounding box tooltip uses an intersection-union matching strategy to quickly locate the object that best matches the target bounding box. The text prompt extracts semantic features of the text using the CLIP model and performs cosine similarity matching with the visual features of the masks of all potential targets, selecting the potential target with the highest similarity to achieve specific target selection based on natural language.

6. The embodied intelligent robot multimodal perception and decision-making method according to claim 1, characterized in that, The generation of precise control commands specifically includes: Based on the 6D pose information of the target object, the angles of each joint of the robotic arm are calculated using an inverse kinematics solver. The RRT path planning algorithm is used to plan collision-free motion trajectories under the conditions of environmental obstacles and joint constraints. The collision-free motion trajectory is converted into a low-level motor control signal to generate precise control commands.

7. The embodied intelligent robot multimodal perception and decision-making method according to claim 1, characterized in that, After the robot body executes the generated precise control commands and completes the corresponding actions, the embodied intelligent robot multimodal perception and decision-making method further includes: The execution status is monitored in real time. If the execution fails, the status information is reported to the cloud, which triggers the cloud to re-plan the task. Specifically, the step of reporting status information to the cloud if execution fails, triggering the cloud to re-plan the task, includes: If the execution fails, the execution result status code and the current environment image are sent back to the cloud. The cloud performs dynamic replanning based on the feedback information and generates an adjusted sequence of atomic actions.

8. A multimodal perception and decision-making system for an embodied intelligent robot, characterized in that, The embodied intelligent robot multimodal perception and decision-making system includes: The cloud-based large model module, deployed on a cloud server, is used to receive user instructions in the cloud, perform task parsing and planning, and generate atomic action sequences. The edge computing module, deployed on the robot body, includes a lightweight perception model and a local decision-maker. It is used to call the local lightweight model to perform real-time environmental perception and multimodal data fusion based on the currently executed atomic action, and generate precise control commands. At the same time, it is used to monitor the execution status in real time. If the execution fails, it will feed back status information and trigger the cloud to re-plan the task. An execution module, deployed on the robot body, is used by the robot body to execute the precise control commands and complete the corresponding actions. The communication module is used for data transmission and command issuance between the cloud-based large model module and the edge computing module, and between the execution module and the edge computing module. The execution module includes at least: The system includes a six-axis collaborative robotic arm, an autonomous navigation chassis, and multimodal sensors, wherein the multimodal sensors include at least a vision sensor, a voice sensor, a lidar, and a depth camera.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the embodied intelligent robot multimodal perception and decision-making method according to any one of claims 1-7.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the embodied intelligent robot multimodal perception and decision-making method according to any one of claims 1-7.

Images