Universal system of intelligent robot with body
Through the core neural network system and multimodal perception fusion technology, the problems of insufficient motion control of embodied intelligent robots are solved, and high-precision motion and natural human-computer interaction in complex environments are realized, thus improving the universality of robots Sex and practicality.
Patent Information
- Application Number
- CN202411208275.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing embodied intelligent robots are not mature enough in motion control, difficult to achieve highly flexible and accurate motion, limited perception and recognition capabilities, and not friendly enough in human-computer interaction interfaces, especially in complex and dynamic environments.
The core neural network system is adopted, including large language modules, memory modules, perception modules and motion control modules, and combined with natural language processing, model prediction control and multimodal perception fusion technology to realize the robot's understanding of users' natural language instructions and environmental perception. Generate precise control instructions to drive the mechanical system to perform tasks.
It improves the motion flexibility and accuracy of the robot in complex environments, enhances perception and recognition capabilities, realizes natural human-computer interaction, is suitable for use scenarios with high precision and dynamic adjustment, and improves versatility and practicality.
Smart Images

Figure CN118906074A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent robots, and in particular relates to a universal system of embodied intelligent robots. Background Art
[0002] Embodied intelligent robots are an important development direction in the field of artificial intelligence. They are guided by the theory of embodied intelligence and integrate mature industrial robot technology with emerging artificial intelligence technology. Embodied intelligent robots refer to intelligent systems or machines that can interact with the environment in real time through perception and interaction. They not only have a high-precision adaptive perception system, but also can make autonomous decisions and automatically perform various tasks. This kind of robot uses the "body" as a carrier to understand and adapt to the environment through perception and action, realizing the deep integration of intelligence and body. Embodied intelligent robots include the following parts: 1. Perception system: using a multimodal pan-sensor system, it can continuously and uninterruptedly perceive the surrounding environment and the robot's own state, providing accurate information for decision-making. 2. Motion system: based on a closed-loop control system, combined with dynamics and kinematics algorithms, high-precision motion control is achieved. For example, the "visual servo" system is used to accurately plan and adjust the robot's motion trajectory. 3. World model: the cognitive framework built by the intelligent body based on its own structural characteristics to interpret the world. It is dynamically changing, and every interaction with the environment will have an impact on it.
[0003] Existing embodied intelligent robots have the following problems: 1. Motion control is not mature enough. Embodied intelligent robots still face challenges in motion control, and it is difficult to achieve highly flexible and precise movements. This is mainly because the existing motion control algorithms and mechanical structures cannot fully simulate the flexibility and precision of humans, which may limit the robots' ability to perform complex tasks. 2. Limited perception and recognition capabilities. Although robots are equipped with a variety of sensory organs (such as cameras, lidar, tactile sensors, etc.), their perception and recognition capabilities of dynamic environments are still not strong enough. This limits the application capabilities of robots in complex and changing environments. 3. The human-computer interaction interface is not friendly enough. Although the current human-computer interaction methods have made some progress, they still need to be optimized. For example, natural language processing technology is not yet fully mature, which may cause misunderstandings when robots understand human language; at the same time, robots lack the ability to communicate emotionally, making it difficult to have natural and fluent conversations like humans.
[0004] Therefore, the present invention provides a general system of embodied intelligent robots to solve some of the problems existing in the above-mentioned background technologies. Summary of the invention
[0005] In view of the problems raised by the above background technology, the purpose of the present invention is to provide a general system of embodied intelligent robots.
[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is as follows:
[0007] A universal system of an embodied intelligent robot, the universal system comprising a core neural network and a main body device, the core neural network comprising a large language module, a memory module, a perception module and a motion control module, the main body device comprising a drive system, a mechanical system and a sensor system;
[0008] The large language module is used to receive interactive tasks issued by the user through natural language dialogue and send feedback confirmation information to the user; the large language module is also used to generate motion planning information and send the motion planning information to the motion control module; the feedback confirmation information includes perception information; the large language module adopts any language model in the BERT series, GPT-3 series or GPT-4 series;
[0009] The memory module is responsible for storing and retrieving knowledge information associated with the interactive task, wherein the knowledge information includes experience information;
[0010] The perception module is used to receive the perception information request sent by the large language module and return the corresponding perception information. The perception module integrates multiple perception methods for comprehensive perception of the environment;
[0011] The motion control module is used to generate control instructions based on the planning information generated by the large language module, combined with the knowledge information stored and retrieved by the memory module and the perception information provided by the perception module, and send the control quality to the drive system in the main device; the motion control module adopts any key technology of model predictive control MPC, whole body control WBC and multimodal perception fusion;
[0012] The drive system is used to receive control instructions sent by the motion control model and generate corresponding control signals;
[0013] The mechanical system includes a robot, the robot is provided with a plurality of joints and components, and the mechanical system is used to receive a control signal sent by a drive system, and then drive the joints and components of the robot according to the control signal;
[0014] The sensor system is used to collect perception information in the environment in real time and send this information to the perception module for processing.
[0015] It is further defined that the multiple perception modes of the perception module include visual perception, auditory perception and tactile perception.
[0016] It is further defined that the large language module adopts a natural language processing (NLP) algorithm to understand the key points of user instructions, and the NLP algorithm includes a word embedding function and an attention mechanism function. The word embedding function is used to convert words or phrases in natural language into vector representations in a high-dimensional space, and send the converted results to a computer for processing; the attention mechanism function is used to help the language model pay more attention to relevant parts in the input sequence when generating an output sequence.
[0017] It is further defined that the word embedding function is implemented based on the neural network model Word2Vec or GloVe.
[0018] It is further defined that the attention mechanism function is implemented based on the Seq2Seq model.
[0019] It is further defined that the model predictive control MPC performs prediction based on a known system model, and the state space model expression of the model predictive control MPC is:
[0020] x k+1 =Ax k +Bu k
[0021] where x k is the state vector of the system at time step k, u k is the control input vector at time step k, A is the state transfer matrix, and B is the input matrix;
[0022] Also includes output expressions:
[0023] y k =Cx k +Du k
[0024] y k is the output vector at time k, C is the output matrix, and D is the direct transfer matrix.
[0025] Further defined, the model predictive control MPC optimizes by predicting the system output for multiple time steps in the future; assuming that at time k, the state and control input for the next N steps are predicted, it can be expressed as:
[0026] x k+1 =Ax k +Bu k ;
[0027] x k+2 =Ax k+1 +Bu k+1 =A 2 x k +Abu k +Buk+1 ;
[0028] x k+N =A N x k +A N-1 Bu k +…+Bu k+N-1 ;
[0029] Where A is the state transfer matrix, B is the input matrix, x k+1 is the state vector of the system at time step K+1, x k+2 is the state vector of the system at time step K+2, x k+N is the state vector of the system at time step K+N.
[0030] It is further defined that the model predictive control MPC determines the optimal control input sequence by minimizing a cost function, wherein the cost function includes a weighted sum of a state error and a control input, wherein the state tracking error is used to measure the gap between the actual state and the target state, and the control input is used to control the size of the input, and the expression is as follows:
[0031]
[0032] Where N is the prediction time domain length, Q is the state error weight matrix, R is the control input weight matrix, and P is the terminal state weight matrix.
[0033] Beneficial effects of the present invention:
[0034] This solution provides a general system of embodied intelligent robots. Compared with the existing technology, the robot motion control is not mature enough and it is difficult to achieve highly flexible and precise motion, mainly because the existing motion control algorithms and mechanical structures cannot fully simulate the flexibility and precision of humans, which may limit the robot's ability to perform complex tasks. The large language module of this solution receives interactive tasks issued by users through natural language dialogues, the memory module stores and retrieves knowledge information associated with the interactive tasks, and the perception module comprehensively perceives the environment. The motion control module generates control instructions based on the planning information generated by the large language module, combined with the knowledge information stored and retrieved by the memory module and the perception information provided by the perception module, and sends the control quality to the robot, thereby realizing human-computer interaction of the robot in a real environment. In addition, the motion control module adopts model predictive control MPC, which makes decisions by predicting future system behavior and solves an optimization problem in each time step to find control inputs to minimize costs and meet constraints. It is very suitable for use scenarios that require high precision and dynamic adjustment, with strong versatility and good practicality. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The present invention can be further illustrated by the non-limiting examples given in the accompanying drawings;
[0036] Figure 1 A system template diagram of a general system embodiment of an embodied intelligent robot according to the present invention;
[0037] Figure 2 This is a step flow chart of embodiment 1 of a general system embodiment of an embodied intelligent robot of the present invention.
[0038] The symbols of the main components are explained as follows: large language module 1, memory module 2, perception module 3, motion control module 4, drive system 5, mechanical system 6, sensor system 7. DETAILED DESCRIPTION
[0039] In order to enable those skilled in the art to better understand the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.
[0040] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0041] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0042] Embodiment 1, as Figure 1As shown, a general system of an embodied intelligent robot of the present invention comprises a core neural network and a main body device, wherein the core neural network comprises a large language module 1, a memory module 2, a perception module 3 and a motion control module 4, and the main body device comprises a drive system 5, a mechanical system 6 and a sensor system 7;
[0043] The large language module 1 is used to receive interactive tasks issued by the user through natural language dialogue and send feedback confirmation information to the user; the large language module 1 is also used to generate motion planning information and send the motion planning information to the motion control module 4; the feedback confirmation information includes perception information; the large language module 1 adopts any language model of the BERT series, GPT-3 series or GPT-4 series;
[0044] The memory module 2 is responsible for storing and retrieving knowledge information associated with the interactive task, wherein the knowledge information includes experience information;
[0045] The perception module 3 is used to receive the perception information request sent by the large language module 1 and return the corresponding perception information. The perception module 3 integrates multiple perception methods for comprehensive perception of the environment;
[0046] The motion control module 4 is used to generate control instructions according to the planning information generated by the large language module 1, combined with the knowledge information stored and retrieved by the memory module 2 and the perception information provided by the perception module 3, and send the control quality to the drive system 5 in the main device; the motion control module 4 adopts any key technology of model predictive control MPC, whole body control WBC and multimodal perception fusion;
[0047] The drive system 5 is used to receive the control instructions sent by the motion control model 4 and generate corresponding control signals;
[0048] The mechanical system 6 includes a robot, which is provided with a plurality of joints and components. The mechanical system 6 is used to receive a control signal from the drive system 5, and then drive the joints and components of the robot according to the control signal;
[0049] The sensor system 7 is used to collect perception information in the environment in real time and send the information to the perception module 3 for processing.
[0050] It is further defined that the multiple perception modes of the perception module 3 include visual perception, auditory perception and tactile perception.
[0051] It is further defined that the large language module 1 adopts a natural language processing (NLP) algorithm to understand the key points of user instructions. The NLP algorithm includes a word embedding function and an attention mechanism function. The word embedding function is used to convert words or phrases in natural language into vector representations in a high-dimensional space, and send the converted results to a computer for processing; the attention mechanism function is used to help the language model pay more attention to relevant parts of the input sequence when generating an output sequence.
[0052] It is further defined that the word embedding function is implemented based on the neural network model Word2Vec or GloVe.
[0053] It is further defined that the attention mechanism function is implemented based on the Seq2Seq model.
[0054] It is further defined that the model predictive control MPC performs prediction based on a known system model, and the state space model expression of the model predictive control MPC is:
[0055] x k+1 =Ax k +Bu k
[0056] where x k is the state vector of the system at time step k, u k is the control input vector at time step k, A is the state transfer matrix, and B is the input matrix;
[0057] Also includes output expressions:
[0058] y k =Cx k +Du k
[0059] y k is the output vector at time k, C is the output matrix, and D is the direct transfer matrix.
[0060] Further defined, the model predictive control MPC optimizes by predicting the system output for multiple time steps in the future; assuming that at time k, the state and control input for the next N steps are predicted, it can be expressed as:
[0061] x k+1 =Ax k +Bu k ;
[0062] x k+2 =Ax k+1 +Bu k+1 =A 2 x k +Abu k +Buk+1 ;
[0063] x k+N =A N x k +A N-1 Bu k +…+Bu k+N-1 ;
[0064] Where A is the state transfer matrix, B is the input matrix, x k+1 is the state vector of the system at time step K+1, x k+2 is the state vector of the system at time step K+2, x k+N is the state vector of the system at time step K+N.
[0065] It is further defined that the model predictive control MPC determines the optimal control input sequence by minimizing a cost function, wherein the cost function includes a weighted sum of a state error and a control input, wherein the state tracking error is used to measure the gap between the actual state and the target state, and the control input is used to control the size of the input, and the expression is as follows:
[0066]
[0067] Where N is the prediction time domain length, Q is the state error weight matrix, R is the control input weight matrix, and P is the terminal state weight matrix.
[0068] The working principle and beneficial effects of this solution are as follows:
[0069] This solution provides a general system of embodied intelligent robots. Compared with the existing technology, the robot motion control is not mature enough and it is difficult to achieve highly flexible and precise motion, mainly because the existing motion control algorithms and mechanical structures cannot fully simulate the flexibility and precision of humans, which may limit the robot's ability to perform complex tasks. The large language module of this solution receives interactive tasks issued by users through natural language dialogues, the memory module stores and retrieves knowledge information associated with the interactive tasks, and the perception module comprehensively perceives the environment. The motion control module generates control instructions based on the planning information generated by the large language module, combined with the knowledge information stored and retrieved by the memory module and the perception information provided by the perception module, and sends the control quality to the robot, thereby realizing human-computer interaction of the robot in a real environment. In addition, the motion control module adopts model predictive control MPC, which makes decisions by predicting future system behavior and solves an optimization problem in each time step to find control inputs to minimize costs and meet constraints. It is very suitable for use scenarios that require high precision and dynamic adjustment, with strong versatility and good practicality.
[0070] The specific example of this embodiment is as follows, including the following process:
[0071] 1. The user issues an interactive task through natural language, such as "get me my phone from the drawer";
[0072] 2. The large language module 1 receives the text instruction and performs route reasoning planning through the visual sensor of the perception module 3;
[0073] Route reasoning includes command operations such as arriving at the drawer, opening the drawer, taking the mobile phone out of the drawer, bringing it to the user, and putting down the item. This text command and command operation are synchronously stored in the memory module 2;
[0074] 3. The motion control module 4 receives the command operation and executes the command operation. The motion control module 4 selects the best path based on the model predictive control MPC, navigates according to the map information stored in the memory module 2, and avoids obstacles according to the navigation path;
[0075] 4. After reaching the drawer, the robot finds a way to open the drawer through the information stored in the memory module 2, finds the mobile phone through the perception module 3, and then grabs the mobile phone through the manipulator of the mechanical system 6;
[0076] 5. Finally, the robot plans a route based on the initial starting position stored in the memory module 2 and executes the route. When the robot reaches the initial position, it searches for the user's position and finds the user's hand position through the perception module 3, and finally hands the mobile phone to the user.
[0077] like Figure 2 Embodiment 2 shown:
[0078] The similarities between this embodiment and the first embodiment are not repeated here. The specific example of this embodiment is as follows, including the following process:
[0079] 1. The user issues interactive tasks through natural language, such as "deliver food to room 301 on the 19th floor";
[0080] 2. The large language module 1 receives the text instruction and performs route reasoning planning through the visual sensor of the perception module 3;
[0081] Route reasoning includes arriving at the food pickup location, confirming receipt of the food, arriving at the elevator, waiting for the elevator to open, entering the elevator, pressing the 19th floor button, waiting for the elevator to arrive at the 19th floor, exiting the elevator, arriving at room 301, waiting for the user to pick up the food, and leaving. This text instruction and instruction operation are synchronously stored in the memory module 2;
[0082] 3. The motion control module 4 receives the command operation and executes the command operation. The motion control module 4 selects the best path based on the model predictive control MPC, navigates according to the map information stored in the memory module 2, and avoids obstacles according to the navigation path;
[0083] 4. When arriving in the elevator, the robot finds out how to press the elevator button 19 through the information stored in the memory module 2, finds the elevator button for the 19th floor through the perception module 3, and then presses the elevator button through the manipulator of the mechanical system 6;
[0084] 5. After arriving at room 301, the robot waits for food to be picked up. The sensing module 3 detects in real time whether there is a user picking up food. After the food is picked up, the robot executes the return route;
[0085] 6. Finally, the robot plans a route based on the initial starting position stored in the memory module 2 and executes the route. When the robot reaches the initial position, it continues to wait for the next user instruction.
[0086] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.
Claims
1. A general system of an embodied intelligent robot, characterized in that: The universal system comprises a core neural network and a main body device, wherein the core neural network comprises a large language module (1), a memory module (2), a perception module (3) and a motion control module (4), and the main body device comprises a drive system (5), a mechanical system (6) and a sensor system (7); The large language module (1) is used to receive an interactive task issued by a user through a natural language dialogue, and send feedback confirmation information to the user; the large language module (1) is also used to generate motion planning information, and send the motion planning information to the motion control module (4); the feedback confirmation information includes perception information; the large language module (1) adopts any language model of the BERT series, the GPT-3 series or the GPT-4 series; The memory module (2) is responsible for storing and retrieving knowledge information associated with the interactive task, wherein the knowledge information includes experience information; The perception module (3) is used to receive the perception information request sent by the large language module (1) and return the corresponding perception information. The perception module (3) integrates multiple perception methods to perform comprehensive perception of the environment; The motion control module (4) is used to generate control instructions based on the planning information generated by the large language module (1), the knowledge information stored and retrieved by the memory module (2) and the perception information provided by the perception module (3), and send the control quality to the drive system (5) in the main device; the motion control module (4) adopts any one of the key technologies of model predictive control MPC, whole body control WBC and multimodal perception fusion; The drive system (5) is used to receive control instructions sent by the motion control model (4) and generate corresponding control signals; The mechanical system (6) includes a robot, the robot is provided with a plurality of joints and components, and the mechanical system (6) is used to receive a control signal sent by the drive system (5), and then drive the joints and components of the robot according to the control signal; The sensor system (7) is used to collect perception information in the environment in real time and send the information to the perception module (3) for processing.
2. A general system of embodied intelligent robots according to claim 1, characterized in that: The multiple perception modes of the perception module (3) include visual perception, auditory perception and tactile perception.
3. The general system of an embodied intelligent robot according to claim 1, characterized in that: The large language module (1) uses a natural language processing (NLP) algorithm to understand the key points of user instructions. The NLP algorithm includes a word embedding function and an attention mechanism function. The word embedding function is used to convert words or phrases in natural language into vector representations in a high-dimensional space, and send the converted results to a computer for processing; the attention mechanism function is used to help the language model pay more attention to relevant parts in the input sequence when generating an output sequence.
4. A general system of embodied intelligent robots according to claim 3, characterized in that: The word embedding function is implemented based on the neural network model Word2Vec or GloVe.
5. The general system of an embodied intelligent robot according to claim 3, characterized in that: The attention mechanism function is implemented based on the Seq2Seq model.
6. The general system of an embodied intelligent robot according to claim 1, characterized in that: The model predictive control MPC performs prediction based on a known system model. The state space model expression of the model predictive control MPC is: x k+1 =Ax k +Bu k where x k is the state vector of the system at time step k, u k is the control input vector at time step k, A is the state transfer matrix, and B is the input matrix; Also includes output expressions: y k =Cx k +You k y k is the output vector at time k, C is the output matrix, and D is the direct transfer matrix.
7. A general system of embodied intelligent robots according to claim 6, characterized in that: The model predictive control MPC is optimized by predicting the system output for multiple time steps in the future; assuming that at time k, the state and control input for the next N steps are predicted, it can be expressed as: x k+1 =Ax k +Bu k 4 x k+2 =Ax k+1 +Bu k+1 =A 2 x k +Ash k +Bu k+1 4 x k+N =A N x k +A N-1 This k +…+This k+N-1 ; Where A is the state transfer matrix, B is the input matrix, x k+1 is the state vector of the system at time step K+1, x k+2 is the state vector of the system at time step K+2, x k+N is the state vector of the system at time step K+N.
8. The general system of an embodied intelligent robot according to claim 6, characterized in that: The model predictive control MPC determines the optimal control input sequence by minimizing a cost function, wherein the cost function includes a weighted sum of a state error and a control input, the state tracking error is used to measure the gap between the actual state and the target state, and the control input is used to control the size of the input, and its expression is as follows: Where N is the prediction time domain length, Q is the state error weight matrix, R is the control input weight matrix, and P is the terminal state weight matrix.
Citation Information
Cited By
Large-model-driven intelligent biosynthesis system and method with body
CN119479775A
Safe flight agent system for executing complex language instructions in high dynamic environment
CN119992387A
Multi-agent collaborative embodied task reasoning and robot scheduling system and method
WO2026152575A1
