Physical intelligent active perception-based body-equipped agent and control method thereof
Through the multimodal embossed model combined with the underlying motion strategy and real-time correction of the environmental interaction module, the deviation and action delay problems of embossed agents in task execution are solved, precise environmental perception and action execution are achieved, and intelligence is improved generalization performance of body.
Patent Information
- Application Number
- CN202411190886.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When the existing embodied intelligent technology combines large pre-training models with agents, it ignores the underlying motion control level, resulting in deviations and motion delays in task execution. Data processing focuses on visual data, unable to accurately perceive environmental changes, and lacks depth. Hierarchical physical interaction and generalization performance.
The control method based on a multimodal embossed model is adopted, including data processing module, perception planning module, physical intelligence module, information output module, environmental interaction module and motion control execution module, through model construction, data acquisition and processing, and task planning , motion planning and motion correction steps, combined with multimodal data and underlying motion strategies, monitor and correct motion movements in real time to achieve accurate environmental perception and action execution.
It realizes precise task execution of embodied agents in complex scenarios, avoids disconnection between task planning and motion and delay in action, enhances the ability to perceive environmental changes, and has strong generalization performance.
Smart Images

Figure CN119045379A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of embodied intelligence technology, and in particular to an embodied intelligent body based on active perception of physical intelligence and a control method thereof. Background Art
[0002] With the explosive iteration of artificial intelligence technology, embodied intelligence, as a typical method and technology in the era of artificial intelligence, has ushered in a huge development opportunity. Embodied intelligence refers to the method of combining intelligent systems with physical entities, which enables physical entities to perceive the environment, make decisions and perform corresponding actions. Therefore, the physical entity combined with embodied intelligence is also called an intelligent body. "Embodiment" is not only abstract algorithms and data, but also interacts with the world through physical form. The application scenarios of embodied intelligence are also extremely wide. Intelligent bodies can be integrated into various vertical fields such as intelligent manufacturing and service industries, such as industrial inspections and housekeeping services, so that embodied intelligence can lead the upgrading of new manufacturing, service and other industries.
[0003] Large models empower robot agents, giving them "embodied" capabilities and characteristics, and have become one of the important paradigms for embodied intelligence technology research. Research on embodied intelligence driven by large language models remains hot and is shifting from initial applications to deep integration, especially the integration of robot motion control. In a variety of complex scene tasks, a single modality is not enough to cope with complex actual scenes, and multiple modalities greatly enhance the robot's perception and understanding capabilities, enabling the robot to understand complex scenes and tasks more accurately and comprehensively. After large-scale data pre-training, multimodal large models have learned a wealth of human knowledge, giving robots the ability to make autonomous plans and decisions.
[0004] At present, the research on the combination of large prediction models and embodied agents tends to focus on high-semantic task planning, thus ignoring the underlying motion control level. As a result, although the agent can quickly make task plans, there will be deviations or action delays in the actual execution process. At the same time, in terms of data processing, most of them focus on visual data, ignoring the changes caused by the movement of the agent to the environment, and cannot accurately perceive the changes in the environment. There is a lack of deeper physical interaction and generalization performance. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an embodied intelligent body based on active perception of physical intelligence and a control method thereof.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] According to one aspect of the present invention, there is provided a method for controlling an embodied intelligent body based on active perception of physical intelligence, the method controlling the embodied intelligent body based on a multimodal embodied large model, the multimodal embodied large model comprising a data processing module, a perception planning module, a physical intelligence module, an information output module, an environment interaction module and a motion control execution module;
[0008] As a preferred technical solution, the method comprises the following steps:
[0009] Model construction: fine-tuning training is performed based on the pre-trained model to obtain a multimodal embodied large model, wherein the multimodal embodied large model also includes an information output module;
[0010] Data collection and processing: The data processing module collects multimodal data and receives semantic information in the initial environment, and performs alignment and fusion operations on the multimodal data and semantic information to obtain processed data;
[0011] Task planning: The perception planning module receives processed data, infers and perceives the environment based on the processed data to obtain perception information, generates task planning instructions based on the perception information, and outputs the perception information and task planning instructions after completion;
[0012] Motion planning: The physical intelligence module receives processed data, sensory information, and task planning instructions, and outputs motion action planning instructions based on the processed data, sensory information, and task planning instructions combined with the underlying motion strategy;
[0013] Motion correction: The information output module receives the perception information and motion action planning instructions, and distributes the perception information to the environment interaction module. The environment interaction module detects the environment in which the intelligent body is located in real time to obtain real-time environmental information, and compares the real-time environmental information with the perception information; the motion control execution module receives the motion action planning instructions and performs motion correction based on the comparison results, and outputs a signal of arrival when the action is completed; the real-time environmental information is the multimodal data and arrival signal collected during the motion of the intelligent body;
[0014] User feedback: The user interaction module receives the arrival signal output by the motion control module and replies text information to the user based on the arrival signal.
[0015] As a preferred technical solution, the model construction specifically includes: creating an instruction adjustment data set based on the task scenario in which the embodied intelligent body is located; inspecting and selecting the model base of the pre-trained model based on the instruction adjustment data set; and training the selected model base using retrieval enhancement generation and supervised fine-tuning methods.
[0016] As a preferred technical solution, the alignment and fusion operation includes the following steps:
[0017] Converting or normalizing the multimodal data and the text instructions to obtain first data, wherein the multimodal data includes a scene image, a voice instruction, force sensor data on an end effector, and text data;
[0018] The same type of first data uses the same feature extractor to extract features, and the extracted features are converted into high-level feature representations using a deep coupled autoencoder;
[0019] By coupling the hidden layers of a plurality of different deep coupled autoencoders to share and fuse information, a joint feature of the first data is extracted to obtain the second data;
[0020] The second data is spatially matched to obtain processed data.
[0021] As a preferred technical solution, the second data is represented in the form of Tokens in the feature space.
[0022] As an optimal technical solution, the motion planning specifically includes: generating underlying motion control instructions and data based on the underlying motion strategy combined with perception information and task planning instructions; encapsulating the underlying motion control instructions and data, projecting the effective information into the space through a linear layer, and outputting the motion instruction sequence at the corresponding moment.
[0023] As a preferred technical solution, the underlying motion strategy includes physical laws and motion posture data learned from multiple time-sequential physical simulators.
[0024] As a preferred technical solution, the effective information includes the state, motion law, and posture change data of the physical object to be executed that conforms to the motion strategy.
[0025] As a preferred technical solution, the motion correction specifically includes: the environment interaction module detects the environment content during the motion in real time and outputs the real-time environment information at time t;
[0026] The real-time environment information at time t is compared with the predicted perception information at time t. If there is an error in the comparison, the environment interaction module transmits the real-time environment information to the perception planning module. The perception planning module corrects the task planning instructions according to the real-time environment information to obtain the task instructions, and outputs the real-time environment information and the task instructions to the physical intelligence module.
[0027] The physical intelligence module receives real-time environment information and task instructions, and corrects the motion action planning instructions based on the real-time environment and task instructions to obtain motion action instructions;
[0028] The motion action instruction is transmitted to the motion control execution module through the information output module, and the module receives the motion action instruction and executes it;
[0029] If no error is found in the comparison, the motion control execution module executes the motion action planning instruction at time t.
[0030] According to another aspect of the present invention, there is provided an embodied intelligent body based on active perception of physical intelligence, which includes a data processing module, a perception planning module, a physical intelligence module, an information output module, an environment interaction module, a motion control execution module and a user interaction module, and control is achieved through the above method.
[0031] Compared with the prior art, the present invention has the following advantages:
[0032] 1) The present invention combines the perception information, task planning instructions and the underlying motion strategy, and generates accurate motion action planning instructions for the task planning instructions based on the combined results, thereby avoiding the problems of the task planning instructions being out of sync with the intelligent body's motion or the action execution being delayed, so that the embodied intelligent body can perform tasks more accurately;
[0033] 2) The present invention establishes an environmental interaction module to monitor the environmental changes caused by the movement of the intelligent body in real time when the intelligent body moves, and feeds back the detected environmental information to the perception planning module for task planning correction, and corrects the movement of the embodied intelligent body according to the corrected task instructions. It can accurately perceive environmental changes, realize deeper physical interaction, and has strong generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is the data flow diagram of the present invention;
[0035] Figure 2 This is a flow chart of the embodied intelligent body control method based on active perception of physical intelligence of the present invention;
[0036] Figure 3 This is a flowchart of the pre-training of the multi-modal embodied large model based on physical intelligence active perception of the present invention. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0038] Embodiment 1:
[0039] In this embodiment, a method for controlling an embodied intelligent body based on active perception of physical intelligence is provided. The method controls the embodied intelligent body based on a multimodal embodied large model. The multimodal embodied large model includes a data processing module, a perception planning module, a physical intelligence module, an information output module, an environment interaction module, and a motion control execution module. The data flows between the modules are as follows: Figure 1 shown.
[0040] The control method flow chart is as follows Figure 2 The method comprises the following steps:
[0041] S1. Model construction, its construction process is as follows Figure 3 As shown:
[0042] Select a Transformer-based pre-trained model, create a small instruction adjustment dataset based on the specific task scenario of the embodied intelligent agent, use this dataset as the corpus of the pre-trained model prompt word project to train the prompt words, and select the pre-trained model as the model base if the answer availability of the selected pre-trained model is greater than or equal to 50%, otherwise re-select the pre-trained model;
[0043] Based on the physical laws and motion posture data learned from the physical simulator, the retrieval-augmented generation method is used to enhance the generation ability of the selected pre-trained model to improve the efficiency and quality of text processing tasks, so that the embodied intelligent agent can quickly and accurately obtain and query relevant information fragments;
[0044] Then use the supervised fine-tuning (SFT) method and select the low-rank adaptation (LoRA) method to fine-tune the model and construct several<input,output> The supervised information of the pre-trained model can significantly reduce the number of parameters and computing resources required for fine-tuning while maintaining the performance of the pre-trained model, thereby improving the efficiency of fine-tuning. The cold start strategy training method is then used to first freeze the action decoder and only adjust the language model, so that the model can quickly learn new language understanding and generation capabilities while maintaining the action generation capability.
[0045] The pre-trained model obtained after the above processing is the multimodal embodied large model.
[0046] S2. Data collection and processing: The data processing module collects multimodal data and receives semantic information in the initial environment, and performs alignment and fusion operations on the multimodal data and semantic information to obtain processed data;
[0047] The specific alignment and fusion operation includes the following steps:
[0048] S21. Use word embedding technology to convert words or subwords into continuous vector representations for text, and convert symbolic text into numerical forms that can be processed by machine learning models; use scaling and cropping to normalize the pixel values of images to eliminate brightness differences between different images; perform denoising and normalization on typical information of the data class to improve data availability;
[0049] S22, the multimodal data processed in step S21 is the first data, the same type of first data is selected by the same feature extractor to perform feature extraction, and the extracted features are converted into high-level feature representations by a deep coupled autoencoder;
[0050] S23, by coupling a plurality of different hidden layers of deep coupled autoencoders to share and fuse information, extracting joint features of the first data to obtain second data, and representing the second data using feature vectors Tokens;
[0051] S24. Convert different types of second data into the same joint embedding space, match and fuse the second data in the form of feature vector Tokens in the space to obtain processed data, thereby supporting cross-modal retrieval, generation and other tasks.
[0052] S3. Task planning: This is carried out in the perception planning module. The perception planning module is divided into the perception part and the planning part. Its workflow is as follows:
[0053] The perception part receives the processed data output from the data processing module, generates the perception information of the initial environment where the embodied intelligent body is located based on the processed data, and transmits the perception information to the planning part;
[0054] The planning part receives the perception information and generates task planning instructions based on the perception information;
[0055] The perception planning module outputs the perception information and task planning instructions to the physical intelligence module and the information output module.
[0056] S4. Motion Planning:
[0057] The physical intelligence module receives processed data, perception information, and mission planning instructions;
[0058] Obtaining an underlying motion strategy, the underlying motion strategy including physical laws and motion posture data learned from multiple sequential physical simulators;
[0059] Generate low-level motion control instructions and data based on low-level motion strategies combined with perception information and task planning instructions;
[0060] The underlying motion control instructions and data are encapsulated, and the state, motion law, and posture change data of the physical objects to be executed that conform to the motion strategy are projected into the space through the linear layer, and the motion instruction sequence at the corresponding moment is output.
[0061] S5, Motion Correction:
[0062] The information output module receives the perception information and motion planning instructions, and sends the perception information to the environment interaction module, which detects the environment content O observed by the agent at time t in real time. t and the action a performed at that moment t , and according to O t and a t Generate real-time environmental information;
[0063] The real-time environment information is compared with the perception information. If there is an error in the comparison, the environment interaction module transmits the real-time environment information to the perception planning module. The perception planning module corrects the task planning instructions according to the real-time environment information to obtain the task instructions, and outputs the real-time environment information and task instructions to the physical intelligence module.
[0064] The physical intelligence module receives real-time environment information and task instructions, and corrects the motion action planning instructions based on the real-time environment and task instructions to obtain the motion action instruction T = [x ′ query ,((a0,O0),...,(a t ,O t )...)];
[0065] Movement command T = [x ′ query ,((a0,O0),...,(a t ,O t )...)] is transmitted to the motion control execution module through the information output module, and the module receives the motion action instruction and executes it;
[0066] If no error is found in the comparison, the motion control execution module executes the motion action planning instruction;
[0067] When the action is completed, the arrival signal is output to the outside.
[0068] S6. User feedback: The user interaction module receives the arrival signal output by the motion control module, and replies text information to the user based on the arrival signal.
[0069] Embodiment 2:
[0070] This embodiment provides an embodied intelligent body based on active perception of physical intelligence. The embodied intelligent body is controlled by the control method of the above embodiment. When it receives instructions from the user, it executes according to steps S1 to S6 of the above embodiment, which will not be described in detail here.
[0071] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A control method for an embodied intelligent body based on active perception of physical intelligence, characterized in that: The method controls an embodied intelligent body based on a multimodal embodied large model, wherein the multimodal embodied large model includes a data processing module, a perception planning module, a physical intelligence module, an environment interaction module, and a motion control execution module; The perception planning module receives the data processed by the data processing module, and outputs perception information and task planning instructions based on the data; the physical intelligence module receives the perception information and task planning instructions and outputs motion action planning instructions, the environmental interaction module collects environmental information in real time and compares it with the perception information, the motion control module receives the motion action planning instructions, and performs motion correction based on the comparison results to realize embodied intelligent body control.
2. According to claim 1, a method for controlling an embodied intelligent body based on active perception of physical intelligence is characterized in that: The method comprises the following steps: Model construction: fine-tuning training is performed based on the pre-trained model to obtain a multimodal embodied large model, wherein the multimodal embodied large model also includes an information output module; Data collection and processing: The data processing module collects multimodal data and receives semantic information in the initial environment, and performs alignment and fusion operations on the multimodal data and semantic information to obtain processed data; Task planning: The perception planning module receives processed data, infers and perceives the environment based on the processed data to obtain perception information, generates task planning instructions based on the perception information, and outputs the perception information and task planning instructions after completion; Motion planning: The physical intelligence module receives processed data, sensory information, and task planning instructions, and outputs motion action planning instructions based on the processed data, sensory information, and task planning instructions combined with the underlying motion strategy; Motion correction: The information output module receives the perception information and motion action planning instructions, and distributes the perception information to the environment interaction module. The environment interaction module detects the environment in which the intelligent body is located in real time to obtain real-time environmental information, and compares the real-time environmental information with the perception information; the motion control execution module receives the motion action planning instructions and performs motion correction based on the comparison results, and outputs a signal of arrival when the action is completed; the real-time environmental information is the multimodal data and arrival signal collected during the motion of the intelligent body; User feedback: The user interaction module receives the arrival signal output by the motion control module and replies text information to the user based on the arrival signal.
3. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 2, characterized in that: The model construction specifically includes: Create instructions to adjust the dataset based on the task scenario in which the embodied agent is located; Adjust the data set according to the instructions to test and select the model base for the pre-trained model; The selected model base is trained using retrieval-augmented generation and supervised fine-tuning methods.
4. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 2, characterized in that: The alignment and fusion operation includes the following steps: Converting or normalizing the multimodal data and the text instructions to obtain first data, wherein the multimodal data includes a scene image, a voice instruction, force sensor data on an end effector, and text data; The same type of first data uses the same feature extractor to extract features, and the extracted features are converted into high-level feature representations using a deep coupled autoencoder; By coupling the hidden layers of a plurality of different deep coupled autoencoders to share and fuse information, a joint feature of the first data is extracted to obtain the second data; The second data is spatially matched to obtain processed data.
5. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 4, characterized in that: The second data is represented in the feature space in the form of Tokens.
6. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 2, characterized in that: The motion planning specifically includes: Generate low-level motion control instructions and data based on low-level motion strategies combined with perception information and task planning instructions; The bottom-level motion control instructions and data are encapsulated, the effective information is projected into the space through the linear layer, and the motion instruction sequence at the corresponding moment is output.
7. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 6, characterized in that: The underlying motion strategy includes physical laws and motion posture data learned from multiple sequential physical simulators.
8. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 6, characterized in that: The effective information includes the state, motion law and posture change data of the physical object to be executed in accordance with the motion strategy.
9. The method for controlling an embodied intelligent body based on active perception of physical intelligence according to claim 2, characterized in that: The motion correction specifically includes: The environmental interaction module detects the environmental content in real time during the movement and outputs the real-time environmental information at time t; The real-time environment information at time t is compared with the predicted perception information at time t. If there is an error in the comparison, the environment interaction module transmits the real-time environment information to the perception planning module. The perception planning module corrects the task planning instructions according to the real-time environment information to obtain the task instructions, and outputs the real-time environment information and the task instructions to the physical intelligence module. The physical intelligence module receives real-time environment information and task instructions, and corrects the motion action planning instructions based on the real-time environment and task instructions to obtain motion action instructions; The motion action instruction is transmitted to the motion control execution module through the information output module, and the module receives the motion action instruction and executes it; If no error is found in the comparison, the motion control execution module executes the motion action planning instruction at time t.
10. An embodied intelligent agent based on active perception of physical intelligence, characterized in that: The intelligent agent comprises: Data processing module: The data processing module collects multimodal data in the environment and receives semantic information, and processes the data to obtain processed data; Perception planning module: receives the processed data output by the data processing module, and outputs perception information and task planning instructions based on the processed data for scene reasoning and specific task planning; Physical intelligence module: receives the processed data output by the data processing module and the perception information and task planning instructions output by the perception planning module, and outputs motion action planning instructions in combination with the underlying motion strategy, wherein the motion action planning instructions are used to guide the movement of the embodied intelligent body; Information output module: as an intermediate medium for data transmission, it receives the perception information output by the perception planning module and the motion action planning instructions output by the physical intelligence module, and distributes the perception information to the environment interaction module and the motion action planning instructions to the motion control execution module; Environmental interaction module: detects the environmental content of the embodied intelligent body during movement in real time, compares it with the perceived information, and transmits the comparison results to the motion control execution module; Motion control execution module: executes motion action planning instructions, or corrects motion actions according to comparison results and executes the corrected motion actions, and outputs arrival signals when the task is completed; User interaction module: receives the arrival signal output by the motion control module and replies text information to the user based on the arrival signal.
Citation Information
Patent Citations
Universal system of intelligent robot with body, construction method and use method
CN117549310A
Control method and system of intelligent robot with body, electronic equipment and storage medium
CN117885082A
Autonomous construction method and system for intelligent bulldozing robot with body
CN118238153A
Cited By
Motion planning method, training method and system of intelligent body model
CN119398105A
Motion planning method, training method and system for embodied intelligence model
CN119398105B
Large-model-driven intelligent biosynthesis system and method with body
CN119479775A
Action generation method, system and equipment based on multi-modal large model and medium
CN119539092A
Inspection robot control system and method based on inspection information
CN119987368A
