Embodied intelligence agent based on physical intelligence and active perception, and control method therefor

Abstract

The present invention relates to an embodied intelligence agent based on physical intelligence and active perception, and a control method therefor. The method controls the embodied intelligence agent on the basis of a multimodal embodiment large model, and the multimodal embodiment large model comprises a data processing module, a perception and planning module, a physical intelligence module, an information transmission module, an environment interaction module, and a motion control execution module. The method comprises: the perception and planning module receives data processed by the data processing module and, according to the data, outputs perception information and a task planning instruction; the physical intelligence module receives the perception information and the task planning instruction, and outputs a motion action planning instruction; the environment interaction module collects environment information in real time and compares same with the perception information; and the motion control module receives the motion action planning instruction and incorporates the comparison result to perform action correction so as to implement control of the embodied intelligence agent. Compared with the prior art, the present invention has the advantages of high generalisation performance and high task execution accuracy.

Description

An embodied intelligent agent based on physical intelligence active perception and its control method Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to an embodied intelligent agent based on physical intelligence active perception and its control method. Background Technology

[0002] With the explosive growth of artificial intelligence (AI) technology, embodied intelligence, as a typical method and technology of the AI ​​era, has ushered in tremendous development opportunities. Embodied intelligence refers to the method of combining intelligent systems with physical entities, enabling physical entities to perceive their environment, make decisions, and perform corresponding actions. Therefore, the physical entities integrated with embodied intelligence are also called intelligent agents. "Embodied" is not only about abstract algorithms and data, but also about interacting with the world through physical forms. The application scenarios of embodied intelligence are also extremely broad, integrating intelligent agents into various vertical fields such as intelligent manufacturing and service industries, such as industrial inspection and domestic services, allowing embodied intelligence to lead a new industrial upgrade in manufacturing, service, and other sectors.

[0003] Large-scale models empower robotic agents, endowing them with "embodied" capabilities and characteristics, and have become one of the important paradigms in embodied intelligence technology research. Research on embodied intelligence driven by large language models remains hot, shifting from initial applications to deep integration, especially in robot motion control. In various complex scenarios, a single modality is insufficient to cope with intricate real-world situations, while multiple modalities significantly enhance a robot's perception and understanding capabilities, enabling it to more accurately and comprehensively comprehend complex scenes and tasks. Multimodal large-scale models, after large-scale data pre-training, learn rich human knowledge, endowing robots with autonomous planning and decision-making capabilities.

[0004] Current research on combining big oracle models and embodied agents tends to focus on high-semantic task planning, neglecting the underlying motion control layer. This leads to issues such as deviations or delays in actual execution, even though the agent can quickly make task plans. Furthermore, in terms of data processing, most studies emphasize visual data, ignoring the changes in the environment caused by the agent's movements. This results in an inability to accurately perceive environmental changes, a lack of deeper physical interaction capabilities, and a deficiency in generalization performance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing an embodied intelligent agent based on physical intelligence active perception and its control method.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to one aspect of the present invention, a control method for an embodied intelligent agent based on physical intelligence active perception is provided. The method controls the embodied intelligent agent based on a multimodal embodied large model, wherein the multimodal embodied large model includes a data processing module, a perception planning module, a physical intelligence module, an information output module, an environmental interaction module, and a motion control execution module.

[0008] As a preferred technical solution, the method includes the following steps:

[0009] Model construction: A multimodal embodied large model is obtained by fine-tuning the pre-trained model. The multimodal embodied large model also includes an information output module.

[0010] Data acquisition and processing: The data processing module acquires multimodal data and receives semantic information from the initial environment, and performs alignment and fusion operations on the multimodal data and semantic information to obtain processed data;

[0011] Task planning: The perception planning module receives the processed data, infers and perceives the environment based on the processed data to obtain perception information, generates task planning instructions based on the perception information, and outputs the perception information and task planning instructions after completion.

[0012] Motion planning: The physical intelligence module receives processed data, perception information, and task planning instructions, and outputs motion planning instructions based on the processed data, perception information, and task planning instructions combined with the underlying motion strategy.

[0013] Motion correction: The information output module receives sensing information and motion planning instructions, and sends the sensing information to the environment interaction module. The environment interaction module detects the environment in which the agent is located in real time to obtain real-time environmental information, and compares the real-time environmental information with the sensing information. The motion control execution module receives the motion planning instructions and performs motion correction based on the comparison results. When the motion is completed, it outputs a positioning signal. The real-time environmental information consists of multimodal data and positioning signals collected during the agent's motion.

[0014] User feedback: The user interaction module receives the arrival signal output by the motion control module and replies with text information to the user based on the arrival signal.

[0015] As a preferred technical solution, the model construction specifically includes: creating an instruction-adjusted dataset based on the task scenario of the embodied agent; verifying and selecting the model base for the pre-trained model based on the instruction-adjusted dataset; and training the selected model base using retrieval-enhanced generation and supervised fine-tuning methods.

[0016] As a preferred technical solution, the alignment and fusion operation includes the following steps:

[0017] The first data is obtained by converting or normalizing the multimodal data and text commands. The multimodal data includes scene images, voice commands, force sensor data on the end effector, and text data.

[0018] For the same type of first data, the same feature extractor is selected for feature extraction, and the extracted features are converted into a high-level feature representation by a deep-coupled autoencoder.

[0019] By coupling the hidden layers of multiple different deep-coupled autoencoders to share and fuse information, the joint features of the first data are extracted to obtain the second data;

[0020] The processed data is obtained by spatial matching of the second data.

[0021] As a preferred technical solution, the second data is represented in the form of Tokens in the feature space.

[0022] As a preferred technical solution, the motion planning specifically includes: generating underlying motion control commands and data based on underlying motion strategies combined with perception information and task planning instructions; encapsulating the underlying motion control commands and data, projecting the effective information through a linear layer onto the space where it is located, and outputting the motion command sequence at the corresponding moment.

[0023] As a preferred technical solution, the underlying motion strategy includes physical laws and motion posture data learned from multiple time-series physical simulators.

[0024] As a preferred technical solution, the effective information includes the state, motion laws, and posture change data of the physical object to be executed in accordance with the motion strategy.

[0025] As a preferred technical solution, the motion correction specifically includes: an environmental interaction module that detects environmental content in real time during the motion process and outputs real-time environmental information at time t;

[0026] The real-time environmental information at time t is compared with the predicted perception information at time t. If an error occurs in the comparison, the environmental interaction module sends the real-time environmental information to the perception planning module. The perception planning module corrects the task planning instructions based on the real-time environmental information to obtain the task instructions, and outputs the real-time environmental information and task instructions to the physical intelligence module.

[0027] The physical intelligence module receives real-time environmental information and task instructions, and corrects the motion planning instructions based on the real-time environment and task instructions to obtain motion instructions.

[0028] The motion command is transmitted to the motion control execution module through the information output module, which receives and executes the motion command.

[0029] If no error is found during the comparison, the motion control execution module executes the motion planning instruction at time t.

[0030] According to another aspect of the present invention, an embodied intelligent agent based on physical intelligence active perception is provided. The intelligent agent includes a data processing module, a perception planning module, a physical intelligence module, an information output module, an environmental interaction module, a motion control execution module, and a user interaction module, and achieves control through the above method.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1) In this invention, sensory information, task planning instructions and underlying motion strategies are combined, and precise motion planning instructions are generated for the task planning instructions based on the combination results. This avoids problems such as the disconnect between task planning instructions and the intelligent agent's motion or the delay in action execution, so that the embodied intelligent agent can execute tasks more accurately.

[0033] 2) This invention establishes an environmental interaction module, which monitors environmental changes caused by the agent's movement in real time and feeds back the detected environmental information to the perception planning module for task planning correction. Based on the corrected task instructions, the module corrects the embodied agent's movement, enabling accurate perception of environmental changes, achieving deeper physical interaction, and possessing strong generalization performance. Attached Figure Description

[0034] Figure 1 is a data flow diagram of the present invention;

[0035] Figure 2 is a flowchart of the embodied intelligent agent control method based on physical intelligence active perception according to the present invention;

[0036] Figure 3 is a flowchart of the pre-training process of the physical intelligence active perception multimodal embodied large model of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] Example 1:

[0039] In this embodiment, a control method for embodied intelligent agents based on physical intelligence active perception is provided. This method controls embodied intelligent agents based on a multimodal embodied large model. The multimodal embodied large model includes a data processing module, a perception planning module, a physical intelligence module, an information output module, an environmental interaction module, and a motion control execution module. The data flow between each module is shown in Figure 1.

[0040] The flowchart of the control method is shown in Figure 2, and includes the following steps:

[0041] S1. Model construction, the construction process of which is shown in Figure 3:

[0042] Select a Transformer-based pre-trained model, create a small instruction adjustment dataset based on the specific task scenario of the embodied agent, and use this dataset as the corpus for the prompt word project of the pre-trained model to train prompt words. If the selected pre-trained model has a response availability greater than or equal to 50%, then select the pre-trained model as the model base; otherwise, reselect the pre-trained model.

[0043] Based on the physical laws and motion posture data learned from the physical simulator, the generation capability of the selected pre-trained model is enhanced by the retrieval enhancement generation method, so as to improve the efficiency and quality of text processing tasks, so that the embodied intelligent agent can quickly and accurately obtain and query relevant information fragments.

[0044] Then, using the Supervised Fine-Tuning (SFT) method, the Low-Rank Adaptation (LoRA) method is selected to fine-tune the model, constructing several...<input,output> The supervised information significantly reduces the number of parameters and computational resources required for fine-tuning while maintaining the performance of the pre-trained model, thus improving fine-tuning efficiency. Then, a cold start strategy is adopted for training. First, the action decoder is frozen, and only the language model is adjusted, so that the model can quickly learn new language understanding and generation capabilities while maintaining its action generation capabilities.

[0045] The pre-trained model obtained after the above processing is the multimodal embodied large model.

[0046] S2. Data Acquisition and Processing: The data processing module acquires multimodal data and receives semantic information from the initial environment, and performs alignment and fusion operations on the multimodal data and semantic information to obtain processed data;

[0047] The specific alignment and blending operation includes the following steps:

[0048] S21. For text, use word embedding technology to convert words or subwords into continuous vector representations, and convert symbolic text into numerical forms that can be processed by machine learning models; for images, use scaling and cropping while normalizing the pixel values ​​of the images to eliminate brightness differences between different images; for typical information in data types, perform denoising and normalization processing to improve data usability.

[0049] S22. The multimodal data processed by step S21 is the first data. The same feature extractor is selected for the first data of the same type to extract features. The extracted features are converted into high-level feature representations by a deep-coupled autoencoder.

[0050] S23. By coupling the hidden layers of multiple different deep-coupled autoencoders to share and fuse information, the joint features of the first data are extracted to obtain the second data, and the second data is represented by feature vectors Tokens.

[0051] S24. Different types of second data are transformed into the same joint embedding space. The second data is matched and fused in the form of feature vector tokens in this space to obtain processed data, thereby supporting cross-modal retrieval, generation and other tasks.

[0052] S3. Task Planning: This is performed in the perception planning module, which is divided into a perception part and a planning part. Its workflow is as follows:

[0053] The perception section receives processed data output from the data processing module, generates perception information of the initial environment in which the embodied intelligent agent is located based on the processed data, and transmits the perception information to the planning section.

[0054] The planning section receives sensing information and generates task planning instructions based on the sensing information.

[0055] The perception planning module outputs perception information and task planning instructions to the physical intelligence module and the information output module.

[0056] S4, Exercise Planning:

[0057] The physical intelligence module receives processed data, sensor information, and task planning instructions;

[0058] Obtain the underlying motion strategy, which includes physical laws and motion posture data learned from multiple time-series physical simulators;

[0059] Based on the underlying motion strategy, combined with perception information and task planning instructions, underlying motion control commands and data are generated;

[0060] The underlying motion control commands and data are encapsulated, and the state, motion laws and attitude change data of the physical object to be executed that conforms to the motion strategy are projected into the space through a linear layer, and the motion command sequence at the corresponding moment is output.

[0061] S5, Motion Correction:

[0062] The information output module receives sensory information and motion planning instructions, and distributes the sensory information to the environment interaction module. The environment interaction module detects the environmental content O observed by the agent at time t in real time. t and the action a performed at that moment t And based on O t and a t Generate real-time environmental information;

[0063] The real-time environmental information is compared with the perceived information. If an error occurs in the comparison, the environmental interaction module sends the real-time environmental information to the perception planning module. The perception planning module corrects the task planning instructions based on the real-time environmental information to obtain the task instructions, and outputs the real-time environmental information and task instructions to the physical intelligence module.

[0064] The physical intelligence module receives real-time environmental information and task instructions, and corrects the motion planning instructions based on the real-time environment and task instructions to obtain the motion instruction T = [x′]. query ,((a0,O0),...,(a t O t )...)];

[0065] Movement instruction T = [x′ query ,((a0,O0),...,(a t O t The information is transmitted to the motion control execution module via the information output module, which receives and executes the motion command.

[0066] If no error is found during the comparison, the motion control execution module executes the motion planning instructions;

[0067] Once the action is completed, a signal indicating that the action is in place is output.

[0068] S6. User Feedback: The user interaction module receives the positioning signal output by the motion control module and replies with text information to the user based on the positioning signal.

[0069] Example 2:

[0070] This embodiment provides an embodied intelligent agent based on physical intelligence active perception. The embodied intelligent agent is controlled by the control method of the above embodiment. When it receives an instruction from the user, it executes the steps S1 to S6 of the above embodiment, which will not be described in detail here.

[0071] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling an embodied intelligent agent based on physical intelligence active perception, characterized in that, This method controls embodied intelligent agents based on a multimodal embodied big model, which includes a data processing module, a perception and planning module, a physical intelligence module, an environmental interaction module, and a motion control execution module. The perception planning module receives the data processed by the data processing module and outputs perception information and task planning instructions based on the data; the physical intelligence module receives the perception information and task planning instructions and outputs motion planning instructions; the environmental interaction module collects environmental information in real time and compares it with the perception information; the motion control module receives the motion planning instructions and performs motion correction based on the comparison results to realize the control of the embodied intelligent agent.

2. The embodied intelligent agent control method based on physical intelligence active perception according to claim 1, characterized in that, The method includes the following steps: Model construction: A multimodal embodied large model is obtained by fine-tuning the pre-trained model. The multimodal embodied large model also includes an information output module. Data acquisition and processing: The data processing module acquires multimodal data and receives semantic information from the initial environment, and performs alignment and fusion operations on the multimodal data and semantic information to obtain processed data; Task planning: The perception planning module receives the processed data, infers and perceives the environment based on the processed data to obtain perception information, generates task planning instructions based on the perception information, and outputs the perception information and task planning instructions after completion. Motion planning: The physical intelligence module receives processed data, perception information, and task planning instructions, and outputs motion planning instructions based on the processed data, perception information, and task planning instructions combined with the underlying motion strategy. Motion correction: The information output module receives sensing information and motion planning instructions, and sends the sensing information to the environment interaction module. The environment interaction module detects the environment in which the agent is located in real time to obtain real-time environmental information, and compares the real-time environmental information with the sensing information. The motion control execution module receives the motion planning instructions and performs motion correction based on the comparison results. When the motion is completed, it outputs a positioning signal. The real-time environmental information consists of multimodal data and positioning signals collected during the agent's motion. User feedback: The user interaction module receives the arrival signal output by the motion control module and replies with text information to the user based on the arrival signal.

3. The embodied intelligent agent control method based on physical intelligence active perception according to claim 2, characterized in that, The model construction specifically includes: Create and adjust the dataset according to the task scenario of the embodied intelligent agent; Adjust the dataset according to the instructions to verify and select the model base for the pre-trained model; The selected model base was trained using retrieval-enhanced generation and supervised fine-tuning methods.

4. The embodied intelligent agent control method based on physical intelligence active perception according to claim 2, characterized in that, The alignment and fusion operation includes the following steps: The first data is obtained by converting or normalizing the multimodal data and text commands. The multimodal data includes scene images, voice commands, force sensor data on the end effector, and text data. For the same type of first data, the same feature extractor is selected for feature extraction, and the extracted features are converted into a high-level feature representation by a deep-coupled autoencoder. By coupling the hidden layers of multiple different deep-coupled autoencoders to share and fuse information, the joint features of the first data are extracted to obtain the second data; The processed data is obtained by spatial matching of the second data.

5. The embodied intelligent agent control method based on physical intelligence active perception according to claim 4, characterized in that, The second data is represented in the feature space as tokens.

6. The embodied intelligent agent control method based on physical intelligence active perception according to claim 2, characterized in that, The aforementioned motion planning specifically includes: Based on the underlying motion strategy, combined with perception information and task planning instructions, underlying motion control commands and data are generated; The underlying motion control commands and data are encapsulated, and the effective information is projected into the space through a linear layer to output the motion command sequence at the corresponding moment.

7. The embodied intelligent agent control method based on physical intelligence active perception according to claim 6, characterized in that, The underlying motion strategy includes physical laws and motion posture data learned from multiple time-series physical simulators.

8. The embodied intelligent agent control method based on physical intelligence active perception according to claim 6, characterized in that, The valid information includes the state, motion patterns, and posture change data of the physical object to be executed, which conforms to the motion strategy.

9. The embodied intelligent agent control method based on physical intelligence active perception according to claim 2, characterized in that, The motion correction specifically includes: The environmental interaction module detects the environmental content in real time during the motion process and outputs the real-time environmental information at time t. The real-time environmental information at time t is compared with the predicted perception information at time t. If an error occurs in the comparison, the environmental interaction module sends the real-time environmental information to the perception planning module. The perception planning module corrects the task planning instructions based on the real-time environmental information to obtain the task instructions, and outputs the real-time environmental information and task instructions to the physical intelligence module. The physical intelligence module receives real-time environmental information and task instructions, and corrects the motion planning instructions based on the real-time environment and task instructions to obtain motion instructions. The motion command is transmitted to the motion control execution module through the information output module, which receives and executes the motion command. If no error is found during the comparison, the motion control execution module executes the motion planning instruction at time t.

10. An embodied intelligent agent based on physical intelligence active perception, characterized in that, The intelligent agent includes: Data processing module: The data processing module collects multimodal data and receives semantic information from the environment, and processes the data to obtain processed data; Perception and Planning Module: Receives processed data from the data processing module and outputs perception information and task planning instructions based on the processed data for scenario reasoning and specific task planning. The physical intelligence module receives processed data from the data processing module and perception information and task planning instructions from the perception and planning module. It then combines these with the underlying motion strategy to output motion planning instructions, which are used to guide the embodied intelligent agent's movement. Information output module: As an intermediate medium for data transmission, it receives the perception information output by the perception planning module and the motion planning instructions output by the physical intelligence module, and distributes the perception information to the environment interaction module and the motion planning instructions to the motion control execution module. Environmental interaction module: Real-time detection of environmental content during the movement of the embodied intelligent agent, comparison with the perceived information, and transmission of the comparison results to the motion control execution module; Motion control execution module: executes motion planning instructions, or corrects motion actions based on comparison results and executes the corrected motion actions, and outputs a position signal upon task completion; User interaction module: Receives the position signal output by the motion control module and replies with text information to the user based on the position signal.

Images