Interactive simulation method and system for intelligent agent with body

Abstract

The invention provides an interactive simulation method and system for an intelligent body. The method comprises the following steps: generating an intelligent body data set based on an intelligent body simulator; the intelligent data set comprises a plurality of interactive simulation samples; the multi-modal model is finely adjusted based on the intelligent data set; based on a preset interactive simulation mechanism, carrying out interactive simulation on the body-equipped intelligent agent through the body-equipped intelligent simulator; the intelligent body is a multi-modal model after fine adjustment; the intelligent emulator comprises a simulation world information generation module which is used for generating simulation world information; the character simulation information generation module is used for generating character simulation information; and the large language model is used for generating an interactive simulation sample according to the simulation world information and the character simulation information. According to the method, the perception, reasoning and action capabilities of the intelligent agent can be enhanced and evaluated, so that the intelligent agent can better understand and execute complex tasks, meanwhile, the method flexibly adapts to environmental changes, and the interactive simulation quality is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an interactive simulation method and system for embodied intelligent agents. Background Technology

[0002] In the field of artificial intelligence, embodied agents are intelligent systems capable of sensing, reasoning, and interacting with their physical or virtual environments. These agents demonstrate enormous application potential in multiple fields, including robotics, autonomous vehicles, smart homes, and virtual assistants.

[0003] However, despite the revolutionary concept of embodied agents, current implementations still have significant limitations. Current embodied agents primarily focus on executing predefined task flows, often lacking a deep understanding of the requirements of complex tasks. Summary of the Invention

[0004] This invention provides an interactive simulation method and system for embodied intelligent agents, which can enhance the perception, reasoning and action capabilities of intelligent agents, enabling them to better understand and execute complex tasks, while flexibly adapting to environmental changes and improving the quality of interactive simulation.

[0005] This invention provides an interactive simulation method for embodied intelligent agents, comprising: generating an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset including multiple interactive simulation samples; fine-tuning a multimodal model based on the embodied intelligent dataset; and performing interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism; the embodied intelligent agent being the fine-tuned multimodal model; wherein the embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating the interactive simulation samples based on the simulation world information and the character simulation information.

[0006] According to the interactive simulation method of an embodied intelligent agent provided by the present invention, the simulation world information generation module is specifically used to: obtain dynamic simulated weather information through a weather simulation system; obtain dynamic simulated traffic information through a traffic simulation system; obtain time information through a time cycle management system; and implement the dynamic simulated weather information, the dynamic simulated traffic information, and the time information through a 3D animation system and a rendering engine to form the simulation world information.

[0007] According to the interactive simulation method of an embodied intelligent agent provided by the present invention, the character simulation information generation module is specifically used for: establishing a character model through a three-dimensional human body modeling system; determining the character's personal information and activity schedule; obtaining the character simulation information based on a behavioral knowledge base, the character's personal information, and the activity schedule; and performing human intention-based action control on the character model based on the character simulation information through a 3D animation system and rendering engine.

[0008] According to the interactive simulation method of embodied intelligent agents provided by the present invention, the embodied intelligent simulator further includes a crowd simulation module; the crowd simulation module is specifically used for: generating simulated character profiles and schedules based on the simulated character information and a large language model under visual and voice navigation tasks; performing time-driven activity management through a high-level planner based on the simulated character profiles and the daily schedule; breaking down the current activity into sub-tasks through a mid-level planner and managing them through a finite state machine; and planning a global path for the current sub-task using the A* algorithm through a low-level planner and employing a social force model for local obstacle avoidance based on the simulated world information and the simulated character information.

[0009] According to the interactive simulation method of embodied intelligent agents provided by the present invention, the data types of the embodied intelligent dataset include 2D / 3D bounding boxes, semantic segmentation and depth maps, panoramic RGB images and dynamic human-computer interaction data.

[0010] According to the present invention, an interactive simulation method for an embodied intelligent agent is provided. The preset interactive simulation mechanism is an evaluation mechanism combining open-loop and closed-loop methods, used to evaluate the high-level capabilities of the embodied intelligent agent under a directional inquiry interactive task. The high-level capabilities include self-evaluation capability, information acquisition capability, and real-time applicability capability. The directional inquiry interactive task is based on visual and voice navigation tasks, and incorporates semantic understanding, proactive questioning, and dialogue feedback.

[0011] The present invention also provides an interactive simulation system for embodied intelligent agents, comprising: a dataset generation module for generating an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset including multiple interactive simulation samples; a fine-tuning module for fine-tuning a multimodal model based on the embodied intelligent dataset; and a simulation module for performing interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism; the embodied intelligent agent being the fine-tuned multimodal model; wherein the embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating the interactive simulation samples based on the simulation world information and the character simulation information.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the interactive simulation method of the embodied intelligent agent as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interactive simulation method of the embodied intelligent agent as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the interactive simulation method of the embodied intelligent agent as described above.

[0015] This invention provides an interactive simulation method and system for embodied intelligent agents. The method includes: generating an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset includes multiple interactive simulation samples; fine-tuning a multimodal model based on the embodied intelligent dataset; and performing interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism; the embodied intelligent agent is the fine-tuned multimodal model; wherein the embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating interactive simulation samples based on the simulation world information and character simulation information. This invention can enhance and evaluate the perception, reasoning, and action capabilities of embodied intelligent agents, enabling them to better understand and execute complex tasks, while flexibly adapting to environmental changes and improving the quality of interactive simulation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an interactive simulation method for an embodied intelligent agent provided by the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the principle of an interactive simulation method for embodied intelligent agents provided by the present invention.

[0019] Figure 3 This is a schematic diagram illustrating the principle of human behavior simulation provided by the present invention.

[0020] Figure 4This is a schematic diagram of the structure of an interactive simulation system for an embodied intelligent agent provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an interactive simulation method for an embodied intelligent agent provided by the present invention.

[0024] This invention provides an interactive simulation method for embodied intelligent agents, comprising: 101: Generate an embodied intelligence dataset based on an embodied intelligence simulator; the embodied intelligence dataset includes multiple interactive simulation samples; 102: Fine-tuning a multimodal model based on an embodied intelligence dataset; 103: Based on a preset interactive simulation mechanism, an embodied intelligent agent is interactively simulated using an embodied intelligent simulator; the embodied intelligent agent is a finely tuned multimodal model; the embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating interactive simulation samples based on the simulation world information and character simulation information.

[0025] Embodied intelligent agents refer to intelligent systems with perception, reasoning, and action capabilities, enabling them to interact with their surroundings in physical or virtual environments. This invention provides an interactive simulation method for embodied intelligent agents. First, an embodied intelligent simulator (FreeAskWorld simulator) is activated. This simulator includes a simulated world information generation module, a character simulation information generation module, and a large language model. The simulated world information generation module is responsible for generating scene information in the virtual environment, including terrain, buildings, weather conditions, etc. For example, it generates a city block, containing multiple streets, buildings, traffic lights, and pedestrians. The character simulation information generation module generates character information in the virtual environment, including the character's appearance, behavior patterns, and schedule. For example, it generates an office worker whose behavior patterns include going to work in the morning, eating lunch at noon, and leaving get off work in the afternoon.

[0026] The large language model generates interactive simulation samples based on information from the simulated world and simulated characters. These samples include the agent's interactions with objects and people in the environment. For example, when an agent is walking on a street and encounters a pedestrian, the agent needs to ask the pedestrian for directions. The interactive simulation samples generated by the large language model include the agent's question, the pedestrian's answer, and the agent's behavior of adjusting its path based on the answer.

[0027] The generated embodied intelligence dataset contains multiple interactive simulation samples, covering six task categories and 16 core object categories, generating over 17 hours of interactive data and more than 60,000 high-quality labeled frames to support the model's comprehensive learning of semantic interaction and spatial awareness capabilities. The multimodal models used in this invention include, for example, the ETPNav and BEVBert models. The multimodal models are fine-tuned based on the FreeAskWorld dataset (embodied intelligence dataset). The embodied agent is the fine-tuned multimodal model. The multimodal model can handle data from multiple modalities, such as visual, linguistic, and action data. During fine-tuning, the model learns how to generate reasonable interactive behaviors based on environmental and human information. For example, the model learns how to adjust navigation paths based on street layout and pedestrian responses. The fine-tuned multimodal model can better understand and execute complex interactive tasks. The fine-tuned multimodal model from the embodied intelligence dataset is loaded into the embodied intelligence simulator to form an embodied agent. Through a pre-defined interactive simulation mechanism, the embodied agent performs interactive simulations in a virtual environment. During the simulation, the embodied agent can actively interact with objects and people in the environment. For example, when searching for a destination, an embodied agent can ask a virtual character for directions and adjust its path based on the response. The simulator records the embodied agent's behavior and interaction results to evaluate its performance and optimize the model.

[0028] The ETPNav model is a hierarchical VLN-CE framework that performs online topology mapping, cross-modal planning using Transformers, and low-level control using a rotation-forward pattern, enhanced by obstacle avoidance heuristics. The BEVBert model is a map-based multimodal pre-trained model that leverages hybrid topology-metric representations to improve spatial reasoning and language-guided navigation capabilities.

[0029] This invention's embodied intelligent simulator constructs highly realistic simulations of human behavior and complex social interaction scenarios by integrating large language models and social intention modeling. Leveraging Unity's High Definition Render Pipeline (HDRP), the platform achieves a virtual environment with detailed lighting and realistic features. Simultaneously, the embodied intelligent simulator supports dynamic scene randomization (such as weather, time, and traffic conditions), significantly improving data diversity and model robustness.

[0030] The embodied intelligent simulator of this invention creates a highly realistic virtual environment through a simulated world information generation module and a character simulation information generation module, and generates rich interactive simulation samples using a large language model. These samples are used to fine-tune the multimodal model, enabling it to better understand and perform complex interactive tasks. Finally, interactive simulations of the embodied intelligent agent are conducted using the embodied intelligent simulator to verify the model's interactive capabilities and adaptability in complex dynamic environments.

[0031] This invention presents an embodied agent interaction simulation method that combines physical simulation with semantic interaction, supporting high-level agent behavior planning and language-driven human-computer interaction. Based on intent recognition and social cognitive mechanisms, this invention enables the agent to understand and respond to human intents, supporting the simulation of social behaviors such as proactive information acquisition and task execution. This invention supports scalable, realistic, and interactive human-computer simulations, constructing a complete data generation pipeline for diverse embodied tasks. For example, it extends visual and language navigation tasks to a semantically rich direction-finding task, enhancing the agent's interactive capabilities in real-world environments through proactive interaction mechanisms, semantic understanding, proactive questioning, and dialogue feedback.

[0032] This invention supports an engine framework structure for scalable human-intelligent agent simulation, which facilitates the integration of new task types, language interaction forms, and environmental resources, thereby enhancing the adaptability of embodied AI models in terms of multi-task versatility and social intelligence.

[0033] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of an interactive simulation method for an embodied intelligent agent provided by the present invention.

[0034] In a preferred embodiment, the simulated world information generation module is specifically used to: acquire dynamic simulated weather information through a weather simulation system; acquire dynamic simulated traffic information through a traffic simulation system; acquire time information through a time cycle management system; and implement the dynamic simulated weather information, dynamic simulated traffic information, and time information through a 3D animation system and rendering engine to form simulated world information.

[0035] In this embodiment, the acquisition of simulated world information is achieved by integrating multiple dynamic simulation systems through a simulated world information generation module, ensuring the realism, dynamism, and diversity of the simulated environment. Through the weather simulation system, the agent can acquire dynamic simulated weather information in the current simulated environment. This system can simulate various weather conditions, such as sunny, rainy, and snowy days, and can dynamically change weather conditions based on time information provided by the time cycle management system. For example, it can simulate weather changes at sunrise and sunset, helping models trained under different lighting and visibility conditions to generalize better. On sunny days, the system generates bright sunlight and a clear sky; on rainy days, it generates raindrops, puddles, and fog effects.

[0036] Traffic simulation systems provide dynamically simulated traffic information. This system can simulate urban traffic flow, including vehicle speed and direction, pedestrian paths, and traffic rules (changes in traffic lights). This traffic information changes as the simulation progresses to simulate the realism and dynamic complexity of traffic flow in the real world, supporting more challenging perception and planning tasks. For example, the traffic simulation system can generate corresponding traffic flow based on different time periods (such as morning rush hour, off-peak hour, and evening rush hour). During the morning rush hour, the system generates a large number of vehicles and hurried pedestrians; during the off-peak hour, traffic flow is relatively reduced.

[0037] The time cycle management system is responsible for providing time information, including the current simulation time and time of day (e.g., morning, afternoon, evening), as well as date and season changes. This time information is crucial for understanding changes in lighting conditions and human activity patterns within the simulated environment.

[0038] The 3D animation system and rendering engine integrate and realize the aforementioned dynamic simulated weather, traffic, and time information to form a complete simulated world information. The 3D animation system is responsible for generating animation effects related to weather and traffic, such as falling raindrops and vehicle movement trajectories. The rendering engine is responsible for rendering these dynamic effects with high quality, making the virtual environment appear highly realistic. In this way, the simulated world information generation module provides a dynamic and realistic virtual environment for the interactive simulation of embodied intelligent agents.

[0039] 3D animation systems, for example, use the Unity animation system, and rendering engines, for example, use Unity's HDRP rendering pipeline.

[0040] The simulation world information generation module in this embodiment generates dynamic simulation world information through a weather simulation system, a traffic simulation system, and a time cycle management system, and visualizes it using a 3D animation system and a rendering engine. This dynamic simulation world information provides a highly realistic virtual environment for the interactive simulation of embodied intelligent agents, enabling the agents to be trained and evaluated in a near-real-world environment.

[0041] As a preferred embodiment, the character simulation information generation module is specifically used for: establishing a character model through a 3D human body modeling system; determining the character's personal information and activity schedule; obtaining character simulation information based on a behavioral knowledge base, the character's personal information, and the activity schedule; and controlling the character model's actions based on human intentions through a 3D animation system and rendering engine.

[0042] In this embodiment, the process of acquiring character simulation information involves creating realistic virtual characters capable of natural interaction with embodied intelligent agents within a simulation environment. The character simulation information generation module establishes character models using a 3D human body modeling system, allowing designers to define the character's physical characteristics, such as height, body shape, facial features, and clothing, and create corresponding 3D models. For example, the system can generate a young office worker characterized as male, 30 years old, 1.75 meters tall, and wearing formal attire. These models are not only visually realistic but also capable of simulating real human movement. Next, the character's personal information and activity schedule are determined. Personal information includes basic character details such as name, age, education, occupation, and personality traits. For example, the office worker's profile might include that they work at a technology company, enjoy reading and sports, etc. This information will influence the character's behavior and interaction patterns. The activity schedule defines the character's daily activities in the simulation environment, such as work, rest, and socializing, as well as the time and location of these activities. The office worker's schedule could include: waking up at 7:00 AM, leaving for work at 7:30 AM, arriving at the office at 8:00 AM, having lunch at 12:00 PM, leaving get off work at 5:00 PM, and going to the gym at 7:00 PM. Then, based on a behavioral knowledge base, the individual's profile, and the schedule, the system generates simulated information about the individual. The behavioral knowledge base contains a large number of predefined sequences of human actions, such as walking, running, and gestures. For example, according to the behavioral knowledge base, the office worker might check their phone, greet colleagues, or show impatience during traffic jams while commuting. Combining the individual's profile and schedule, the system can generate specific behavioral sequences. For example, when leaving home at 7:30 AM, the office worker might put on a suit, pick up their briefcase, and lock the door; during their commute, they might take the subway and check news on their phone.

[0043] Using a 3D animation system and rendering engine, the system controls the actions of a character model based on simulated information, allowing for the manipulation of human intentions. The 3D animation system translates the character's behavioral sequences into specific animated movements, such as walking, waving, and nodding. The rendering engine then renders these movements with high quality, making the character's actions appear natural and realistic. For example, when an office worker checks their phone on the subway, the system generates animations of them taking their phone out of their pocket, unlocking the screen, and browsing news, which are then rendered by the rendering engine to make these actions appear realistic and fluid in the virtual environment.

[0044] To support consistent embodied and context-sensitive behavior, MotionX was used as the SMSLX animation library, with its motion structured into high-level categories and subcategories, allowing the selection of semantically relevant actions for each activity. A custom SMSLX animation driver plugin was also developed, allowing seamless access to the full range of animations in the library. To enhance animation fidelity and character realism, blending functionality was integrated into the motion controller, enabling smooth transitions between actions.

[0045] Of course, regional familiarity and personality can also be incorporated into a person's profile, and navigation styles can be categorized based on four key characteristics: landmark level, direction type, distance level, and utterance length. This allows the multimodal model to generate context-based navigation labels and instructions based on the person's profile, task, and simulated world information.

[0046] In this embodiment, the character simulation information generation module establishes a highly detailed character model using a 3D human modeling system, determines the character's personal information and activity schedule, and generates character simulation information based on a behavioral knowledge base. Through a 3D animation system and rendering engine, the module performs human-intentional motion control on the character model, making the virtual character's behavior in the virtual environment highly realistic and consistent with human behavioral patterns. This character simulation information provides rich interactive objects for the interactive simulation of embodied intelligent agents, enabling the agents to be trained and evaluated in near-real-world conditions.

[0047] This invention provides a highly immersive simulation environment, enhancing the model's adaptability to real-world environments. It supports real-time interactive training and evaluation for complex tasks, including dynamic navigation and direction querying. Diverse scene designs and a modular architecture give the simulator excellent scalability, making it suitable for a variety of embodied intelligence tasks.

[0048] Please refer to Figure 3 , Figure 3 A schematic diagram illustrating the principle of human behavior simulation provided by this invention.

[0049] As a preferred embodiment, the embodied intelligent simulator also includes a crowd simulation module; the crowd simulation module is specifically used for: generating simulated character profiles and schedules based on large language models according to character simulation information under visual and voice navigation tasks; managing time-driven activities through a high-level planner based on simulated character profiles and daily schedules; breaking down the current activity into sub-tasks through a mid-level planner and managing them through a finite state machine; and planning a global path for the current sub-task using the A* algorithm through a low-level planner, based on simulated world information and character simulation information, and using a social force model for local obstacle avoidance.

[0050] In this embodiment, the crowd simulation module, under visual and voice navigation tasks, generates simulated character profiles (personal data) and schedules using a large language model based on the simulated character information. These profiles include information such as the character's name, occupation, personality traits, and daily activities, providing the intelligent agent with interactive objects and context. For example, assuming the task scenario is a large shopping mall, the module will generate multiple virtual characters, each with a detailed profile including age, gender, occupation, hobbies, and a specific schedule. For instance, generating a 35-year-old woman whose occupation is freelancer and hobbies include shopping and watching movies, her schedule might be arriving at the shopping mall at 10:00 AM, having lunch at a restaurant at 12:00 PM, and going to see a movie at 2:00 PM.

[0051] Next, based on the simulated person's profile and schedule, the advanced planner performs time-driven activity management. The advanced planner selects the person's activities for the current time point according to the schedule and the person's activity plan. For example, for the woman mentioned above, at 10:00 AM, the advanced planner would select "Arrive at the shopping mall" as the current activity. The advanced planner dynamically adjusts the order of the person's activities based on the schedule and activity priorities to ensure that the person's behavior conforms to the schedule.

[0052] The intermediate planner then breaks down the current activity into subtasks and manages them using a finite state machine. For example, the activity "arrive at the shopping mall" can be broken down into subtasks such as "walk from the parking lot to the shopping mall entrance" and "enter the shopping mall." The finite state machine manages the execution order and state transitions of these subtasks. For instance, once the person completes the subtask "walk from the parking lot to the shopping mall entrance," the finite state machine will trigger the execution of the subtask "enter the shopping mall."

[0053] Finally, based on the current simulated world information and character simulation information, the low-level planner uses the A* algorithm to plan a global path for the current subtask and employs a social force model for local obstacle avoidance. For example, for the subtask of "walking from the parking lot to the shopping mall entrance," the low-level planner will first use the A* algorithm to plan the optimal path from the parking lot to the shopping mall entrance. During path planning, the low-level planner considers factors such as obstacles in the simulated world, the positions and directions of other characters, etc. Simultaneously, a social force model is used for local obstacle avoidance to ensure that the character can naturally avoid other characters and obstacles while walking. For example, when the character approaches a pedestrian, the social force model calculates an appropriate avoidance action so that the character can pass smoothly without a collision.

[0054] Finite state machines (FSMs) can determine the next state and corresponding output based on the input and the current state. In interactive simulation methods for embodied agents, FSMs can be used for subtask selection and management to achieve more complex behavioral patterns and decision-making processes. The intermediate planner determines the next subtask based on the agent's current state and environmental information and passes it to the lower-level planner.

[0055] The A* algorithm is an algorithm for finding the shortest path from a starting point (Source) to a target point (Goal) in a graph with multiple nodes. It belongs to the heuristic search algorithm and is highly efficient and accurate in path planning and graph search problems.

[0056] The core idea of ​​the A* algorithm is to use a priority queue to store nodes to be explored. Nodes in the priority queue are sorted according to the sum of the path cost from the starting point to the current node (g value) and the estimated cost from the current node to the target point (h value). g value is the actual cost from the starting point to the current node, and h value is the estimated cost from the current node to the target point, typically estimated using heuristics. The A* algorithm explores by repeatedly selecting the node with the lowest cost from the priority queue until the target point is found or the queue is empty. The A* algorithm guarantees finding the shortest path if h value is an acceptable heuristic (i.e., h value does not overestimate the actual cost) and if the edge weights in the graph are non-negative.

[0057] The social force model is a simulation of crowd behavior that views a crowd as a collection of individuals, each influenced by physical forces. These forces include: self-motivation (the tendency of individuals to move to reduce the time required to reach their goal), interpersonal repulsion (repulsion between individuals to avoid collisions), interpersonal attraction (individuals may be attracted to each other, such as during conversations or walking together), and environmental attraction (individuals may be attracted to certain features in the environment, such as exits or stairs). The social force model determines the acceleration and velocity of individuals by calculating the resultant force of these forces, thus simulating their trajectories. This model can generate relatively natural crowd flow patterns and is suitable for crowd simulation, robot path planning, and obstacle avoidance.

[0058] In this embodiment, the crowd simulation module generates detailed simulated character profiles and schedules using a large language model. The high-level planner manages activities according to the schedule, the mid-level planner breaks down activities into sub-tasks and manages them using a finite state machine, and the low-level planner is responsible for global path planning and local obstacle avoidance. These modules work together to make the virtual character's behavior highly realistic and consistent with human behavioral patterns. In this way, the crowd simulation module provides rich interactive objects and complex dynamic environments for the interactive simulation of embodied intelligent agents, enabling the agents to be trained and evaluated in near-real-world conditions.

[0059] As a preferred embodiment, the data types of the embodied intelligence dataset include 2D / 3D bounding boxes, semantic segmentation and depth maps, panoramic RGB images, and dynamic human-computer interaction data.

[0060] In this embodiment, the data types of the embodied intelligence dataset include 2D / 3D bounding boxes, semantic segmentation and depth maps, panoramic RGB images, and dynamic human-computer interaction data. These data types provide the agent with rich visual and semantic information, enabling it to better understand and interact with the simulated world.

[0061] 2D / 3D bounding box data supports target localization tasks, used to identify the position and shape of objects in a simulated environment. These bounding boxes can be static, such as walls and furniture, or dynamic, such as moving pedestrians and vehicles. Using these bounding boxes, agents can identify and locate key objects in the environment, enabling effective navigation and interaction.

[0062] Semantic segmentation and depth maps enhance scene understanding by providing semantic and depth information about the scene. Semantic segmentation maps assign each pixel in an image to a specific category, such as roads, buildings, and vegetation, helping the agent understand the composition of the scene. Depth maps provide distance information from each pixel in the scene to the camera, enabling the agent to perceive the distance of objects and their spatial layout.

[0063] Panoramic RGB images enhance visual navigation by providing comprehensive visual information about the simulated environment. These images can be static or dynamic, capturing changes in lighting and the movement of objects within the environment. Panoramic images enable agents to observe their environment from multiple perspectives, enhancing their spatial awareness.

[0064] Dynamic human-computer interaction data supports multi-turn dialogue and navigation tasks, recording the interaction process between the intelligent agent and the virtual character, including dialogue content, action sequences, and interaction results. This data helps the intelligent agent learn how to respond to human instructions and behaviors, improving its interaction capabilities.

[0065] The diversity and high-quality annotations of the embodied intelligence dataset in this invention provide rich data support for training embodied intelligence tasks. Multimodal perspectives (such as panoramic views and depth maps) enhance the model's environmental understanding capabilities. The semantic richness of the embodied intelligence dataset compensates for the shortcomings of existing datasets in modeling dynamic interactions and social behaviors.

[0066] The embodied intelligence dataset of this invention includes various data types such as 2D / 3D bounding boxes, semantic segmentation and depth maps, panoramic RGB images, and dynamic human-computer interaction data. These data types are generated through different modules of the embodied intelligence simulator, providing rich multimodal information for the training and evaluation of embodied agents. 2D / 3D bounding box data helps the agent accurately locate target objects, semantic segmentation and depth map data enhance its understanding of the environment, panoramic RGB images improve visual navigation, and dynamic human-computer interaction data supports complex multi-turn dialogue and navigation tasks. The comprehensive use of these data enables embodied agents to perform highly realistic interactive simulations in virtual environments, thereby better adapting to complex tasks in the real world.

[0067] As a preferred embodiment, the preset interactive simulation mechanism is an evaluation mechanism that combines open-loop and closed-loop methods, used to evaluate the high-level capabilities of the embodied intelligent agent under the directional inquiry interaction task; the high-level capabilities include self-evaluation ability, information acquisition ability, and real-time applicability ability; the directional inquiry interaction task is based on visual and voice navigation tasks, and introduces semantic understanding, proactive questioning, and dialogue feedback.

[0068] In this embodiment, the directional inquiry interaction task is based on the visual and voice navigation tasks, and introduces semantic understanding, proactive questioning, and dialogue feedback to enhance the agent's interaction capabilities in real-world environments.

[0069] For directional inquiry tasks, the following high-level capabilities of the embodied agent are evaluated through an active interaction mechanism: Self-assessment ability to judge the status of task completion.

[0070] Information acquisition capabilities, such as proactively requesting navigation information from human agents.

[0071] Real-time adaptability, adjusting path planning in dynamic scenarios.

[0072] The interactive simulation mechanism of this invention can be a combined open-loop and closed-loop evaluation mechanism to compare the performance of human and machine models in real-world interaction scenarios, measuring their semantic understanding, behavioral planning, and interaction capabilities. Open-loop evaluation involves the agent or system performing a task without real-time feedback. In this evaluation, the agent executes tasks according to pre-set instructions or plans, regardless of dynamic changes in the environment or further interactions with it. Open-loop evaluation is typically used to test the agent's performance under specific conditions, such as navigating to a target location along a pre-set path. Closed-loop evaluation, on the other hand, is a more complex and dynamic evaluation method involving real-time interaction between the agent and the environment. In closed-loop evaluation, the agent's behavior adjusts based on real-time feedback from the environment to adapt to changes in the environment. This invention's evaluation mechanism (interactive simulation mechanism), by combining open-loop and closed-loop evaluation, provides a comprehensive framework for evaluating the performance of embodied agents. Open-loop evaluation helps to understand the agent's basic performance under ideal conditions, while closed-loop evaluation reveals the agent's adaptability and intelligence in the face of uncertainty and dynamic changes. This comprehensive evaluation approach helps develop more intelligent, flexible, and adaptable artificial intelligence systems that can effectively perform tasks in the real world and interact naturally with humans.

[0073] This invention significantly improves the navigation success rate and semantic understanding ability of embodied agents in complex and dynamic scenarios. It enhances the model's predictive accuracy and adaptability, supporting more natural human-computer interaction. It also provides a new benchmark for evaluating the social behavior capabilities of embodied agents.

[0074] Furthermore, this invention can also generate highly realistic virtual worlds using the Genie 3 world model. Interactive 3D scenes can be dynamically created and modified via text prompts, such as adding characters or objects in real time.

[0075] This invention supports continuous path planning and can dynamically adjust behavior. By integrating multimodal data and social cognitive theory, it fills the gaps in existing technologies regarding semantic interaction and high-level planning capabilities. A high-fidelity simulation environment and rich datasets enhance the applicability of the trained model in real-world scenarios.

[0076] The interactive simulation system of embodied intelligent agents provided by the present invention is described below. The interactive simulation system of embodied intelligent agents described below can be referred to in correspondence with the interactive simulation method of embodied intelligent agents described above.

[0077] Please refer to Figure 4 , Figure 4This is a schematic diagram of the structure of an interactive simulation system for an embodied intelligent agent provided by the present invention.

[0078] This invention also provides an interactive simulation system for embodied intelligent agents, comprising: a dataset generation module 401, used to generate an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset includes multiple interactive simulation samples; a fine-tuning module 402, used to fine-tune a multimodal model based on the embodied intelligent dataset; and a simulation module 403, used to perform interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism; the embodied intelligent agent is the fine-tuned multimodal model; wherein the embodied intelligent simulator includes: a simulation world information generation module, used to generate simulation world information; a character simulation information generation module, used to generate character simulation information; and a large language model, used to generate interactive simulation samples based on the simulation world information and the character simulation information.

[0079] Figure 5 An example is a schematic diagram of the structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute an interactive simulation method for an embodied intelligent agent. This method includes: generating an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset includes multiple interactive simulation samples; fine-tuning a multimodal model based on the embodied intelligent dataset; and performing interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism. The embodied intelligent agent is a fine-tuned multimodal model. The embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating interactive simulation samples based on the simulation world information and the character simulation information.

[0080] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the interactive simulation method for embodied intelligent agents provided by the above methods. The method includes: generating an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset includes multiple interactive simulation samples; fine-tuning a multimodal model based on the embodied intelligent dataset; and performing interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism; the embodied intelligent agent is a fine-tuned multimodal model; wherein the embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating interactive simulation samples based on the simulation world information and the character simulation information.

[0082] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an interactive simulation method for embodied intelligent agents provided by the above methods. The method includes: generating an embodied intelligent dataset based on an embodied intelligent simulator; the embodied intelligent dataset including multiple interactive simulation samples; fine-tuning a multimodal model based on the embodied intelligent dataset; and performing interactive simulation on the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism; the embodied intelligent agent is the fine-tuned multimodal model; wherein the embodied intelligent simulator includes: a simulation world information generation module for generating simulation world information; a character simulation information generation module for generating character simulation information; and a large language model for generating interactive simulation samples based on the simulation world information and the character simulation information.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive simulation method for embodied intelligent agents, characterized in that, include: An embodied intelligence dataset is generated based on an embodied intelligence simulator; the embodied intelligence dataset includes multiple interactive simulation samples. Fine-tuning of multimodal models based on embodied intelligence datasets; Based on a preset interactive simulation mechanism, the embodied intelligent agent is interactively simulated through the embodied intelligent simulator. The embodied intelligent agent is a finely tuned multimodal model; The embodied intelligent simulator includes: The simulation world information generation module is used to generate simulation world information; The character simulation information generation module is used to generate character simulation information; A large language model is used to generate the interactive simulation samples based on the simulated world information and the character simulation information.

2. The interactive simulation method for embodied intelligent agents according to claim 1, characterized in that, The simulated world information generation module is specifically used for: Dynamic simulated weather information is obtained through a weather simulation system; Dynamic simulated traffic information is obtained through a traffic simulation system; Obtain time information through a time cycle management system; The simulated weather information, simulated traffic information, and time information are implemented using a 3D animation system and rendering engine to form the simulated world information.

3. The interactive simulation method for embodied intelligent agents according to claim 1, characterized in that, The character simulation information generation module is specifically used for: A human figure model is created using a 3D human body modeling system; Determine the individual's profile and activity schedule; Based on the behavioral knowledge base, the individual's personal information, and the activity schedule, the simulated information of the individual is obtained; Using a 3D animation system and rendering engine, the character model is controlled to perform actions in accordance with human intentions based on the character simulation information.

4. The interactive simulation method for embodied intelligent agents according to claim 1, characterized in that, The embodied intelligent simulator also includes a crowd simulation module; the crowd simulation module is specifically used for: In visual and voice navigation tasks, based on the character simulation information, simulated character profiles and schedules are generated using a large language model. Based on the simulated character profile and daily schedule, time-driven activity management is performed through an advanced planner. The current activity is broken down into subtasks using an intermediate planner and managed using a finite state machine; Based on the simulated world information and the simulated character information, a global path is planned for the current subtask using the A* algorithm through a low-level planner, and local obstacle avoidance is performed using a social force model.

5. The interactive simulation method for embodied intelligent agents according to claim 1, characterized in that, The data types of the embodied intelligence dataset include 2D / 3D bounding boxes, semantic segmentation and depth maps, panoramic RGB images, and dynamic human-computer interaction data.

6. The interactive simulation method for embodied intelligent agents according to any one of claims 1 to 5, characterized in that, The preset interactive simulation mechanism is an evaluation mechanism that combines open-loop and closed-loop methods. It is used to evaluate the high-level capabilities of the embodied intelligent agent under the direction query interaction task. The high-level capabilities include self-evaluation ability, information acquisition ability, and real-time applicability. The direction query interaction task is based on visual and voice navigation tasks and introduces semantic understanding, proactive questioning, and dialogue feedback.

7. An interactive simulation system for an embodied intelligent agent, characterized in that, include: A dataset generation module is used to generate an embodied intelligence dataset based on an embodied intelligence simulator; the embodied intelligence dataset includes multiple interactive simulation samples. The fine-tuning module is used to fine-tune multimodal models based on embodied intelligence datasets; The simulation module is used to perform interactive simulation of the embodied intelligent agent through the embodied intelligent simulator based on a preset interactive simulation mechanism. The embodied intelligent agent is a finely tuned multimodal model; The embodied intelligent simulator includes: The simulation world information generation module is used to generate simulation world information; The character simulation information generation module is used to generate character simulation information; A large language model is used to generate the interactive simulation samples based on the simulated world information and the character simulation information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the interactive simulation method of the embodied intelligent agent as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the interactive simulation method of the embodied intelligent agent as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the interactive simulation method of the embodied intelligent agent as described in any one of claims 1 to 6.

Images