Open scene intelligent interaction method and system based on multi-modal large model

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an open scene tool intelligent interaction method and system based on a multi-modal large model, and the method comprises the steps: generating a navigation grid and an action feasibility matrix related to a new environment; generating matched candidate actions in response to a natural language instruction input by a user; calculating success probabilities of the candidate actions according to the action feasibility matrix; optimizing the candidate action according to the success probability of the candidate action to determine a final execution action; aligning the interaction point to the geometric center of the target object according to the visual observation image of the new environment and the natural language instruction; and controlling the action execution of the intelligent agent according to the attribute of the final execution action. Thus, after the agent enters a new environment, through autonomous perception and analysis reasoning of scenes and semantics, decision-making of final execution actions is realized based on the action feasibility matrix, and action execution of the agent is controlled based on the attributes of the final execution actions, so that semantic planning and physical execution are better linked, and the agent can better adapt to an unknown environment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, such as open-scene embodied intelligent interaction methods and systems based on multimodal large models. Background Technology

[0002] Currently, visual navigation methods primarily rely on predefined semantic maps and discretized waypoints for path planning. While this simplifies the learning process, it exhibits significant rigidity in open-world environments. Specifically, when moving obstacles or dynamically changing layouts exist, map-based methods struggle to adjust in real-time, leading to a substantial decrease in adaptability. Furthermore, this field faces a fundamental trade-off between generalization ability and operational accuracy: modular frameworks based on reinforcement learning can achieve proficiency in specific tasks but require extensive retraining in new environments; while zero-shot methods based on Large Language Models (LLMs) avoid training, their text-based prompts often ignore crucial spatial visual context, resulting in navigation failures in complex geometric environments. Existing unified models (such as Uni-NaVid) attempt to integrate these approaches, but still employ discrete action tokenization, limiting their application in continuous real-world scenarios.

[0003] The physical interaction capabilities of existing embodied intelligence systems are typically limited to a very small number (often fewer than 10) of atomic actions, such as simple "grabbing" or "pushing." This limited action space cannot meet the demands of complex tasks. More importantly, when faced with novel objects in open scenes, traditional methods, using predefined "action-object" mappings, often introduce "feasibility ambiguity." That is, the system struggles to determine whether a semantically reasonable action (such as "open") is physically feasible when faced with an object with unknown physical properties or geometric configurations. This over-reliance on prior knowledge leads to frequent interaction failures for the agent when encountering novel object configurations.

[0004] Current task decomposition strategies exhibit significant fragmentation, with a lack of effective coordination mechanisms between high-level semantic planning and low-level physical control, leading to severe semantic-physical divergence. Multimodal large language models (MLLMs), acting as high-level planners, often prioritize semantic completion of the task while neglecting physical constraints, frequently proposing actions that violate physical laws or are currently unenforceable (e.g., suggesting "operate a switch" when too far away). Furthermore, existing embodied intelligence benchmarks suffer from gaps: one type focuses on high-level planning in simplified environments (e.g., VisualAgentBench), while the other emphasizes low-level control without task context (e.g., EB-Manipulation), lacking comprehensive evaluation of agents at the full-stack level—that is, their continuous collaborative capabilities from long-range navigation to fine-grained physical manipulation. This fragmentation makes it difficult to solve complex tasks such as "drawing the curtains and turning off the bedroom light," which require long-term spatial reasoning and continuous action sequences.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides an open-scene embodied intelligent interaction method and system based on a multimodal large model to improve the agent's ability to connect semantic planning and physical execution, as well as its adaptability to unknown environments.

[0008] In some embodiments, the open-scene embodied intelligent interaction method based on a multimodal large model includes: generating a navigation mesh and an action feasibility matrix for a new environment; generating candidate actions matching a natural language instruction input by a user; calculating the success probability of the candidate actions based on the action feasibility matrix; optimizing the candidate actions based on their success probabilities to determine the final action to be executed; aligning the interaction point to the geometric center of a target object based on a visual observation image of the new environment and the natural language instruction; and controlling the agent's action execution based on the attributes of the final action to be executed.

[0009] In some embodiments, the open scene embodied intelligent interaction system based on a multimodal large model includes: a generation module configured to generate a navigation mesh and an action feasibility matrix for a new environment; a response module configured to generate candidate actions matching a natural language instruction input by a user; a calculation module configured to calculate the success probability of the candidate actions based on the action feasibility matrix; a determination module configured to optimize the candidate actions based on the success probability of the candidate actions to determine the final action to be executed; an alignment module configured to align the interaction points to the geometric center of a target object based on a visual observation image of the new environment; and an execution module configured to control an agent to execute the final action based on the attributes of the final action to be executed.

[0010] In some embodiments, the intelligent agent includes the aforementioned open-scene embodied intelligent interaction system based on a multimodal large model.

[0011] The open-scene embodied intelligent interaction method and system based on a multimodal large model provided in this disclosure can achieve the following technical effects: When an agent enters a new environment, it first generates a navigation grid and action feasibility matrix for that environment. Upon receiving natural language commands from the user, it generates matching candidate actions and identifies the target object to be interacted with. Based on the action feasibility matrix, it calculates the success rate of the candidate actions and optimizes them to determine the final action to improve the success rate of the interaction. Then, based on visual observations of the new environment and natural language commands, it aligns the interaction point to the geometric center of the target object to facilitate smooth interaction. Finally, it controls the agent's action execution based on the attributes of the final action, enabling the agent to perform movements such as movement and manipulation smoothly and accurately. In this way, after entering a new environment, the agent autonomously perceives and analyzes the scene and semantics, makes decisions about the final action based on the action feasibility matrix, and controls the agent's action execution based on the attributes of the final action. This allows for better integration of semantic planning and physical execution, making it more adaptable to unknown environments.

[0012] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0013] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of the first open scene embodied intelligent interaction method based on a multimodal large model provided in this disclosure embodiment; Figure 2 This is a schematic diagram illustrating the generation of a navigation mesh and action feasibility matrix for a new environment, provided in an embodiment of this disclosure. Figure 3 This is a schematic diagram illustrating the generation of candidate actions matching a natural language instruction in response to a user's input, provided in an embodiment of this disclosure. Figure 4 This is a schematic diagram illustrating the calculation of the success probability of a candidate action based on an action feasibility matrix, provided in an embodiment of this disclosure. Figure 5 This is a schematic diagram illustrating how an interaction point is aligned to the geometric center of a target object based on a visual observation image of a new environment and natural language instructions, according to an embodiment of this disclosure. Figure 6 This is a schematic diagram of the second open scene embodied intelligent interaction method based on a multimodal large model provided in this disclosure embodiment; Figure 7 This is a schematic diagram of an open scene embodied intelligent interaction system based on a multimodal large model provided in this embodiment of the disclosure.

[0014] Figure label: 10. An open-scene embodied intelligent interaction system based on a multimodal large model; 11. Generation module; 12. Response module; 13. Calculation module; 14. Determination module; 15. Alignment module; 16. Execution module; 17. Monitoring module; 18. Update module. Detailed Implementation

[0015] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0016] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0017] Unless otherwise stated, the term "multiple" means two or more.

[0018] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0019] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0020] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0021] Combination Figure 1 As shown, this disclosure provides a first open-scene embodied intelligent interaction method based on a multimodal large model, including: S1, Generate a navigation grid and action feasibility matrix for the new environment; S2, in response to the user's input of a natural language command, generates candidate actions that match the natural language command; S3, Calculate the success probability of candidate actions based on the action feasibility matrix; S4, Optimize the candidate actions based on their success probabilities, and determine the final action to be executed; S5 aligns the interaction point to the geometric center of the target object based on the visual observation image of the new environment and natural language instructions. S6 controls the execution of the agent's actions based on the attributes of the final action.

[0022] When an agent enters a new environment, it automatically generates a navigation mesh and action feasibility matrix corresponding to the new environment. Specialized navigation mesh generation algorithms, such as the Recast algorithm, can be used to perform steps such as scene acquisition, voxelization, and region growing in the new environment, automatically dividing the traversable area into polygons. Action feasibility matrix. It consists of the initial confidence level of one or more actions performed on an object.

[0023] The system acquires and responds to natural language commands input by the user, and generates candidate actions that match the environmental layout. It's understandable that an agent, starting from its current location, may need to execute multiple consecutive decomposed actions before interacting with a target object. For example, if the agent is currently in the living room and the target object is in the kitchen, the agent needs to execute the following decomposed actions in sequence: walk straight, turn left, walk straight again, turn right, and then perform actions such as picking up, pressing, or pulling the target object. However, when the agent enters a new environment, it cannot be 100% certain to execute each of these decomposed actions. Therefore, one or more candidate actions are generated for each decomposed action, and the candidate actions corresponding to the same decomposed action constitute an independent set of candidate actions.

[0024] Based on the action feasibility matrix, the success probability of these candidate actions interacting with the target object is calculated. The optimal candidate action is then selected based on these success probabilities and chosen as the final action to be executed. It is understood that each set of candidate actions will generate a final action, and these final actions are also continuous in the time dimension.

[0025] When an agent enters a new environment, it simultaneously acquires visual observation images of that environment. It parses natural language commands to identify the target object, determines the object's position in the visual observation image, and aligns the interaction points to the object's geometric center.

[0026] Determine the attributes of the final action, such as whether it involves movement or manipulation. Based on these attributes, control the agent's actions to ensure smooth interaction between the agent and the target object.

[0027] The open-scene embodied intelligent interaction method based on a multimodal large model provided in this disclosure first generates a navigation grid and action executability matrix for the new environment when the agent enters it. After receiving natural language commands input by the user, matching candidate actions are generated, and the target object to be interacted with is parsed. The success rate of the candidate actions is calculated based on the action feasibility matrix, and the candidate actions are optimized to determine the final action to improve the success rate of the interaction. Then, based on the visual observation image and natural language commands of the new environment, the interaction point is aligned to the geometric center of the target object so that smooth interaction with the target object is possible. Finally, the agent's action execution is controlled based on the attributes of the final action, enabling the agent to perform movement, manipulation, and other actions smoothly and accurately. In this way, after entering a new environment, the agent autonomously perceives and analyzes the scene and semantics, makes decisions on the final action based on the action feasibility matrix, and controls the agent's action execution based on the attributes of the final action. This allows for better integration of semantic planning and physical execution, making it more adaptable to unknown environments.

[0028] Optionally, combined Figure 2 As shown, S1 generates a navigation mesh and action feasibility matrix for the new environment, including: S11, scan the scene of the new environment, generate a global navigation mesh, and mark walkable surfaces and static obstacles in the scene in the global navigation mesh; S12 uses the MLLM model to perform visual analysis on objects in the new environment, identifies object attributes, and constructs an initial action feasibility matrix based on object attributes and prior knowledge.

[0029] Environment initialization and prior knowledge construction Navigation Mesh Generation: When an agent enters a new environment, it first scans the scene to generate a global navigation mesh. This mesh marks walkable surfaces and static obstacles in the scene, providing basic geometric constraints for subsequent path planning.

[0030] Action Feasibility Matrix Initialization: An MLLM (Multimodal Large Language Model) is used to perform visual analysis of objects in the scene and identify their attributes. Based on common-sense prior knowledge (e.g., "handles are usually used to pull," "buttons are usually used to press"), an initial action feasibility matrix is ​​constructed. Elements in the matrix Indicates the object Execute action The initial confidence level. Optionally, the initial confidence level is set to a baseline value, such as 0.5.

[0031] Optionally, combined Figure 3 As shown, S2, in response to a natural language instruction input by the user, generates candidate actions that match the natural language instruction, including: S21, parse the natural language instructions to obtain the target task to be executed; S22, use the MLLM model to analyze the context of the target execution task and generate a set of semantically reasonable candidate actions; S23, output the probability of each candidate action based on semantic relevance.

[0032] Task parsing and candidate action generation (semantic stage) It receives natural language commands input by the user (such as "Go to the kitchen and get me an apple") and visual observation images of the current environment.

[0033] Parse natural language instructions to obtain the target execution task. Utilize an MLLM model as a high-level planner to analyze the context of the target execution task and generate a semantically reasonable set of candidate actions for each decomposed action. .

[0034] The MLLM model simultaneously outputs the probability of each candidate action based on semantic relevance. For example, targeting (When you turn on the TV), the probability of generating the action "press the power button" is 0.7, and the probability of generating "use the remote control" is 0.6.

[0035] Optionally, combined Figure 4 S3, Based on the action feasibility matrix, calculate the success probability of candidate actions, including: S31, query the current action feasibility matrix corresponding to the current moment, and obtain the physical interaction confidence of the current action feasibility matrix with respect to the target object; S32, calculate the power of candidate actions based on the probability of semantic relevance and the confidence of physical interaction.

[0036] Physical feasibility weighting and action reasoning (physical phase) The agent queries the feasibility matrix of the current action at time t. , obtain the physical interaction confidence of the current action feasibility matrix with respect to the current target object.

[0037] Combining the semantic relevance probability obtained from S23 with the physical interaction confidence, the adjusted action probability is calculated for each candidate action. : ; Existing technologies often employ discrete waypoint navigation or single-modal planning methods, which are prone to failure due to accumulated errors in long-running tasks. This embodiment integrates the high-level semantic reasoning capabilities of the MLLM model with a low-level verification mechanism based on a physical action feasibility matrix, constructing a closed-loop system of "perception-reasoning-execution-verification," effectively solving the problem of the disconnect between semantic planning and physical execution. This ensures that the selected action is not only semantically logical but also has a high physical execution success rate, significantly improving the success rate and robustness of complex long-running tasks.

[0038] Optionally, S4, the candidate actions are optimized based on their success probabilities to determine the final action to be executed, including: To prevent the agent from getting trapped in local optima and to encourage it to explore new strategies, a stochastic factor is introduced for action selection. The final formula for calculating the action to be executed is: ; in, For the final action to be performed; The probability of success for a candidate action; The randomness factor is a random noise value that follows a specific distribution. This allows the agent to occasionally try actions with lower confidence but potentially feasible actions while having a high probability of choosing high-confidence actions, in order to adapt to new environments.

[0039] Optionally, combined Figure 5 As shown, S5, based on the visual observation image of the new environment and natural language instructions, aligns the interaction point to the geometric center of the target object, including: S51, acquire depth information of the new environment, and project the two-dimensional pixel coordinates in the visual observation image into three-dimensional spatial coordinates; S52, compare the features of the detected target object with the expected target in the target's task; S53, If the comparison result indicates that the interaction point falls on the wrong object, then calculate the coordinate correction amount; S54, realign the interaction points to the geometric center of the target object based on the target correction amount.

[0040] Visual guidance and spatial alignment Coordinate mapping: Determines the final action to be performed. Then, using the depth information of the new environment obtained by the depth camera, the two-dimensional pixel coordinates in the visual observation image are projected and converted into three-dimensional spatial coordinates.

[0041] Semantic consistency verification: The agent compares the features of detected target objects with the expected targets in the task. If it finds that the interaction point falls on an adjacent incorrect object (e.g., the target is "cup," but the interaction point falls on "table"), the agent automatically calculates the coordinate correction. The interaction points are corrected based on coordinate adjustments, and then realigned to the geometric center of the semantically correct target object, eliminating erroneous operations caused by visual overlap.

[0042] Optionally, S6, based on the attributes of the final action, controls the agent's action execution, including: If the attribute of the final action is that the action involves movement, then a path is planned on the navigation grid, and a dynamic obstacle avoidance algorithm is used to avoid moving obstacles; spline interpolation is used to smooth the path.

[0043] If the attribute of the final action is that the action involves manipulation, then the corresponding parameterized primitive in the atomic action library is called, and the amplitude and trajectory of the action are adaptively adjusted according to the three-dimensional geometry of the target object to execute the pre-designed animation sequence.

[0044] Action execution (navigation and manipulation) Smooth navigation: If the final action involves movement, the agent plans a path on the global navigation grid and uses a dynamic obstacle avoidance algorithm to avoid moving obstacles. Unlike traditional polyline paths, spline interpolation is used to smooth the path, enabling the agent to move smoothly along the tangent direction of the path, simulating the natural gait of humans.

[0045] Atomic Action Execution: If the final action involves manipulation, the agent calls the corresponding parameterized primitive from the atomic action library. This library contains 23 predefined actions, such as pull, push, grab, press, toggle, and rotate. The agent adaptively adjusts the amplitude and trajectory of the action based on the three-dimensional geometry of the target object, executing a pre-designed animation sequence.

[0046] This solves the alignment problem between visual perception and physical space, and optimizes the motion path using spline interpolation technology, significantly improving the accuracy and efficiency of operation, and achieving high-precision operation control and an efficient and natural interactive experience. At the same time, thanks to continuous and smooth navigation path planning (eliminating mechanical pauses) and precise atomic action invocation, task completion time can be greatly shortened, establishing a new paradigm for efficient embodied interaction in open-world scenarios.

[0047] Optionally, combined Figure 6As shown, this disclosure provides a second open-scene embodied intelligent interaction method based on a multimodal large model, including: S1, Generate a navigation grid and action feasibility matrix for the new environment; S2, in response to the user's input of a natural language command, generates candidate actions that match the natural language command; S3, Calculate the success probability of candidate actions based on the action feasibility matrix; S4, Optimize the candidate actions based on their success probabilities, and determine the final action to be executed; S5 aligns the interaction point to the geometric center of the target object based on the visual observation image of the new environment and natural language instructions. S6 controls the execution of the agent's actions based on the attributes of the final action; S7 monitors the results of the action execution; S8. Update the feasibility matrix based on monitoring results: If the monitoring results indicate that an action has failed, the confidence level corresponding to the action is penalized exponentially to eliminate infeasible solutions. If the monitoring results indicate that an action has succeeded, the confidence level corresponding to the action increases rapidly to reinforce correct experiences.

[0048] Feedback and Matrix Update The agent monitors the result (success or failure) of the final action execution and updates the action feasibility matrix accordingly. To achieve self-optimization.

[0049] Failure Update (Exponential Decay): If an action fails, the confidence level corresponding to the action is penalized exponentially to quickly eliminate infeasible solutions. ; in, This represents the number of consecutive failures of the action. This is the attenuation factor. Optionally, The value is 0.8.

[0050] Successful Update (Enhanced Growth): If the action is successful, the confidence level corresponding to the action increases rapidly, reinforcing correct experience. ; This formula achieves a growth pattern of "halving the distance to the maximum value (1.0)," ensuring that confidence can be significantly improved after the first success.

[0051] Finally, the updated matrix is ​​based on the object feature set. Normalization is performed to complete one interactive loop.

[0052] In this way, to address the problem that traditional intelligent agents have difficulty adapting to new environments or new objects, this embodiment introduces a dynamic evolutionary feasibility matrix and a feature-based normalization mechanism, enabling the intelligent agent to have a "learning from experience" ability similar to humans. It can quickly adapt to unknown scenarios through a small amount of trial and error, giving the intelligent agent a strong environmental adaptability and cross-object generalization ability.

[0053] Combination Figure 7 As shown, this disclosure provides an open-scene embodied intelligent interaction system 10 based on a multimodal large model, including: a generation module 11, a response module 12, a calculation module 13, a determination module 14, an alignment module 15, an execution module 16, a monitoring module 17, and an update module 18. The generation module 11 is configured to generate a navigation mesh and an action feasibility matrix for the new environment. The response module 12 is configured to generate candidate actions matching natural language commands input by the user. The calculation module 13 is configured to calculate the success probability of the candidate actions based on the action feasibility matrix. The determination module 14 is configured to optimize the candidate actions based on their success probabilities and determine the final execution action. The alignment module 15 is configured to align the interaction points to the geometric center of the target object based on a visual observation image of the new environment. The execution module 16 is configured to control the agent to execute the final execution action based on its attributes. The monitoring module 17 is configured to monitor the result of the action execution. The update module 18 is configured to update the feasibility matrix based on the monitoring results.

[0054] The open-scene embodied intelligent interaction system based on a multimodal large model, as provided in this disclosure, constructs a closed-loop interaction system: an MLLM model serves as the high-level "brain" for semantic understanding and task decomposition, combined with a reusable atomic physical action library containing 23 parameterized primitives as the low-level "limbs," and a dynamically maintained "Feasibility Matrix" serves as an experience memory module to bridge the two. During operation, the system not only generates candidate actions based on visual semantics but also weights the selection of actions based on historical experience of physical feasibility. Through a "perception-reasoning-execution-feedback" cycle, the agent can continuously optimize the interaction strategy between objects and actions in unknown environments through trial and error and reinforcement, thereby achieving full-stack capabilities from long-range smooth navigation to fine-grained physical manipulation.

[0055] This disclosure provides an intelligent agent, including the aforementioned open-scene embodied intelligent interaction system based on a multimodal large model.

[0056] The following will use a home living environment as an example to explain in detail the workflow of the intelligent agent provided in the embodiments of this disclosure: 1. After entering a new home environment, the intelligent agent scans the living room, bedroom, kitchen, bathroom, and other scenes to generate a global navigation grid. Walkable surfaces and static obstacles are marked in the global navigation grid. Static obstacles include furniture, appliances, building structures and fixtures, decorations and storage items, etc. The size and location of static obstacles are proportionally reproduced in the grid.

[0057] The MLLM model is used to perform visual analysis on objects in the scene, identifying object attributes, i.e., identifying what the objects are. Based on prior knowledge, an initial action feasibility matrix is ​​established for these objects. .

[0058] 2. The intelligent agent receives the user's natural language command "Turn on the TV" and simultaneously acquires visual observation images of the current home environment.

[0059] Using an MLLM model to analyze the context of "turning on the TV", the agent's overall action is decomposed into six parts. For each decomposed action, a set of semantically reasonable candidate actions is generated as follows: {Turn left, turn right, go straight} {Turn left, turn right, go straight} {Turn left, turn right, go straight} {Turn left, turn right, go straight} {Find the TV, find the remote} {Press the power button to use the remote control}.

[0060] For the action "turn on the TV", the MLLM model simultaneously outputs the probability of each candidate action in the set of candidate actions based on semantic relevance. For example, for a set of candidate actions In terms of probability, the probability of generating the action "press the power button" is 0.7, and the probability of generating "use the remote control" is 0.6.

[0061] 3. Query the current time Action Feasibility Matrix Obtain the confidence level of physical interactions with "TV" (initially all are 0.5).

[0062] Combining semantic probability and physical confidence, according to the formula The adjusted action probabilities are calculated using the candidate action set. For example: The adjusted probability of "pressing the power button" is The probability of "using the remote control" is .

[0063] 4. Introduce a randomness factor According to the formula The final actions to be performed are determined in the following order: "Go straight", "Turn left", "Go straight", "Turn right", "Find the TV", "Press the power button".

[0064] 5. The intelligent agent acquires depth information through its own configured depth camera and projects the two-dimensional pixel coordinates in the visual image into three-dimensional spatial coordinates.

[0065] The agent compares the visually detected "television" with the expected target by feature comparison. If it finds that the interaction point falls on the "table", it automatically calculates the coordinate correction amount and realigns the interaction point to the geometric center of the "television".

[0066] 6. Based on the final execution actions of "go straight", "turn left", "go straight", "turn right" and "find TV", it can be determined that these actions involve movement. The agent plans a path on the global navigation grid and avoids moving obstacles by combining dynamic obstacle avoidance algorithm. Spline interpolation is used to smooth the path so that the agent can move smoothly along the tangent direction of the path, simulating the natural movement gait of humans.

[0067] Based on the fact that the final action is "press the power button", it can be determined that the action involves manipulation. The agent then calls the corresponding parameterized primitive in the atomic action library and adaptively adjusts the amplitude and trajectory of the action based on the three-dimensional geometry of the "television" to execute the pre-designed animation sequence.

[0068] 7. Check if the action of "pressing the power button" was successfully executed. If yes, the confidence level corresponding to this action increases. If no, the confidence level corresponding to this action is penalized exponentially. This process updates the action feasibility matrix.

[0069] The updated action executability matrix is ​​normalized based on the feature set of "television" to complete one interaction loop.

[0070] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0071] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included or substituted for parts and features of other embodiments. Furthermore, the terminology used herein is for descriptive purposes only and is not intended to limit the claims. Without further limitations, an element defined by the phrase "comprising a…" does not exclude the presence of additional identical elements in the process, method, or apparatus that includes said element. Throughout this document, each embodiment may emphasize differences from other embodiments, and similar or identical parts between embodiments may be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant details may be referred to the description of the method section.

[0072] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0073] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. An open-scene embodied intelligent interaction method based on a multimodal large model, characterized in that, include: Generate a navigation grid and action feasibility matrix for the new environment; In response to a natural language instruction input by the user, a candidate action matching the natural language instruction is generated; Calculate the success probability of the candidate action based on the action feasibility matrix; The candidate actions are optimized based on their success probabilities to determine the final action to be executed. Based on the visual observation image of the new environment and the natural language instructions, align the interaction point to the geometric center of the target object; The agent's action execution is controlled based on the attributes of the final action.

2. The open-scene embodied intelligent interaction method based on a multimodal large model according to claim 1, characterized in that, The generation of the navigation mesh and action feasibility matrix for the new environment includes: The scene of the new environment is scanned to generate a global navigation mesh, and walkable surfaces and static obstacles in the scene are marked in the global navigation mesh; The MLLM model is used to perform visual analysis on objects in the scene of the new environment, identify object attributes, and construct an initial action feasibility matrix based on the object attributes and prior knowledge.

3. The open-scene embodied intelligent interaction method based on a multimodal large model according to claim 1, characterized in that, The generation of candidate actions that match the natural language instruction includes: Parse the natural language instructions to obtain the target task to be executed; The context of the target task is analyzed using an MLLM model to generate a set of semantically reasonable candidate actions; Output the probability of each candidate action based on semantic relevance.

4. The open-scene embodied intelligent interaction method based on a multimodal large model according to claim 3, characterized in that, The step of calculating the success probability of the candidate action based on the action feasibility matrix includes: Query the current action feasibility matrix corresponding to the current moment to obtain the physical interaction confidence of the current action feasibility matrix for the target object; The power of the candidate action is calculated based on the probability of semantic relevance and the confidence level of physical interaction.

5. The open-scene embodied intelligent interaction method based on a multimodal large model according to claim 1, characterized in that, The step of optimizing the candidate actions based on their success probabilities to determine the final action to be executed includes: Introducing a randomness factor, the formula for calculating the final action is as follows: ； in, This is the final action to be performed; The randomness factor is a random noise value that follows a specific distribution; The success probability of the candidate action.

6. The open-scene embodied intelligent interaction method based on a multimodal large model according to claim 1, characterized in that, The step of aligning the interaction points to the geometric center of the target object based on the visual observation image of the new environment includes: The depth information of the new environment is obtained, and the two-dimensional pixel coordinates in the visual observation image are projected and converted into three-dimensional spatial coordinates. The detected target object is compared with the expected target in the target's task by feature comparison; If the comparison results indicate that the interaction point falls on an incorrect object, then calculate the coordinate correction amount; The interaction point is realigned to the geometric center of the target object based on the target correction amount.

7. The open-scene embodied intelligent interaction method based on a multimodal large model according to claim 1, characterized in that, The step of controlling the agent to execute the final action based on the attributes of the final action includes: If the attribute of the final action is that the action involves movement, then a path is planned on the navigation grid, and a dynamic obstacle avoidance algorithm is used to avoid moving obstacles; wherein, spline interpolation is used to smooth the path; If the attribute of the final action is that the action involves manipulation, then the corresponding parameterized primitive in the atomic action library is called, and the amplitude and trajectory of the action are adaptively adjusted according to the three-dimensional geometry of the target object to execute the pre-designed animation sequence.

8. The open-scene embodied intelligent interaction method based on a multimodal large model according to any one of claims 1 to 7, characterized in that, The method further includes: Monitor the results of the action execution; Update the feasibility matrix based on the monitoring results: If the monitoring results indicate that the action has failed, the confidence level corresponding to the action is penalized according to an exponential law, and infeasible solutions are excluded. If the monitoring results indicate that the action was successful, the confidence level corresponding to the action will increase rapidly, thus consolidating correct experience.

9. An open-scene embodied intelligent interaction system based on a multimodal large model, characterized in that, include: The generation module is configured to generate a navigation grid and action feasibility matrix for the new environment; The response module is configured to generate candidate actions that match the natural language instructions input by the user in response to such instructions. Calculation module; Configured to calculate the success probability of the candidate action based on the action feasibility matrix; The determination module is configured to optimize the candidate actions based on their success probabilities to determine the final action to be executed. The alignment module is configured to align the interaction points to the geometric center of the target object based on a visual observation image of the new environment. The execution module is configured to control the agent to execute the final execution action based on the attributes of the final execution action.

10. An intelligent agent, characterized in that, This includes the open-scene embodied intelligent interaction system based on a multimodal large model as described in claim 9.

Images