A somatic intelligent robot interaction simulation method and system

Abstract

The application provides a kind of body intelligent robot interaction simulation method and system, it is related to intelligent robot technical field, the method first obtains visual information in environment, contact force sensation information and language instruction information, and carries out time synchronization and space alignment processing, subsequently constructs the cause and effect semantic field including physical state layer, semantic availability layer and cause and effect graph layer, realizes the unified expression to environment and interactive relationship, then generates unified interaction representation vector by multi-modal unified embedding model, and carries out reasoning analysis to candidate action based on cause and effect relationship structure, carries out multi-branch interaction simulation to candidate action, obtains execution result under different environmental disturbance conditions, carries out comprehensive evaluation to simulation result by multi-objective evaluation model, determines optimal interaction strategy, and it is output for driving robot execution or for training control model, system can improve robot decision accuracy and execution stability under complex dynamic environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, specifically to an embodied intelligent robot interactive simulation method and system. Background Technology

[0002] With the continuous development of artificial intelligence technology and the widespread application of service robots, industrial robots, and human-robot collaborative systems, embodied intelligence has gradually become an important development direction for intelligent systems. Embodied intelligence emphasizes that intelligent agents, supported by computer systems, complete complex interactive tasks in real environments through the close coupling of perception, decision-making, and execution. Especially in service scenarios, intelligent manufacturing, and public environments, higher requirements are placed on robots' understanding, decision-making, and execution stability in dynamic environments. Currently, research on embodied intelligence mainly focuses on multimodal perception, path planning, and reinforcement learning. Among them, multimodal perception technology enhances environmental understanding by integrating visual, force, and linguistic information; path planning algorithms are used to generate the optimal path from the starting point to the target point; and reinforcement learning methods optimize decision-making strategies through trial and error mechanisms. These technologies have, to a certain extent, promoted the development of robot autonomy.

[0003] However, existing technologies face the following technical problems when dealing with complex dynamic environments and tasks with high interaction requirements: 1. Existing technologies lack the ability to uniformly express and deeply semantically fuse multimodal information. Current multimodal processing methods mostly adopt simple data splicing or independent feature extraction methods. For example, visual information and force information are input into different models for processing, and then fused through post-processing. This approach makes it difficult to express the intrinsic relationship between different modalities in a unified feature space. As a result, the system's understanding of the environment remains at the "perception layer" and lacks the ability to deeply semantically model the functional attributes and interactive meanings of objects. Therefore, it is easy to have misunderstanding biases or inaccurate decision-making in complex tasks. 2. Existing decision-making methods mainly rely on correlation-based prediction mechanisms and lack the ability to model causal relationships. Traditional path planning or reinforcement learning methods usually make decisions based on statistical patterns in historical data, such as optimizing paths through reward functions or predicting action outcomes through probability models. However, this method is difficult to distinguish between "causal relationships" and "chance correlations". When the environment changes or unseen situations occur, it is easy to cause decision failure or instability. Especially in tasks involving physical interaction, such as handling liquids or delicate operations, it is impossible to accurately predict the impact of actions on the results, thereby reducing the reliability of the system. 3. Existing technologies lack a pre-verification mechanism for multiple possible execution paths, making it difficult to cope with uncertainties in dynamic environments. In practical applications, environments often involve uncertainties such as personnel movement and changes in obstacles. Traditional methods typically generate only a single execution path and execute it directly, lacking the ability to pre-simulate and evaluate different execution scenarios. When the actual environment differs from the prediction, it can easily lead to task failure or security risks. Especially in highly dynamic scenarios, the system struggles to adjust its strategy in a timely manner, resulting in insufficient robustness and limiting its application effectiveness in complex real-world environments.

[0004] To address the aforementioned issues, there is an urgent need for an embodied intelligent robot interaction simulation method and system that can achieve unified expression of multimodal information and introduce causal reasoning and multi-branch simulation mechanisms. This would enable closed-loop optimization of environmental understanding, decision generation, and execution verification, thereby improving the robot's interaction stability, decision reliability, and overall robustness in complex dynamic environments. Summary of the Invention

[0005] Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides an embodied intelligent robot interaction simulation method and system, which solves the problems mentioned in the background technology above.

[0007] Technical solution To achieve the above objectives, the present invention provides the following technical solution: an embodied intelligent robot interactive simulation method, the method comprising the following steps: Sp1: Acquire multimodal data of the target environment, including environmental visual information, tactile force information, and externally input language command information, and perform time synchronization and spatial alignment processing on the multimodal data; Sp2: Construct a causal semantic field based on the multimodal data, wherein the causal semantic field includes a physical state layer, a semantic availability layer, and a causal relationship layer. The physical state layer is used to describe the spatial location, motion state, and physical attributes of each object in the environment. The semantic availability layer is used to describe the types of interactive actions and functional attributes that the objects can perform. The causal relationship layer is used to describe the causal dependency between actions and results. Sp3: Input the multimodal data into the multimodal unified embedding model to generate a unified interactive representation vector, so that visual features, force features and language semantic features are mapped to the same feature space; Sp4: Based on the causal relationship layer, perform causal reasoning on the robot candidate actions to obtain a set of multiple candidate actions that satisfy the task constraints; Sp5: Perform multi-branch interactive simulation for the candidate action set, construct multiple possible execution paths for each candidate action and generate corresponding simulation results; Sp6: Evaluate the simulation results and determine the optimal interaction strategy based on task completion, interaction stability, and execution consistency; Sp7: Output the optimal interaction strategy to drive the robot to execute or to train the robot control model.

[0008] Preferably, the causal relationship layer in Sp2 is constructed by learning the causal structure of historical interaction data, and the causal path between actions and changes in environmental state is determined by utilizing the dependencies between variables.

[0009] Preferably, the multimodal unified embedding model in Sp3 uses a cross-modal feature alignment mechanism to jointly encode data from different modalities, so that visual features, force features, and language features have consistent semantic representations in a unified vector space.

[0010] Preferably, the causal reasoning in Sp4 includes predicting the execution results of candidate actions based on the current environmental state, and filtering a set of actions that meet preset target constraints by comparing different action paths.

[0011] Preferably, the multi-branch interactive simulation in Sp5 includes performing multi-scenario simulations for each candidate action, generating multiple result branches under different environmental disturbance conditions, and evaluating the task success probability of each branch to assess the execution effect of the action under different conditions.

[0012] Preferably, the determination of the optimal interaction strategy in Sp6 is based on a multi-objective evaluation function, which includes a task success rate index, an action execution stability index, and a resource consumption index.

[0013] Preferably, the simulation results in Sp5 are used to generate new training scenario data, and the training scenario data is used to update the causal semantic field to improve the system's adaptability to complex environments.

[0014] Preferably, the system includes: The multimodal data acquisition module is used to collect visual information, tactile force information and externally input language command information from the environment, and to perform time synchronization and spatial registration processing on the collected data. The causal semantic field construction module is used to construct a unified representation model containing physical state information, semantic availability information, and causal relationship structure based on the multimodal data, so as to characterize the attributes of environmental objects and their interaction relationships. The multimodal unified embedding module is used to jointly encode visual data, force data, and language data, mapping information from different modalities to a unified feature representation space to generate a unified representation vector for interactive decision-making. The causal reasoning module is used to perform reasoning analysis on the robot's executable actions based on the causal relationship structure, generate a set of candidate actions that meet the task objectives, and predict the execution results of each candidate action. The multi-branch simulation module is used to perform multi-path simulation calculations on the candidate action set and generate multiple simulation execution results under different environmental conditions and disturbance factors to reflect the uncertainty of action execution. The strategy evaluation module is used to evaluate the simulation results, determine the optimal interaction strategy based on task completion, execution stability and consistency indicators, and output the results for robot control or model training.

[0015] Preferably, the causal semantic field construction module includes a causal relationship learning unit, which is used to extract causal dependencies between variables from historical interaction data and update the causal relationship structure.

[0016] Preferably, the multi-branch simulation module includes an environmental disturbance generation unit, which is used to construct simulation environments under different parameter conditions to improve the coverage and robustness of simulation results.

[0017] Beneficial effects This invention provides a method and system for interactive simulation of embodied intelligent robots. It has the following beneficial effects: 1. This invention constructs a causal semantic field and a multimodal unified embedding mechanism to collaboratively express visual information, force information, and linguistic semantic information in the same feature space. This transforms the relationship between different modalities from a simple splicing relationship to a unified representation system with structural correlation. As a result, the system can simultaneously characterize the physical attributes, functional attributes, and interactive semantic relationships of environmental objects, significantly improving the robot's overall understanding and semantic expression capabilities of complex environments. This enables the robot to form more accurate and consistent environmental cognition results when facing multi-object, multi-constraint interactive tasks.

[0018] 2. This invention introduces a causal relationship layer and a causal reasoning mechanism to upgrade the mapping relationship between actions and changes in environmental state from statistical correlation to causal dependence. This enables the system to explicitly consider the causal chain of actions leading to results during the decision-making process, thereby achieving forward-looking prediction of the consequences of candidate actions. This mechanism effectively avoids the decision-making bias problem that traditional methods are prone to in environmental changes or unknown scenarios, enabling robots to more accurately judge the feasibility and potential risks of actions in complex physical interaction tasks, thereby significantly improving the reliability and robustness of decision-making.

[0019] 3. This invention constructs multiple possible execution paths for each candidate action through a multi-branch interactive simulation mechanism and performs simulation verification under different disturbance conditions and environmental changes. This allows for a comprehensive evaluation of multiple possible outcomes before actual execution. This mechanism enables the system to identify high-risk paths in advance and select execution strategies with better stability, avoiding the failure risk caused by direct execution of a single path. At the same time, it improves the system's adaptability and fault tolerance in dynamic environments, enabling the robot to maintain a high task completion rate and execution stability in complex scenarios. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This is a diagram of the interactive simulation system architecture of the present invention; Figure 3 This is a display of the main interface of the interactive simulation system of the present invention. Figure 1 ; Figure 4 This is a display of the main interface of the interactive simulation system of the present invention. Figure 2 ; Figure 5 This is a user interface demonstration of the system operation process of the present invention. Figure 1 ; Figure 6 This is a user interface demonstration of the system operation process of the present invention. Figure 2 . Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific Implementation Example 1: Please see Figure 1 and Figure 2 As shown, an embodied intelligent robot interaction simulation method and system includes the following steps: Sp1, as the starting point of the entire method, plays a crucial role in transforming multi-source information from the real environment into unified input data. Since embodied intelligent robots rely on multiple information sources such as vision, force, and language when performing interactive tasks, and these different information sources differ significantly in terms of acquisition methods, data structures, and temporal and spatial dimensions, a specially designed multimodal data acquisition and alignment mechanism is needed to achieve unified data processing. To achieve this goal, the system is structured with a multimodal data acquisition module, which is built in a distributed deployment manner. The vision acquisition unit is located at the robot's head or the end of the manipulator to acquire environmental images and depth information, the force sensing unit is integrated at the end of the manipulator to acquire real-time changes in contact force, and the language parsing unit runs in the central processing unit to receive and parse user-input task commands. At the same time, a data synchronization and alignment unit is set as the core scheduling component to communicate with each acquisition unit, thereby forming a collaborative structure of "perception acquisition - centralized processing - unified output" at the system level, so that different modal data can be orderly integrated into the subsequent processing flow.

[0023] After the above modules are deployed, the system first enters the multimodal data acquisition process. During this process, the visual acquisition unit continuously acquires environmental images through camera devices and generates visual data containing color and spatial depth information, thereby reflecting the appearance characteristics and spatial distribution relationships of objects in the target environment. Simultaneously, the force sensing unit outputs contact force and torque data in real time when the robot makes contact actions. This data reflects the physical interaction state between the robot and the environment. Further, the language parsing unit receives natural language commands input by the user and transforms discrete text sequences into continuous semantic vectors through a language encoding model. In this process, an attention-based encoding model is used for semantic modeling, and its core calculation formula is: ; in, The term matrix is ​​used to represent the current semantic query. The key matrix is ​​used to represent the features in the input sequence. Value matrix, representing the corresponding semantic information. The dimension of the key vector is used for normalization to prevent excessively large values. The core function of this formula is to calculate the similarity between the query vector and the key vector and perform normalization to obtain the weight distribution. Then, the value vectors are weighted and summed to obtain the encoding result that highlights the key semantic information. This model is used to convert the natural language instructions input by the user into semantic vectors, enabling the system to understand the task objective and provide input for subsequent semantic availability analysis. Through the above process, the system completes the preliminary collection and encoding of visual information, force information and language information, thereby obtaining the original multimodal data set.

[0024] However, after data acquisition, it was found that different modalities exhibit significant inconsistencies in the time dimension. Visual data is typically acquired at a fixed frame rate, while force data has a higher sampling frequency, and language data displays discrete input characteristics. Directly inputting these data into subsequent models would lead to incorrect descriptions of the environmental state at any given moment, thus affecting decision-making accuracy. Therefore, time synchronization is necessary to establish a unified time reference. Based on this, the system calls the data synchronization and alignment unit to match the timestamps of each modality's data. A unified time point is determined by constructing a time error minimization objective, calculated using the following formula: ; in , , These represent timestamps for visual, force, and language data, respectively. This represents a unified time reference. The formula aims to find a point in time that minimizes the time deviation of all modal data, thus achieving time alignment. In practical applications, this point in time is used to construct a unified time frame, enabling different modal data to represent the environmental state at the same moment. After time matching, linear interpolation is used to compensate for data with higher sampling frequencies. The expression for this is: ; This formula is used to estimate the data value at the target time point when the data at adjacent time points are known, thereby ensuring that each modality has a corresponding value at a unified time point. Through this time synchronization process, the system achieves a consistent representation of multimodal data in the time dimension, providing a reliable foundation for subsequent fusion.

[0025] After unifying the temporal dimension, the next step is to address the inconsistency in spatial coordinates. Visual data is typically located in the camera coordinate system, while force data is located in the robotic arm coordinate system. Without a unified transformation, the true spatial relationships of objects in the environment cannot be accurately described. Therefore, the system continues with spatial alignment processing. During this process, the data synchronization and alignment unit calls pre-calibrated coordinate transformation parameters to map data from different sensor coordinate systems to a unified world coordinate system. A rigid body transformation model is used, and its expression is: ; in, Points in the sensor coordinate system A point in the world coordinate system. Rotation matrix Translation vectors achieve spatial coordinate unification through rigid body transformation. Furthermore, to improve computational efficiency and consistency of expression, homogeneous coordinates can be used for matrix operations, thereby unifying rotation and translation in the same matrix structure. In actual operation, the transformation parameters are corrected by minimizing the spatial error function to ensure alignment accuracy. When the spatial error exceeds a preset threshold, the system automatically triggers a recalibration mechanism to ensure the accuracy of the coordinate mapping relationship. Through the above spatial alignment process, different modal data achieve unified expression in spatial dimensions, enabling visual and force information to be correlated and analyzed in the same physical reference frame.

[0026] After completing time synchronization and spatial alignment, although the multimodal data has been unified in time and space, there are still differences in the numerical range and expression of various data. Therefore, further unified data representation processing is required to facilitate efficient processing by subsequent modules. At this stage, the system performs normalization and standardization operations on each modal data to ensure that different data dimensions have a consistent scale distribution. Visual information, force information, and linguistic semantic information are encapsulated according to a unified data structure to form a standardized data input set. This unified data set not only preserves the integrity of multimodal information but also has good structural consistency, making it directly usable as input for the subsequent causal semantic field construction module.

[0027] After completing Sp1, the system obtains a unified multimodal dataset aligned in both time and space dimensions. Although this dataset is structurally consistent, it remains at the "perceptual data layer" and lacks a deep expression of the relationships between environmental object attributes, interactive capabilities, and behavioral outcomes. Therefore, it cannot directly support complex interactive decisions. For this reason, the system needs to further structure the multimodal data in Sp2 to construct a unified representation system that can express "physical state - semantic function - causal relationship," namely, a causal semantic field. This step introduces a causal semantic field construction module to transform the original perceptual data into a structured knowledge representation with reasoning capabilities, thereby providing basic support for subsequent action generation and simulation.

[0028] In terms of system architecture, Sp2 is implemented by a causal semantic field construction module, which is deployed in the central computing unit and directly connected to the data interface output by Sp1, thereby achieving seamless data transfer. This module is further integrated into a physical modeling unit, a semantic availability analysis unit, and a causal relationship learning unit. The physical modeling unit is responsible for extracting the spatial and dynamic attributes of environmental objects from multimodal data, the semantic availability analysis unit is responsible for identifying the interactive capabilities of objects, and the causal relationship learning unit is responsible for establishing the causal dependency structure between actions and results. Through the collaborative work of the above three sub-units, the system can gradually advance from the "data layer" to the "semantic layer" and the "causal layer" to form a multi-level unified representation.

[0029] In the specific implementation process, the system first enters the physical state layer construction stage. Since Sp1 has completed spatial alignment processing, the visual data and force data are already in a unified coordinate system. At this point, physical attribute modeling of each object in the environment can be directly performed. In this process, the system first extracts the spatial position of each object in the environment from the visual data through object detection and 3D reconstruction algorithms, and calculates its 3D coordinates by combining depth information. At the same time, it estimates the velocity and motion state of the objects by combining continuous frame data, thereby forming a set of physical state representations. To achieve this process, the system uses a state estimation model to dynamically model the objects, and the physical state is represented by a set of vectors: ; in, : No. The position vector of an object, : Velocity vector Acceleration : quality or equivalent quality. Furthermore, to improve the stability of state estimation, the system introduces a state update model on the time series: ; ; By fusing continuous observation data, the physical state not only reflects instantaneous observation results but also has temporal continuity. Through the above process, the system completes the construction of the physical state layer, enabling each object in the environment to have a computable physical description.

[0030] After completing the physical state modeling, the system further enters the semantic availability layer construction phase. This layer is necessary because simple physical attributes cannot express what an object "can do." For example, the shape information of a cup cannot directly indicate its ability to be "grabbed" or "poured." Therefore, it is necessary to model the functional attributes of the object. In this process, the system calls the semantic availability analysis unit to jointly process visual features and linguistic information. The trained availability recognition model then classifies the object functionally. This model is essentially a multi-class classification model, and its expression is: ; in: : Object feature vector Feature extraction function Model weights The probability that an object will perform action 'a'. The function is defined as: ; Through this model, the system can identify the set of executable actions corresponding to an object, such as recognizing that an object has interactive capabilities such as grasping, pushing, or rotating. At the same time, linguistic information plays a constraining role at this stage. By semantically parsing the task instructions, the system can filter out the availability related to the task, thereby preventing irrelevant actions from entering the subsequent reasoning process. Through the construction of this semantic availability layer, the system achieves a semantic leap from "physical attributes" to "functional attributes".

[0031] After completing semantic layer modeling, the system further enters the causal relationship layer construction stage. The reason for constructing causal relationships is that in actual interactions, different actions lead to different results, and traditional correlation-based models cannot distinguish between "causal drive" and "accidental association." Therefore, causal modeling must clarify the dependency between actions and results. In this process, the system calls the causal relationship learning unit to analyze historical interaction data, extract the causal structure between variables, and construct a directed graph model to represent this relationship. This model can be represented as: ; in: A collection of variables, including object state, action variables, and result variables. : Causal edge set. Furthermore, to describe the functional relationships between variables, the system employs a structural causal model, whose basic form is: ; in: :variable The set of parent nodes, Causal function The noise term in this model represents variables determined by their causal parent nodes, thus enabling causal inference. Through this model, the system can make causal inferences about the outcome of an action under a specific state, rather than relying solely on statistical correlation. In practice, the system learns the causal structure by minimizing the joint loss function of prediction error and structural complexity, thereby obtaining a stable causal graph. The formula is as follows: ; in: Data fitting error Structural constraints are used to ensure the sparsity and rationality of the graph. : Weight coefficients. This loss function is used to ensure prediction accuracy while avoiding an overly complex causal graph. Once the model converges, the causal graph can be used for subsequent action inference and result prediction.

[0032] After the three layers mentioned above are constructed in sequence, the system integrates the physical state layer, the semantic availability layer, and the causal relationship layer to form a complete causal semantic field representation. This representation not only includes the physical attributes of objects in the environment, but also the functional capabilities of the objects and the causal relationship between actions and results, enabling the system to understand the environmental state from multiple perspectives. Furthermore, to ensure the reliability of the model, the system performs consistency detection on the causal semantic field. It judges whether the model is stable by evaluating the error between the causal prediction results and the actual observation results. When the error exceeds the threshold, the model update mechanism will be triggered to relearn the causal structure.

[0033] After completing Sp2, the system obtains a causal semantic field representation that includes a physical state layer, a semantic availability layer, and a causal relationship layer. This representation structurally realizes multi-level modeling of the environment. However, the data at this time exists in the form of discrete structures or multi-source features, and no unified numerical expression is formed between different modalities. This will make it difficult to directly carry out subsequent reasoning and optimization processes. Therefore, in order to achieve efficient fusion and unified computation of cross-modal information, the system introduces a multimodal unified embedding module in Sp3, which maps information from vision, force perception, and language to the same feature space, thereby generating a unified interactive representation vector, providing standardized input for subsequent causal reasoning and simulation.

[0034] In terms of system architecture, the multimodal unified embedding module is deployed inside the central computing unit and is directly connected to the causal semantic field construction module, thereby receiving the structured data output by Sp2. The multimodal unified embedding model is an end-to-end trained multi-input single-output mapping model, and its overall expression is as follows: ; in, Representing visual data, Representing force data, Represents the semantic vector of a language. Represents physical state layer information. Represents semantic availability information. This represents the structure of a cause-and-effect diagram. The model represents the unified interactive representation vector output. It achieves semantic alignment and feature fusion of data from different modalities through a unified learning mechanism, thereby forming a consistent vector representation. This module is constructed using a "multi-encoder + fusion network" architecture, which includes a visual encoding unit, a force encoding unit, a language encoding unit, and a cross-modal fusion unit. Each encoding unit is responsible for converting data from different modalities into feature vectors, while the fusion unit is responsible for aligning and integrating these features in a unified space. Through the collaborative work of this module, the system can achieve the transformation from "structured semantic representation" to "continuous vector representation".

[0035] In the specific implementation process, the system first performs modal encoding on the input data. Since the data output by Sp2 contains the physical state information, semantic labels, and causal relationship information of the objects, in the input stage, visual data mainly corresponds to the appearance and spatial structure features of the environment, force data corresponds to the dynamic feedback information in the interaction process, and language data corresponds to the task objective constraints. In the visual encoding process, the system uses a convolutional neural network to extract features from the image, the basic expression of which is: ; in This represents the input image data. Represents the convolution mapping function, output The visual feature vector encodes shape, texture, and spatial relationship information in the environment. After visual feature extraction, the system further encodes the force data. Since force data is typically a low-dimensional continuous signal, a multilayer perceptron is used for nonlinear mapping, expressed as follows: ; in This represents the input force vector. This represents the mapping function of the multilayer perceptron, which transforms the raw force sensory signal into a high-dimensional feature representation, thereby enhancing its expressive power. Simultaneously, the language encoding unit further abstracts and expresses the input semantic vector. Since the initial encoding is completed in Sp1, feature enhancement at this stage is mainly achieved through a deep semantic model, expressed as follows: ; in Represents the input language vector, This represents a semantic mapping function based on an attention mechanism. This process extracts key semantic information from the task and strengthens its weight in the overall representation. Furthermore, to enhance the model's semantic expressive power, the system also encodes structured information in the causal semantic field, resulting in: ; This allows unified embedding to include not only perceptual information, but also physical and semantic structural information.

[0036] After obtaining the feature vectors for visual, force, and linguistic features respectively, the system further enters the cross-modal fusion stage. This stage is necessary because the distributions of features from different modalities differ. Directly concatenating or superimposing them would lead to inconsistencies in the feature space, thus affecting subsequent inference performance. Therefore, the system achieves unified embedding through a cross-modal alignment mechanism. In this process, an attention-weighted fusion model is introduced, the core of which is as follows: ; in Representing different modal features, This represents the corresponding weight coefficient, which is calculated through the attention mechanism and is expressed as: ; in Represents the global query vector. The key vectors representing the features of each modality can be dynamically adjusted based on the current task requirements through the above calculations, thereby achieving adaptive fusion. To further achieve consistent representation of multimodal features in the same space, the system introduces cross-modal alignment constraints during the training phase, so that semantically similar samples from different modalities have closer representations in the embedding space. This constraint is usually implemented through contrastive learning, and its loss function is expressed as: ; in Represents the similarity function. Using temperature as the parameter, minimizing this loss function allows semantically consistent data from different modalities to be brought closer together in the feature space, thus achieving cross-modal alignment. After completing the above fusion and alignment process, the system finally outputs a unified interaction representation vector, which takes the form: This vector integrates visual environment information, physical interaction information, and task semantic information, and is the core input for subsequent causal reasoning and simulation calculations.

[0037] After completing Sp3, the system obtains a unified interaction representation vector. This vector integrates visual information, force information, linguistic semantics, and structural information of the causal semantic field in the same feature space, enabling data from different sources to have a consistent expression. Based on this unified representation, the system has the foundation for high-level decision-making reasoning. However, at this point, it lacks the decision result of "what action should be performed". Therefore, it is necessary to further combine causal relationships to reason and filter actions, thereby obtaining a set of candidate actions that meet the task objectives. Based on this requirement, the system introduces a causal reasoning module in Sp4, which realizes the mapping from state representation to action decision by utilizing the causal relationship layer.

[0038] In terms of system architecture, the causal inference module is deployed inside the central computing unit and is directly connected to the multimodal unified embedding module, thereby receiving the unified interaction representation vector output by Sp3. At the same time, this module maintains a data interface connection with the causal relationship layer constructed by Sp2, so that the inference process not only depends on the current state representation, but also can utilize the causal structure information learned from history. The module includes an action generation unit, a causal inference unit, and an action filtering unit. The action generation unit is used to construct a candidate action space, the causal inference unit is used to predict the possible results of each action, and the action filtering unit filters and sorts the actions according to the task constraints, thereby outputting the final set of candidate actions.

[0039] In the specific implementation process, the system first enters the candidate action generation stage. Since the unified interaction representation vector already contains the current environment state and task semantic information, the system can construct an initial action set based on this vector. This process can be represented as: ,in, This represents the initial candidate action set, which originates from a predefined action library or is generated through a policy network. To enhance the adaptability of action generation, the system typically combines the current state vector with conditional generation, enabling the action set to adapt to different task requirements. After constructing the candidate action set, the system further enters the causal inference stage. Causal inference is necessary because different actions will produce different results under different environmental conditions, and relying solely on statistical correlation cannot accurately determine the true effect of an action. Therefore, a causal model is needed to predict the action outcome. In this process, the system calls the causal relationship layer and performs intervention analysis on the action based on the structural causal model. Its core calculation form is as follows: ; in, Indicates the current state. Indicates the execution of an action , This expression represents the outcome variable and differs from traditional conditional probability by introducing... Operators represent active intervention in the system, thereby characterizing the true causal impact of actions on results. Furthermore, in the specific calculation process, the system unfolds reasoning paths based on a causal graph structure and estimates the results through a structural causal model, which is expressed as: ; in, This represents the set of parent nodes for the result variable. Represents a causal mapping function. This represents the noise term. Through this model, the system can predict the changes in the outcome variable given the action and the current state.

[0040] After predicting the causal outcome, the system further scores each candidate action to measure its effectiveness in the current task. The scoring function can be expressed as: ; in, This represents the task reward or goal completion rate. By calculating the expected value of the causal prediction results, an evaluation value for each action can be obtained. This score considers not only the direct effect of the action but also its long-term impact, thereby improving decision-making quality. After obtaining the scores for all candidate actions, the system enters the action filtering stage. Since actual tasks usually have constraints, such as stability constraints, safety constraints, or task completion requirements, actions need to be filtered. This process is represented as follows: ; in, The selection threshold is used to filter out low-quality actions or actions that do not meet the constraints, thus retaining the set of valid candidate actions. Furthermore, to improve the robustness of the decision-making, the system sorts the filtered actions and selects the highest-scoring actions to proceed to the subsequent simulation stage. This sorting process is represented as follows: ; Through the above steps, the system ultimately obtains a set of multiple candidate actions that satisfy the task constraints.

[0041] After completing Sp4, the system reasoned and screened candidate actions based on the causal relationship layer, obtaining a set of candidate actions that met the task constraints. However, the action evaluation at this stage still mainly relied on the prediction results of the causal model. In the real environment, the action execution process is often affected by uncertain disturbances, modeling errors, and dynamic changes in the environment. Therefore, relying solely on a single causal prediction may still result in bias. Based on this problem, the system needs to further dynamically verify the candidate actions in Sp5. By introducing a multi-branch simulation module, the execution effect of each candidate action under various possible environmental conditions is simulated, thereby obtaining more comprehensive and robust evaluation results and providing a basis for subsequent strategy selection.

[0042] In terms of system architecture, the multi-branch simulation module is deployed in the central computing unit and directly connected to the causal inference module, thereby receiving the candidate action set and unified interaction representation vector output by Sp4. The module includes an environment modeling unit, a disturbance generation unit, a trajectory simulation unit, and a result aggregation unit. The environment modeling unit is used to construct a computable simulation environment, the disturbance generation unit is used to generate various uncertainties, the trajectory simulation unit is used to simulate the action execution path, and the result aggregation unit is used to organize the simulation results under different branches. Through the synergistic effect of the above units, the system can realize the transformation from "single action evaluation" to "multi-scenario path evaluation".

[0043] In the specific implementation process, the system first constructs the initial simulation state based on the unified interactive representation vector. Since Sp3 encodes environmental state, semantic information, and causal information in a unified manner, this vector can be directly used as the initial input of the simulation environment, and it is represented as follows: ; Building upon this, the system enters the environment modeling phase, constructing state transition functions by calling the physics simulation engine to describe the evolution of the environment under the influence of actions. Its basic expression is as follows: ; in, Indicates the current state. Indicates the execution of an action. Indicates environmental disturbance. This represents the state transition function, which is implemented by a physical model or a neural network model and is used to characterize the dynamic impact of actions on the environmental state.

[0044] After completing the simulation environment construction, the system further enters the multi-branch generation stage. The reason for needing multiple branches is that in a real environment, the same action may produce different results under different perturbation conditions. Therefore, the system constructs multiple perturbation variable samples through perturbation generation units, expressed as follows: ; in, This indicates the number of branches. Each perturbation variable represents a possible environmental change, such as a change in the friction coefficient, external force interference, or sensing error. By combining different perturbation variables with candidate actions, the system can construct multiple execution paths. Based on this, the system performs multi-path simulation for each candidate action. For any given action... The system generates a corresponding set of multi-branch trajectories: ; Each trajectory consists of a sequence of states, defined as follows: ; By iteratively applying the state transition function, the state evolution path under each branch can be obtained, thereby simulating the execution process of actions under different environmental conditions.

[0045] After obtaining multiple trajectories, the system further evaluates the simulation results. To measure the performance of the action under different conditions, the system defines a task reward function, which is expressed as follows: ; in, This represents the instant reward function. As a discount factor, the overall performance of this branch can be obtained by accumulating the results across the entire trajectory. Furthermore, to comprehensively consider the results of different branches, the system performs expectation calculations on the multi-branch results of the same action, thereby obtaining the overall performance index of the action, which is expressed as: ; In addition, to reflect the stability of the action, the system also calculates the variance of the result: ; Among them, the smaller the variance, the more stable the action is under different environmental conditions. After completing the above calculation, the system integrates the multi-branch simulation results corresponding to each candidate action through the result aggregation unit, thereby forming an evaluation result set that includes expected return and stability index. This set not only reflects the average performance of the action, but also characterizes its robustness, providing a more comprehensive basis for subsequent decision-making.

[0046] After completing Sp5, the system generated multi-branch simulation results for each candidate action and obtained the trajectory set and corresponding reward distribution of each action under different perturbation conditions. These results reflect the execution effect of the action in various scenarios from a dynamic evolution perspective. However, the data obtained at this time still belongs to the "branch-level result set" and has not yet formed a unified decision basis. Therefore, it is necessary to further comprehensively evaluate these simulation results and conduct quantitative analysis of the action from multiple dimensions to determine the optimal interaction strategy. Based on this requirement, the system introduces a strategy evaluation module in Sp6. By constructing a multi-objective evaluation model, it conducts joint analysis of task completion, interaction stability, and execution consistency, thereby realizing the transformation from "simulation result set" to "optimal strategy output".

[0047] In terms of system architecture, the strategy evaluation module is deployed inside the central computing unit and is directly connected to the multi-branch simulation module, thereby receiving the trajectory set and statistical results output by Sp5. This module includes an index calculation unit, a multi-objective evaluation unit, and a strategy selection unit. The index calculation unit is used to extract the evaluation index of each action under different branches, the multi-objective evaluation unit is used to perform unified modeling of different indexes, and the strategy selection unit is used to output the final optimal interaction strategy. Through the collaborative work of the above units, the system can achieve multi-dimensional information fusion and decision output.

[0048] In the specific implementation process, the system first enters the index calculation stage. Since Sp5 obtains multiple trajectories corresponding to each candidate action, the system can calculate different evaluation indices based on the trajectory set. First, for the task completion index, the system measures the difference between the final state of the trajectory and the target state, which is expressed as: ; in, This indicates the termination state of the i-th trajectory. Let I(·) represent the target state set, and let I(·) be the indicator function used to determine whether the task has been completed. This indicator measures the success probability of an action under different branches. After calculating the task completion rate, the system further calculates the interaction stability index. Stability reflects the degree of fluctuation in the performance of actions under different disturbance conditions, and therefore, the return variance is used as a measure, expressed as: ; in, This represents the average return. Taking a negative value increases the evaluation value for higher stability, facilitating unified optimization. After calculating stability, the system further calculates the execution consistency index, which measures the similarity between different branch trajectories. The core idea is that if multiple branch trajectories are relatively similar, it indicates consistent action execution. This index is expressed as: ; in, The trajectory distance function can be represented by Euclidean distance or dynamic time warped distance. Taking a negative value indicates higher consistency and a higher score. After obtaining the above three core indicators, the system enters the multi-objective evaluation stage. Since there are trade-offs between different indicators (e.g., high completion may be accompanied by low stability), a unified evaluation function needs to be constructed to fuse multiple indicators. This evaluation function is expressed as: ; in, The weighting coefficients are used to reflect the importance of different indicators. These weights are set according to task requirements or automatically adjusted through learning. Through this multi-objective function, the system can uniformly map the evaluation results of different dimensions into a single scalar value, thereby achieving comparability.

[0049] Furthermore, to avoid biases caused by a single weighting method, the system introduces the Pareto optimization concept to perform multi-objective ranking of the action set. In this process, the system constructs the Pareto optimal set: ; in, This indicates that the action is not inferior in all metrics and is superior in at least one metric. This method can be used to select action sets that perform well across multiple objectives.

[0050] After completing the multi-objective evaluation, the system enters the strategy selection phase. In this phase, based on the comprehensive evaluation results, the optimal action is selected from the candidate action set, which is expressed as: ; Through this optimization process, the system can output the optimal interaction strategy.

[0051] After completing Sp6, the system determines the optimal interaction strategy from the candidate action set through a multi-objective evaluation model. This strategy exists in the form of a unified interaction representation vector and action parameters, which can theoretically achieve the optimal task execution effect. However, at this point, the strategy is still at the "decision result layer" and has not yet been truly applied to the robot body or used to improve the model's capabilities. Therefore, it is necessary to further transform the strategy into executable control instructions or use it to train the control model. Based on this requirement, the system introduces a strategy output and execution module in Sp7. By parsing, transforming, and distributing the optimal interaction strategy, a closed loop "from decision to execution" is achieved, and when necessary, it is transformed into training data to optimize the robot's control capabilities.

[0052] In terms of system architecture, the strategy output and execution module is deployed in the central computing unit and directly connected to the strategy evaluation module, thereby receiving the optimal interaction strategy output by Sp6. This module includes a strategy parsing unit, a control mapping unit, an execution scheduling unit, and a model training unit. The strategy parsing unit is used to transform abstract strategy vectors into structured action descriptions, the control mapping unit is used to map high-level actions into low-level control instructions, the execution scheduling unit is used to coordinate the robot's various actuators to complete actions, and the model training unit is used to transform simulation results and strategy information into training samples to continuously optimize the control model. Through the synergistic effect of the above units, the system realizes a dual-path output of the strategy, namely the "direct execution path" and the "model training path".

[0053] In the specific implementation process, the system first enters the strategy parsing stage. Since the optimal interaction strategy is usually represented in vector form, which contains information such as action type, target object, and execution parameters, it needs to be converted into a structured action expression, which is represented as follows: ; in, Indicates the object being operated on. Indicates the action type. This represents the action parameters. Through this parsing process, the system can determine the specific behavior that the robot needs to perform. After completing the policy parsing, the system enters the control mapping stage. Since the robot's execution depends on low-level control signals, such as joint angles, velocities, or torques, it is necessary to convert high-level actions into control inputs. In this process, the system uses a policy function for mapping, which is expressed as: ; in, Indicates the current state. This represents the control output strategy function. This represents the output control signal. The function is implemented by a neural network model. It takes the current state and target action information as input and outputs continuous control variables to drive the robot's execution. Furthermore, during execution, the system introduces a closed-loop control mechanism to adjust the actions in real time, expressed as follows: ; in, Indicates the desired state. Indicates the current state. This represents the control gain. Feedback adjustment can reduce execution errors, thereby improving motion accuracy and stability. After generating the control signal, the system distributes control commands to various robot actuators, such as the robotic arm and end effector, through the execution scheduling unit, thus completing the actual motion execution. During execution, the system continuously collects feedback data and compares it with the expected results to determine whether the execution effect meets expectations. When the deviation exceeds a threshold, the system triggers a strategy correction mechanism to achieve dynamic adjustment. Simultaneously, to further enhance the system's learning ability, the system also uses the optimal interaction strategy and its corresponding simulation trajectory for model training. In this process, the system constructs a training dataset in the following form: ; in, This represents the corresponding reward value. Based on this dataset, the system uses reinforcement learning to update the control model. That is, during policy optimization, the policy gradient method can be used, and its update formula is: ; By continuously optimizing strategy parameters, the model can directly generate better actions in future tasks, thereby reducing the reliance on simulation. The entire method forms a complete closed loop from perception, modeling, reasoning, simulation to execution and learning, enabling embodied intelligent robots to have adaptive interaction and continuous evolution capabilities.

[0054] Furthermore, during system operation, the central control unit performs unified scheduling, and each functional module executes sequentially according to a preset process to form a closed-loop control structure. Specifically, the system first acquires environmental information through a multimodal data acquisition module and transmits the processed unified data to a causal semantic field construction module for structured modeling. Subsequently, the multimodal unified embedding module fuses various types of information to generate a unified interaction representation vector. Based on this, the causal reasoning module generates a set of candidate actions and inputs it into a multi-branch simulation module for multi-scenario verification. The simulation results are further transmitted to a strategy evaluation module for comprehensive analysis. Finally, the strategy output module generates the optimal control command and drives the robot to execute it. During execution, the system continuously receives real-time feedback information from the perception layer and dynamically adjusts the current strategy based on the feedback results, thus forming a closed-loop operation mechanism of "perception-decision-execution-feedback".

[0055] In terms of data flow, the system adopts a layered transmission and unified interface mechanism for management. The modules communicate with each other through standardized data structures. Specifically, the multimodal data acquisition module outputs perception data in a unified format and transmits it as input to the causal semantic field construction module. This module outputs a structured environment representation and further transmits it to the multimodal unified embedding module for vectorization processing. The generated unified interaction representation vector serves as the core input and is called by the causal inference module to generate a set of candidate actions. Subsequently, the set of candidate actions and the environment state are transmitted to the multi-branch simulation module for simulation calculation. The simulation results are then input into the policy evaluation module for multi-objective analysis. Finally, the optimal policy is transformed into control instructions by the policy output module and fed back to the execution layer. Through the above data flow design, the system realizes the orderly transmission and efficient utilization of information between modules.

[0056] Furthermore, after completing the full technical process from Sp1 to Sp7, to further illustrate the feasibility and technical effectiveness of this method in real-world scenarios, the following section provides a comprehensive explanation of the entire solution using a specific application case: This implementation case is built in a smart restaurant service scenario. The restaurant has frequent personnel flow, limited passage space, and dynamic interference factors, such as customer movement, waiters walking around, and changes in the position of tables and chairs. The robot needs to complete the task of "smoothly delivering a bowl of hot soup from the serving area to table number 5 while avoiding sudden obstacles in the process". This task places high demands on path safety, motion stability, and real-time decision-making capabilities.

[0057] In terms of hardware architecture, the system adopts a layered connection structure for deployment. The mobile service robot serves as the main execution body, with a mobile chassis at the bottom for path movement and a robotic arm and tray at the top for carrying food. The robot's head is equipped with a visual perception device to acquire environmental images and depth information. The tray and the end of the robotic arm integrate force sensors to monitor the force and tilt status of the soup bowl during transportation. Meanwhile, the central computing unit is deployed inside the robot body, responsible for running multimodal data processing, causal reasoning, and simulation decision-making modules. In addition, auxiliary positioning devices or wireless communication nodes can be deployed in the restaurant environment to provide location information and task scheduling support. The various hardware modules are connected through an internal bus and wireless communication. The perceived data is transmitted to the central computing unit in real time for processing, and the decision results are then sent down to the mobile chassis and execution mechanism through the control interface, thus forming a closed-loop control system of "perception-decision-execution".

[0058] At the start of the task, the system first enters the environmental perception phase. The robot scans the restaurant interior using its vision system, identifying the location of the food delivery area, the location of table number 5, and the distribution of surrounding customers. Simultaneously, it detects obstacles in the floor passageways. At the same time, the system receives the task instruction "deliver the hot soup to table number 5" and combines this with the current environmental state to form unified input information. After completing the initial perception, the system performs semantic understanding of the environment, not only recognizing spatial structures but also analyzing the interaction relationships between various objects. For example, customers are dynamic obstacles, tables and chairs are static obstacles, and the soup bowl is an object requiring high stability, needing to avoid violent vibrations or tilting during transport. The system also analyzes the potential consequences of actions; for example, sharp turns may cause the soup to slosh, and narrow paths may increase the risk of collisions, thus establishing a correlation between actions and consequences. Subsequently, the system integrates the environmental state, task objective, and interaction relationships to form a unified task representation. Based on this, it generates multiple candidate execution plans, such as: one plan is to choose the shortest path to quickly reach the target location; another plan is to detour through densely populated areas to improve safety; and yet another plan is to slow down in critical areas to improve stability.

[0059] After obtaining multiple candidate solutions, the system does not execute them directly. Instead, it verifies each solution through an internal simulation mechanism. During this process, the system simulates various possible scenarios, such as customers suddenly crossing the aisle, waiters temporarily stopping, and localized congestion on the ground. It also predicts the performance of different solutions under these circumstances. Through this process, it can be found that, for example, the shortest path solution is more efficient under ideal conditions, but it is prone to congestion when there are many people, while the detour solution, although slightly longer, is safer and more stable overall. After completing multi-scenario verification, the system comprehensively evaluates each solution, focusing on whether it can successfully complete the task, whether the transportation process is smooth, and whether it has consistent performance under various conditions, thereby selecting the optimal execution strategy. For example, in this case, the system ultimately chooses a combination strategy of "detour + deceleration in key areas" to achieve a balance between efficiency and safety.

[0060] After determining the strategy, the system translates it into specific control actions, driving the robot to move to the food outlet and pick up the food. Then, it proceeds to table number 5 according to the planned path. During transportation, the system continuously monitors the stability of the soup bowl. If any shaking is detected, the system automatically adjusts its speed or path to ensure that the soup does not spill. At the same time, when a sudden obstacle is detected ahead, the system can quickly adjust its path or stop briefly to safely avoid it. Finally, the robot successfully delivers the hot soup to table number 5 and completes the delivery action. The entire process is smooth and reliable.

[0061] As can be seen from the above implementation process, this system has significant advantages in complex restaurant environments: First, in dynamic pedestrian environments, the system can avoid high-risk paths in advance through multi-scheme generation and simulation verification mechanisms, thereby significantly reducing the possibility of collisions or blockages. Second, in tasks with high stability requirements, such as transporting hot soup, the system can dynamically adjust its movement strategy according to environmental changes, effectively preventing liquid sloshing or spillage and improving service quality. Third, by predicting and evaluating various scenarios, the system's output decisions are more reasonable and stable, maintaining high consistency even in unexpected situations. In addition, because the system continuously receives feedback and makes adjustments during execution, it can continuously optimize its service strategy over long-term operation, improving overall efficiency and reliability.

[0062] This embodiment demonstrates that in a highly dynamic and complex interactive restaurant service environment, the embodied intelligent robot interaction simulation method can effectively select the optimal service path and execution strategy through multimodal perception, causal relationship modeling, and multi-branch simulation verification. This improves task completion efficiency while ensuring safety, and has good practical application value and promotion prospects.

[0063] For further details, please refer to Figure 3 As shown, it provides the main interface display of the embodied intelligent robot interactive simulation system. Figure 1 This section displays the overall system status and core function access points, including the system control panel, language command input area, and task description area. It also features a complete system workflow and a system reset icon for easy user operation and control. Please refer to [link / reference]. Figure 4 As shown, it provides the main interface display of the embodied intelligent robot interactive simulation system. Figure 2 This is used to display the overall system operating status and core function entry points, including a simulation evaluation index display area, a candidate action set, and causal semantic field information, etc. Please refer to [link / reference]. Figure 5 and Figure 6 As shown, it is in Figure 3 and Figure 4 The interface diagram after selecting the operation process is used to reflect the dynamic data changes during the actual operation of the system. Various indicators are updated and displayed in real time, including task success rate, execution stability, resource efficiency and causal reasoning accuracy in simulation evaluation indicators, as well as parameters such as the number of physical state layer objects, the number of semantic availability objects, the number of causal relationships and the unified embedding dimension in causal semantic field information, so as to intuitively reflect the system's decision-making and reasoning capabilities.

[0064] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0065] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for embodied intelligent robot interaction simulation, the method comprising: The method includes the following steps: ​ Sp1: Acquire multimodal data of the target environment, including environmental visual information, tactile force information, and externally input language command information, and perform time synchronization and spatial alignment processing on the multimodal data; Sp2: Construct a causal semantic field based on the multimodal data, wherein the causal semantic field includes a physical state layer, a semantic availability layer, and a causal relationship layer. The physical state layer is used to describe the spatial location, motion state, and physical attributes of each object in the environment. The semantic availability layer is used to describe the types of interactive actions and functional attributes that the objects can perform. The causal relationship layer is used to describe the causal dependency between actions and results. Sp3: Input the multimodal data into the multimodal unified embedding model to generate a unified interaction representation vector; Sp4: Based on the causal relationship layer, perform causal reasoning on the robot candidate actions to obtain a set of multiple candidate actions that satisfy the task constraints; Sp5: Perform multi-branch interactive simulation for the candidate action set, construct multiple possible execution paths for each candidate action and generate corresponding simulation results; Sp6: Evaluate the simulation results and determine the optimal interaction strategy; Sp7: Output the optimal interaction strategy, which is used to drive the robot to execute or to train the robot control model.

2. The embodied intelligent robot interaction simulation method of claim 1, wherein: The causal relationship layer in Sp2 is constructed by learning the causal structure of historical interaction data and using the dependencies between variables to determine the causal path between actions and changes in environmental state.

3. The embodied intelligent robot interaction simulation method of claim 1, wherein: The Sp3 multimodal unified embedding model uses a cross-modal feature alignment mechanism to jointly encode data from different modalities, enabling visual features, force features, and language features to have consistent semantic representations in a unified vector space.

4. The embodied intelligent robot interaction simulation method of claim 1, wherein: The causal reasoning in Sp4 includes predicting the execution results of candidate actions based on the current environmental state, and filtering the set of actions that meet the preset target constraints by comparing different action paths.

5. The embodied intelligent robot interaction simulation method of claim 1, wherein: The multi-branch interactive simulation in Sp5 includes performing multi-scenario simulations for each candidate action, generating multiple result branches under different environmental disturbance conditions, and evaluating the success probability of each branch to assess the execution effect of the action under different conditions.

6. The embodied intelligent robot interaction simulation method of claim 1, wherein: The determination of the optimal interaction strategy in Sp6 is based on a multi-objective evaluation function, which includes a task success rate index, an action execution stability index, and a resource consumption index.

7. The embodied intelligent robot interaction simulation method of claim 1, wherein: The simulation results in Sp5 are used to generate new training scenario data, and the training scenario data is used to update the causal semantic field to improve the system's adaptability to complex environments.

8. The system corresponding to the embodiment of the method for embodied conversational agent interaction simulation according to any one of claims 1-7, wherein: The system includes: The multimodal data acquisition module is used to collect visual information, tactile force information and externally input language command information from the environment, and to perform time synchronization and spatial registration processing on the collected data. The causal semantic field construction module is used to construct a causal semantic field containing a physical state layer, a semantic availability layer, and a causal relationship layer based on the multimodal data, so as to characterize the attributes of environmental objects and their interaction relationships. The physical state layer is used to describe the spatial location, motion state, and physical attributes of each object in the environment. The semantic availability layer is used to describe the types of interactive actions and functional attributes that the objects can perform. The causal relationship layer is used to describe the causal dependency relationship between actions and results. The multimodal unified embedding module is used to jointly encode visual data, force data, and language data, mapping information from different modalities to a unified feature representation space to generate a unified representation vector for interactive decision-making. The causal reasoning module is used to perform reasoning analysis on the robot's executable actions based on the causal relationship structure, generate a set of candidate actions that meet the task objectives, and predict the execution results of each candidate action. The multi-branch simulation module is used to perform multi-path simulation calculations on the candidate action set and generate multiple simulation execution results under different environmental conditions and disturbance factors to reflect the uncertainty of action execution. The strategy evaluation module is used to evaluate the simulation results and determine the optimal interaction strategy based on task completion, execution stability and consistency indicators. The strategy output and execution module is used to convert the optimal interaction strategy into robot control instructions and output them to the motion control system to drive the robot to complete the actual interaction task. At the same time, the execution results are fed back to the system for subsequent model updates and optimizations. 9.The system corresponding to the embodiment of the method for embodied intelligent robot interaction simulation according to claim 8, wherein: The causal semantic field construction module includes a causal relationship learning unit, which is used to extract causal dependencies between variables from historical interaction data and update the causal relationship structure.

10. The system corresponding to the embodiment of the method for embodied conversational agent interaction simulation according to claim 8, wherein: The multi-branch simulation module includes an environmental disturbance generation unit, which is used to construct simulation environments under different parameter conditions to improve the coverage and robustness of simulation results.

Images