Multi-mode perception and interaction method and device in personal environment

Abstract

The invention relates to the technical field of artificial intelligence and robots. According to the multi-modal perception and interaction method and device in the body environment, the method comprises the steps that environment entropy estimation processing is carried out through a dynamic weighting multi-modal feature fusion algorithm, and an environment entropy value representing the disorder degree of the environment is generated; performing cross-modal alignment processing to generate a fusion environment understanding map; performing dynamic decision processing through the task adaptive reinforcement learning model to generate an interaction instruction; driving an execution mechanism to execute the interaction action to obtain an execution result of the interaction action; carrying out dynamic adjustment processing on the weight of the environment entropy value to generate an updated environment entropy weight; and performing local knowledge node incremental updating processing on the meta-knowledge base to generate an optimized meta-knowledge base so as to solve the problems of insufficient consistency of cross-modal data and poor environmental understanding robustness caused by large distribution deviation between virtual features generated by a generation model in a noise or data missing scene and a real environment in related technologies.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and robotics technology, and in particular to a multimodal perception and interaction method and device in an embodied environment. Background Art

[0002] With the rapid development of artificial intelligence and robotics, there is a growing demand for embodied agents (such as service robots and industrial robotic arms) to operate autonomously in complex physical environments. This is particularly true in unstructured scenarios (such as disaster relief and warehouse sorting), where challenges such as highly dynamic environments, conflicting sensory data from multiple sources, and changing mission objectives must be overcome.

[0003] However, existing technologies have the following problems in multimodal perception and interaction: cross-modal data consistency is poor, and the virtual features generated by the generative model in noisy or data-missing scenarios have significant deviations from the real environment distribution, resulting in insufficient robustness in environmental understanding. Summary of the Invention

[0004] Based on this, it is necessary to provide a multimodal perception and interaction method and device in an embodied environment to address the above-mentioned technical problems, so as to solve the problems of insufficient consistency of cross-modal data in related technologies, and the large deviation between the virtual features generated by the generative model and the real environment distribution in noise or data missing scenarios, resulting in poor robustness of environmental understanding.

[0005] In a first aspect, the present application provides a multimodal perception and interaction method in an embodied environment, the method comprising:

[0006] Based on the environmental data collected by multimodal sensors, the environmental entropy is estimated and processed through a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the degree of environmental disorder. Based on the environmental entropy value, the pre-built meta-knowledge base is subjected to entropy-knowledge matching to generate cross-scenario modal association rules.

[0007] Based on cross-scenario modal association rules, a cross-modal generative adversarial network is used to generate virtual features for modal data that is missing or noisy, generating supplementary modal features that are consistent with the real environment distribution.

[0008] The supplementary modal features and real sensor data are cross-modally aligned through a spatiotemporal self-attention mechanism to generate a fused environmental understanding graph.

[0009] Based on the fusion of the environment understanding map and the task objectives, dynamic decision-making is performed through the task-adaptive reinforcement learning model to generate interactive instructions. Based on the interactive instructions, the actuator is driven to perform the interactive action and the execution result of the interactive action is obtained.

[0010] Based on the execution results of interactive actions, the weight of the environmental entropy value is dynamically adjusted through an incremental meta-learning framework to generate an updated environmental entropy weight; based on the updated environmental entropy weight, the local knowledge nodes of the meta-knowledge base are incrementally updated to generate an optimized meta-knowledge base.

[0011] Furthermore, based on the execution results of the interactive actions, the weight of the environmental entropy value is dynamically adjusted through the incremental meta-learning framework to generate an updated environmental entropy weight, including:

[0012] Based on the execution results of interactive actions, the task success rate and action stability indicators are extracted through the task performance quantification module;

[0013] Based on the task success rate and action stability indicators, the weight correction gradient of the environmental entropy value is calculated through the entropy weight dynamic regulator;

[0014] Based on the weight correction gradient, the weight of the environmental entropy value is incrementally iteratively optimized through a lightweight meta-learning framework to generate an updated environmental entropy weight.

[0015] Furthermore, based on the task success rate and action stability indicators, the weight correction gradient of the environmental entropy value is calculated through the entropy weight dynamic regulator, including:

[0016] Based on the task success rate and action stability indicators, dynamic correlation parameters between task relevance and environmental entropy are generated through task performance quantification parameter mapping processing;

[0017] Based on the dynamic correlation parameters, the gradient direction calibration module performs partial derivative calculation on the weight of the environmental entropy value to generate the initial gradient vector;

[0018] Based on the initial gradient vector, normalization and threshold truncation are performed through the gradient amplitude constraint module to generate the weight correction gradient.

[0019] Furthermore, based on the updated environmental entropy weights, the meta-knowledge base is incrementally updated with local knowledge nodes to generate an optimized meta-knowledge base, including:

[0020] Based on the updated environmental entropy weight, the knowledge graph topology relationship analysis module is used to filter local knowledge nodes related to the current task scenario;

[0021] Based on local knowledge nodes, the update amplitude of knowledge nodes is constrained through the meta-learning framework to generate incremental update parameters;

[0022] Based on the incremental update parameters, the updated meta-knowledge base is generalized and verified through the knowledge node consistency verification module to generate an optimized meta-knowledge base.

[0023] Furthermore, based on the fusion of the environment understanding map and the task objectives, dynamic decision-making is performed through the task-adaptive reinforcement learning model to generate interactive instructions, including:

[0024] Based on the task objectives, the task parsing module generates strategy constraint parameters and defines the feasible domain and priority rules of the interactive actions.

[0025] Based on the fusion environment understanding of the object material characteristics and spatial topological relationships in the atlas, a dynamic policy network is used to generate a reward evaluation matrix for candidate interaction actions;

[0026] Based on the policy constraint parameters and the reward evaluation matrix, the action decision engine selects the interaction instructions that maximize the task benefits.

[0027] Furthermore, based on the material characteristics and spatial topological relationships of objects in the fusion environment understanding graph, a reward evaluation matrix for candidate interaction actions is generated through a dynamic policy network, including:

[0028] Based on the material characteristics of the object and the spatial topological relationship, the material-action compatibility parameters and spatial accessibility constraint parameters are extracted through the physical interaction rule library;

[0029] Based on the compatibility parameters and reachability constraint parameters, a multi-objective optimization weight adaptation module is used to generate a balance weight between task safety and efficiency.

[0030] Based on the balance weights, the physical feasibility and task benefits of candidate interaction actions are jointly evaluated through a dynamic policy network to generate a multi-dimensional reward evaluation matrix.

[0031] Furthermore, based on the environmental data collected by the multimodal sensor, the environmental entropy is estimated and processed by a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value representing the degree of environmental disorder, including:

[0032] Based on the visual, tactile and lidar data collected by multimodal sensors, multimodal feature extraction and processing are performed to generate visual texture complexity features, tactile pressure distribution features and lidar point cloud density features;

[0033] Based on the visual texture complexity features, tactile pressure distribution features and lidar point cloud density features, the dynamic weight allocation module generates the environmental complexity evaluation parameters of each modal feature;

[0034] Based on the environmental complexity evaluation parameters, the multimodal features are weighted and aggregated through the entropy fusion module to generate the environmental entropy value that represents the environmental disorder.

[0035] In a second aspect, the present application further provides a multimodal perception and interaction device in an embodied environment, the device comprising:

[0036] The dynamic entropy fusion-knowledge matching module is used to estimate environmental entropy based on environmental data collected by multimodal sensors using a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the degree of environmental disorder. Based on the environmental entropy value, it performs entropy-knowledge matching on the pre-built meta-knowledge base to generate cross-scenario modal association rules.

[0037] The cross-modal adversarial compensation module is used to generate virtual features for missing or noisy modal data based on cross-scene modal association rules through a cross-modal generative adversarial network, generating supplementary modal features that are consistent with the real environment distribution;

[0038] The spatiotemporal self-attention alignment module is used to perform cross-modal alignment processing on the supplementary modal features and the real sensor data through the spatiotemporal self-attention mechanism to generate a fused environment understanding map;

[0039] The task-adaptive reinforcement decision-making module is used to understand the graph and task objectives based on the fusion environment, perform dynamic decision-making through the task-adaptive reinforcement learning model, and generate interactive instructions. Based on the interactive instructions, it drives the actuator to perform interactive actions and obtain the execution results of the interactive actions.

[0040] The incremental meta-learning-knowledge base optimization module is used to dynamically adjust the weight of the environment entropy value based on the execution results of interactive actions through the incremental meta-learning framework to generate updated environment entropy weights; based on the updated environment entropy weights, the local knowledge nodes of the meta-knowledge base are incrementally updated to generate an optimized meta-knowledge base.

[0041] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any method in the first aspect of the present application are implemented.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method in the first aspect of the present application when the computer program is executed by a processor.

[0043] The technical solution provided by the present application includes the following technical effects: by providing a multimodal perception and interaction method and device in an embodied environment, the method includes: based on the environmental data collected by the multimodal sensor, the environmental entropy is estimated by a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the disorder of the environment; based on the environmental entropy value, the pre-built meta-knowledge base is entropy-matched to generate cross-scene modal association rules; based on the cross-scene modal association rules, the modal data that is missing or interfered with by noise is virtualized by a cross-modal generative adversarial network to generate supplementary modal features that are consistent with the real environment distribution; the supplementary modal features and the real sensor data are cross-modally aligned by a spatiotemporal self-attention mechanism to generate The method forms a fused environmental understanding map; based on the fused environmental understanding map and the task objectives, dynamic decision-making is performed through the task-adaptive reinforcement learning model to generate interactive instructions; based on the interactive instructions, the actuator is driven to perform interactive actions to obtain the execution results of the interactive actions; based on the execution results of the interactive actions, the weights of the environmental entropy values are dynamically adjusted through the incremental meta-learning framework to generate updated environmental entropy weights; based on the updated environmental entropy weights, the meta-knowledge base is incrementally updated with local knowledge nodes to generate an optimized meta-knowledge base, so as to solve the problems of insufficient cross-modal data consistency in related technologies and poor robustness of environmental understanding caused by large deviations between the virtual features generated by the generative model and the real environment distribution in noise or data missing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flowchart of a multimodal perception and interaction method in an embodied environment in one embodiment of the present invention;

[0046] Figure 2 FIG. 4 is a structural diagram of a multimodal perception and interaction device in an embodied environment in one embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present application more clearly understood, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to fully understand the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0048] like Figure 1 As shown, the present application provides a multimodal perception and interaction method in an embodied environment, the method comprising:

[0049] S101: Based on the environmental data collected by multimodal sensors, the environmental entropy is estimated and processed through a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the degree of environmental disorder; based on the environmental entropy value, the pre-built meta-knowledge base is subjected to entropy-knowledge matching processing to generate cross-scenario modal association rules.

[0050] Specifically, multimodal sensors are used to collect visual, tactile, auditory and other data in the environment to perceive the state of the environment. Afterwards, the collected multimodal data is processed using a dynamic weighted multimodal feature fusion algorithm. This algorithm dynamically adjusts the weights of each modal feature according to the complexity of the environment to more accurately reflect the disorder of the environment. The algorithm extracts features of each modal data, such as visual texture complexity and tactile pressure distribution, and calculates environmental complexity evaluation parameters based on the above features. Based on the above parameters, the multimodal features are weighted and aggregated through the entropy fusion module, and then an environmental entropy value representing the disorder of the environment is generated. Then, based on the generated environmental entropy value, an entropy-knowledge matching process is performed on the pre-built meta-knowledge base. By matching the environmental entropy value with the knowledge nodes in the knowledge base, cross-scenario modal association rules applicable to the current environment are generated. The above rules can guide effective perception and interaction in different scenarios.

[0051] S102: Based on the cross-scene modal association rules, virtual feature generation processing is performed on the missing or noise-interfered modal data through a cross-modal generative adversarial network to generate supplementary modal features that are consistent with the real environment distribution.

[0052] Specifically, the collected multimodal data (such as vision, touch, and hearing) are input into a cross-modal generative adversarial network (GAN). The generative adversarial network consists of a generator and a discriminator. The generator is responsible for generating virtual features based on association rules and existing modal data to fill in the missing or noisy modal data. The discriminator is used to distinguish the differences between the generated virtual features and the real modal data, and continuously optimize the output of the generator through adversarial training. During the training process, the generator gradually adjusts the generated virtual features based on the feedback from the discriminator to make them as close as possible to the distribution of the real environment. Afterwards, the generated supplementary modal features are consistent in distribution with the real sensor data, ensuring that subsequent environmental understanding and interaction tasks can be processed based on complete and accurate multimodal data.

[0053] S103: The supplementary modal features and real sensor data are cross-modally aligned through the spatiotemporal self-attention mechanism to generate a fusion environment understanding map.

[0054] Specifically, the supplementary modal features generated by a cross-modal generative adversarial network and real sensor data are input into the spatiotemporal self-attention mechanism module. This self-attention mechanism dynamically learns the weight relationships between features from different modalities, thereby achieving cross-modal feature alignment. The spatiotemporal self-attention mechanism captures the correlations between features from different modalities in both time and space, aligning the feature distributions of different modalities by assigning attention weights. After feature alignment, the aligned features are fused to generate a comprehensive feature representation. This fused feature more comprehensively reflects the multimodal information of the environment, providing a more accurate data foundation for subsequent environmental understanding and interaction tasks. Subsequently, a fused environmental understanding graph is generated based on these fused features to guide the agent's perception and interaction in complex environments.

[0055] S104: Based on the fusion environment understanding map and task objectives, dynamic decision processing is performed through the task adaptive reinforcement learning model to generate interaction instructions; based on the interaction instructions, the actuator is driven to perform the interaction action to obtain the execution result of the interaction action.

[0056] Specifically, the fused environmental understanding graph is used as input, and dynamic decision-making is performed using a task-adaptive reinforcement learning model. This model adjusts its policy in real time based on the current environmental state and task objectives to ensure optimal decision-making. By learning from historical interaction data, a policy network capable of adapting to different environments and tasks is constructed. Based on the input fused environmental understanding graph, this network generates a series of candidate interaction actions, evaluates these actions, and selects the optimal one as output. The generated interaction instructions are transmitted to the actuator, driving it to perform the corresponding interaction action. Based on the instructions, the actuator, through its mechanical structure and control system, performs specific physical actions such as grasping, moving, and rotating. During execution, the actuator monitors the execution of the action in real time to ensure its accuracy and stability.

[0057] After the interaction is complete, sensor feedback is used to obtain the results of the interaction. These results include information such as whether the action was successfully completed and whether the environment state has changed. These results are fed back and used for further decision-making and learning to optimize future interaction strategies. This process enables continuous adjustment and optimization of behavior based on dynamic changes in the environment and task objectives.

[0058] S105: Based on the execution results of the interactive action, the weight of the environmental entropy value is dynamically adjusted through the incremental meta-learning framework to generate an updated environmental entropy weight; based on the updated environmental entropy weight, the local knowledge nodes of the meta-knowledge base are incrementally updated to generate an optimized meta-knowledge base.

[0059] Specifically, based on the execution results of interactive actions, the task success rate and action stability indicators are extracted through the task performance quantification module. Then, based on the above indicators, the weight correction gradient of the environmental entropy value is calculated through the entropy weight dynamic regulator. Among them, the entropy weight dynamic regulator will generate dynamic correlation parameters between task relevance and environmental entropy value based on the task success rate and action stability indicator. The gradient direction calibration module calculates the partial derivative of the weight of the environmental entropy value to generate an initial gradient vector, and the gradient amplitude constraint module performs normalization and threshold truncation to generate a weight correction gradient. Then, based on the weight correction gradient, the lightweight meta-learning framework is used to perform incremental iterative optimization on the weight of the environmental entropy value to generate an updated environmental entropy weight.

[0060] Based on the updated environment entropy weight, the knowledge graph topology relationship analysis module is used to screen local knowledge nodes related to the current task scenario. Based on the above local knowledge nodes, the meta-learning framework is used to constrain the update amplitude of the knowledge nodes and generate incremental update parameters. After that, the updated meta-knowledge base is verified for generalization performance through the knowledge node consistency verification module to generate an optimized meta-knowledge base.

[0061] An embodiment of the present application provides a multimodal perception and interaction method in an embodied environment, including: based on environmental data collected by multimodal sensors, performing environmental entropy estimation processing through a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that characterizes the disorder of the environment; based on the environmental entropy value, performing entropy-knowledge matching processing on a pre-built meta-knowledge base to generate cross-scene modal association rules; based on the cross-scene modal association rules, performing virtual feature generation processing on modal data that is missing or interfered with by noise through a cross-modal generative adversarial network to generate supplementary modal features that are consistent with the real environment distribution; performing cross-modal alignment processing on the supplementary modal features and the real sensor data through a spatiotemporal self-attention mechanism to generate a fused environment understanding map. ; Based on the fusion of environmental understanding map and task objectives, dynamic decision-making processing is performed through the task adaptive reinforcement learning model to generate interactive instructions; based on the interactive instructions, the actuator is driven to perform interactive actions to obtain the execution results of the interactive actions; based on the execution results of the interactive actions, the weights of the environmental entropy values are dynamically adjusted through the incremental meta-learning framework to generate updated environmental entropy weights; based on the updated environmental entropy weights, the meta-knowledge base is incrementally updated with local knowledge nodes to generate an optimized meta-knowledge base, so as to solve the problems of insufficient cross-modal data consistency in related technologies, and the large deviation between the virtual features generated by the generative model in noise or data missing scenarios and the real environment distribution, resulting in poor robustness of environmental understanding.

[0062] Furthermore, based on the execution results of the interactive actions, the weight of the environmental entropy value is dynamically adjusted through the incremental meta-learning framework to generate an updated environmental entropy weight, including:

[0063] Based on the execution results of interactive actions, the task success rate and action stability indicators are extracted through the task performance quantification module;

[0064] Based on the task success rate and action stability indicators, the weight correction gradient of the environmental entropy value is calculated through the entropy weight dynamic regulator;

[0065] Based on the weight correction gradient, the weight of the environmental entropy value is incrementally iteratively optimized through a lightweight meta-learning framework to generate an updated environmental entropy weight.

[0066] Specifically, based on the execution results of interactive actions, the task performance quantification module extracts task success rates and action stability indicators. The task success rate measures the completion of interactive actions, while the action stability indicator assesses the stability of the execution process. Based on these indicators, the entropy weight dynamic regulator calculates the weight-corrected gradient of the environmental entropy value. Based on the task success rate and action stability indicator, the entropy weight dynamic regulator generates dynamic correlation parameters between task relevance and environmental entropy value. The gradient direction calibration module calculates the partial derivative of the environmental entropy weight to generate an initial gradient vector. The gradient amplitude constraint module performs normalization and threshold truncation to generate a weight-corrected gradient. Based on the weight-corrected gradient, the lightweight meta-learning framework incrementally iteratively optimizes the environmental entropy weights to generate updated environmental entropy weights. This process ensures that the environmental entropy weights can be dynamically adjusted based on the interaction results, more accurately reflecting the disorder of the environment and providing a more precise basis for subsequent environmental understanding and decision-making.

[0067] Furthermore, based on the task success rate and action stability indicators, the weight correction gradient of the environmental entropy value is calculated through the entropy weight dynamic regulator, including:

[0068] Based on the task success rate and action stability indicators, dynamic correlation parameters between task relevance and environmental entropy are generated through task performance quantification parameter mapping processing;

[0069] Based on the dynamic correlation parameters, the gradient direction calibration module performs partial derivative calculation on the weight of the environmental entropy value to generate the initial gradient vector;

[0070] Based on the initial gradient vector, normalization and threshold truncation are performed through the gradient amplitude constraint module to generate the weight correction gradient.

[0071] Specifically, based on task success rate and action stability indicators, these indicators are converted into dynamic correlation parameters between task relevance and environmental entropy through task performance quantification parameter mapping. The task relevance parameter measures the correlation between the current task and the environmental entropy value, while the dynamic correlation parameter reflects the impact of the environmental entropy value on the task execution results. Using these dynamic correlation parameters, the gradient direction calibration module then calculates the partial derivative of the environmental entropy weight to generate an initial gradient vector. The partial derivative calculation determines the direction of weight adjustment to ensure that the adjustment maximizes the task benefit. Based on the initial gradient vector, the gradient amplitude constraint module then performs normalization to ensure that the gradient vector amplitude is within a reasonable range. A threshold truncation is then performed to prevent the gradient from being too large or too small, thereby generating a weight correction gradient. This process ensures that the environmental entropy weight can be dynamically adjusted based on the task execution results, improving adaptability and robustness.

[0072] Furthermore, based on the updated environmental entropy weights, the meta-knowledge base is incrementally updated with local knowledge nodes to generate an optimized meta-knowledge base, including:

[0073] Based on the updated environmental entropy weight, the knowledge graph topology relationship analysis module is used to filter local knowledge nodes related to the current task scenario;

[0074] Based on local knowledge nodes, the update amplitude of knowledge nodes is constrained through the meta-learning framework to generate incremental update parameters;

[0075] Based on the incremental update parameters, the updated meta-knowledge base is generalized and verified through the knowledge node consistency verification module to generate an optimized meta-knowledge base.

[0076] Specifically, based on the updated environmental entropy weights, the knowledge graph topology analysis module identifies local knowledge nodes closely related to the current task scenario. These nodes are the parts of the meta-knowledge base most relevant to the current environment and task objectives. A meta-learning framework is then used to constrain the update amplitude of these local knowledge nodes, ensuring that the update process is stable and does not deviate excessively from the existing knowledge structure. This framework generates incremental update parameters to guide the direction and extent of knowledge node updates. The updated meta-knowledge base is then verified for generalization performance using a knowledge node consistency verification module. This module checks the consistency and validity of the updated knowledge nodes across different scenarios, ensuring the robustness and adaptability of the meta-knowledge base. After verification, an optimized meta-knowledge base is generated, enabling it to better support intelligent agents' decision-making and interactions in complex environments.

[0077] Furthermore, based on the fusion of the environment understanding map and the task objectives, dynamic decision-making is performed through the task-adaptive reinforcement learning model to generate interactive instructions, including:

[0078] Based on the task objectives, the task parsing module generates strategy constraint parameters and defines the feasible domain and priority rules of the interactive actions.

[0079] Based on the fusion environment understanding of the object material characteristics and spatial topological relationships in the atlas, a dynamic policy network is used to generate a reward evaluation matrix for candidate interaction actions;

[0080] Based on the policy constraint parameters and the reward evaluation matrix, the action decision engine selects the interaction instructions that maximize the task benefits.

[0081] Specifically, the task parsing module conducts an in-depth analysis of the task objectives and extracts key policy constraint parameters. These parameters define the feasible domain and priority rules for interactive actions, ensuring that they meet the basic requirements and constraints of the task. Next, a dynamic policy network is used, combined with the material characteristics and spatial topological relationships of objects in the fusion environment understanding graph, to generate a reward evaluation matrix for candidate interactive actions. This matrix comprehensively considers the physical feasibility of the action, the task payoff, and the environmental state, assigning a reward value to each candidate action. The action decision engine then uses an optimization algorithm based on the policy constraint parameters and the reward evaluation matrix to select the interactive instruction that maximizes the task payoff. This instruction is then used to drive the actuator, ensuring that it can complete the task efficiently and accurately in complex environments.

[0082] Furthermore, based on the material characteristics and spatial topological relationships of objects in the fusion environment understanding graph, a reward evaluation matrix for candidate interaction actions is generated through a dynamic policy network, including:

[0083] Based on the material characteristics of the object and the spatial topological relationship, the material-action compatibility parameters and spatial accessibility constraint parameters are extracted through the physical interaction rule library;

[0084] Based on the compatibility parameters and reachability constraint parameters, a multi-objective optimization weight adaptation module is used to generate a balance weight between task safety and efficiency.

[0085] Based on the balance weights, the physical feasibility and task benefits of candidate interaction actions are jointly evaluated through a dynamic policy network to generate a multi-dimensional reward evaluation matrix.

[0086] Specifically, material-action compatibility parameters and spatial accessibility constraint parameters are extracted from the physical interaction rule library. The material-action compatibility parameters measure the degree of compatibility between different materials and interactive actions, while the spatial accessibility constraint parameters evaluate the feasibility of interactive actions under the current spatial layout. Subsequently, a multi-objective optimization weight adaptation module dynamically generates balance weights based on the safety and efficiency requirements of the task. This module automatically adjusts the weights based on task priority to ensure maximum task efficiency while ensuring safety. Based on the generated balance weights, the dynamic policy network jointly evaluates the physical feasibility and task benefits of candidate interactive actions to generate a multi-dimensional reward evaluation matrix. This matrix comprehensively considers the physical constraints of the action, the task benefits, and the environmental state, and assigns a comprehensive reward value to each candidate action, which is used to guide the action decision engine in selecting the optimal interaction instruction.

[0087] Furthermore, based on the environmental data collected by the multimodal sensor, the environmental entropy is estimated and processed by a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value representing the degree of environmental disorder, including:

[0088] Based on the visual, tactile and lidar data collected by multimodal sensors, multimodal feature extraction and processing are performed to generate visual texture complexity features, tactile pressure distribution features and lidar point cloud density features;

[0089] Based on the visual texture complexity features, tactile pressure distribution features and lidar point cloud density features, the dynamic weight allocation module generates the environmental complexity evaluation parameters of each modal feature;

[0090] Based on the environmental complexity evaluation parameters, the multimodal features are weighted and aggregated through the entropy fusion module to generate the environmental entropy value that represents the environmental disorder.

[0091] Specifically, multimodal feature extraction is performed to generate visual texture complexity features, tactile pressure distribution features, and lidar point cloud density features. These features represent the visual, tactile, and spatial information of the environment, respectively. Next, the environmental complexity of each modal feature is evaluated through a dynamic weight allocation module to generate environmental complexity assessment parameters for each modal feature. These parameters reflect the importance and contribution of each modal feature in the current environment. Subsequently, based on the environmental complexity assessment parameters, the multimodal features are weighted and aggregated through an entropy fusion module to generate an environmental entropy value that represents the degree of environmental disorder. The above process ensures that the environmental entropy value can dynamically reflect the complexity and disorder of the environment, providing an accurate quantitative basis for subsequent environmental understanding and decision-making.

[0092] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0093] In one embodiment, if Figure 2 As shown, the present application also provides a multimodal perception and interaction device 200 in an embodied environment, the device 200 comprising:

[0094] Dynamic entropy fusion-knowledge matching module 201 is used to estimate environmental entropy based on environmental data collected by multimodal sensors using a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the degree of disorder in the environment; based on the environmental entropy value, it performs entropy-knowledge matching on a pre-built meta-knowledge base to generate cross-scenario modal association rules;

[0095] A cross-modal adversarial compensation module 202 is configured to generate virtual features for missing or noisy modal data using a cross-modal generative adversarial network based on cross-scenario modal association rules, thereby generating supplementary modal features that are consistent with the real environment distribution.

[0096] The spatiotemporal self-attention alignment module 203 is used to perform cross-modal alignment processing on the supplementary modal features and the real sensor data through the spatiotemporal self-attention mechanism to generate a fused environment understanding map;

[0097] The task adaptive reinforcement decision module 204 is used to perform dynamic decision processing based on the fusion environment understanding map and task objectives through the task adaptive reinforcement learning model to generate interaction instructions; based on the interaction instructions, it drives the actuator to perform the interaction action and obtain the execution result of the interaction action;

[0098] The incremental meta-learning-knowledge base optimization module 205 is used to dynamically adjust the weight of the environmental entropy value based on the execution results of the interactive action through the incremental meta-learning framework to generate an updated environmental entropy weight; based on the updated environmental entropy weight, the local knowledge nodes of the meta-knowledge base are incrementally updated to generate an optimized meta-knowledge base.

[0099] Specifically, the dynamic entropy fusion-knowledge matching module 201 uses multimodal sensors to collect environmental data, estimates environmental entropy through a dynamic weighted multimodal feature fusion algorithm, quantifies the degree of environmental disorder, and uses the environmental entropy value to match the pre-built meta-knowledge base to generate cross-scene modal association rules. Based on the above rules, the cross-modal adversarial compensation module 202 generates supplementary modal features for missing or noisy modal data through a cross-modal generative adversarial network to ensure the integrity and consistency of the data. The spatiotemporal self-attention alignment module 203 aligns the supplementary modal features with the real sensor data to generate a fused environmental understanding map, providing more comprehensive environmental information for subsequent decision-making. The task adaptive reinforcement decision module 204 dynamically generates interactive instructions based on the fused environmental understanding map and task objectives through a task adaptive reinforcement learning model, drives the actuator to perform interactive actions, and obtains the execution results. Afterwards, the incremental meta-learning-knowledge base optimization module 205 dynamically adjusts the environment entropy weight based on the interaction results through the incremental meta-learning framework, and locally updates the meta-knowledge base to optimize performance, forming a closed-loop optimization and improving the autonomous operation capability of the intelligent agent in complex environments.

[0100] The incremental meta-learning-knowledge base optimization module 205 is further used to:

[0101] Based on the execution results of interactive actions, the task success rate and action stability indicators are extracted through the task performance quantification module;

[0102] Based on the task success rate and action stability indicators, the weight correction gradient of the environmental entropy value is calculated through the entropy weight dynamic regulator;

[0103] Based on the weight correction gradient, the weight of the environmental entropy value is incrementally iteratively optimized through a lightweight meta-learning framework to generate an updated environmental entropy weight.

[0104] The incremental meta-learning-knowledge base optimization module 205 is further used to:

[0105] Based on the task success rate and action stability indicators, dynamic correlation parameters between task relevance and environmental entropy are generated through task performance quantification parameter mapping processing;

[0106] Based on the dynamic correlation parameters, the gradient direction calibration module performs partial derivative calculation on the weight of the environmental entropy value to generate the initial gradient vector;

[0107] Based on the initial gradient vector, normalization and threshold truncation are performed through the gradient amplitude constraint module to generate the weight correction gradient.

[0108] The incremental meta-learning-knowledge base optimization module 205 is further used to:

[0109] Based on the updated environmental entropy weight, the knowledge graph topology relationship analysis module is used to filter local knowledge nodes related to the current task scenario;

[0110] Based on local knowledge nodes, the update amplitude of knowledge nodes is constrained through the meta-learning framework to generate incremental update parameters;

[0111] Based on the incremental update parameters, the updated meta-knowledge base is generalized and verified through the knowledge node consistency verification module to generate an optimized meta-knowledge base.

[0112] The task adaptive reinforcement decision module 204 is further configured to:

[0113] Based on the task objectives, the task parsing module generates strategy constraint parameters and defines the feasible domain and priority rules of the interactive actions.

[0114] Based on the fusion environment understanding of the object material characteristics and spatial topological relationships in the atlas, a dynamic policy network is used to generate a reward evaluation matrix for candidate interaction actions;

[0115] Based on the policy constraint parameters and the reward evaluation matrix, the action decision engine selects the interaction instructions that maximize the task benefits.

[0116] The task adaptive reinforcement decision module 204 is further configured to:

[0117] Based on the material characteristics of the object and the spatial topological relationship, the material-action compatibility parameters and spatial accessibility constraint parameters are extracted through the physical interaction rule library;

[0118] Based on the compatibility parameters and reachability constraint parameters, a multi-objective optimization weight adaptation module is used to generate a balance weight between task safety and efficiency.

[0119] Based on the balance weights, the physical feasibility and task benefits of candidate interaction actions are jointly evaluated through a dynamic policy network to generate a multi-dimensional reward evaluation matrix.

[0120] The dynamic entropy fusion-knowledge matching module 201 is further used for:

[0121] Based on the visual, tactile and lidar data collected by multimodal sensors, multimodal feature extraction and processing are performed to generate visual texture complexity features, tactile pressure distribution features and lidar point cloud density features;

[0122] Based on the visual texture complexity features, tactile pressure distribution features and lidar point cloud density features, the dynamic weight allocation module generates the environmental complexity evaluation parameters of each modal feature;

[0123] Based on the environmental complexity evaluation parameters, the multimodal features are weighted and aggregated through the entropy fusion module to generate the environmental entropy value that represents the environmental disorder.

[0124] In one embodiment, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0125] In one embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned method embodiments when the computer program is executed by a processor.

[0126] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0127] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A multimodal perception and interaction method in an embodied environment, characterized in that: The method comprises: Based on the environmental data collected by multimodal sensors, the environmental entropy is estimated and processed using a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the degree of disorder in the environment; based on the environmental entropy value, an entropy-knowledge matching process is performed on the pre-built meta-knowledge base to generate cross-scenario modal association rules; Based on the cross-scene modal association rules, a cross-modal generative adversarial network is used to generate virtual features for the missing or noisy modal data to generate supplementary modal features that are consistent with the real environment distribution; The supplementary modal features and real sensor data are cross-modally aligned using a spatiotemporal self-attention mechanism to generate a fused environment understanding graph. Based on the fusion environment understanding map and the task objectives, dynamic decision processing is performed through a task-adaptive reinforcement learning model to generate interaction instructions; based on the interaction instructions, the actuator is driven to perform the interaction action to obtain the execution result of the interaction action; Based on the execution result of the interactive action, the weight of the environmental entropy value is dynamically adjusted through an incremental meta-learning framework to generate an updated environmental entropy weight; based on the updated environmental entropy weight, the local knowledge nodes of the meta-knowledge base are incrementally updated to generate an optimized meta-knowledge base.

2. The multimodal perception and interaction method in an embodied environment according to claim 1, characterized in that: The step of dynamically adjusting the weight of the environment entropy value based on the execution result of the interactive action through an incremental meta-learning framework to generate an updated environment entropy weight includes: Based on the execution results of the interactive actions, the task success rate and action stability indicators are extracted through the task performance quantification module; Based on the task success rate and action stability index, a weight correction gradient of the environment entropy value is calculated by an entropy weight dynamic regulator; Based on the weight correction gradient, the weight of the environmental entropy value is incrementally iteratively optimized through a lightweight meta-learning framework to generate an updated environmental entropy weight.

3. The multimodal perception and interaction method in an embodied environment according to claim 2, characterized in that: The weight correction gradient of the environment entropy value is calculated by the entropy weight dynamic regulator based on the task success rate and the action stability index, including: Based on the task success rate and action stability indicators, dynamic correlation parameters of task relevance and environmental entropy are generated through task performance quantification parameter mapping processing; Based on the dynamic correlation parameters, a gradient direction calibration module is used to perform partial derivative calculation processing on the weight of the environmental entropy value to generate an initial gradient vector; Based on the initial gradient vector, normalization and threshold truncation processing are performed through a gradient amplitude constraint module to generate the weight correction gradient.

4. The multimodal perception and interaction method in an embodied environment according to claim 1, characterized in that: The step of performing incremental updating of local knowledge nodes on the meta-knowledge base based on the updated environmental entropy weight to generate an optimized meta-knowledge base includes: Based on the updated environmental entropy weight, the local knowledge nodes related to the current task scenario are screened through the knowledge graph topology relationship analysis module; Based on the local knowledge nodes, constraining the update amplitude of the knowledge nodes through a meta-learning framework to generate incremental update parameters; Based on the incremental update parameters, the updated meta-knowledge base is subjected to generalization performance verification processing by a knowledge node consistency verification module to generate the optimized meta-knowledge base.

5. The multimodal perception and interaction method in an embodied environment according to claim 1, characterized in that: The method of performing dynamic decision-making based on the fusion environment understanding map and the task objectives through the task adaptive reinforcement learning model to generate interactive instructions includes: Based on the task objectives, the task parsing module generates strategy constraint parameters and defines the feasible domain and priority rules of the interactive actions; Based on the material characteristics and spatial topological relationships of objects in the fusion environment understanding graph, a reward evaluation matrix for candidate interaction actions is generated through a dynamic policy network; Based on the policy constraint parameters and the reward evaluation matrix, the interaction instruction that maximizes the task benefit is selected through an action decision engine.

6. The multimodal perception and interaction method in an embodied environment according to claim 5, characterized in that: The method of generating a reward evaluation matrix for candidate interactive actions through a dynamic policy network based on the object material features and spatial topological relationships in the fusion environment understanding graph includes: Based on the material characteristics of the object and the spatial topological relationship, material-action compatibility parameters and spatial accessibility constraint parameters are extracted through a physical interaction rule library; Based on the compatibility parameter and the reachability constraint parameter, a balance weight between task safety and efficiency is generated through a multi-objective optimization weight adaptation module; Based on the balance weights, the physical feasibility and task benefits of the candidate interactive actions are jointly evaluated through a dynamic strategy network to generate the multi-dimensional reward evaluation matrix.

7. The multimodal perception and interaction method in an embodied environment according to claim 1, characterized in that: The environmental data collected by the multimodal sensor is subjected to environmental entropy estimation processing by a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value representing the degree of environmental disorder, including: Based on the visual, tactile and lidar data collected by the multimodal sensor, multimodal feature extraction processing is performed to generate visual texture complexity features, tactile pressure distribution features and lidar point cloud density features; Based on the visual texture complexity feature, the tactile pressure distribution feature and the lidar point cloud density feature, a dynamic weight allocation module is used to generate an environmental complexity assessment parameter for each modal feature; Based on the environmental complexity evaluation parameters, the multimodal features are weightedly aggregated by an entropy fusion module to generate the environmental entropy value representing the degree of environmental disorder.

8. A multimodal perception and interaction device in an embodied environment, characterized in that: The device comprises: The dynamic entropy fusion-knowledge matching module is used to estimate the environmental entropy based on the environmental data collected by multimodal sensors through a dynamic weighted multimodal feature fusion algorithm to generate an environmental entropy value that represents the degree of disorder in the environment. Based on the environmental entropy value, the module performs entropy-knowledge matching on the pre-built meta-knowledge base to generate cross-scenario modal association rules. A cross-modal adversarial compensation module is used to generate virtual features for the missing or noisy modal data through a cross-modal generative adversarial network based on the cross-scene modal association rules, and generate supplementary modal features that are consistent with the real environment distribution; A spatiotemporal self-attention alignment module is used to perform cross-modal alignment processing on the supplementary modal features and the real sensor data through the spatiotemporal self-attention mechanism to generate a fused environment understanding map; A task adaptive reinforcement decision module is used to perform dynamic decision processing based on the fusion environment understanding map and task objectives through a task adaptive reinforcement learning model to generate interaction instructions; based on the interaction instructions, drive the actuator to perform the interaction action and obtain the execution result of the interaction action; The incremental meta-learning-knowledge base optimization module is used to dynamically adjust the weight of the environmental entropy value based on the execution result of the interactive action through the incremental meta-learning framework to generate an updated environmental entropy weight; based on the updated environmental entropy weight, the local knowledge nodes of the meta-knowledge base are incrementally updated to generate an optimized meta-knowledge base.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the processor implements the steps of a multimodal perception and interaction method in an embodied environment according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a multimodal perception and interaction method in an embodied environment according to any one of claims 1 to 7 are implemented.

Images