Information interaction method and system of intelligent robot with body

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an information interaction method and system for an intelligent robot with a body, and the method comprises the steps: achieving the precise time alignment of multi-source sensor data through a hardware clock synchronization protocol, carrying out the cross-modal feature fusion through a self-adaptive weight distribution mechanism, and constructing a unified environment state expression; analyzing the user instruction in combination with the language model and the dynamic knowledge graph, and generating an execution intention containing environmental adaptation parameters; and generating an action strategy considering a task target and environmental adaptability based on a multi-target optimization algorithm, and converting the action strategy into a control instruction through a hierarchical execution mechanism. The problems that in the prior art, multi-source sensing data synchronization precision is insufficient, and heterogeneous data fusion efficiency is low are effectively solved, and the interaction accuracy, the response speed and the self-adaptive capacity of the robot in the complex dynamic environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an information interaction method and system for an embodied intelligent robot. Background Technology

[0002] In current intelligent robotics technology, information interaction methods for embodied intelligent robots are a key research area. This technology utilizes the principle of embodied cognition to directly link behavior and perception in the physical environment, enabling robots to acquire environmental information through their own sensors and understand and execute user commands in conjunction with their own context. However, in practical applications, how to accurately interpret and follow human commands in different or special scenarios still faces many challenges.

[0003] Existing technology, such as CN119260754A, discloses an information interaction method for embodied intelligent robots. This method includes: receiving information containing task instructions from a user; parsing the information based on a contextual understanding module to identify the specific meaning of the task instructions and environmental conditions; executing operations corresponding to the specific meaning and adjusting execution parameters in a specific environment to ensure accurate implementation of the task instructions. This technology enhances environmental perception and adaptability, integrating multiple sensory information such as vision, hearing, and touch to achieve comprehensive environmental perception, improves the accuracy of instruction understanding and execution, and enables the robot to more accurately parse complex user instructions, achieving real-time multimodal data fusion and decision optimization.

[0004] However, the aforementioned information interaction methods for embodied intelligent robots still have significant shortcomings. These methods employ a simple exponentially weighted moving average filter to process sensor data, lacking an effective cross-modal timestamp alignment mechanism, resulting in insufficient synchronization accuracy of multi-source sensing data. Furthermore, the lack of a unified mapping relationship between the sensor data in the feature space leads to low heterogeneous data fusion efficiency, affecting the system's interaction performance and response speed in complex environments. These issues limit the robot's practical application effectiveness in highly dynamic scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide an information interaction method and system for embodied intelligent robots, which solves the problems of insufficient synchronization accuracy of multi-source sensing data and low efficiency of heterogeneous data fusion in existing information interaction methods for embodied intelligent robots.

[0006] To achieve the above objectives, the present invention provides an information interaction method for an embodied intelligent robot, comprising the following steps:

[0007] Receive sensing data from multiple sources and perform time alignment processing on the sensing data;

[0008] Feature extraction and fusion are performed on time-aligned multimodal sensing data to generate a unified representation of the environment state;

[0009] Parse the user's input interaction commands, combine them with the environmental state representation for semantic understanding, and generate an execution intent that includes environmental adaptation parameters;

[0010] Based on the execution intent and the current environmental state, generate action strategies that take into account both task objectives and environmental adaptability;

[0011] Action strategies are translated into control commands to drive robots to perform interactive tasks.

[0012] The process includes receiving sensing data from multiple sensors and performing time alignment on the sensing data. Specific steps include:

[0013] The data from each sensor is timestamped using a hardware clock synchronization protocol;

[0014] Time-series alignment of multi-source sensing data based on timestamps;

[0015] Perform time window matching processing on the time-aligned data.

[0016] The process involves extracting and fusing features from time-aligned multimodal sensing data to generate a unified representation of the environment state. Specific steps include:

[0017] Feature representations of the visual, auditory, and tactile modalities are extracted separately;

[0018] Establish spatial mapping relationships for cross-modal features;

[0019] An adaptive weight allocation mechanism is used for feature fusion.

[0020] The process involves parsing user-inputted interaction commands, combining them with environmental state representations for semantic understanding, and generating execution intents that include environmental adaptation parameters. Specific steps include:

[0021] Extract the literal semantics of instructions using a language model;

[0022] Semantic expansion is performed by combining historical interaction records and contextual information;

[0023] The semantic understanding results are adjusted based on environmental state parameters.

[0024] The process involves generating an action strategy that balances task objectives and environmental adaptability based on the execution intent and the current environmental state. Specific steps include:

[0025] Construct an optimization objective function that includes environmental adaptability and task completion rate;

[0026] Short-term action sequences are generated using model prediction methods;

[0027] Optimize long-term decision-making strategies by combining reinforcement learning methods.

[0028] Among them, based on the execution intention and the current environmental state, an action strategy that takes into account both task objectives and environmental adaptability is generated.

[0029] During the execution of interactive tasks, the system monitors changes in environmental status in real time, dynamically adjusts action strategies based on these changes, records task execution results, and updates environmental adaptation parameters.

[0030] The specific steps involved in translating action strategies into control commands to drive the robot to perform interactive tasks include:

[0031] Decompose high-level strategies into basic, executable action units;

[0032] Perform a feasibility check on the robot's current state;

[0033] Optimize the execution timing and resource allocation of control commands.

[0034] An information interaction system for an embodied intelligent robot includes an alignment module, a fusion module, a parsing module, an action module, and a control module. The fusion module is connected to the alignment module, the parsing module is connected to the fusion module, the action module is connected to the parsing module, and the control module is connected to the action module.

[0035] The alignment module is used to receive sensing data from multiple source sensors and perform time alignment processing on the sensing data;

[0036] The fusion module is used to extract and fuse features from time-aligned multimodal sensing data to generate a unified representation of the environment state.

[0037] The parsing module is used to parse the interactive commands input by the user, combine them with the environmental state representation for semantic understanding, and generate the execution intent containing environmental adaptation parameters.

[0038] The action module is used to generate action strategies that take into account both task objectives and environmental adaptability, based on the execution intent and the current environmental state.

[0039] The control module is used to translate action strategies into control commands, driving the robot to perform interactive tasks.

[0040] This invention discloses an information interaction method and system for an embodied intelligent robot. It achieves precise time alignment of multi-source sensor data through a hardware clock synchronization protocol, employs an adaptive weight allocation mechanism for cross-modal feature fusion, and constructs a unified environmental state representation. It combines a language model and dynamic knowledge graph to parse user commands, generating execution intentions that include environmental adaptation parameters. Based on a multi-objective optimization algorithm, it generates action strategies that balance task objectives and environmental adaptability, and transforms these into control commands through a hierarchical execution mechanism. This effectively solves the problems of insufficient synchronization accuracy of multi-source sensor data and low efficiency of heterogeneous data fusion in existing technologies, significantly improving the robot's interaction accuracy, response speed, and adaptability in complex dynamic environments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0042] Figure 1 This is a flowchart of the information interaction method of the embodied intelligent robot according to the first embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the information interaction system of the embodied intelligent robot according to the second embodiment of the present invention.

[0044] In the diagram: 201 - Alignment module, 202 - Fusion module, 203 - Parsing module, 204 - Action module, 205 - Control module. Detailed Implementation

[0045] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0046] The first embodiment of this application is as follows:

[0047] Please see Figure 1 ,in, Figure 1 This is a flowchart of the information interaction method of the embodied intelligent robot according to the first embodiment of the present invention.

[0048] This invention provides an information interaction method for an embodied intelligent robot, comprising the following steps:

[0049] S101: Receive sensing data from multiple source sensors and perform time alignment processing on the sensing data;

[0050] Specifically, the first step is to configure and coordinate the operating states of various sensors. Vision sensors acquire environmental images and depth information, audio sensors acquire voice signals, tactile sensors record contact force data, and motion sensors detect pose changes. These sensors have different physical characteristics and operating modes, including varying sampling frequencies, data formats, and transmission delays. To coordinate these differences, this invention employs a hierarchical synchronization strategy, constructing a complete timing alignment system from the hardware layer to the application layer. At the hardware level, a unified time reference is established for each sensor through a clock synchronization circuit. This circuit uses a master-slave architecture, where the master clock node periodically broadcasts a synchronization signal, and the slave nodes eliminate errors caused by transmission delays through time compensation algorithms. Each sensor's data packet is marked with a precise timestamp, generated based on the same clock reference, providing a reliable timing reference for subsequent processing. At the data processing level, corresponding alignment algorithms are designed for the sampling characteristics of different sensors. For high-frequency sampling sensors (such as tactile sensors), a combination of downsampling and interpolation is used; for low-frequency sampling sensors (such as vision sensors), motion compensation and prediction algorithms are used to fill time gaps. Specifically, for continuous signals such as speech, a dynamic alignment method based on acoustic features can effectively handle the time-varying characteristics of speech signals. Time window management is another crucial step in achieving accurate alignment. The system maintains a dynamically adjustable time window, automatically optimizing its size based on the current environmental complexity and computational load. Within this window, various sensor data are aligned using a timestamp matching algorithm, while an anomaly detection mechanism identifies and handles timestamp anomalies to ensure the reliability of the alignment results. To ensure processing efficiency, this invention designs a hierarchical caching architecture. Raw sensor data first enters a high-speed cache for preliminary time stamping, then is transferred to the alignment processing queue, ultimately generating a time-aligned multimodal data stream. The entire process employs a pipelined design, with each processing stage operating in parallel, ensuring both processing timeliness and data consistency. This time alignment processing method has the following advantages: it can adapt to the differences in temporal characteristics of different types of sensors; it can effectively handle random delays caused by network transmission; it supports real-time adjustments in dynamic environments; and it provides a precise time reference for subsequent feature extraction and fusion. These technical means have completely solved the problem of data fusion distortion caused by time asynchrony in traditional methods, laying a solid foundation for building a reliable robot environmental perception system.

[0051] S102: Extract and fuse features from time-aligned multimodal sensing data to generate a unified representation of the environment state;

[0052] Specifically, the data from each modality is first standardized and preprocessed. Visual data is processed using a convolutional neural network to extract spatial features, including basic visual elements such as edges, textures, and object contours. Auditory data undergoes Fourier transform to calculate Mel-frequency cepstral coefficients, obtaining the spectral characteristics of sound. Tactile data is processed using statistical methods to extract physical characteristics such as pressure distribution and vibration frequency. In the feature fusion stage, a feature-level fusion strategy is adopted. The feature vectors of different modalities are normalized and then weighted and concatenated to form a joint feature representation, where the weight coefficients are dynamically adjusted according to the signal-to-noise ratio of each modality. For spatiotemporal correlation features, canonical correlation analysis (CCA) is applied to establish cross-modal feature mapping relationships and discover potential correlations between visual scenes and sound events. The final environmental state representation is modeled using a probabilistic graphical model, where nodes represent environmental elements and edges represent spatial and semantic relationships between elements. The node state is continuously optimized through a message passing algorithm. This processing flow transforms heterogeneous multimodal perception data into a unified, structured environmental state representation with clear physical meaning, providing a reliable environmental cognitive foundation for subsequent decision-making.

[0053] S103: Parse the user's input interaction commands, combine them with the environmental state representation to perform semantic understanding, and generate an execution intent that includes environmental adaptation parameters;

[0054] Specifically, the process begins with basic language processing of the original speech or text commands, including standard natural language processing steps such as word segmentation, part-of-speech tagging, and syntactic analysis, extracting semantic elements such as action predicates, target objects, and spatial relationships. Addressing referential and ellipsis issues in the commands, semantic completion is achieved through a referential resolution algorithm combined with dialogue history, a process that draws upon the theory of contextual understanding in linguistics. In the environment state fusion stage, the parsed semantic elements are matched with entities in the environment model, establishing a correspondence between commands and the environment based on spatial relationships and object attributes; for example, mapping "the red box on the left" to a specific object instance. The execution intent generation stage comprehensively considers environmental constraints and task requirements, balancing operational efficiency, safety, and energy consumption through a multi-criteria decision-making method, ultimately outputting an executable solution that includes adaptive parameters such as force adjustment, movement speed, and interaction mode. The entire processing flow adopts a modular design; the language models, referential resolution algorithms, and multi-criteria decision-making methods used in each stage are standard technologies in the fields of natural language processing and artificial intelligence, and their theoretical foundations and implementation details are discussed in detail in relevant academic literature. This environment-aware semantic understanding method can transform abstract user instructions into specific operational plans that take into account actual environmental constraints, ensuring both the smoothness of natural interaction and the effectiveness of task execution.

[0055] S104: Based on the execution intent and the current environmental state, generate an action strategy that takes into account both task objectives and environmental adaptability;

[0056] Specifically, a multi-objective optimization function is first constructed, which includes two core evaluation dimensions: a task completion index that quantitatively evaluates the contribution of the action sequence to the achievement of the objective, and an environmental adaptability index that measures the degree of matching between the strategy and environmental constraints. At the short-term decision-making level, a model predictive control method is adopted to continuously generate the optimal action sequence within a finite future time domain based on the current environmental state (such as object positions, obstacle distribution, etc.). This method can consider the robot's kinematic and dynamic constraints in real time. At the long-term policy optimization level, a reinforcement learning framework is combined to evaluate the long-term benefits of different strategies through a value function, and the policy gradient method is used to continuously improve the decision model. During policy execution, the system continuously monitors changes in the environmental state (such as the addition of obstacles, changes in lighting conditions, etc.). When a significant change is detected, a dynamic adjustment mechanism is triggered: for local disturbances, online replanning through model predictive control provides a rapid response; for global environmental changes, a reinforcement learning strategy is invoked to generate a new action plan. After each task execution, the system records the actual results (such as completion time, energy consumption data, etc.) and updates the weight coefficients of the environmental adaptability parameters accordingly, forming a continuously optimizing closed-loop learning system. The entire policy generation process integrates mature methods from control theory and machine learning. By organically combining short-term precise control with long-term policy optimization, it ensures efficient operation in stable environments while adapting to the challenges brought by dynamic changes, enabling the robot to maintain excellent interactive performance in complex scenarios. Multi-objective optimization ensures a balance between task execution and environmental adaptation, model predictive control guarantees real-time performance, reinforcement learning achieves long-term optimization, and environmental monitoring and parameter update mechanisms further contribute to the formation of an adaptive intelligent decision-making system.

[0057] S105: Transforms action strategies into control commands to drive the robot to perform interactive tasks.

[0058] Specifically, the process begins with kinematic decomposition of the high-level action strategy. Inverse kinematics calculations transform the abstract action description into concrete trajectory points in joint space, adhering to fundamental principles of robot kinematics. During trajectory planning, a polynomial interpolation algorithm generates smooth joint space motion curves, while considering the robot arm's acceleration and torque constraints to ensure the motion conforms to dynamic constraints. For fine-grained operation of the end effector, a sensor-feedback-based force / position hybrid control strategy is introduced, using an impedance control algorithm to adjust the interaction forces between the robot arm and the environment. For navigation tasks on the mobile platform, a sampling-based local path planning method generates obstacle avoidance trajectories in real time, while combining odometry and visual data for pose correction. In the command distribution phase, a real-time scheduling algorithm coordinates the timing of actions across multiple actuators, ensuring collaborative operation of all subsystems. The entire transformation process integrates standard methods from the field of robot control, including kinematics solving, trajectory optimization, force control, and multi-axis coordination techniques, which have been successfully implemented in industrial and service robot applications. This standardized instruction conversion process enables the high-level strategies generated by the intelligent decision-making system to be accurately translated into control commands that can be understood by the underlying actuators, achieving a complete closed loop from perception and cognition to physical action, and ensuring that the robot can reliably and accurately complete various interactive tasks.

[0059] It effectively solves the problems of insufficient synchronization accuracy of multi-source sensing data and low efficiency of heterogeneous data fusion in existing technologies, and significantly improves the robot's interaction accuracy, response speed and adaptive ability in complex dynamic environments.

[0060] The second embodiment of this application is as follows:

[0061] Based on the first embodiment, please refer to Figure 2 ,in, Figure 2 This is a schematic diagram of the information interaction system of the embodied intelligent robot according to the second embodiment of the present invention.

[0062] This embodiment of an information interaction system for an embodied intelligent robot includes an alignment module 201, a fusion module 202, a parsing module 203, an action module 204, and a control module 205.

[0063] In this specific embodiment, the fusion module 202 is connected to the alignment module 201, the parsing module 203 is connected to the fusion module 202, the action module 204 is connected to the parsing module 203, and the control module 205 is connected to the action module 204.

[0064] Alignment module 201 is used to receive sensing data from multi-source sensors and perform time alignment processing on the sensing data;

[0065] The fusion module 202 is used to extract and fuse features from time-aligned multimodal sensing data to generate a unified environmental state representation;

[0066] The parsing module 203 is used to parse the interactive commands input by the user, combine them with the environmental state representation to perform semantic understanding, and generate an execution intent that includes environmental adaptation parameters;

[0067] Action module 204 is used to generate action strategies that take into account both task objectives and environmental adaptability based on the execution intent and the current environmental state.

[0068] The control module 205 is used to convert action strategies into control commands to drive the robot to perform interactive tasks.

[0069] Using the information interaction system of an embodied intelligent robot in this embodiment, the alignment module 201 is responsible for the time synchronization processing of multi-source sensor data; the fusion module 202 completes multimodal feature extraction and environmental state modeling; the parsing module 203 realizes semantic understanding and environmental adaptation of natural language commands; the action module 204 generates task strategies that consider environmental constraints; and the control module 205 finally converts the strategies into executable commands. Each module uses a standardized interface protocol for data transmission, forming a coherent information processing chain. Through a layered and progressive processing architecture, key problems such as insufficient multimodal data fusion and poor environmental adaptability are effectively solved, enabling the robot to accurately understand user intentions and generate execution plans adapted to environmental conditions, significantly improving the reliability of interaction and task completion efficiency in complex scenarios. The system as a whole achieves efficient integration of perceptual data, accurate parsing of command intentions, and reasonable generation of action strategies, providing a complete technical implementation solution for natural human-computer interaction.

[0070] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. An information interaction method for an embodied intelligent robot, characterized in that, Includes the following steps: Receive sensing data from multiple sources and perform time alignment processing on the sensing data; Feature extraction and fusion are performed on time-aligned multimodal sensing data to generate a unified representation of the environment state; Parse the user's input interaction commands, combine them with the environmental state representation for semantic understanding, and generate an execution intent that includes environmental adaptation parameters; Based on the execution intent and the current environmental state, generate action strategies that take into account both task objectives and environmental adaptability; Action strategies are translated into control commands to drive robots to perform interactive tasks.

2. The information interaction method for an embodied intelligent robot as described in claim 1, characterized in that, The process involves receiving sensing data from multiple sensors and performing time alignment on the sensing data. Specific steps include: The data from each sensor is timestamped using a hardware clock synchronization protocol; Time-series alignment of multi-source sensing data based on timestamps; Perform time window matching processing on the time-aligned data.

3. The information interaction method for an embodied intelligent robot as described in claim 2, characterized in that, The time-aligned multimodal sensing data is then subjected to feature extraction and fusion to generate a unified representation of the environment state. Specific steps include: Feature representations of the visual, auditory, and tactile modalities are extracted separately; Establish spatial mapping relationships for cross-modal features; An adaptive weight allocation mechanism is used for feature fusion.

4. The information interaction method for an embodied intelligent robot as described in claim 3, characterized in that, Parse the user's input interaction commands, combine them with the environmental state representation for semantic understanding, and generate an execution intent that includes environmental adaptation parameters. Specific steps include: Extract the literal semantics of instructions using a language model; Semantic expansion is performed by combining historical interaction records and contextual information; The semantic understanding results are adjusted based on environmental state parameters.

5. The information interaction method for an embodied intelligent robot as described in claim 4, characterized in that, Based on the execution intent and the current environmental state, an action strategy that balances task objectives and environmental adaptability is generated. Specific steps include: Construct an optimization objective function that includes environmental adaptability and task completion rate; Short-term action sequences are generated using model prediction methods; Optimize long-term decision-making strategies by combining reinforcement learning methods.

6. The information interaction method for an embodied intelligent robot as described in claim 5, characterized in that, Based on the execution intent and the current environmental state, an action strategy is generated that balances task objectives and environmental adaptability. During the execution of interactive tasks, the system monitors changes in environmental status in real time, dynamically adjusts action strategies based on these changes, records task execution results, and updates environmental adaptation parameters.

7. The information interaction method for an embodied intelligent robot as described in claim 6, characterized in that, The action strategy is translated into control commands to drive the robot to perform interactive tasks. The specific steps include: Decompose high-level strategies into basic, executable action units; Perform a feasibility check on the robot's current state; Optimize the execution timing and resource allocation of control commands.

8. An information interaction system for an embodied intelligent robot, applicable to the information interaction method for an embodied intelligent robot as described in claim 1, characterized in that, It includes an alignment module, a fusion module, a parsing module, an action module, and a control module. The fusion module is connected to the alignment module, the parsing module is connected to the fusion module, the action module is connected to the parsing module, and the control module is connected to the action module. The alignment module is used to receive sensing data from multiple source sensors and perform time alignment processing on the sensing data; The fusion module is used to extract and fuse features from time-aligned multimodal sensing data to generate a unified representation of the environment state. The parsing module is used to parse the interactive commands input by the user, combine them with the environmental state representation for semantic understanding, and generate the execution intent containing environmental adaptation parameters; The action module is used to generate action strategies that take into account both task objectives and environmental adaptability, based on the execution intent and the current environmental state. The control module is used to translate action strategies into control commands, driving the robot to perform interactive tasks.

Images