Multi-modal human-computer interaction and instruction accurate execution method for robot dog

Abstract

The application discloses a kind of machine dog multimodal man-machine interaction and instruction accurate execution method, comprising the following steps: build multimodal interaction data acquisition module, with 20Hz frequency Synchronous acquisition multi-class interaction data and auxiliary data;The multimodal data collected is preprocessed in view of;Improved CNN+LSTM multimodal instruction recognition model is constructed, and input multimodal fusion feature matrix is accurately recognized for instruction;Instruction semantics is analyzed using NLP technology, distinguish instruction type and extract key parameters, and construct instruction priority evaluation system;Combined with instruction analysis result, machine dog state and scene environment, action manifold learning is used to plan execution path and action sequence, and real-time execution control is realized by PID control algorithm;Multiple scene adaptation mode is preset, automatically identify job scene and dynamically adjust parameters.The application can realize multiple natural interaction, improve instruction recognition accuracy and execution precision, enhance scene adaptation capability, and improve man-machine collaborative work efficiency.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology for legged robots, and more specifically to a method for multimodal human-computer interaction and precise command execution for a robot dog. Background Technology

[0002] With the continuous evolution of legged robot technology, robot dogs have shown broad application prospects in many fields such as inspection and security, special detection, and service companionship, thanks to their excellent adaptability to unstructured terrain and flexible movement characteristics. Their human-computer interaction performance is the core indicator that determines operational efficiency and scene adaptability. Current human-computer interaction technology for robotic dogs still has significant shortcomings. Interaction methods are generally limited, relying heavily on traditional modes such as preset button commands and fixed speech commands. They lack multimodal natural interaction capabilities and flexible command adaptation mechanisms, making it difficult to meet the diverse interaction needs in complex scenarios. Affected by environmental noise interference, insufficient semantic understanding, and limited environmental perception accuracy, robotic dogs have low accuracy in recognizing user commands, especially for non-standardized commands, ambiguous expressions, and real-time commands in dynamic scenarios, which are prone to recognition errors and misjudgments. Simultaneously, the coordination and matching between the command parsing module and the motion execution module are insufficient, and the response accuracy and robustness of the control link need improvement. This leads to significant deviations between the actual actions of the robotic dog and the expected goals, making it difficult to guarantee execution stability in complex terrain and dynamic task scenarios. These technical deficiencies collectively result in a poor command interaction link and high feedback correction costs during human-computer collaboration, leading to low overall human-computer collaboration efficiency and severely restricting the application depth and widespread adoption of robotic dogs in complex work scenarios. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a method for multimodal human-computer interaction and precise command execution in robot dogs. Through a full-link design that integrates multimodal data acquisition, feature fusion and recognition, command scheduling, closed-loop execution, and scene adaptive optimization, the method solves the problems of existing robot dogs, such as single interaction methods, low command recognition accuracy, poor execution coordination, and insufficient scene adaptability.

[0004] To achieve the above objectives, this invention provides the following technical solution: a method for multimodal human-computer interaction and precise command execution in a robot dog, comprising the following steps: Step 1, constructing a multimodal interaction data acquisition module to synchronously acquire multiple types of interaction data and auxiliary data at a frequency of 20Hz; Step 2, performing targeted preprocessing on the multimodal data acquired in Step 1 to eliminate environmental interference and data differences; Step 3, constructing an improved CNN+LSTM multimodal command recognition model, inputting a multimodal fusion feature matrix, and performing precise command recognition; Step 4, using NLP technology to parse command semantics, distinguishing between basic control commands and task commands, and extracting key parameters; constructing a command priority evaluation system, with the following ranking rules: Emergency commands > Operational task commands > Basic control commands. The priority can be dynamically adjusted based on the robot dog's remaining battery power and attitude deviation. If multiple commands conflict, the highest priority command is automatically selected, and lower priority commands are cached. Step five: Combining the command parsing results, robot dog status, and scene environment, the optimal execution path and action sequence are planned using the action manifold learning method, and real-time execution control is achieved through the PID control algorithm. Step six: Preset adaptation modes for inspection, rescue, and security-related scenarios. The operation scenario is automatically identified through scene feature matching, and modal weights and execution parameters are dynamically adjusted. After each command execution is completed, relevant data is recorded to establish an iterative optimization dataset, and the execution parameters are dynamically corrected through reinforcement learning.

[0005] As a further improvement of the present invention, the data in step one includes voice interaction data, gesture interaction data, visual interaction data, electroencephalogram (EEG) signal data, and the robot dog's own state data.

[0006] As a further improvement of the present invention, the specific method for targeted preprocessing of the multimodal data collected in step two is as follows: Speech data is denoised using the GAN algorithm and converted into spectral features via STFT; gesture / visual data is denoised using Gaussian filtering, image segmentation, and pose normalization; EEG signal data is denoised and normalized; then, multimodal data is time-aligned using timestamps, and multimodal features are fused using a channel attention mechanism to generate a multimodal fusion feature matrix. in: This is a multimodal fusion feature matrix; The number of modes; Let be the attention weights for the i-th modality, satisfying ; For a single feature vector of the i-th modality, the attention weights are... The calculation formula is as follows: In the formula, , This is the output layer weight matrix. The hidden layer weight matrix is... It is the Sigmoid activation function. For hidden layer bias terms, This is the output layer bias term.

[0007] As a further improvement of the present invention, the improved CNN+LSTM multimodal instruction recognition model constructed in step three includes an input layer, a CNN branch for extracting spatial features, an LSTM branch for extracting temporal features, a dropout layer with a dropout rate of 0.2, an attention fusion layer, a fully connected layer, and an output layer. It employs a cross-entropy loss function combined with L1 regularization to avoid overfitting. in: This represents the total loss value of the model. The cross-entropy loss function; The regularization coefficient is used. This is an L1 regularization term.

[0008] As a further improvement of the present invention, in step five, combining the instruction parsing results, the robot dog's state, and the scene environment, the optimal execution path and action sequence are planned using the action manifold learning method. The specific method for achieving real-time execution control through the PID control algorithm is as follows: A closed-loop mechanism of "execution-feedback-correction" is established, execution status data is collected every 50ms, execution deviation is calculated and dynamically corrected, and the execution deviation is calculated as follows: Corrected joint control commands: in , , Ensure that the execution deviation is ≤3% and the total response time is <2 seconds.

[0009] The beneficial effects of this invention are as follows: By combining multimodal synchronous acquisition and attention fusion recognition, it overcomes the limitations of a single interaction mode, supporting diverse natural interaction methods such as voice, gesture, vision, and EEG, adapting to the diversified interaction needs in complex scenarios; the improved CNN+LSTM dual-branch model combined with multimodal feature fusion effectively reduces the interference of environmental noise and dynamic scenes on command recognition, improving the recognition accuracy of non-standardized and fuzzy commands; the command semantic parsing and priority scheduling mechanism combined with action manifold path planning and PID closed-loop control establishes a collaborative link between command parsing and motion execution, improving control response accuracy and robustness, reducing execution deviation, and ensuring execution stability in complex terrains; scene adaptive matching and reinforcement learning iterative optimization enable dynamic adjustment of modal weights and execution parameters under different operating scenarios, continuously optimizing execution efficiency, improving overall human-machine collaboration efficiency, and expanding the application depth of robot dogs. Multimodal differentiated preprocessing and temporal alignment further ensure the effectiveness of feature fusion, regularized loss functions enhance model generalization ability, and millisecond-level closed-loop control further improves the real-time performance and reliability of execution. Detailed Implementation

[0010] The present invention will be further described in detail below with reference to the given embodiments.

[0011] The multimodal human-computer interaction and precise command execution method for robot dogs in this embodiment includes the following steps: Step 1, building a multimodal interaction data acquisition module to synchronously acquire multiple types of interaction data and auxiliary data at a frequency of 20Hz; Step 2, performing targeted preprocessing on the multimodal data acquired in Step 1 to eliminate environmental interference and data differences; Step 3, constructing an improved CNN+LSTM multimodal command recognition model, inputting a multimodal fusion feature matrix, and performing precise command recognition; Step 4, using NLP technology to parse command semantics, distinguishing between basic control commands and task commands, and extracting key parameters; constructing a command priority evaluation system, with the ranking rule being emergency commands > task commands. The command is set to the basic control command, and its priority can be dynamically adjusted according to the robot dog's remaining battery power and attitude deviation. If multiple commands conflict, the highest priority command is automatically selected and the lower priority commands are cached. In step five, the optimal execution path and action sequence are planned using the action manifold learning method, combined with the command parsing results, robot dog status and scene environment, and real-time execution control is achieved through the PID control algorithm. In step six, the adaptation modes for inspection, rescue and security-related scenarios are preset, and the operation scenario is automatically identified through scene feature matching, and the modal weights and execution parameters are dynamically adjusted. After each command execution is completed, relevant data is recorded to establish an iterative optimization dataset, and the execution parameters are dynamically corrected through reinforcement learning.

[0012] This method achieves synchronous acquisition of multi-dimensional interactive data through a data acquisition module, breaking the limitations of traditional single-interaction methods and meeting the diverse interaction needs of complex scenarios. After preprocessing to eliminate environmental noise and data heterogeneity, it achieves accurate judgment of instructions through a fusion recognition model, reducing the misjudgment rate in dynamic environments. The semantic parsing and priority scheduling module realizes the classification management and conflict resolution of instructions, ensuring the priority of instruction response in emergency scenarios. The action planning and closed-loop control module realizes accurate mapping from instructions to motion execution, improving execution accuracy and response speed. The scenario adaptation and iterative optimization module realizes automatic adaptation to the work scenario and parameter self-optimization, continuously improving the system's adaptability and execution efficiency in different scenarios. It solves the problems of single interaction, inaccurate recognition, insufficient collaboration, and weak adaptability in existing technologies from the entire chain of interaction, recognition, scheduling, execution, and optimization, and achieves an overall improvement in human-machine collaboration efficiency.

[0013] Furthermore, the data in step one includes voice interaction data, gesture interaction data, visual interaction data, EEG signal data, and the robot dog's own state data. By covering four interaction modalities—voice, gesture, vision, and EEG—and combining them with the robot dog's own state auxiliary data, a comprehensive interactive data input system is constructed. This system supports multi-channel natural interaction with users and can also adapt commands based on the device's own state, helping to improve the rationality of command recognition and execution. It also achieves the diversification of interaction methods and the integration of state perception, further expanding the scope of adaptability to interaction scenarios.

[0014] Furthermore, the specific preprocessing methods for the multimodal data collected in Step 1 in Step 2 are as follows: Speech data is denoised using the GAN algorithm and converted into spectral features via STFT; gesture / visual data undergoes Gaussian filtering for denoising, image segmentation, and pose normalization; EEG signal data is filtered for denoising and normalized; then, multimodal data is time-aligned using timestamps, and multimodal features are fused using a channel attention mechanism to generate a multimodal fusion feature matrix. in: This is a multimodal fusion feature matrix; The number of modes; Let be the attention weights for the i-th modality, satisfying ; For a single feature vector of the i-th modality, the attention weights are... The calculation formula is as follows: In the formula, , This is the output layer weight matrix. The hidden layer weight matrix is... It is the Sigmoid activation function. For hidden layer bias terms, This is the output layer bias term.

[0015] Differentiated preprocessing methods are adopted to address the noise characteristics of different modal data, effectively filtering out environmental interference and unifying the data format. Temporal alignment ensures the temporal consistency of multimodal data. The channel attention mechanism can adaptively allocate feature weights for different modalities, helping to improve the feature extraction efficiency and recognition accuracy of the instruction recognition model. Furthermore, adaptive weighted fusion of modal features is implemented, which strengthens the feature contribution of high-information modalities and further enhances the recognition robustness in complex environments.

[0016] Furthermore, the improved CNN+LSTM multimodal instruction recognition model constructed in step three includes an input layer, a CNN branch for extracting spatial features, an LSTM branch for extracting temporal features, a dropout layer with a dropout rate of 0.2, an attention fusion layer, a fully connected layer, and an output layer. It employs a cross-entropy loss function combined with L1 regularization to avoid overfitting. in: This represents the total loss value of the model. The cross-entropy loss function; The regularization coefficient is used. This is an L1 regularization term.

[0017] By extracting spatial features through CNN branches and temporal features through LSTM branches, the dual-branch structure fully explores the spatiotemporal features of multimodal data. Dropout layers and L1 regularization jointly suppress model overfitting, helping to improve the generalization ability and recognition accuracy of the instruction recognition model. In addition, the joint extraction of spatiotemporal features is achieved, which enhances the model's ability to discriminate dynamic temporal instructions and further reduces the recognition error of non-standardized instructions.

[0018] Furthermore, in step five, combining the instruction parsing results, the robot dog's state, and the scene environment, the optimal execution path and action sequence are planned using the action manifold learning method. The specific method for achieving real-time execution control through the PID control algorithm is as follows: An "execution-feedback-correction" closed-loop mechanism is established, execution status data is collected every 50ms, execution deviation is calculated, and dynamic correction is performed. The execution deviation is calculated as follows: Corrected joint control commands: in , , Ensure that the execution deviation is ≤3% and the total response time is <2 seconds.

[0019] The motion manifold learning method can generate adaptive motion sequences for complex terrains. PID closed-loop control collects execution status at high frequency and corrects it in real time, ensuring the accuracy and response speed of motion execution. It helps to solve the problem of insufficient coordination between instruction parsing and motion execution. In addition, it realizes millisecond-level closed-loop feedback correction, which further improves the stability and response speed of instruction execution in dynamic scenarios.

[0020] In summary, the multimodal human-computer interaction and precise command execution method for robot dogs of the present invention adopts: 1. Attention-based multimodal fusion algorithm To address the issues of poor adaptability and low recognition accuracy in single-modal interactions, a channel attention mechanism is employed to deeply fuse multimodal data. This mechanism dynamically allocates the weights of each modal feature and adjusts the weight ratios in real-time according to the operational scenario. This overcomes the shortcomings of poor multi-source data fusion in complex environments, providing high-quality feature support for command recognition. The core formula and weight calculation method are described in step 2 of the invention, effectively improving the anti-interference capability and adaptability of multimodal interactions.

[0021] 2. Improved CNN+LSTM multimodal instruction recognition model Combining the advantages of CNN and LSTM, a dual-branch recognition model is constructed. The CNN branch extracts spatial features from gesture and visual data, while the LSTM branch extracts temporal features from speech and EEG signals. A dropout mechanism is introduced to avoid overfitting, and cross-entropy loss function combined with L1 regularization is used to optimize training performance. Through lightweight optimization, the model parameter size is ≤1.2MB, inference latency is ≤1ms, and command recognition accuracy is ≥95%, still reaching over 92% in complex environments, representing an improvement of over 12% compared to existing single-modal recognition algorithms.

[0022] 3. Dynamic feedback correction mechanism for instruction execution To address the issues of large command execution deviations and lack of dynamic adjustment in existing robotic dogs, a closed-loop "execution-feedback-correction" mechanism is established. Execution status data is collected at 50ms intervals, and a PID control algorithm dynamically corrects joint control commands and execution paths, keeping execution deviations within 3%. Simultaneously, response time allocation is optimized to ensure a total response time of less than 2 seconds, resolving the pain point of high response latency in emergency operation scenarios.

[0023] 4. Multi-scenario adaptive adaptation strategy It has multiple preset scene adaptation modes and automatically identifies the operation scene through scene feature matching algorithm (core formula: Sim=Xscene·Xtemplate‖Xscene‖·‖Xtemplate‖). It dynamically adjusts the multimodal interaction weight and instruction execution parameters. It can adapt to different scenarios such as inspection, rescue, and security without modifying the core algorithm. It has strong versatility and low deployment cost.

[0024] 5. Reinforcement learning-driven iterative optimization mechanism To address the issue of decreased algorithm adaptability due to dynamic changes in work scenarios, an iterative optimization mechanism based on reinforcement learning is established. This mechanism records the interaction data, recognition results, and execution deviations for each instruction execution, dynamically corrects the multimodal fusion weights, recognition model parameters, and execution control parameters, thereby continuously improving algorithm performance and making the algorithm more intelligent with use, adapting to the dynamic changes in scenario requirements.

[0025] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for multimodal human-computer interaction and precise command execution in a robot dog, characterized in that: Includes the following steps: Step 1: Build a multimodal interactive data acquisition module to synchronously acquire multiple types of interactive data and auxiliary data at a frequency of 20Hz; Step 2: Perform targeted preprocessing on the multimodal data collected in Step 1 to eliminate environmental interference and data differences; Step 3: Construct an improved CNN+LSTM multimodal instruction recognition model, input the multimodal fusion feature matrix, and perform accurate instruction recognition; Step 4: Use NLP technology to parse the semantics of the instructions, distinguish between basic control instructions and task instructions, and extract key parameters; construct an instruction priority evaluation system, with the sorting rule being emergency instructions > task instructions > basic control instructions, and the priority can be dynamically adjusted according to the robot dog's remaining battery power and attitude deviation; if there are multiple conflicting instructions, the highest priority instruction is automatically filtered out and the low priority instructions are cached. Step 5: Combining the instruction parsing results, the robot dog's status, and the scene environment, the optimal execution path and action sequence are planned using the action manifold learning method, and real-time execution control is achieved through the PID control algorithm; Step six: Preset adaptation modes for inspection, rescue, and security scenarios; automatically identify operation scenarios through scenario feature matching; dynamically adjust modal weights and execution parameters; and record relevant data to build an iterative optimization dataset after each instruction execution, and dynamically correct execution parameters through reinforcement learning.

2. The method for multimodal human-computer interaction and precise command execution of a robot dog according to claim 1, characterized in that: The data in step one includes voice interaction data, gesture interaction data, visual interaction data, electroencephalogram (EEG) signal data, and the robot dog's own state data.

3. The method for multimodal human-computer interaction and precise command execution of a robot dog according to claim 2, characterized in that: The specific preprocessing method for the multimodal data collected in step one in step two is as follows: Speech data is denoised using the GAN algorithm and converted into spectral features via STFT; gesture / visual data is denoised using Gaussian filtering, image segmentation, and pose normalization; EEG signal data is denoised and normalized; then, multimodal data is time-aligned using timestamps, and multimodal features are fused using a channel attention mechanism to generate a multimodal fusion feature matrix. in: This is a multimodal fusion feature matrix; The number of modes; Let be the attention weights for the i-th modality, satisfying ; For a single feature vector of the i-th modality, the attention weights are... The calculation formula is as follows: In the formula, , This is the output layer weight matrix. The hidden layer weight matrix is... It is the Sigmoid activation function. For hidden layer bias terms, This is the output layer bias term.

4. The method for multimodal human-computer interaction and precise command execution of a robot dog according to claim 3, characterized in that: The improved CNN+LSTM multimodal instruction recognition model constructed in step three includes an input layer, a CNN branch for extracting spatial features, an LSTM branch for extracting temporal features (dropout rate = 0.2), an attention fusion layer, a fully connected layer, and an output layer. It employs a cross-entropy loss function combined with L1 regularization to avoid overfitting. in: This represents the total loss value of the model. The cross-entropy loss function; The regularization coefficient is used. This is an L1 regularization term.

5. The method for multimodal human-computer interaction and precise command execution of a robot dog according to claim 4, characterized in that: In step five, combining the instruction parsing results, the robot dog's state, and the scene environment, the optimal execution path and action sequence are planned using the action manifold learning method. The specific method for achieving real-time execution control through the PID control algorithm is as follows: A closed-loop mechanism of "execution-feedback-correction" is established, execution status data is collected every 50ms, execution deviation is calculated, and dynamic correction is performed. The execution deviation is calculated as follows: Corrected joint control commands: ,in , , Ensure that the execution deviation is ≤3% and the total response time is <2 seconds.

Images