Intelligent voice recognition and natural language interaction method based on quadruped robot

Abstract

The invention discloses an intelligent voice recognition and natural language interaction method based on a quadruped robot, and the method comprises the following steps: S1, collecting a user voice instruction, generating a standardized voice text, and extracting a semantic keyword set; s2, collecting multi-source sensing data of the quadruped robot and generating a structured state data tensor; s3, constructing a multi-modal collaborative modeling mechanism, and generating a multi-modal joint semantic embedding vector; s4, constructing a semantic map based on semantic embedding and generating an action chain plan structure; s5, executing each sub-action in the action chain, and performing path analysis and execution monitoring; s6, storing the interaction task as a multi-modal semantic behavior memory unit; and S7, performing similarity retrieval based on current semantic input and historical memory to realize behavior migration and action chain multiplexing. The method has the advantages of accurate semantic understanding, intelligent interaction response, high behavior migration capability and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot control and navigation, and particularly relates to an intelligent speech recognition and natural language interaction method based on a quadruped robot. BACKGROUND

[0002] In the development of current quadruped robot natural language interaction systems, the limitations of speech recognition and task understanding capabilities seriously restrict their autonomous behavior generation capabilities in complex environments. Existing speech recognition methods rely heavily on supervised training models, and have weak generalization capabilities for unstructured speech instructions, making it difficult to adapt to the changes in tone, semantic ambiguity and instruction diversity in natural language, resulting in recognition errors or incomplete task intent analysis in the interaction process.

[0003] At the same time, existing robot interaction systems generally use a single modal perception strategy, lack a collaborative modeling mechanism for speech, vision, spatial state and other multi-modal information, and cannot achieve dynamic fusion between semantic information and environmental state, resulting in spatial understanding bias and behavior decision delay when executing complex instructions. Especially in dynamic environments, the lack of semantic instruction and scene state linkage modeling capability makes it difficult for robots to generate accurate action chains and context-aware responses.

[0004] In addition, traditional systems lack a semantic memory mechanism and cannot structure the storage and reuse of behavior processes in historical interaction tasks, resulting in the need to re-analyze and model each task, which is inefficient and cannot support cross-task behavior migration. For frequently repeated or structurally similar speech instructions, current systems generally lack intelligent reuse capabilities, affecting the stability and interaction efficiency of long-term autonomous operation of robots.

[0005] It can be seen that the existing technology still has obvious defects in the robustness of speech understanding, multi-modal collaborative modeling capability and task reuse mechanism, and there is an urgent need for a new interaction method with stronger semantic understanding capability, spatial perception capability and memory migration mechanism.

[0006] Therefore, how to provide an intelligent speech recognition and natural language interaction method based on a quadruped robot is a problem that those skilled in the art need to solve. SUMMARY

[0007] One purpose of the present application is to propose an intelligent voice recognition and natural language interaction method based on a quadruped robot, which extracts instruction intent in a manner combining voice recognition and semantic analysis, constructs a voice-scene-state multi-modal dynamic collaborative model by fusing multi-source perception data, generates an executable action chain structure, and drives action planning and execution through a semantic graph. The semantic memory mechanism is introduced to realize behavior migration and action chain reuse of historical interaction. The method has the advantages of accurate instruction understanding, strong environmental adaptability, and efficient interaction response, and can improve the autonomous interaction ability and execution efficiency of the quadruped robot in complex task scenarios.

[0008] According to an embodiment of the present application, an intelligent voice recognition and natural language interaction method based on a quadruped robot comprises the following steps:

[0009] S1, collect the voice instruction input by the user, perform feature encoding and decoding processing on the collected voice signal through a self-supervised voice recognition model, generate corresponding standardized voice text instructions, and extract semantic keywords composed of action targets, spatial descriptions, and logical relationships based on grammar analysis technology;

[0010] S2, collect multi-source perception data of the quadruped robot at the current time, which includes environmental visual image data, inertial measurement unit attitude data, spatial coordinate data, and SLAM space map, and generate structured time sequence state data tensor;

[0011] S3, based on the semantic keyword set and the time sequence state data tensor, construct a voice-scene-state multi-modal dynamic collaborative modeling mechanism, use a semantic embedding module to extract a semantic representation vector, use a multi-channel perception encoding module to extract a modal feature representation vector, and through a cross-modal attention structure, jointly model the semantic representation and the state representation to generate a multi-modal joint semantic embedding vector;

[0012] S4, based on the multi-modal joint semantic embedding vector, construct a semantic graph representation structure, and extract semantic nodes and their dependency relationships to generate an initial action chain plan structure containing multiple sub-action nodes; further, organize the semantic keyword set, multi-source perception state tensor, and actual executed action chain plan structure collected in the historical interaction process into a "semantic-state-action" triple form as expert demonstration trajectory data; based on the above expert trajectory, use a behavior cloning algorithm to train an action generation strategy network, and in the training process, introduce a posture control target function and a gait stability target function to optimize the mapping ability from the semantic graph structure to the action chain sequence; the constructed action generation model accepts the current multi-modal joint semantic embedding vector and spatial semantic constraint as input, and outputs an action chain plan structure that is reasonable in structure, stable in posture, and consistent in semantics, realizing efficient mapping from natural language interaction intent to action control instruction;

[0013] S5, execute each sub-action node based on the action chain plan structure, and realize path adaptive scheduling in combination with the condition variable auto-encoder and the virtual execution model, specifically comprising:

[0014] Based on the multi-modal joint semantic embedding vector and the multi-source perception data, a condition variable auto-encoder model is constructed, the current semantic instruction, the perception state and the initial action chain are used as the condition input, and an action chain structure constraint term is introduced. A candidate action chain path set with semantic consistency and execution diversity is generated by sampling in the latent space, which enhances the breadth and adaptability of the path scheme. In the virtual execution model, the accessibility, posture stability and execution cost of each candidate path are simulated and evaluated, and the optimal path is selected. In the execution stage, inertial and image data are collected in real time to construct an online monitoring flow to monitor the path deviation and stability changes. If an abnormality occurs, the system quickly switches to the backup path to ensure the robust execution and adaptive control of the action sequence in a dynamic environment.

[0015] S6, based on the standardized speech text instruction, multi-source perception data and action chain plan structure, the three are organized into a "semantic instruction-perception context-execution behavior" triple form, and the triple is constructed into a multi-modal semantic behavior memory unit and stored in a semantic memory module for subsequent behavior migration and action chain reuse in similar speech interaction tasks.

[0016] S7, when the quadruped robot receives a new speech instruction, the semantic keyword set of the instruction is extracted and a multi-modal joint semantic embedding vector is generated. Based on the multi-modal joint semantic embedding vector, the historical interaction records are retrieved in the stored semantic memory module, the similarity score of the current instruction and the historical semantic instruction is calculated, and it is judged whether the instruction is reused according to the similarity score. If there is a historical triple with a similarity score higher than a preset threshold, the corresponding action chain plan structure is extracted as a candidate execution scheme, and the environment adaptability is verified in combination with the current perception state data. If the verification is passed, the historical action chain plan structure is directly reused, otherwise the action chain structure is reconstructed and the semantic memory module is updated.

[0017] Optionally, the step S1 specifically comprises:

[0018] S11, collect the natural speech instruction of the user in the interaction process, the speech instruction is input in the form of continuous audio signal, a multi-channel microphone array is used for signal sampling and noise reduction processing, and a clear original speech signal data sequence is generated;

[0019] S12, based on the original speech signal data sequence, input it into a speech recognition model constructed based on self-supervised learning, the speech recognition model includes a feature encoding module, a context modeling module and a decoding module, wherein:

[0020] The feature encoding module extracts frame-level acoustic features through stacked one-dimensional convolutional layers and layer normalization layers; the context modeling module adopts an improved Transformer structure, supports unsupervised mask reconstruction tasks, and optimizes hidden representations using self-supervised pre-training objectives; the decoding module maps the context representations to word-level prediction outputs, combines a CTC loss function to complete time alignment learning of characters or word fragments, and outputs original text sequences;

[0021] S13, based on the original text sequence output by the decoding module, performing standardization processing, including text error correction, punctuation completion, case unification, mood word filtering and language style normalization, to generate a standardized speech text instruction corresponding to the original speech semantics as a semantic expression basis for the current interaction task;

[0022] S14, performing sentence segmentation, word segmentation and syntax analysis on the standardized speech text instruction, extracting action verbs, target entities, spatial orientation words and logical connection words, and constructing a semantic keyword set composed of several semantic units, which is used to represent the core behavior target and scene structure of the current speech instruction, as the semantic input of the multi-modal modeling mechanism in subsequent step S3.

[0023] Optionally, the step S2 specifically comprises:

[0024] S21, acquiring the environmental visual image data of the quadruped robot at the current time, the image data being acquired by a depth camera or an RGB-D sensor installed at the front of the robot, the image frame frequency and the time sequence synchronization module being consistent, for perceiving spatial structure information such as obstacles, boundaries and target objects in the environment;

[0025] S22, acquiring the inertial measurement unit attitude data of the quadruped robot body, the IMU data including three-axis acceleration, three-axis angular velocity and attitude angle information, recording the current motion state and attitude change of the robot;

[0026] S23, acquiring the spatial coordinate data obtained by the robot based on the positioning module, the spatial coordinate data including two-dimensional or three-dimensional position information of the robot in the local coordinate system, and being time-aligned with the environmental visual image data, for calibrating the absolute position of the image data in space;

[0027] S24, constructing a SLAM space map based on the environmental visual image data and the spatial coordinate data, the map establishing a spatial topology structure through feature point extraction and inter-frame matching, for representing the reachable area and obstacle distribution structure of the environment around the robot;

[0028] S25, based on the multi-source perception data collected in the above steps, time stamp alignment and structured organization are performed, and a unified time sequence state data tensor is generated, the time sequence state data tensor organizes each modal perception feature according to a time step, and the time sequence state data tensor is used as a state input basis of a multi-modal dynamic collaborative modeling mechanism in a subsequent step S3.

[0029] Optionally, the step S3 specifically comprises:

[0030] S31, based on the time sequence state data tensor, the time sequence state data tensor is divided into a plurality of modal channels, respectively corresponding to environmental visual image data, inertial measurement unit attitude data, spatial coordinate data and a SLAM space map, each modal channel is respectively sent into an independent perception coding branch, and a modal feature representation vector is extracted;

[0031] S32, based on the semantic keyword set, the semantic keyword set is converted into a semantic embedding vector, the semantic embedding vector is obtained through a publicly disclosed pre-trained language model, the language model supports context encoding of instruction type natural language text, and a BERT, MiniLM or a Transformer model fine-tuned on instruction corpus can be optionally used, and the context dependency relationship of action verbs, spatial orientation words and target entities in the original semantic structure is reserved;

[0032] S33, a cross-modal attention fusion module is constructed, the modal feature representation vector and the semantic embedding vector are input into the fusion module for joint modeling, the fusion module includes a modal weighting submodule and an attention alignment submodule, wherein:

[0033] The modal weighting submodule jointly constructs a modal weight distribution based on the semantic embedding vector and the scene complexity vector, splices the scene complexity vector and the semantic embedding vector to input a weight scoring network to generate a fusion weight of each modal, and obtains a final modal weighting coefficient through softmax normalization, and then performs weighted fusion on all modal feature representation vectors, and outputs a perception weighted representation vector;

[0034] The scene complexity vector is calculated from the obstacle density, the passage channel width and the local space closure degree extracted from the SLAM space map, and is used to measure the complexity of the current environment,

[0035] The attention alignment submodule constructs an attention distribution between the semantic embedding vector and the perception weighted representation vector based on a Cross-Attention mechanism, introduces a space guiding bias term in the calculation process, the space guiding bias term is generated by matching the spatial orientation words in the semantic keyword set and the local space anchor points in the SLAM space map, and is used to adjust the correlation score between the semantic words and the perception regions in the attention matrix, and finally outputs a multi-modal fusion representation vector, which retains the correspondence relationship between the semantic structure and the spatial position, and is used as an input of a subsequent semantic position binding mechanism.

[0036] wherein the local spatial anchor point refers to a region marker point generated by combining the SLAM space map, the environmental visual image data, and the semantic label, which can be mapped by the semantic target and has a clear spatial position, and is used to realize the alignment relationship between the spatial position expression in the voice instruction and the robot perception environment; the semantic label refers to a structured category identifier assigned to core semantic elements such as action verbs, target entities, spatial directions, and logical conjunctions in the natural language instruction analysis process, and is used to guide the function implementation of key modules such as semantic understanding, cross-modal representation alignment, semantic graph construction, and action chain generation;

[0037] S34, connect the multi-modal fusion representation vector and the spatial semantic binding vector, and input them into a multi-layer feedforward network for nonlinear mapping and feature compression to generate a multi-modal joint semantic embedding vector as an input basis for subsequent steps S4 of semantic graph construction and action chain plan generation; wherein the spatial semantic binding vector refers to a semantic embedding representation that fuses spatial position information and is generated by the Cross-Attention mechanism through a spatial guide bias term based on the matching relationship between the spatial direction words in the semantic keyword set and the local spatial anchor points in the SLAM space map in the cross-modal attention alignment process, and is used to express the binding position of the spatial semantic elements in the natural language in the current robot perception environment;

[0038] S35, based on the multi-modal joint semantic embedding vector and the SLAM space map, construct a spatial semantic anchoring mechanism to anchor and match the spatial direction words and the target entities in the semantic keyword set, and identify their local spatial anchor points in the SLAM map. Combine the visual perception results to generate a scene complexity vector, label the semantic label and path accessibility information of each anchor point, form a spatial constraint marking structure, and use it for execution parameter setting of the target sub-action node in the subsequent action chain.

[0039] Optionally, the step S4 specifically comprises:

[0040] S41, construct a semantic graph representation structure based on the multi-modal joint semantic embedding vector, extract the semantic keyword set, the semantic relationship edge, and the spatial accessibility information, and generate an initial action chain plan structure, wherein the action chain plan structure is composed of multiple sub-action nodes and includes the semantic label, execution constraint, and spatial target of each sub-action node;

[0041] S42, collect and construct historical voice interaction records and action chain trajectory data of successful task execution to generate an expert demonstration trajectory dataset, and use a behavior cloning algorithm to train the dataset to form an action generation model, which accepts the multi-modal joint semantic embedding vector and the spatial semantic constraint as input and outputs an executable action chain plan structure.

[0042] S43, in the action generation model training process, introduce the posture control objective function L p With the gait stability objective function L g For constraint policy optimization direction, wherein:

[0043]

[0044] Where R i Indicates the actual posture quaternion of the robot, Indicates the expected posture quaternion generated by the expert trajectory, and N is the total number of frames;

[0045] The formula is derived from:

[0046] This formula is formally a quaternion attitude error minimization method, which is a general objective function form from the fields of robot control, computer graphics, SLAM, etc.

[0047] The improvement point of the formula is:

[0048] For the training process of the behavior cloning policy network, the action generated by the model is guided to keep consistent with the posture demonstrated by the expert; The posture loss is combined with the following stability loss to embed into the objective function of the action generation model, which embodies the structural control of the execution quality; Embedding posture control in the natural language interaction task of quadruped robots combines task semantic driving with low-level posture control;

[0049]

[0050] Wherein And Respectively indicate the contact center positions of the left and right support feet at time t, v t Indicates the center of mass velocity, λ is the weighting factor, and T is the total number of time steps;

[0051] The formula is derived from:

[0052] This formula combines the symmetry control of left and right support foot points (through ‖c l -c r ‖) and the center of mass velocity suppression in form;

[0053] The improvement point of the formula is:

[0054] Introduce this gait loss into the behavior cloning training process, combine it with the semantic atlas control model, and form a closed-loop supervision path of "semantic-action-dynamic execution effect"; In the gait objective function, the control term λ·‖v t ‖ provides a constraint on gait smoothness, which improves the execution reliability of quadruped robots in natural scenes;

[0055] S44, optimizing L p with L g , training the action generation model to enhance the posture control accuracy and gait stability, and finally forming an action chain planning structure output result with posture robustness and dynamic adaptability.

[0056] Optionally, the step S5 specifically comprises:

[0057] S51, based on the action chain planning structure, the multi-modal joint semantic embedding vector and the multi-source perception data, a conditional variable auto-encoder model is constructed, wherein the encoder takes the current semantic instruction, the perception state and the initial action chain as the conditional input to extract the semantic conditional representation; an action chain structure constraint term is introduced, so that when sampling the latent variable, not only the semantic conditional input is relied on, but also the node topology structure and the execution order information of the action chain are considered; the decoder takes the semantic conditional representation and the Gaussian random vector as the input to generate the candidate action chain path, and finally outputs a candidate action chain path set with semantic consistency and execution diversity;

[0058] S52, the candidate action chain path set is input into a virtual execution model, an SLAM space map and a real-time visual image fusion scene are constructed in a simulation environment, a dynamics simulation model is called to simulate the physical behavior of the quadruped robot executing each candidate path, and the accessibility index, the posture stability score and the execution cost evaluation of the corresponding path are output;

[0059] S53, based on the three dimensions of accessibility, posture stability and execution cost, a path evaluation index vector is constructed, a linear weighting strategy is adopted for comprehensive scoring, and the optimal path is automatically selected as the final execution path;

[0060] S54, in the actual action chain execution process, inertial measurement unit data and visual image data are collected in real time to construct an online monitoring stream, and path deviation analysis and posture stability monitoring are performed;

[0061] S55, when the path deviation exceeds the limit or the posture stability decreases in the monitoring stream, the standby path in the candidate action chain path set is automatically switched to, the path reconstruction and action connection control are completed, and the robustness and adaptability of the action sequence execution in the complex dynamic environment are ensured.

[0062] Optionally, the step S6 specifically comprises:

[0063] S61, the standardized speech text instruction generated in the current interaction task is extracted, and based on the grammar analysis result, the action verb, the target entity, the spatial orientation word and the logical connection word are obtained to construct a semantic keyword set as the current semantic instruction information;

[0064] S62, acquire multi-source perception data corresponding to the current semantic instruction execution moment, the perception data including environment visual image, SLAM space map, IMU attitude data and space coordinate information, and construct perception context content based on the multi-modal feature representation vector output by the perception coding module;

[0065] S63, call the action chain plan structure generated by the current round of voice interaction, the action chain structure being composed of multiple sub-action nodes, each node containing target position, action intention, expected attitude and execution constraint parameter as execution behavior information;

[0066] S64, combine the semantic instruction, perception context and execution behavior into a "semantic instruction-perception context-execution behavior" triple form, and construct a behavior memory unit based on the triple, wherein the multi-modal joint semantic embedding vector of the current interaction task is used as the retrieval key, and the triple ontology is used as the storage value;

[0067] S65, write the constructed behavior memory unit into the semantic memory module, the semantic memory module supporting semantic vector retrieval, task label indexing and execution feedback archiving functions, and being stored in a structured manner according to the task execution time sequence;

[0068] S66, attach interaction execution metadata to each behavior memory unit, including instruction initiation time, path accessibility score, execution state label and interaction round number, for supporting adaptability matching and execution verification in the behavior migration and action chain reuse process of subsequent similar semantic tasks.

[0069] Optionally, the step S7 specifically comprises:

[0070] S71, when the user inputs a new natural language voice instruction, the semantic keyword set and the corresponding multi-modal joint semantic embedding vector of the current interaction task are extracted according to the step of claim 2;

[0071] S72, based on the multi-modal joint semantic embedding vector, the semantic memory module is called for retrieval query, and the similarity scores of each multi-modal semantic behavior memory unit in the historical interaction record are calculated;

[0072] S73, the similarity score adopts the following weighted cosine similarity formula, which is used to measure the similarity between the multi-modal joint semantic embedding vector x = [x1, x2,..., x n ] of the current semantic instruction and the embedding vector y = [y1, y2,..., y n ] in the historical semantic memory unit. The mathematical expression is as follows:

[0073]

[0074] wherein:

[0075] x i y i Let w represent the embedding components of the i-th semantic dimension respectively; i ∈[0,1] represents the i-th dimension feature weight obtained by the semantic keyword weight generation network, indicating the degree of attention the current semantic keyword pays to this dimension of the semantic space; the numerator is the inner product operation after introducing the weight, and the denominator is the product of the modulus after introducing the weight, ensuring that the calculation result is normalized to the interval [-1,1].

[0076] Formula source:

[0077] The weighted cosine similarity calculation formula is based on the basic definition of classical cosine similarity and is a publicly available formula in the fields of linear algebra and information retrieval.

[0078] Standard cosine similarity is defined as follows, used to measure the cosine of the angle between two vectors in a high-dimensional space:

[0079]

[0080] in,

[0081] A and B represent two vectors to be compared; A·B represents the dot product of the two vectors; ||A|| and ||B|| represent their respective Euclidean norms.

[0082] Formula improvement and derivation:

[0083] The weighted cosine similarity proposed in this invention, based on the aforementioned classic form, introduces a semantic importance weight factor w to enhance its discriminative ability in the multimodal semantic vector matching process. The specific formula is as follows:

[0084]

[0085] in:

[0086] u=(u1,u2,...,u d ): The multimodal joint semantic embedding vector of the current voice command; v = (v1, v2, ..., v d ): The embedding vector corresponding to the historical semantic instruction; w = (w1, w2, ..., w d ): Dimension-weighted vector, used to represent the importance of each semantic feature, derived from statistical indicators such as part-of-speech and position weight in the semantic keyword set; d: Dimension of the semantic vector.

[0087] Compared to the standard cosine similarity, this formula introduces a weighting factor w in both the numerator and denominator. i In essence, different matching intensities are assigned to semantic components of different dimensions, which effectively improves the model's sensitivity to key semantic features.

[0088] Formula logic derivation process:

[0089] Standard form is:

[0090]

[0091] Introducing semantic weighting term w i , for representing the importance of features of different semantic dimensions:

[0092] The numerator becomes: ∑w i u i v i ; the denominator becomes:

[0093] The final expression is constructed, embodying the similarity measurement method under the semantic weighting mechanism, preserving the normalization property of the cosine distance, and avoiding the interference of the module length on the result.

[0094] S74, according to the set threshold value, screening out the candidate historical triple set with high similarity, and extracting the corresponding action chain plan structure as a candidate execution scheme;

[0095] S75, adaptive verification is performed on the candidate action chain plan structure and the current perception state data to determine whether the path accessibility, obstacle distribution and posture matching degree meet the current environmental requirements;

[0096] S76, if the verification is passed, the historical action chain plan structure is taken as the execution scheme of the current instruction, and there is no need to re-construct the semantic graph and the action chain structure;

[0097] If the verification is not passed, the action chain plan structure is regenerated according to the steps of claim 5, and the step of updating the semantic memory module of claim 7 is called;

[0098] S77, after completing the behavior migration, the new execution result is recorded in the semantic memory module according to the time stamp, for continuously optimizing the experience accumulation and behavior reuse efficiency of the interactive task.

[0099] The beneficial effects of the present application are:

[0100] Firstly, the present application realizes the deep modeling of the correlation between the spatial orientation words, action targets and environmental states in natural language by constructing a multi-modal dynamic collaborative modeling mechanism of voice-scene-state, effectively solves the problem of lack of understanding ability of the current perception environment and state of the robot in the traditional voice interaction system, and improves the accuracy and environmental adaptability of instruction understanding;

[0101] Secondly, the application introduces a multi-modal semantic graph construction and action chain planning mechanism, which can convert the semantic structure in the user voice instruction into a structured behavior sequence containing execution sequence, action constraint and environmental condition, realizes accurate control from language input to multi-step physical execution, and improves the execution efficiency and reliability of natural language instructions;

[0102] In addition, the application designs a semantic memory module and a behavior migration mechanism, which constructs the semantic instructions, environmental perception states and execution behaviors in the historical interaction process into triples and stores them, supports similar task matching and action chain reuse in subsequent interactions, greatly reduces the cost of repeated interactions, and improves the generalization ability and intelligence level of the quadruped robot in dynamic voice interaction scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0103] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application, and do not limit the application. In the drawings:

[0104] Fig. 1 A whole flow chart of an intelligent voice recognition and natural language interaction method based on a quadruped robot is proposed in the application;

[0105] Fig. 2 A structural schematic diagram of a "voice-scene-state multi-modal dynamic cooperative modeling mechanism" in the application;

[0106] Fig. 3 A flow chart of a "multi-modal semantic graph generation and action chain planning mechanism" in the application. DETAILED DESCRIPTION

[0107] The application will now be described in further detail, by way of example only, with reference to the accompanying drawings. These drawings are not to scale and are included herewith for illustrative purposes only. They show, schematically, the basic structure of the application and, therefore, only show the components relevant to the application.

[0108] REFERENCE Figs. 1-3 An intelligent voice recognition and natural language interaction method based on a quadruped robot, comprising the following steps:

[0109] S1, collect the voice instruction input by the user, perform feature encoding and decoding processing on the collected voice signal through a self-supervised voice recognition model, generate the corresponding standardized voice text instruction, and extract the semantic keywords set composed of action targets, spatial descriptions and logical relationships based on grammar analysis technology;

[0110] S2, collect the multi-source perception data of the quadruped robot at the current time, the data including environmental visual image data, inertial measurement unit attitude data, spatial coordinate data and SLAM space map, and generate a structured time sequence state data tensor;

[0111] S3. Based on the semantic keyword set and temporal state data tensor, a speech-scene-state multimodal dynamic collaborative modeling mechanism is constructed. The semantic embedding module is used to extract the semantic representation vector, the multi-channel perception coding module is used to extract the modal feature representation vector, and the semantic representation and state representation are jointly modeled through a cross-modal attention structure to generate a multimodal joint semantic embedding vector.

[0112] S4. Based on multimodal joint semantic embedding vectors, a semantic graph representation structure is constructed, and semantic nodes and their dependencies are extracted to generate an initial action chain plan structure containing multiple sub-action nodes. Furthermore, the semantic keyword set collected during historical interactions, the multi-source perception state tensor, and the actual executed action chain plan structure are organized into a "semantic-state-action" triplet form as expert demonstration trajectory data. Based on the aforementioned expert trajectory, a behavior cloning algorithm is used to train the action generation strategy network. During training, a posture control objective function and a gait stability objective function are introduced to optimize the mapping capability from the semantic graph structure to the action chain sequence. The constructed action generation model accepts the current multimodal joint semantic embedding vector and spatial semantic constraints as input, and outputs a structurally reasonable, posture-stable, and semantically consistent action chain plan structure, achieving efficient mapping from natural language interaction intentions to action control commands.

[0113] S5. Execute each sub-action node based on the action chain plan structure, and achieve adaptive path scheduling by combining a conditionally variable autoencoder and a virtual execution model, specifically including:

[0114] Based on multimodal joint semantic embedding vectors and multi-source perception data, a conditionally variable autoencoder model is constructed. The current semantic instruction, perception state, and initial action chain are used as conditional inputs, and action chain structure constraints are introduced. Through latent space sampling, a set of candidate action chain paths with semantic consistency and execution diversity is generated, enhancing the breadth and adaptability of path schemes. In a virtual execution model, the accessibility, attitude stability, and execution cost of each candidate path are simulated and evaluated, and the optimal path is selected. During the execution phase, inertial and image data are collected in real time to construct an online monitoring stream, monitoring path deviation and stability changes. If an anomaly occurs, the system quickly switches to a backup path, ensuring robust execution and adaptive control of action sequences in dynamic environments.

[0115] S6. Based on standardized voice and text commands, multi-source perception data and action chain planning structure, the three are organized into a triplet form of "semantic command-perception context-execution behavior". The triplet is constructed into a multimodal semantic behavior memory unit and stored in the semantic memory module for behavior transfer and action chain reuse in subsequent similar voice interaction tasks.

[0116] S7, when the quadruped robot receives a new voice instruction, a set of semantic keywords of the instruction is extracted and a multi-modal joint semantic embedding vector is generated, a historical interaction record is retrieved in the stored semantic memory module based on the multi-modal joint semantic embedding vector, a similarity score of the current instruction and the historical semantic instruction is calculated, if there is a historical triple with a similarity higher than a preset threshold, the corresponding action chain plan structure is extracted as a candidate execution scheme, and the environment adaptability is verified combined with the current perception state data, if the verification is passed, the historical action chain plan structure is directly reused, otherwise, the action chain structure is reconstructed and the semantic memory module is updated.

[0117] The application proposes an intelligent voice recognition and natural language interaction method based on a quadruped robot, which is developed around a multi-modal collaborative modeling mechanism of voice-scene-state, and a closed-loop process from voice instruction collection, state perception, semantic modeling, graph construction, action execution, behavior memory to interaction transfer is constructed.

[0118] In the embodiment, the step S1 specifically comprises:

[0119] S11, collecting natural voice instructions of a user in an interaction process, the voice instructions are input in the form of continuous audio signals, a multi-channel microphone array is used for signal sampling and noise reduction processing, and a clear original voice signal data sequence is generated;

[0120] S12, based on the original voice signal data sequence, inputting the original voice signal data sequence into a voice recognition model constructed based on self-supervised learning, the voice recognition model comprising a feature encoding module, a context modeling module and a decoding module, wherein:

[0121] The feature encoding module extracts frame-level acoustic features through a stacked one-dimensional convolution layer and a layer normalization layer; the context modeling module adopts an improved Transformer structure, supports an unsupervised mask reconstruction task, and uses a self-supervised pre-training target to optimize the hidden representation; the decoding module maps the context representation to word-level prediction output, combines a CTC loss function to complete time alignment learning of characters or word fragments, and outputs an original text sequence;

[0122] S13, based on the original text sequence output by the decoding module, performing standardization processing on the original text sequence, the processing including text error correction, punctuation completion, case unification, mood word filtering and language style normalization, generating a standardized voice text instruction corresponding to the original voice semantics as a semantic expression basis of a current interaction task;

[0123] S14, perform sentence segmentation, word segmentation and syntax analysis on the standardized speech text instruction, extract action verbs, target entities, spatial orientation words and logical connection words, and construct a semantic keyword set composed of a plurality of semantic units, the set being used to represent a core behavior target and a scene structure of the current speech instruction as semantic input of a multi-modal modeling mechanism in a subsequent step S3.

[0124] The speech recognition model used includes a feature encoding module, a context modeling module and a decoding module, wherein:

[0125] The feature encoding module is used to perform preliminary feature extraction on the original speech signal data sequence, performs a local perception operation on the input sequence through a stacked one-dimensional convolutional layer, extracts frame-level acoustic feature representations, and introduces layer normalization operations after each convolutional layer to improve training stability and feature expression consistency, and finally generates a stable acoustic encoding vector sequence; the context modeling module adopts an improved Transformer structure to build long-distance context dependency. In the training process, the module introduces an unsupervised mask reconstruction task to predict and recover after partially masking the encoding vector, improving the modeling ability of the model for temporal dependence and semantic continuity in the speech sequence; the training target adopts a multi-task self-supervised optimization strategy to jointly minimize the mask reconstruction loss and the context consistency loss, thereby optimizing the expression ability of the intermediate hidden representation; the decoding module receives the hidden representation sequence output by the context modeling module, converts it into a word-level prediction probability distribution through a linear mapping layer and a softmax output layer, and combines a CTG (Connectionist Temporal Classification) loss function to complete the time alignment learning between speech and text, effectively modeling the inconsistent pronunciation length, pause variation and other phenomena in speech, and finally outputting the original text sequence consistent with the semantic of the speech instruction.

[0126] In the embodiment, the step S2 specifically comprises:

[0127] S21, collect the environmental visual image data of the quadruped robot at the current time, the image data being obtained through a depth camera or an RGB-D sensor installed at the front of the robot, the image frame frequency and the time sequence synchronization module being consistent, and being used to perceive the spatial structure information such as obstacles, boundaries and target objects in the environment;

[0128] S22, collect the inertial measurement unit attitude data of the quadruped robot body, the IMU data including three-axis acceleration, three-axis angular velocity and attitude angle information, recording the current motion state and attitude change of the robot;

[0129] S23, the robot acquires spatial coordinate data based on the positioning module, which includes two-dimensional or three-dimensional position information of the robot in the local coordinate system, and is time-aligned with the environmental visual image data for calibrating the absolute position of the image data in space;

[0130] S24, based on the environmental visual image data and the spatial coordinate data, a SLAM space map is constructed, which establishes a spatial topology structure through feature point extraction and inter-frame matching, and is used to represent the reachable area and obstacle distribution structure of the environment around the robot;

[0131] S25, based on the multi-source perception data collected in the above steps, time stamp alignment and structured organization are performed to generate a unified time sequence state data tensor, which organizes each modal perception feature according to time steps, and the time sequence state data tensor is used as the state input basis of the multi-modal dynamic collaborative modeling mechanism in the subsequent step S3.

[0132] This step focuses on the four-legged robot body and its environmental perception, collects multi-source modal data and generates structured time sequence state data tensor, which covers visual image, inertial attitude, spatial coordinate and SLAM map information, organizes different modal features in a unified format, provides high timeliness and consistency for subsequent multi-modal modeling and state fusion, and reflects the time sequence and structural advantages of environmental modeling.

[0133] In this embodiment, the step S3 specifically includes:

[0134] S31, based on the time sequence state data tensor, it is divided into multiple modal channels, respectively corresponding to environmental visual image data, inertial measurement unit attitude data, spatial coordinate data and SLAM space map, each modal channel is respectively sent into an independent perception coding branch to extract modal feature representation vector;

[0135] S32, based on the semantic keyword set, it is converted into a semantic embedding vector, the semantic embedding vector is obtained through a publicly disclosed pre-trained language model, the language model supports context encoding of instruction type natural language text, and can optionally use BERT, MiniLM or Transformer model fine-tuned on instruction corpus, which retains the context dependency relationship of action verbs, spatial orientation words and target entities in the original semantic structure;

[0136] S33, a cross-modal attention fusion module is constructed, the modal feature representation vector and the semantic embedding vector are input into the fusion module for joint modeling, the fusion module includes a modal weighting submodule and an attention alignment submodule, wherein:

[0137] The modal weighting submodule jointly constructs a modal weight distribution based on a semantic embedding vector and a scene complexity vector, splices the scene complexity vector and the semantic embedding vector to input a weight scoring network to generate a fusion weight of each modality, and obtains a final modal weighting coefficient through softmax normalization, and then performs weighted fusion on all modal feature representation vectors to output a perception weighted representation vector;

[0138] The scene complexity vector is calculated based on the obstacle density, the passage width, and the local space closure degree extracted from the SLAM space map, and is used to measure the complexity of the current environment,

[0139] The attention alignment submodule constructs an attention distribution between the semantic embedding vector and the perception weighted representation vector based on the Cross-Attention mechanism, and introduces a spatial guide bias item in the calculation process. The spatial guide bias item is generated by matching the spatial orientation words in the semantic keyword set with the local space anchor points in the SLAM space map, and is used to adjust the correlation score between the semantic words and the perception regions in the attention matrix. Finally, a multi-modal fusion representation vector is output, which retains the correspondence between the semantic structure and the spatial position, and is used as the input of the subsequent semantic position binding mechanism.

[0140] The local space anchor point refers to a region marker point that can be mapped by a semantic target and has a clear spatial position, which is generated by jointly using the SLAM space map, environmental visual image data, and semantic labels, and is used to realize the alignment relationship between the spatial position expression in the voice instruction and the perception environment of the robot. The semantic label refers to a structured category identifier assigned to core semantic elements such as action verbs, target entities, spatial orientations, and logical conjunctions in the natural language instruction analysis process, which is used to guide the function implementation of key modules such as semantic understanding, cross-modal representation alignment, semantic graph construction, and action chain generation.

[0141] S34, connect the multi-modal fusion representation vector and the spatial semantic binding vector, input into a multi-layer feedforward network for nonlinear mapping and feature compression to generate a multi-modal joint semantic embedding vector, which is used as the input basis for the subsequent steps S4 of semantic graph construction and action chain planning generation. The spatial semantic binding vector refers to a semantic embedding representation that fuses spatial position information, which is generated by regulating the Cross-Attention mechanism based on the matching relationship between the spatial orientation words in the semantic keyword set and the local space anchor points in the SLAM space map through the spatial guide bias item, and is used to express the binding position of the spatial semantic element in the natural language in the current perception environment of the robot.

[0142] S35, based on the multi-modal joint semantic embedding vector and the SLAM space map, a space semantic anchoring mechanism is constructed, the space orientation words and target entities in the semantic keyword set are anchored and matched, and the local space anchor points in the SLAM map are recognized. A scene complexity vector is generated by combining the visual perception result, the semantic label and the path accessibility information of each anchor point are labeled, and a space constraint label structure is formed, which is used for setting the execution parameters of the target sub-action node in the subsequent action chain.

[0143] This step proposes a "voice-scene-state" multi-modal dynamic collaborative modeling mechanism. By introducing semantic embedding models, perception coding branches, cross-modal attention structures, and space semantic binding mechanisms, the semantic keyword set and the time series state tensor are deeply fused, and a multi-modal joint semantic embedding vector expressing the semantic-space-perception association relationship is constructed. This enhances the robot's multi-dimensional perception and understanding of targets, orientations, and intentions in natural language, and embodies the modeling innovation of semantic-driven and perception-collaborative.

[0144] In this embodiment, the step S4 specifically comprises:

[0145] S41, based on the multi-modal joint semantic embedding vector, a semantic graph representation structure is constructed, the semantic keyword set, the semantic relationship edge and the space accessibility information are extracted, and an initial action chain plan structure is generated, wherein the action chain plan structure is composed of multiple sub-action nodes and includes the semantic label, the execution constraint and the space target of each sub-action node;

[0146] S42, collect and construct the historical voice interaction records and action chain trajectory data of successful task execution, generate an expert demonstration trajectory dataset, and use a behavior cloning algorithm to train the dataset to form an action generation model. The model accepts the multi-modal joint semantic embedding vector and the space semantic constraint as input, and outputs an executable action chain plan structure;

[0147] S43, in the training process of the action generation model, a pose control target function L p and a gait stability target function L g are introduced to constrain the optimization direction of the policy, wherein:

[0148]

[0149] wherein R i represents the actual pose quaternion of the robot during execution, represents the expected pose quaternion generated by the expert trajectory, and N is the total number of frames;

[0150] The formula is derived from:

[0151] This formula is formally a quaternion attitude error minimization method, which is a general objective function form from the fields of robot control, computer graphics, SLAM, etc.

[0152] Formula improvement points:

[0153] The training process for the behavior cloning policy network guides the actions generated by the model to remain consistent with the expert demonstrated attitude; this attitude loss is combined with the following stability loss to embed into the action generation model's objective function, reflecting the structural control of execution quality; embedding attitude control in the quadruped robot natural language interaction task combines task semantic driving with low-level attitude control;

[0154]

[0155] where and respectively represent the contact center positions of the left and right support feet at time t, v t represents the center of mass velocity, λ is the weighting factor, and T is the total number of time steps.

[0156] Formula source:

[0157] This formula formally combines left and right support foot point symmetry control (through ‖c l -c r ‖) and center of mass velocity suppression.

[0158] Formula improvement points:

[0159] This gait loss is introduced into the behavior cloning training process and combined with the semantic atlas control model to form a "semantic-action-dynamic execution effect" closed-loop supervision path; in the gait objective function, the control term λ·‖v t ‖ provides a constraint on gait smoothness, improving the execution reliability of quadruped robots in natural scenes.

[0160] S44, by jointly optimizing L p and L g , the action generation model is trained to enhance attitude control accuracy and gait stability, ultimately forming an action chain plan structure output result with attitude robustness and dynamic adaptability.

[0161] The behavior cloning algorithm is used for strategy network training of the data set, specifically including the following processes: first, the successfully executed user voice instructions, environmental perception state and corresponding action chain result in the historical interaction process are organized as a "semantic-state-action" triple, and an expert demonstration data set is constructed. The semantic part comes from the semantic keyword set generated by the standardized voice text instruction, the state information includes the current multi-modal perception input of the robot, such as visual image, depth map and inertial measurement data, and the action part is the sub-action node sequence in the action chain plan structure corresponding to the semantic instruction and state. Then, the expert data set is used as a supervised sample and input into the strategy neural network model to be trained. The strategy network takes the semantic keyword set and spatial semantic constraint as input, extracts joint feature representation through embedding mapping, and generates an action chain plan structure combined with state information. During the training process, a supervised learning method is used to minimize the difference between the action chain output by the strategy network and the expert demonstration action chain, and a posture control objective function and a gait stability objective function are introduced as additional constraint terms for training, to ensure that the strategy network not only accurately matches the target semantic intent when generating an action chain, but also has good posture rationality and gait continuity. Through the above training process, a behavior cloning strategy model that can directly predict the action chain structure from multi-modal input is finally obtained, which is used to respond to new instructions and generate stable and executable action plans in actual operation.

[0162] In the embodiment, the step S5 specifically includes:

[0163] S51, based on the action chain plan structure, the multi-modal joint semantic embedding vector and the multi-source perception data, a conditional variable auto-encoder model is constructed, wherein the encoder takes the current semantic instruction, the perception state and the initial action chain as the conditional input to extract the semantic condition representation; an action chain structure constraint term is introduced, so that when the latent variable is sampled, not only the semantic condition input is relied on, but also the node topology structure and execution order information of the action chain are considered; the decoder takes the semantic condition representation and a Gaussian random vector as input to generate a candidate action chain path, and finally outputs a candidate action chain path set with semantic consistency and execution diversity;

[0164] S52, the candidate action chain path set is input into a virtual execution model, a SLAM spatial map and a real-time visual image fusion scene are constructed in a simulation environment, a dynamics simulation model is called to simulate the physical behavior of the quadruped robot executing each candidate path, and the accessibility index, the posture stability score and the execution cost evaluation of the corresponding path are output;

[0165] S53, based on the three dimensions of accessibility, posture stability and execution cost, a path evaluation index vector is constructed, a linear weighting strategy is used for comprehensive scoring, and the optimal path is automatically selected as the final execution path;

[0166] S54, during the actual action chain execution process, real-time collection of inertial measurement unit data and visual image data constructs an online monitoring stream, and path deviation analysis and posture stability monitoring are performed;

[0167] S55, when path deviation exceeding limit or posture stability decline is detected in the monitoring stream, a backup path in the candidate action chain path set is automatically switched to, path reconstruction and action connection control are completed, and robustness and adaptability of action sequence execution in a complex dynamic environment are ensured.

[0168] The dynamics simulation model is used to virtually evaluate the feasibility of the candidate action chain path in the real physical environment, and its construction process combines the physical parameters of the robot body (including mass distribution, joint freedom, inertia matrix, etc.) and the spatial topological structure information in the environment model. In the action chain path evaluation process, the joint control parameters, posture targets and execution timing information of each action node in the path are accepted as input, and based on the kinematics and dynamics joint modeling mechanism, the continuous motion response of the robot under the given control instruction is simulated. The physical engine module and the collision detection mechanism are included in the simulation engine, which is used to dynamically calculate the posture change, stability response and spatial accessibility index of the robot during path execution, and output results including posture deviation, execution stability score, contact force change curve and path accessibility score.

[0169] In order to improve the adaptability of the evaluation process to the target environment, the dynamics simulation model also combines the three-dimensional spatial point cloud information from the perception module and the SLAM map generated by the mapping system, and reconstructs the obstacle layout and feasible channel information consistent with the real environment in the simulation, so as to ensure that the path evaluation has real space constraints. In the simulation process, if a certain candidate path appears posture imbalance, stability index is lower than the threshold value or pre-collision event occurs with obstacles, the path will be marked as an unusable path and excluded from the candidate set. Finally, the dynamics simulation model provides accurate path optimization basis through parallel evaluation of multiple candidate paths, and provides a safe, stable and feasible path scheme for action chain execution in the real scene.

[0170] In the embodiment, the step S6 specifically includes:

[0171] S61, extracting the standardized speech text instruction generated in the current interaction task, and based on the grammar parsing result, obtaining action verbs, target entities, spatial orientation words and logical connection words, constructing a semantic keyword set as the current semantic instruction information;

[0172] S62, acquire multi-source perception data corresponding to the current semantic instruction execution moment, the perception data including environmental visual image, SLAM space map, IMU attitude data and spatial coordinate information, and construct perception context content based on the multi-modal feature representation vector output by the perception coding module;

[0173] S63, call the action chain plan structure generated by the current round of voice interaction, the action chain structure being composed of multiple sub-action nodes, each node containing target position, action intention, expected posture and execution constraint parameter as execution behavior information;

[0174] S64, combine the semantic instruction, perception context and execution behavior into a "semantic instruction-perception context-execution behavior" triple form, and construct a behavior memory unit based on the triple, wherein the multi-modal joint semantic embedding vector of the current interaction task is used as the retrieval key, and the triple ontology is used as the storage value;

[0175] S65, write the constructed behavior memory unit into the semantic memory module, the semantic memory module supporting semantic vector retrieval, task label indexing and execution feedback archiving functions, and being stored in a structured manner according to the task execution time sequence;

[0176] S66, attach interaction execution metadata to each behavior memory unit, including instruction initiation time, path accessibility score, execution state label and interaction round number, for supporting adaptability matching and execution verification in the behavior migration and action chain reuse process of subsequent similar semantic tasks.

[0177] The semantic memory module is used for storing and managing multi-modal semantic behavior memory units, and supports behavior migration and reuse of action chain structure in natural language interaction tasks. The internal structure of the semantic memory module includes an index coding layer, a storage unit pool and a similarity retrieval interface: the index coding layer is used to convert the multi-modal joint semantic embedding vector generated by each round of interaction task into a fixed-dimensional hash vector or a high-dimensional dense vector as a unique index key; the storage unit pool is used to save the semantic behavior triple content, including semantic instruction information, perception context content and action chain plan structure, and is attached with interaction metadata such as time stamp, execution success rate and task label; the similarity retrieval interface uses weighted cosine similarity as the retrieval measurement method, supports fast calculation of the semantic similarity between the new semantic embedding vector and the historical record when the new semantic embedding vector comes, so as to filter out the migratable action execution scheme. In addition, the module also supports semantic clustering and structure compression of historical interaction samples, realizes dynamic update of long-term memory and preferential reservation of important behavior patterns, and significantly improves the adaptability and semantic migration efficiency of the quadruped robot in continuous dialogue and repetitive task scenarios.

[0178] In the embodiment, the step S7 specifically includes:

[0179] S71. When the user inputs a new natural language voice command, process it according to the steps of claim 2, and extract the semantic keyword set of the current interaction task and the corresponding multimodal joint semantic embedding vector.

[0180] S72. Based on the multimodal joint semantic embedding vector, use it as the retrieval query, call the semantic memory module, and calculate its similarity score with each multimodal semantic behavior memory unit in the historical interaction record;

[0181] S73. The similarity score uses the following weighted cosine similarity formula to measure the multimodal joint semantic embedding vector x = [x1, x2, ..., x...]. n [The embedded vector y = [y1, y2, ..., y] in the historical semantic memory unit n The degree of similarity between [ ]. Its mathematical expression is as follows:

[0182]

[0183] in:

[0184] x i y i Let w represent the embedding components of the i-th semantic dimension respectively; i ∈[0,1] represents the i-th dimension feature weight obtained by the semantic keyword weight generation network, indicating the degree of attention the current semantic keyword pays to this dimension of the semantic space; the numerator is the inner product operation after introducing the weight, and the denominator is the product of the modulus after introducing the weight, ensuring that the calculation result is normalized to the interval [-1,1].

[0185] Formula source:

[0186] The weighted cosine similarity calculation formula is based on the basic definition of classical cosine similarity and is a publicly available formula in the fields of linear algebra and information retrieval.

[0187] Standard cosine similarity is defined as follows, used to measure the cosine of the angle between two vectors in a high-dimensional space:

[0188]

[0189] in,

[0190] A and B represent two vectors to be compared; A·B represents the dot product of the two vectors; ||A|| and ||B|| represent their respective Euclidean norms.

[0191] Formula improvement and derivation:

[0192] The weighted cosine similarity introduced in the application introduces a semantic importance weight factor w on the basis of the classical form, and improves the discrimination ability in the multi-modal semantic vector matching process.

[0193]

[0194] Among them:

[0195] u = (u1, u2,..., u d ): multi-modal joint semantic embedding vector of the current voice instruction; v = (v1, v2,..., v d ): embedding vector corresponding to the historical semantic instruction; w = (w1, w2,..., w d ): dimension weight vector, used to represent the importance of each dimension semantic feature, derived from the part of speech, position weight and other statistical indicators in the semantic keyword set; d: dimension of semantic vector.

[0196] Compared with the standard cosine similarity, the formula introduces a weight factor w i in the numerator and denominator, which essentially gives different matching strengths to different dimensions of semantic components, effectively improving the sensitivity of the model to key semantic features.

[0197] Formula logical derivation process

[0198] The standard form is:

[0199]

[0200] Introducing a semantic weight term w i , which is used to represent the importance of different semantic dimensions:

[0201] The numerator becomes: ∑w i u i v i ; and the denominator becomes:

[0202] The final expression is constructed, which embodies the similarity measurement method under the semantic weighting mechanism, retains the normalization property of the cosine distance, and avoids the interference of the module length on the result.

[0203] S74, according to the set threshold θ, screening out the candidate historical triple set with similarity higher than θ, and extracting the corresponding action chain plan structure as the candidate execution scheme;

[0204] S75, adaptively verifying the candidate action chain plan structure and the current perception state data, and judging whether the path accessibility, obstacle distribution and posture matching degree meet the current environmental requirements;

[0205] S76, if the verification passes, the history action chain plan structure is taken as the execution scheme of the current instruction, without rebuilding the semantic graph and the action chain structure;

[0206] If the verification fails, the action chain plan structure is regenerated according to the step of claim 5, and the step of claim 7 is called to update the semantic memory module;

[0207] S77, after completing the behavior migration, the new execution result is recorded in the semantic memory module according to the time stamp, for continuously optimizing the experience accumulation and behavior reuse efficiency of the interactive task.

[0208] This step realizes the behavior migration and execution optimization based on the semantic memory module. When a new instruction is received, the semantic features are extracted and the high-similarity task is retrieved in the history memory. The current environment state is combined for adaptation verification, so as to realize the reuse of the history action chain and improve the decision efficiency, which embodies the adaptive interactive migration mechanism and high-efficiency task scheduling ability under the language semantic driving.

[0209] Embodiment 1:

[0210] In order to verify the feasibility of the application in implementation, the application is applied to an actual service four-legged robot interactive scene, a certain commercial office building hall is selected as a test environment, the area is about 600 square meters, the environment includes different types of obstacles (including sofas, reception desks, green plants, ornaments, experimental personnel), the scene light condition is variable and the personnel flow is frequent. The four-legged robot carries the intelligent speech recognition and natural language interaction system proposed in the application, and completes the tasks such as navigation, guidance and information broadcast through voice instructions. The goal is to realize the closed-loop control of semantic understanding, path planning, task execution and behavior reuse under natural language interaction.

[0211] In this test scene, the user issues the following instruction to the robot through natural voice: "Please take me to the No. 3 conference room". The voice acquisition module starts the multi-channel microphone array to perform voice sampling and noise reduction processing, extracts the clear voice signal, and converts it into standardized voice text through the self-supervised speech recognition model. The system further extracts the keyword set (including "take", "I", "No. 3 conference room" and the spatial orientation word "to") through the text semantic analysis module.

[0212] At the same time, the robot collects the environment image stream through the visual camera, and obtains the current attitude and motion state through the inertial measurement unit, and constructs a synchronous SLAM map, which labels the obstacle distribution, reachable area and path accessibility index between the current position and the target space. On the basis of the structured state data tensor and the semantic keyword set, the system constructs a multi-modal collaborative modeling module to generate a multi-modal joint semantic embedding vector.

[0213] Subsequently, the system constructs an initial structure of a semantic graph according to the semantic vector, nodes in the graph include semantic elements such as "belt", "I", "No. 3 conference room", etc., connection edges between nodes are constructed through syntactic dependency relations, and nodes in the graph are labeled with spatial attributes according to the reachable path in the SLAM map, including the coordinates of the target position, the path length, the obstacle density, etc. After optimizing the graph structure, the main path is extracted and a plan structure of the action chain is generated, which includes a series of sub-actions such as turning, gait adjustment, path selection and target confirmation for the current map.

[0214] Each sub-action node is assigned specific execution parameters, such as target position coordinates (x = 12.3, y = 5.7), expected turning angle 72°, action constraint that the avoidance radius is not less than 0.5 meters, etc. During path planning, the robot dynamically detects the position changes of surrounding people and objects, and adjusts its gait trajectory in real time. For the occupied area appearing in the original path, the system can automatically generate voice feedback and wait for further instructions or re-plan the path from the user.

[0215] During the entire interaction process, the system automatically organizes the interaction task of this round into a semantic triple including "semantic instruction - perception context - execution behavior", and stores it into the semantic memory module. The recorded data includes interaction time, target position (No. 3 conference room, ID#003), path accessibility score (0.93), number of action chain nodes (a total of 11 steps), and execution completion time. In the subsequent test round, when another user issues a similar voice instruction "I want to go to No. 3 conference room", the system performs weighted cosine similarity matching based on the current semantic embedding vector and the historical records in the semantic memory module, and the score reaches 0.94 (higher than the set threshold 0.9), so the system does not need to rebuild the semantic graph, but directly reuses the historical action chain plan structure, and successfully completes the task execution after adaptive verification of the current environment.

[0216] To comprehensively evaluate the effectiveness of the system in real complex environments, we have conducted 5 days of scene testing, covering 38 users and 132 natural voice instructions. The system successfully recognized and executed 124 tasks, with a semantic analysis accuracy of 93.9%, an action chain execution success rate of 90.2%, an average path planning response time of 2.1 seconds, and a semantic memory reuse rate of 41.7%. In terms of behavior migration, all reused action chain tasks saved more than 3.6 seconds on average compared to re-generation.

[0217] This embodiment verifies the interaction robustness, task response efficiency and semantic behavior migration ability of the method in real complex spaces, fully embodies the innovation advantages of the method in natural language understanding, multi-modal fusion modeling, graph construction and behavior reuse, and can provide efficient, intelligent and continuous semantic interaction support for service-oriented quadruped robots.

[0218] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent substitutions or changes according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for intelligent speech recognition and natural language interaction based on a quadruped robot, characterized in that, The method comprises the following steps: S1, collecting the voice instruction input by the user, performing feature coding and decoding processing on the collected voice signal, generating corresponding standardized voice text instruction, and extracting a semantic keyword set; S2, collecting the multi-source perception data of the quadruped robot at the current time, the data including environment visual image data, inertial measurement unit attitude data, spatial coordinate data and SLAM space map, generating a structured time sequence state data tensor; S3, based on the semantic keyword set and the time sequence state data tensor, constructing a voice-scene-state multi-modal dynamic collaborative modeling mechanism, extracting a modal feature representation vector, and generating a multi-modal joint semantic embedding vector; S4, based on the multi-modal joint semantic embedding vector, constructing a semantic graph representation structure, generating an initial action chain plan structure; at the same time, organizing the successfully implemented historical interaction records into expert demonstration trajectory data, training an action generation model using a behavior cloning algorithm, and introducing a posture control target function and a gait stability target function to improve the dynamic adaptability of the robot in a complex environment, the model accepting the current multi-modal joint semantic embedding vector and spatial semantic constraint as input, and outputting an action chain plan structure; S5, based on the action chain plan structure, executing each sub-action node, and combining a conditionally variable autoencoder and a virtual execution model to realize path adaptive scheduling, specifically including: Based on the multi-modal joint semantic embedding vector and the multi-source perception data, a conditionally variable autoencoder model is constructed, the current semantic instruction, perception state and initial action chain are used as conditional input, and an action chain structure constraint term is introduced to generate a candidate action chain path set with semantic consistency and execution diversity; in the virtual execution model, the accessibility, posture stability and execution cost of each candidate path are simulated and evaluated, and the optimal path is selected; in the execution stage, inertial and image data are collected in real time to construct an online monitoring flow, monitor the path deviation and stability change, and if an abnormality occurs, quickly switch to a backup path to ensure the robust execution and adaptive control of the action sequence in a dynamic environment. S6, based on the standardized voice text instruction, multi-source perception data and action chain plan structure, the three are organized into a "semantic instruction-perception context-execution behavior" triple form, and the triple is constructed into a multi-modal semantic behavior memory unit and stored in a semantic memory module for subsequent behavior migration and action chain reuse in similar voice interaction tasks; S7, when the quadruped robot receives a new voice instruction, the semantic keyword set is extracted and a multi-modal joint semantic embedding vector is generated, based on the multi-modal joint semantic embedding vector, the historical interaction records are retrieved, the similarity score of the current instruction and the historical semantic instruction is calculated, and whether the instruction is reused is determined according to the score. 2.The method of claim 1, wherein, The step S1 specifically includes: S11, collecting the natural voice instruction of the user in the interaction process, using a multi-channel microphone array to perform signal sampling and noise reduction processing, and generating a clear raw voice signal data sequence; S12, based on the raw voice signal data sequence, inputting it into a voice recognition model constructed based on self-supervised learning, and outputting a raw text sequence; S13, based on the original text sequence output by the decoding module, standardizing the original text sequence to generate a standardized voice text instruction corresponding to the original voice semantics; S14, performing sentence segmentation, word segmentation and syntax analysis on the standardized voice text instruction, extracting action verbs, target entities, spatial orientation words and logical connection words, and constructing a semantic keyword set composed of a plurality of semantic units. 3.The method of claim 1, wherein, The step S2 specifically comprises: S21, collecting the environmental visual image data of the quadruped robot at the current time, the image frame frequency and the time sequence synchronization module being consistent, for perceiving the spatial structure information in the environment; S22, collecting the inertial measurement unit attitude data of the quadruped robot body, recording the current motion state and attitude change of the robot; S23, collecting the spatial coordinate data obtained by the robot based on the positioning module, and time aligning the environmental visual image data, for calibrating the absolute position of the image data in space; S24, constructing a SLAM space map based on the environmental visual image data and the spatial coordinate data, for representing the reachable area and obstacle distribution structure of the environment around the robot; S25, based on the multi-source perception data collected in the above steps, time stamp alignment and structured organization are performed to generate a unified time sequence state data tensor. 4.The method of claim 1, wherein, The step S3 specifically comprises: S31, based on the time sequence state data tensor, dividing it into a plurality of modal channels, respectively corresponding to the environmental visual image data, the inertial measurement unit attitude data, the spatial coordinate data and the SLAM space map, each modal channel being respectively sent into an independent perception coding branch to extract a modal feature representation vector; S32, based on the semantic keyword set, converting it into a semantic embedding vector; S33, constructing a cross-modal attention fusion module, inputting the modal feature representation vector and the semantic embedding vector into the fusion module for joint modeling, the fusion module including a modal weighting submodule and an attention alignment submodule, the modal weighting submodule performing weighted fusion on all modal feature representation vectors to output a perception weighted representation vector; The attention alignment submodule constructs the attention distribution between the semantic embedding vector and the perception weighted representation vector, and outputs a multi-modal fusion representation vector; S34, connecting the multi-modal fusion representation vector and the spatial semantic binding vector, and inputting them into a multi-layer feedforward network for nonlinear mapping and feature compression to generate a multi-modal joint semantic embedding vector; S35, based on the multi-modal joint semantic embedding vector and the SLAM space map, constructing a spatial semantic anchoring mechanism, anchoring and matching the spatial orientation words and target entities in the semantic keyword set to form a spatial constraint label structure.

5. The method for intelligent speech recognition and natural language interaction based on quadruped robot according to claim 1, characterized in that, The step S4 specifically comprises: S41, constructing a semantic graph representation structure based on the multi-modal joint semantic embedding vector, extracting the semantic keyword set, semantic relationship edges and spatial accessibility information, and generating an initial action chain plan structure; S42, collect and construct the history voice interaction record and action chain track data of successful task execution, generate expert demonstration track data set, train the data set by using behavior cloning algorithm to form action generation model, the model accepts multi-modal joint semantic embedding vector and spatial semantic constraint as input, and outputs executable action chain plan structure; S43, in the action generation model training process, introduce the posture control target function L p With the gait stability target function L g For constraint policy optimization direction, wherein: wherein R i represents the pose quaternion of the robot when actually executed, represents the desired pose quaternion generated by the expert trajectory, and N is the total number of frames. wherein with respectively represent the contact center positions of the supporting feet on both sides around time t, v t denotes the centroid velocity, λ is a weighting factor, and T is the total number of time steps. S44, optimizing L p With L g , training the action generation model to enhance the posture control accuracy and gait stability. 6.The method of intelligent speech recognition and natural language interaction based on quadruped robot according to claim 1, characterized in that, The step S5 specifically comprises: S51, based on the action chain plan structure, the multi-modal joint semantic embedding vector and the multi-source perception data, a conditional variable auto-encoder model is constructed, wherein the encoder takes the current semantic instruction, the perception state and the initial action chain as the conditional input to extract the semantic condition representation; an action chain structure constraint term is introduced, so that when the latent variable is sampled, not only the semantic condition input is relied on, but also the node topological structure and the execution order information of the action chain are considered; the decoder takes the semantic condition representation and the Gaussian random vector as the input to generate the candidate action chain path, and finally outputs the candidate action chain path set with semantic consistency and execution diversity; S52, input the candidate action chain path set into the virtual execution model, construct the SLAM space map and the real-time visual image fusion scene in the simulation environment, call the dynamics simulation model to simulate the physical behavior of the quadruped robot executing each candidate path, and output the accessibility index, the attitude stability score and the execution cost evaluation of the corresponding path; S53, based on the three dimensions of accessibility, attitude stability and execution cost, a path evaluation index vector is constructed, a linear weighting strategy is adopted for comprehensive scoring, and the optimal path is automatically selected as the final execution path; S54, in the actual action chain execution process, inertial measurement unit data and visual image data are collected in real time to construct an online monitoring stream, path deviation analysis and attitude stability monitoring are performed; S55, when the path deviation exceeds the limit or the attitude stability decreases in the monitoring stream, the standby path in the candidate action chain path set is automatically switched to, path reconstruction and action connection control are completed, and the robustness and adaptability of the action sequence execution in the complex dynamic environment are ensured.

7. The method for intelligent speech recognition and natural language interaction based on quadruped robot according to claim 1, characterized in that, The step S6 specifically comprises: S61, extract the standardized voice text instruction and the semantic keyword set in the current interaction task, and extract the semantic instruction information; S62, acquire the multi-source perception data corresponding to the current semantic instruction, and extract the perception context content; S63, call the current round action chain plan structure, and extract the execution behavior content; S64, form a triple structure by combining the semantic instruction information, the perception context content and the execution behavior content, construct a multi-modal semantic behavior memory unit, and write the semantic memory module with the multi-modal joint semantic embedding vector of the current interaction task as the index key; S65, additional interaction metadata is stored in a structured manner according to the execution time sequence, which is used for subsequent behavior migration and action chain reuse. 8.The method of claim 1, wherein, The step S7 specifically comprises: S71, when a new natural language voice instruction is input by the user, the semantic keyword set of the current interaction task and the corresponding multi-modal joint semantic embedding vector are extracted; S72, based on the multi-modal joint semantic embedding vector, the semantic memory module is called for the retrieval query to calculate the similarity score with each multi-modal semantic behavior memory unit in the historical interaction record; S73, according to the set threshold, the candidate historical triple set with the similarity higher than the threshold is screened out, and the corresponding action chain plan structure is extracted as the candidate execution scheme; S74, the adaptive verification is performed on the candidate action chain plan structure and the current perception state data to judge whether the path accessibility, obstacle distribution and posture matching degree meet the current environment requirement; S75, if the verification is passed, the historical action chain plan structure is taken as the execution scheme of the current instruction, and it is not necessary to re-construct the semantic graph and the action chain structure; if the verification is not passed, the action chain plan structure is re-generated, and the semantic memory module is updated; S76, after the behavior migration is completed, the new execution result is recorded to the semantic memory module according to the time stamp.

Images