A voice interaction method and system based on a humanoid robot and a storage medium

Abstract

The present application relates to a kind of voice interaction method, system and storage medium based on humanoid robot, relate to humanoid robot technical field;Voice interaction method includes: the voice signal in the interaction interval of collection and judgment is determined, target character position is determined in combination with preset wake-up keyword, and target interaction object is identified;The voice feature of the target interaction object is extracted, and target interaction atlas is constructed in combination with interaction object image;According to the target interaction atlas, the corresponding voice interaction block in the preset voice interaction template library is matched;The voice content of the target interaction object is analyzed, and compliant reply text is generated, and corresponding interaction sentence is output according to the voice interaction block;Through end-to-end multi-modal modeling, the naturalness of interaction is improved, combined with dynamic checking mechanism, risk output is reduced, while supporting dialect and special population adaptation, reduce overall response delay time, improve the compliance of reply content.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot technology, and in particular to a voice interaction method, system and storage medium based on a humanoid robot. Background Technology

[0002] With the explosive growth in demand for flexible manufacturing, the adaptability bottleneck of traditional automated equipment in complex and unstructured environments is becoming increasingly prominent. In particular, solutions with human spatial cognition capabilities are urgently needed in fields such as logistics sorting and precision assembly. At the same time, the requirement for cross-scenario intelligent collaboration is driving the evolution of robotics technology toward environmental understanding and autonomous decision-making. This has made humanoid robots that combine three-dimensional environmental perception and real-time cognitive computing the core carrier for breaking through the limitations of dynamic operation scenarios, in order to meet the needs of precise and unmanned operation of highly mixed material flow.

[0003] Humanoid robots are a dimension that urgently needs to be broken through in current development. Although many different types of humanoid robots have emerged at present, there are still limitations. These limitations are numerous, such as the inability to achieve a human-like appearance, or their control and application being rigid, logically confused, or lacking in precision, etc.

[0004] Application number CN202510142422.0 discloses a multilingual speech recognition and interaction method for humanoid robots, specifically relating to the field of speech recognition and interaction technology. It effectively separates background noise and target speech signals by using a multi-channel high-sensitivity microphone array combined with an adaptive noise cancellation algorithm, improving the accuracy of speech signal recognition in noisy environments. Through in-depth analysis of the speech signal and background noise feature data of the target interaction object, it calculates the abnormal coefficient of speech rate fluctuation, the coefficient of pitch deviation, and the coefficient of background noise influence. It adjusts the sentence segmentation model according to the different speakers' speech rate, pitch, pauses, and other habits to avoid inaccurate sentence segmentation under different users or contexts. Based on the sentence segmentation adaptation optimization model, it generates a sentence segmentation adaptation optimization index, and combines this with a threshold to determine whether the parameters of the sentence segmentation model need adjustment, enabling real-time response and optimization, reducing misjudgments or omissions, and improving the overall fluency and accuracy of the interaction.

[0005] The existing technical solutions mentioned above have the following drawbacks: 1. The multimodal feature fusion model has a lack of historical behavior data in the cold start scenario (first interaction), which leads to deviation in the prediction of interaction intent; 2. The current dual-channel verification mechanism relies on a static rule base and cannot be updated in real time to meet changes in industry regulations, resulting in compliance delays. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a voice interaction method, system, and storage medium based on a humanoid robot. This method improves the naturalness of interaction through end-to-end multimodal modeling, reduces risk output by combining dynamic verification mechanisms, supports adaptation to dialects and special populations, reduces overall response latency, and improves the compliance of response content.

[0007] The objective of this invention is achieved through the following technical solution: A voice interaction method based on a humanoid robot includes: Collect and judge the voice signals within the preset interaction range, combine them with the preset wake-up keywords to determine the location of the target character, and identify the target interaction object; Extract the speech features of the target interactive object and combine them with the image of the interactive object to construct a target interaction map; Based on the target interaction graph, the corresponding voice interaction module is matched from the preset voice interaction template library; Analyze the voice content of the target interactive object, generate compliant response text, and output the corresponding interactive statements according to the voice interaction module.

[0008] By employing the aforementioned technical solution, the location of the target interaction object is determined through a sound source localization (DOA) algorithm combined with wake word recognition. Voice features are extracted using a Conformer-Transformer model and fused with image features extracted by ArcFace via a cross-attention mechanism to construct a multimodal interaction graph. Based on features such as age and emotion in the graph, voice interaction modules from a pre-defined template library are matched. DeepSeek deep thinking logic is used to generate response content, and OpenAI's dual-channel verification mechanism (factual verification + compliance screening) ensures output security. Finally, multilingual TTS synthesized speech is output. By integrating voiceprint recognition (x-vector), visual analysis (ResNet-50), multimodal attention mechanisms, and a federated learning update system, accurate cross-modal feature matching, real-time compliance assurance, and personalized interactive experience are achieved, reducing overall response latency and improving the compliance of response content.

[0009] The present invention is further configured such that: the specific steps of collecting and judging the voice signal within the preset interaction interval, determining the target character position by combining it with the preset wake-up keyword, and identifying the target interaction object include: According to the preset interaction range, the robot collects all sound wave signals to obtain voice signals; The speech signal is subjected to signal-to-noise separation to obtain an environmental noise signal and a mixed sound source signal; The mixed sound source signal is filtered and converted to obtain multi-target mixed speech content; The multi-target mixed speech content is decomposed frame by frame to obtain several audio frames, and the corresponding audio energy value is calculated and compared with a preset energy threshold. If the audio energy value is greater than or equal to the energy threshold, the current audio frame is determined to be a non-silent interval, and all the non-silent intervals are aggregated to obtain a new audio. The new audio is subjected to timbre recognition and voiceprint separation to obtain several character audio segments; The audio clips of all the characters are converted into text and matched with preset wake-up keywords to obtain several target audio clips; The target audio segment is located by calculating the timestamp difference between adjacent target audio segments to obtain the target character's position; the target character's position includes the character's orientation angle and the target character's distance. The voice signal of the target character is enhanced based on the distance to the target character and the environmental noise signal to generate a directional target voice beam, and the target interaction object is determined by combining the character's orientation angle.

[0010] By adopting the above technical solution, mixed sound sources are obtained through sound wave acquisition and signal-to-noise separation. After filtering and conversion, non-silent intervals are aggregated using audio frame energy threshold detection. Character audio segments are extracted by combining timbre recognition and voiceprint separation technology. Then, target segments are located by text conversion and wake word matching. Finally, the character's orientation angle and distance are calculated based on the timestamp difference. Environmental noise signals are fused to enhance the target speech and generate directional beams. This achieves accurate spatial positioning and noise suppression of specific interactive objects in multi-sound source scenarios, significantly improving the robustness of voice interaction in complex environments.

[0011] The present invention is further configured such that: the specific steps of extracting the speech features of the target interactive object and combining them with the image of the interactive object to construct the target interactive graph include: The target speech signal is obtained by collecting the speech of the target interactive object based on the directional target speech beam; Speech features are extracted from the target speech signal to obtain the target language, target timbre, target fundamental frequency, and target speech rate; The target timbre, the target fundamental frequency, and the target speech rate are deeply fused and extracted to obtain a spectral feature vector and a prosodic feature vector; The target voiceprint feature matrix is ​​obtained by performing several iterations of training on the spectral feature vector and the prosodic feature vector. The target interactive object is image captured to obtain an image of the interactive object; Facial features are extracted and pose features are analyzed from the image of the interactive object to obtain facial morphology vectors and pose interaction vectors. The facial shape vector and the posture interaction vector are trained iteratively several times to obtain the target behavior feature matrix; The target voiceprint feature matrix and the target behavior feature matrix are weighted and fused using a cross-attention mechanism to obtain the local feature matrix of the character. Graph operations are performed on the local feature matrix of the character in conjunction with the target language to obtain the target interaction graph.

[0012] By employing the above technical solution, the spectral features (MFCC) and prosodic features (F0, speech rate) of the speech signal are extracted using the Conformer model. These are then combined with x-vector voiceprint features to generate a voiceprint matrix. ResNet-50 is used to extract facial morphology vectors, and OpenPose is used to obtain posture interaction vectors. These are then iteratively trained using Transformer to form a behavioral feature matrix. A cross-attention mechanism is used to fuse voiceprint and behavioral features, constructing a local feature matrix for the character. Finally, a GNN graph neural network is used to generate a target interaction map, combining the target language (LangID). By dynamically aligning heterogeneous features through multimodal feature fusion (speech, vision, behavior) and the attention mechanism, the accuracy of character identification and the robustness of interaction intent parsing are significantly improved. Furthermore, real-time interaction across languages ​​(through LangID adaptation) and across scenes (noise / lighting changes) is supported.

[0013] The present invention is further configured such that: the specific steps of matching the corresponding voice interaction module from the preset voice interaction template library according to the target interaction graph include: Information is parsed and extracted from the target interaction graph to obtain the target age and interaction logic; The target age is matched and mapped according to a preset age level to obtain a speech parameter block; the speech parameter block includes fundamental frequency, speech rate, frequency of onomatopoeia, frequency of popular internet slang, pause time, frequency of abbreviations and number of times information is repeated; The interaction logic is template matched according to the preset interaction task type to obtain the interaction template framework; Based on the voice interaction architecture of the preset voice interaction template library, the voice parameter block and the interaction template framework are cross-fused to obtain the voice interaction module.

[0014] By employing the above technical solution, the BERT model is used to analyze the target interaction graph, extracting the target age and interaction logic. The K-means clustering algorithm is then used to map the target age to a preset age level, generating speech parameter blocks containing parameters such as fundamental frequency and speech rate. An RNN model is then used to match the interaction logic with a preset interaction task type, generating an interaction template framework. Finally, a DNN deep neural network is used to cross-integrate the speech parameter blocks and the interaction template framework to generate the voice interaction module. Through multimodal deep learning algorithms (BERT, K-means, RNN, DNN), accurate interaction intent parsing and personalized voice interaction generation are achieved, significantly improving the personalization and naturalness of the user experience.

[0015] The present invention is further configured such that: the specific steps of analyzing the voice content of the target interactive object, generating compliant reply text, and outputting corresponding interactive statements according to the voice interaction module include: The speech content of the target interactive object is subjected to tone recognition and text translation to obtain a syllable sequence list and dialogue text. The dialogue text is classified and analyzed according to a preset multilingual stop word database to obtain a dialogue vocabulary and a dialogue grammar table. Match each phrase in the communication vocabulary list according to the preset multilingual keyword list to obtain several communication keywords; Recursively decompose all the aforementioned communication keywords in conjunction with the communication syntax table to obtain several semantic reasoning requests; The semantic reasoning request is retrieved sequentially based on a multi-source knowledge base to obtain relevant knowledge files; Based on the self-thinking logic, text extraction and content generation were performed on the relevant knowledge documents to obtain several draft response texts; All draft response texts are verified according to a preset dual-channel verification mechanism to select compliant response texts; The compliant response text is translated based on the voice interaction module, and the corresponding interactive statements are output.

[0016] By employing the above technical solutions, the Conformer model is used to achieve tone recognition and ASR text translation of speech signals. BERT is combined for stop word filtering and dependency parsing to generate a communication grammar table. A BiLSTM-CRF model is used to match multilingual keywords and recursively decompose them into semantic reasoning requests. The FAISS vector retrieval engine is used to retrieve relevant knowledge documents from multi-source knowledge bases (such as Wikipedia and industry databases). The DeepSeek large-scale model is used for content generation and logical reasoning to generate draft responses. Finally, a dual-channel verification process using a rule engine and a deep learning model (such as factual verification + compliance screening) ensures output security, and interactive sentences are synthesized based on multilingual TTS. By integrating end-to-end speech recognition, multimodal semantic parsing, and dynamic knowledge retrieval, combined with a real-time compliance verification mechanism, output security is guaranteed. Personalized interaction across languages ​​(Chinese / English / Japanese / Korean) and dialects is supported, reducing response latency.

[0017] The present invention is further configured such that the specific steps of extracting text and generating content from the relevant knowledge documents based on self-thinking logic to obtain several draft response texts include: All related files are deconstructed based on data type to obtain text, images, and tables; Events are extracted from the text to obtain an event processing overview; the image and table are identified and parsed to obtain a data analysis file. Entity association and graph construction are performed on the event handling overview and the data analysis file to obtain a global event graph; Based on the self-thinking logic and communication keywords, all graph nodes in the global event graph are matched and located, the event matching degree is calculated, and compared with the preset matching threshold. If the event matching degree is greater than the matching threshold, the current graph node is marked and aggregated to obtain the event processing thought chain; The event handling thought chain is used to generate content, resulting in an event handling draft. The matching accuracy is calculated and compared with a preset accuracy threshold. If the matching accuracy is greater than the accuracy threshold, the content of the current event processing draft is determined to be accurate, and the sentence coherence is calculated and compared with the preset sentence consistency threshold. If the coherence of the statement is greater than the statement consistency threshold, then the logic of the current event processing draft is determined to be correct, and it is used as the response draft text.

[0018] By employing the above technical solution, the QGA-EE model is used to extract events from text and construct a global event graph (based on a GNN graph neural network). The DeepSeek large model's self-thinking logic is combined to semantically associate OCR-recognized image / table data. A graph attention network (GAT) is used to dynamically calculate event matching degree. An event processing draft is generated through a Transformer architecture, and a rule engine and a BERT coherence detection model are used for joint verification. Finally, a response draft that meets preset thresholds is output. By integrating event extraction, multimodal parsing, and dynamic graph inference, a hierarchical verification mechanism (matching degree ≥95%, coherence ≥92%) ensures content accuracy and logical rigor, supporting collaborative processing of cross-modal data (text / image / table) and accurate response generation in complex scenarios.

[0019] The present invention is further configured such that: the specific steps of verifying all the draft response texts according to the preset dual-channel verification mechanism and filtering compliant response texts include: Based on the factual channel in the preset dual-channel verification mechanism, entity extraction is performed on all draft response texts to obtain entity names and operation sequences; The entity names are matched according to the preset domain knowledge graph, the entity matching degree is calculated, and the entity confidence degree is obtained by combining the graph node weights and comparing it with the preset confidence threshold. If the confidence level of the entity is less than the confidence threshold, the current draft text of the response is determined to have failed verification, and a rejection is triggered. Otherwise, conflict detection is performed on the order of each step in the operation sequence. If there is a contradiction between the steps, the contradictory steps are corrected according to the domain knowledge graph to obtain a corrected step sequence. The draft response text is then corrected to obtain the communication response text. Based on the compliance channel in the preset dual-channel verification mechanism, and combined with the preset risk rule base, the communication response text is corrected for compliance, resulting in a de-identified response text; The verification strength of the desensitized response text is calculated based on the target age and the risk profile adaptation mechanism, and then compared with the preset age verification range. If the verification strength is less than the age verification range, the current anonymized reply text is determined to be unethical, and the corresponding reply content voice is output according to the legal weight and user risk level; otherwise, the current anonymized reply text is determined to be compliant.

[0020] By adopting the above technical solution, entity extraction and entity matching degree calculation are performed using the BERT-BiLSTM-CRF model (based on GNN graph node weights). A rule engine and dependency parser are used to detect and correct operation sequence conflicts, generating communication response text. Compliance de-identification processing is completed using regular expressions and rule matching mechanisms. Finally, a risk profiling adaptation mechanism (an age verification model based on logistic regression) dynamically adjusts the verification strength to ensure that the output content meets standards in three dimensions: factual accuracy (entity confidence ≥ 98%), logical consistency (conflict correction rate ≥ 99%), and ethical compliance (de-identification coverage 100%). The advantage of this solution lies in its integration of deep learning entity recognition, graph reasoning, and dynamic risk adaptation, enabling cross-domain knowledge verification and personalized compliance control, supporting multi-level defense (fact-logic-ethics) and real-time response.

[0021] Secondly, the present invention also provides a voice interaction system based on a humanoid robot, employing the following technical solution: A voice interaction system based on a humanoid robot, used to implement the voice interaction method of the humanoid robot, includes: The identification and positioning module is used to collect and judge voice signals, combine them with preset wake-up keywords to determine the location of the target character, and identify the target interaction object; The image construction module is used to extract the voice features of the target interactive object and combine them with the image of the interactive object to construct the target interactive graph. The logic matching module is used to match the corresponding voice interaction module from the preset voice interaction template library according to the target interaction graph; The self-checking module is used to extract text and generate content from relevant knowledge documents based on self-thinking logic, resulting in several draft response texts. It is then combined with a dual-channel verification mechanism to verify and select compliant response texts. The voice interaction module is used to analyze the voice content of the target interaction object and generate and translate the interactive statements of the compliant response text according to the voice interaction module.

[0022] By employing the above technical solution, the target role's location is determined through the DOA sound source localization algorithm combined with wake word recognition (based on the Conformer-Transformer model). ArcFace is used to extract image features of the interactive object and construct a target interaction graph by cross-attention with speech features. Personalized voice interaction modules are extracted from a template library based on the BERT-K-means dual-mode matching algorithm. A QGA-EE event extraction model combined with a DeepSeek large model is used to generate a draft response. The solution employs BERT-BiLSTM-CRF entity verification and a risk profiling adaptation mechanism (logistic regression dynamically adjusts the verification strength) to achieve dual verification of factual accuracy and compliance. Finally, interactive sentences are synthesized through multilingual TTS. The advantage of this solution lies in its support for dialect / special population adaptation and real-time multilingual interaction through multimodal feature fusion (speech, vision, behavior) and an end-to-end deep learning architecture.

[0023] Thirdly, the present invention also provides an electronic device, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above scheme.

[0024] Fourthly, the present invention also provides a storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the voice interaction method based on a humanoid robot as described above.

[0025] In summary, the beneficial technical effects of the present invention are as follows: 1. Humanoid robots improve the naturalness of interaction through end-to-end multimodal modeling, and reduce risk output by combining dynamic verification mechanisms; 2. Supports adaptation to dialects and special populations, reducing overall response latency and improving the compliance of response content; Attached Figure Description Figure 1 This is a flowchart illustrating a humanoid robot voice interaction method according to one embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating a humanoid robot voice interaction method according to one embodiment of the present invention.

[0027] Figure 3 This is a flowchart illustrating a humanoid robot voice interaction method according to one embodiment of the present invention.

[0028] Figure 4This is a schematic diagram of the structure of a humanoid robot voice interaction method according to one embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings.

[0030] Example 1: Reference Figure 1 The present invention discloses a voice interaction method based on a humanoid robot, comprising: S1: Collect and judge the voice signal within the preset interaction range, combine it with the preset wake-up keyword to determine the location of the target character, and identify the target interaction object; S2: Extract the speech features of the target interactive object and combine them with the image of the interactive object to construct a target interactive graph; S3: Based on the target interaction map, match the corresponding voice interaction module from the preset voice interaction template library; S4: Analyze the voice content of the target interactive object, generate compliant reply text, and output the corresponding interactive statements according to the voice interaction module.

[0031] In this embodiment, the humanoid robot can receive sounds within a 2-meter radius around its body, identify the language of the sounds through an AI database, and automatically eliminate interference from non-human languages. The elimination logic is to not further identify or act on non-human languages; this part needs optimization as there is a risk of misjudgment. After identifying the language, such as Chinese, the language recognition module further identifies the language content. After confirming the semantics, AI generates response logic, and then performs a self-check using AI after generating the response content. Here, the response generation logic uses DeepSeek's deep thinking logic, while the self-check uses another system, such as OpenAI. After the response content passes self-checking, the voice component can translate the generated response into the received language. During this process, if unrecognizable content or content that cannot be generated due to local laws cannot be responded to, the user can inquire about the direction of the sound source or express that they cannot respond in the received language. This part of the response does not require quality control and can be generated directly.

[0032] The implementation principle of this embodiment is as follows: The humanoid robot collects voice signals within a 2-meter radius through a microphone array and determines the spatial location of the target interactive object by combining the DOA sound source localization algorithm. At the same time, it performs wake word recognition and language detection based on the Conformer model. After confirming that the language is a human language (such as Chinese), it further analyzes the user's intent through the speech recognition and semantic understanding module. Then, it generates response content based on the DeepSeek large model and performs content verification and risk screening through an independent OpenAI-like model or a BERT-based compliance detection system to ensure that the output content meets the requirements in terms of factuality, logic, and ethical compliance. If the content cannot be recognized or cannot be generated due to local regulations, the robot can directly use a preset response template that does not require quality inspection (such as "I cannot answer this question at the moment") to provide feedback to the sound source in the original language. At the same time, it can combine the speech rate, tone, and emotion parameters of the voice interaction module to provide personalized output to improve the naturalness of the interaction and user-friendliness. The entire process achieves an efficient, safe, and natural voice interaction experience through multimodal fusion, end-to-end generation, and multi-system collaborative verification mechanism.

[0033] Example 2: Step S1 includes: According to the preset interaction range, the robot collects all sound wave signals to obtain voice signals; The speech signal is subjected to signal-to-noise separation to obtain an environmental noise signal and a mixed sound source signal; The mixed sound source signal is filtered and converted to obtain multi-target mixed speech content; The multi-target mixed speech content is decomposed frame by frame to obtain several audio frames, and the corresponding audio energy value is calculated and compared with a preset energy threshold. If the audio energy value is greater than or equal to the energy threshold, the current audio frame is determined to be a non-silent interval, and all the non-silent intervals are aggregated to obtain a new audio. The new audio is subjected to timbre recognition and voiceprint separation to obtain several character audio segments; The audio clips of all the characters are converted into text and matched with preset wake-up keywords to obtain several target audio clips; The target audio segment is located by calculating the timestamp difference between adjacent target audio segments to obtain the target character's position; the target character's position includes the character's orientation angle and the target character's distance. The voice signal of the target character is enhanced based on the distance to the target character and the environmental noise signal to generate a directional target voice beam, and the target interaction object is determined by combining the character's orientation angle.

[0034] The implementation principle of this embodiment is as follows: The robot collects sound wave signals within the interaction zone through a microphone array, separates environmental noise and mixed sound source signals based on a deep learning signal-to-noise separation model (such as Conv-TasNet), extracts multi-target speech content using masking excitation technology, optimizes non-silent zone aggregation by combining short-time energy detection and zero-crossing rate analysis, enhances the target character's speech using a multi-channel beamforming algorithm (such as delayed summation beamforming), and extracts character audio segments through an end-to-end voiceprint separation model (such as DCCRN). Accurate positioning is achieved by combining contextual semantic analysis and wake word matching. After positioning calculation, the character's orientation angle and distance are further corrected by adaptive filter banks and spatial spectrum estimation (such as the MUSIC algorithm), and spatial filtering technology is used to suppress interference signals. Finally, a high signal-to-noise ratio directional speech beam is generated, which, combined with a language model (such as Transformer), realizes semantic-level recognition and interactive response of the target interactive object, thereby achieving highly robust multi-target speech separation and positioning interaction in complex acoustic environments.

[0035] Example 3: Step S2 includes: The target speech signal is obtained by collecting the speech of the target interactive object based on the directional target speech beam; Speech features are extracted from the target speech signal to obtain the target language, target timbre, target fundamental frequency, and target speech rate; The target timbre, the target fundamental frequency, and the target speech rate are deeply fused and extracted to obtain a spectral feature vector and a prosodic feature vector; The target voiceprint feature matrix is ​​obtained by performing several iterations of training on the spectral feature vector and the prosodic feature vector. The target interactive object is image captured to obtain an image of the interactive object; Facial features are extracted and pose features are analyzed from the image of the interactive object to obtain facial morphology vectors and pose interaction vectors. The facial shape vector and the posture interaction vector are trained iteratively several times to obtain the target behavior feature matrix; The target voiceprint feature matrix and the target behavior feature matrix are weighted and fused using a cross-attention mechanism to obtain the local feature matrix of the character. Graph operations are performed on the local feature matrix of the character in conjunction with the target language to obtain the target interaction graph.

[0036] The implementation principle of this embodiment is as follows: the robustness of speech features is enhanced through adaptive normalization processing, a 3D pose estimation model (such as OpenPose) is introduced to improve the dimension of image features, and a dynamic routing algorithm (such as Capsule Network) is combined to optimize the multimodal attention weight allocation. The spatiotemporal graph convolutional network (ST-GCN) is used to dynamically update the interaction graph structure. In the feature fusion stage, cross-modal long-range dependency modeling is achieved through the Transformer-XL architecture, and the correlation between voiceprint and behavioral features is calibrated in real time based on an online incremental learning mechanism. Finally, a dynamic interaction graph containing semantic-behavioral-spatial relationships is generated through iterative propagation of graph neural networks (GNN), realizing the temporal consistency alignment and multidimensional semantic association of cross-modal features, and improving the accuracy and stability of role identification and interaction intent understanding in complex scenarios.

[0037] Example 4: Step S3 includes: Information is parsed and extracted from the target interaction graph to obtain the target age and interaction logic; The target age is matched and mapped according to a preset age level to obtain a speech parameter block; the speech parameter block includes fundamental frequency, speech rate, frequency of onomatopoeia, frequency of popular internet slang, pause time, frequency of abbreviations and number of times information is repeated; The interaction logic is template matched according to the preset interaction task type to obtain the interaction template framework; Based on the voice interaction architecture of the preset voice interaction template library, the voice parameter block and the interaction template framework are cross-fused to obtain the voice interaction module.

[0038] The implementation principle of this embodiment is as follows: By introducing a context-aware age prediction model (such as dynamic age estimation based on spatiotemporal graph convolutional networks) and a reinforcement learning-driven interaction logic optimizer, the extraction accuracy of the target age and interaction logic is calibrated in real time; combined with an online incremental learning mechanism, the speech parameter blocks (such as fundamental frequency, speech rate, onomatopoeia frequency, etc.) are adaptively adjusted to adapt to changes in user behavior and environmental noise; in the template matching stage, semantic similarity calculation and fuzzy logic are integrated to dynamically select the optimal interaction template framework from the preset library; and a Transformer-based generative model is applied to perform multimodal deep fusion, cross-optimizing the speech parameter blocks and template framework to generate personalized voice interaction modules, achieving high naturalness and robustness of voice response across age groups and task scenarios, and improving the accuracy of understanding interaction intent and user satisfaction.

[0039] Example 5: Reference Figure 2 Step 4 includes: S41: Perform tone recognition and text translation on the speech content of the target interactive object to obtain a syllable sequence list and dialogue text; S42: Classify and parse the dialogue text according to the preset multilingual stop word library to obtain a dialogue vocabulary list and a dialogue grammar list; S43: Match each phrase in the communication vocabulary according to the preset multilingual keyword list to obtain several communication keywords; S44: Recursively decompose all the communication keywords in conjunction with the communication syntax table to obtain several semantic reasoning requests; S45: Retrieve relevant knowledge files by sequentially searching the semantic reasoning request based on the multi-source knowledge base; S46: Based on the self-thinking logic, extract text from the relevant knowledge documents and generate content to obtain several draft response texts; S47: Verify all the draft response texts according to the preset dual-channel verification mechanism, and filter out compliant response texts; S48: Translate the compliant response text according to the voice interaction module and output the corresponding interactive statements.

[0040] The implementation principle of this embodiment is as follows: By introducing a dialogue history encoder based on the Transformer architecture, the coherence and accuracy of semantic reasoning are improved. The reliability of knowledge retrieval is enhanced by combining the BERT-entity link joint verification mechanism. In the response generation stage, personalized parameters driven by user profiles (such as sentiment tendency and interaction preference) are introduced to optimize text style adaptation. Finally, through a dual-channel verification mechanism that integrates rule engines (such as logistic regression risk assessment) and deep learning models (such as OpenAI-like models), the multi-dimensional compliance of the response content in terms of factuality, compliance, and sentiment adaptation is ensured. At the same time, through a multilingual TTS synthesis module (supporting prosodic transfer and sentiment embedding), the compliant response text is translated into natural speech interaction sentences that conform to the language habits and emotional needs of the target interaction object, thereby realizing a closed loop of intelligent dialogue across languages ​​and scenarios.

[0041] Example 6: Reference Figure 3 Step S45 includes: All related files are deconstructed based on data type to obtain text, images, and tables; Events are extracted from the text to obtain an event processing overview; the image and table are identified and parsed to obtain a data analysis file. Entity association and graph construction are performed on the event handling overview and the data analysis file to obtain a global event graph; Based on the self-thinking logic and communication keywords, all graph nodes in the global event graph are matched and located, the event matching degree is calculated, and compared with the preset matching threshold. If the event matching degree is greater than the matching threshold, the current graph node is marked and aggregated to obtain the event processing thought chain; The event handling thought chain is used to generate content, resulting in an event handling draft. The matching accuracy is calculated and compared with a preset accuracy threshold. If the matching accuracy is greater than the accuracy threshold, the content of the current event processing draft is determined to be accurate, and the sentence coherence is calculated and compared with the preset sentence consistency threshold. If the coherence of the statement is greater than the statement consistency threshold, then the logic of the current event processing draft is determined to be correct, and it is used as the response draft text.

[0042] The implementation principle of this embodiment is as follows: By introducing a multimodal data fusion framework (such as CLIP, M6) to achieve collaborative parsing of text, images and tables, a dynamic threshold adjustment mechanism (based on online learning algorithms) is combined to optimize event matching and semantic coherence evaluation, and semantic similarity calculation (such as Sentence-BERT) is used to enhance the matching accuracy of graph nodes. At the same time, a context-aware Transformer-XL architecture is embedded in the draft generation stage to ensure logical coherence. Finally, the system threshold and matching strategy are dynamically optimized through a reinforcement learning-driven feedback loop, realizing end-to-end intelligent processing of cross-modal data-driven event graph construction and high-accuracy, high-coherence response draft generation.

[0043] Example 7: Reference Figure 4 Step S46 includes: Based on the factual channel in the preset dual-channel verification mechanism, entity extraction is performed on all draft response texts to obtain entity names and operation sequences; The entity names are matched according to the preset domain knowledge graph, the entity matching degree is calculated, and the entity confidence degree is obtained by combining the graph node weights and comparing it with the preset confidence threshold. If the confidence level of the entity is less than the confidence threshold, the current draft text of the response is determined to have failed verification, and a rejection is triggered. Otherwise, conflict detection is performed on the order of each step in the operation sequence. If there is a contradiction between the steps, the contradictory steps are corrected according to the domain knowledge graph to obtain a corrected step sequence. The draft response text is then corrected to obtain the communication response text. Based on the compliance channel in the preset dual-channel verification mechanism, and combined with the preset risk rule base, the communication response text is corrected for compliance, resulting in a de-identified response text; The verification strength of the desensitized response text is calculated based on the target age and the risk profile adaptation mechanism, and then compared with the preset age verification range. If the verification strength is less than the age verification range, the current anonymized reply text is determined to be unethical, and the corresponding reply content voice is output according to the legal weight and user risk level; otherwise, the current anonymized reply text is determined to be compliant.

[0044] The implementation principle of this embodiment is as follows: By introducing a multimodal knowledge graph (such as Graph-BERT combined with visual-text joint reasoning) to enhance the context-awareness of entity verification, embedding a causal reasoning model (such as CausalTransformer) to optimize the dynamic correction strategy for operation sequence conflict detection, and integrating the context-aware Transformer-XL architecture in the compliance channel to achieve time-sensitive verification of risk rules, while dynamically updating the domain knowledge graph and risk rule base through an incremental learning mechanism (such as Federated Learning), and combining user behavior time-series modeling (such as LSTM-Attention) to achieve real-time adaptive adjustment of age verification intensity, and finally constructing a closed-loop feedback system (such as a threshold optimization strategy based on reinforcement learning) in the factual and compliance dual channels to ensure the accuracy and ethical compliance of cross-modal and cross-scenario entity verification, and achieve intelligent, dynamic, and highly robust response generation.

[0045] Example 8: A voice interaction system based on a humanoid robot, applied to the aforementioned fault detection method, includes: The identification and positioning module is used to collect and judge voice signals, combine them with preset wake-up keywords to determine the location of the target character, and identify the target interaction object; The image construction module is used to extract the voice features of the target interactive object and combine them with the image of the interactive object to construct the target interactive graph. The logic matching module is used to match the corresponding voice interaction module from the preset voice interaction template library according to the target interaction graph; The self-checking module is used to extract text and generate content from relevant knowledge documents based on self-thinking logic, resulting in several draft response texts. It is then combined with a dual-channel verification mechanism to verify and select compliant response texts. The voice interaction module is used to analyze the voice content of the target interaction object and generate and translate the interactive statements of the compliant response text according to the voice interaction module.

[0046] In this embodiment, an environmental perception module (which achieves noise suppression and spatial localization through microphone array and SLAM technology), a dynamic modeling module (which updates the semantic association of the interaction graph in real time based on incremental graph neural network), an emotion adaptation module (which uses a multimodal emotion recognition model to analyze user micro-expressions and voice emotions), and a collaborative decision-making module (which introduces federated reinforcement learning to optimize the cross-scene generalization ability of the template library) are added to the existing modules. At the same time, a knowledge tracing mechanism is embedded in the self-checking system module (which tracks the source of the credibility of knowledge files through attention weights).

[0047] The implementation principle of this embodiment is as follows: the environmental perception module integrates sound source localization and spatial modeling technology to accurately identify the location of the target role; the dynamic modeling module continuously optimizes the semantic relevance of the interaction graph through incremental graph learning; the logical matching module adaptively selects interaction modules from the federated optimized template library with the help of a collaborative decision-making mechanism; the self-checking system module combines knowledge tracing and dual-channel verification to achieve multi-dimensional content compliance verification; and the emotion adaptation module synchronously analyzes the user's emotional state and injects it into the prosody generation model of the voice interaction module, ultimately forming a closed-loop interaction system of environmental perception, dynamic modeling, and emotional feedback, achieving highly robust anthropomorphic voice interaction.

[0048] Example: An electronic device, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above scheme.

[0049] Example: A storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the humanoid robot-based voice interaction method as described above.

[0050] The embodiments described herein are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A voice interaction method based on a humanoid robot, characterized in that, include: Collect voice signals within a preset interaction range, combine them with preset wake-up keywords to determine the location of the target character, and identify the target interaction object; The specific steps for extracting the speech features of the target interaction object and combining them with the image of the interaction object to construct the target interaction map include: The target speech signal is obtained by collecting the speech of the target interactive object based on the directional target speech beam; Speech features are extracted from the target speech signal to obtain the target language, target timbre, target fundamental frequency, and target speech rate; The target timbre, the target fundamental frequency, and the target speech rate are deeply fused and extracted to obtain a spectral feature vector and a prosodic feature vector; The target voiceprint feature matrix is ​​obtained by performing several iterations of training on the spectral feature vector and the prosodic feature vector. The target interactive object is image captured to obtain an image of the interactive object; Facial features are extracted and pose features are analyzed from the image of the interactive object to obtain facial morphology vectors and pose interaction vectors. The facial shape vector and the posture interaction vector are trained iteratively several times to obtain the target behavior feature matrix; The target voiceprint feature matrix and the target behavior feature matrix are weighted and fused using a cross-attention mechanism to obtain the local feature matrix of the character. Graph operations are performed on the local feature matrix of the character and the target language to obtain the target interaction graph. The specific steps for matching the corresponding voice interaction module from the preset voice interaction template library based on the target interaction graph include: Information is parsed and extracted from the target interaction graph to obtain the target age and interaction logic; The target age is matched and mapped according to a preset age level to obtain a speech parameter block; The interaction logic is template matched according to the preset interaction task type to obtain the interaction template framework; Based on the voice interaction architecture of the preset voice interaction template library, the voice parameter block and the interaction template framework are cross-fused to obtain the voice interaction module; Analyze the voice content of the target interactive object, generate compliant response text, and output the corresponding interactive statements according to the voice interaction module.

2. The voice interaction method based on a humanoid robot according to claim 1, characterized in that, The specific steps for collecting voice signals within a preset interaction range, combining them with preset wake-up keywords to determine the target character's location, and identifying the target interaction object include: According to the preset interaction range, the robot collects all sound wave signals to obtain voice signals; The speech signal is subjected to signal-to-noise separation to obtain an environmental noise signal and a mixed sound source signal; The mixed sound source signal is filtered and converted to obtain multi-target mixed speech content; The multi-target mixed speech content is decomposed frame by frame to obtain several audio frames, and the corresponding audio energy value is calculated and compared with a preset energy threshold. If the audio energy value is greater than or equal to the energy threshold, the current audio frame is determined to be a non-silent interval, and all the non-silent intervals are aggregated to obtain a new audio. The new audio is subjected to timbre recognition and voiceprint separation to obtain several character audio segments; The audio clips of all the characters are converted into text and matched with preset wake-up keywords to obtain several target audio clips; The target audio segment is located by calculating the timestamp difference between adjacent target audio segments to obtain the target character's position; the target character's position includes the character's orientation angle and the target character's distance. The voice signal of the target character is enhanced based on the distance to the target character and the environmental noise signal to generate a directional target voice beam, and the target interaction object is determined by combining the character's orientation angle.

3. The voice interaction method based on a humanoid robot according to claim 1, characterized in that, The specific steps for analyzing the voice content of the target interactive object, generating compliant response text, and outputting corresponding interactive statements according to the voice interaction module include: The speech content of the target interactive object is subjected to tone recognition and text translation to obtain a syllable sequence list and dialogue text. The dialogue text is classified and analyzed according to a preset multilingual stop word database to obtain a dialogue vocabulary and a dialogue grammar table. Match each phrase in the communication vocabulary list according to the preset multilingual keyword list to obtain several communication keywords; Recursively decompose all the aforementioned communication keywords in conjunction with the communication syntax table to obtain several semantic reasoning requests; The semantic reasoning request is retrieved sequentially based on a multi-source knowledge base to obtain relevant knowledge files; Based on self-thinking logic, text extraction and content generation were performed on the relevant knowledge documents to obtain several draft response texts; All draft response texts are verified according to a preset dual-channel verification mechanism to select compliant response texts; The compliant response text is translated based on the voice interaction module, and the corresponding interactive statements are output.

4. The voice interaction method based on a humanoid robot according to claim 3, characterized in that, The specific steps for extracting text and generating content from the relevant knowledge documents based on self-thinking logic to obtain several draft response texts include: All related files are deconstructed based on data type to obtain text, images, and tables; Events are extracted from the text to obtain an event processing overview; the image and table are identified and parsed to obtain a data analysis file. Entity association and graph construction are performed on the event handling overview and the data analysis file to obtain a global event graph; Based on the self-thinking logic and communication keywords, all graph nodes in the global event graph are matched and located, the event matching degree is calculated, and compared with the preset matching threshold. If the event matching degree is greater than the matching threshold, the current graph node is marked and aggregated to obtain the event processing thought chain; The event handling thought chain is used to generate content, resulting in an event handling draft. The matching accuracy is calculated and compared with a preset accuracy threshold. If the matching accuracy is greater than the accuracy threshold, the content of the current event processing draft is determined to be accurate, and the sentence coherence is calculated and compared with the preset sentence consistency threshold. If the coherence of the statement is greater than the statement consistency threshold, then the logic of the current event processing draft is determined to be correct, and it is used as the response draft text.

5. The voice interaction method based on a humanoid robot according to claim 3, characterized in that, The specific steps for verifying all the draft response texts according to the preset dual-channel verification mechanism and selecting compliant response texts include: Based on the factual channel in the preset dual-channel verification mechanism, entity extraction is performed on all draft response texts to obtain entity names and operation sequences; The entity names are matched according to the preset domain knowledge graph, the entity matching degree is calculated, and the entity confidence degree is obtained by combining the graph node weights and comparing it with the preset confidence threshold. If the confidence level of the entity is less than the confidence threshold, the current draft text of the response is determined to have failed verification, and a rejection is triggered. Otherwise, conflict detection is performed on the order of each step in the operation sequence. If there is a contradiction between the steps, the contradictory steps are corrected according to the domain knowledge graph to obtain a corrected step sequence. The draft response text is then corrected to obtain the communication response text. Based on the compliance channel in the preset dual-channel verification mechanism, and combined with the preset risk rule base, the communication response text is corrected for compliance, resulting in a de-identified response text; The verification strength of the desensitized response text is calculated based on the target age and the risk profile adaptation mechanism, and then compared with the preset age verification range. If the verification strength is less than the age verification range, the current anonymized reply text is determined to be unethical, and the corresponding reply content voice is output according to the legal weight and user risk level; otherwise, the current anonymized reply text is determined to be compliant.

6. A voice interaction system based on a humanoid robot, used to implement the voice interaction method as described in any one of claims 1-5, characterized in that, include: The identification and positioning module is used to collect and judge voice signals, combine them with preset wake-up keywords to determine the location of the target character, and identify the target interaction object; The image construction module is used to extract the voice features of the target interactive object and combine them with the image of the interactive object to construct the target interactive graph. The logic matching module is used to match the corresponding voice interaction module from the preset voice interaction template library according to the target interaction graph; The self-checking module is used to extract text and generate content from relevant knowledge documents based on self-thinking logic, resulting in several draft response texts. It is then combined with a dual-channel verification mechanism to verify and select compliant response texts. The voice interaction module is used to analyze the voice content of the target interaction object and generate and translate the interactive statements of the compliant response text according to the voice interaction module.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the voice interaction method based on a humanoid robot as described in any one of claims 1 to 5.

Images