Voice interaction method and apparatus, and XR device

Abstract

The invention provides a voice interaction method and device and XR equipment, and relates to the technical field of voice processing, and the method comprises the steps: obtaining a to-be-recognized voice and a current voice recognition text; according to the to-be-recognized voice and the current voice recognition text, performing retrieval to obtain historical voice similar information; splicing the current voice recognition text and the historical voice similar information to obtain spliced information; and reasoning the spliced information to generate a voice interaction result. According to the invention, the problems of low interaction success rate and poor user experience caused by poor understanding accuracy of XR equipment on personalized pronunciations of different users in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the field of voice processing technology, and in particular to a voice interaction method, apparatus, and XR device. Background Art

[0002] With the recent proliferation of XR (Extended Reality) devices, voice interaction has become a crucial human-machine interface for smart devices like XR glasses. Users can use voice to adjust XR glasses parameters (e.g., volume, screen brightness) or perform tasks (e.g., travel route planning, shopping, translation, etc.). However, existing XR voice interaction systems typically use fixed speech recognition models, which are poorly adaptable to user accents, pronunciation characteristics, and personalized spoken expressions. This results in poor speech recognition accuracy and a degraded user experience.

[0003] Therefore, how to improve the accuracy of XR devices' understanding of different users' personalized pronunciations has become a key technical bottleneck in improving the human-computer interaction experience of XR devices. Summary of the Invention

[0004] The present invention provides a voice interaction method, apparatus, and XR device to solve the problem in the prior art that XR devices have poor accuracy in understanding the personalized pronunciations of different users, resulting in low interaction success rate and poor user experience.

[0005] The present invention provides a voice interaction method, comprising: Get the speech to be recognized and its current speech recognition text; Retrieving historical speech similarity information based on the speech to be recognized and the current speech recognition text; Splicing the current speech recognition text and the historical speech similarity information to obtain splicing information; The splicing information is inferred to generate a voice interaction result.

[0006] According to a voice interaction method provided by the present invention, retrieving historical voice similarity information based on the voice to be recognized and the current voice recognition text includes: Generate a current semantic vector based on the speech to be recognized and the current speech recognition text, and generate a current pronunciation vector based on the current speech recognition text; Retrieving a first historical similar speech from a preset experience database according to the current semantic vector and obtaining a first similarity; Retrieving a second historical similar speech from the preset experience library according to the current pronunciation vector, and obtaining a second similarity; determining a target historical similar speech according to the first historical similar speech, the first similarity, the second historical similar speech, and the second similarity; The historical speech similarity information corresponding to the target historical similar speech is obtained from the preset experience database.

[0007] According to a speech interaction method provided by the present invention, generating a current semantic vector based on the speech to be recognized and the current speech recognition text includes: The speech to be recognized is subjected to noise suppression, echo cancellation, pre-emphasis, endpoint detection, and frame windowing processing in sequence, and the Mel spectrum features of each frame of speech data are calculated; Splicing the Mel-spectrogram features to generate a speech feature matrix; Segmenting the current speech recognition text, mapping the segmentation results to a pre-trained vocabulary to obtain a word unit sequence; Projecting the speech feature matrix and the word unit sequence into the same hidden dimension space to obtain a speech embedding sequence and a text embedding sequence; Fusing the speech embedding sequence with the text embedding sequence to obtain a fused embedding sequence; The fused embedding sequence is encoded to obtain a current semantic vector.

[0008] According to a voice interaction method provided by the present invention, generating a current pronunciation vector according to the current voice recognition text includes: Converting the current speech recognition text into a pronunciation symbol sequence; Mapping each pronunciation symbol in the pronunciation symbol sequence to a pronunciation embedding vector of a preset dimension through a preset embedding lookup table; Average pooling is performed on the pronunciation embedding vector to generate a current pronunciation vector.

[0009] According to a voice interaction method provided by the present invention, determining a target historical similar voice based on the first historical similar voice, the first similarity, the second historical similar voice, and the second similarity includes: Normalizing the first similarity to obtain a third similarity, and normalizing the second similarity to obtain a fourth similarity; wherein the third similarity includes a third semantic similarity and a third pronunciation similarity, and the fourth similarity includes a fourth semantic similarity and a fourth pronunciation similarity; performing weighted summation on the third semantic similarity and the third pronunciation similarity according to a preset weight to obtain a first fused similarity; performing a weighted summation of the fourth semantic similarity and the fourth pronunciation similarity according to the preset weight to obtain a second fused similarity; According to the first fusion similarity and the second fusion similarity, a target historical similar speech is screened from the first historical similar speech and the second historical similar speech.

[0010] According to a voice interaction method provided by the present invention, after inferring the splicing information and generating a voice interaction result, the method further includes: Outputting an execution operation according to the voice interaction result and obtaining interaction feedback information; generating a feedback tag according to the interactive feedback information; The current speech recognition text, the current semantic vector, the current pronunciation vector and the feedback label are stored in the preset experience library.

[0011] According to a voice interaction method provided by the present invention, generating a feedback tag according to the interaction feedback information includes: If the type of the interactive feedback information is voice feedback, performing semantic recognition on the interactive feedback information, and determining a semantic feedback label according to the semantic recognition result; If the type of the interaction feedback information is behavior feedback, matching the interaction feedback information with a preset mapping relationship between the behavior type and the behavior feedback label to obtain a behavior feedback label; If the type of the interaction feedback information is interface feedback, the interaction feedback information is matched with a preset mapping relationship between the interface feedback type and the interface feedback label to obtain the interface feedback label.

[0012] According to a voice interaction method provided by the present invention, after generating a feedback tag according to the interaction feedback information, the method further includes: If the feedback tag is determined to be positive feedback, the number of times the target historical similar voice is used and the most recent time of use in the preset experience database are updated; The preset experience library is cleaned up regularly based on the updated usage count and the updated most recent usage time.

[0013] The present invention also provides a voice interaction device, comprising: An acquisition module is used to obtain the speech to be recognized and its current speech recognition text; A retrieval module, configured to retrieve historical speech similarity information based on the speech to be recognized and the current speech recognition text; A splicing module, configured to splice the current speech recognition text and the historical speech similarity information to obtain splicing information; The interaction module is used to infer the splicing information and generate a voice interaction result.

[0014] The present invention also provides an XR device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the voice interaction method described above is implemented.

[0015] The voice interaction method, apparatus, and XR device provided by the present invention obtain a speech to be recognized and its current speech recognition text; retrieve historical speech similarity information based on the speech to be recognized and the current speech recognition text; concatenate the current speech recognition text with the historical speech similarity information to obtain concatenated information; and perform inference on the concatenated information to generate a speech interaction result. In the present invention, by utilizing the speech to be recognized and the current speech recognition text to retrieve historical interaction experience and using this historical interaction experience as a real-time calibration source, the accuracy of the speech interaction results can be significantly improved, thereby improving the interaction success rate and user experience. Furthermore, by intelligently concatenating the current speech recognition text with the historical speech similarity information, complete and formatted concatenated information is constructed as inference input to generate a speech interaction result. This approach avoids retraining complex deep learning models, requiring only low-complexity, lightweight inference to generate high-quality speech interaction results. This significantly reduces computing resource consumption and response latency, significantly improving the interaction success rate and robustness in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a system architecture diagram of the voice interaction system provided by the present invention; Figure 2 This is one of the flow charts of the voice interaction method provided by the present invention; Figure 3 This is the second flow chart of the voice interaction method provided by the present invention; Figure 4 This is the third flow chart of the voice interaction method provided by the present invention; Figure 5 It is a structural diagram of the voice interaction device provided by the present invention; Figure 6 It is a structural schematic diagram of the XR device provided by the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] With the recent proliferation of XR (Extended Reality) devices, voice interaction has become a crucial human-machine interface for smart devices like XR glasses. Users can use voice to adjust XR glasses parameters (e.g., volume, screen brightness) or perform tasks (e.g., travel route planning, shopping, translation, etc.). However, existing XR voice interaction systems typically use fixed speech recognition models, which are poorly adaptable to user accents, pronunciation characteristics, and personalized spoken expressions. This results in poor speech recognition accuracy and a degraded user experience.

[0020] Therefore, how to improve the accuracy of XR devices' understanding of different users' personalized pronunciations has become a key technical bottleneck in improving the human-computer interaction experience of XR devices.

[0021] Based on the above problems, the present invention proposes a voice interaction method, device and XR device. Figures 1-6 Provide a description.

[0022] like Figure 1 As shown, the voice interaction system includes an extended reality (XR) device 01 and a terminal 02.

[0023] XR devices 01 include, but are not limited to, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and MR (Mixed Reality) devices, and are used to implement the voice interaction method of the present invention. Terminals 02 can be, but are not limited to, the XR devices themselves, various personal computers, laptops, smartphones, tablets, and IoT devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices, and are used to execute corresponding operations based on control commands output by the XR devices.

[0024] For example, for voice home control scenarios, users can say "turn on the TV", "raise the air conditioner temperature", etc. The system adapts to the user's dialect pronunciation, accurately recognizes the voice interaction results, and then controls smart home devices such as smart TVs and smart air conditioners based on the voice interaction results.

[0025] Figure 2 This is one of the flow charts of the voice interaction method provided by the present invention, such as Figure 2 As shown, the voice interaction method includes: step S110, step S120, step S130 and step S140.

[0026] Step S110: Acquire the speech to be recognized and its current speech recognition text.

[0027] In this embodiment, the voice interaction method is applied to XR devices.

[0028] XR devices are wearable or portable devices that achieve human-computer interaction by integrating virtual and real environments through hardware and software technologies. XR devices include, but are not limited to, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and MR (Mixed Reality) devices. VR devices use computer technology to simulate and generate three-dimensional virtual spaces, allowing users to immerse themselves in and interact with them for an immersive experience. AR devices use technology to fuse virtual information with the real world, overlaying it onto real scenes in real time to enhance the sensory experience. MR devices blend the real and virtual worlds to create a new visual environment that combines physical entities and virtual information, allowing users to interact with these entities and virtual information in real time.

[0029] This voice interaction method is suitable for voice interaction scenarios such as in-vehicle voice interaction, voice home control, assisted interaction for people with disabilities, industrial site operations in noisy environments, multilingual learning and translation, etc.

[0030] The speech to be recognized is obtained and recognized through a speech recognition model, such as an ASR (Automatic Speech Recognition) model, to obtain speech recognition text, which is recorded as the current speech recognition text to distinguish it from the historical speech recognition text.

[0031] Step S120 , retrieving historical speech similarity information based on the speech to be recognized and the current speech recognition text.

[0032] Based on the speech to be recognized and the current speech recognition text, historical speech similarity information is retrieved from a preset experience database. The preset experience database is a pre-built experience database used to store historical speech interaction information, including but not limited to: timestamp, speech, semantic vector, pronunciation vector, speech recognition text, speech correction text, feedback result, feedback type, speech feedback subtype, behavioral feedback action, system execution status, used speech recognition model version, used LLM model version, number of uses, and last use time.

[0033] It should be noted that in the initial phase when there is no voice interaction information, the preset experience library can be populated using preset "template phrases" or sample questions and answers. As voice interaction information is subsequently generated, the preset experience library can be incrementally updated based on successful interactions or high-confidence samples explicitly confirmed by the user. Low-confidence or repeatedly failed records are regularly filtered and cleaned to prevent the accumulation of erroneous samples from negatively impacting the voice interaction results.

[0034] As an implementation method, a current semantic vector is generated based on the speech to be recognized and the current speech recognition text; based on the current semantic vector, a first historical similar speech is retrieved from a preset experience library as the target historical similar speech; and historical speech similarity information corresponding to the target historical similar speech is obtained from the preset experience library.

[0035] As another implementation method, a current semantic vector is generated based on the speech to be recognized and the current speech recognition text, and a current pronunciation vector is generated based on the current speech recognition text; based on the current semantic vector, a first historical similar speech is retrieved from a preset experience library, and based on the current pronunciation vector, a second historical similar speech is retrieved from the preset experience library, and the first historical similar speech and the second historical similar speech are used as target historical similar speech; and historical speech similarity information corresponding to the target historical similar speech is obtained from the preset experience library.

[0036] As another embodiment, a current semantic vector is generated based on the speech to be recognized and the current speech recognition text, and a current pronunciation vector is generated based on the current speech recognition text; based on the current semantic vector, a first historical similar speech is retrieved from a preset experience library, and a first similarity is obtained; based on the current pronunciation vector, a second historical similar speech is retrieved from the preset experience library, and a second similarity is obtained; based on the first historical similar speech, the first similarity, the second historical similar speech, and the second similarity, a target historical similar speech is determined; and historical speech similarity information corresponding to the target historical similar speech is obtained from the preset experience library.

[0037] Step S130 , concatenating the current speech recognition text and the historical speech similarity information to obtain concatenated information.

[0038] Here, the historical speech similarity information includes historical speech recognition text, and also includes at least one of a timestamp, historical speech correction text, feedback result, behavior feedback action, and system execution status.

[0039] The current speech recognition text and historical speech similarity information are concatenated according to a preset template to generate concatenated information. The preset template can take the form of "currently heard + past similar experience," and the output format can be selected as "direct answer" or "reverse question confirmation" based on a single or multiple candidate intents. This approach enhances the model's robustness to ambiguous accents or ambiguous commands.

[0040] As an implementation method, the preset template is as follows: Currently hearing: "{T_cur}".

[0041] Past interactions with similar accents are as follows (up to 3 records in reverse chronological order): {Time_i1}, hear "{T_i1}", the user corrects it to "{T_i1 }", [Execution: {OK / FAIL}]; {Time_i2}, after hearing "{T_i2}", the user clicks the "Confirm" button, [Behavior Feedback: CONFIRM]; {Time_i3}, after hearing "{T_i3}", the user selected the "Cancel" option, [Behavior feedback: CANCEL].

[0042] Based on the above experience, determine and directly respond to the action that best matches the user's current intent: If there is only one clear intention, use the "You want to..." format; If there are two or more possibilities, use the rhetorical question format: "Would you like to... or...?" In the above preset template, {T_cur} represents the current speech recognition text, {Time_i#} represents the timestamp of the #th historical interaction, {T_i#} represents the historical speech recognition text, {T_i#} represents the } represents the historical speech correction text (i.e. the text corrected by the user), [Execution: {OK / FAIL}] indicates whether the system was successfully executed at that time, and [Behavior Feedback: ...] indicates the feedback given by the user through buttons / gestures, etc.

[0043] Furthermore, it is recommended that the total number of characters in the generated concatenated information should not exceed 300 characters to ensure the response delay of the subsequent voice model. That is, after the concatenated information is generated, it is detected whether the number of characters in the concatenated information exceeds 300 characters. If it exceeds, the concatenated information is simplified to increase the speed of generating subsequent voice interaction results and reduce the response delay. For example, the part of the preset template that selects "direct answer" or "reverse question confirmation" based on a single or multiple candidate intents can be simplified.

[0044] Step S140: inferring the splicing information to generate a voice interaction result.

[0045] The pre-set language model is used to infer the spliced ​​information and generate a voice interaction result. Specifically, the spliced ​​information is input into the pre-set language model to obtain the voice interaction result output by the pre-set language model.

[0046] Among them, the preset language model is a large language model (Large Language Model, LLM for short), such as GPT (Generative Pre-trained Transformer), LLaMA (Large LanguageModel Meta AI‌), Gemma model, etc.

[0047] For example, the current speech recognition text is "Navigate to Wanxiang City". Based on the acquired historical speech similarity information, splicing is performed according to the preset template to obtain the following splicing information: Currently heard: "Navigate to Wanxiang City."

[0048] Past interactions with similar accents are as follows: [2024-11-25T10:16:05], after hearing "Navigate to MixC", the user clicked the "Confirm" button. [Behavior feedback: CONFIRM]; [2025-01-06T10:15:30], after hearing "Navigate to Wanxi City", the user corrected it to "Navigate to Wanxiang City", [Execute: OK]; [2025-1-21T10:14:10], after hearing "Navigate to MixC", the user selected the "Cancel" option, [Behavior feedback: CANCEL].

[0049] Based on the above experience, determine and directly respond to the action that best matches the user's current intent. If there is only one clear intent, the response format is: Do you want to...? If there are multiple possibilities, do not output the aforementioned response and instead ask a rhetorical question in the format of: Do you want to... or...?

[0050] Correspondingly, the voice interaction result is: Do you want to start navigating to "Vanke Mall" or cancel navigation? For example, the current speech recognition text is "play for seven days". Based on the acquired historical speech similarity information, splicing is performed according to the preset template to obtain the following splicing information: Currently heard: "Play for seven days."

[0051] Past interactions with similar accents are as follows: [2025-03-09T09:30:12], after hearing "Play for seven days", the user corrected it to "Play sunny days", [Execute: OK]; [2025-01-02T09:28:50], after hearing "Play Sunny Day", the user clicked the "Confirm" button, [Behavior Feedback: CONFIRM]; [2024-11-05T09:27:20], after hearing "Play please listen", the system prompts the option, and the user skips by swiping left, [Behavior feedback: SWIPE LEFT].

[0052] Based on the above experience, determine and directly respond to the action that best matches the user's current intent: If there is only one clear intention, use the "You want to..." format; If there are two or more possibilities, use the rhetorical question format: "Would you like to... or...?" Correspondingly, the voice interaction result is: You want to play Jay Chou's "Sunny Day".

[0053] The voice interaction method provided by the present invention obtains a speech to be recognized and its current speech recognition text; retrieves historical speech similarity information based on the speech to be recognized and the current speech recognition text; concatenates the current speech recognition text with the historical speech similarity information to obtain concatenated information; and infers the concatenated information to generate a speech interaction result. In the present invention, by utilizing the speech to be recognized and the current speech recognition text to retrieve historical interaction experience and using this historical interaction experience as a real-time calibration source, the accuracy of the speech interaction results can be significantly improved, thereby improving the interaction success rate and user experience. Furthermore, by intelligently concatenating the current speech recognition text and the historical speech similarity information, complete and formatted concatenated information is constructed as inference input to generate the speech interaction result. This approach avoids retraining complex deep learning models, requiring only low-complexity, lightweight inference to generate high-quality speech interaction results. This significantly reduces computing resource consumption and response latency, significantly improving the interaction success rate and robustness in complex scenarios.

[0054] Figure 3 This is the second flow chart of the voice interaction method provided by the present invention, such as Figure 3 As shown, step S120 includes: step S121, step S122, step S123, step S124 and step S125.

[0055] It should be noted that the execution order of step S122 and step S123 is not particular.

[0056] Step S121 , generating a current semantic vector based on the speech to be recognized and the current speech recognition text, and generating a current pronunciation vector based on the current speech recognition text.

[0057] In this embodiment, a semantic vector is generated based on the speech to be recognized and the current speech recognition text, which is recorded as the current semantic vector , and generate a pronunciation vector based on the current speech recognition text, recorded as the current pronunciation vector .

[0058] The generation process of the current semantic vector is as follows: the recognized speech is subjected to noise suppression, echo cancellation, pre-emphasis, endpoint detection and frame windowing processing in turn, and the Mel spectrum features of each frame of speech data are calculated; the Mel spectrum features are spliced ​​to generate a speech feature matrix; the current speech recognition text is segmented, and the segmentation results are mapped to the pre-trained vocabulary to obtain a word unit sequence; the speech feature matrix and the word unit sequence are projected into the same hidden dimension space to obtain a speech embedding sequence and a text embedding sequence; the speech embedding sequence and the text embedding sequence are fused to obtain a fused embedding sequence; the fused embedding sequence is encoded to obtain the current semantic vector.

[0059] The generation process of the current pronunciation vector is as follows: the current speech recognition text is converted into a sequence of pronunciation symbols; through a preset embedding lookup table, each pronunciation symbol in the pronunciation symbol sequence is mapped to a pronunciation embedding vector of a preset dimension; the pronunciation embedding vector is average pooled to generate the current pronunciation vector.

[0060] Step S122: According to the current semantic vector, a first historical similar speech is retrieved from a preset experience library, and a first similarity is obtained.

[0061] Step S123 : According to the current pronunciation vector, a second historical similar speech is retrieved from the preset experience library, and a second similarity is obtained.

[0062] As an implementation, the RAG (Retrieval-augmented Generation) reasoning framework based on experience vectors can be used during retrieval. This technology significantly enhances the system's ability to capture the latest knowledge. It dynamically retrieves historical speech similarity information related to the current input from a preset experience library and uses it in the generation of speech interaction results, thereby improving the accuracy of semantic interaction results.

[0063] As an implementation method, a dual-channel search indexing method may be used during retrieval to improve retrieval efficiency and retrieval accuracy.

[0064] Among them, in the "Semantic Channel Index", the current semantic vector For a query, the top K most similar speech recordings (those with the highest semantic similarity) are retrieved and recorded as the first historical similar speech recordings. The corresponding similarity scores are then obtained, also known as the first similarity. The first similarity includes the first semantic similarity and the first pronunciation similarity. The first semantic similarity can be retrieved from the "Semantic Channel Index," while the first pronunciation similarity can be retrieved from the "Pronunciation Channel Index" by searching the first historical similar speech recordings.

[0065] In the "Voice Channel Index", the current pronunciation vector For the query, the top K most similar speech recordings (those with the greatest pronunciation similarity) are retrieved and recorded as the second historical similar speech recordings. The corresponding similarity scores are then obtained, also known as the second similarity. The second similarity includes the second semantic similarity and the second pronunciation similarity. The second pronunciation similarity can be retrieved from the "pronunciation channel index," while the second semantic similarity can be retrieved from the "semantic channel index" by searching the second historical similar speech recordings.

[0066] Step S124 : determining a target historical similar speech according to the first historical similar speech, the first similarity, the second historical similar speech, and the second similarity.

[0067] Then, the similarities obtained from the two retrievals (i.e., the first similarity and the second similarity) are normalized to the interval [0, 1] respectively to make them comparable. Then, the target historical similar speech is determined based on the normalized first similarity and the normalized second similarity.

[0068] Specifically, the first similarity is normalized to obtain a third similarity, and the second similarity is normalized to obtain a fourth similarity; wherein the third similarity includes a third semantic similarity and a third pronunciation similarity, and the fourth similarity includes a fourth semantic similarity and a fourth pronunciation similarity; then, according to a preset weight, the third semantic similarity and the third pronunciation similarity are weighted and summed to obtain a first fusion similarity, and according to a preset weight, the fourth semantic similarity and the fourth pronunciation similarity are weighted and summed to obtain a second fusion similarity; wherein the preset weight includes a preset semantic weight and a preset pronunciation weight, and the preset semantic weight is greater than the preset pronunciation weight. For example, the preset semantic weight can be set to 70%, and the preset pronunciation weight can be set to 30%. Finally, based on the first fusion similarity and the second fusion similarity, the target historical similar speech is screened from the first historical similar speech and the second historical similar speech. wherein the target historical similar speech can include one or more.

[0069] Step S125 , obtaining historical speech similarity information corresponding to the target historical similar speech from the preset experience database.

[0070] Acquire historical speech similarity information corresponding to the target historical similar speech from a preset experience database, wherein the historical speech similarity information includes historical speech recognition text and at least one of a timestamp, historical speech correction text, feedback result, behavioral feedback action, and system execution status.

[0071] In this embodiment, the current semantic vector and current pronunciation vector are generated based on the speech to be recognized and the current speech recognition text. Then, during retrieval, the "semantic channel index" and "pronunciation channel index" are used in parallel to identify historically similar speech from both semantic and pronunciation dimensions. This approach ensures the accuracy of retrieval results and, in turn, improves the accuracy of voice interaction results.

[0072] Further, based on the above embodiment, the step of "generating a current semantic vector based on the speech to be recognized and the current speech recognition text" includes: step S1211, step S1212, step S1213, step S1214, step S1215 and step S1216.

[0073] It should be noted that the execution order of steps S1211-S1212 and step S1213 is not specific.

[0074] Step S1211 , performing noise suppression, echo cancellation, pre-emphasis, endpoint detection, and frame windowing processing on the speech to be recognized in sequence, and calculating the Mel spectrum features of each frame of speech data.

[0075] Step S1212: concatenate the mel-spectrogram features to generate a speech feature matrix.

[0076] Noise suppression uses digital signal processing to isolate and attenuate background noise (such as wind and keyboard sounds). Spectral subtraction is used to suppress ambient noise. By performing noise suppression on the speech being recognized, the signal-to-noise ratio (SNR) is improved, thereby enhancing the quality of the speech signal being recognized.

[0077] Echo cancellation is an adaptive filtering technology used to eliminate feedback (i.e., echo) from the speaker's playback content in the microphone's captured signal. This can further improve the quality of the speech signal being recognized.

[0078] Pre-emphasis is a signal preprocessing technique that compensates for the high-frequency attenuation caused by the speech signal being radiated through the lips. This is achieved through a first-order high-pass filter. This compensates for the high-frequency components of speech, improving consonant clarity while also balancing spectral energy and stabilizing the calculation of Mel-spectral features.

[0079] Endpoint detection accurately determines the start and end points of speech segments by analyzing the time and frequency domain characteristics of speech signals. This eliminates silent segments and non-speech segments (such as breathing), reducing inefficient computation and saving power. This is crucial for XR devices. It also prevents non-speech segments from contaminating the Mel-spectrum signature.

[0080] Frame windowing is a preprocessing technique that converts continuous speech signals into short, stationary sequences. This process converts non-stationary speech into quasi-stationary segments, eliminating spectral leakage caused by frame edge truncation. Specifically, frames are divided into 25ms frames with a 10ms frame shift, and a Hamming window is applied to suppress spectral leakage.

[0081] After the above processing, the Mel spectrum features of each frame of speech data can be calculated by the 40-dimensional Mel filter bank, and then spliced ​​into a feature matrix of length T , The matrix M is a real matrix with a dimension of T × 40. T is the time dimension, which is the total number of frames of speech data obtained after frame segmentation and windowing, determined by the speech recognition, frame length, and frame shift of the speech to be recognized, and 40 is the feature dimension.

[0082] Step S1213: segment the current speech recognition text, map the segmentation results to a pre-trained vocabulary, and obtain a word-unit sequence.

[0083] While generating a speech feature matrix based on the speech to be recognized, the current speech recognition text is segmented, and the segmentation results are mapped to a pre-trained vocabulary to obtain a token sequence.

[0084] The pre-trained vocabulary is a pre-trained BERT (Bidirectional Encoder Representations from Transformers) vocabulary. Mapping to the pre-trained BERT vocabulary ensures that speech and text embedding sequences are projected into the same semantic space, facilitating multimodal fusion processing.

[0085] Assume that the length of the token sequence is L, and record the token sequence as .

[0086] Step S1214: Project the speech feature matrix and the word unit sequence into the same hidden dimension space to obtain a speech embedding sequence and a text embedding sequence.

[0087] Specifically, the speech feature matrix is ​​transformed into Projected to the same hidden dimension d as the text embedding sequence, we get the speech embedding sequence as follows: ; Where A represents the speech embedding sequence, represents the time dimension after convolution, d represents the hidden dimension, represents one-dimensional convolution, Represents a linear transformation.

[0088] At the same time, the embedding layer of the pre-trained BERT will be Project to , to align with the dimensions of the speech embedding sequence.

[0089] Step S1215: Fusing the speech embedding sequence with the text embedding sequence to obtain a fused embedding sequence.

[0090] Then, the speech embedding sequence A is fused with the text embedding sequence W to obtain the fused embedding sequence , as follows: .

[0091] Step S1216: Encode the fused embedding sequence to obtain a current semantic vector.

[0092] When encoding, a multimodal Transformer layer can be constructed. The input is the above fused embedding sequence. After N layers of self-attention and feedforward networks, the output of the first position (or the specially inserted [CLS] position) is taken as the overall semantic representation, which is recorded as the current semantic vector:

[0093] in, Represents the overall semantic representation, and semantic is the current semantic vector.

[0094] By leveraging Transformerr’s self-attention mechanism to dynamically learn cross-modal dependencies, error self-correction can be achieved.

[0095] Furthermore, we can first perform contrastive learning or mask prediction tasks on large-scale speech-text pairs, so that Simultaneously encodes semantic and phonetic features.

[0096] Furthermore, the positive examples (i.e. successful interactions) in the preset experience library can be combined Perform lightweight fine-tuning, such as using LoRA (Low-Rank Adaptation) technology to adjust only some weights of the multimodal Transformer to gradually adapt to the user's accent.

[0097] In this embodiment, the speech to be recognized is first processed to generate a speech feature matrix, and the current speech recognition text is processed at the same time to obtain a word unit sequence; then, the speech feature matrix and the word unit sequence are projected into the same hidden dimension space to obtain a speech embedding sequence and a text embedding sequence, which are fused to obtain a fused embedding sequence. Through the above same-dimensional projection, multimodal fusion of speech and text can be achieved. The fused embedding sequence is then encoded to obtain the current semantic vector, realizing cross-modal semantic encoding. In this way, the heterogeneity problem of multimodal data in XR scenarios is solved, providing core technical support for low-latency, high-robustness interaction.

[0098] Further, based on the above embodiment, the step of “generating a current pronunciation vector according to the current speech recognition text” includes: step S1217, step S1218 and step S1219.

[0099] Step S1217: convert the current speech recognition text into a pronunciation symbol sequence.

[0100] Since the current speech recognition text may be in different language types, the pronunciation symbols may include phoneme symbols, syllable symbols or pinyin symbols.

[0101] As an implementation method, the current speech recognition text can be converted through an acoustic prediction network.

[0102] For example, if the current speech recognition text is in Chinese, the current speech recognition text can be converted from Chinese characters to pinyin to obtain a pinyin symbol sequence, which is recorded as .

[0103] Step S1218: Mapping each pronunciation symbol in the pronunciation symbol sequence to a pronunciation embedding vector of a preset dimension through a preset embedding lookup table.

[0104] Step S1219: perform average pooling on the pronunciation embedding vector to generate a current pronunciation vector.

[0105] Here, the preset embedding lookup table is a mapping matrix from discrete symbols to continuous vectors, denoted as E∈R V×m , where V represents the size of the phonetic symbol vocabulary and m represents the embedding dimension.

[0106] Through the preset embedding lookup table, each pronunciation symbol in the pronunciation symbol sequence is mapped to a pronunciation embedding vector of preset dimension m, which is recorded as .

[0107] Then, the pronunciation embedding vector Perform average pooling to obtain the current pronunciation vector, as follows: ; Among them, P represents the current pronunciation vector and K represents the length of the pronunciation symbol sequence.

[0108] Furthermore, as the number of user-corrected pinyin-text pairs in the preset experience library continues to accumulate, the preset embedding lookup table or acoustic prediction network can be fine-tuned to make the pronunciation vector more accurately reflect the user's accent.

[0109] In this embodiment, the current speech recognition text is converted into a sequence of pronunciation symbols to directly capture the essence of pronunciation. This method is naturally robust to homophones. At the same time, for users with different dialects or accents, the pronunciation symbols can more accurately reflect the actual pronunciation. Then, through a preset embedding lookup table, each pronunciation symbol in the pronunciation symbol sequence is mapped to a pronunciation embedding vector of a preset dimension; then, the pronunciation embedding vector is average pooled to generate the current pronunciation vector. In the above manner, the current pronunciation vector is generated for subsequent retrieval of historical similar speech based on the pronunciation dimension, which can improve the accuracy of the historical similar speech retrieval results, and thus improve the accuracy of the speech interaction results.

[0110] Further, based on the above embodiment, step S124 includes: step S1241, step S1242, step S1243 and step S1244.

[0111] It should be noted that the execution order of step S1242 and step S1243 is not specific.

[0112] Step S1241 , normalizing the first similarity to obtain a third similarity, and normalizing the second similarity to obtain a fourth similarity; wherein the third similarity includes a third semantic similarity and a third pronunciation similarity, and the fourth similarity includes a fourth semantic similarity and a fourth pronunciation similarity.

[0113] The first similarity is normalized to the interval [0, 1] to obtain a third similarity, and the second similarity is normalized to the interval [0, 1] to obtain a fourth similarity; wherein the third similarity includes a third semantic similarity and a third pronunciation similarity, and the fourth similarity includes a fourth semantic similarity and a fourth pronunciation similarity.

[0114] Through normalization processing, the third similarity and the fourth similarity can be made comparable.

[0115] Step S1242 : performing weighted summation on the third semantic similarity and the third pronunciation similarity according to preset weights to obtain a first fusion similarity.

[0116] Here, the preset weight includes a preset semantic weight and a preset pronunciation weight, and the preset semantic weight>the preset pronunciation weight. For example, the preset semantic weight can be set to 70%, and the preset pronunciation weight can be set to 30%.

[0117] According to the preset semantic weight and the preset pronunciation weight, the third semantic similarity and the third pronunciation similarity are weighted and summed to obtain a fused similarity, which is recorded as the first fused similarity.

[0118] Specifically, the first fusion similarity = preset semantic weight The third semantic similarity + preset pronunciation weight Third, pronunciation similarity.

[0119] Step S1243 : performing weighted summation on the fourth semantic similarity and the fourth pronunciation similarity according to the preset weight to obtain a second fusion similarity.

[0120] According to the preset semantic weight and the preset pronunciation weight, the fourth semantic similarity and the fourth pronunciation similarity are weighted and summed to obtain a fused similarity, which is recorded as the second fused similarity.

[0121] Specifically, the second fusion similarity = preset semantic weight Fourth semantic similarity + preset pronunciation weight Fourth, pronunciation similarity.

[0122] Step S1244 : Filter and obtain a target historical similar speech from the first historical similar speech and the second historical similar speech according to the first fusion similarity and the second fusion similarity.

[0123] After calculating the first fusion similarity and the second fusion similarity, the first fusion similarity and the second fusion similarity are sorted in descending order, and a preset number N (usually N≤3) of fusion similarities are taken as target fusion similarities according to the sorting results. Then, based on the target fusion similarity, the target historical similar speech is screened from the first historical similar speech and the second historical similar speech to obtain the target historical similar speech.

[0124] In this embodiment, by using "semantic vector retrieval" and "pronunciation vector retrieval" in parallel, comparing the two retrieval results and weighted fusion, so as to take into account the differences in user accents and semantic consistency, the retrieval and reasoning accuracy in special accent scenarios can be significantly improved.

[0125] Figure 4 This is the third flow chart of the voice interaction method provided by the present invention, such as Figure 4 As shown, after the above step S140, the voice interaction method may further include: step S150, step S160 and step S170.

[0126] Step S150: outputting an execution operation according to the voice interaction result and obtaining interaction feedback information.

[0127] After the voice interaction result is generated, the operation is performed according to the output of the voice interaction result. Specifically, if the voice interaction result only involves a single intent, the relevant application is directly called according to the voice interaction result, and the corresponding operation is performed. For example, in the above example, the voice interaction result is: "You want to play Jay Chou's "Sunny Day", then you can directly call the music player to play Jay Chou's "Sunny Day". If the voice interaction result only involves multiple intentions, the relevant application is called according to the voice interaction result, and the confirmation operation interface is displayed. For example, in the above example, the voice interaction result is: "Do you want to start navigating to "Vanke Mall" or cancel navigation? " At this time, the map SDK (Software Development Kit) is called and the UI (User Interface) button is displayed for the user to select and confirm.

[0128] Then, the user's feedback on the system's operations is monitored to obtain interactive feedback information. User feedback includes but is not limited to: 1) clicking "Confirm / Cancel"; 2) manually entering or speaking the corrected text; 3) giving up the execution.

[0129] Step S160: Generate a feedback tag according to the interactive feedback information.

[0130] After obtaining the interactive feedback information, a feedback tag is generated according to the interactive feedback information.

[0131] If the type of the interactive feedback information is voice feedback, semantic recognition is performed on the interactive feedback information, and a semantic feedback label is determined according to the semantic recognition result.

[0132] If the type of the interaction feedback information is behavior feedback, the interaction feedback information is matched with a preset mapping relationship between the behavior type and the behavior feedback label to obtain the behavior feedback label.

[0133] Furthermore, in addition to obtaining user interaction feedback information, system execution feedback results can also be obtained, and system execution tags can be determined based on the system execution feedback results. System execution tags include but are not limited to: system execution status, system execution failure type, and remark information of system execution failure.

[0134] Step S170: storing the current speech recognition text, the current semantic vector, the current pronunciation vector and the feedback label in the preset experience library.

[0135] The current speech recognition text, current semantic vector, current pronunciation vector and feedback label are stored in a preset experience library in a preset format for subsequent optimization.

[0136] Of course, it can be understood that in actual applications, in addition to the above information, other relevant information, such as the record ID (Identity document), the timestamp of the voice to be recognized, the system execution status, the error code, the voice recognition model version used, the LLM model version used, the number of uses and the most recent use time, can also be stored in the preset experience library.

[0137] As an implementation method, the data structure written into the preset experience database is shown as follows: { "record_id": "uuid-xxxxx", / / unique record ID "user_id": "user_1023", / / Optional: User ID "timestamp": "2025-05-14T20:32:10Z", / / ISO 8601 timestamp / / === Embedding vector part === "semantic_embedding": [0.32, -0.11, …], / / semantic embedding "pronunciation_embedding": [0.12, 0.04, …], / / pronunciation vector / / === Text content === "asr_text": "Play Jay Chou's Seven Days", / / Voice recognition text "final_text": "Play Jay Chou's Sunny Day", / / Voice correction text (if any) / / === Behavior Feedback === "feedback": { "result": "FAIL", / / success / failure / cancellation / termination, etc. "type": "USER_SPEECH", / / semantic feedback / behavioral feedback / interface feedback "subtype": "NEGATION", / / Semantic feedback subtype: NEGATION ("Not this"), CONFIRMATION ("OK, thank you"), etc. / Interface feedback subtype: MAP_API, etc. "action": "VOICE_REPEAT" / / Behavior feedback action: VOICE_REPEAT, CONFIRM_BUTTON, SWIPE_LEFT, etc. }, / / === Execution Status === "execution": { "status": "OK", / / System execution status: OK / ERROR / TIMEOUT "error_code": null, / / If the execution fails, you can fill in the interface exception code "notes": null / / Notes, such as "There is no such place on the map" }, / / === Meta Information and Auditing === "source": { "asr_engine": "CloudASR-v2.1", / / ASR engine version used "llm_model": "ChatLLM-3B", / / LLM model version used for inference "retrieval_topk": 4 / / Number of candidates for this retrieval }, "usage_count": 1, / / The number of times this experience has been retrieved / used "last_used": "2025-05-14T20:45:03Z" / / The time when it was last used as a prompt word } The types and meanings of the various fields in the data structure are shown in Table 1 below.

[0138] Table 1 Field Description Table

[0139] In the data structure above, user semantic feedback and behavioral feedback are stored together in the feedback field, making them easier to distinguish and process later. The execution field records system call results, supporting performance monitoring of the interface or model itself. The source and timestamp fields facilitate tracking the origin and timeliness of each experience.

[0140] In this embodiment, feedback tags are generated based on the user's interactive feedback information, and the current speech recognition text, current semantic vector, current pronunciation vector, and feedback tags are then stored in a preset experience library in a preset format. Through this approach, after each speech interaction, the current semantic vector and current pronunciation vector are used to form an "experience vector," which is then stored in the vectorized experience library along with the current speech recognition text and the feedback results of that interaction. This creates a complete closed-loop learning path and experience accumulation, facilitating the subsequent generation of accurate speech interaction results based on the historical speech information in the preset experience library.

[0141] Furthermore, in one embodiment, the above step S160 includes step S161, step S162 and step S163.

[0142] Step S161: If the type of the interactive feedback information is voice feedback, semantic recognition is performed on the interactive feedback information, and a semantic feedback tag is determined according to the semantic recognition result.

[0143] If the type of the interactive feedback information is voice feedback, semantic recognition is performed on the interactive feedback information.

[0144] As an implementation method, semantic recognition can be performed through an NLP (Natural Language Processing) model.

[0145] Then, a semantic feedback label is determined based on the semantic recognition result, wherein the semantic feedback label includes the semantic feedback result and the semantic feedback subtype.

[0146] If the speech recognition result is determined to be an "unsatisfactory" result, for example, the user says "not this", "you said it wrong", etc., then the semantic feedback result is determined to be a failure.

[0147] If the speech recognition result is determined to be "unsatisfactory," the user's speech is further acquired, and the speech interaction method of the present invention is continued to re-engage the interaction. If the semantic feedback result after the interaction is successful is obtained, the speech recognition text of the newly acquired user's speech is used as the speech correction text.

[0148] If the speech recognition result is determined to be a "satisfactory" result, for example, the user says "good, thank you", "OK", "ok", etc., then the semantic feedback result is determined to be successful.

[0149] If the speech recognition result is determined to be a "re-speak" result, for example, the speech recognition result is consistent with the current speech recognition text, or the similarity is greater than a preset value, it is determined that the user is repeating the same voice command. At this time, the semantic feedback result is determined to be a failure.

[0150] Furthermore, when the voice feedback tag is written into the preset experience library, the feedback.result, feedback.type, and feedback.subtype fields may be updated.

[0151] For example, if the user feedback says "not this", the corresponding writing logic of the preset experience library is: feedback.result=FAIL, type=USER_SPEECH / subtype=NEGATION. The above "failure + user correction" record can facilitate the avoidance of similar misidentifications in subsequent searches.

[0152] Step S162: If the type of the interaction feedback information is behavior feedback, the interaction feedback information is matched with a preset mapping relationship between behavior types and behavior feedback labels to obtain a behavior feedback label.

[0153] If the interaction feedback information is of the behavioral feedback type, the interaction feedback information is matched against the pre-set mapping between the behavior type and the behavioral feedback tag to obtain the behavioral feedback tag. Behavioral feedback types may include, but are not limited to, system behavior interruption and user confirmation. The behavioral feedback tag includes the behavioral feedback result and the behavioral feedback action.

[0154] For example, if the user manually exits or changes routes during navigation through application state change events / interruption identification, the behavior feedback result is determined to be a failure. If no task interaction feedback information is obtained within a preset time (e.g., 3 seconds), the behavior feedback result is determined to be a success.

[0155] For example, when the user clicks the "Confirm" button to continue execution through UI event monitoring, the behavior feedback result is determined to be successful.

[0156] Furthermore, when the voice feedback tag is written into the preset experience library, the feedback.result and feedback.action fields may be updated.

[0157] For example, the system prompts "Navigate to MixC?" and the user clicks the "Confirm" button to continue. Correspondingly, the preset experience database write logic is: feedback.result=OK, action=CONFIRM_BUTTON.

[0158] Step S163: If the type of the interaction feedback information is interface feedback, the interaction feedback information is matched with a preset mapping relationship between the interface feedback type and the interface feedback label to obtain an interface feedback label.

[0159] Considering the possibility of system execution failures, such as underlying execution failures in navigation and query requests, the system monitors the failure signals in the system response chain. Correspondingly, if the interactive feedback information is of the interface feedback type, the interactive feedback information is matched with the preset mapping between the interface feedback type and the interface feedback tag to obtain the interface feedback tag. The interface feedback tag includes the interface feedback subtype, system execution status, and error code.

[0160] For example, if the map API request fails or the interface does not return, the interface feedback subtype is determined to be MAP_API (map interface), the system execution status is ERROR (error), and the error code is MAP_TIMEOUT (map interface timeout).

[0161] Furthermore, when the interface feedback tag is written into the preset experience library, the feedback.type, feedback.subtype, execution.status, and execution.error_code fields may be updated.

[0162] For example, in a navigation scenario, the speech recognition text is correct, but the map interface times out and becomes unresponsive, meaning the map call fails and the underlying layer returns an error. In this case, write the following logic: feedback.type=API_ERROR / subtype=MAP_API, execution.status=ERROR / error_code=MAP_TIMEOUT, for subsequent interface stability analysis and optimization tips.

[0163] In this embodiment, user feedback is subdivided into "semantic feedback" (such as denial, confirmation, and repetition) and "behavioral feedback" (such as button clicks, gesture operations, and long stays), as well as "API / system link feedback" (such as interface timeouts and LLM abnormal responses), and feedback tags (including feedback results, feedback types, feedback subtypes, behavioral feedback actions, system execution status, and error codes) are structured and stored to support refined calls and model optimization during subsequent retrieval.

[0164] Furthermore, in one embodiment, after the above step S160, the voice interaction method further includes: Step S180: If the feedback tag is determined to be positive feedback, the number of times the target historical similar voice is used and the most recent use time in the preset experience database are updated.

[0165] If the feedback tag is positive, the usage count (usage_count) and last usage time (last_used) of the target similar voice in the preset experience database are updated. Specifically, the usage count is increased by 1, and the last usage time is adjusted to the time of the current search and use.

[0166] In this embodiment, by updating the usage count and the most recent usage time of only the samples with positive feedback for subsequent model updates, it is possible to prevent the accumulation of erroneous samples from negatively impacting the model.

[0167] Step S190: regularly clearing the preset experience database according to the updated usage count and the updated most recent usage time.

[0168] Then, the preset experience library is cleaned up regularly based on the updated usage count and the updated most recent usage time.

[0169] In this embodiment, template samples that have not been used for a long time are deleted based on the updated number of uses and the updated most recent use time, so as to eliminate historical voice information that has not been used for a long time, improve the quality of historical voice samples in the preset experience library, and further prevent the accumulation of erroneous samples from having a negative impact on the model.

[0170] The following uses XR glasses as an example to illustrate the application scenarios of the voice interaction method provided by the present invention.

[0171] For example, in the in-vehicle voice interaction scenario, the driver issues commands such as "play XX song", "switch to XX radio station" or "navigate to XX" in his or her own accent while driving. XR glasses (or head-up display) combine retrieval experience to achieve accurate recognition and low-latency feedback, thereby improving driving safety.

[0172] For example, for voice home control scenarios, when speaking dialects or with a heavy accent at home, users can say "turn on the living room lights" or "raise the air conditioner temperature", etc. The system will adapt to the dialect pronunciation and accurately identify and control smart home devices.

[0173] For example, in scenarios where users with disabilities are assisted in interacting, when they stutter, have unclear pronunciation, or have language barriers, they can speak commands (such as "call Mom") through XR glasses. The system, combined with historical error correction experience, can always correctly understand and execute them, while displaying status feedback through the glasses.

[0174] For example, for industrial site operation scenarios in noisy environments, workers wearing XR glasses can issue voice commands such as "start device A" and "record temperature sensor readings" in dialects with different accents in the high-noise environment of factory workshops or construction sites. The system can quickly recognize and execute them, and supports audio-text retrieval with adaptive accents.

[0175] For example, in multilingual learning and translation scenarios, language learners wear XR glasses to imitate pronunciation, and the system corrects their pronunciation errors in real time and provides examples; in cross-language communication scenarios, the accent adaptation mechanism is used to accurately identify non-native pronunciations and instantly translate them.

[0176] The voice interaction device provided by the present invention is described below. The voice interaction device described below and the voice interaction method described above can be referenced to each other.

[0177] Figure 5 This is a schematic diagram of the structure of the voice interaction device provided by the present invention. Figure 5 As shown, the device includes an acquisition module 510, a retrieval module 520, a splicing module 530 and an interaction module 540; wherein: An acquisition module 510 is used to acquire the speech to be recognized and the current speech recognition text; A retrieval module 520 is configured to retrieve historical speech similarity information based on the speech to be recognized and the current speech recognition text; A splicing module 530 is configured to splice the current speech recognition text and the historical speech similarity information to obtain splicing information; The interaction module 540 is configured to infer the splicing information and generate a voice interaction result.

[0178] The voice interaction device provided by the present invention obtains a speech to be recognized and its current speech recognition text; retrieves historical speech similarity information based on the speech to be recognized and the current speech recognition text; concatenates the current speech recognition text with the historical speech similarity information to obtain concatenated information; and infers the concatenated information to generate a speech interaction result. In the present invention, by utilizing the speech to be recognized and the current speech recognition text to retrieve historical interaction experience and using this historical interaction experience as a real-time calibration source, the accuracy of the speech interaction results can be significantly improved, thereby enhancing the interaction success rate and user experience. Furthermore, by intelligently concatenating the current speech recognition text with the historical speech similarity information, complete and formatted concatenated information is constructed as inference input to generate the speech interaction result. This approach avoids retraining complex deep learning models, requiring only low-complexity, lightweight inference to generate high-quality speech interaction results. This significantly reduces computing resource consumption and response latency, significantly improving the interaction success rate and robustness in complex scenarios.

[0179] It should be noted here that the above-mentioned voice interaction device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned voice interaction method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0180] Figure 6 An example of a physical structure diagram of an XR device is shown below. Figure 6 As shown, the XR device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may invoke logic instructions in the memory 630 to execute a voice interaction method, which includes: obtaining a speech to be recognized and its current speech recognition text; retrieving historical speech similarity information based on the speech to be recognized and the current speech recognition text; concatenating the current speech recognition text with the historical speech similarity information to obtain concatenated information; and inferring the concatenated information to generate a voice interaction result.

[0181] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0182] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the voice interaction method provided by the above methods, which includes: obtaining the voice to be recognized and its current voice recognition text; retrieving historical voice similarity information based on the voice to be recognized and the current voice recognition text; splicing the current voice recognition text and the historical voice similarity information to obtain splicing information; and reasoning on the splicing information to generate a voice interaction result.

[0183] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the voice interaction method provided by the above-mentioned methods, the method including: obtaining the voice to be recognized and its current voice recognition text; retrieving historical voice similarity information based on the voice to be recognized and the current voice recognition text; splicing the current voice recognition text and the historical voice similarity information to obtain splicing information; and reasoning on the splicing information to generate a voice interaction result.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0185] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A voice interaction method, characterized in that: include: Get the speech to be recognized and its current speech recognition text; Retrieving historical speech similarity information based on the speech to be recognized and the current speech recognition text; Splicing the current speech recognition text and the historical speech similarity information to obtain splicing information; The splicing information is inferred to generate a voice interaction result.

2. The voice interaction method according to claim 1, characterized in that: The retrieving historical speech similarity information based on the speech to be recognized and the current speech recognition text includes: Generate a current semantic vector based on the speech to be recognized and the current speech recognition text, and generate a current pronunciation vector based on the current speech recognition text; Retrieving a first historical similar speech from a preset experience database according to the current semantic vector and obtaining a first similarity; Retrieving a second historical similar speech from the preset experience library according to the current pronunciation vector, and obtaining a second similarity; determining a target historical similar speech according to the first historical similar speech, the first similarity, the second historical similar speech, and the second similarity; The historical speech similarity information corresponding to the target historical similar speech is obtained from the preset experience database.

3. The voice interaction method according to claim 2, characterized in that: The generating a current semantic vector according to the speech to be recognized and the current speech recognition text includes: The speech to be recognized is subjected to noise suppression, echo cancellation, pre-emphasis, endpoint detection, and frame windowing processing in sequence, and the Mel spectrum features of each frame of speech data are calculated; Splicing the Mel-spectrogram features to generate a speech feature matrix; Segmenting the current speech recognition text, mapping the segmentation results to a pre-trained vocabulary to obtain a word unit sequence; Projecting the speech feature matrix and the word unit sequence into the same hidden dimension space to obtain a speech embedding sequence and a text embedding sequence; Fusing the speech embedding sequence with the text embedding sequence to obtain a fused embedding sequence; The fused embedding sequence is encoded to obtain a current semantic vector.

4. The voice interaction method according to claim 2, wherein: Generating a current pronunciation vector according to the current speech recognition text includes: Converting the current speech recognition text into a pronunciation symbol sequence; Mapping each pronunciation symbol in the pronunciation symbol sequence to a pronunciation embedding vector of a preset dimension through a preset embedding lookup table; Average pooling is performed on the pronunciation embedding vector to generate a current pronunciation vector.

5. The voice interaction method according to claim 2, wherein: The step of determining a target historical similar speech according to the first historical similar speech, the first similarity, the second historical similar speech, and the second similarity includes: Normalizing the first similarity to obtain a third similarity, and normalizing the second similarity to obtain a fourth similarity; wherein the third similarity includes a third semantic similarity and a third pronunciation similarity, and the fourth similarity includes a fourth semantic similarity and a fourth pronunciation similarity; performing weighted summation on the third semantic similarity and the third pronunciation similarity according to a preset weight to obtain a first fused similarity; performing a weighted summation of the fourth semantic similarity and the fourth pronunciation similarity according to the preset weight to obtain a second fused similarity; According to the first fusion similarity and the second fusion similarity, a target historical similar speech is screened from the first historical similar speech and the second historical similar speech.

6. The voice interaction method according to any one of claims 2 to 5, characterized in that: After the splicing information is inferred and a voice interaction result is generated, the method further includes: Outputting an execution operation according to the voice interaction result and obtaining interaction feedback information; generating a feedback tag according to the interactive feedback information; The current speech recognition text, the current semantic vector, the current pronunciation vector and the feedback label are stored in the preset experience library.

7. The voice interaction method according to claim 6, characterized in that: Generating a feedback tag according to the interactive feedback information includes: If the type of the interactive feedback information is voice feedback, performing semantic recognition on the interactive feedback information, and determining a semantic feedback label according to the semantic recognition result; If the type of the interaction feedback information is behavior feedback, matching the interaction feedback information with a preset mapping relationship between the behavior type and the behavior feedback label to obtain a behavior feedback label; If the type of the interaction feedback information is interface feedback, the interaction feedback information is matched with a preset mapping relationship between the interface feedback type and the interface feedback label to obtain the interface feedback label.

8. The voice interaction method according to claim 6, characterized in that: After generating the feedback tag according to the interactive feedback information, the method further includes: If the feedback tag is determined to be positive feedback, the number of times the target historical similar voice is used and the most recent time of use in the preset experience database are updated; The preset experience library is cleaned up regularly based on the updated usage count and the updated most recent usage time.

9. A voice interaction device, characterized in that: include: An acquisition module is used to obtain the speech to be recognized and its current speech recognition text; A retrieval module, configured to retrieve historical speech similarity information based on the speech to be recognized and the current speech recognition text; A splicing module, configured to splice the current speech recognition text and the historical speech similarity information to obtain splicing information; The interaction module is used to infer the splicing information and generate a voice interaction result.

10. An XR device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the voice interaction method according to any one of claims 1 to 8 is implemented.

Images