Voice interaction method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides a voice interaction method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a first interactive audio input by a target object, and performing phoneme analysis on the first interactive audio to obtain a phoneme sequence; obtaining a historical interaction text associated with the target object, and inputting the phoneme sequence and the historical interaction text into a large language model for semantic analysis to obtain a first text recognition result corresponding to the phoneme sequence; performing grammar correction on the first text recognition result to obtain a second text recognition result; through a large language model, based on the first text recognition result and the second text recognition result, a corresponding target feedback text is generated, and the target feedback text comprises grammar correction information and reply information corresponding to the first interaction audio; and outputting a corresponding second interaction audio to the target object based on the target feedback text. Therefore, the reliability and effectiveness of voice interaction teaching can be improved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a voice interaction method, device, computer equipment, and readable storage medium. Background Technology

[0002] A voice-interactive teaching device is a tool that enables interactive language learning by recognizing and processing the voice input of a target audience. Its core lies in the intelligent analysis of the target audience's expressed meaning and the generation of corresponding guiding feedback, thereby assisting the target audience in improving their comprehensive ability to use the target language.

[0003] In related technologies, the target's speech is typically converted into text recognition results first, and then grammatical correction and responses are performed based on these results. However, the accuracy of the evaluation highly depends on the correctness of the front-end recognition. If text conversion errors occur due to factors such as pronunciation or accent, subsequent grammatical correction will be based on erroneous text, leading to incorrect grammatical analysis conclusions and misleading feedback. This not only fails to help learners correct their grammar but may also exacerbate learners' confusion due to incorrect feedback, ultimately reducing the reliability and effectiveness of voice-interactive teaching. Summary of the Invention

[0004] This application proposes a voice interaction method, device, computer equipment, and readable storage medium, which can improve the reliability and effectiveness of voice interaction teaching.

[0005] To achieve the above objectives, a first aspect of this application proposes a voice interaction method, the method comprising: The first interactive audio input from the target object is obtained, and phoneme analysis is performed on the first interactive audio to obtain a phoneme sequence; Obtain the historical interaction text associated with the target object, and input the phoneme sequence and the historical interaction text into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence; The first text recognition result is grammatically corrected to obtain the second text recognition result; Based on the first text recognition result and the second text recognition result, the large language model generates a corresponding target feedback text, wherein the target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio. Based on the target feedback text, the corresponding second interactive audio is output to the target object.

[0006] Accordingly, a second aspect of the embodiments of this application provides a voice interaction device, the device comprising: The acquisition module is used to acquire the first interactive audio input by the target object, and to perform phoneme analysis on the first interactive audio to obtain a phoneme sequence; The analysis module is used to obtain the historical interaction text associated with the target object, and input the phoneme sequence and the historical interaction text into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence; The correction module is used to perform grammatical correction on the first text recognition result to obtain the second text recognition result; The generation module is used to generate corresponding target feedback text based on the first text recognition result and the second text recognition result through the large language model, wherein the target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio; The output module is used to output the corresponding second interactive audio to the target object based on the target feedback text.

[0007] In some embodiments, the voice interaction device further includes an acoustic feature analysis module, used for: Based on the first text recognition result, acoustic feature analysis is performed on the first interactive audio to obtain pronunciation recognition result; The pronunciation recognition results are input into the large language model to obtain corresponding pronunciation correction suggestions. Output the pronunciation correction suggestion information to the target object.

[0008] In some embodiments, the acoustic feature analysis module is further used for: Based on the first text recognition result, the start time index and end time index of each word element in the first interactive audio are determined, and the pronunciation segment corresponding to each word element is determined from the first interactive audio based on the start time index and the end time index. By performing acoustic feature analysis on each word element and its corresponding pronunciation segment, a pronunciation recognition sub-result for each word element is obtained; Based on the pronunciation recognition sub-results corresponding to each word element contained in the first text recognition result, the pronunciation recognition result corresponding to the first text recognition result is obtained.

[0009] In some implementations, the analysis module is further configured to: Obtain a phoneme pronunciation profile of the target object, wherein the phoneme pronunciation profile is used to record the mapping relationship between the confused pronunciation phonemes output by the target object and the standard phonemes; Based on the phoneme pronunciation profile, the corresponding target confused pronunciation phoneme is extracted from the phoneme sequence, and the target standard phoneme corresponding to the target confused pronunciation phoneme in the phoneme pronunciation profile is extracted. Based on the target standard phoneme, replace the target confused pronunciation phoneme in the phoneme sequence to obtain at least one candidate phoneme sequence; The at least one candidate phoneme sequence, the phoneme sequence, and the historical interactive text are input into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence.

[0010] In some embodiments, the correction module is further configured to: Obtain preset grammatical structure rules, and analyze the first text recognition result based on the grammatical structure rules to obtain the corresponding grammatical error elements; The interaction scene corresponding to the first interactive audio is obtained, and based on the interaction scene and the grammatical error elements, the grammatical correction of the first text recognition result is performed to obtain the corresponding second text recognition result.

[0011] In some implementations, the generation module is further configured to: Based on the second text recognition result, positive grammar correction information corresponding to the first text recognition result is generated using the large language model. Obtain the interaction scene corresponding to the first interactive audio, and perform dialogue extension and dialogue response on the second text recognition result based on the interaction scene to obtain the response information corresponding to the first interactive audio; Based on the positive grammar correction information and the response information, a corresponding target feedback text is generated.

[0012] In some implementations, the generation module is further configured to: Obtain the intonation information of the first interactive audio, and determine the current emotion category and emotion intensity value of the target object based on the intonation information; Based on the emotion category and the emotion intensity value, a corresponding emotion state feature is generated; The first text recognition result, the second text recognition result, and the emotional state features are input into the large language model to obtain the corresponding target feedback text.

[0013] Accordingly, a third aspect of the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the voice interaction method of any one of the embodiments of the first aspect of the present application.

[0014] Accordingly, a fourth aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the voice interaction method of any one of the embodiments of the first aspect of this application.

[0015] This application obtains a phoneme sequence by acquiring a first interactive audio input from a target object and performing phoneme analysis on the first interactive audio; it then acquires the target object's associated historical interactive text and inputs the phoneme sequence and historical interactive text into a large language model for semantic analysis to obtain a first text recognition result corresponding to the phoneme sequence; it performs grammatical correction on the first text recognition result to obtain a second text recognition result; and through the large language model, it generates a corresponding target feedback text based on the first and second text recognition results. The target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio; finally, it outputs the corresponding second interactive audio to the target object based on the target feedback text. In this way, the accuracy of text recognition can be improved through phoneme sequence and context-aware semantic analysis, reducing recognition errors caused by inaccurate pronunciation or accents, and achieving natural, non-interrupted interactive teaching feedback through the fusion of grammatical correction and semantic response. Specifically, phoneme analysis preserves the original acoustic features, avoiding text conversion errors caused by inaccurate pronunciation or accents in traditional automatic speech recognition; the large language model, combined with historical interactive text, performs semantic reasoning on phoneme sequences, effectively improving robustness in recognizing the user's true intent; thus, grammatical correction based on accurate first text recognition results effectively prevents erroneous text from propagating to subsequent analyses; the generated feedback text naturally integrates grammatical correction information into the scenario response, enabling feedback to maintain dialogue coherence while specifically pointing out errors, improving the accuracy of teaching and interactive guidance and the target audience's experience. In summary, this application can improve the reliability and effectiveness of voice-interactive teaching. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of the voice interaction system provided in the embodiments of this application; Figure 2 This is a flowchart of the voice interaction method provided in the embodiments of this application; Figure 3 This is a general flowchart of the voice interaction method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the functional modules of the voice interaction device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] A voice-interactive teaching device is a tool that enables interactive language learning by recognizing and processing the voice input of a target audience. Its core lies in the intelligent analysis of the target audience's expressed meaning and the generation of corresponding guiding feedback, thereby assisting the target audience in improving their comprehensive ability to use the target language.

[0021] In related technologies, the target's speech is typically converted into text recognition results first, and then grammatical correction and responses are performed based on these results. However, the accuracy of the evaluation highly depends on the correctness of the front-end recognition. If text conversion errors occur due to factors such as pronunciation or accent, subsequent grammatical correction will be based on erroneous text, leading to incorrect grammatical analysis conclusions and misleading feedback. This not only fails to help learners correct their grammar but may also exacerbate learners' confusion due to incorrect feedback, ultimately reducing the reliability and effectiveness of voice-interactive teaching.

[0022] Based on this, embodiments of this application provide a voice interaction method, apparatus, computer device, and readable storage medium, which can improve the reliability and effectiveness of voice interaction teaching.

[0023] The voice interaction method, apparatus, computer equipment, and readable storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the voice interaction system in the embodiments of this application is described.

[0024] Please refer to Figure 1 In some embodiments, this application provides a voice interaction system, including a terminal 11 and a server 12.

[0025] In some implementations, terminal 11 can be used to collect first interactive audio input from the target object and perform preliminary audio processing and result presentation. For example, terminal 11 can be a smartphone, tablet computer, smart learning machine, or a dedicated interactive device equipped with a microphone and speaker.

[0026] Furthermore, the terminal 11 can perform phoneme analysis on the first interactive audio through a built-in or integrated processing module to obtain a phoneme sequence, and upload the phoneme sequence to the server 12 for further analysis; alternatively, it can directly upload the first interactive audio to the server 12 for analysis. The terminal 11 can also receive feedback information from the server 12 to play the second interactive audio to the target object, thereby realizing the front-end human-computer interaction interface function.

[0027] In some implementations, server 12 can be used to perform computationally intensive semantic understanding, syntax analysis, and content generation tasks. For example, it can be a cloud server cluster or a locally deployed high-performance computing device.

[0028] Furthermore, the server 12 can run a large language model, receive phoneme sequences or first interactive audio from the terminal 11, and obtain the historical interactive text corresponding to the relevant user for semantic analysis to obtain the first text recognition result, thereby completing grammatical correction and generating target feedback text that integrates correction information and scenario response to provide core language processing and intelligent teaching capabilities.

[0029] Terminal 11 and server 12 can communicate and exchange commands via wired or wireless networks. Terminal 11 uploads the generated phoneme sequence and associated context to server 12. After completing intelligent processing, server 12 can send the target feedback text to terminal 11, thereby collaboratively completing a complete interactive teaching loop from voice input to intelligent feedback.

[0030] The voice interaction method in this application can be illustrated through the following embodiments.

[0031] It should be noted that in various specific embodiments of this application, when processing data related to the identity or characteristics of the target object, such as target object information, target object behavior data, target object historical data, and target object location information, the target object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application need to obtain sensitive personal information of the target object, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0032] In this embodiment, the description will focus on the voice interaction device, which can be integrated into a computer device. See also... Figure 2 , Figure 2 This is a flowchart illustrating the steps of the voice interaction method provided in this application embodiment. Taking the voice interaction device specifically integrated into a terminal or server as an example, the specific process when the processor on the terminal or server executes the program instructions corresponding to the voice interaction method is as follows: Step 101: Obtain the first interactive audio input from the target object, and perform phoneme analysis on the first interactive audio to obtain a phoneme sequence.

[0033] In some implementations, in order to accurately understand the user's voice intent and avoid recognition errors caused by inaccurate pronunciation in traditional automatic speech recognition, the first interactive audio input from the target object can be acquired and its phonemes analyzed to obtain the original phoneme sequence, so as to preserve the acoustic features of the user's pronunciation and provide basic data for subsequent large language models to guess the user's true intent based on context.

[0034] The target audience can be users who engage in interactive Chinese language learning, such as language learners who converse with artificial intelligence in simulated scenarios.

[0035] The first interactive audio can be voice data input by the target object through a microphone in the interactive scenario, such as a recording of sentences or words spoken by the user in a food ordering or interview scenario.

[0036] Among them, the phoneme sequence can be a pinyin sequence with tone obtained after performing low-level acoustic analysis on the first interactive audio, such as the "mian4 pao1" sequence that reflects the user's actual pronunciation characteristics, which can be used by subsequent large language models to guess the text that the user is actually expressing.

[0037] In some embodiments, a sentence or a continuous speech uttered by a target object in a specific teaching scenario (such as ordering food in a simulated coffee shop) can be received through the microphone module of a user device (such as a smart phone, a computer, or an interactive robot), and digitized into an audio signal (such as PCM format), which is the first interactive audio.

[0038] Specifically, the first interactive audio can be preprocessed and acoustic features can be extracted. The preprocessing can include operations such as pre-emphasis, framing, and windowing to enhance high-frequency components and reduce spectral leakage. Subsequently, acoustic features that can characterize the essential characteristics of the speech are extracted. For example, the Mel-frequency cepstral coefficients of the first interactive audio are extracted to obtain an acoustic feature sequence. For example, for a 2-second speech segment, after framing with a frame length of 25 milliseconds and a step size of 10 milliseconds, approximately 200 frames can be extracted, and each frame corresponds to a 39-dimensional acoustic feature sequence.

[0039] Furthermore, the extracted acoustic feature sequence can be input into a pre-trained acoustic model, and the acoustic model calculates the posterior probability of each frame belonging to each phoneme state. Finally, the decoder searches the probability sequence output by the acoustic model in combination with a pronunciation dictionary and a language model (optional or not) to find the most likely phoneme sequence corresponding to the first interactive audio. Different from the traditional automatic speech recognition (ASR) that directly outputs Chinese character text, the decoding process of this application defines the output unit as phonemes (and tones), so as to obtain a pure phoneme sequence without semantic inference.

[0040] Exemplarily, even if the user pronounces the initial consonant / b / of "bread (bāo)" as / p / , the phoneme recognition module will also truly output a sequence reflecting its actual acoustic features, such as "mian4 pao1" (where "4" and "l" represent tones), rather than the possibly incorrect Chinese character "cotton-padded jacket". In this way, the acoustic information of the user's original pronunciation can be retained to the greatest extent, providing a high-fidelity input for subsequent semantic guessing based on context without being interfered by traditional ASR error correction.

[0041] In some embodiments, in order to better accommodate users with different regional accents (such as Cantonese accent, Hakka accent, Wu dialect accent) or dialect backgrounds, the phoneme recognition module can integrate acoustic models corresponding to multiple dialect accents. Specifically, each acoustic model can be pre-trained so that the acoustic model learns the mapping relationship between the pronunciation corresponding to each dialect accent and phonemes. Thus, when the target object inputs the first interactive audio, the corresponding acoustic model can be called, and the acoustic model outputs the corresponding phoneme sequence based on the first interactive audio. In this way, the interference of the accent on the underlying phoneme capture can be reduced, and the reliability of the phoneme sequence can be improved.

[0042] By using the above methods, the original acoustic information of the user's pronunciation can be accurately captured, avoiding the semantic bias that may be introduced by traditional automatic speech recognition directly converting to text, thus laying the foundation for the robustness of the entire system's speech understanding and the fluency of interaction.

[0043] Step 102: Obtain the historical interaction text associated with the target object, and input the phoneme sequence and historical interaction text into the large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence.

[0044] In some implementations, in order to accurately infer the true semantic intent, the historical interaction text associated with the target object can be obtained, and the phoneme sequence reflecting the acoustic features and the historical interaction text can be input together into a large language model for joint semantic analysis. This allows the model to use its contextual understanding and common sense reasoning capabilities to guess and output the text result that best matches the context and user intent from the potentially inaccurate phoneme sequence.

[0045] Among them, historical interaction text can be the interaction text content generated in the current dialogue scenario before the target object's current input, such as the AI ​​system's previous question to the user, "What would you like to eat?".

[0046] Among them, large language models can be artificial intelligence models that have been trained and have strong contextual understanding and text generation capabilities, such as Tongyi Qianwen, Kimi model, GPT series models, etc.

[0047] The first text recognition result can be the Chinese text output by the large language model after semantic analysis of the phoneme sequence, which is inferred to be the actual expression of the target object. For example, the output "bread" is the phoneme sequence "mian4 pao1" combined with the context of "ordering food".

[0048] In some implementations, the system can maintain a session cache that stores, in sequence, the multi-turn dialogue text between the target object and the system in the current scenario. When processing the user's current input (i.e., the first interactive audio), the system extracts the dialogue text from the cache preceding this input, within a preset number of turns (e.g., the most recent 3 to 5 turns), and concatenates or formats it as historical interaction text. For example, in a coffee shop ordering scenario, if the system's previous question was "What would you like to eat?", that text would be extracted as historical interaction text. This provides immediate and accurate context for understanding the current user's intent, helping to resolve semantic ambiguity caused by unclear pronunciation.

[0049] In some implementations, all earlier dialogues beyond a preset number of rounds can be processed by a large language model to generate a summary text that condenses key information and the main theme of the dialogue. This summary text is then combined with at least one recent historical interaction text. In this way, the context length input to the large language model can be controlled without losing important background information, balancing understanding depth and computational efficiency.

[0050] For example, historical interaction text (as known context) and the phoneme sequence to be analyzed can be combined to construct a prompt text with a clear task. For instance: "Based on the following dialogue history and the pinyin sequence spoken by the user, please infer what the user most likely wants to express in Chinese. Dialogue history: [historical interaction text]. User pinyin: [phoneme sequence]. Inference result:." This prompt text can then be input into a large language model. The large language model encodes and understands the prompt text based on its internal parameters and autoregressively generates the most likely subsequent text sequence, i.e., the inferred Chinese sentence. For example, for the phoneme sequence "mian4 pao1" and the historical text "What do you want to eat?", the large language model will output "bread" as the first text recognition result, rather than the strictly phoneme-corresponding "mianpao". This effectively isolates and alleviates the core problem of traditional end-to-end systems directly outputting incorrect Chinese characters due to inaccurate user pronunciation, significantly improving the robustness and accuracy of speech understanding.

[0051] In some implementations, in addition to historical interaction text, visual descriptive text of the current scene (such as "There is a coffee table and menu in front of the user") may also be included. Then, the visual descriptive text, historical interaction text, and phoneme sequence are input into a large language model to obtain the first text recognition result corresponding to the phoneme sequence, making the guessing of the user's voice intent more comprehensive and accurate.

[0052] By combining pure acoustic information with rich dialogue context, intelligent semantic reasoning can be achieved. This can effectively correct phoneme sequence deviations caused by inaccurate pronunciation by the user, resulting in text that more accurately reflects the user's intent. This provides a reliable and consistent text foundation for subsequent parallel grammar correction, ensuring the coherence and accuracy of the entire interactive teaching process.

[0053] In some implementations, to more accurately identify user speech intent, especially when the user has specific pronunciation habits, a phoneme pronunciation profile of the target object cached by the system can be obtained. Based on this profile, confused pronunciation phonemes are extracted from the phoneme sequence and replaced with standard phonemes to generate a candidate phoneme sequence. The candidate sequence, along with the original phoneme sequence and historical interaction text, is then input into a large language model for semantic analysis. This utilizes a personalized pronunciation correction mechanism to improve the accuracy and robustness of semantic reasoning. For example, step 102, "inputting the phoneme sequence and historical interaction text into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence," may include: (102.1) Obtain the phoneme pronunciation profile of the target object, wherein the phoneme pronunciation profile is used to record the mapping relationship between the confused pronunciation phonemes output by the target object and the standard phonemes; (102.2) Based on the phoneme pronunciation profile, extract the corresponding target confused pronunciation phoneme from the phoneme sequence, and the target standard phoneme corresponding to the target confused pronunciation phoneme in the phoneme pronunciation profile; (102.3) Replace the target confused pronunciation phoneme in the phoneme sequence with the target standard phoneme to obtain at least one candidate phoneme sequence; (102.4) Input at least one candidate phoneme sequence, the phoneme sequence and historical interactive text into the large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence.

[0054] Among them, the phoneme pronunciation profile can be a data structure that records the historical pronunciation error patterns of the target object, such as a mapping table or database, storing the correspondence between the standard phoneme "bao" and "pao" that users often mispronounce. It can be used for personalized identification and correction of user pronunciation deviations.

[0055] Among them, confused pronunciation phonemes can be incorrect phonemes that are inconsistent with the standard phonemes in the actual pronunciation of the target object. For example, the phoneme "pao" that a user pronounces as "bao" can be used to identify the specific location of the pronunciation error.

[0056] Among them, the standard phoneme can be the correct pronunciation phoneme corresponding to the confused pronunciation phoneme, such as the standard pronunciation of "bao" for "bao", which can be used as a reference benchmark for correcting the user's pronunciation.

[0057] Among them, the target confused pronunciation phoneme can be a confused pronunciation phoneme identified from the current phoneme sequence that matches the phoneme pronunciation profile. For example, “pao1” extracted from the phoneme sequence “mian4 pao1” can be used for targeted phoneme replacement.

[0058] The target standard phoneme can be the standard phoneme that is mapped to the target confused phoneme in the phoneme pronunciation profile. For example, "pao1" is mapped to "bao1" according to the profile.

[0059] Among them, the candidate phoneme sequence can be a phoneme sequence variant obtained by replacing the target confused pronunciation phoneme with the target standard phoneme. For example, the sequence after replacing "mian4 pao1" with "mian4 bao1" can be used to provide a variety of possible semantic inputs for large language models.

[0060] In some implementations, the system can continuously record and analyze the differences between the actual phonemes (output by the phoneme recognition module) of the target object in previous interactions and the standard phonemes corresponding to the standard text ultimately guessed by the large language model. For example, in multiple ordering scenarios, when a user wants to express "bread," the phoneme recognition module repeatedly outputs "mian4 pao1" (where "pao1" is a confused phoneme), while the large language model, combined with the context, repeatedly corrects it to the standard phoneme "bao1" corresponding to "bao" (package). The system will statistically analyze such consistently occurring error mappings, forming a mapping table with confused phonemes as keys and standard phonemes as values. For example, the phoneme pronunciation profile can be recorded as: {"pao1":"bao1"; "si4":"shi4"}. In this way, the system can capture and model the user's unique, systematic pronunciation habits in a personalized way, providing a data foundation for subsequent accurate correction. The phoneme pronunciation profile can be dynamically and robustly updated as the user's pronunciation level changes.

[0061] In some implementations, a phoneme sequence (e.g., "wo3 si4 xue2 sheng1") can be compared one-to-one with keys (i.e., confused phonemes, such as "si4") in a phoneme pronunciation profile. When a phoneme in the phoneme sequence (including initials, finals, and tones) completely matches a key in the phoneme pronunciation profile or is within a certain edit distance (e.g., allowing different tones), it can be marked as the target confused phoneme. Simultaneously, the value corresponding to that key is retrieved from the profile as the target standard phoneme. For example, the target confused phoneme "si4" can be extracted from the phoneme sequence "wo3 si4 xue2 sheng1," and its target standard phoneme "shi4" can be obtained.

[0062] Furthermore, the phoneme sequence can be viewed as a sequence of nodes, with each possible phoneme position to be replaced (i.e., the target confused phoneme) serving as a decision point. Starting from the first decision point, attempts are made to retain the original phoneme and replace it with the target standard phoneme, generating different phoneme sequences. Subsequent decision points are processed sequentially, eventually forming a binary tree (if each position has only two choices). Traversing all paths from the root to the leaves yields all possible candidate phoneme sequences. For example, for the sequence "mian4 pao1" with the profile {"pao1":"bao1"}, two candidate sequences will be generated: the original "mian4pao1" and the replaced "mian4 bao1".

[0063] In some implementations, a special text prompt can be constructed, presenting the original phoneme sequence and all candidate phoneme sequences (labeled with their sources) side-by-side to the large language model, requiring it to comprehensively determine the user's most likely intent. For example, the prompt text could be: "The user may be saying one of the following pinyin. Please determine what he most likely wants to express based on the dialogue history. History: [Historical Interaction Text]. Original Pinyin: [Phonetic Sequence]. Possible Corrected Pinyin 1: [Candidate Phoneme Sequence 1] (Reason: The user often pronounces 'b' as 'p'). Possible Corrected Pinyin 2: [Candidate Phoneme Sequence 2]. Please output the most reasonable Chinese sentence:." Thus, the large language model integrates all pinyin variations, historical context, and common sense to output the most reasonable first text recognition result. This creatively combines offline-built personalized pronunciation profiles with online large language model reasoning, proactively considering and attempting to correct habitual pronunciation errors when guessing user intent, effectively improving the efficiency and accuracy of reasoning, and helping to solve the long-term recognition bottleneck caused by persistent pronunciation errors.

[0064] By using the above methods, common pronunciation errors of users can be corrected in a personalized way, and phoneme sequence variants that are more in line with standard pronunciation can be generated. This enhances the reasoning efficiency and accuracy of large language models when performing semantic analysis in conjunction with context.

[0065] Step 103: Perform grammatical correction on the first text recognition result to obtain the second text recognition result.

[0066] In some implementations, in order to improve the grammatical accuracy of user input text, the first text recognition result can be analyzed and corrected according to grammatical rules and contextual information to obtain a second text recognition result that conforms to language norms. This eliminates grammatical errors in the original recognition result and provides an accurate textual basis for generating natural and authentic dialogue responses.

[0067] Among them, the second text recognition result can be the text that has been grammatically corrected from the first text recognition result and conforms to the grammatical norms. For example, correcting the text with missing grammar "I went to an apple yesterday" to "I went to buy an apple yesterday" which contains the correct verb and tense particle.

[0068] In some embodiments, a dedicated grammar correction Agent (or module) can be used to grammatically correct the first text recognition result. The Agent can identify and correct grammar errors in the text, including but not limited to word order errors, component omissions (such as missing verbs, particles), inappropriate word usage, collocation errors, and tense-aspect misuses, etc.

[0069] Specifically, the input first text recognition result can be analyzed based on preset grammar structure rules. The grammar structure rules can be obtained from a standard linguistic knowledge base or abstracted from a statistical model trained with a large amount of correct corpus. When analyzing, dependency syntax analysis or constituent syntax analysis techniques can be used to parse the grammatical relationships (such as subject-predicate, verb-object, attributive-middle, etc.) between the components in the sentence, and compare them with the correct patterns in the grammar structure rules, so as to locate potential grammar error elements.

[0070] Exemplarily, for the first text recognition result "I went to an apple yesterday", through dependency analysis, it is found that there is a lack of a proper semantic relationship (improper verb-object collocation) between the verb "went" and the noun "apple", and there is a tense mismatch between the time adverbial "yesterday" and the verb "went" without the aspect marker "了", thus identifying the missing verb and the missing aspect marker as grammar error elements. In this way, it is possible to accurately locate the specific problem points of non-native expressions or violations of grammar rules.

[0071] In some embodiments, the corrected text can be generated based on the detected grammar error elements and the context information of the current interaction scenario. The interaction scenario can be obtained from the system's preset scenario labels (such as "ordering food", "asking for directions") or automatically induced from the conversation history. For example, for "I went to an apple yesterday", combined with the scenario of describing past activities, the system generates the corrected second text recognition result as "I went to buy an apple yesterday". In this way, the deep integration of grammar correction and semantic scenarios can be achieved, making the correction result not only grammatically correct but also conform to the expression habits of specific situations, laying a high-quality text foundation for generating natural and fluent conversation responses in the future.

[0072] In some embodiments, in order to accurately correct the grammar of the user text, the grammatical error elements in the first text recognition result can be analyzed by obtaining the preset grammatical structure rules, and context-adaptive grammar correction can be performed in combination with the interaction scenario corresponding to the first interaction audio, so as to generate a second text recognition result that not only conforms to the grammatical norms but also fits the scenario semantics. Exemplarily, step 103 may include: (103.1) Obtain the preset grammatical structure rules, and based on the grammatical structure rules, analyze the first text recognition result to obtain the corresponding grammatical error elements; (103.2) Obtain the interaction scenario corresponding to the first interaction audio, and based on the interaction scenario and the grammatical error elements, perform grammar correction on the first text recognition result to obtain the corresponding second text recognition result.

[0073] Among them, the grammatical structure rules can be a preset set of target language grammar specifications. For example, a rule library including the integrity of the subject-predicate-object structure, the usage rules of tense auxiliaries, and the tone collocation rules is a benchmark for analyzing the grammatical correctness of the text.

[0074] Among them, the grammatical error elements can be specific grammatical problems identified by analyzing the first text recognition result through the grammatical structure rules. For example, error types such as missing verbs in a sentence or improper use of tones can be used to guide subsequent targeted grammar correction.

[0075] Among them, the interaction scenario can be the dialogue background environment where the first interaction audio is located, such as a dining scene, an interview scene, or a taxi-hailing scene.

[0076] In some embodiments, the preset grammatical structure rules can be loaded from a structured linguistic knowledge base. The linguistic knowledge base can encode the core grammar rules of the target language (such as Chinese, or other languages such as English, etc.) in a computer-readable form. The grammatical structure rules can include but are not limited to: syntactic rules (such as the basic structure of "subject + predicate + object"), word class collocation rules (such as "adjective + 'de' + noun"), function word usage rules (such as "le" is used to indicate the perfective aspect), and common error pattern rules (such as "shi" and "hen" cannot be directly connected).

[0077] Furthermore, the grammatical structure rules can be expressed as "condition-action" pairs or constraint templates. Exemplarily, a grammatical structure rule regarding the perfective aspect can be expressed as: If the sentence semantics represents a completed action and the main verb is not followed by the dynamic auxiliary "le", then trigger the "missing aspect marker" error element.

[0078] In some embodiments, the first text recognition result can be segmented and tagged with word types, and then dependency syntactic analysis can be performed to obtain a syntactic dependency relationship tree between words in the sentence. Subsequently, the syntactic dependency relationship tree is pattern-matched and constraint-checked against the loaded syntactic rules. Specifically, by traversing the syntactic dependency relationship tree, it is checked whether the word type collocations between parent and child nodes conform to the rules, and whether the structure of the entire sentence satisfies the integrity constraints. For example, for the sentence "I went to an apple yesterday", the dependency analysis will establish "went" as the core predicate, with its object being "apple", and there is an adverbial "yesterday". The system then checks the rules and finds that as a transitive verb, the typical object of "went" should be a location noun or an object noun with the meaning of "acquisition", and "apple" as a simple thing noun does not conform to the typical collocation, and when "went" is collocated with "yesterday", the aspect marker "了" is usually required. Therefore, the system will generate syntactic error elements such as {"error type": "inappropriate verb-object collocation and aspect marker missing", "position": [["went", "apple"]], "suggested action": "add appropriate verb and aspect marker"}. In this way, syntactic problems can be accurately located at a deep level of sentence structure and semantic relationships.

[0079] In some embodiments, when starting a practice session, the system will explicitly set a scenario label (such as "restaurant ordering", "hotel check-in", "job interview"), and the scenario label can serve as the interaction scenario of the session. Therefore, the interaction scenario corresponding to the first interaction audio can be directly read from the session metadata.

[0080] In some embodiments, the interaction scenario can also be dynamically inferred from the conversation history through a large language model. This application does not limit the determination method of the interaction scenario.

[0081] In some embodiments, the first text recognition result, syntactic error elements (such as the type and position of missing components), and the interaction scenario can be used as conditions and input into a conditional text generation model (for example, a fine-tuned Seq2Seq model or a large language model directly guided in the prompt). The model is trained or guided to generate a sentence that corrects the specified error and conforms to the scenario expression. Exemplarily, the input conditions are: original sentence: "I went to an apple yesterday". Error: inappropriate verb-object collocation ("went to apple"), aspect marker missing. Scenario: describing past activities / shopping. The correction model may then output: "I went to buy an apple yesterday". Here, the addition of "buy" solves the collocation problem, the addition of "了" solves the aspect marker problem, and the entire sentence conforms to the scenario semantics of "describing past shopping activities". In this way, formal syntactic error analysis can be combined with semantically rich scenario information to drive the generation of corrected text that is both syntactically standard and situationally appropriate.

[0082] In some embodiments, for the correction of colloquial and informal expressions, the judgment of语体appropriateness can be introduced. Before correction, the system first judges the degree of formality of the语体required by the current interaction scenario (e.g., a formal语体is required in the "interview" scenario, and an informal语体is acceptable in the "chat with friends" scenario). For expressions that appear in an informal scenario and are slightly loose in grammar but conform to colloquial habits (such as "I went to an apple store yesterday"), the grammar correction module can appropriately relax the rules or generate a comparison suggestion between a more formal version and an orally acceptable version, rather than forcing a correction into a completely standard written sentence.

[0083] Through the above method, the formal grammar rules can be combined with the specific scenario semantics for comprehensive analysis. In this way, more accurate grammar correction that better meets the actual communication needs can be achieved, ensuring that the corrected text is both grammatically correct and contextually appropriate, and enhancing the practicality and naturalness of teaching feedback.

[0084] Step 104: Based on the first text recognition result and the second text recognition result, generate a corresponding target feedback text through a large language model, where the target feedback text includes grammar correction information of the second text recognition result relative to the first text recognition result and reply information corresponding to the first interactive audio.

[0085] In some embodiments, in order to naturally integrate grammar correction into scenario conversations and generate feedback that combines teaching guidance and interactive coherence, a large language model can be used to generate a target feedback text containing grammar correction information and scenario reply content based on the comparison between the first text recognition result and the second text recognition result, so as to achieve heuristic and non-interruptive grammar teaching while maintaining the fluency of the conversation.

[0086] Among them, the target feedback text can be the complete reply content generated by the large language model by integrating grammar correction and scenario conversations. For example, in the dining scenario, for the user's grammar error, the generated "That sounds great! By the way, you just said 'I went to an apple yesterday'. Actually, a more idiomatic way is 'I went to buy an apple yesterday'. Are the apples you bought sweet?" can be used to provide grammar guidance and continue the scenario conversation to the user at the same time, increasing the user's interest in the conversation.

[0087] Among them, the grammar correction information can be the content in the target feedback text that explains and corrects the grammar errors in the first text recognition result. For example, the part in the above reply that compares the original statement with the idiomatic statement and explains the usage of the auxiliary word "了".

[0088] Among them, the reply information can be the content that continues the conversation related to the current interaction scenario in the target feedback text. For example, the question "Are the apples you bought sweet?" in the above reply that pulls the conversation back to the dining scenario can be used to maintain the coherence and immersion of the conversation.

[0089] In some embodiments, the first text recognition result (the user's original expression), the second text recognition result (the corrected expression), and the historical context of the current conversation can be organized into a prompt text in a predetermined format. The prompt text can clearly instruct the large language model to play a dual role. For example, one role can be the conversation role in the current scenario (such as a waiter), and the other role can be a patient language coach.

[0090] Specifically, the prompt template can be designed as follows: "You are playing a [interaction scenario, e.g., coffee shop waiter]. Please make a natural and friendly response to the user's latest sentence based on the conversation history. At the same time, there was a minor grammar issue in the user's previous expression, and we have optimized it into a more native way of saying it. In your response, first affirm the user's meaning, then very naturally and in a suggestive way introduce the more native way of saying it (don't say 'you're wrong'), and finally continue the conversation.

[0091] Conversation history: [historical interaction text]; User's original words: [first text recognition result]; More native way of saying it: [second text recognition result]; Please generate your response: ".

[0092] After receiving this prompt, the large language model will, based on its powerful language generation and context understanding capabilities, integrate the four elements of "interaction scenario", "historical interaction text", "first text recognition result", and "second text recognition result" to generate a text that meets the requirements. Exemplarily, the input is: interaction scenario = coffee shop waiter, historical interaction text = "What would you like to order?", first text recognition result = "I went to an apple yesterday", second text recognition result = "I went to buy an apple yesterday". The large language model may generate: "That sounds great! By the way, you said 'I went to an apple yesterday' just now. Actually, if you want to express that something has been completed, we usually say 'I went to buy an apple yesterday'. Adding the particle '了' makes it more natural. Is the apple you bought sweet?" In this way, through carefully designed prompt engineering, the rigid grammar correction instruction can be transformed into an internal and empathetic conversation strategy of the large language model, thus achieving a seamless and natural integration of corrected information and the conversation flow, providing users with an immersive experience of "learning in conversation".

[0093] In some embodiments, in order to cultivate the learner's metacognitive ability, that is, the awareness and regulation of their own cognitive processes, a stepped prompting strategy can be designed. Specifically, the system can determine the prompting degree of this feedback based on the user's historical error correction acceptance and the current error type. For example, the prompting text input into the large language model can be divided into three levels: The first level is direct embedding. As shown in the above example, a more idiomatic expression is directly given. The second level is heuristic questioning. The prompt becomes: "There may be one place in the user's original words that can be improved. Can you find it? Please first guide the user to think in the form of a question and then give suggestions." The model may generate: "You said 'I went to an apple yesterday'. Think about it, is there a word missing that means 'completed'? … Right, it's 'le'. So we usually say 'I went to buy an apple yesterday'." The third level is metacognitive guidance. The prompt requires the model to guide the user to reflect on the expression process, such as: "Please guide the user to review the gap between what they just wanted to express and the sentence they actually said." In this way, by dynamically adjusting the prompting strategy, the system can gradually transition from imparting to guiding, promoting the user to think actively and internalize language rules.

[0094] Through the above method, the grammar correction information can be organically incorporated into the reply information of the scenario dialogue. In this way, while maintaining the natural fluency of the dialogue, grammar guidance can be provided to the user, avoiding abruptly interrupting the dialogue process, and thus creating an immersive language learning experience of learning while using for the user, improving the interaction effect and user participation.

[0095] In some embodiments, in order to naturally incorporate grammar correction into the scenario dialogue in a positive and constructive manner and avoid blunt correction, the large language model can generate positive grammar correction information based on the second text recognition result, and combine the interaction scenario corresponding to the first interaction audio to extend the dialogue and generate a reply, and finally synthesize the target feedback text to provide encouraging and heuristic grammar teaching while maintaining the coherence and immersion of the dialogue. Exemplarily, step 104 may include: (104.a1) Through the large language model, according to the second text recognition result, generate positive grammar correction information corresponding to the first text recognition result; (104.a2) Obtain the interaction scenario corresponding to the first interaction audio, and extend the dialogue and generate a dialogue reply based on the interaction scenario for the second text recognition result to obtain the reply information corresponding to the first interaction audio; (104.a3) Based on the positive grammar correction information and the reply information, generate the corresponding target feedback text.

[0096] Among them, the positive grammar correction information can be grammar guidance content generated by a large language model and presented in a positive and constructive way. For example, by comparing the user's original statement with a more natural statement to imply the correct usage, such as "Actually, a more natural way to say it is...", which can be used for implicit grammar teaching while protecting the user's confidence.

[0097] Among them, the interaction scenario can be the specific conversation background or role-playing environment where the first interaction audio is located, such as an ordering scenario, an interview scenario, or a taxi-hailing scenario, to provide context basis for the conversation reply and ensure that the reply information conforms to the scenario logic and the role identity.

[0098] In some embodiments, positive grammar correction information can be generated by producing prompts of constructive comparison rather than error correction. The first text recognition result (original sentence) and the second text recognition result (corrected sentence) can be used as inputs, and the large language model can be instructed to focus on analyzing the differences between the two and describe this improvement in an encouraging and suggestive tone.

[0099] Specifically, the following prompt text can be constructed and input into the large language model: "Please act as a friendly language coach. Your task is to compare the user's original sentence and the improved sentence, and generate a short explanation in a positive and encouraging tone to show why the improved way of saying is more natural or more idiomatic. Please do not use negative words such as 'wrong' or 'incorrect', and focus on 'better expression'."

[0100] Original sentence: [First text recognition result]; Improved sentence: [Second text recognition result]; Please generate your teaching prompt: ".

[0101] Based on this instruction, the large language model can generate positive grammar correction information such as "Great expression! If we adjust 'I went to an apple yesterday' a bit to 'I went to buy an apple yesterday', adding 'buy' and 'le' (了 in Chinese), it can more clearly express the completed action and sound more idiomatic." In this way, the dry text differences can be transformed into a language guidance material with a teaching touch and easy for users to accept.

[0102] In some embodiments, to improve the pertinence of the correction information, specific error type labels can be injected into the prompt text. For example, the prompt text can be increased with: "The grammar points involved: addition of the perfective marker 'le' and verb complementation." This can guide the large language model to generate more professional and accurate explanations.

[0103] In some embodiments, users can be segmented, and prompt texts with adjustable styles and complexities can be used for different users. Specifically, different prompt texts can be selected according to the user's learning profile (such as learning stage, age, preferences). For example, for children or novice learners, use the first prompt text that contains more emojis and simple vocabulary: "I understand what you just said! If we say 'I went to buy an apple yesterday' like this, doesn't it feel more like the story? The word 'le' here is like putting a full stop to the story." For advanced learners, a more concise and professional second prompt text can be used: "From 'I went to an apple yesterday' to 'I went to buy an apple yesterday', the main optimization lies in supplementing the specific verb 'buy' and adding the aspect marker 'le', making the completion of the action more explicit." In addition, the directness of the prompt can be dynamically adjusted according to the user's feedback on historical corrections (such as acceptance rate, repeated error rate), achieving a balance between gentle hints and "explicit suggestions" to achieve an accurate match between the corrective information and the user's cognitive level and psychological preferences.

[0104] In some embodiments, the interaction scenario corresponding to the first interaction audio can be directly obtained from the conversation metadata or through a scenario classification model, and a reply message that conforms to the role of the interaction scenario can be generated therefrom. Specifically, the following scenario dialogue prompt can be constructed: "You are playing the role of an [interaction scenario, e.g., coffee shop waiter]. Please generate a natural and friendly reply that conforms to your role identity based on the latest speech of the following user to continue the conversation. Please focus on the content of the conversation itself and do not conduct any grammar teaching for the time moment.

[0105] User speech: [Second text recognition result]; Please generate your reply: ".

[0106] Specifically, the large language model can fuse the positive grammar correction information and the reply message to generate the corresponding target feedback text. For example, the correction information can be "Great expression!... It will sound more native." and the reply message can be "Sounds fresh! Our apple pie in the store is also very popular. Do you want to try it today?". The large language model can generate: "Great expression! If we make a slight adjustment... It will sound more native. Speaking of apples, sounds fresh! Our apple pie in the store is also very popular. Do you want to try it today?". In this way, the positive grammar correction information can be connected to the dialogue part to output a complete target feedback text that is fluent, natural, and serves two purposes.

[0107] In some implementations, only one prompt text can be designed to guide the large language model to directly output a target feedback text containing positive grammatical correction information and response information based on the first text recognition result, the second text recognition result, and the interaction scenario. Specifically, the prompt text for generating positive grammatical correction information and response information described above can be integrated, which will not be elaborated here.

[0108] By using the above methods, grammar correction information can be organically integrated into the responses of scenario dialogues in a positive and encouraging manner, avoiding direct criticism of user errors. This maintains the natural flow of the dialogue and the user's learning enthusiasm, while effectively conveying grammar knowledge. In turn, it creates a supportive and immersive language learning environment for users, improving teaching effectiveness and interactive experience.

[0109] In some implementations, to adaptively adjust the tone and manner of teaching feedback based on the user's current emotional state to provide a more considerate and supportive interactive experience, the user's emotional category and intensity can be identified by analyzing the intonation information of the first interactive audio, generating emotional state features, and then inputting these features along with the text recognition results into a large language model to generate emotionally adapted target feedback text. For example, step 104 may further include: (104.b1) Obtain the intonation information of the first interactive audio, and determine the current sentiment category and sentiment intensity value of the target object based on the intonation information; (104.b2) Generate corresponding emotional state features based on emotional category and emotional intensity value; (104.b3) Input the first text recognition result, the second text recognition result and the emotional state features into the large language model to obtain the corresponding target feedback text.

[0110] The intonation information can be acoustic parameters extracted from the first interactive audio that reflect the prosodic features of speech, such as pitch curves, speech rate and volume changes, which can be used to analyze the speaker's emotional state.

[0111] Among them, the emotion category can be the user's main emotion type identified by analyzing tone information, such as joy, calmness, tension or frustration, which can be used to qualitatively describe the user's current emotional tendency.

[0112] Among them, the emotional intensity value can be a numerical indicator used to quantify the degree of emotional expression. For example, the tension level calculated by the algorithm is 0.8 (range 0-1), which can be used to measure the strength of a user's emotions.

[0113] Among them, the emotional state features can be structured emotional description data generated by combining emotional category and emotional intensity value. For example, a data structure containing "emotional category: tension, intensity: 0.8" can be used as input conditions for large language models to generate emotionally adapted responses.

[0114] In some implementations, the first interactive audio can be preprocessed, such as through pre-emphasis, framing, and windowing. Subsequently, for each frame or each syllable / word unit, a set of acoustic features that effectively reflect speech rhythm and emotion is extracted. These acoustic features include at least one of the following: fundamental frequency and its statistics, used to reflect pitch changes, such as mean, range, and standard deviation; pitch is typically higher and more variable when the user is excited; energy / loudness, used to reflect volume; energy is typically higher when the user is angry or emphasizing; speech rate, which can be calculated by the number of syllables per unit time or by detecting speech activity; speech rate may increase when tense or anxious; and spectral features, such as statistics of Mel-frequency cepstral coefficients, which can indirectly reflect the tension of the pronunciation.

[0115] For example, for a 2-second speech segment, the system can calculate its average fundamental frequency as 220Hz, its standard deviation as 45Hz, its average energy as high, and its speech rate as 6 syllables per second. Then, a pre-trained emotion classification model can be used to determine the current emotion category and intensity value of the target object.

[0116] Furthermore, the sentiment classification model can be a classifier based on deep neural networks (such as convolutional neural networks or recurrent neural networks). Its input is the aforementioned acoustic feature sequence (or statistical feature vectors extracted from the sequence), and its output is a discrete sentiment category label (such as "positive / neutral / negative" or more granular "joy, calm, frustrated, tense") and a continuous sentiment intensity value (such as 0.0 to 1.0). The training samples for the sentiment classification model can consist of a large number of speech samples with sentiment labels. In practical applications, the acoustic feature sequence (e.g., intonation feature vector) of the first interactive audio is input into the model to obtain the predicted sentiment category and sentiment intensity value. This achieves an objective and quantitative perception of the user's current emotional state, providing a key input for personalized feedback.

[0117] For example, the emotion category and emotion intensity value can be transformed into a structured representation to facilitate input into a large language model for conditional control. For instance, it can be converted into textual information, such as: "The current user's emotion category is tense, and the emotion intensity is 0.75."

[0118] In some implementations, the first text recognition result, the second text recognition result, and emotional state features and / or interaction scenario can be jointly input into a large language model to obtain the corresponding target feedback text. For example, the following prompt text can be constructed: "You are a [role] working in an [interactive scenario, such as a coffee shop], and also a meticulous language coach. Please generate a response for the user based on the following information:" 1. User's original expression: [First text recognition result]; 2. More idiomatic expression suggestions: [Second text recognition results]; 3. User's current emotional state: [Emotional state characteristics, such as: sounds somewhat nervous]; Please generate a response with the following requirements: First, use an appropriate tone of voice based on the user's emotional state (e.g., offer more encouragement to nervous users).

[0119] Then, incorporate more authentic expressions into the conversation in a very natural and unobtrusive way.

[0120] Finally, based on the characters' identities in the scene, continue the dialogue in a friendly manner.

[0121] After understanding this prompt, the large language model simultaneously considers semantic content (original sentence and corrected sentence), teaching task (integration suggestions), and emotional constraints (tone adjustment) to generate a three-in-one response. For example, for the emotional state feature "nervous, intensity 0.75", the model might generate: "Don't be nervous, just speak slowly. What you just said, 'I went to an apple yesterday,' is very clear. If you changed it to 'I bought an apple yesterday,' it would sound more fluent. Relax, just like chatting. So, how did that apple taste?" In this way, by using emotional state as a generation condition, the large language model can dynamically adjust the "social-emotional" dimension of its feedback, upgrading from simple content correctness feedback to a holistic teaching interaction that combines emotional support and content guidance.

[0122] In some implementations, a short-term emotional state sequence can be maintained (e.g., the emotional category and intensity values ​​of the last three rounds of conversation). Based on this emotional state sequence, the system can infer the trend of changes in the user's emotions (e.g., a continuously increasing level of tension, a shift from frustration to calm, etc.). When generating target feedback text, the emotional state features are expanded to include a description of the trend. For example, if the emotional state feature indicates that the user's tension is increasing, the large language model will adjust its strategy accordingly. For instance, for a user whose tension is continuously increasing, the system may decide to temporarily suspend grammar correction in the current turn and prioritize using simpler, more supportive dialogue to alleviate anxiety (e.g., "It's okay, we'll take it slow. You want to talk about apples, right?"). This dynamic decision-making based on emotional trends makes teaching interventions more intelligent and humane.

[0123] In the above way, the user's emotional state can be perceived in real time, and this non-verbal information can be incorporated into the feedback generation process. Thus, the grammar corrections and scenario responses generated by the large language model can be made to match the user's emotions in terms of tone, diction, and encouragement level, thereby achieving a more empathetic and personalized teaching interaction and enhancing the user's psychological security and engagement during the language learning process.

[0124] Step 105: Output a corresponding second interactive audio to the target object based on the target feedback text.

[0125] In some embodiments, in order to convert the text feedback containing grammar corrections and scenario dialogues into natural speech interaction to complete the teaching loop and enhance the user experience, the target feedback text can be converted into a corresponding second interactive audio through text-to-speech technology and played to the target object to achieve multi-modal and immersive language teaching interaction.

[0126] Among them, the second interactive audio can be a voice output that simulates an artificial intelligence dialogue character generated after converting the target feedback text through a text-to-speech engine.

[0127] Specifically, through text-to-speech (TTS) technology, the target feedback text can be converted into a natural and fluent second interactive audio. For example, front-end text analysis can be performed on the target feedback text, including text regularization (such as converting the number "123" to "one hundred and twenty-three"), word segmentation, part-of-speech tagging, and disambiguation of polyphonic characters. Then, prosody prediction is carried out, and a prosody model (usually based on deep learning) is used to predict the pronunciation duration, fundamental frequency (pitch) contour, and energy (volume) change of each word in the text. These information together determine whether the rhythm and intonation of the synthesized speech are natural. Next, acoustic feature generation is carried out, and an acoustic model (such as a sequence-to-sequence model like Tacotron, FastSpeech, etc.) is used to generate intermediate acoustic features (such as Mel spectrogram) based on the text and the predicted prosody information. Finally, a vocoder (such as WaveNet, HiFi-GAN) is used to convert the Mel spectrogram into the final audio waveform, that is, the second interactive audio. Exemplarily, for the target feedback text "That sounds great! By the way, you just said...", the TTS engine finally outputs a natural speech of about 5 seconds. In this way, the conversion from the intelligently generated target feedback text to the perceivable speech feedback can be completed, realizing the closed-loop of human-computer interaction and providing the user with an immersive learning experience in the auditory dimension.

[0128] In some implementations, the target feedback text can also be converted into a second interactive audio using other technologies, such as end-to-end neural speech synthesis technology (e.g., VITS, NaturalSpeech), streaming speech synthesis technology, etc. This application does not limit the specific conversion method.

[0129] In some implementations, the generation of the second interactive audio can reflect the user's emotional category. Specifically, when invoking the TTS engine, in addition to providing the target feedback text, a style embedding vector can be provided. This style embedding vector can be derived from the user's emotional state (such as "nervous") and mapped to a "gentle, soothing" speech style. The TTS engine's acoustic model, trained on multi-style data, can synthesize speech with corresponding emotional coloring or role characteristics based on the input style embedding vector. For example, for a nervous user, a slower, softer, and encouraging speech can be synthesized. In this way, the second interactive audio can achieve a high degree of consistency in content, emotion, and role identity, greatly enhancing the realism and immersion of the interaction.

[0130] This application obtains a phoneme sequence by acquiring a first interactive audio input from a target object and performing phoneme analysis on the first interactive audio; it then acquires the target object's associated historical interactive text and inputs the phoneme sequence and historical interactive text into a large language model for semantic analysis to obtain a first text recognition result corresponding to the phoneme sequence; it performs grammatical correction on the first text recognition result to obtain a second text recognition result; and through the large language model, it generates a corresponding target feedback text based on the first and second text recognition results. The target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio; finally, it outputs the corresponding second interactive audio to the target object based on the target feedback text. In this way, the accuracy of text recognition can be improved through phoneme sequence and context-aware semantic analysis, reducing recognition errors caused by inaccurate pronunciation or accents, and achieving natural, non-interrupted interactive teaching feedback through the fusion of grammatical correction and semantic response. Specifically, phoneme analysis preserves the original acoustic features, avoiding text conversion errors caused by inaccurate pronunciation or accents in traditional automatic speech recognition; the large language model, combined with historical interactive text, performs semantic reasoning on phoneme sequences, effectively improving robustness in recognizing the user's true intent; thus, grammatical correction based on accurate first text recognition results effectively prevents erroneous text from propagating to subsequent analyses; the generated feedback text naturally integrates grammatical correction information into the scenario response, enabling feedback to maintain dialogue coherence while specifically pointing out errors, improving the accuracy of teaching and interactive guidance and the target audience's experience. In summary, this application can improve the reliability and effectiveness of voice-interactive teaching.

[0131] In some embodiments, in order to convert technical pronunciation diagnosis into easy-to-understand and executable teaching guidance, the acoustic features of the user's audio can be analyzed in detail based on the standard text, and the analysis results can be input into a large language model to be converted into natural language suggestions for the user and then output, so as to complete the intelligent conversion from machine evaluation to humanized teaching feedback. Exemplarily, after outputting the corresponding second interactive audio to the target object based on the target feedback text, it further includes: (A.1) Analyze the acoustic features of the first interactive audio based on the first text recognition result to obtain a pronunciation recognition result; (A.2) Input the pronunciation recognition result into the large language model to obtain the corresponding pronunciation correction suggestion information; (A.3) Output the pronunciation correction suggestion information to the target object.

[0132] Among them, the pronunciation recognition result can be structured analysis data generated by aligning and comparing the first interactive audio, including an overall score and detailed scores and error types for each phoneme. For example, a JSON object containing a list of overall_score, word_scores, and phoneme_scores can be used to accurately quantify the quality of the user's pronunciation in dimensions such as initials, finals, and tones.

[0133] Among them, the pronunciation correction suggestion information can be pronunciation improvement guidance text generated by the large language model based on the pronunciation recognition result, expressed in an encouraging tone and easy-to-understand language. For example, "You need to pay attention to the initial of the word'shì'. The standard pronunciation'sh' requires the tip of the tongue to curl up..." It can be used to intuitively explain the pronunciation problem to the user and provide specific practice methods.

[0134] Exemplarily, the forced alignment algorithm can be used to precisely match the first text recognition result (such as "I am a student") as the "standard answer" and its corresponding standard pinyin sequence (such as "wǒ shì xué shēng") with the first interactive audio on the time axis. This algorithm can calculate the most likely start and end times of each phoneme (initial, final, tone) in the audio waveform through a hidden Markov model or a deep neural network. Exemplarily, it can allocate the time period 0.15 - 0.55 seconds for the word "wǒ", 0.60 - 0.95 seconds for the word "shì", etc.

[0135] Subsequently, within the time period determined by forced alignment, acoustic features of the user's actual pronunciation can be extracted, such as Mel-frequency cepstral coefficients, fundamental frequency curves, etc. For each phoneme unit, the extracted acoustic features are input into a pre-trained pronunciation quality assessment model (such as a model based on the GOP algorithm). This model calculates the pronunciation quality score of the phoneme by comparing the likelihood of the user's features with the standard pronunciation model (trained from a large amount of standard pronunciation data). The evaluation dimensions cover the accuracy of initials, finals, and the correctness of tones.

[0136] Finally, the system aggregates the scores of all phonemes, syllables, and even words to generate a structured pronunciation recognition result. This result can be in JSON format, including the overall score, the scores of each word, the detailed scores of each phoneme, error types (such as "initial substitution", "tone error"), and a description of possible acoustic deviations. Exemplarily, structured data as shown in the legend of the disclosure document is output, precisely indicating that the initial "sh" of the character "是" is pronounced as "s", and the tone of the character "学" is incorrect. In this way, the transformation from the original audio to refined and interpretable pronunciation analysis data can be achieved. <着

[0137] In some embodiments, the technical pronunciation recognition result (such as JSON data) can be converted into a natural language description, or directly used as part of a structured prompt and input into a large language model. For example: "You are a Chinese pronunciation coach. Please generate an encouraging and easy-to-understand pronunciation improvement suggestion based on the following pronunciation analysis report of a learner. Please focus on one or two points that need improvement the most, explain the error in simple language, and give specific practice methods.

[0138] [[ID=(11]]Pronunciation diagnosis report: [Insert the textual summary or key data of the pronunciation recognition result here]; Please generate your suggestion:".

[0139] Furthermore, based on its language generation ability and teaching knowledge, the large language model can output pronunciation correction suggestion information such as "Your pronunciation is great overall! For the word '是', you can pay a little more attention. Its initial is 'sh', and you need to curl the tip of your tongue up. You pronounced it a bit like's'. You can try saying '诗-人' to feel the pronunciation position of'sh'. Additionally, the tone of the character '学' is the second tone and should rise. You read it a bit like the first tone." In this way, the cold numerical scores can be transformed into warm and actionable teaching guidance, enhancing the user's learning experience and correction efficiency.

[0140] In some implementations, pronunciation correction suggestions can be directly output to the user as a text report. Alternatively, the pronunciation correction suggestions can be converted into interactive audio and output to the user, using the same method as described above for converting the target feedback text into second interactive audio. Other conversion methods may also be used, which will not be elaborated here.

[0141] In some implementations, the generation of pronunciation correction suggestions can incorporate a personalized dynamic threshold mechanism. The system dynamically adjusts the score threshold for judging pronunciation errors based on the target user's historical pronunciation level. For beginners, the score threshold is relaxed, focusing only on the most serious errors; for advanced users, the score threshold is tightened, enabling the identification of more subtle deviations. When generating pronunciation correction suggestions, the large language model is informed of the current user level and the type of error being focused on, thus generating suggestions that match the difficulty and emphasis. For example, for advanced users, instead of explaining the tongue position of "sh," the system directly compares the pronunciation with the formant of the standard vowel. This achieves precise personalization of the suggestions.

[0142] By using the above methods, professional and abstract acoustic parameters and scores can be transformed into concrete and vivid teaching language. This significantly lowers the barrier for users to understand pronunciation analysis results and provides actionable suggestions, thereby achieving effective transmission and teaching transformation of pronunciation assessment results and improving learning efficiency and user experience in the pronunciation correction process.

[0143] In some implementations, to achieve fine-grained and accurate text-based evaluation of user pronunciation, the word elements in the first text recognition result can be forcibly aligned with the first interactive audio on the time axis to determine the pronunciation time segment of each word element. The acoustic features of each segment can then be analyzed to obtain independent pronunciation recognition sub-results for each word element, which are then aggregated to obtain a comprehensive pronunciation recognition result. For example, (A.1) may include: (A.1.1) Based on the first text recognition result, determine the start time index and end time index of each word element in the first interactive audio, and determine the pronunciation segment corresponding to each word element from the first interactive audio based on the start time index and end time index; (A.1.2) By performing acoustic feature analysis on each word element and its corresponding pronunciation segment, the pronunciation recognition sub-result of each word element is obtained; (A.1.3) Based on the pronunciation recognition sub-results corresponding to each word element contained in the first text recognition result, the pronunciation recognition result corresponding to the first text recognition result is obtained.

[0144] Among them, word elements can be the smallest semantic or grammatical units that constitute the first text recognition result, such as a Chinese character, a word, or a syllable, which can be used as the basic unit for pronunciation evaluation and analysis.

[0145] Among them, the start time index can be the time point identifier when the word element starts to be pronounced in the first interactive audio. For example, the word "面" starts to be pronounced at 0.15 seconds in the audio obtained through the forced alignment algorithm, which can be used to locate the starting point of the pronunciation segment.

[0146] Among them, the end time index can be the time point identifier when the word element ends to be pronounced in the first interactive audio. For example, the word "面" ends at 0.55 seconds in the audio, which can be used to locate the ending point of the pronunciation segment.

[0147] Among them, the pronunciation segment can be the audio segment corresponding to a specific word element intercepted from the first interactive audio according to the start time index and the end time index. For example, the audio segment from 0.15 seconds to 0.55 seconds corresponds to the pronunciation of the word "面", which can be used for independent acoustic analysis.

[0148] Among them, the pronunciation recognition sub-result can be the pronunciation quality evaluation data of the word element obtained through acoustic feature analysis of the word element and its corresponding pronunciation segment. For example, the structured data containing phoneme scores, error types, and correction suggestions can be used to reflect the specific pronunciation situation of the word element.

[0149] Specifically, preprocessing such as word segmentation can be performed on the first text recognition result to obtain a series of word elements (such as the word "苹果"), and generate its standard phoneme sequence. At the same time, frame-level acoustic features (such as MFCC) are extracted from the first interactive audio. Then, a decoding network is constructed using a pre-trained acoustic model and a pronunciation dictionary, and the Viterbi algorithm (such as the Viterbi algorithm) is used to search for the optimal path in the decoding network. The optimal path assigns precise start time indices and end time indices to each word element and its phonemes. Based on these time indices, the system can intercept the corresponding pronunciation segment from the first interactive audio. For example, the word element "苹果" may be aligned and intercepted as an audio segment in the time interval [1.0s, 1.5s]. In this way, the fine-grained time alignment between text and speech is achieved, laying a foundation for subsequent unitary analysis.

[0150] Furthermore, for the pronunciation segment of each word element, fine acoustic features such as its Mel spectrum and fundamental frequency curve can be extracted and compared with the standard pronunciation acoustic model of the word element (or phoneme). A pronunciation scoring model (such as based on the GOP algorithm) is used to calculate the pronunciation quality score. Specifically, the phoneme-level score can be calculated by the following formula: Score(phoneme)=(L_user-L_ubm) / N_frames; Among them, Score(phoneme) is the phoneme score, L_user is the log-likelihood of the user features under the standard model, L_ubm is the log-likelihood under the universal background model, and N_frames is the number of frames.

[0151] By analyzing the score and feature differences (such as the deviation of the fundamental frequency curve from the standard tone template), the specific error types of initial consonants, vowels or tones can be analyzed, and a structured pronunciation recognition sub-result can be generated for each word element. For example, for the word "是", its pronunciation recognition sub-result can be: {"word": "是", "score": 45.0, "error_detail": {"initial_consonant": {"actual": "s", "expected": "sh"}}}.

[0152] Exemplarily, the pronunciation recognition sub-results of all word elements in the first text recognition result can be integrated into a hierarchical data structure (such as JSON) in word order to form the final pronunciation recognition result. The pronunciation recognition result includes the overall score of the sentence, fluency metrics, and a list of detailed scores and error analyses that can be drilled down to each word and phoneme. In this way, a comprehensive and structured pronunciation analysis report is generated, supporting a full range of analysis from a macro overview to micro error localization.

[0153] In some embodiments, in order to more accurately detect fluency problems such as involuntary repetition of the user, relevant features of speech pathology can be introduced during acoustic feature analysis. In addition to conventional features, the system additionally calculates the coefficient of variation of vowel duration, consonant occlusion duration, frequency and duration of silent pauses, etc. for each pronunciation segment. By comparing with the user's historical baseline or the standard fluent speech model, abnormal hesitation, repetition or prolongation patterns can be detected. When generating the pronunciation recognition sub-result, an additional "fluency anomaly flag" and description (such as "There is an abnormal pause of 300ms before '苹果', which may affect fluency") will be included. This enables the system to not only evaluate pronunciation accuracy but also diagnose expression fluency problems, with a more comprehensive teaching dimension.

[0154] Through the above methods, the user's overall speech input can be decomposed into pronunciation units that precisely correspond to the text for independent analysis. In this way, fine-grained and high-precision pronunciation quality data can be obtained, accurately locating specific errors at the level of initial consonants, vowels or tones, and further providing a solid data foundation for generating personalized and actionable pronunciation diagnosis reports and feedback suggestions, improving the effectiveness of pronunciation teaching.

[0155] Please refer to Figure 3 and below in combination with Figure 3The following describes the overall embodiments of this application. A user provides first interactive audio through a voice input device (such as a microphone), and a user voice processing module preprocesses and extracts features from the first interactive audio. Specifically, a phoneme recognition module performs phoneme analysis on the first interactive audio and outputs a phoneme sequence reflecting the user's actual pronunciation characteristics. This phoneme sequence, along with historical interactive text obtained from the conversation history, is input into a Large Language Model (LLM) for semantic recognition (LLM recognition). By combining the context, the user's intent is inferred to obtain the first text recognition result.

[0156] Furthermore, processing of the grammar and pronunciation channels can be performed in parallel. In the grammar channel, the first text recognition result can be grammatically corrected based on preset grammar rules to generate a second text recognition result that conforms to language norms. In the pronunciation channel, the original audio can be finely aligned and acoustically analyzed based on the first text recognition result. For example, a structured pronunciation evaluation report can be generated using the Goodness of Pronunciation (GOP) algorithm to quantitatively evaluate the pronunciation quality of each phoneme or word element.

[0157] Furthermore, while playing a specific dialogue role (as played by the LLM), the large language model can integrate the results of the first text recognition, the second text recognition, and an optional pronunciation evaluation report to generate a summary text (LLM summary) that incorporates positive grammar correction information, contextualized response content, and optional pronunciation guidance suggestions. This text is then converted into a second interactive audio output to the user via a TTS (Text-to-Speech) engine, completing the voice interaction loop. Optionally, the evaluation report generated by the pronunciation channel can also be converted into easily understandable pronunciation correction suggestions (pronunciation report) by the large language model and fed back to the user in text or voice format.

[0158] This embodiment preserves the original acoustic information through phoneme sequences, enhances the robustness of semantic recognition by leveraging the contextual understanding capabilities of a large language model, and achieves accurate and personalized language correction and teaching guidance while maintaining natural and fluent dialogue through parallel processing and fusion feedback of grammar and pronunciation channels, thereby improving the reliability and teaching effectiveness of the voice interaction system.

[0159] Please see Figure 4 This application also provides a voice interaction device that can implement the above-described voice interaction method. The voice interaction device includes: The acquisition module 41 is used to acquire the first interactive audio input by the target object and perform phoneme analysis on the first interactive audio to obtain a phoneme sequence; Analysis module 42 is used to obtain the historical interaction text associated with the target object, and input the phoneme sequence and historical interaction text into the large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence; Correction module 43 is used to perform grammatical correction on the first text recognition result to obtain the second text recognition result; The generation module 44 is used to generate corresponding target feedback text based on the first text recognition result and the second text recognition result through a large language model. The target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio. Output module 45 is used to output the corresponding second interactive audio to the target object based on the target feedback text.

[0160] The specific implementation of this voice interaction device is basically the same as the specific embodiments of the voice interaction method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the voice interaction device may also be equipped with other functional modules to implement the voice interaction method in the above embodiments.

[0161] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described voice interaction method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0162] Please see Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 51 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 52 can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 52 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 52 and is called and executed by the processor 51 to execute the voice interaction method of the embodiments of this application. Input / output interface 53 is used to implement information input and output; The communication interface 54 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 55 transmits information between various components of the device (e.g., processor 51, memory 52, input / output interface 53, and communication interface 54); The processor 51, memory 52, input / output interface 53, and communication interface 54 are connected to each other within the device via bus 55.

[0163] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described voice interaction method.

[0164] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0166] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0169] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0170] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0171] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0172] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A voice interaction method, characterized in that, The method includes: The first interactive audio input from the target object is obtained, and phoneme analysis is performed on the first interactive audio to obtain a phoneme sequence; Obtain the historical interaction text associated with the target object, and input the phoneme sequence and the historical interaction text into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence; The first text recognition result is subjected to grammatical correction to obtain the second text recognition result; Based on the first text recognition result and the second text recognition result, the large language model generates a corresponding target feedback text, wherein the target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio. Based on the target feedback text, the corresponding second interactive audio is output to the target object.

2. The voice interaction method according to claim 1, characterized in that, After outputting the corresponding second interactive audio to the target object based on the target feedback text, the method further includes: Based on the first text recognition result, acoustic feature analysis is performed on the first interactive audio to obtain pronunciation recognition result; The pronunciation recognition results are input into the large language model to obtain corresponding pronunciation correction suggestions. Output the pronunciation correction suggestion information to the target object.

3. The voice interaction method according to claim 2, characterized in that, The step of performing acoustic feature analysis on the first interactive audio based on the first text recognition result to obtain pronunciation recognition results includes: Based on the first text recognition result, the start time index and end time index of each word element in the first interactive audio are determined, and the pronunciation segment corresponding to each word element is determined from the first interactive audio based on the start time index and the end time index. By performing acoustic feature analysis on each word element and its corresponding pronunciation segment, a pronunciation recognition sub-result for each word element is obtained; Based on the pronunciation recognition sub-results corresponding to each word element contained in the first text recognition result, the pronunciation recognition result corresponding to the first text recognition result is obtained.

4. The voice interaction method according to claim 1, characterized in that, The step of inputting the phoneme sequence and the historical interactive text into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence includes: Obtain a phoneme pronunciation profile of the target object, wherein the phoneme pronunciation profile is used to record the mapping relationship between the confused pronunciation phonemes output by the target object and the standard phonemes; Based on the phoneme pronunciation profile, the corresponding target confused pronunciation phoneme is extracted from the phoneme sequence, and the target standard phoneme corresponding to the target confused pronunciation phoneme in the phoneme pronunciation profile is extracted. Based on the target standard phoneme, replace the target confused pronunciation phoneme in the phoneme sequence to obtain at least one candidate phoneme sequence; The at least one candidate phoneme sequence, the phoneme sequence, and the historical interactive text are input into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence.

5. The voice interaction method according to claim 1, characterized in that, The step of performing grammatical correction on the first text recognition result to obtain the second text recognition result includes: Obtain preset grammatical structure rules, and analyze the first text recognition result based on the grammatical structure rules to obtain the corresponding grammatical error elements; The interaction scene corresponding to the first interactive audio is obtained, and based on the interaction scene and the grammatical error elements, the grammatical correction of the first text recognition result is performed to obtain the corresponding second text recognition result.

6. The voice interaction method according to claim 1, characterized in that, The step of generating corresponding target feedback text based on the first text recognition result and the second text recognition result using the large language model includes: Based on the second text recognition result, positive grammar correction information corresponding to the first text recognition result is generated using the large language model. Obtain the interaction scene corresponding to the first interactive audio, and perform dialogue extension and dialogue response on the second text recognition result based on the interaction scene to obtain the response information corresponding to the first interactive audio; Based on the positive grammar correction information and the response information, a corresponding target feedback text is generated.

7. The voice interaction method according to claim 1, characterized in that, The step of generating corresponding target feedback text based on the first text recognition result and the second text recognition result using the large language model includes: Obtain the intonation information of the first interactive audio, and determine the current emotion category and emotion intensity value of the target object based on the intonation information; Based on the emotion category and the emotion intensity value, a corresponding emotion state feature is generated; The first text recognition result, the second text recognition result, and the emotional state features are input into the large language model to obtain the corresponding target feedback text.

8. A voice interaction device, characterized in that, The device includes: The acquisition module is used to acquire the first interactive audio input by the target object, and to perform phoneme analysis on the first interactive audio to obtain a phoneme sequence; The analysis module is used to obtain the historical interaction text associated with the target object, and input the phoneme sequence and the historical interaction text into a large language model for semantic analysis to obtain the first text recognition result corresponding to the phoneme sequence; The correction module is used to perform grammatical correction on the first text recognition result to obtain the second text recognition result; The generation module is used to generate corresponding target feedback text based on the first text recognition result and the second text recognition result through the large language model, wherein the target feedback text includes grammatical correction information of the second text recognition result relative to the first text recognition result and response information corresponding to the first interactive audio; The output module is used to output the corresponding second interactive audio to the target object based on the target feedback text.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the voice interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the voice interaction method according to any one of claims 1 to 7.

Images