Intelligent voice dialogue method and system based on large language model

Abstract

This application discloses an intelligent voice dialogue method and system based on a large language model, belonging to the field of speech processing technology. The method includes: acquiring an audio stream of the user's surrounding environment; processing the audio stream by frame and determining whether it contains speech to obtain a valid audio stream; converting the valid audio stream into a text string using a deep learning model; performing deep analysis using a large language model to dynamically parse the user's immediate state, potential needs, and emotional inclinations, generating natural language response text, and converting it into an audio stream for playback using speech synthesis technology. This method leverages the powerful natural language understanding capabilities of the large language model to dynamically and deeply understand the user's immediate state, potential needs, and emotional tone, forming a richer, more vivid, and context-dependent user understanding. This results in a more natural, considerate, and accurate response generation and service provision, achieving a more human-centered human-computer voice interaction experience.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to an intelligent speech dialogue method and system based on a large language model. Background Technology

[0002] Currently, in the field of user profile building, especially for the initial profile creation stage that requires active user participation, the "tag selection method" is a widely used basic approach. The core mechanism of this method is: pre-define a series of tags, keywords, or interest categories that are highly relevant to a specific business area; new users (usually after registration) are guided to a selection interface to browse and select preset options that match their own characteristics, interests, or needs; the system then aggregates and forms a preliminary user profile based on the user's selection results, and provides initial personalized services accordingly.

[0003] Although the label selection method is widely used because of its simplicity and clear logic, it also shows some inherent technical limitations in practical applications: (1) This method requires users to accurately understand the meaning of each preset label and the category it represents. When the label system is large in scale and complex in hierarchy, or when the label description is relatively abstract and professional, users need to invest significant cognitive resources in identification and screening. This process objectively increases the user's learning cost and cognitive burden; (2) The effective interaction of the label selection interface depends heavily on the user's understanding and selection ability. For children whose cognitive abilities are still developing and elderly people whose cognitive abilities have declined, For users with specific cognitive impairments, understanding the meaning of the label text and locating their own preferences among many options often pose significant challenges. This results in the method being insufficient in coverage and inclusiveness when serving a diverse range of users in the whole society. (3) Faced with a large number of options, users may make quick, unthinking selections or even random choices due to lack of patience or pursuit of operational efficiency. This behavior pattern will directly affect the accuracy and completeness of the constructed profile. At the same time, the entire selection process is usually task-driven and has weak interactivity and fun, which may reduce users' willingness to participate actively and overall experience satisfaction. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this application provides an intelligent voice dialogue method and system based on a large language model, which solves the problems of existing user profile construction methods in terms of ease of use, inclusiveness, information richness, data robustness, and subsequent processing.

[0005] To achieve the aforementioned objectives, the technical solution adopted in this application is as follows: First aspect: This application provides an intelligent voice dialogue method based on a large language model, including: S1: Obtain the audio stream of the user's surrounding environment; S2: Perform frame segmentation on the audio stream and determine whether the segmented audio stream contains speech to obtain a valid audio stream containing speech. S3: Receive a valid audio stream containing speech and convert the valid audio stream into a text string using a deep learning model; S4: Based on a large language model, acquire and analyze data from text strings, dialogue history, and semi-structured question inputs, and generate natural language response text based on the analysis results; S5: Converts natural language response text into a synthesized speech audio stream; S6: Play back the synthesized speech audio stream.

[0006] Further, S2 includes: S201: Receive a continuous audio stream and perform frame segmentation on the continuous audio stream; S202: Extract speech features and non-speech features from each frame of the audio stream based on its acoustic features; S203: Based on the extracted features, determine whether the current frame contains speech; S204: Based on the continuous frame-level judgment results, determine the start and end of the speech segment to obtain a valid audio stream containing speech.

[0007] Further, S3 includes: S301: Receive the valid audio stream containing speech using an acoustic model, and map the valid audio stream containing speech into an acoustic unit to obtain all text symbol sequences corresponding to the valid audio stream; S302: Based on historical text, use a language model to rearrange all text symbol sequences to obtain rearranged text symbol sequences; S303: Based on all text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model, a text string is generated using a decoder.

[0008] Furthermore, the text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model are used to generate text strings using a decoder, including: The beam search algorithm is used to calculate the joint score of all text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model, and the text symbol sequence with the highest score is taken as the final text string.

[0009] Further, S4 includes: S401: Combine the text string, dialogue history, and semi-structured question input data to obtain combined multi-source information; S402: Based on the Transformer network architecture, perform natural language understanding on the combined multi-source information; S403: Based on the understood multi-source information, gain a deeper understanding of the context to obtain the core content of the next action and the next response; S404: Integrate the core content of the response to obtain the natural language response text.

[0010] Furthermore, the input data for the semi-structured problem is one or more external information sources, including: Relevant information fragments obtained from knowledge base or database retrieval based on text strings, user history profiles or preferences, task templates or API descriptions required for specific tasks, and system status or external environment status.

[0011] Furthermore, it also includes: Based on the semi-structured input data and user intent, perform structured output tasks to obtain structured data or instructions; The structured data or instructions include: API call parameters, database operation instructions, form filling, and formatted data output.

[0012] Further, S5 includes: S501: Perform text front-end processing on the generated natural language response text; S502: Based on an acoustic model, the processed natural language response text is converted into an acoustic feature sequence; S503: Based on a vocoder, acoustic feature sequences are synthesized into audible audio waveforms to obtain a synthesized speech audio stream.

[0013] Furthermore, the text front-end processing includes: text normalization, word segmentation, part-of-speech tagging, polyphonic character disambiguation, and prosody prediction.

[0014] The second aspect: This application provides an intelligent voice dialogue system based on a large language model, including: a voice activity monitoring module, a speech-to-text module, a large language model processing core module, a text-to-speech module, and a playback module; The voice activity monitoring module receives the raw audio stream of the user's surrounding environment in real time, and identifies the speech segments and non-speech segments of the raw audio stream based on a neural network model to obtain an effective audio stream containing speech. The speech-to-text module includes an acoustic model, a language model, and a decoder. The acoustic model receives a valid audio stream containing speech and maps the valid audio stream containing speech to acoustic units to obtain all corresponding text symbol sequences. The language model rearranges the text symbol sequences to obtain rearranged text symbol sequences. The decoder combines the outputs of the acoustic model and the language model and uses a beam search algorithm to obtain a text string. The core module of the large language model processing acquires and analyzes text strings, dialogue history, and semi-structured question inputs from external information sources, and generates natural language response text based on the analysis results; The text-to-speech module includes a text front-end processing module, an acoustic module, and a vocoder. The text front-end processing module processes the natural language response text, and the acoustic model converts the processed natural language response text into an acoustic feature sequence. Based on the vocoder, the acoustic feature sequence is converted into a synthesized speech audio stream. The playback module plays the synthesized audio stream.

[0015] The beneficial effects of this application are: This application provides an intelligent voice dialogue method and system based on a large language model. Through the powerful natural language understanding capabilities of the large language model, it can dynamically and deeply understand the user's immediate state, potential needs, emotional tone, and contextual relationships from the user's real-time voice dialogue, historical dialogue, and semi-structured question input data, forming a richer, more vivid, and context-dependent user understanding. Based on this deeper and more dynamic understanding of the user, this invention can generate responses and provide services in a more natural, considerate, and accurate manner. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart illustrating an intelligent voice dialogue method based on a large language model, provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0019] Example 1: This application provides an intelligent voice dialogue method based on a large language model, which can be found in [reference needed]. Figure 1 , Figure 1 The diagram shown is a flowchart illustrating an intelligent voice dialogue method based on a large language model provided in an embodiment of this application, including: S1: Obtain the audio stream of the user's surrounding environment.

[0020] In one embodiment of this application, the audio stream may be an audio stream that is captured in real time from the user's surrounding environment by a device such as a microphone.

[0021] S2: Perform frame segmentation on the audio stream and determine whether the segmented audio stream contains speech to obtain a valid audio stream containing speech.

[0022] Further, S2 includes: S201: Receive a continuous audio stream and perform frame segmentation on the continuous audio stream; S202: Extract speech features and non-speech features from each frame of the audio stream based on its acoustic features; S203: Based on the extracted features, determine whether the current frame contains speech; S204: Based on the continuous frame-level judgment results, determine the start and end of the speech segment to obtain a valid audio stream containing speech.

[0023] In one embodiment of this application, the acquisition of the effective audio stream can be a conventional method: based on acoustic features of the audio signal, such as energy (e.g., short-time energy, logarithmic energy), zero-crossing rate (ZCR), fundamental frequency (pitch), spectral entropy, linear predictive coding (LPC) residuals, a threshold or a simple state machine (e.g., silence -> possible speech -> speech -> possible silence -> silence) is used for judgment. For example, when the energy of several consecutive frames exceeds a preset threshold, it is determined as the start of speech; otherwise, it is determined as the end of speech (usually there is a silence duration parameter, such as 700ms-1500ms, to confirm the end of a sentence). Alternatively, it can be based on machine learning or deep learning methods: using support vector machines (SVM), Gaussian mixture models (GMM), or more advanced deep neural networks such as recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), convolutional neural networks (CNN), or Transformer architectures. These models are typically trained on large amounts of audio data labeled with both speech and non-speech segments. They can learn to distinguish speech and non-speech features directly from the raw audio waveform or its spectrogram (such as a Mel spectrogram), resulting in higher accuracy and noise robustness. For example, Google WebRTC VAD or Silero-VAD are applications of this technology.

[0024] In one embodiment of this application, the audio stream is processed by framing, which may be 10ms, 20ms or 30ms per frame.

[0025] S3: Receives a valid audio stream containing speech and uses a deep learning model to convert the valid audio stream into a text string.

[0026] Further, S3 includes: S301: Receive the valid audio stream containing speech using an acoustic model, and map the valid audio stream containing speech into an acoustic unit to obtain all text symbol sequences corresponding to the valid audio stream; S302: Based on historical text, use a language model to rearrange all text symbol sequences to obtain rearranged text symbol sequences; S303: Based on all text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model, a text string is generated using a decoder.

[0027] Furthermore, the text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model are used to generate text strings using a decoder, including: The beam search algorithm is used to calculate the joint score of all text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model, and the text symbol sequence with the highest score is taken as the final text string.

[0028] In one embodiment of this application, the acoustic model is the core component, responsible for mapping audio features (such as Mel-frequency cepstral coefficients (MFCC) and filter bank energy (FBank) to acoustic units (such as phonemes, syllables, or characters). Commonly used model structures include RNNs, LSTMs, or GRUs based on Connectionist Temporal Classification (CTC) loss, or Encoder-Decoder structures based on attention mechanisms (such as Listen, Attend, and Spell-LAS, or Transformer Transducer). The language model is used to improve the fluency and accuracy of recognition. It predicts the probability of word sequences by learning from a large amount of text corpus. It can be integrated with the acoustic model for decoding (shallow fusion) or for reordering multiple candidate results output by the acoustic model (deep fusion). The decoder combines the outputs of the acoustic model and the language model, using algorithms such as Beam Search to find the most likely text sequence.

[0029] To achieve real-time dialogue, streaming processing can usually be selected, which means receiving audio chunks while recognizing them, and being able to output intermediate (non-final) recognition results and final recognition results.

[0030] S4: Based on a large language model, acquire and analyze data from text strings, dialogue history, and semi-structured question inputs, and generate natural language response text based on the analysis results.

[0031] Further, S4 includes: S401: Combine the text string, dialogue history, and semi-structured question input data to obtain combined multi-source information; S402: Based on the Transformer network architecture, perform natural language understanding on the combined multi-source information; S403: Based on the understood multi-source information, gain a deeper understanding of the context to obtain the core content of the next action and the next response; S404: Integrate the core content of the response to obtain the natural language response text.

[0032] The semi-structured problem input data consists of one or more external information sources, including: Relevant information fragments obtained from knowledge base or database retrieval based on text strings, user history profiles or preferences, task templates or API descriptions required for specific tasks, and system status or external environment status.

[0033] Furthermore, it also includes: performing structured output work based on semi-structured input data and user intent to obtain structured data or instructions; The structured data or instructions include: API call parameters, database operation instructions, form filling, and formatted data output.

[0034] In one embodiment of this application, the output user text, along with the necessary dialogue history, is sent to a large language model. Simultaneously, based on the current dialogue context or a preset task flow, the system obtains or prepares "semi-structured question input" from external sources (such as retrieving relevant information from a knowledge base, loading user profile data, or calling a specific task template), and provides it to the large language model. This large language model is typically based on a Transformer network architecture (including a self-attention mechanism) and obtains powerful language understanding and generation capabilities through pre-training on massive amounts of text and code data. Representative models include the GPT series, LLaMA series, and PaLM series. The large language model comprehensively understands, reasons, and makes decisions based on all received information, specifically: ① Natural Language Understanding (NLU): performing intent recognition, entity extraction, and sentiment analysis on the output text; ② If necessary, planning and executing "structured output tasks" based on semi-structured input and user intent, such as preparing API call parameters (generating a compliant JSON request body for a specific API), database operation instructions (generating SQL query statements, which should be used with caution and validated), form filling (filling information extracted from the dialogue into a predefined form structure), and formatting specific data (outputting processing results according to preset patterns such as JSON, XML, and CSV, which is usually achieved through sophisticated prompt engineering or the model's function calling capabilities; the LLM is instructed or trained to output content in a specific format under specific conditions); ③ Generating coherent, relevant, and tone-appropriate natural language response text. It can be seen that the output of the large language model includes two parts: a) natural language response text for the user; b) if applicable, structured data or instructions generated after executing structured tasks. Perform the corresponding operation based on the instruction type, such as calling an external API, writing or reading data to or from a database, updating the internal state of the system, or populating a user interface form.

[0035] S5: Converts natural language response text into a synthesized speech audio stream.

[0036] Further, S5 includes: S501: Perform text front-end processing on the generated natural language response text; S502: Based on an acoustic model, the processed natural language response text is converted into an acoustic feature sequence; S503: Based on a vocoder, acoustic feature sequences are synthesized into audible audio waveforms to obtain a synthesized speech audio stream.

[0037] The text front-end processing includes: text normalization, word segmentation, part-of-speech tagging, polyphonic character disambiguation, and prosody prediction.

[0038] In one embodiment of this application, text normalization can be the conversion of numerical abbreviations to text, prosodic prediction can be the prediction of pauses and stresses, and the acoustic model converts the processed text feature sequence into an acoustic feature representation (such as a Mel spectrogram). Commonly used models include the Tacotron series, FastSpeech series, Transformer TTS, etc. Then, based on the vocoder, the acoustic features (Mel spectrogram) are synthesized into an audible audio waveform. Commonly used models include WaveNet, WaveGlow, HiFi-GAN, MB-MelGAN, etc. These models can generate high-quality natural speech.

[0039] In one embodiment of this application, in order to reduce user waiting time, streaming synthesis can also be used, that is, synthesizing and playing speech while receiving text segments.

[0040] S6: Play back the synthesized speech audio stream.

[0041] In one embodiment of this application, the synthesized speech is played to the user through a speaker or other device to complete one round of interaction, and then returns to S1 to continue listening to the user's next round of input.

[0042] Example 2: This application provides an intelligent voice dialogue system based on a large language model, including: a voice activity monitoring module, a speech-to-text module, a large language model processing core module, a text-to-speech module, and a playback module; The voice activity monitoring module receives the raw audio stream of the user's surrounding environment in real time, and identifies the speech segments and non-speech segments of the raw audio stream based on a neural network model to obtain an effective audio stream containing speech.

[0043] In one embodiment of this application, the Voice Activity Detection (VAD) module monitors the input audio stream in real time to accurately determine whether the user is speaking, as well as the start and end points of the speech. The VAD sensitivity level (e.g., WebRTC VAD levels 0-3), the energy or confidence threshold for speech determination, and the silence duration threshold for the end of the speech (e.g., 700ms-1500ms) are used. When VAD detects the start of speech, the system begins to buffer or directly stream audio data to the Speech-to-Text (STT) module. If the Text-to-Speech (TTS) module is playing a system response at this time, a "barge-in" mechanism is triggered to pause or stop TTS playback. When VAD detects that speech has not reappeared within the set silence duration threshold, it determines that the user has finished speaking.

[0044] The speech-to-text module includes an acoustic model, a language model, and a decoder. The acoustic model receives a valid audio stream containing speech and maps the valid audio stream containing speech to acoustic units to obtain the corresponding sequence of text symbols. The language model rearranges the text symbol sequence to obtain the rearranged text symbol sequence. The decoder combines the outputs of the acoustic model and the language model and uses a beam search algorithm to obtain the text string.

[0045] In one embodiment of this application, the speech-to-text (STT) module converts audio segments containing human speech, as identified by the VAD module, into computer-readable text strings. Modern STT systems typically employ end-to-end deep learning models. The STT module utilizes its acoustic and language models to convert audio data into text strings and uses algorithms such as Beam Search based on the decoder to find the most likely text sequence.

[0046] The core module of the large language model processing acquires and analyzes text strings, dialogue history, and semi-structured question inputs from external information sources, and generates natural language response text based on the analysis results.

[0047] In one embodiment of this application, the Large Language Model (LLM) processing core module is the "brain" of the system, responsible for understanding user intent, combining context, invoking external knowledge or tools, generating natural language responses, and processing semi-structured question inputs and generating structured outputs according to the design of this invention. The technical implementation process includes: ① Infrastructure: Typically based on a Transformer network architecture (including a self-attention mechanism), it obtains powerful language understanding and generation capabilities through pre-training on massive amounts of text and code data. Representative models include the GPT series, LLaMA series, and PaLM series. ② Natural Language Understanding (NLU): Performing intent recognition, entity extraction, and sentiment analysis on the text output by STT. ③ Dialogue Management (DM): Maintaining the dialogue state, tracking context, and determining the next action. ④ Response Generation (NLG): Generating coherent, relevant, and tone-appropriate natural language responses. ⑤ Semi-structured Question Input Processing: LLM receives not only the user's current text but also a source of "semi-structured question input," which can be one or more external information sources, such as: knowledge base or database query results: Based on the user's question, the system may first perform a knowledge retrieval, providing relevant information fragments (text, tabular data, etc.) as semi-structured input to LLM to achieve Retrieval Enhanced Generation (RAG); User profiles or preferences: Including information such as user history and preference settings, input in JSON, XML, or specific text formats to help LLM personalize responses; Task templates or API descriptions: For specific tasks (such as booking or querying), parameter templates or functional descriptions of relevant APIs (such as subsets of the OpenAPI specification) can be provided, allowing LLM to more accurately extract parameters from the user's intent; System status or external sensor data: For example, current device status, environmental information, etc. LLM, through its powerful contextual understanding capabilities, integrates this semi-structured information with the user's direct text input for more accurate reasoning and decision-making. ⑥ Structured Output: LLM not only generates natural language text for TTS broadcasting, but also generates structured data or executes specific instructions based on commands or understood task requirements. For example: API call parameter generation: generating formatted JSON request bodies for specific APIs; database operation instructions: generating SQL query statements (use with caution and validation); form filling: filling information extracted from the dialogue into predefined form structures; formatted data output: outputting processing results according to preset patterns (such as JSON, XML, CSV). This is typically achieved through sophisticated prompt engineering or the model's function calling capabilities, where the LLM is instructed or trained to output content in a specific format under specific conditions.

[0048] The parameters of the core module of the Large Language Model (LLM) processing include: the model selected by the LLM (such as GPT-4, LLaMA-3, etc.), the Prompt template, the temperature coefficient used to control the randomness of the generated text, and the maximum number of generated tokens.

[0049] The text-to-speech module includes a text front-end processing module, an acoustic module, and a vocoder. The text front-end processing module processes the natural language response text, and the acoustic model converts the processed natural language response text into an acoustic feature sequence. Based on the vocoder, the acoustic feature sequence is converted into a synthesized speech audio stream.

[0050] In one embodiment of this application, a text-to-speech (TTS) module converts the natural language response text generated by LLM into human speech that sounds natural and carries appropriate emotion and rhythm. Modern TTS systems also widely employ deep learning.

[0051] The playback module plays the synthesized audio stream.

[0052] This application provides an intelligent voice dialogue method and system based on a large language model. Through the powerful natural language understanding capabilities of the large language model, it can dynamically and deeply understand the user's immediate state, potential needs, emotional tone, and contextual relationships from the user's real-time voice dialogue, historical dialogue, and semi-structured question input data, forming a richer, more vivid, and context-dependent user understanding. Based on this deeper and more dynamic understanding of the user, this invention can generate responses and provide services in a more natural, considerate, and accurate manner.

[0053] It should be noted that those skilled in the art will recognize that the embodiments described herein are for the purpose of helping readers understand the principles of this application, and should be understood as not limiting the scope of protection of this application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this application without departing from the essence of this application, and these modifications and combinations are still within the scope of protection of this application.

Claims

1. An intelligent voice dialogue method based on a large language model, characterized in that, include: S1: Obtain the audio stream of the user's surrounding environment; S2: Perform frame segmentation on the audio stream and determine whether the segmented audio stream contains speech to obtain a valid audio stream containing speech. S3: Receive a valid audio stream containing speech and convert the valid audio stream into a text string using a deep learning model; S4: Based on a large language model, acquire and analyze data from text strings, dialogue history, and semi-structured question inputs, and generate natural language response text based on the analysis results; S5: Converts natural language response text into a synthesized speech audio stream; S6: Play back the synthesized speech audio stream.

2. The intelligent voice dialogue method based on a large language model according to claim 1, characterized in that, S2 includes: S201: Receive a continuous audio stream and perform frame segmentation on the continuous audio stream; S202: Extract speech features and non-speech features from each frame of the audio stream based on its acoustic features; S203: Based on the extracted features, determine whether the current frame contains speech; S204: Based on the continuous frame-level judgment results, determine the start and end of the speech segment to obtain a valid audio stream containing speech.

3. The intelligent voice dialogue method based on a large language model according to claim 1, characterized in that, S3 includes: S301: Receive the valid audio stream containing speech using an acoustic model, and map the valid audio stream containing speech into an acoustic unit to obtain all text symbol sequences corresponding to the valid audio stream; S302: Based on historical text, use a language model to rearrange all text symbol sequences to obtain rearranged text symbol sequences; S303: Based on all text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model, a text string is generated using a decoder.

4. The intelligent voice dialogue method based on a large language model according to claim 3, characterized in that, The text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model are used to generate text strings using a decoder, including: The beam search algorithm is used to calculate the joint score of all text symbol sequences output by the acoustic model and the rearranged text symbol sequences output by the language model, and the text symbol sequence with the highest score is taken as the final text string.

5. The intelligent voice dialogue method based on a large language model according to claim 1, characterized in that, The S4 includes: S401: Combine the text string, dialogue history, and semi-structured question input data to obtain combined multi-source information; S402: Based on the Transformer network architecture, perform natural language understanding on the combined multi-source information; S403: Based on the understood multi-source information, gain a deeper understanding of the context to obtain the core content of the next action and the next response; S404: Integrate the core content of the response to obtain the natural language response text.

6. The intelligent voice dialogue method based on a large language model according to claim 5, characterized in that, The semi-structured problem input data is from one or more external information sources, including: Relevant information fragments obtained from knowledge base or database retrieval based on text strings, user history profiles or preferences, task templates or API descriptions required for specific tasks, and system status or external environment status.

7. The intelligent voice dialogue method based on a large language model according to claim 6, characterized in that, Also includes: Based on the semi-structured input data and user intent, perform structured output tasks to obtain structured data or instructions; The structured data or instructions include: API call parameters, database operation instructions, form filling, and formatted data output.

8. The intelligent voice dialogue method based on a large language model according to claim 1, characterized in that, The S5 includes: S501: Perform text front-end processing on the generated natural language response text; S502: Based on an acoustic model, the processed natural language response text is converted into an acoustic feature sequence; S503: Based on a vocoder, acoustic feature sequences are synthesized into audible audio waveforms to obtain a synthesized speech audio stream.

9. The intelligent voice dialogue method based on a large language model according to claim 8, characterized in that, The text front-end processing includes: text normalization, word segmentation, part-of-speech tagging, polyphonic character disambiguation, and prosody prediction.

10. An intelligent voice dialogue system based on a large language model, characterized in that, include: The system includes a voice activity monitoring module, a speech-to-text module, a large language model processing core module, a text-to-speech module, and a playback module. The voice activity monitoring module receives the raw audio stream of the user's surrounding environment in real time, and identifies the speech segments and non-speech segments of the raw audio stream based on a neural network model to obtain an effective audio stream containing speech. The speech-to-text module includes an acoustic model, a language model, and a decoder. The acoustic model receives a valid audio stream containing speech and maps the valid audio stream containing speech to acoustic units to obtain all corresponding text symbol sequences. The language model rearranges the text symbol sequences to obtain rearranged text symbol sequences. The decoder combines the outputs of the acoustic model and the language model and uses a beam search algorithm to obtain a text string. The core module of the large language model processing acquires and analyzes text strings, dialogue history, and semi-structured question inputs from external information sources, and generates natural language response text based on the analysis results; The text-to-speech module includes a text front-end processing module, an acoustic module, and a vocoder. The text front-end processing module processes the natural language response text, and the acoustic model converts the processed natural language response text into an acoustic feature sequence. Based on the vocoder, the acoustic feature sequence is converted into a synthesized speech audio stream. The playback module plays the synthesized audio stream.

Images