A voice interaction method and system

Abstract

This application provides a voice interaction method and system, including receiving a user-input query request voice interaction request; invoking a large language model to generate corresponding response content, wherein the large language model outputs the response content block by block in a streaming manner during the generation process; while the large language model outputs each data block, it performs streaming parsing on the output data blocks to obtain several texts to be played with complete semantics; the texts to be played are sequentially submitted to a speech synthesizer for speech synthesis to obtain corresponding audio data, and the audio data is played. This invention achieves the purpose of simultaneous response and playback by using a streaming interaction method of simultaneous generation and transmission of the large language model, parsing and converting the response content generated by the large language model block by block, thereby meeting the real-time requirements of interaction.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and more specifically, to a voice interaction method and system. Background Technology

[0002] With the deep integration of artificial intelligence and the Internet of Things (IoT) technologies, smart kitchens are gradually moving from concept to reality. Users increasingly expect to receive natural, fluent, and intelligent voice guidance during the cooking process, much like a human assistant.

[0003] In existing technologies, a "request-complete generation-response" model is used for voice guidance. When a user issues a complex cooking instruction ("Give me a detailed recipe for Peking duck at home"), this model requires waiting for the Large Language Model (LLM) to completely generate all steps, ingredients, etc. before the voice broadcast can begin. This results in interaction delay and cannot meet the need for real-time interaction. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a voice interaction method and system to improve the immediacy of voice interaction.

[0005] Firstly, a voice interaction method is provided, including: Receive user input for voice interaction requests; The large language model is invoked to generate the corresponding response content, during which the large language model outputs the response content in a streaming manner, block by block. While the large language model outputs each data block, it performs streaming parsing on the output data blocks to obtain several texts with complete semantics to be broadcast. The text to be broadcast is submitted sequentially to the speech synthesis unit for speech synthesis, resulting in the corresponding audio data, which is then played.

[0006] Optionally, before invoking the large language model, the following may also be included: Construct prompt words and specify the text format of the large language model output as Markdown format in the prompt words; The prompt and the user's voice interaction request are sent to the server together.

[0007] Optionally, the method further includes: The prompt words are enhanced based on predefined metadata tags; among them, the metadata tags include semantic style tags, which are used to guide the large language model to inject control information related to the speech broadcast style into the generated response content; The output data blocks are stream-parsed to obtain several texts with complete semantics to be broadcast, including: Parse the data block containing control information related to the speech broadcast style to obtain the text to be broadcast and speech style parameters with complete semantics; The text to be broadcast is submitted sequentially to the speech synthesis unit for speech synthesis, and the corresponding audio data obtained includes: The text to be broadcast and its corresponding speech style parameters are submitted sequentially to the speech synthesis party to synthesize audio data containing speech style parameters.

[0008] Optionally, the method further includes: While the audio data corresponding to the current text to be played is being played, a preloading mechanism is used to continuously initiate speech synthesis requests for subsequent texts to be played, thereby obtaining a preloaded audio stream. The preloading mechanism is as follows: based on the real-time measured network round-trip time and the processing latency of the speech synthesis service, the preloading window is adaptively adjusted, and subsequent text to be played is preloaded based on the preloading window; The preloaded audio stream is cached in the local audio buffer; the local audio buffer has a maximum cache capacity, and the least accessed audio data is automatically removed when the buffer is full.

[0009] Optionally, the output data blocks are stream-parsed to obtain several texts with complete semantics to be broadcast, including: Identify Markdown tags in each data block; The pre-configured stack-based finite state machine is triggered by Markdown tags to enter the corresponding state, and the data block content is parsed in the corresponding state to obtain several texts with complete semantics to be broadcast.

[0010] Optionally, in the corresponding state, the data block content is parsed to obtain several texts to be broadcast, including: When the data block is a table structure or a nested list, in response to the stacked finite state machine entering the table header parsing state, the column headers in the table header are extracted and stored as an array of column names. When the stack-based finite state machine enters the table row parsing state, the cell data of the current row is semantically bound to the column name array to generate structured data in the form of key-value pairs; It is converted into natural language broadcast text based on predefined voice broadcast templates; When the data block is unstructured text, remove non-semantic Markdown tags and retain the plain text content as the broadcast text. Optionally, the method also includes: Listen for voice interruption events and generate a session snapshot when a voice interruption event is detected. The session snapshot encapsulates the text that has been played, is being played, and is to be played; the ID of the text currently being played and the timestamp of the audio stream breakpoint corresponding to that text; When the interruption is resumed, a session snapshot is loaded; The location of the interruption is determined based on information in the session snapshot; Resume voice broadcasting based on the interruption location.

[0011] Optionally, the method further includes: Listen for preset wake words; When a wake word is detected, pause the current audio playback and capture the user's interruption. The intent of the interrupted content is identified based on a pre-trained intent recognition model. Based on the user's intent and the preset mapping relationship between intent and interaction operation, the corresponding interaction operation is determined and executed. The interaction operation includes at least one of the following: jumping to a specified step, replaying the current step, continuing playback, pausing playback, querying information of the current step, or returning to the previous level of process.

[0012] Optionally, the method may also include the following before invoking the large language model: Receive information from other modalities during the voice interaction process, including at least images; Integrate information from other modalities with voice interaction requests.

[0013] Secondly, a voice interaction system is provided, including a user input interface, a central control module, a streaming communication module, a streaming semantic parser, a TTS control module, an audio playback module, and a large language model; The user input interface is used to collect users' voice interaction requests; The streaming communication module sends voice interaction requests to the large language model and receives the response content returned by the large language model; During the generation process, the large language model returns the response content in chunks of data. The streaming semantic parser is used to perform streaming parsing on the already output data blocks while the large language model outputs each data block, to obtain several texts to be broadcast. The TTS control module is used to sequentially submit the text to be played to the speech synthesis unit for speech synthesis to obtain the corresponding audio data. The audio playback module is used to play audio data; The central control module serves two purposes: firstly, it generates control commands based on voice interaction requests and sends these commands to the streaming communication module; secondly, it triggers the streaming semantic parser to perform semantic parsing based on the return results of the large language model, and triggers the TTS control module to perform speech synthesis.

[0014] This invention provides a voice interaction method and system. The method involves receiving a user-input query request for voice interaction; invoking a large language model to generate corresponding response content, wherein the large language model outputs the response content block by block in a streaming manner during the generation process; simultaneously, the large language model performs streaming parsing on the output data blocks to obtain several texts to be played with complete semantics; and then sequentially submits the texts to be played to a speech synthesizer for speech synthesis to obtain corresponding audio data, which is then played back. This invention achieves the goal of simultaneous response and playback by using a streaming interaction method where the large language model generates and transmits data block by block, parsing and converting the response content generated by the large language model into text-to-speech, thus meeting the real-time requirements of interaction.

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This diagram illustrates the architecture of the voice interaction system provided in an embodiment of the present invention. Figure 2 A schematic diagram of the architecture of a voice interaction system provided in another embodiment of the present invention is shown; Figure 3 This is a structural schematic diagram illustrating the architecture of a voice interaction system provided in another embodiment of the present invention; Figure 4 The diagram shows a flowchart of a voice interaction method provided by an embodiment of the present invention.

[0018] Among them, 100 is the terminal device; 200 is the server; 101 is the user input interface; 102 is the central control module; 103 is the streaming communication module; 104 is the streaming semantic parser; 105 is the TTS control module; 106 is the audio playback module; 107 is the status management module; and 108 is the wake word listener. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] With the deep integration of artificial intelligence and Internet of Things technologies, smart kitchens are gradually moving from concept to reality. Users increasingly expect to receive natural, fluent, and intelligent voice guidance during the cooking process, just like a human assistant.

[0021] In existing technologies, a "request-complete generation-response" model is used for voice guidance. When a user issues a complex cooking instruction ("Give me a detailed recipe for Peking duck at home"), this model requires waiting for the Large Language Model (LLM) to completely generate all steps, ingredients, etc. before the voice broadcast can begin. This results in interaction delay and cannot meet the need for real-time interaction.

[0022] Based on this, embodiments of the present invention provide a voice interaction system that can be applied to voice interaction in cooking scenarios, as well as to voice interaction in other smart home scenarios. The system includes a user input interface, a central control module, a streaming communication module, a streaming semantic parser, a TTS control module, an audio playback module, and a large language model.

[0023] In one specific implementation, such as Figure 1As shown, the user input interface, central control module, streaming communication module, streaming semantic parser, TTS control module, and audio playback module can be deployed on the terminal side, while the large language model and speech synthesis unit can be deployed on the server side. The terminal can be a mobile phone, tablet, smart speaker, or smart kitchen appliance with screen and voice functions, etc. Users input their voice interaction requests through the terminal, which then sends these requests to the server. The server sends the generated response content back to the terminal for voice playback. To achieve streaming interaction between the terminal and server, a streaming data channel is established between the terminal device and the server using protocols such as WebSocket. This protocol allows the server to push the generated data to the terminal one "data block" after another, while simultaneously generating content. For example, each meaningful data block (e.g., a complete sentence) generated by the server is immediately sent to the terminal device through this streaming channel, without waiting for the large language model to generate complete content.

[0024] In another implementation, when the terminal device has computing resources to support the operation of a large language model, the large language model can be deployed on the terminal side, thereby generating natural language response content locally without relying on a remote server.

[0025] Specifically, the user input interface includes a voice input unit and / or a user interface. The voice input unit consists of a voice acquisition unit and an ASR (Automatic Speech Recognition Engine). The voice acquisition unit is used to collect the user's voice interaction requests, and the ASR is used to convert the user's voice into text. The user interface is, for example, a touch screen, on which the user can directly input cooking-related text and voice interaction requests.

[0026] The central control module is used to coordinate and manage the core logic of the entire voice interaction process. Its functions include: receiving raw requests from the user input interface; generating corresponding control commands according to the request type and sending them to the streaming communication module to initiate interaction with the server. The TTS control module and streaming semantic parser are subject to unified scheduling and state management, including but not limited to: triggering the start and stop of the semantic parsing process; controlling the timing of TTS audio synthesis and playback, etc.

[0027] The central control module serves as the system's hub, ensuring efficient collaboration and consistent status among all functional modules during streaming processing, thereby achieving a low-latency, highly coherent intelligent voice interaction experience.

[0028] The streaming communication module uses protocols such as WebSocket to establish a streaming data channel with the server, receives data blocks sent by the server, and sends them to the streaming semantic parser.

[0029] The streaming semantic parser is used to perform semantic parsing and splicing on the received data blocks, obtain several text units with complete semantics, and store them in the semantic cache queue.

[0030] In one example, for instance, the content generated by the large language model is " Prepare ingredients : 500 grams of chicken wings, one can of cola. Blanch : Put the chicken wings into cold water, add ginger slices, and remove them from the pot and wash them after boiling. Fry : Pour oil into the pot and fry the chicken wings until both sides are golden brown." Due to streaming transmission, this content is sent in 4 data blocks: Data block 1: "1. Prepare ingredients : 500 grams of chicken wings, co" Data block 2: "la one can of cola.\n2. Blanch : Put the chicken wings in co" Data block 3: "ld water, add ginger slices, and remove them from the pot and wash them after boiling."; Data block 4: "\n3. Fry : Pour oil into the pot and fry the chicken wings until both sides are golden brown."; In the embodiment of the present invention, the working process of the streaming semantic parser is as follows: First step: Receive data block 1 and perform parsing; The parsing result is the start of the first item of an ordered list, but it is obviously incomplete (there should be more words after the word "co"), and it is temporarily stored in the internal buffer without outputting a text unit.

[0031] Second step: Receive data block 2 and add it to the internal buffer. Now the content in the buffer is: "1. Prepare ingredients : 500 grams of chicken wings, one can of cola.\n2. Blanch "Cold the chicken wings"; Step 3: Analysis: \n indicates the end of the first item. "One can of cola." forms a complete sentence.

[0032] Step 4: Extract the first complete list item: "1". Prepare ingredients Ingredients: 500g chicken wings, one can of cola. The list item is then pushed into the sentence buffer queue as a text unit.

[0033] Subsequent data blocks are processed according to the above process to obtain other text units with complete semantics. The extracted text units are then sent to the first-in-first-out sentence buffer queue in the order they were generated.

[0034] The TTS control module retrieves text units from the sentence buffer queue and sends them to a third-party TTS service via the streaming communication module. It also receives the audio stream returned by the third-party TTS service and notifies the audio playback module to play it.

[0035] The audio playback module is used to play synthesized audio streams.

[0036] Speech synthesis typically takes hundreds of milliseconds to several seconds to complete. Text unit generation, however, is often much faster than speech synthesis, especially when LLM streams short sentences. If a serial mode is used, where the speech synthesized from each text unit is played before requesting the synthesis of the next text unit, the user will perceive noticeable speech interruptions, resulting in a choppy interaction.

[0037] Therefore, in another embodiment of the present invention, a preloading mechanism and a local audio buffer are configured in the TTS control module. While the current text unit is playing, a TTS synthesis request is initiated in advance for the subsequent text unit, and the preloaded synthesized audio stream is cached in the local audio buffer. When the playback pointer reaches the next text unit, its audio is ready and there is no need to wait.

[0038] The preload window size (i.e., the number of text units preloaded each time) is dynamically adjusted. In one feasible implementation, this is done dynamically based on the following two real-time metrics: Network round-trip time: Measured by sending lightweight heartbeat packets (such as HTTP HEAD requests) to a third-party TTS service; TTS service processing latency: Records the time from submitting a text unit to receiving the first audio unit.

[0039] In one specific implementation, the size of the preload window can be adjusted based on the sliding average of these two metrics.

[0040] Specifically, take a preset number of indicator samples collected in real time recently, calculate the average value of the two indicators respectively, and compare the average value with their respective preset thresholds.

[0041] If the network round-trip time exceeds the preset threshold, the network is poor; otherwise, the network is good. If the TTS service processing latency exceeds the preset threshold, the service response is slow; otherwise, the service response is fast.

[0042] When network conditions are good and service response is fast, the window can be expanded to 3-4 units to maximize buffer depth; when the network is poor, it should be reduced to 1 unit to avoid wasting bandwidth on long-term requests that may be unavailable due to interruption.

[0043] In this embodiment of the invention, the local audio buffer is managed using the Least Recently Used (LRU) algorithm. When it is necessary to play an audio unit (one audio unit corresponds to one text unit), the local audio buffer is checked first. If a match is found, the audio is played directly.

[0044] In addition, to prevent excessive memory usage during prolonged use, a capacity limit is set for the local audio buffer (for example, storing the 20 most recent audio units). When new audio data needs to be cached but the capacity is full, the system will automatically evict the least recently accessed audio data.

[0045] The embodiments of the present invention, through adaptive preloading and cache management, can maximize the continuity and fluency of voice broadcasting in various network environments.

[0046] The large language model is used to receive users' text and voice interaction requests and generate corresponding responses.

[0047] Third-party TTS services, such as Google Cloud TTS and Azure TTS, are used to convert each text unit sent by the terminal into an audio stream. Additionally, the server-side is configured with API interfaces that support streaming output.

[0048] This invention establishes a streaming data channel between the terminal and the server, enabling streaming transmission. Based on this, a streaming semantic parser and a TTS control module are constructed, allowing the parser to parse, concatenate, and convert text to speech blocks of the streaming data, thereby achieving the goal of generating and playing a large language model simultaneously and meeting the real-time requirements of interaction.

[0049] The content generation process of a large language model is essentially a probability-based sequence construction process that outputs tokens one by one. Therefore, the choice of output format directly determines the efficiency, cost, and difficulty of subsequent processing.

[0050] Therefore, based on the above embodiments, the text format output by the large language model is further defined. In another embodiment of the present invention, the text output by the large language model adopts the Markdown format.

[0051] Because if a large language model deployed on the server directly outputs binary or complex markup formats such as .docx (Word) or rich text (RTF), the following problems will occur: First, these formats are binary or have extremely complex nested tokens, making it difficult for large language models to generate such byte streams directly and efficiently. The generation process becomes extremely slow and prone to errors. Second, the token overhead of these formats is huge. Expressing a simple bold or list may require tens or even hundreds of tokens to generate complex tags and attributes, which greatly increases the computational cost. Third, such complex formats must wait for the entire file to be generated before they can be processed, which cannot meet the requirements of streaming parsing.

[0052] Markdown text uses simple markup syntax to format text, for example: Use # to indicate heading levels, such as #heading1, ##heading2, etc. Use numbers and dots (such as 1.) to create ordered lists, etc.

[0053] Markdown, with its plain text and lightweight markup, allows large language models to efficiently and accurately express rich document structures with minimal token overhead (e.g., ## represents a second-level heading). This format not only significantly improves the generation speed and reduces the cost of large language models, but also ensures that data blocks are parallel and not nested, providing an unambiguous data foundation for subsequent streaming parsing and semantic extraction.

[0054] Correspondingly, the streaming semantic parser is internally configured with a stacked finite state machine parsing engine, with built-in Markdown syntax rules and custom tag processing logic, and can support nested structures (such as nested lists) within a single data block.

[0055] Specifically, the stack-based finite state machine defines a set of core parsing states, including but not limited to: IDLE (idle), PARSING_HEADING (parsing headers), PARSING_LIST_ITEM (parsing list items), PARSING_TABLE_HEADER (parsing table headers), PARSING_TABLE_ROW (parsing table rows), and PARSING_CODE_BLOCK (parsing code blocks). These states represent different semantic contexts in which the streaming semantic parser operates when processing Markdown streams.

[0056] The transitions between states are driven by specific character sequences (called "trigger symbols") in the received data stream. In one example: The detection of the character "##" triggered the entry into the PARSING_HEADING state.

[0057] The character "-" was detected. "Triggering entry into the PARSING_LIST_ITEM state."

[0058] The detection of the character "|" triggers entry into the PARSING_TABLE_HEADER or PARSING_TABLE_ROW state.

[0059] The detection of the character "``` triggers entry into or exit from the PARSING_CODE_BLOCK state.

[0060] In a specific example, when the streaming semantic parser detects a character indicating the start of nesting (such as a higher indentation level -) in the current context (e.g., parsing a main list item in the PARSING_LIST_ITEM state), it does the following: Push the current parsing state (such as PARSING_LIST_ITEM) and its related context information (such as list type and indentation level) onto the context stack.

[0061] Enter a new PARSING_LIST_ITEM state to process the sublist items. At this point, the working state of the streaming semantic parser remains unchanged, but its semantic level has been deepened.

[0062] When the nested structure ends (for example, when the indentation level of subsequent text is reduced or disappears, no longer conforming to the format of the sublist), the parser terminates the current PARSING_LIST_ITEM state, pops the previously saved state and context from the context stack, restores the parsing state and context of the previous level, and thus continues to process the remaining content of the parent structure.

[0063] This mechanism ensures that nested structures in Markdown text are correctly parsed layer by layer and can seamlessly restore the previous parsing environment after the structure is closed, achieving accurate and unambiguous streaming semantic extraction of complex Markdown document structures.

[0064] This invention further limits the output format of the large language model on the server side to Markdown format, thereby improving the text generation speed and reducing costs from the source. A stacked finite state machine capable of parsing this format is configured in the streaming semantic parser on the terminal side for efficient and accurate streaming parsing, achieving near synchronization between text content generation and voice playback, reaching a response time within seconds.

[0065] In addition, Markdown contains a large number of formatting tags, such as the heading symbol # and list item symbols. Or -, bold symbol Linking syntax such as [](). If this raw Markdown text is provided directly to a third-party TTS service without processing, the third-party TTS service will also convert these non-semantic formatting symbols into speech (for example, incorrectly converting "##preparation work" into "hash hash preparation work"), generating a lot of meaningless noise and interfering with the user's understanding of the core information.

[0066] Therefore, based on the above embodiments, the streaming semantic parser is also used to preprocess the initially extracted text units.

[0067] Continuing from the previous example, let's say the initial text unit is "1". Prepare ingredients Ingredients: 500g chicken wings, one can of cola. Pre-process the ingredients: Remove the bolded marker from item number 1. The resulting plain text is: "Prepare ingredients: 500g chicken wings, one can of cola." This text unit is then pushed into the sentence buffer queue.

[0068] The embodiments of the present invention reduce noise in synthesized speech data.

[0069] When voice broadcasts are interrupted due to factors such as checking phone notifications, answering calls, the device automatically locking the screen to save power, or network issues, it is often impossible to resume from the exact point of interruption. Users need to manually find the progress or listen from the beginning, resulting in low interaction efficiency.

[0070] Therefore, as Figure 2 As shown, another embodiment of the present invention further includes a state management module, which is used to listen for interruption events, generate a session snapshot, and write the session snapshot to local storage.

[0071] In this embodiment of the invention, interruption events mainly include two categories: one is expected interruption, such as page switching, page closing, and page freezing; the other is unexpected interruption, such as network disconnection and WebSocket connection abnormality.

[0072] In this embodiment of the invention, the session snapshot encapsulates the information required to resume the interaction in JSON format: all text units in the sentence buffer queue, including those that have been played, are being played, and are yet to be played. Playback position information, including the ID of the currently playing text unit in the queue and the precise timestamp (milliseconds) of the audio stream breakpoint corresponding to that text unit. In this embodiment of the invention, the ID and audio stream timestamp of each text unit are pre-configured in the following manner: First: Text Unit ID Binding When the streaming semantic parser generates each text unit, the system assigns it a globally unique ID. This ID is used throughout the entire data flow process. In the UI layer, this ID is used as the data-id attribute of the DOM element to enable content addressing.

[0073] At the audio layer, this ID is logically bound to the corresponding audio stream to quickly locate the audio segment containing that ID.

[0074] Second: Word-by-word timestamp recording When the TTS control module calls a third-party TTS service, it explicitly requests the return of word-by-word timestamps through API parameters, which record the start and end times (in milliseconds) of each word in the audio stream.

[0075] The session snapshot also includes a list describing the state of the local audio buffer, listing which text units' audio data has been successfully preloaded.

[0076] When an interruption is resolved (such as when a page becomes visible or a restart occurs), the central control module executes a seamless recovery mechanism: Step 1: Load the latest session snapshot from local storage.

[0077] Step 2: Environmental inspection.

[0078] Verify if the network connection has been restored. If not, enter the "Waiting for network" pause state; if restored, proceed to the next step.

[0079] Step 3: Queue Reconstruction: Reconstruct the sentence buffer queue using all text units encapsulated in the session snapshot.

[0080] After an interruption, the data in the sentence buffer queue may be lost or invalidated. Therefore, rebuilding the sentence buffer queue ensures seamless playback. Furthermore, by rebuilding already played text units, backtracking is supported, such as repeating the first step.

[0081] Step 4: Audio cache recovery.

[0082] Based on the list describing the local audio buffer status in the session snapshot, check which text units' audio streams have been cached and which have not. For text units that have not been cached, continue to send requests to the third-party TTS service to obtain the audio stream data that has not yet been converted to speech.

[0083] Step 5: Precise addressing and playback.

[0084] Send a RESUME command to the TTS control module, instructing it to continue playing the previous session content. This command includes playback position information.

[0085] After receiving the RESUME command, the TTS control module performs the following steps: Step 1: Quickly find the target audio stream containing the TD based on the ID of the text unit.

[0086] Step 2: Query the timestamp information of the breakpoint in the target audio stream.

[0087] Step 3: Instruct the audio playback module to start playing from this timestamp information. This achieves a natural and seamless transition.

[0088] This seamless recovery mechanism ensures the accuracy of the recovery operation, providing users with a seamless experience regardless of the cause and duration of the interruption.

[0089] Most existing systems are one-way information broadcasters, unable to handle users' immediate follow-up questions and instruction changes during the guidance process. This keeps the interaction at a simple question-and-answer level and cannot meet the dynamic needs of real cooking scenarios.

[0090] For example, a user might interrupt the voice broadcast. The system cannot accurately distinguish their intention. It cannot determine whether the user wants to pause playback, inquire about a specific detail, or completely change the task.

[0091] Unable to accurately identify the user's intent, the system doesn't know how to handle the user's "interruption"—should it pause and wait? Or should it discard the current task? Or should it "insert" a new subtask into the current task? This forces the system to adopt only the simplest strategies, such as ignoring the user's interruption or abruptly interrupting the entire process and requiring the user to start over.

[0092] Therefore, in order to achieve a natural and seamless interactive experience, in another embodiment of the present invention, such as Figure 3 As shown, the streaming interactive system for assisting cooking also includes a wake word listener located on the terminal device side; and a trained intent recognition model deployed in the central control module.

[0093] The wake-word listener is driven by dedicated low-power hardware (such as a DSP or MCU) that continuously monitors ambient sounds and recognizes only preset wake-words (such as "Hey, Chef"). It has extremely low power consumption and can operate 24 / 7.

[0094] Intent recognition models are used to identify the intent behind user interruptions. In one example, lightweight Transformer models or convolutional network models can be used. The training method is similar to that of existing classification models, the difference being the training corpus and the corresponding intent labels, which will not be elaborated upon here.

[0095] These two elements work together to allow users to "interrupt" with their voices at any time during the system's voice broadcasts, thus breaking the limitations of traditional polling-based interaction.

[0096] When the wake word listener detects a wake word during voice broadcasting, the ASR engine in the above embodiment converts the user's interruption into text, and the central control module executes the following decision-making process: Step 1: Pause the current audio broadcast.

[0097] Regardless of what the user says, the system first sends a PAUSE command to the TTS control module and the voice playback module to immediately stop audio playback.

[0098] At the same time, the state management module is triggered to create a session snapshot, which records key information such as the current playback position and task context.

[0099] Step 2: Intent Classification.

[0100] The user's voice content is quickly classified using a pre-trained intent recognition model.

[0101] In one example, user intents fall into three categories: Category A: local control instructions (such as “pause,” “louder,” “repeat”); Category B: contextual queries (such as “how finely should the ginger be cut?”); and Category C: global instructions (such as “change the recipe”).

[0102] Step 3: Execution and Response Based on the classification results, the system performs the corresponding operations.

[0103] For category A, such as "repeat the first step", the central control module will obtain the ID of the text unit in the session snapshot based on the content, generate a repeat playback instruction based on the ID, and send it to the TTS module. The TTS control module will then find the corresponding audio stream in the audio buffer area based on the ID and trigger the voice playback module to repeat the audio stream.

[0104] For Category B: Pause the main task flow, package the user's query with the current context (e.g., context:{current_step:"s001",recipe:"Cola Chicken Wings"}), and initiate a separate, high-priority LLM request. After receiving a response and completing the broadcast, ask the user: "Would you like to continue with the previous steps?"

[0105] If the user chooses to continue the original task, the system will execute the seamless recovery mechanism described in the above embodiments.

[0106] In a specific example: the system announces, "...put the chicken wings in the pan and fry until golden brown on both sides...", and the user interjects, "Wait a minute, should I add oil?"; The voice playback module is immediately muted and a session snapshot is created.

[0107] The intent recognition model identifies the intent as a "contextual query," and the central control module sends a new request to the server.

[0108] Receive and broadcast the response from the large language model: "Yes, a small amount of oil needs to be added to the pan first." After playing, ask, "Did you understand? Do you need me to continue?" User: "Continue." The central control module loads a session snapshot, locating the end of the word "yellow" in "golden yellow." Then it seamlessly announces: "...yellow. Then add cola...".

[0109] For category C: This is a task reset. Clear all states of the current task and start a completely new conversation.

[0110] User input requests may include images, such as "Please determine whether the pork belly in this picture is evenly marbled and suitable for braised pork?", in order to meet this multimodal scenario requirement.

[0111] In another embodiment of the present invention, a Vision-Language Model (VLM) is deployed on the server side. This Vision-Language Model is built based on LLM and can process both images and text simultaneously; it is also a type of large language model. The output format of this Vision-Language Model is also Markdown. The Markdown format data stream is then sent to the streaming semantic parser via a streaming communication module, executing the same process as in the above embodiment.

[0112] In one feasible implementation, the central control module performs some preprocessing operations before sending the image data to the server to optimize performance and cost: Step A: Scale the image proportionally to a preset size suitable for visual analysis (e.g., the longest side does not exceed 1024 pixels).

[0113] Step B: Format conversion and compression.

[0114] For example, by uniformly converting to JPEG format and applying a medium-quality compression ratio (such as 75%), the data load can be significantly reduced while ensuring recognition accuracy.

[0115] The central control module is also used to dynamically construct prompts based on the context of the current interaction.

[0116] In one example, the prompt word consists of three parts: 1. System instructions: Define the roles and tasks of the visual language model, for example: "You are a professional cooking assistant, analyzing images and questions provided by the user."

[0117] 2. Contextual information: Provide the background of the current cooking task, such as: "The user is making 'braised pork' and is currently in step two: preparing the pork belly."

[0118] 3. User's voice interaction request: For example: "Please determine whether the pork belly in this picture is evenly marbled with fat and lean meat, and whether it is suitable for making braised pork?"

[0119] The embodiments of the present invention fulfill the requirements of multimodal interaction and enrich application scenarios.

[0120] In the above embodiments, the TTS control module extracts plain text units from the sentence buffer queue and provides them to a third-party TTS service for speech synthesis. The synthesized audio data has the following problems: 1. It is impossible to control the speech rate, pauses, stress, and other speech styles of the text, resulting in mechanical and monotonous speech output; 2. If specific content (such as code, comments, internal IDs) is converted into speech, the resulting audio information will be redundant and unclear.

[0121] Based on this, another embodiment of the present invention provides a broadcast control mechanism based on metadata tags, the principle of which is as follows: Step 1: Define a set of metadata tags.

[0122] Metadata tags include semantic style tags and context tags.

[0123] Specifically, HTML comment syntax can be used to define tags. TTS control tags include, for example:<!--TTS:emphasis=strong--> : Indicates emphasis;<!--TTS:pause=0.5s--> : indicates a pause of 0.5 seconds;<!--TTS:ignore_start--> This means "From this point onward, all subsequent text should not be synthesized into speech and should be discarded."<!--TTS:ignore_end--> : This indicates that the text will be discarded from here onwards, and text generation will continue.

[0124] Contextual tags, such as [METADATA:step_id=s001,estimated_time=5m], indicate that the current step is the first step and the time required to complete this step is 5 minutes. This tag can be used to pass structured context information for UI synchronization, progress management, etc.

[0125] Step 2: Generate prompt words for a large language model based on these labels.

[0126] In one example, the prompt word is as follows: You are a smart cooking assistant. Please output your response in the following format: Organize content using standard Markdown format; Use key guiding words (such as "firstly", "must", "note")<!--TTS:emphasis=strong--> pack; Add after each major step<!--TTS:pause=0.5s--> ; Place internal development notes or debugging information in<!--TTS:ignore_start--> and<!--TTS:ignore_end--> between; Add a [METADATA:step_id=xxx,estimated_time=Xm] tag to each header line; Additional information within parentheses in list items should not be emphasized and should be read in a light tone.

[0127] Step 3: Input the user's voice interaction request along with the prompt word into the server's large language model; Step 4: The large language model injects the corresponding labels into the generated data blocks based on the prompt words.

[0128] In one example, if a user's query is "how to make cola chicken wings", the large language model generates the following content: ##Preparation Phase [METADATA:step_id=s001,estimated_time=5m] <!--TTS:emphasis=strong--> first<!--TTS:emphasis=reset--> We need to prepare the following ingredients: Chicken wings (middle section): 500g Coca-Cola: 1 can<!--TTS:pause=0.5s--> (Approximately 330 ml) Ginger: 3 slices <!--TTS:ignore_start--> Developer's Note: Plain Coca-Cola is recommended here; Coca-Cola Zero is not recommended.

[0129] <!--TTS:ignore_end--> .

[0130] Step 5: The streaming semantic parser parses the data blocks injected with metadata tags and generates a semantic action queue.

[0131] Each semantic action in this semantic action queue is an executable instruction object, including the action type, the text to be read, and the speech style.

[0132] Example: The parsing process and output queue of the Markdown stream above: Input data block 1: ##Preparation stage [METADATA:step_id=s001,estimated_time=5m]; The streaming semantic parser parses this data block as follows: First, the character "##" is detected and identified as a title, triggering the stack-based finite state machine to enter the PARSING_HEADING state; Then, the "preparation phase" is detected, and it is cached as plain text. [METADATA:step_id=s001,estimated_time=5m] was detected and identified as a metadata tag; This data block 1 contains two types of information: the "preparation phase," which requires voice prompts (generating the SPEAK action). Context metadata: step_id, estimated_time; not used for broadcasting, but required to update system status (generating SET_METADATA action).

[0133] The specific actions are as follows: Action 1 (Speak): {type:'SPEAK',payload:{text:'Preparation Phase',style:'HEADING_2'},source_id:'block_0'}; Action 2 (SetMetadata): {type:'SET_METADATA',payload:{step_id:'s001',estimated_time:'5m'}}; The generated semantic actions are stored in a first-in-first-out semantic action sequence.

[0134] Input data block 2:<!--TTS:emphasis=strong--> first<!--TTS:emphasis=reset--> We need to prepare the following ingredients: The specific parsing process for this data block will not be elaborated here. Referring to the parsing process for data block 1 described above, semantic action 3 is obtained directly: Action 3 (Speak): {type:'SPEAK',payload:{text:'First',style:'EMPHASIS_STRONG'},source_id:'block_1'}; Input data block 3: Chicken wings (middle section): 500g Coca-Cola: 1 can<!--TTS:pause=0.5s--> (Approximately 330 ml); The analysis yielded the following actions: Action 4 (Speak): {type:'SPEAK',payload:{text:'500g chicken wings', '1 can of cola',style:'LIST_ITEM'},source_id:'block_2_part_1'}; Action 5 (Pause): {type:'PAUSE', payload:{duration:0.5s}}; Action 6 (Speak): {type:'SPEAK',payload:{text:'(approximately 330 ml)',style:'LIST_ITEM'},source_id:'block_2_part_2'}; Input stream:<!--TTS:ignore_start--> Developer's Note: Plain Coca-Cola is recommended here; Coca-Cola Zero is not recommended.<!--TTS:ignore_end--> ; The streaming semantic parser has entered the detection range.<!--TTS:ignore_start--> This triggers the entry into the IN_IGNORE_BLOCK state, where all characters between the two tags are discarded without any action.

[0135] It should be noted that formatting tags in Markdown data blocks, such as the heading symbol # and list item symbols, are subject to change. Or -, bold symbol Link syntax such as []() is directly discarded. There's no need to use the `ignore` tag mentioned above; instead, the streaming semantic parser actively discards them after detection.

[0136] Because ## is a Markdown syntax symbol, it has a clear semantic structure (representing a second-level heading). You can actively parse its structure, extract the text, and discard the formatting symbols, rather than passively blocking out the entire block of content with comment tags.

[0137] If you use `ignore` to wrap the "##Preparation" phase, it will also skip the title text "Preparation Phase," meaning nothing will play. Only comments without a fixed syntax structure, such as "Developer's Note: Plain Coke is recommended here; Coke Zero is not effective," need to be discarded using the `ignore` tag.

[0138] The semantic actions generated above are written into the semantic action sequence in the order of their generation.

[0139] Step 6: The TTS control module extracts semantic actions sequentially from the semantic action sequence and performs subsequent operations based on the extracted action type.

[0140] In one example, the mapping relationship between semantic action types and their corresponding execution action logic is shown in the table below:

[0141] Step 7: A third-party TTS service synthesizes a layered speech based on the text and corresponding style parameters.

[0142] In one example, the smart broadcast effect is as follows: (Raising tone) "Preparation stage." (After a brief pause, emphasizing the point) "First, we need to prepare:" "Chicken wings—500 grams." (Half-second pause) "Cola—one can." It should be noted that in this embodiment of the invention, the sentence buffer queue is replaced by a semantic action sequence, but a dynamic preloading mechanism can also be used to extract semantic actions.

[0143] This invention, through the introduction of semantic style tags (such as...),<!--TTS:emphasis=strong--> ,<!--TTS:pause=0.5s--> Features such as speech rate variation (fast / slow reading), volume adjustment (soft / emphasis), and pause duration (short pauses / paragraph intervals) enable fine-grained control over text, achieving a rhythmic quality close to that of a natural voice and avoiding a mechanical, monotonous "robot reading," significantly improving the user's auditory experience. Furthermore, redundant content such as developer comments is not read aloud, improving the accuracy of voice playback.

[0144] Based on the above system embodiments, this invention provides a voice interaction method, such as... Figure 4 As shown, the method includes the following steps: Step S401: Receive a voice interaction request input by the user.

[0145] Step S402: Call the large language model to generate the corresponding response content, wherein the large language model outputs the response content block by block in a streaming manner during the generation process.

[0146] Step S403: While the large language model outputs each data block, the output data blocks are parsed in a streaming manner to obtain several texts with complete semantics to be broadcast. In this embodiment of the invention, each text to be broadcast is cached in a sentence cache queue as a text unit.

[0147] Step S404: Submit the text to be broadcast to the speech synthesis unit in sequence to obtain the corresponding audio data, and play the audio data.

[0148] This invention utilizes a streaming interactive approach that generates and transmits data blocks one by one using a large language model. This approach parses data blocks and converts text into speech, thereby achieving the goal of responding and broadcasting simultaneously and meeting the real-time requirements of interaction.

[0149] Speech synthesis typically takes hundreds of milliseconds to several seconds to complete. Text unit generation, however, is often much faster than speech synthesis, especially when LLM streams short sentences. If a serial mode is used, where the speech synthesized from each text unit is played before requesting the synthesis of the next text unit, the user will perceive noticeable speech interruptions, resulting in a choppy interaction.

[0150] Based on the above embodiments, the embodiments of the present invention further include the following steps: Step S405: While the audio data corresponding to the current text to be played is being played, a preloading mechanism is used to continuously initiate speech synthesis requests for subsequent texts to be played, thereby obtaining a preloaded audio stream.

[0151] The preloading mechanism involves adaptively adjusting the preloading window based on real-time measured network round-trip time and speech synthesis service processing latency, and preloading subsequent text to be played based on the preloading window. A detailed explanation of this preloading mechanism can be found in the system embodiment described above, and will not be repeated here.

[0152] Step S406: Cache the preloaded synthesized audio stream in the local audio buffer.

[0153] The local audio buffer has a maximum cache capacity set by default. When the cache is full, the least accessed audio data will be automatically removed.

[0154] The TTS synthesis request is to convert the content of a text unit into audio data.

[0155] The embodiments of the present invention, through an adaptive preloading window mechanism and intelligent caching management of local audio, enable smooth playback to be maintained adaptively under various network and service load conditions, avoiding stuttering or excessive caching, and ensuring the continuity and smoothness of voice broadcasting.

[0156] The content generation process of a large language model is essentially a probability-based sequence construction process that outputs tokens one by one. Therefore, the choice of output format directly determines the efficiency, cost, and difficulty of subsequent processing.

[0157] Therefore, based on the above implementation, the following steps are also included before invoking the large language model for voice interaction: Step S402A: Construct prompt words and specify the text format of the large language model output as Markdown in the prompt words.

[0158] Step S402B sends the prompt word and the user's voice interaction request together to the large language model.

[0159] Markdown format, with its plain text and lightweight markup, allows large language models to efficiently and accurately express rich document structures with minimal token overhead (e.g., ## represents a second-level heading). This format not only significantly improves the generation speed and reduces the cost of large language models, but also ensures that data blocks are parallel and not nested, providing an unambiguous data foundation for subsequent streaming parsing and semantic extraction. Correspondingly, step S403 specifically includes the following steps: Step S403A: Identify Markdown tags in each data block; Step S403B: Trigger the pre-configured stack-based finite state machine to enter the corresponding state based on Markdown tags, and parse the data block content in the corresponding state to obtain several texts with complete semantics to be broadcast.

[0160] This embodiment uses a stack-based finite state machine to accurately identify stream interruption points and structural boundaries, and extract complete semantic units (such as a complete step) without ambiguity, providing reliable input for subsequent TTS and serving as a key parsing guarantee for achieving high-quality streaming.

[0161] In one feasible implementation, for data blocks with complex structures, such as Markdown tables, the terminal-side parsing process includes: Step A: In response to the stack-based finite state machine entering the table header parsing state (PARSING_TABLE_HEADER), extract the column headers from the table header and store them as an array of column names.

[0162] The column headings are, for example, (Ingredients | Quantity | Notes).

[0163] Step B: In response to the stack-based finite state machine entering the table row parsing state, the cell data of the current row is semantically bound to the array to generate structured data in the form of key-value pairs.

[0164] Specifically, when entering the PARSING_TABLE_ROW state, the data cells are parsed row by row, and semantic binding is performed using the saved header array to construct each row into a key-value pair structure.

[0165] For example: {Ingredients: "Eggs", Quantity: "3", Note: "Room temperature"}.

[0166] Step C: Convert the speech into natural language text based on a predefined speech template.

[0167] In one example, the template format is as follows: First, a summary: "Next is an ingredient list with three columns: Ingredients, Quantity, and Remarks." Then broadcast line by line: "The first line states the ingredient is eggs, the quantity is 3, and the note is 'room temperature'." Additionally, Markdown includes numerous formatting tags, such as the heading symbol # and list item symbols. Or -, bold symbol Linking syntax such as [](). If this raw Markdown text is provided directly to a third-party TTS service without processing, the third-party TTS service will also convert these non-semantic formatting symbols into speech (for example, incorrectly converting "##preparation work" into "hash hash preparation work"), generating a lot of meaningless noise and interfering with the user's understanding of the core information.

[0168] Therefore, in another embodiment, the following steps are also included: Step S407: When the data block is unstructured text, remove the non-semantic Markdown tags and retain the plain text content as the broadcast text.

[0169] This embodiment transforms complex structured data into human-readable narrative language while eliminating format noise, making complex information such as ingredient lists and step lists naturally understandable, greatly expanding the information carrying capacity of the voice assistant.

[0170] When voice broadcasts are interrupted due to factors such as checking phone notifications, answering calls, the device automatically locking the screen to save power, or network issues, it is often impossible to resume from the exact point of interruption. Users need to manually find the progress or listen from the beginning, resulting in low interaction efficiency.

[0171] Therefore, based on the above embodiments, the embodiments of the present invention further include the following steps: Step S408: Listen for voice interruption events and generate a session snapshot when a voice interruption event is detected.

[0172] In this embodiment of the invention, interruption events mainly include two categories: one is expected interruption, such as page switching, page closing, and page freezing; the other is unexpected interruption, such as network disconnection and WebSocket connection abnormality.

[0173] The session snapshot encapsulates the text that has been played, is being played, and is yet to be played; the ID of the text currently being played and the timestamp of the audio stream breakpoint corresponding to that text.

[0174] Step S409: When the interruption is resumed, load the session snapshot.

[0175] Step S410: Determine the interruption location based on the information in the session snapshot.

[0176] Step S411: Resume voice broadcasting based on the interruption location.

[0177] The method for determining the interruption location is described in the above system embodiment and will not be repeated here.

[0178] This embodiment generates a session snapshot when a voice interruption event occurs and restores the process based on the session snapshot. This solves the pain point of interrupted interaction flow in interrupted scenarios. After switching tasks, users can resume from the precise breakpoint, ensuring the accuracy and continuity of the recovery operation and greatly improving the user experience.

[0179] Most existing systems are one-way information broadcasters, unable to handle users' immediate follow-up questions and instruction changes during the guidance process. This keeps the interaction at a simple question-and-answer level and cannot meet the dynamic needs of real cooking scenarios.

[0180] For example, a user might interrupt the voice broadcast. The system cannot accurately distinguish their intention. It cannot determine whether the user wants to pause playback, inquire about a specific detail, or completely change the task.

[0181] Unable to accurately identify the user's intent, the system doesn't know how to handle the user's "interruption"—should it pause and wait? Or should it discard the current task? Or should it "insert" a new subtask into the current task? This forces the system to adopt only the simplest strategies, such as ignoring the user's interruption or abruptly interrupting the entire process and requiring the user to start over.

[0182] Therefore, in order to achieve a natural and seamless interactive experience, in another embodiment of the present invention, the following steps are also included: Step S412: Listen for the preset wake word.

[0183] Step S413: When the wake word is detected, pause the current voice broadcast and obtain the user's interruption.

[0184] Step S414: Perform intent recognition on the interrupted content based on the pre-trained intent recognition model.

[0185] Step S415: Determine and execute the corresponding interactive operation based on the user's intent and the preset intent-interaction mapping relationship.

[0186] Interactive operations include at least one of the following: jumping to a specified step, replaying the current step, continuing playback, pausing playback, querying information about the current step, or returning to the previous step.

[0187] The mapping relationship between the preset intent and the executed operation is described in the above system embodiment and will not be repeated here.

[0188] This embodiment breaks the limitations of polling-based interaction, supports natural interruption and context-aware response, and allows users to effectively intervene at any stage of the task, enabling complex interactions such as pausing, asking for details, and changing instructions. It can adapt to the needs of dynamic and changing scenarios such as cooking, greatly improving intelligence and practicality.

[0189] In the above embodiments, plain text units are extracted to synthesize speech, and the synthesized audio data has the following problems: 1. It is impossible to control the speech rate, pauses, stress, and other speech styles of the text, resulting in mechanical and monotonous speech output; 2. If specific content (such as code, comments, internal IDs) is converted into speech, the resulting audio information will be redundant and unclear.

[0190] Based on this, embodiments of the present invention further include the following steps: Step S416: Enhance the prompt words based on predefined metadata tags.

[0191] Among them, metadata tags include semantic style tags, which are used to guide the large language model to inject control information related to speech broadcasting style into the generated response content.

[0192] Predefined metadata tags also include context tags.

[0193] The enhanced prompts and the user's voice interaction request are sent to the server so that the large language model can inject the corresponding metadata tags into the generated data block based on the enhanced prompts.

[0194] Correspondingly, the output data blocks are stream-parsed to obtain several texts with complete semantics to be broadcast, including: Step S403C: Parse the data block containing control information related to the speech broadcast style to obtain the text to be broadcast and speech style parameters with complete semantics.

[0195] In this step, the text to be played and the speech style parameters are a semantic action and are cached in the semantic action queue. Each semantic action in the semantic action queue is an executable instruction object, including the action type, the text to be played, and the speech style parameters.

[0196] The text to be broadcast is submitted sequentially to the speech synthesis unit for speech synthesis, and the corresponding audio data obtained includes: Step S404A: Submit the text to be broadcast and the speech style parameters to the speech synthesis party in sequence to synthesize the speech containing the speech style parameters.

[0197] This invention, through the introduction of semantic style tags, enables fine-grained control of text, achieving fine-grained speech style adjustment (emphasis / pause / skip), giving the broadcast a rhythm and emphasis closer to a real person, avoiding mechanical and monotonous "robot reading," and significantly improving the user's auditory experience. Furthermore, redundant content such as developer comments is discarded, improving the accuracy of voice playback.

[0198] Based on the above embodiments, the method further includes the following before invoking the large language model: Receive information from other modalities during the voice interaction process, wherein the information from other modalities includes at least images; Information from other modalities is fused with the voice interaction request.

[0199] In this embodiment, by fusing the user's voice interaction requests with other modal information acquired simultaneously (such as images of ingredients or the cooking process captured by the terminal's camera), the interaction is no longer limited to single-modal interaction of pure text or voice. Instead, it enables joint understanding and multi-dimensional analysis of voice commands and image content. For example, it can determine whether the lean-to-fat ratio of pork belly is suitable for braising or identify the degree of caramelization of ingredients in the pot. This multimodal input is then provided to a large language model (such as a visual large language model).

[0200] Thus, it evolves from a traditional "voice assistant" into a "visual intelligent agent" with environmental awareness capabilities, significantly expanding its application boundaries and practical value in real-world cooking scenarios. The method provided in this embodiment of the invention shares the same implementation principle and technical effects as the aforementioned system embodiment. For the sake of brevity, any aspects not mentioned in the method embodiment can be referred to the corresponding content in the aforementioned system embodiment, and will not be repeated here.

[0201] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces. Indirect couplings or communication connections between systems or units may be electrical, mechanical, or other forms.

[0202] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0203] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0204] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0206] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A voice interaction method, characterized in that, include: Receive user input for voice interaction requests; The large language model is invoked to generate the corresponding response content, wherein the large language model outputs the response content block by block in a streaming manner during the generation process; while the large language model outputs each data block, the output data blocks are parsed in a streaming manner to obtain several texts to be broadcast with complete semantics. The text to be broadcast is submitted sequentially to the speech synthesis unit for speech synthesis to obtain the corresponding audio data, and then the audio data is played.

2. The method according to claim 1, characterized in that, Before invoking the large language model for voice interaction, the following is also included: Construct prompt words, and specify the text format of the large language model output as Markdown format in the prompt words; The prompt word and the voice interaction request are sent together to the large language model.

3. The method according to claim 2, characterized in that, The method further includes: The prompt words are enhanced based on predefined metadata tags; wherein, the metadata tags include semantic style tags, which are used to guide the large language model to inject control information related to the voice broadcast style into the generated response content; The output data blocks are stream-parsed to obtain several texts with complete semantics to be broadcast, including: Parse the data block containing control information related to the speech broadcast style to obtain the text to be broadcast and speech style parameters with complete semantics; The text to be broadcast is sequentially submitted to the speech synthesis unit for speech synthesis, and the corresponding audio data obtained includes: The text to be played and its corresponding voice style parameters are submitted sequentially to the voice synthesis party to synthesize audio data containing the voice style parameters.

4. The method according to claim 3, characterized in that, The method further includes: While the audio data corresponding to the current text to be played is being played, a preloading mechanism is used to continuously initiate speech synthesis requests for subsequent texts to be played, thereby obtaining a preloaded audio stream. The preloading mechanism is as follows: based on the real-time measured network round-trip time and the processing latency of the speech synthesis service, the preloading window is adaptively adjusted, and the subsequent text to be played is preloaded based on the preloading window; The preloaded audio stream is cached in a local audio cache area; the local audio cache area has a preset maximum cache capacity, and when the cache is full, the least accessed audio data is automatically removed.

5. The method according to claim 2, characterized in that, The output data blocks are stream-parsed to obtain several texts with complete semantics to be broadcast, including: Identify Markdown tags in each data block; Based on the Markdown markers, a pre-configured stack-based finite state machine is triggered to enter the corresponding state, and the data block content is parsed in the corresponding state to obtain several texts with complete semantics to be broadcast.

6. The method according to claim 5, characterized in that, Parse the data block content in the corresponding state to obtain several texts to be broadcast, including: When the data block is a table structure or a nested list, in response to the stacked finite state machine entering the table header parsing state, the column headers in the table header are extracted and stored as an array associated with the current table; When the stacked finite state machine enters the table row parsing state, the cell data of the current row is semantically bound to the array to generate structured data in the form of key-value pairs; It is converted into natural language broadcast text based on predefined voice broadcast templates; When the data block is unstructured text, remove the non-semantic Markdown tags and retain the plain text content as the broadcast text.

7. The method according to claim 1, characterized in that, The method further includes: Listen for voice interruption events and generate a session snapshot when the voice interruption event is detected; the session snapshot encapsulates the text that has been played, is being played, and is to be played; the ID of the text currently being played and the timestamp of the audio stream breakpoint corresponding to the text being played; When the interruption is resumed, the session snapshot is loaded; The location of the interruption is determined based on the information in the session snapshot; Voice broadcasting is resumed based on the interruption location.

8. The method according to claim 1, characterized in that, The method further includes: Listen for preset wake words; When a wake word is detected, pause the current audio playback and capture the user's interruption. The intent of the interrupted content is determined by a pre-trained intent recognition model. Based on the user intent and the preset intent-interaction mapping relationship, the corresponding interaction operation is determined and executed. The interaction operation includes at least one of the following: jumping to a specified step, replaying the current step, continuing playback, pausing playback, querying the current step information, or returning to the previous level process.

9. The method according to claim 1, characterized in that, The method further includes the following steps before invoking the large language model: Receive information from other modalities during the voice interaction process, wherein the information from other modalities includes at least images; Information from other modalities is fused with the voice interaction request.

10. A voice interaction system, characterized in that, It includes a user input interface, a central control module, a streaming communication module, a streaming semantic parser, a TTS control module, an audio playback module, and a large language model; The user input interface is used to collect users' voice interaction requests; The streaming communication module sends the voice interaction request to the large language model and receives the response content returned by the large language model; The large language model returns the response content block by block in the form of data during the generation process; The streaming semantic parser is used to perform streaming parsing on the output data blocks while the large language model outputs each data block to obtain several texts to be broadcast. The TTS control module is used to sequentially submit the text to be played to the speech synthesis unit for speech synthesis to obtain the corresponding audio data. The audio playback module is used to play the audio data; The central control module is used, on the one hand, to generate control commands based on voice interaction requests and send the control commands to the streaming communication module; on the other hand, it is used to trigger the streaming semantic parser to perform semantic parsing based on the return results of the large language model, and to trigger the TTS control module to perform speech synthesis.

Images