Dual-path real-time voice conversation recovery method, system, device and medium

Abstract

The application relates to a dual-path real-time voice conversation recovery method, system, device and storage medium. The method comprises the following steps: obtaining user voice data, and determining valid speech audio from the user voice data through local endpoint detection; submitting the valid speech audio to a first processing path to generate and return reply audio, and starting a second processing path to return user transcription text; determining a first completion state for indicating completion of the first processing path and a second completion state for indicating completion of the second processing path, and waiting for the second completion state within a preset bounded time period when the first completion state arrives earlier than the second completion state; after the second completion state arrives or the preset bounded time period expires, associating the obtained user transcription text and reply text corresponding to the reply audio as a dialogue round, and writing the dialogue round into a recovery window with a preset capacity.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and in particular to a dual-channel real-time voice conversation recovery method, system, device and storage medium. Background Technology

[0002] Real-time voice interaction terminals typically need to complete recording, voice endpoint detection, speech recognition, semantic processing, response generation, speech synthesis, and playback sequentially. If response processing cannot begin until the complete speech-recognized text is returned, the transcription delay will be directly added to the delay of the first segment of the response audio, making the pause easily perceptible to the user.

[0003] Some real-time conversational services can directly understand the input audio and output a response audio, while simultaneously returning the transcribed text to the user. However, the response completion event and the user transcription completion event may come from different asynchronous processing pipelines, and their arrival order is uncertain. If history is recorded immediately upon response completion, user-side content may be lost because the transcription has not yet returned; if the transcription is waited for indefinitely, it will block the next round of interaction.

[0004] Additionally, network connectivity on the mobile device may be interrupted, and changes in output parameters or timbre may require re-establishing the session. If historical or current audio is sent before the session configuration takes effect during reconstruction, the remote session may process the data using default parameters; if the connection is re-established and the entire history is re-fed in every round, it will increase network load, processing latency, and context overhead. Summary of the Invention

[0005] In view of the above, this application provides a dual-channel real-time voice conversation recovery method, system, device and storage medium, the purpose of which is to solve the above-mentioned technical problems.

[0006] In a first aspect, this application provides a dual-channel real-time voice conversation recovery method, the method comprising: Acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The valid speech audio is submitted to the first processing channel to generate and return the response audio, while the second processing channel is activated to return the user's transcribed text. A first completion state for indicating the completion of the first processing path and a second completion state for indicating the completion of the second processing path are determined. When the first completion state arrives before the second completion state, the system waits for the second completion state within a preset bounded time period. After the second completion state is reached or the preset bounded time period expires, the obtained user-transcribed text and the reply text corresponding to the reply audio are associated as a dialogue turn and written into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; When a session connection needs to be rebuilt, a session configuration message is sent to the rebuilt connection. After confirming that the session configuration message has entered the sending sequence of the rebuilt connection, the connection status is updated to available. The message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. Before submitting new valid speech audio through the reconstructed connection, at least one dialogue turn within the recovery window is sent to the reconstructed connection to restore the session context.

[0007] In some embodiments, determining valid speech audio from the user speech data via local endpoint detection includes: The start and end points of the voice activity are determined based on the temporal amplitude characteristics of the user's voice data. Extract the audio segment located between the start and end points of the voice activity from the user's voice data as the initial valid segment; Obtain the pre-rolled audio frames cached before the start of the speech activity, and concatenate the pre-rolled audio frames to the beginning of the initial valid segment to form the valid speech audio.

[0008] In some embodiments, obtaining the pre-rolled audio frames cached before the start of the voice activity and concatenating the pre-rolled audio frames to the beginning of the initial valid segment includes: Maintain a pre-read cache queue, which continuously stores the most recent N frames of user voice data, where N is a preset positive integer; When the start point of the voice activity is determined, all N frames are extracted from the pre-rolled cache queue as the pre-rolled audio frames. The pre-rolled audio frames are spliced ​​together in chronological order before the initial valid segment, and the spliced ​​complete audio sequence is output as the valid speech audio to the first processing path and the second processing path.

[0009] In some embodiments, the terminal stops playing the response audio during the acquisition of the valid speech audio and stops acquiring user voice data during the playback of the response audio, so as to form a half-duplex audio path; The user-transcribed text returned by the second processing path is also used to perform at least one auxiliary control, which includes interface text display, pause intent detection, session resumption content generation, and fault diagnosis; the auxiliary control does not trigger the secondary generation of the response audio.

[0010] In some embodiments, when the second completion state has not been reached within the preset bounded time period, a timeout processing operation is performed, the timeout processing operation including: Destroy the asynchronous waiting object corresponding to this round, and unbind the second completion state from the waiting callback function of this round; Release the timer resources occupied by the preset bounded time period and switch the state machine of this round to the timeout completion state; Associate the partially received user transcribed text or empty placeholders with the reply text as the dialogue round and write it into the recovery window, releasing the asynchronous mutex lock used to block subsequent audio acquisition; When the second completion state arrives after the timeout processing operation is completed, the complete user transcription text carried by the second completion state is re-associated with the reply text of the corresponding round in the recovery window, and the corresponding entry in the recovery window is updated in a copy-on-write manner without blocking the ongoing subsequent dialogue rounds.

[0011] In some embodiments, destroying the asynchronous waiting object corresponding to the current round and unbinding the second completion state from the waiting callback function of the current round includes: Obtain the asynchronous event handle associated with this round, call the underlying operating system interface to set the asynchronous event handle to an invalid state, and reclaim the kernel object resources occupied by the asynchronous event handle; Iterate through the registry of callback functions for the second completion state, using the round identifier of this round as the index to find the bound waiting callback functions; When the waiting callback function is found, the waiting callback function is removed from the callback function registry, and the corresponding entry in the callback function registry is marked as invalid; Send a reconciliation confirmation message to the second processing path so that the second processing path will no longer trigger the unbound waiting callback function when the second completion state is reached.

[0012] In some embodiments, releasing the asynchronous mutex lock used to block subsequent audio acquisition includes: Obtain the current lock state of the asynchronous mutex, the current lock state including the lock holder identifier and the lock waiting queue; After verifying that the lock holder identifier is consistent with the thread identifier of the currently executing thread, the holder identifier of the asynchronous mutex lock is cleared, and the first waiting thread in the lock waiting queue is removed from the lock waiting queue. The scheduling state of the first waiting thread is switched from blocked to ready, and the first waiting thread is submitted to the operating system scheduler to resume execution. Before releasing the asynchronous mutex, the atomic operation that switches the state machine to the timeout completion state in the current round is merged with the operation that releases the asynchronous mutex into the same critical section transaction to ensure that the state machine switch and the lock release cannot be interrupted or separated in a multi-threaded environment.

[0013] Secondly, this application provides a dual-channel real-time voice session recovery system, which includes: The acquisition module is used to acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The first processing module is used to submit the valid speech audio to the first processing channel to generate and return the response audio, and at the same time start the second processing channel to return the user's transcribed text; The determination module is used to determine a first completion state indicating the completion of the first processing path and a second completion state indicating the completion of the second processing path. When the first completion state arrives before the second completion state, the module waits for the second completion state within a preset bounded time period. The second processing module is used to associate the obtained user-transcribed text and the reply text corresponding to the reply audio as a dialogue turn after the second completion state is reached or the preset bounded time period expires, and write it into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; The reconnection module is used to send a session configuration message to the reconstructed connection when the session connection needs to be reconstructed. After confirming that the session configuration message has entered the sending sequence of the reconstructed connection, the connection status is updated to available. The message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. The sending module is used to send at least one dialogue turn within the recovery window to the reconstructed connection before submitting new valid speech audio through the reconstructed connection, in order to restore the session context.

[0014] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the dual-channel real-time voice session recovery method described in any embodiment of the first aspect.

[0015] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the dual-channel real-time voice session recovery method as described in any embodiment of the first aspect.

[0016] The technical solutions provided in this application have the following advantages compared with the prior art: By decoupling the generation of response audio from the user-transcribed text and processing them in parallel, the first processing path can output response audio without waiting for the second processing path, significantly reducing the impact of transcription delay on the timing of the first response audio segment. By using independent completion states and bounded waiting mechanisms to adapt to the asynchronous out-of-order arrival of response events and transcription events, the system avoids blocking the session due to transcription failures while preserving the complete history as much as possible. By feeding back a limited history window only when the connection is rebuilt and explicitly prioritizing the entry of session configuration messages into the sending sequence, the system ensures that the remote session has completed parameter settings and history loading before processing the current audio, reducing the amount of repeated context transmission and improving the context continuity and parameter consistency after reconnection. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a preferred embodiment of the dual-channel real-time voice conversation recovery method of this application; Figure 2 This is an exemplary flowchart illustrating the execution of timeout handling operations as shown in some embodiments of this application; Figure 3 This is a schematic diagram of the dual-channel real-time voice session recovery system structure shown in some embodiments of this application; Figure 4The following are exemplary timing diagrams illustrating asynchronous coordination timing in some embodiments of this application; Figure 5 This is a schematic diagram illustrating the connection reconstruction and history restoration process in some embodiments of this application; Figure 6 This is a schematic diagram of a preferred embodiment of the dual-channel real-time voice conversation recovery system of this application; Figure 7 This is a schematic diagram of a preferred embodiment of the electronic device of this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0022] Reference Figure 1 The diagram shown is a flowchart illustrating an embodiment of the dual-channel real-time voice session recovery method of this application. The method is executed by an electronic device, which can be implemented by a software system and / or a hardware system. The dual-channel real-time voice session recovery method includes: Step 101: Acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection.

[0023] User voice data refers to the raw pulse code modulated audio signal collected by the terminal device through the audio input device.

[0024] Local endpoint detection refers to performing voice activity detection on user voice data locally on the terminal to identify valid regions containing human voices.

[0025] Effective speech audio refers to audio segments extracted from user voice data that actually contain the user's speech content.

[0026] The terminal stops playing the response audio while collecting the valid speech audio, and stops collecting user voice data while playing the response audio, thus forming a half-duplex audio path.

[0027] In some embodiments, the processor acquires user voice data by sampling audio at a 24,000 Hz sampling rate, 16-bit quantization, and dual-channel audio acquisition, reading one frame every 50 milliseconds, and averaging the left and right channel samples to obtain single-channel audio.

[0028] In some embodiments, the processor determines valid speech audio through local endpoint detection by: calculating the root mean square amplitude and peak value for each single-channel frame; storing the most recent frames before entering the speaking state in a pre-roll queue; determining the start of a speech when the root mean square amplitude or peak value of at least two consecutive frames reaches a starting threshold, and writing the pre-roll frame into the speech buffer to avoid swallowing the first syllable; after entering the speaking state, determining the end of a speech when the speech duration reaches the minimum duration and the continuous silence reaches the end duration; forcibly ending the speech when the maximum speech duration is reached; and determining the audio segment between the start and end of the speech as valid speech audio. The pre-roll queue stores approximately 300 milliseconds of audio, the shortest speech duration is 600 milliseconds, the end silence duration is 500 milliseconds, and the longest speech duration is 12000 milliseconds.

[0029] In some embodiments, determining valid speech audio from the user speech data via local endpoint detection includes the following operations: S11, determine the start and end points of the voice activity based on the temporal amplitude characteristics of the user's voice data.

[0030] Time-domain amplitude characteristics refer to the amplitude variation characteristics of an audio signal in the time domain, including parameters such as root mean square amplitude, short-time energy, peak amplitude, and zero-crossing rate.

[0031] The starting point of voice activity refers to the moment when a user begins to speak, that is, the dividing point in the user's voice data from a silent state to an effective voice state.

[0032] The endpoint of voice activity refers to the moment when a user ends speaking, that is, the dividing point in the user's voice data from an active voice state to a silent state.

[0033] In some embodiments, the processor determines the start and end points of voice activity based on the temporal amplitude characteristics of user voice data, including: calculating the root mean square amplitude and peak value for each single-channel frame; storing the most recent frames before entering the speaking state in a pre-roll queue; determining the moment as the start of voice activity when the root mean square amplitude or peak value of at least two consecutive frames reaches a start threshold; continuously monitoring the root mean square amplitude and peak value of each frame after entering the speaking state; and determining the moment as the end of voice activity when the effective speech duration reaches the minimum duration and the root mean square amplitude and peak value of consecutive silent frames are lower than the end threshold and reach the end duration. The start and end thresholds can be calibrated based on the device noise floor.

[0034] S12, extract the audio segment located between the start point and the end point of the voice activity in the user voice data as the initial valid segment.

[0035] The initial valid segment refers to a continuous audio fragment extracted from the user's speech data, starting from the beginning of the speech activity and ending at the end of the speech activity. It contains the main content of a complete speech by the user but does not include the pre-written portion before the beginning of the speech activity.

[0036] In some embodiments, the processor extracts an audio segment from the user's speech data located between the start and end points of the speech activity as an initial valid segment, comprising: obtaining a first timestamp of the start point of the speech activity in the user's speech data and a second timestamp of the end point of the speech activity in the user's speech data; the processor truncating all audio samples between the first timestamp and the second timestamp from the buffer of the user's speech data, and writing the truncated result as the initial valid segment into the speech buffer. The initial valid segment is in a single-channel pulse code modulation audio format.

[0037] In some embodiments, the processor maintains a circular audio buffer to continuously store recently acquired user voice data; when the processor determines the start point of a voice activity, it records the start index of the start point in the circular buffer; when the processor determines the end point of a voice activity, it records the end index of the end point in the circular buffer; the processor reads all audio samples between the start and end indices from the circular buffer as the initial valid segment. When the user voice data is multi-channel pulse code modulation audio, the processor first mixes the multi-channel audio into a single-channel audio before extracting the initial valid segment.

[0038] S13, obtain the pre-rolled audio frames cached before the start of the speech activity, and splice the pre-rolled audio frames to the beginning of the initial valid segment to form the valid speech audio.

[0039] Pre-rolled audio frames refer to the most recent frames of user voice data that are continuously cached in the pre-roll queue before the start of the voice activity is determined.

[0040] Effective speech audio refers to the complete audio sequence obtained by splicing pre-rolled audio frames to the beginning of the initial effective segment. This complete audio sequence retains the complete transition information of the speech beginning.

[0041] In some embodiments, the processor acquires pre-rolled audio frames cached before the start of the voice activity and concatenates these pre-rolled audio frames to the beginning of the initial valid segment. This includes: maintaining a pre-rolled cache queue to continuously store the most recent N frames of user voice data; upon determining the start of the voice activity, the processor extracts all N frames from the pre-rolled cache queue as pre-rolled audio frames, where N is a preset positive integer; the processor concatenates the pre-rolled audio frames in chronological order before the initial valid segment and outputs the concatenated complete audio sequence as valid speech audio to the first processing path and the second processing path. Here, N corresponds to approximately 300 milliseconds of audio duration.

[0042] In this embodiment, by saving the pre-rolled audio frame before the start of the speech activity and splicing the pre-rolled frame to the beginning of the initial valid segment, the first syllable of the user's speech is not swallowed due to detection delay, thereby ensuring the integrity of the valid speech audio submitted to the first processing path and the second processing path, and improving the accuracy of response audio generation and user transcribed text.

[0043] In some embodiments, obtaining the pre-rolled audio frames cached before the start of the voice activity and concatenating the pre-rolled audio frames to the beginning of the initial valid segment includes the following operations: S21, maintain a pre-roll cache queue, which continuously stores the most recent N frames of user voice data, where N is a preset positive integer.

[0044] The pre-loaded buffer queue refers to a first-in, first-out (FIFO) buffer allocated in the terminal device's memory. This buffer is specifically used to temporarily store user voice data frames acquired before the start of voice activity is determined. The most recent N frames refer to the N newest audio frames retained by this queue according to the FIFO rule, where N is a pre-set positive integer corresponding to the desired audio duration. User voice data refers to the raw pulse-code modulated audio signal acquired by the terminal device through the audio input device.

[0045] In some embodiments, the processor maintains a pre-read cache queue by: allocating a fixed-size circular buffer in memory as the pre-read cache queue, the circular buffer having a storage capacity of N frames of audio data; the processor continuously writing each frame of user speech data into the circular buffer before entering a speaking state; and when the circular buffer is full, the processor overwrites the oldest frame with the latest frame to ensure that the queue always stores the most recent N frames of user speech data. For example, N is 6, corresponding to an audio duration of 300 milliseconds.

[0046] S22, when determining the starting point of the voice activity, extract all N frames from the pre-rolled cache queue as the pre-rolled audio frames.

[0047] The speech activity start point refers to the moment when the processor detects the start of the user's speech based on the temporal amplitude characteristics of the user's speech data.

[0048] Pre-rolled audio frames refer to all N frames of user speech data extracted from the pre-rolled buffer queue. These pre-rolled audio frames are located before the start of speech activity and are used to complete the audio content missed by the detected delay at the beginning of the speech.

[0049] In some embodiments, the processor extracts all N frames as pre-rolled audio frames from the pre-rolled buffer queue when determining the start of voice activity, including: determining the start of voice activity when the root mean square amplitude or peak value of at least two consecutive frames reaches a start threshold; the processor immediately freezes the pre-rolled buffer queue to stop receiving new frames; the processor sequentially reads all N frames of audio data starting from the head of the pre-rolled buffer queue and uses the read N frames as a whole as a pre-rolled audio frame; after reading is completed, the processor clears the pre-rolled buffer queue to prepare for the next round of pre-rolling.

[0050] S23, the pre-rolled audio frames are spliced ​​together in chronological order before the initial valid segment, and the spliced ​​complete audio sequence is output as the valid speech audio to the first processing path and the second processing path.

[0051] The time sequence refers to the order in which the pre-rolled audio frames are acquired and their temporal relationship relative to the initial valid segment.

[0052] A complete audio sequence refers to a single continuous audio data stream obtained by concatenating pre-rolled audio frames and initial valid segments in chronological order.

[0053] The first processing path refers to a real-time inference service that receives audio input and directly outputs streaming response audio and response text.

[0054] The second processing path refers to the speech transcription service that works in parallel with the first processing path.

[0055] In some embodiments, the user-transcribed text returned by the second processing path is also used to perform at least one auxiliary control, the auxiliary control including interface text display, pause intent detection, session resumption content generation, and fault diagnosis; the auxiliary control does not trigger secondary generation of the response audio.

[0056] In some embodiments, the processor splices pre-rolled audio frames sequentially before the initial valid segment, including: reading the acquisition timestamps of each frame in the pre-rolled audio frames, arranging all pre-rolled audio frames into a first audio sequence according to the acquisition timestamps from earliest to latest; placing the arranged first audio sequence before the initial valid segment, and continuously connecting the last frame of the first audio sequence with the first frame of the initial valid segment at the sampling point; and the processor using the spliced ​​complete audio sequence as valid speech audio, and simultaneously outputting it to the transmission buffers of the first processing path and the second processing path.

[0057] In some embodiments, before the processor splices the pre-rolled audio frames sequentially to the initial valid segment, the processor further includes: storing the pre-rolled audio frames and the initial valid segment in two consecutive memory blocks respectively; creating a new consecutive memory block with a total length equal to the sum of the length of the pre-rolled audio frames and the length of the initial valid segment; copying the pre-rolled audio frames sequentially to the beginning of the new memory block in their original frame order, and then copying the initial valid segment to the position immediately following the pre-rolled audio frames in the new memory block; and distributing all the data in the new memory block as valid speech audio to the first processing path and the second processing path simultaneously through a parallel transmission interface.

[0058] In this embodiment, by continuously caching the most recent N frames of user speech data before the start point of speech activity is determined and extracting all cached frames immediately after the start point is determined and splicing them into the initial valid segment, it is ensured that the truncated syllables of speech due to endpoint detection delay are completely preserved, thereby avoiding the problem of losing the first character of valid speech audio and improving the input audio quality of the first processing path response audio generation and the second processing path user transcribed text.

[0059] Step 102: Submit the valid speech audio to the first processing channel to generate and return the response audio, and at the same time start the second processing channel to return the user's transcribed text.

[0060] Response audio refers to a sequence of audio data blocks generated by the first processing path based on valid speech audio and returned in a streaming manner.

[0061] User-transcribed text refers to the text content output after the second processing pathway performs speech recognition on valid speech audio.

[0062] In some embodiments, the processor submits valid speech audio to the first processing path by: encoding the valid speech audio into an audio append message according to a preset block size and sending it continuously, followed by sending an audio submission message and a reply creation message.

[0063] The processor generates and returns response audio through the first processing path, including: the remote first processing path directly processes the audio content and returns multiple response audio blocks in the order of arrival; the processor immediately writes the response audio blocks into the audio output buffer for playback after receiving them, without waiting for the complete response to end.

[0064] The processor initiates a second processing path to return the transcribed text to the user, which includes: the processor sending an asynchronous transcription request to the second processing path for the same valid speech audio; the second processing path completing speech recognition independently of the first processing path and then returning the transcribed text to the user.

[0065] In other embodiments, the first processing path and the second processing path are two parallel processing pipelines within the same remote session service. The processor submits valid speech audio to both the first and second processing paths simultaneously through a single connection, wherein the response audio generation of the first processing path and the transcription text generation of the second processing path are decoupled and executed in parallel without blocking each other.

[0066] Step 103: Determine a first completion state to indicate the completion of the first processing path and a second completion state to indicate the completion of the second processing path. When the first completion state arrives before the second completion state, wait for the second completion state within a preset bounded time period.

[0067] The first completion status refers to the event flag used to indicate that the first processing path has completed the generation of the current round of response audio.

[0068] The second completion status refers to the event flag used to indicate that the second processing path has completed the generation of the user's transcribed text in the current round.

[0069] The preset bounded time period refers to the maximum time length that is pre-set for waiting for the second completion state to arrive, and its value can be dynamically adjusted according to network conditions.

[0070] In some embodiments, the processor maintains a first completion flag and a second completion flag for each dialogue round, with both flags initially unset.

[0071] The processor determines the first completion status by setting the first completion flag when it receives a response completion event from the first processing path.

[0072] The processor determines the second completion status by setting the second completion flag when it receives a transcription completion event or transcription failure event returned by the second processing path. When the first completion flag is set before the second completion flag, the processor waits for a preset bounded period of 1500 milliseconds using a condition variable.

[0073] When the second completion flag is set within the preset bounded time period, the processor stops waiting and continues subsequent processing; when the preset bounded time period expires and the second completion flag is still not set, the processor determines that the waiting time has expired.

[0074] Step 104: After the second completion state is reached or the preset bounded time period expires, the obtained user-transcribed text and the reply text corresponding to the reply audio are associated as a dialogue round and written into a recovery window with a preset capacity.

[0075] The response text refers to the textual description corresponding to the content of the response audio, returned by the first processing path at the same time as generating the response audio.

[0076] A dialogue round refers to a complete dialogue unit consisting of the content spoken by the user and the corresponding response from the assistant in the same round of interaction, including the pairing relationship between the user's transcribed text and the response text.

[0077] The recovery window is a cache structure used to cache the most recent M conversation rounds, where M is a positive integer greater than 1. The recovery window can be used to provide historical context when the connection is re-established.

[0078] The preset capacity refers to the maximum number of dialog turns that the restore window can store.

[0079] In some embodiments, when the second completion state is reached, the processor combines the user-transcribed text and the response text into the same dialogue round and writes it to the recovery window. After a preset bounded timeout, the processor combines empty transcription or partially received transcription with the response text into the recovery window and writes it to the recovery window, then begins the next recording to prevent asynchronous transcription failures from blocking the real-time session. The recovery window is a first-in, first-out sliding window structure. After writing each new dialogue round, the processor checks the total number of rounds in the window; when the total number of rounds exceeds a preset capacity, the processor deletes the earliest written dialogue round. The preset capacity is the most recent 20 dialogue rounds.

[0080] Step 105: When a session connection needs to be rebuilt, a session configuration message is sent to the rebuilt connection. After confirming that the session configuration message has entered the sending sequence of the rebuilt connection, the connection status is updated to available.

[0081] A session connection refers to a network communication link established between a terminal and a remote session service for real-time voice interaction.

[0082] A re-established connection refers to a new connection that is established after the original connection has been closed due to reasons such as network disconnection, changes in session parameters, or changes in output tone.

[0083] A session configuration message is an instruction message that carries configuration parameters such as input audio format, output audio format, output timbre, processing model, response modality, and whether remote endpoint detection is disabled. The message content of a session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version.

[0084] A sending sequence refers to an ordered queue of messages to be sent on the same connection, with messages being sent sequentially according to the order in which they entered the sending sequence.

[0085] Connection status refers to a shared flag indicating whether the current connection can be used by the audio worker thread.

[0086] In some embodiments, when the processor detects a network disconnection, a change in output tone, or other changes in session parameters that require reconnection, it actively closes the current connection while retaining the recovery window. After establishing a new connection, the processor immediately calls the blocking write interface to send session configuration messages in the new connection open callback.

[0087] The processor confirms that the session configuration message has entered the transmission sequence of the rebuilt connection by: after receiving a transmission success event from the underlying transport protocol, the processor marks the session configuration message as enqueued. After the session configuration message is marked as enqueued, the processor sets the shared connection availability flag and notifies the audio worker threads waiting for a connection. This ensures that even if the audio worker threads are immediately awakened, subsequent messages are still sent after the configuration message, preventing audio from being processed by the remote default configuration.

[0088] Step 106: Before submitting new valid speech audio through the reconstructed connection, at least one dialogue turn in the recovery window is sent to the reconstructed connection to restore the session context.

[0089] New valid speech audio refers to user speech audio that is newly collected by the terminal and determined by endpoint detection after the connection is rebuilt.

[0090] Historical messages refer to a sequence of input messages marked with user and assistant roles, which are transformed from the dialogue turns within the recovery window.

[0091] Session context refers to the information from previous rounds of interaction that a remote session service needs to understand the current conversation, including historical user input and corresponding assistant responses.

[0092] In some embodiments, the processor wakes up the audio worker thread after the connection status is updated to available. The audio worker thread checks whether there are any historical messages to be re-fetched. If there are, the processor converts the dialogue turn in the recovery window into at least one historical input message with user role and assistant role tags, sends the historical input message first, and then sends the audio append message, audio commit message, and reply creation message of the current turn in sequence. After sending, the processor clears the re-fetching flag. During the sending of historical messages, the processor prohibits the insertion of messages carrying new valid utterance audio into the sending queue until all historical messages have been sent.

[0093] In this embodiment, by decoupling and parallelizing the generation of response audio with the user-transcribed text, the first processing path can output response audio without waiting for the second processing path, significantly reducing the impact of transcription delay on the timing of the first response audio segment. By using independent completion states and bounded waiting mechanisms to adapt to the asynchronous out-of-order arrival of response events and transcription events, the system avoids blocking the session due to transcription failures while preserving the complete history as much as possible. By only feeding back a limited history window when the connection is rebuilt and explicitly prioritizing the entry of session configuration messages into the sending sequence, the system ensures that the remote session has completed parameter settings and history loading before processing the current audio, reducing the amount of repeated context transmission and improving the context continuity and parameter consistency after reconnection.

[0094] Reference Figure 2 The diagram shown is an exemplary flowchart illustrating the execution of timeout handling operations according to some embodiments of this application. In some embodiments, the process includes the following operations: Step 201: Destroy the asynchronous waiting object corresponding to this round and unbind the second completion state from the waiting callback function of this round.

[0095] An asynchronous wait object is an instance of a synchronization primitive provided by the operating system for inter-thread synchronization. This instance allows the current thread to enter a blocked state that can be awakened before the second completion state is reached.

[0096] The callback function awaited in this round refers to a pre-registered function pointer that is automatically called by the system when the second completion state is reached. This function is used to notify the current thread that the rewriting is complete.

[0097] In some embodiments, the processor destroys the asynchronous wait object corresponding to the current round and unbinds the second completion state from the wait callback function of the current round, including: obtaining the handle of the condition variable object associated with the current round, calling the condition variable's destruction interface to release the object's wait queue resource in the kernel; the processor traverses the callback function registration list of the second completion state, searching for the bound callback function entry using the round identifier of the current round as the key; after finding the entry, the processor removes it from the list and sets it to an invalid state; the processor sends a rebinding request to the second processing path so that the second processing path will not trigger the unbound callback function when the second completion state is subsequently generated.

[0098] Step 202: Release the timer resources occupied by the preset bounded time period and switch the state machine of this round to the timeout completion state.

[0099] Timer resources refer to timer objects or timed task entries allocated by the operating system to manage timeout waits. These resources are created or occupied when a preset bounded time period begins.

[0100] A state machine is a set of states in different completion stages of a round and the rules for transitions between states. It includes at least the waiting state, the timed-out completion state, and the normal completion state.

[0101] The timeout completion state refers to a specific state value in the state machine that represents the end of the waiting period due to the expiration of a preset bounded time period.

[0102] In some embodiments, the processor releasing the timer resources occupied by a preset bounded time period and switching the state machine of the current round to the timeout completed state includes: the processor calling the timer cancellation function to terminate the running timer, releasing the timer's time wheel slot and expiration callback queue in the kernel; the processor atomically switching the state variable of the current round from the waiting state to the timeout completed state, the atomic switch being implemented by comparing and swapping instructions to prevent multi-threaded contention; and the processor sending a release notification to the timer manager after the state variable switch is completed, causing the timer manager to remove the timeout record from the list of active timers.

[0103] Step 203: Associate the partially received user transcribed text or empty placeholders in this round with the reply text as the dialogue round and write it into the recovery window, and release the asynchronous mutex lock used to block subsequent audio acquisition.

[0104] Partially transcribed text refers to the intermediate results of user speech recognition that have been returned by the second processing pathway within a preset bounded time period but are not yet complete. These intermediate results may contain some recognized words or sentence fragments.

[0105] Empty placeholders are predefined markers used to replace user-transcribed content in a dialog round when the transcribed text is completely missing.

[0106] In some embodiments, the processor associates the partially received user-transcribed text or empty placeholders with the response text in the current round as a dialogue round and writes it into the recovery window. Releasing the asynchronous mutex lock used to block subsequent audio acquisition includes: the processor checking whether the partially received user-transcribed text in the current round is empty; if it is not empty, it generates a dialogue round by pairing the partially transcribed text with the response text; if it is empty, it generates a dialogue round by pairing an empty placeholder with the response text; the processor writes the generated dialogue round to the end of the recovery window and increments the window counter; the processor obtains the current holder identifier of the asynchronous mutex lock, verifies that the holder identifier is consistent with the current thread identifier, and then calls the unlock operation to release the mutex lock, allowing the audio acquisition thread waiting for the lock to continue execution.

[0107] Step 204: When the second completion state arrives after the timeout processing operation is completed, the complete user transcription text carried by the second completion state is re-associated with the reply text of the corresponding round in the recovery window, and the corresponding entry in the recovery window is updated in a copy-on-write manner without blocking the ongoing subsequent dialogue rounds.

[0108] Delayed arrival refers to the situation where the second completion status is received by the processor only after the timeout processing operation has been completed.

[0109] Complete user-transcribed text refers to the final transcription result containing complete semantic content returned by the second processing pathway after all speech recognition is completed.

[0110] Copy-on-write refers to a method where the processor, when updating an entry in the recovery window, does not directly modify the content of the original storage location. Instead, it creates a copy of the entry, performs the modification operation on the copy, and then points the pointer in the recovery window to the new copy to complete the update.

[0111] In some embodiments, when the second completion state arrives late, the processor re-associates the complete user-transcribed text with the response text of the corresponding round in the recovery window, and updates the corresponding entry in the recovery window in a copy-on-write manner, including: the processor receiving the second completion state and extracting the complete user-transcribed text and round identifier from it; the processor locating the corresponding dialogue round entry in the recovery window according to the round identifier; the processor creating a complete copy of the dialogue round entry and replacing the user-transcribed text field in the copy with the complete user-transcribed text; the processor using an atomic pointer swap operation to replace the pointer in the recovery window pointing to the original entry with a pointer pointing to the new entry; and the processor marking the original entry as reclaimable and placing it in a delayed release queue for subsequent reclamation.

[0112] In this embodiment, a timeout handling operation destroys the waiting object, releases timer resources and mutex locks when transcription is not completed in time, ensuring that subsequent audio acquisition is not blocked and guaranteeing the smoothness of the session. The state machine switching accurately records the state of the current round ending due to timeout, avoiding repeated waiting. By updating the corresponding entry in the recovery window with copy-on-write when the delay arrives, it is ensured that the written dialogue rounds can eventually obtain complete transcribed text for correction, while not blocking subsequent dialogue rounds that are in progress, thus achieving a balance between system fault tolerance and timely response.

[0113] In some embodiments, destroying the asynchronous waiting object corresponding to the current round and unbinding the second completion state from the waiting callback function of the current round includes the following operations: S31, obtain the asynchronous event handle associated with this round, call the underlying operating system interface to set the asynchronous event handle to an invalid state, and reclaim the kernel object resources occupied by the asynchronous event handle.

[0114] An asynchronous event handle is an identifier assigned by the operating system kernel for this timeout wait operation. This identifier is used to reference a specific wait event object between user mode and kernel mode. Its type can be an event object handle, a timer object handle, or a synchronization object handle.

[0115] The underlying operating system interface refers to the system call functions provided by the operating system for managing kernel objects.

[0116] Kernel object resources refer to the storage space, queue structure, and scheduling-related data structures allocated by the operating system kernel to maintain asynchronous event handles.

[0117] In some embodiments, the processor acquiring the asynchronous event handle associated with the current round and calling the underlying operating system interface to invalidate it and reclaim kernel object resources includes: the processor reading the saved event object handle value from the current round context structure, which is created and returned by the operating system when waiting for the operation to start; the processor calling the handle-closing system call provided by the operating system, passing the event object handle to the call to make the kernel mark the handle as invalid; after receiving the handle-closing request, the operating system kernel removes the event object from the handle table of the current process, reclaims the kernel event block memory occupied by the event object, and sets the handle value to an unavailable state.

[0118] S32, traverse the registry of callback functions in the second completion state, and use the round identifier of this round as the index to find the bound waiting callback function.

[0119] The callback function registry is a structured data collection used to store all callback function registration information corresponding to the second completion state. This collection is organized with round identifiers as keys and callback function pointers and related context information as values.

[0120] A turn identifier is a number or string that uniquely identifies a turn in a conversation, and this identifier remains unique within the current session.

[0121] A callback function is a pre-registered function pointer and its execution context that need to be called when the second completion state is reached. This function is used to notify the waiting thread that the transcribing is complete.

[0122] In some embodiments, the processor traverses the callback function registry of the second completion state and searches for bound waiting callback functions using the round identifier of the current round as an index, including: the processor acquiring a mutex lock on the callback function registry of the second completion state to prevent the registry from being modified during traversal; the processor sequentially reading the round identifier field of each registration entry starting from the header of the registry and comparing it with the round identifier of the current round; when the round identifier matches successfully, the processor extracts the callback function pointer and context parameters stored in the entry, and records the position of the entry in the registry as the position to be removed.

[0123] S33, when the waiting callback function is found, the waiting callback function is removed from the callback function registry, and the corresponding entry in the callback function registry is marked as invalid.

[0124] Marking an entry as invalid means that after the removal operation is completed, the original entry is marked with an unusable status value so that the entry will not be mistakenly regarded as valid registration information.

[0125] In some embodiments, when a processor finds a waiting callback function, removing the waiting callback function from the callback function registry and marking the corresponding entry as invalid includes: the processor holding a write lock on the callback function registry, reconnecting the predecessor and successor nodes of the target entry from the linked list of the registry, so that the target entry is removed from the linked list; the processor switching the valid flag bit in the removed entry from a valid value to an invalid value, and clearing the timestamp field of the entry to zero; the processor placing the invalid entry in a delayed recycling queue, waiting for all ongoing traversal operations to be completed before releasing the memory occupied by the entry.

[0126] S34, send a reconciliation confirmation message to the second processing path so that the second processing path will no longer trigger the unbound waiting callback function when the second completion state is reached.

[0127] The callback unbinding confirmation message is a notification message sent by the terminal to the second processing path. This message is used to inform the second processing path that the callback function in a specific round has been unbound.

[0128] In some embodiments, the processor sending a rebinding confirmation message to the second processing path so that the second processing path will no longer trigger the unbound waiting callback function when the second completion state arrives includes: the processor constructing a message after the completion callback function is removed from the registry, the message containing the round identifier of the current round and the unbound flag; the processor sending the message to the second processing path through the established network connection; the second processing path deleting the corresponding entry in its local callback mapping table according to the round identifier after receiving the message, and checking the mapping table when the second completion state is subsequently generated to confirm whether the callback is triggered.

[0129] In this embodiment, by acquiring the asynchronous event handle, invalidating it, and reclaiming kernel object resources, kernel resource leakage caused by residual waiting objects after timeout is avoided. By traversing the callback function registry to find and remove bound callback functions and sending an unbinding confirmation message to the second processing path, it is ensured that the second processing path will not trigger unbound callback functions when the transcribing is completed, thereby avoiding system exceptions caused by wild pointer access or invalid callback execution, and improving the security and reliability of asynchronous waiting resource management.

[0130] In some embodiments, releasing the asynchronous mutex lock used to block subsequent audio acquisition includes the following operations: S41, obtain the current lock state of the asynchronous mutex lock, the current lock state includes the lock holder identifier and the lock waiting queue.

[0131] An asynchronous mutex lock is a mutual exclusion synchronization object used in a multi-threaded environment to protect shared resources from concurrent access conflicts. While the lock object is held by one thread, it will prevent other threads from entering the critical section to perform subsequent audio acquisition operations.

[0132] The current lock state refers to the complete set of attribute information of the asynchronous mutex at a certain moment, which includes at least which thread currently holds the lock and which threads are waiting to acquire the lock.

[0133] The lock holder identifier is a unique identifier or thread identifier assigned by the operating system to the thread currently holding the lock.

[0134] A lock waiting queue is a first-in-first-out ordered queue formed by threads that are blocked by the operating system because they are unable to successfully acquire the asynchronous mutex lock. The thread at the front of the queue will acquire the lock first.

[0135] In some embodiments, the processor acquires the current lock state of the asynchronous mutex lock by: calling a lock state query interface provided by the operating system, which returns a structure containing the lock holder thread identifier and the length of the waiting queue; the processor reads the holder identifier field from the returned structure as the lock holder identifier, and obtains the control block address of the first waiting thread pointed to by the head pointer of the waiting queue to access the lock waiting queue. The processor holds a read lock on the lock while acquiring the lock state to ensure that the lock state is not modified during the state reading process.

[0136] S42, after verifying that the lock holder identifier is consistent with the thread identifier of the currently executing thread, the holder identifier of the asynchronous mutex lock is cleared, and the first waiting thread in the lock waiting queue is removed from the lock waiting queue.

[0137] The currently executing thread refers to the processor thread that is performing a timeout handling operation; this thread is the one that currently holds the asynchronous mutex lock.

[0138] A thread identifier is an identifier used by the operating system to uniquely identify each thread. Clearing means setting the lock holder identifier field to a null or invalid value to indicate that the lock is not currently held by any thread.

[0139] The first waiting thread is the thread that is at the front of the lock waiting queue and has been waiting the longest. This thread will be the first to acquire the lock after it is released.

[0140] In some embodiments, after verifying that the lock holder identifier is consistent with the thread identifier of the currently executing thread, the processor clears the holder identifier of the asynchronous mutex lock and removes the first waiting thread from the lock waiting queue, which includes: the processor reads the thread identifier of the currently executing thread, compares it with the lock holder identifier obtained in step 1, and confirms that the current thread legally holds the lock when the two are equal; the processor atomically writes the holder identifier field in the lock object to a null value, so that subsequent threads querying the lock status believe that the lock is in an unheld state; the processor takes the control block of the first waiting thread from the head of the lock waiting queue and marks the waiting status of the control block as having acquired the lock.

[0141] S43, switch the scheduling state of the first waiting thread from blocked state to ready state, and submit the first waiting thread to the operating system scheduler to resume its execution.

[0142] Scheduling state refers to the running state category of a thread's lifecycle in the operating system kernel, which includes at least two states: blocked state and ready state.

[0143] A blocked state refers to a state in which a thread is unable to continue execution because it is waiting for a resource or event and is removed from the CPU scheduling list by the operating system.

[0144] The ready state refers to the state in which a thread has all the conditions for execution and is waiting for the operating system scheduler to allocate a CPU time slice before it can start running.

[0145] The operating system scheduler is the core component in the operating system kernel responsible for managing thread CPU time allocation and context switching.

[0146] In some embodiments, the processor switching the scheduling state of the first waiting thread from blocked to ready and submitting the first waiting thread to the operating system scheduler to resume execution includes: the processor obtaining the thread control block of the first waiting thread, modifying the thread state field stored in the control block from the value corresponding to the blocked state to the value corresponding to the ready state; the processor inserting the modified thread control block into the tail of the ready queue of the operating system scheduler; and the operating system scheduler selecting the thread from the ready queue and performing a context switch to resume its execution during the next time slice allocation.

[0147] S44, before releasing the asynchronous mutex lock, the atomic operation of switching the state machine to the timeout completion state in this round is merged with the operation of releasing the asynchronous mutex lock into the same critical section transaction, so as to ensure that the state machine switching and the lock release cannot be interrupted or separated in a multi-threaded environment.

[0148] A state machine is a set of states in different completion stages of a round and the rules for transitions between states. It includes at least a waiting state and a timed-out completion state.

[0149] The timeout completion state refers to a specific state value in the state machine that represents the end of the waiting period due to the expiration of a preset bounded time period.

[0150] An atomic operation is an indivisible unit of operation that cannot be interrupted by other threads or interrupt handlers during execution.

[0151] A critical section transaction refers to a sequence of operations that need to be executed continuously and are not interfered with by other threads during their execution. All operations in this sequence either complete completely or fail to complete completely.

[0152] In some embodiments, the atomic operation of switching the state machine to the timeout completion state in the current round and the operation of releasing the asynchronous mutex lock before releasing the asynchronous mutex lock are combined into the same critical section transaction. This includes: the processor disabling thread preemption on the current processor core before entering the critical section, so that the current thread will not be preempted by other threads during the execution of the critical section operation; the processor sequentially executing the state machine state value write operation and the lock object release operation within the critical section, without inserting any instructions that may trigger a context switch between the two operations; and the processor re-enabling thread preemption when exiting the critical section.

[0153] In this embodiment, the lock release operation is only performed after verifying that the lock holder's identifier matches the thread identifier of the currently executing thread, thus avoiding data structure corruption caused by non-holding threads mistakenly releasing the lock. By switching the scheduling state of the first waiting thread from blocked to ready and submitting it to the operating system scheduler, it is ensured that the thread waiting for audio acquisition can resume execution in a timely manner after the lock is released without permanent blocking. By merging the atomic operation of switching the state machine to the timeout completion state with the operation of releasing the asynchronous mutex lock into the same critical section transaction, it is ensured that the state machine switching and lock release cannot be interrupted or separated in a multi-threaded environment, thus avoiding inconsistent intermediate states where other threads read the updated state machine but the lock has not yet been released under race conditions.

[0154] Reference Figure 3 The diagram shown is a schematic diagram of the dual-channel real-time voice session recovery system structure as illustrated in some embodiments of this application.

[0155] like Figure 3 As shown, the system includes audio acquisition and endpoint detection 301, first processing path interface 302, second processing path interface 303, streaming audio playback 304, asynchronous state coordination 305, recovery window 306, connection control 307, and auxiliary control 308.

[0156] The audio acquisition and endpoint detection 301 can acquire pulse-code modulated audio from a preferred external microphone. One feasible parameter combination is: 24000 Hz sampling rate, 16-bit quantization, dual-channel acquisition, reading one frame every 50 ms, and averaging the left and right channel samples to obtain a single-channel audio. These parameters are for illustrative purposes only; sampling rates from 16000 Hz to 48000 Hz and frame lengths from 10 ms to 100 ms can also be used.

[0157] For each single-channel frame, calculate the root mean square amplitude and peak value. Before entering the speaking state, store the most recent frames in the pre-reading queue; when the root mean square amplitude or peak value of at least two consecutive frames reaches the start threshold, determine the start of the speech, and write the pre-reading frame into the speech buffer to avoid swallowing the first syllable. After entering the speaking state, determine the end of the speech when the speech duration reaches the minimum duration and the continuous silence reaches the end duration; force an end when the maximum speech duration is reached.

[0158] The example can save approximately 300 ms of pre-rolled audio, with a minimum speech duration of 600 ms, an end-of-silence duration of 500 ms, and a maximum speech duration of 12000 ms. The threshold can be calibrated based on the device noise floor or calculated dynamically, and is not limited to a fixed value.

[0159] Reference Figure 4 The diagram shown is an exemplary timing diagram illustrating asynchronous coordination timing in some embodiments of this application.

[0160] like Figure 4 As shown, at the start of a dialogue round, asynchronous state coordination 305 clears the first and second completion flags, and clears the user transcription cache and response text cache for this round. The audio block and response text increment of the first processing path can arrive sequentially; the transcription completion event of the second processing path arrives independently.

[0161] Upon receiving a completion response, the first completion flag is set. If the second completion flag is not yet set, the system waits for a preset bounded period using short polling, a condition variable, a semaphore, or an asynchronous wait object. In one embodiment, the bounded period is 1500 ms; its value can be dynamically adjusted based on network statistics.

[0162] When a transcription completion event is received within a bounded time period, the transcribed text and the response text are grouped into the same dialogue round and written to the response window. When a transcription failure event is received, a second completion flag is also set to avoid meaningless waiting. When the wait times out, a round can be formed with an empty transcription or a partially received transcription, and the next recording will begin.

[0163] The transcribed text returned by the second processing path can also be normalized by the auxiliary control module 308, for example, by converting it to lowercase, removing punctuation and spaces, and then compared with a conservative set of pause phrases. When a clear pause intent is detected, data collection stops after the current round ends and waits for manual resumption; this detection does not call the generative model separately, thus avoiding adding an extra session inference link.

[0164] Reference Figure 5 The diagram shown is a schematic representation of the connection reconstruction and history restoration process in some embodiments of this application.

[0165] like Figure 5 As shown, after creating a new connection, connection controller 307 immediately sends a session configuration message in the new connection's open callback. Session configuration can include parameters such as input audio format, output audio format, output timbre, processing model, response modality, commands, and whether to disable remote endpoint detection.

[0166] After the session configuration message enters the send queue of the same connection, connection control 307 sets the shared connection availability flag and notifies the audio worker threads waiting for a connection. This ensures that even if the audio worker thread is immediately awakened, subsequent messages are still sent after the configuration message, preventing audio from being processed by the remote default configuration.

[0167] The audio worker thread checks the pending rewind flag. If restoration is required, the restoration window is converted into at least one historical input message with user and assistant role tags. This historical input message is sent first, followed by the audio append message, audio commit message, and reply creation message for the current round, sent sequentially. The pending rewind flag is cleared after sending is complete.

[0168] When the output tone changes and the remote session does not allow tone switching within a connection that has already output audio, the system actively closes the current connection, retains the recovery window, and establishes a new session through the same reconstruction process. Therefore, there is no need to design another set of recovery logic for tone switching.

[0169] Reference Figure 6 The diagram shown is a functional module schematic of the dual-channel real-time voice conversation recovery system 100 of this application.

[0170] The dual-channel real-time voice conversation recovery system 100 described in this application is installed in an electronic device. Depending on its functions, the dual-channel real-time voice conversation recovery system 100 includes an acquisition module 110, a first processing module 120, a determination module 130, a second processing module 140, a reconnection module 150, and a transmission module 160. These modules can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0171] In this embodiment, the functions of each module / unit are as follows: The acquisition module 110 is used to acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The first processing module 120 is used to submit the valid speech audio to the first processing channel to generate and return the response audio, and at the same time start the second processing channel to return the user's transcribed text. The determining module 130 is used to determine a first completion state indicating the completion of the first processing path and a second completion state indicating the completion of the second processing path. When the first completion state arrives before the second completion state, it waits for the second completion state within a preset bounded time period. The second processing module 140 is used to associate the obtained user-transcribed text and the reply text corresponding to the reply audio as a dialogue turn after the second completion state is reached or the preset bounded time period expires, and write it into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; The reconnection module 150 is used to send a session configuration message to the reconstructed connection when the session connection needs to be reconstructed, and update the connection status to available after confirming that the session configuration message has entered the sending sequence of the reconstructed connection; the message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. The sending module 160 is used to send at least one dialogue turn within the recovery window to the reconstructed connection before submitting new valid speech audio through the reconstructed connection, so as to restore the session context.

[0172] The specific implementation of the dual-channel real-time voice session recovery system in this application is largely the same as the specific implementation of the dual-channel real-time voice session recovery method described above, and will not be repeated here.

[0173] Reference Figure 6 The diagram shown is a schematic representation of a preferred embodiment of the electronic device of this application.

[0174] The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. The memory 113 is used to store computer programs, such as a dual-channel real-time voice session recovery program; In some embodiments, the processor 111 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 111 is typically used to control the overall operation of the electronic device, such as performing data interaction or communication-related control and processing. In this embodiment, the processor 111 is used to run program code stored in the memory 113 or process data.

[0175] The communication interface 112 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The communication interface 112 may also be used to establish a communication connection between the electronic device and other electronic devices.

[0176] The memory 113 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 113 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the memory 113 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. of the electronic device. Of course, the memory 113 may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory 113 is typically used to store the operating system and various computer programs installed on the electronic device, such as the program code of a dual-channel real-time voice session recovery program. In addition, the memory 113 may also be used to temporarily store various types of data that have been output or will be output.

[0177] Figure 6 Only an electronic device having a processor 111, a communication interface 112, a memory 113 and a communication bus 114 is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0178] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the dual-channel real-time voice session recovery method provided in any of the foregoing method embodiments, including: Acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The valid speech audio is submitted to the first processing channel to generate and return the response audio, while the second processing channel is activated to return the user's transcribed text. A first completion state for indicating the completion of the first processing path and a second completion state for indicating the completion of the second processing path are determined. When the first completion state arrives before the second completion state, the system waits for the second completion state within a preset bounded time period. After the second completion state is reached or the preset bounded time period expires, the obtained user-transcribed text and the reply text corresponding to the reply audio are associated as a dialogue turn and written into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; When a session connection needs to be rebuilt, a session configuration message is sent to the rebuilt connection. After confirming that the session configuration message has entered the sending sequence of the rebuilt connection, the connection status is updated to available. The message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. Before submitting new valid speech audio through the reconstructed connection, at least one dialogue turn within the recovery window is sent to the reconstructed connection to restore the session context.

[0179] For a detailed explanation of the above steps, please refer to the above. Figure 1 A flowchart illustrating an embodiment of a dual-channel real-time voice session recovery method.

[0180] Furthermore, this application also proposes a computer-readable storage medium that is both non-volatile and volatile. This computer-readable storage medium is any one or any combination of several of the following: hard disk, multimedia card, SD card, flash memory card, SMC, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area. The program storage area stores a dual-channel real-time voice session recovery program, which, when executed by a processor, performs the following operations: Acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The valid speech audio is submitted to the first processing channel to generate and return the response audio, while the second processing channel is activated to return the user's transcribed text. A first completion state for indicating the completion of the first processing path and a second completion state for indicating the completion of the second processing path are determined. When the first completion state arrives before the second completion state, the system waits for the second completion state within a preset bounded time period. After the second completion state is reached or the preset bounded time period expires, the obtained user-transcribed text and the reply text corresponding to the reply audio are associated as a dialogue turn and written into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; When a session connection needs to be rebuilt, a session configuration message is sent to the rebuilt connection. After confirming that the session configuration message has entered the sending sequence of the rebuilt connection, the connection status is updated to available. The message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. Before submitting new valid speech audio through the reconstructed connection, at least one dialogue turn within the recovery window is sent to the reconstructed connection to restore the session context.

[0181] The specific implementation of the computer-readable storage medium in this application is largely the same as the specific implementation of the dual-channel real-time voice session recovery method described above, and will not be repeated here.

[0182] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware simulation platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0184] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for restoring a dual-channel real-time voice conversation, characterized in that, The method includes: Acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The valid speech audio is submitted to the first processing channel to generate and return the response audio, while the second processing channel is activated to return the user's transcribed text. A first completion state for indicating the completion of the first processing path and a second completion state for indicating the completion of the second processing path are determined. When the first completion state arrives before the second completion state, the system waits for the second completion state within a preset bounded time period. After the second completion state is reached or the preset bounded time period expires, the obtained user-transcribed text and the reply text corresponding to the reply audio are associated as a dialogue turn and written into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; When a session connection needs to be rebuilt, a session configuration message is sent to the rebuilt connection. After confirming that the session configuration message has entered the sending sequence of the rebuilt connection, the connection status is updated to available. The message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. Before submitting new valid speech audio through the reconstructed connection, at least one dialogue turn within the recovery window is sent to the reconstructed connection to restore the session context.

2. The dual-channel real-time voice session recovery method as described in claim 1, characterized in that, The step of determining valid speech audio from the user's speech data through local endpoint detection includes: The start and end points of the voice activity are determined based on the temporal amplitude characteristics of the user's voice data. Extract the audio segment located between the start and end points of the voice activity from the user's voice data as the initial valid segment; Obtain the pre-rolled audio frames cached before the start of the speech activity, and concatenate the pre-rolled audio frames to the beginning of the initial valid segment to form the valid speech audio.

3. The dual-channel real-time voice session recovery method as described in claim 2, characterized in that, The step of acquiring the pre-rolled audio frames cached before the start of the voice activity and concatenating the pre-rolled audio frames to the beginning of the initial valid segment includes: Maintain a pre-read cache queue, which continuously stores the most recent N frames of user voice data, where N is a preset positive integer; When the start point of the voice activity is determined, all N frames are extracted from the pre-rolled cache queue as the pre-rolled audio frames. The pre-rolled audio frames are spliced ​​together in chronological order before the initial valid segment, and the spliced ​​complete audio sequence is output as the valid speech audio to the first processing path and the second processing path.

4. The dual-channel real-time voice session recovery method as described in claim 1, characterized in that, The terminal stops playing the response audio while collecting the valid speech audio, and stops collecting user voice data while playing the response audio, so as to form a half-duplex audio path; The user-transcribed text returned by the second processing path is also used to perform at least one auxiliary control, which includes interface text display, pause intent detection, session resumption content generation, and fault diagnosis; the auxiliary control does not trigger the secondary generation of the response audio.

5. The dual-channel real-time voice session recovery method as described in claim 1, characterized in that, If the second completion state has not been reached within the preset bounded time period, a timeout processing operation is performed, the timeout processing operation including: Destroy the asynchronous waiting object corresponding to this round, and unbind the second completion state from the waiting callback function of this round; Release the timer resources occupied by the preset bounded time period and switch the state machine of this round to the timeout completion state; Associate the partially received user transcribed text or empty placeholders with the reply text as the dialogue round and write it into the recovery window, releasing the asynchronous mutex lock used to block subsequent audio acquisition; When the second completion state arrives after the timeout processing operation is completed, the complete user transcription text carried by the second completion state is re-associated with the reply text of the corresponding round in the recovery window, and the corresponding entry in the recovery window is updated in a copy-on-write manner without blocking the ongoing subsequent dialogue rounds.

6. The dual-channel real-time voice session recovery method as described in claim 5, characterized in that, Destroying the asynchronous waiting object corresponding to this round and unbinding the second completion state from the waiting callback function of this round includes: Obtain the asynchronous event handle associated with this round, call the underlying operating system interface to set the asynchronous event handle to an invalid state, and reclaim the kernel object resources occupied by the asynchronous event handle; Iterate through the registry of callback functions for the second completion state, using the round identifier of this round as the index to find the bound waiting callback functions; When the waiting callback function is found, the waiting callback function is removed from the callback function registry, and the corresponding entry in the callback function registry is marked as invalid; Send a reconciliation confirmation message to the second processing path so that the second processing path will no longer trigger the unbound waiting callback function when the second completion state is reached.

7. The dual-channel real-time voice session recovery method as described in claim 5, characterized in that, The release of the asynchronous mutex lock used to block subsequent audio acquisition includes: Obtain the current lock state of the asynchronous mutex, the current lock state including the lock holder identifier and the lock waiting queue; After verifying that the lock holder identifier is consistent with the thread identifier of the currently executing thread, the holder identifier of the asynchronous mutex lock is cleared, and the first waiting thread in the lock waiting queue is removed from the lock waiting queue. The scheduling state of the first waiting thread is switched from blocked to ready, and the first waiting thread is submitted to the operating system scheduler to resume execution. Before releasing the asynchronous mutex, the atomic operation that switches the state machine to the timeout completion state in the current round is merged with the operation that releases the asynchronous mutex into the same critical section transaction to ensure that the state machine switch and the lock release cannot be interrupted or separated in a multi-threaded environment.

8. A dual-channel real-time voice conversation recovery system, characterized in that, The system includes: The acquisition module is used to acquire user voice data and determine valid speech audio from the user voice data through local endpoint detection; The first processing module is used to submit the valid speech audio to the first processing channel to generate and return the response audio, and at the same time start the second processing channel to return the user's transcribed text; The determination module is used to determine a first completion state indicating the completion of the first processing path and a second completion state indicating the completion of the second processing path. When the first completion state arrives before the second completion state, the module waits for the second completion state within a preset bounded time period. The second processing module is used to associate the obtained user-transcribed text and the reply text corresponding to the reply audio as a dialogue turn after the second completion state is reached or the preset bounded time period expires, and write it into a recovery window with a preset capacity; the reply text is the text description corresponding to the reply audio content returned by the first processing path at the same time as generating the reply audio; the recovery window is used to cache and store the most recent M dialogue turns, where M is a positive integer greater than 1; The reconnection module is used to send a session configuration message to the reconstructed connection when the session connection needs to be reconstructed. After confirming that the session configuration message has entered the sending sequence of the reconstructed connection, the connection status is updated to available. The message content of the session configuration message includes at least one of the following: audio input format, audio output format, speech synthesis timbre, speech synthesis model version, and speech recognition model version. The sending module is used to send at least one dialogue turn within the recovery window to the reconstructed connection before submitting new valid speech audio through the reconstructed connection, in order to restore the session context.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in a memory, it implements the dual-channel real-time voice session recovery method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dual-channel real-time voice session recovery method as described in any one of claims 1 to 7.

Images