Streaming voice service concurrent processing method based on soft switch

Abstract

The invention relates to a streaming voice service concurrent processing method based on soft switching, and the method comprises the steps: building and maintaining a global voice recognition channel with an ASR server, continuously forwarding a user voice stream to the ASR server during a whole session period, and receiving a recognition result; processing the recognition result by adopting a finite-state machine and a large language model to generate a response text for multiple rounds of dialogues; and concurrently executing TTS processing while the global speech recognition channel is continuously operated: segmenting a response text into a plurality of segments according to semantics to perform streaming speech synthesis, and immediately sending a pseudo response after receiving first speech data so as to trigger a soft switch system to pre-load a playing buffer area in advance. Through global reception and concurrent processing, smooth multi-round dialogues which can be interrupted at any time are realized, and the interaction delay is reduced from a second level to a millisecond level by using a text fragmentation and early media triggering technology, so that the user experience and the interaction efficiency of an intelligent call center are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice soft switching communication, in particular to a concurrent processing method for streaming voice service based on soft switching. BACKGROUND

[0002] With the progress of artificial intelligence technology, call centers are evolving from traditional key or simple instruction interaction to intelligent and multi-round dialogue based on automatic speech recognition, text-to-speech and large language models. In this evolution process, achieving natural and smooth real-time communication between man and machine has become the core goal of improving user experience. One of the key prerequisites for achieving this goal is to ensure that the system has a low end-to-end interaction delay in the entire process of receiving user voice, intelligent understanding and generating voice response.

[0003] However, many current intelligent transformations are carried out on the basis of existing soft switching communication architecture. Such architecture has its inherent media processing mode in design. For example, when performing speech synthesis tasks, the system usually needs to wait for the backend service to process all the text content and generate a complete audio file before starting the playback process to the user. Similarly, in the speech recognition link, the system may need to wait for the user to finish speaking a whole sentence or detect a significant pause before returning the recognition result to the business logic layer. This block or round processing method will introduce a certain basic delay in each round of interaction.

[0004] When a large language model is integrated into this interaction link, the delay problem will further accumulate. The time consumption of the three main links of user voice recognition, network transmission and model reasoning of the recognition result to the large language model, and speech synthesis of the model returned text will be added together. Even if the delay of each link is within an acceptable range, the sum may reach several seconds, resulting in a significant pause in the user's conversation. This pause destroys the immediacy and continuity of communication, so that the intelligent service has a higher level of intelligence, but in the interaction experience, it still lags behind and has a gap with the truly smooth conversation. SUMMARY

[0005] In order to effectively reduce the total delay of end-to-end interaction within the existing soft switching system, the present application provides a concurrent processing method for streaming voice service based on soft switching.

[0006] The concurrent processing method for streaming voice service based on soft switching provided by the present application adopts the following technical solution: A concurrent processing method for streaming voice service based on soft switching, comprising the following steps: S1. Establishing and maintaining a persistent connection with the ASR service end as a global voice recognition channel; S2. During the establishment of the communication session, a user voice stream received from the soft switch system is forwarded to the ASR server in real time based on a global voice recognition channel, and an identification result returned by the ASR server is received; S3. The identification result is processed by using a finite state machine, when the confidence of the identification result is not lower than a preset threshold, a large language model is called to perform intent recognition on the identification result, to generate a response text and build a multi-round dialogue context; S4. After the response text is generated, a TTS processing step is performed concurrently while the global voice recognition channel is maintained to continuously run: S41. The response text is segmented into multiple text segments according to semantic units; S42. The multiple text segments are sent to a streaming TTS server one by one through an asynchronous transmission mechanism, and streaming voice data corresponding to each text segment returned by the server is received; S43. After a first byte or a first voice frame of the first streaming voice data is received, a pseudo response is sent to the soft switch system to trigger the soft switch system to pre-load a play buffer; S44. The streaming voice data is transcoded and time-aligned, converted into an audio format adapted to the soft switch system, and sent to the soft switch system for playing after being encapsulated.

[0007] By using the above technical solution, the recognition result of automatic speech recognition is taken as an input of calling a large language model, to generate a response text related to a dialogue context. This processing flow connects the voice input of a user with the text response generated by the system, and constitutes the basis of an intelligent multi-round dialogue.

[0008] To convert the response text into voice that can be heard by a user, the method does not use the conventional mode of waiting for a complete text to be synthesized before playing. The text-to-voice processing step first segments the response text into semantic segments, and sends them to a streaming server one by one for processing. This segmented streaming synthesis mechanism enables the voice synthesis operation to start and return an audio stream as soon as the first text segment is received, avoiding the initial delay caused by waiting for a long text to be processed.

[0009] On this basis, the method sends a pseudo response to the soft switch system as soon as the first byte or frame of the returned voice data is received. The role of the pseudo response is to trigger the soft switch system to pre-load a play buffer in advance. The combination of the text segmentation and pseudo response pre-loading steps respectively overcomes the processing delay of the voice synthesis server and the buffer delay of the soft switch system, shortens the end-to-end time from generating a response text to a user hearing the first sound from several seconds in the traditional mode to milliseconds, thereby realizing low-delay real-time voice interaction.

[0010] Optionally, the persistent connection is a WebSocket connection; the soft switch system and the global voice recognition channel communicate based on a media resource control protocol and a real-time transport protocol; and the way of maintaining the persistent connection in the S1 comprises: periodically sending a heartbeat keep-alive frame through the WebSocket connection.

[0011] By using the above technical solution, the WebSocket connection provides a standardized bidirectional real-time communication channel between the server and the ASR server, and the periodically sent heartbeat keep-alive frame actively maintains the activity of the long connection, preventing accidental disconnection due to network node timeout. This solution provides a specific protocol layer implementation for the continuity and stability of the global voice recognition channel, ensuring the reliability of the communication link in long-time sessions.

[0012] Optionally, the S2 comprises the following sub-steps: S21. receiving an RTP voice stream in a first audio format sent by the soft switch system; S22. performing resampling and protocol encapsulation processing on the RTP voice stream to convert it into a second audio format supported by the ASR server; S23. sending the converted voice stream in the second audio format to the ASR server through the WebSocket connection.

[0013] By using the above technical solution, the forwarding process of the user voice stream is divided into a media processing pipeline. This pipeline receives the RTP voice stream in the first audio format from the soft switch system, performs resampling and protocol encapsulation processing to convert it into the second audio format supported by the ASR server, and finally sends it through the WebSocket connection. This series of sub-steps solves the media format incompatibility problem between the soft switch system and the ASR server. It defines a clear media conversion path from the telecommunication protocol domain to the Internet protocol domain, ensuring that the original user voice data can be correctly adapted and transmitted.

[0014] Optionally, the first audio format is a G.711 a-law or μ-law encoding format; and the second audio format is a 16 kHz sampling rate, 16 bit bit depth pulse code modulation format.

[0015] By adopting the technical scheme, the first audio format is defined as G.711 a-law or mu-law coding commonly used in the telecommunication field, and the second audio format is defined as a high-fidelity 16 kHz, 16 bit pulse code modulation format commonly required by modern speech recognition engines. This format conversion from narrowband to wideband improves the quality of audio data input to the ASR server.

[0016] Optionally, the S42 comprises the following sub-steps: S421. Starting an independent asynchronous listening thread, which is dedicated to receiving all streaming voice data returned by the streaming TTS server; S422. Encapsulating the plurality of text segments into independent WebSocket requests at a preset time interval of no more than 10 milliseconds, and sequentially sending them to the streaming TTS server.

[0017] By adopting the technical scheme, starting an independent asynchronous listening thread dedicated to data reception separates the sending action of text segments from the receiving action of voice data, avoiding blocking caused by waiting for a response. Sending WebSocket requests of text segments one by one at an interval of no more than 10 milliseconds ensures that the data input to the streaming TTS server is smooth and continuous.

[0018] Optionally, the S43 comprises the following sub-steps: S431. Continuously monitoring the data channel connected with the streaming TTS server; S432. When detecting the presence of the first data unit in the data channel, triggering the construction process of the pseudo-response immediately; S433. Sending the constructed pseudo-response to the softswitch system.

[0019] Optionally, the pseudo-response sent to the softswitch system in the S43 is an MRCP SPEAK pseudo-response.

[0020] By adopting the technical scheme, MRCP is a standardized protocol used by the softswitch system to control media resources, and SPEAK is a voice playback instruction defined by it. Using the MRCP SPEAK pseudo-response is to trigger the buffer preload of the softswitch system using its native protocol interface.

[0021] Optionally, the S44 comprises the following sub-steps: S441. Receiving the streaming voice data in the first coding format through the WebSocket protocol and placing it in a ring buffer for caching to achieve time alignment; S442. reading voice data from the ring buffer and performing codec conversion to convert the first codec format into a second codec format suitable for the soft switching system; S443. after encapsulating the voice data in the second codec format into RTP data packets, sending the voice data to the soft switching system.

[0022] By using the above technical solution, the received WebSocket voice data is placed in the ring buffer, solving the problem of uneven RTP data packet timing caused by network jitter, and realizing smooth audio stream output. The subsequent codec conversion and RTP encapsulation steps complete the conversion from Internet media format to telecommunications media format.

[0023] Optionally, the first codec format is wav or MP3; and the second codec format is PCMA or PCMU.

[0024] By using the above technical solution, the first codec format is limited to wav or MP3, etc. which are high-fidelity or high-efficiency formats commonly used in modern streaming media services, and the second codec format is limited to PCMA or PCMU, etc. which are standard telephone encoding formats widely supported by soft switching systems.

[0025] Optionally, when detecting voice data packet loss or transmission stall during receiving or forwarding the streaming voice data, a silence stream packet matching the length of the lost voice data is generated, and the silence stream packet is inserted into the output sequence of the streaming voice data to ensure the continuity of the voice stream received by the soft switching system and avoid playback interruption.

[0026] By using the above technical solution, a network packet loss compensation mechanism is added. When detecting voice data packet loss or transmission stall, the system automatically generates a silence stream packet matching the length of the lost data and inserts it into the output sequence. This ensures the continuity of the RTP voice stream sent to the soft switching system in terms of timing, and avoids the playback interruption or abnormality that may occur in the player due to data loss.

[0027] In summary, the present application includes at least one of the following beneficial technical effects: 1. The response text slice is cut for streaming voice synthesis, and a pseudo response is sent after receiving the first data frame. This combination overcomes the processing delay of the voice synthesis server and the playback buffer delay of the soft switching system, respectively, thereby reducing the end-to-end interaction delay from seconds to milliseconds, realizing real-time voice interaction. 2. Maintaining a global speech recognition channel throughout the communication session and running it concurrently with the text-to-speech announcement process. This concurrent processing mechanism enables the system to receive and recognize the user's speech input synchronously during its own announcement, thus supporting the user to interrupt at any moment, changing the traditional turn-taking mode of one question and one answer, and improving the fluency and naturalness of multi-turn dialogue; 3. Built-in media processing pipeline, which realizes real-time conversion between the protocols and encoding formats of the telecommunication domain, such as RTP and G.711, and the protocols and encoding formats of the Internet domain, such as WebSocket. This method solves the media compatibility problem between heterogeneous systems and guarantees the continuity of audio stream under unstable network through the packet loss compensation mechanism, providing a compatible and reliable technical solution for integrating streaming AI services on the existing softswitch architecture. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A flowchart of the concurrent processing method of the softswitch-based streaming voice service in an embodiment of the present application is shown.

[0029] Figure 2 A softswitch interfacing streaming TTS system architecture diagram in an embodiment of the present application is shown.

[0030] Figure 3 A softswitch interfacing streaming ASR system architecture diagram in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The present application will be further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0032] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this description, some of the diagrams in the present disclosure represent structures and devices in block diagram form in order to avoid obscuring the principles of the disclosed principles. Not all features of a practical implementation are necessarily described or implied. Also, the language used in the present disclosure has been principally selected for readability and instructional purposes and can not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such an inventive subject matter. References in the present disclosure to "one implementation" or "an implementation" means that a particular feature, structure, or characteristic described is included in at least one implementation, and multiple references to "one implementation" or "an implementation" do not necessarily all refer to the same implementation.

[0033] The terms “a,” “an,” and “the” are not intended to refer to singular entities but include the general class of which a specific example can be used for illustration. The use of the terms “a” or “an” can mean any number of including “one,” “one or more,” “at least one,” and “one or more than one.” The term “or” means any one of the alternatives, as well as any combination of the alternatives, including all of the alternatives, unless the alternatives are expressly indicated to be mutually exclusive. The phrase “at least one of’ in combination with a list of items is intended to refer to a single item from the list of items or any combination of items from the list of items. The phrase does not require all of the listed items, unless expressly so limited.

[0034] Currently, call centers serve as an important node for enterprises to interact with customers, and their technical form is undergoing a profound evolution from traditional interactive voice response to intelligent and conversational services. This evolution aims to handle more complex business scenarios through automation, improving service efficiency and user satisfaction. The core driving force behind this transformation is the maturity of voice processing and natural language understanding technologies.

[0035] To achieve smooth and intelligent human-machine voice dialogue, a combined system composed of multiple core technologies is usually required. First, the softswitch system, as the communication infrastructure, is responsible for handling the underlying telephone signaling and call connection, serving as a bridge between users and service systems. Second, automatic speech recognition technology, referred to as ASR in this process, converts the user's voice stream into text format in real-time during the call, serving as input for subsequent processing. Third, large language models, referred to as LLM in this process, serve as the cognitive core of the system, responsible for semantic understanding and intent recognition of the text converted by ASR, and generating corresponding response text according to business logic. Finally, text-to-speech technology, referred to as TTS in this process, is responsible for synthesizing the response text generated by the large language model into natural speech and playing it to the user through the softswitch system. These four work together to form a complete interaction link from hearing to understanding and then responding.

[0036] In complex business consultation scenarios that require multiple rounds of interaction, such as when a user handles a business that requires step-by-step confirmation of information, the dialogue between the user and the system is no longer a simple question-and-answer. The user may immediately generate new questions or modify previously provided information based on the content heard during the system's information broadcast. This behavior of speaking in between the other party's speech, commonly found in interpersonal communication, is also known as interrupting.

[0037] To provide effective services in such a scenario, the intelligent system must have the corresponding technical capabilities. First, to support user interruption, the system needs to continuously monitor and process user voice input while making voice announcements, which requires the system to have the ability to concurrently process both "listen" and "speak" tasks. Second, to ensure smooth dialogue, the system's response time after the user finishes speaking must be as short as possible to avoid long silences that make the user uncomfortable. Therefore, the low-latency real-time response capability and the concurrent processing capability to support user interruption together constitute the core technical requirements in this application scenario.

[0038] When integrating AI services with existing softswitch systems, the conventional method usually follows mature telecommunication protocol standards. The softswitch system sends control instructions to external ASR or TTS media service resources through the Media Resource Control Protocol (MRCP). The actual user voice stream or synthesized voice stream to be played is transmitted through the Real-time Transport Protocol (RTP). This technical architecture is reliable in operation, but its media processing mechanism is mostly turn-based or block-based.

[0039] The essence of this processing mode is serial operation and waiting. In the speech recognition process, the system usually needs to rely on the Voice Activity Detection (VAD) algorithm to determine that the user has stopped speaking before sending the entire speech to be recognized and returning the final result. In the speech synthesis process, the system needs to submit all the text to be announced to the TTS service at once and wait for it to generate complete audio data before starting playback. The silence waiting of ASR and the complete synthesis waiting of TTS constitute the basic source of interaction delay in the traditional integration scheme.

[0040] In the end-to-end interactive process integrated with large language models, the delays in each link will have a cumulative effect. First, from the user stopping speaking to the ASR system returning the recognition result, there is a voice activity detection and recognition processing delay. Then, the recognition result is transmitted to the large language model through the network, the model performs intent recognition and response text generation, introducing network delay and model inference delay. Next, the complete response text generated is sent to the TTS service for synthesis, resulting in a speech synthesis delay. Finally, after the synthesized audio data stream reaches the softswitch system, the system itself needs to perform a certain amount of buffering to ensure smooth playback, adding playback buffering delay. These delay links are connected in series, causing the total time from the user finishing speaking to hearing the system's response to be significantly lengthened, which constitutes a technical bottleneck for improving user experience.

[0041] The embodiments of the present application disclose a concurrent processing method for a softswitch-based streaming voice service, which refers to Figure 1, including steps S1-S4. The core idea of the method is to logically decouple the continuous automatic speech recognition processing from the text-to-speech processing triggered by specific events, thereby forming a concurrent processing model. The global speech recognition channel is a logical channel that is continuously activated during the entire communication session and is dedicated to receiving and processing user speech. The concurrent TTS processing step is a speech synthesis and playback task that is executed in parallel in the background of the continuous operation of the recognition channel. The purpose of separating the "listen" and "speak" processes is to enable the system to continuously monitor and process user speech input while performing its own speech playback, which provides the basis for implementing the user interruption function.

[0042] To better understand the method proposed in the present application, a specific system implementation architecture will be described below with reference to Figure 2 and Figure 3 The system architecture mainly includes three parts: a softswitch system located at the user access side, a streaming AI server as a third-party service, and a streaming media processing middle platform that implements the method of the present application and is deployed as a bridge between the two. The streaming media processing middle platform is the core carrier of the method of the present application.

[0043] The streaming media processing middle platform is internally composed of multiple logically cooperating modules to achieve end-to-end concurrent processing and media conversion. These modules include an ASR media adaptation module, a dialogue management module, a TTS streaming synthesis module, and a TTS media adaptation module.

[0044] The ASR media adaptation module is responsible for processing the uplink media stream sent to the ASR server. Its input is the RTP stream carrying user speech from the softswitch system, and the audio format of the speech stream is usually G.711. The module performs real-time resampling and format conversion on the received audio to obtain the high-fidelity PCM format supported by the ASR server. Its output is the converted PCM audio stream, which is sent out through a persistent WebSocket connection established with the ASR server.

[0045] The dialogue management module is the core of the entire interaction logic. Its input is the text recognition result returned by the ASR server. The module has a built-in finite state machine to manage the flow of multiple rounds of dialogue and performs confidence filtering on the input recognition result. For recognition results that meet the threshold, the module calls a large language model service to perform intent recognition and obtain the response text generated by the model. The output of the module is the response text.

[0046] The TTS streaming synthesis module is responsible for initiating and managing the speech synthesis process. Its input is the response text generated by the dialogue management module. The module first splits the response text into multiple text segments, and then sends these segments one by one to the streaming TTS server through an asynchronous sending mechanism. At the same time, the module is also responsible for starting a separate listening thread, whose output is the sequence of text segments sent to the TTS server, and receiving the streaming speech data returned by the TTS server in real time.

[0047] The TTS media adaptation module is responsible for processing the downlink media stream from the TTS server. Its input is the streaming speech data received by the TTS streaming synthesis module. The module performs timing alignment on the input data stream through a ring buffer, and performs codec conversion to convert it into the PCMA or PCMU format supported by the softswitch system. The final output of the module is the voice stream encapsulated as an RTP data packet, which is sent to the softswitch system for playback. In addition, the module is also responsible for constructing and sending a pseudo response at a specific time.

[0048] Based on the above system, referring to Figure 1 , the steps of the concurrent processing method of the softswitch-based streaming voice service are as follows: S1. Establish and maintain a persistent connection with the ASR server as a global speech recognition channel.

[0049] The ASR server refers to a network server that provides automatic speech recognition technology services, and its instances can include streaming voice recognition services provided by mainstream cloud service providers such as Ali Cloud, Baidu, or Xunfei. The persistent connection described in this step is preferably a WebSocket connection. The WebSocket protocol provides a full-duplex communication channel based on a single TCP connection, which has lower signaling overhead and can exchange real-time data with lower latency compared to traditional HTTP polling or long connection alternatives, and as a web standard, it has good compatibility. The HTTP long connection solution has the disadvantages of large repeated connection overhead or non-true duplex.

[0050] In the implementation of the step, the soft switch system and the streaming media processing platform implementing the method communicate through a media resource control protocol and a real-time transport protocol. The media resource control protocol, MRCP, is a signaling layer protocol used by the soft switch system to issue control instructions to the media processing resource, such as "start recognition" or "play voice". The real-time transport protocol, RTP, is a data transmission layer protocol used to carry actual media data, such as the original voice stream of a user. The purpose of the step is to pre-establish a voice recognition uplink channel that is ready at all times throughout the entire communication session, laying a foundation for subsequent global and uninterrupted voice recognition and fundamentally eliminating the time delay caused by repeatedly establishing an identification session to respond to each user speech in the traditional solution.

[0051] Optionally, the manner of maintaining the persistent connection in the S1 includes: Periodically sending a heartbeat keep-alive frame through the WebSocket connection.

[0052] Maintaining the persistent connection is crucial to ensure that the global voice recognition channel is not interrupted during a silent period without user speech. If it is not maintained, a firewall or network address translation device in the network may trigger a timeout mechanism due to detecting an idle connection, thereby unilaterally closing the connection. Once the connection is accidentally disconnected, subsequent user speech cannot be processed in real time, and the system needs to re-authenticate and establish a connection, which will introduce significant delay and may cause the recognition task to fail.

[0053] By periodically sending a heartbeat keep-alive frame, such as a PING frame through the WebSocket connection every 30 seconds, the network intermediate device can be convinced that the connection is still active, thereby resetting the timeout timer and avoiding the connection from being recycled. In a specific implementation, the method can add a session maintenance flag bit in the initial MRCP request, combined with the heartbeat keep-alive mechanism of the WebSocket, to achieve a global voice state of up to 24 hours after completing a single service authentication, ensuring the long-term availability of the global voice recognition channel.

[0054] S2. During the communication session, based on the global voice recognition channel, the user voice stream received from the soft switch system is forwarded to the ASR server in real time, and the recognition result returned by the ASR server is received.

[0055] This step describes the main task after the global voice recognition channel established in S1 enters the actual working state. This step defines a continuous bidirectional data flow: one is the uplink voice data flow from the soft switch system to the ASR server, and the other is the downlink text recognition result flow returned from the ASR server. This step is the execution phase of realizing global and uninterrupted voice monitoring of the user, which processes all voice inputs of the user in the call process in real time through the pre-established channel.

[0056] Specifically, S2 includes the following sub-steps S21-S23.

[0057] S21. Receive the RTP voice stream in the first audio format sent by the soft switch system.

[0058] S22. Perform resampling and protocol encapsulation processing on the RTP voice stream to convert it into the second audio format supported by the ASR server.

[0059] S23. Send the converted voice stream in the second audio format to the ASR server through the WebSocket connection.

[0060] In this embodiment, the first audio format is usually the standard audio coding format in the telecommunication network environment, examples of which include G.711 a-law or μ-law coding format, i.e., PCMA or PCMU. Such formats have the characteristics of low code rate, usually 8 kHz sampling rate, and can meet the basic call demand. The second audio format is a high-fidelity, wideband audio format universally supported by modern ASR servers to ensure recognition accuracy, an example of which is the 16 kHz sampling rate, 16 bit depth pulse code modulation format, i.e., PCM.

[0061] The advantage of this media conversion process is to improve the accuracy of voice recognition. For example, when the soft switch system sends a voice stream in the 8 kHz PCMU coding through the RTP protocol, the ASR media adaptation module of the present method will decode and upsample the voice data to convert it into a 16 kHz 16 bit PCM raw stream. Compared with the original narrowband audio, the converted wideband audio retains more voice detail information such as high-frequency harmonics, providing a higher quality input source to the ASR server, thereby effectively reducing the recognition error rate.

[0062] For example, in a specific call, the user's speech is encapsulated by the softswitch system into a series of RTP data packets and sent. The ASR media adaptation module of the method continuously receives these data packets, extracts the PCMA or PCMU payload, performs resampling from 8 kHz to 16 kHz, and adapts the resulting data stream to the format required by the streaming ASR server, such as encapsulating it into a WAV container format or sending it directly as a PCM naked stream, in chunks through a WebSocket connection to the server.

[0063] S3. Processing the recognition result using a finite state machine, when the confidence of the recognition result is not lower than a preset threshold, calling a large language model to perform intent recognition on the recognition result to generate a response text and build a multi-turn dialogue context.

[0064] A finite state machine is a computational model whose core idea is that the system is in a well-defined state at any given time point, and transitions from one state to another according to external inputs or events. In this method, the finite state machine is used as a controller of the dialogue flow, where each state represents a specific stage of the dialogue, such as a state waiting for the user's initial question or a state waiting for the user to provide specific information.

[0065] When the recognition result returned by S2 arrives as an input event, the finite state machine decides the subsequent action and the new state to be transitioned to according to its current state and the content of the input event, to ensure that the entire multi-turn dialogue can follow the preset business logic and proceed in an orderly manner. For example, upon receiving a recognition result with a confidence not lower than a preset threshold, the finite state machine will trigger the action of calling a large language model.

[0066] After receiving the recognition result text, the large language model performs intent recognition on it. The process of intent recognition can include classifying the text to determine the user's core purpose, as well as extracting key information entities in the text, also known as slot filling. For example, for the text "query my savings account balance", the large language model can identify the core intent of "querying the balance" and extract the key information of "account type" as "savings".

[0067] The purpose of building a multi-turn dialogue context is to enable the system to remember and understand the historical information of the dialogue, so as to correctly handle natural language phenomena such as reference and omission, and achieve coherent and logical communication. The global speech recognition channel established by S1 and S2 provides a foundation for this, as it ensures that all the user's utterances in the entire conversation can be continuously captured, providing an uninterrupted input source for building a complete dialogue history.

[0068] The construction of context can be achieved by the dialogue management module maintaining a data structure containing historical interaction records during the session. The structure stores the user's previous recognition results and the system's previous response texts in order. When processing new user input, the system can provide part or all of the historical records together with the current input to the large language model, enabling it to generate more accurate and relevant response texts based on a full understanding of the cause and effect.

[0069] In the example above, the ASR server processed the PCM raw stream and returned a recognition result with a confidence of 0.95, the content of which is "check the weather today". The finite state machine in the dialogue management module receives this result because its confidence is higher than the preset threshold of 0.8, and sends the text to the large language model. The large language model identifies the user's intent as querying the weather and generates the response text "OK, querying for you. Today's local weather is sunny, temperature 25 degrees Celsius, and gentle breeze." At the same time, the dialogue management module stores the user's question and the system's answer in the context of this session. At this time, S3 is completed, and the generated response text will be processed by S4.

[0070] S4. After generating the response text, while maintaining the global speech recognition channel running continuously, concurrently execute the TTS processing step.

[0071] This step is a concurrent implementation, that is, the start and execution of the text-to-speech processing process will not interrupt or pause the global speech recognition channel established and running in S1 and S2. Since the recognition channel is continuously running, the system can synchronously receive and process the user's speech input at any time during the process of playing the response text through TTS.

[0072] Specifically, the TTS processing step of S4 includes S41-S44.

[0073] S41. The response text is divided into multiple text segments according to semantic units.

[0074] S42. The multiple text segments are sent one by one to the streaming TTS server through an asynchronous transmission mechanism, and the corresponding streaming voice data returned by the server is received.

[0075] Here, the response text is divided according to semantic units by using the syntax analysis algorithm in natural language processing technology. This algorithm can identify sentence boundaries, clause structures or natural pause points in the text, and based on this, a long response text is divided into multiple logically coherent and shorter text segments, for example, each segment has an average length of no more than 10 characters.

[0076] The asynchronous transmission mechanism is used to realize non-blocking communication. After sending a request for a text segment through the WebSocket protocol, the sending end does not need to wait for the response from the server, and can continue to send the next text segment, thereby forming a continuous request sending stream. The streaming TTS server is a network service that can receive a text data stream and synchronously generate an audio data stream. It can start synthesis and transmit audio without receiving all the text. Sending text segments one by one is to cooperate with and utilize the capability of the streaming TTS server to start the speech synthesis process as soon as possible. Streaming voice data refers to a series of audio data units generated in real time by the server and continuously transmitted. The format can be pulse code modulation (PCM) raw data.

[0077] In the example above, the response text generated by S3 is "OK, querying for you. Today, the local weather is sunny, the temperature is 25 degrees Celsius, and there is a gentle breeze." In S41, the natural language processing algorithm in the TTS streaming synthesis module divides the text into four text segments: "OK, querying for you.", "Today, the local weather is sunny,", "the temperature is 25 degrees Celsius,", and "and there is a gentle breeze." In S42, the module immediately sends the first segment "OK, querying for you." to the streaming TTS server through an asynchronous WebSocket request, and then prepares and sends the second segment, and so on. Almost at the same time as the first segment is sent, an asynchronous thread dedicated to receiving starts to receive the streaming voice data corresponding to the first segment returned in real time by the TTS server.

[0078] Specifically, S42 includes the following sub-steps S421-S422.

[0079] S421. Start an independent asynchronous listening thread dedicated to receiving all the streaming voice data returned by the streaming TTS server.

[0080] S422. Encapsulate the multiple text segments as independent WebSocket requests at a preset time interval of no more than 10 milliseconds, and send them to the streaming TTS server in turn.

[0081] The listening thread responsible for receiving voice data is designed as an independent asynchronous thread to realize parallel processing of sending and receiving, and to avoid process blocking. If the "send-wait-receive" mode is used in the same thread, the thread must wait for the corresponding voice data to return after sending the first text segment, before it can continue to send the second text segment. By starting an independent thread dedicated to receiving through S421, the sending task in the main process does not need to wait for the response, and can continuously send all the text segments at a short preset time interval, thereby maximizing the efficiency of data transmission.

[0082] Continuing the example above, after the TTS stream synthesis module has sliced the response text, it first starts an asynchronous listening thread according to S421, which begins to wait for receiving audio data on the WebSocket connection. Then, the module encapsulates the first text segment "OK, querying for you." into a WebSocket request and sends it according to S422, waits for an interval of, for example, 10 milliseconds, and then continues to send the second segment "It is sunny today in your local area.", and so on until all segments are sent. During this sending process, the independent listening thread may have already started to receive the audio stream corresponding to the first segment, and the two processes run in parallel without contradiction, ensuring full-duplex utilization of the communication link.

[0083] S43. Immediately after receiving the first byte or the first voice frame of the first stream voice data, a pseudo response is sent to the soft switch system to trigger the soft switch system to pre-load the play buffer.

[0084] In a conventional process, the soft switch system needs to receive a sufficient amount of audio data and confirm that the media stream is valid before allocating resources and loading the play buffer, which itself introduces a certain amount of delay. This step sends a "pseudo response" to "inform" the soft switch system that the media stream is about to arrive, prompting it to pre-complete the preparation work of the buffer.

[0085] Continuing the example described above, the asynchronous listening thread started by S421 is listening to the WebSocket connection. When the stream TTS service side completes the synthesis of the first text segment "OK, querying for you." and begins to return the audio stream, the listening thread immediately triggers the execution of S43 upon receiving the first byte or the first data frame of the audio stream. The TTS media adaptation module will immediately construct a pseudo response and send it to the soft switch system. After the soft switch system receives the response, it will mistakenly believe that a complete audio stream is ready, thereby starting its internal play buffer loading process in advance. In this way, the method overlaps the soft switch's own buffering delay with the first packet data transmission delay of the TTS service in time. Tests show that this can reduce the end-to-end delay from 5 seconds to 280 milliseconds.

[0086] Specifically, the pseudo response sent to the soft switch system in S43 is an MRCP SPEAK pseudo response.

[0087] The specific form of the pseudo response is limited to the MRCP SPEAK pseudo response, in order to utilize the native media control protocol of the softswitch system to achieve maximum compatibility. MRCP is the standard signaling protocol for the softswitch system to interact with media resources, and the SPEAK instruction is used to initiate speech synthesis and announcement. The pseudo response constructed by the method complies with the MRCP protocol specification in terms of format, and for the softswitch system, it is a legal and expected event. By sending this standard protocol event, the method can reliably trigger the softswitch system to perform its standard media playback preparation process, i.e. buffer preloading, without the need for any customized modification to the softswitch system itself.

[0088] Specifically, the S43 includes the following sub-steps S431-S433.

[0089] S431. Continuously monitor the data channel connected to the streaming TTS server.

[0090] S432. Upon detecting the occurrence of the first data unit in the data channel, immediately trigger the construction process of the pseudo response.

[0091] S433. Send the constructed pseudo response to the softswitch system.

[0092] S431 specifies that the system is in a continuous ready state, and the asynchronous listening thread started in S421 is responsible for uninterrupted listening on the data channel. Step S432 defines the trigger condition, i.e. the occurrence of the first data unit on the data channel, such as the first byte or the first frame of audio data, as the precise signal to trigger the pseudo response construction process.

[0093] S44. Encode and decode the streaming voice data and perform timing alignment, convert it to an audio format suitable for the softswitch system, and send it to the softswitch system for playback after encapsulation.

[0094] This step constitutes the final link in the downlink media stream processing, and its core task is to convert the streaming voice data transmitted in the Internet Protocol domain into a standard media stream in the telecommunications protocol domain.

[0095] Specifically, the S44 includes the following sub-steps S441-S443.

[0096] S441. Receive the streaming voice data in a first codec format through the WebSocket protocol and store it in a ring buffer for buffering to achieve timing alignment.

[0097] S442. Read the voice data from the ring buffer and perform codec conversion to convert the first codec format to a second codec format suitable for the softswitch system.

[0098] S443. encapsulate the voice data in the second codec format into RTP packets and send to the softswitch system.

[0099] Optionally, the first codec format is wav or MP3; and the second codec format is PCMA or PCMU.

[0100] For example, in a specific implementation, the TTS media adaptation module receives an audio stream from a streaming TTS server via WebSocket protocol, which may use wav as its first codec format for high quality. The module stores the received wav data frames into a ring buffer for smoothing processing. Then, the next stage of the processing pipeline reads the wav data from the buffer, performs codec conversion, decodes and re-encodes it into PCMU supported by the softswitch system as its second codec format. Finally, the generated PCMU data is encapsulated into standard RTP packets and sent to the softswitch system. This scheme has the advantage that it allows the system to utilize modern efficient encoders such as wav when interacting with AI services to obtain more natural synthesized speech, while converting to the most widely compatible PCMA or PCMU format when interacting with telecommunication devices, thus balancing speech quality and system integration.

[0101] Optionally, when detecting packet loss or transmission stall of the streaming voice data during receiving or forwarding, a silence stream packet matching the duration of the lost voice data is generated and inserted into the output sequence of the streaming voice data to ensure the continuity of the voice stream received by the softswitch system and avoid playback interruption.

[0102] This step provides a network packet loss compensation mechanism to improve the robustness of operation in unstable network environments. When receiving streaming voice data from a TTS server via the Internet, packet loss may occur due to network congestion, etc. If the audio stream with data gaps is directly forwarded to the softswitch system, it may cause the user to hear stuttering, noise, or even cause the player to interrupt due to discontinuous data stream. This method generates a silence data packet matching the duration of the lost data and inserts it into the output sequence when detecting data packet loss, ensuring that the RTP stream sent to the softswitch system is continuous in timing and sequence number. For the softswitch system, it receives a complete media stream, thus avoiding playback anomalies and ensuring the smoothness of the user listening experience.

[0103] In addition, to further improve the adaptability of the system in a complex network environment, the application also provides a dynamic code rate adaptation mechanism. Specifically, the TTS and ASR server monitors the network delay, packet loss rate and bandwidth fluctuation of the data channel during voice data interaction through the WebSocket connection. When the average delay of consecutive frames is detected to exceed the preset threshold, or the packet loss rate exceeds the preset proportion, the system will automatically reduce the audio code rate, for example, from 128kbps to 64kbps or 32kbps, to reduce the data volume and ensure real-time performance; when the network condition returns to normal, gradually restore to the original code rate. Through the above dynamic adjustment, the delay and stuttering risk in a weak network environment can be effectively reduced, and the call quality can be stabilized.

[0104] The application also includes a dual protocol stack middleware for realizing the bidirectional conversion of the SIP / MRCPv2 protocol of the soft switching side and the WebSocket protocol of the TTS / ASR service side. The middleware includes a session management module, a media stream adaptation module and a protocol conversion module, wherein the session management module is used for maintaining the call session state and the global speech recognition channel; the media stream adaptation module is used for completing the resampling, format conversion and buffer management of the audio stream; and the protocol conversion module is responsible for translating the MRCPSPEAK, RECOGNIZE and other control instructions into WebSocket messages, and encapsulating the returned recognition results or voice stream into MRCP responses for the soft switching system to recognize and play. Through the middleware, the differences between the underlying protocols can be shielded, the system integration complexity can be reduced, and seamless cross-protocol connection can be realized.

[0105] In addition, to support the continuity of multi-round dialogue, the application sets a context state management module for caching historical recognition results and response texts. When a new recognition result arrives, the module will splice it with the session history and submit it as a complete input to the large language model to generate a response related to the context. This design ensures that the context is not lost during voice interaction, allowing users to have a natural multi-round dialogue without repeating the previous information.

[0106] To describe the technical solutions of the application as a whole, the following will take a complete user interaction process as an example to describe the technical features described in the foregoing steps in series, and explain the principles and creativity.

[0107] In one specific application scenario, when a user accesses a call center through a phone call, the call is first established by a softswitch system. At this time, the streaming media processing middleware implementing the method immediately performs S1, proactively establishing a persistent WebSocket connection with a preset ASR server, such as an Ali Cloud voice recognition service. Once the connection is established, it remains active as a global voice recognition channel throughout the entire call, ready to receive user voice at all times, which fundamentally avoids the delay caused by the need to reestablish an identification session each time the user speaks in the traditional solution.

[0108] Subsequently, the user issues a voice instruction to the system, such as "check the weather today". The softswitch system encapsulates the user's voice in an RTP data packet in the real-time 8kHz PCMU encoding format it collects and sends it to the streaming media processing middleware. The ASR media adaptation module in the middleware performs S2, continuously receiving the RTP voice stream. The module decodes and up-samples the received PCMU data, converting it to the 16kHz 16bit PCM format required by the ASR server to ensure high recognition rates, and then real-time forwards the high-quality PCM audio stream to the ASR server through the established global voice recognition channel, i.e., the WebSocket connection. This media adaptation process ensures the accuracy of voice recognition. After the ASR server is processed, it returns the recognition result in text format "check the weather today".

[0109] After receiving the recognition result, the dialogue management module of the middleware begins to perform S3. The module first confirms that the confidence of the result is higher than a preset threshold, then calls the large language model service and inputs the recognized text. After the large language model performs intent recognition, it generates the corresponding response text: "OK, querying for you. Today's local weather is sunny, temperature 25 degrees Celsius, and a gentle breeze." At the same time, the module records the user's question and the system's answer together, constructing the context of this session and providing support for subsequent possible multi-turn dialogues.

[0110] After generating the response text, the method immediately enters S4 and performs the TTS processing procedure concurrently. It is important to emphasize that the execution of this step does not interrupt the global voice recognition channel maintained by S1 and S2, which continues to listen to the user's voice input, thus ensuring that the user can interrupt the system's broadcast at any time.

[0111] After the TTS processing flow is started, the TTS streaming synthesis module first divides the long response text into four semantically coherent text segments using natural language processing algorithms. The first text segment "OK, querying for you." is then sent to the streaming TTS server through an asynchronous WebSocket request, and a separate thread is started to listen for the returned data. The principle of this kind of segmented streaming is that it divides a long text synthesis task into multiple short tasks, so that the TTS server can start working immediately and return the audio of the first segment.

[0112] When the listening thread receives the first byte of audio data from the TTS server, the TTS media adaptation module quickly constructs an MRCP SPEAK pseudo response and sends it to the softswitch system. It uses the native protocol of the softswitch system to insert an early start signal into the process of waiting for the complete audio, cleverly eliminating the playback buffer delay of the softswitch itself.

[0113] Finally, the listening thread continuously receives subsequent streaming voice data, and the TTS media adaptation module converts the input audio stream in the wav format to the PCMA format through its internal media processing pipeline, and then encapsulates it into RTP data packets and sends it to the softswitch system that has completed preloading. Subsequent text segments will also be processed through the complete process of "segmentation - sending - receiving - preloading - conversion - playing" in turn, and the user will finally hear a coherent voice that is responsive and smooth.

[0114] It should be understood that the size of the sequence number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0116] The above examples are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing examples, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for concurrent processing of softswitch-based streaming voice services, characterized in that, The method comprises the following steps: S1. Establishing and maintaining a persistent connection as a global speech recognition channel with an ASR server; S2. During the establishment of a communication session, based on the global speech recognition channel, forwarding a user speech stream received from a soft switch system to the ASR server in real time, and receiving a recognition result returned by the ASR server; S3. Processing the recognition result by using a finite state machine, when the confidence of the recognition result is not lower than a preset threshold, calling a large language model to perform intent recognition on the recognition result, to generate a response text and construct a multi-round dialogue context; S4. After the response text is generated, while maintaining the global speech recognition channel to continue running, performing a TTS processing step concurrently: S41. Cutting the response text into multiple text segments according to semantic units; S42. Sending the multiple text segments to a streaming TTS server one by one through an asynchronous transmission mechanism, and receiving streaming voice data corresponding to each text segment returned by the server; S43. After receiving the first byte or the first voice frame of the first streaming voice data, immediately sending a pseudo response to the soft switch system to trigger the soft switch system to pre-load a play buffer; S44. Performing codec conversion and timing alignment on the streaming voice data, converting the streaming voice data into an audio format suitable for the soft switch system, and then sending the converted streaming voice data to the soft switch system for playing. 2.The softswitch-based concurrent processing method of streaming voice service according to claim 1, characterized in that, The persistent connection is a WebSocket connection; the soft switch system and the global speech recognition channel communicate based on a media resource control protocol and a real-time transport protocol; The way of maintaining the persistent connection in S1 comprises: Periodically sending a heartbeat keep-alive frame through the WebSocket connection. 3.The softswitch-based concurrent processing method of streaming voice services according to claim 1, wherein, S2 comprises the following sub-steps: S21. Receiving an RTP voice stream sent by the soft switch system in a first audio format; S22. Performing resampling and protocol encapsulation processing on the RTP voice stream, to convert it into a second audio format supported by the ASR server; S23. Sending the converted voice stream in the second audio format to the ASR server through the WebSocket connection. 4.The softswitch-based concurrent processing method of streaming voice service according to claim 3, wherein, The first audio format is a G.711 a-law or μ-law encoding format; the second audio format is a 16 kHz sampling rate, 16 bit bit depth pulse code modulation format. 5.The softswitch-based concurrent processing method of streaming voice service according to claim 1, wherein, S42 comprises the following sub-steps: S421. Starting an independent asynchronous listening thread, which is dedicated to receiving all streaming voice data returned by the streaming TTS server; S422. At a preset time interval of no more than 10 milliseconds, encapsulating the multiple text segments into independent WebSocket requests, and sending them to the streaming TTS server one by one. 6.The softswitch-based concurrent processing method of streaming voice services according to claim 1, wherein, S43 comprises the following sub-steps: S431. Continuously listening to a data channel connected with the streaming TTS server; S432. When detecting the appearance of the first data unit in the data channel, immediately triggering a construction process of a pseudo response; S433. The pseudo response is sent to the soft switch system. 7.The softswitch-based concurrent processing method of streaming voice services according to claim 1, wherein, The pseudo response sent to the soft switch system in S43 is an MRCP SPEAK pseudo response. 8.The softswitch-based concurrent processing method of streaming voice service according to claim 1, wherein, S44 includes the following sub-steps: S441. The streaming voice data in the first codec format is received through the WebSocket protocol and cached in a ring buffer for timing alignment; S442. The voice data is read from the ring buffer and converted to the second codec format suitable for the soft switch system; S443. The voice data in the second codec format is encapsulated into an RTP data packet and sent to the soft switch system. 9.The softswitch-based concurrent processing method of streaming voice services according to claim 8, wherein, The first codec format is wav or MP3; and the second codec format is PCMA or PCMU. 10.The softswitch-based concurrent processing method of streaming voice service according to claim 1, wherein, When packet loss or transmission stall of the streaming voice data is detected during the receiving or forwarding process, a silence stream packet matching the length of the lost voice data is generated and inserted into the output sequence of the streaming voice data to ensure the continuity of the voice stream received by the soft switch system and avoid playback interruption.

Images