Voice conversation method and system, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a voice conversation method and system, electronic equipment, a storage medium and a program product. One method is implemented as follows: a voice dialogue stream (which can be an inquiry voice stream input by a user for a target commodity) of a user is subjected to streaming response, a plurality of dialogue text segments are generated in a segmented manner in the streaming response process, and the dialogue text segments are output in real time after being generated and stored in a first cache queue; and when it is detected that the reply text fragment is stored in the first cache queue, the stored reply text fragment is converted into a reply voice fragment in real time and played. The streaming response may be performed by an agent. Visibly, according to the response of the voice dialogue stream of the user, the dialogue is played while being generated, the waiting time from text to voice synthesis can be greatly shortened, and therefore the response time can be shortened.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a voice dialogue method, system, electronic device, storage medium, and program product. Background Art

[0002] Currently, intelligent voice dialogue systems have been widely used in various fields, including customer service and marketing. Traditionally, conversational service logic is orchestrated through methods such as the SPD canvas (a visual interface or tool) to drive conversation execution, resulting in relatively human-like intelligent voice dialogue systems. However, due to the simple classification models used as intermediate models and the fixed SPD canvas process rules, these traditional intelligent voice dialogue systems lack the flexibility to address complex and ever-changing user intent, making them difficult to meet the growing demand for intelligent services. In recent years, machine models such as language models have demonstrated significant advantages in natural language understanding and generation tasks. However, the introduction of language models into voice dialogue systems also presents new challenges, particularly their high response time (RT), which severely impacts the real-time performance and user experience of voice dialogue systems. Summary of the Invention

[0003] Multiple embodiments of the present application provide a voice conversation method, system, electronic device, storage medium, and program product to improve conversation response time.

[0004] In a first embodiment, the present application provides a voice conversation method. The method includes:

[0005] Receive the user's voice conversation stream;

[0006] Performing a streaming response to the voice dialogue stream, wherein during the streaming response process, reply text segments are generated in segments, and after the reply text segments are generated, they are output in real time to be stored in a first cache queue;

[0007] When detecting that the reply text segment is stored in the first cache queue, converting the stored reply text segment into a reply voice segment in real time;

[0008] Play the reply voice clip

[0009] In a second embodiment, the present application provides a voice conversation method. The method includes:

[0010] Determine the response generation method based on the conversation text corresponding to the user's voice conversation flow;

[0011] While determining the conversation generation method, calling a plurality of first preset models, and respectively using the plurality of first preset models to perform streaming responses for the conversation text;

[0012] According to the reply generation method, converting the reply text segments output by one of the first preset models in the process of executing the streaming response into reply voice segments;

[0013] Play the reply voice clip.

[0014] In a third embodiment, the present application provides a voice conversation method. The method includes:

[0015] Determine the response generation method based on the conversation text corresponding to the user's voice conversation flow;

[0016] According to the conversation generation method, one of the plurality of first preset models is called to perform a stream response for the conversation text.

[0017] Play the response voice segment based on the segmented response text segment output during the streaming response process.

[0018] In a fourth embodiment, the present application provides a voice conversation method. The method includes:

[0019] Receive the user's voice conversation stream;

[0020] Based on the dialogue text corresponding to the voice dialogue flow, triggering a third preset model to perform a streaming response;

[0021] Converting the reply text segments output in segments by the third preset model when executing the streaming response into reply voice segments;

[0022] Playing the reply voice clip;

[0023] The third preset model is deployed on a node in the rule engine, and the rule engine controls the human-computer voice dialogue according to the set dialogue service rule logic.

[0024] In a fifth embodiment, the present application provides a voice conversation method. The method includes:

[0025] Using the second preset model, monitoring the user's voice conversation flow;

[0026] When the voice dialogue flow is detected to have reached the end of a sentence, the agent is triggered to perform a streaming response based on the dialogue text corresponding to the voice dialogue flow;

[0027] Playing a reply voice segment based on the reply text segment output by the agent in the process of executing the streaming response;

[0028] The second preset model is a trained lightweight model.

[0029] In a sixth embodiment, the present application provides a voice conversation method. The method includes:

[0030] Obtain the user's voice query stream for the target product;

[0031] Based on the text information corresponding to the query voice stream, trigger the intelligent agent to perform a streaming response;

[0032] Playing a reply voice segment based on the reply text segment output by the agent in the process of executing the streaming response, so as to respond to the inquiry voice stream;

[0033] Among them, the intelligent agent performs streaming response including: while determining the reply generation method based on the text information, using multiple first preset models to perform streaming response for the text information respectively; the reply text segment is output in segments by one of the first preset models during the execution of the streaming response.

[0034] In a seventh embodiment, the present application provides a voice dialogue system. The system includes:

[0035] The client is used to collect the user's voice dialogue flow during the human-computer voice conversation and send the voice dialogue flow to the server;

[0036] The server is used to execute the steps in the various method embodiments provided in this application.

[0037] In an eighth embodiment, the present application provides an electronic device. The electronic device includes: a memory and a processor, wherein the memory is configured to store executable instructions; and the processor is coupled to the memory and configured to execute the executable instructions stored in the memory to implement the steps of each method embodiment provided above.

[0038] In a ninth embodiment, the present application provides a computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by a computer, can implement the steps of each method embodiment provided in the present application.

[0039] In a tenth embodiment, the present application provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the above-mentioned method embodiments provided in the present application.

[0040] The solutions provided by the various embodiments of the present application adopt a streaming response to the user's voice dialogue stream (which can be a voice flow of inquiry input by the user for a target product). During the streaming response process, multiple reply text segments are generated in segments, and after the reply text segments are generated, they are output in real time to be stored in the first cache queue. When the reply text segments are detected in the first cache queue, the stored reply text segments are converted into reply voice segments and played in real time. It can be seen that the present application achieves the simultaneous generation and playback of replies in response to the user's voice dialogue stream, which can greatly shorten the waiting time of text-to-speech synthesis, thereby reducing the response time. Specifically, the implementation scheme for streaming response to the voice dialogue stream includes the following:

[0041] One implementation involves determining a response generation method based on the conversation text corresponding to the voice conversation flow while simultaneously invoking multiple first preset models and utilizing each of these models to perform streaming responses to the conversation text. Subsequently, based on the response generation method, the segmented response text segments output by one of the first preset models during the streaming response are converted into and played as response voice segments. By implementing this multi-tasking approach (including determining the response generation method and performing streaming responses using multiple first preset models) in parallel, the present application reduces the response time (RT) of a conversation round to the time taken by one of the multiple parallelized tasks, further effectively reducing overall response time.

[0042] Another implementation involves determining a response generation method based on the text corresponding to the user's voice conversation flow. Based on this response generation method, one of multiple first preset models is invoked to perform a streaming response to the conversation text. The response audio segments are then played based on the segmented response text segments output during the streaming response process. This multi-task sequential execution approach can significantly reduce the overall response time of a conversation.

[0043] Another implementation involves triggering a third preset model deployed on a node in the rule engine to execute a streaming response based on the user's voice conversation flow. The text segments output by the third preset model during the streaming response are then converted into and played as voice responses. The rule engine described above is used to control human-computer voice conversations according to the logic of the established conversation service rules. By deploying the third preset model on a node in the rule engine, this application maintains the ability to strictly control the conversation service process through rules while also leveraging the model's capabilities to respond to conversations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 A schematic diagram of the structure of a customer service system provided by an exemplary embodiment of the present application;

[0046] Figure 2 A flowchart of a voice conversation method provided by an exemplary embodiment of the present application;

[0047] Figure 3 A schematic structural diagram of a voice duplex module provided in an exemplary embodiment of the present application;

[0048] Figure 4a A schematic diagram of the structure of an intelligent dialogue module provided by an exemplary embodiment of the present application;

[0049] Figure 4b A schematic structural diagram of an intelligent dialogue module provided by another exemplary embodiment of the present application;

[0050] Figure 4c A schematic structural diagram of a voice duplex module and an intelligent dialogue module provided in another exemplary embodiment of the present application;

[0051] Figure 5 A time series data diagram of a conversation response time provided by an exemplary embodiment of the present application;

[0052] Figure 6 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] With the continuous advancement of technologies such as artificial intelligence, speech recognition, and natural language processing, intelligent voice dialogue systems have been widely used in various fields, including customer service and marketing. Traditional intelligent voice dialogue systems typically rely on rule-driven or shallow classification models for intent recognition and dialogue management, with their core logic controlled by pre-set flow charts or state machines. For example, on some service platforms, such as e-commerce platforms, relatively human-like intelligent voice dialogue systems have been established based on automatic speech recognition (ASR) and text-to-speech synthesis (TTS) technologies, using the SPD canvas for service logic orchestration to drive dialogue execution. These systems are used in scenarios such as consumer online consultations (inbound calls) and marketing information or service progress notifications (outbound calls). These traditional intelligent voice dialogue systems offer high stability and maintainability in implementing fixed processes and structured interactions. However, due to the simple classification models in the intermediate models and the fixed SPD canvas process rules, these systems often fail to better understand complex and changing user intent, hindering the ability to implement more flexible dialogues and thus failing to meet the growing demand for intelligent services. In recent years, the rapid development of large language models (LLMs) has demonstrated significant advantages in natural language understanding and generation tasks. They can more accurately capture speech information, support open-ended conversations, and possess excellent contextual reasoning capabilities. Therefore, implementing intelligent voice dialogue systems based on language models is a key measure to improve the intelligence and flexibility of system conversations. However, since voice conversations have high timeliness requirements and language models are known to have high real-time response (RT), directly integrating language models into intelligent voice dialogue systems may have the opposite effect, causing the RT to be even higher, affecting the real-time performance of intelligent voice dialogue systems and the user conversation experience. Therefore, it is necessary to accelerate the entire conversation chain from both the algorithm and engineering perspectives to achieve the ideal conversation real-time response (RT).

[0054] Currently, there is a solution that implements a Thousand Machines Digital Human for intelligent sales services. Specifically, this Thousand Machines Digital Human reduces latency through solutions such as language model (LLM) reasoning acceleration, key-value (KV) cache management, dynamic batch processing, knowledge fragment fusion, and high-frequency information compression. The core focus is on optimizing the algorithm model's RT, with less engineering optimization. Furthermore, the voice duplex module in this solution is completely handed over to cloud communications, making it impossible to improve the voice duplex module level.

[0055] To sum up, there is an urgent need to propose a solution that can effectively integrate language models and realize intelligent upgrades while ensuring the real-time performance of the voice dialogue system, so as to promote the development of intelligent voice dialogue systems in a more flexible, natural and efficient direction.

[0056] To this end, this application provides a solution to the problem that when a language model-based agent is connected to an intelligent voice dialogue system, the RT of the voice dialogue will become high, which affects the user dialogue experience. The basic ideas of this solution include: on the engineering side, using TTS streaming, parallelization of modules in the Agent, etc. to accelerate dialogue requests and reduce the overall RT of the dialogue. At the same time, in the model call, KVCache, PD separation and OpenAI call format are optimized to improve the model call speed, thereby reducing the dialogue RT; in addition, a second preset model is provided through relevant algorithms to detect the end of the user's voice, thereby replacing the existing ASR that implements sentence end judgment by configuring a fixed silence duration (that is, the existing ASE often uses a fixed VAD (voice endpoint detection) duration). This can reduce the ASR recognition waiting time. As a result, this application can reduce the RT of the dialogue to about 1.3-1.7s, and the fluency of the dialogue is significantly improved.

[0057] Regarding KVCache, PD separation and OpenAI calling format, etc., they are described in detail in other embodiments below and will not be described in detail here.

[0058] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] It should be noted that, in the case of user information involved in the embodiments of the present application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0060] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0061] First, the terms used in the embodiments of the present application are explained. It should be understood that this explanation is for a clearer understanding of the embodiments of the present application and does not necessarily constitute a limitation on the embodiments of the present application.

[0062] An intelligent voice dialogue system (capable of online human-computer voice dialogue, such as a customer service system) is a human-computer voice interaction system based on speech recognition (ASR), natural language processing (NLP), and text-to-speech synthesis (TTS) technologies. This system enables machines to understand human voice input, process it, and provide corresponding services or information, ultimately providing feedback to the user in the form of voice. ASR converts user voice input into text. NLP understands and analyzes the converted text and generates appropriate responses. TTS converts the generated responses into voice output. In other words, it uses speech synthesis technology to convert text into audible sound, enabling the machine to "speak" and complete a voice conversation with the user.

[0063] An agent is an intelligent entity capable of perceiving its environment, developing plans, and taking actions to achieve its goals. It consists of three core modules: brain, perception, and action. The brain, typically comprised of a language model (LLM), is responsible for planning and decision-making; the perception module is responsible for multimodal perception of the environment; and the action module executes specific actions based on the plan using tools.

[0064] The preset model is a trained machine learning model. In this application, the preset model includes multiple first preset models, second preset models, and third preset models. Multiple first preset models and third preset models are all machine learning models for generating natural language output. For example, multiple first preset modules include an explanation model (Explain model) and a retrieval increase model (Rag model). The detailed description of the Explain model and the Rag model will be described in detail in other embodiments below and will not be described in detail here. The third preset module may be a language module (LLM). Among them, the language model (LLM): is a deep learning model with a large number of parameters, mainly used to process natural language processing (NLP) tasks. LLM is based on the transformer (Transformer) architecture and can understand and generate high-quality human language text. And, the second preset model is a lightweight model for sentence end detection of voice dialogue streams. The embodiment of this application does not limit the number of model parameters supported by each preset model, with the goal of meeting application requirements.

[0065] KVCache (Key-Value Cache) is a key technology used to store and reuse intermediate calculation results during the language model (LLM) reasoning process;

[0066] Streaming response: A technology that gradually generates and returns results, suitable for scenarios with progressive output. Its characteristics include real-time performance (results are returned as they are computed, without waiting for all results to be completed) and low latency (users can quickly see partial output).

[0067] Streaming TTS: Combining the powerful text generation capabilities of language models with real-time speech synthesis technology, it enables efficient conversion from streaming text to streaming speech, thereby improving the user experience. Specifically, traditional TTS technology typically requires converting the entire text into speech before playback, which causes a certain delay. Streaming TTS, on the other hand, can start playback as the text is gradually converted into speech, significantly reducing user waiting time and making voice conversation interactions more natural and smooth.

[0068] PD separation: This technology separates the prefill and decode phases during model inference. This separation can significantly improve inference efficiency, especially in large-scale model applications.

[0069] Voice duplex module: refers to the technology or component that can support two-way voice communication at the same time.

[0070] The following method embodiments provided in this application are implemented based on the voice dialogue system provided in this application, which is an intelligent voice dialogue system with online human-computer voice dialogue function. Figure 1 The hardware structure of the voice dialogue system is shown as an example. Figure 1 As shown, the voice dialogue system includes: a server 100 and a client 200. The server can be a server, a service cluster, a virtual server, or a cloud, etc., and this embodiment does not specifically limit this. The server provides corresponding functional services to the client, such as intelligent customer service, intelligent question and answer services, etc. The user can initiate a voice call through the browser, application (APP), web application H5 (HyperText Markup Language 5, the fifth generation of HTML, Hypertext Markup Language), light application (also known as mini-program, a lightweight application) or cloud application on the client. After the voice call is connected to the voice dialogue system on the service, an online human-computer voice conversation will be established between the client and the server. When the user speaks during the human-computer voice conversation, the user's voice will be collected by the audio pickup device (such as a microphone) on the client to form a corresponding voice conversation stream, and then sent to the server via the voice conversation stream. The above-mentioned client can be, but is not limited to, a terminal device such as a smartphone, a smart wearable device, a tablet computer, a laptop computer, a desktop computer, etc.

[0071] During specific implementation, each method embodiment provided in this application is applied to the above-mentioned server 100.

[0072] For detailed description of the specific functions of the server 100 and the client 200, please refer to the relevant content in the embodiments below, which will not be described in detail here.

[0073] Figure 1 This is a flow chart of a voice dialogue method provided by an exemplary embodiment of the present application. Figure 1 , the voice dialogue method comprises the following steps:

[0074] 101. Receive a user's voice conversation stream;

[0075] 102. Perform a streaming response on the voice dialogue stream, wherein, during the streaming response process, a plurality of reply text segments are generated in segments, and the reply text segments are output in real time after being generated to be stored in a first cache queue;

[0076] 103. When detecting that the reply text segment is stored in the first cache queue, convert the stored reply text segment into a reply voice segment in real time;

[0077] 104. Play the reply voice clip.

[0078] In the above 101, the voice dialogue stream is a data stream formed by the voice emitted by the user during the human-computer voice conversation through the client. Specifically, the voice dialogue stream can be generated by the audio pickup device such as a microphone on the client to collect the voice emitted by the user. The human-computer voice conversation can be initiated actively by the user or responded to passively. For example, in the e-commerce scenario, each e-commerce platform often provides an intelligent sales service function, which can simulate human voice conversation to help solve user problems or provide the required information, etc., so that the user can actively initiate a voice call through the intelligent sales service entrance provided on an e-commerce platform for voice consultation; or, the e-commerce platform can also actively initiate a voice call to the user through the intelligent sales service on it, and inform the user of service progress, marketing information, etc. by voice.

[0079] Regardless of whether the user initiates a human-machine voice conversation or responds passively, a unique session identifier is assigned during the establishment process. This uniquely identifies the human-machine voice conversation, allowing the system to identify and track requests belonging to the same human-machine voice conversation and maintain session status. In addition, the session identifier has other functions in this application, which will be discussed in detail below and will not be detailed here.

[0080] Based on this, the method provided in this application may further include the following steps:

[0081] 100a. Allocate a unique session identifier for the human-machine voice conversation (also called a human-machine voice call) with the user. The session identifier is, for example, a session ID (chat_id), which may be allocated during the process of establishing the human-machine voice conversation.

[0082] Furthermore, in the above 101, the voice conversation stream is the voice stream captured by the client during the human-machine voice conversation and sent to the server. That is, the voice conversation stream mentioned in the above 101 is the current voice conversation stream of the user during the human-machine voice conversation.

[0083] During a human-machine voice conversation, response time (RT) is crucial to ensuring a positive user experience. Responding to a user's voice conversation flow often involves the following steps:

[0084] 1) Through ASR technology, the user's voice dialogue flow is recognized to obtain the corresponding dialogue text.

[0085] 2) Generate a request based on the conversation text and send it to the intelligent conversation module. The intelligent conversation module is built based on a corresponding preset model (such as a language model). It recognizes and analyzes the conversation text carried in the request, understands the user's conversational intent, and responds to the request.

[0086] 3) Convert the text content output by the intelligent dialogue module in response to the request into speech and play it. Among them, TTS technology is often used to convert the text content output by the intelligent dialogue module into speech.

[0087] As can be seen above, during the human-machine voice conversation process, the time consumed by intelligent voice outbound calls is primarily concentrated in three parts: ASR, TTS, and the intelligent conversation module. Traditional TTS technology is typically non-streaming, requiring the text to be synthesized to be processed and sent all at once. In other words, it is necessary to wait until the intelligent conversation module has completely generated the entire text before converting it to speech. When using the intelligent conversation module for conversations, if the model outputs all text at once, it will take at least 2 to 5 seconds to retrieve the entire text content from the intelligent conversation module. After that, using TTS technology to convert this entire text content to speech can take around 400 milliseconds at the fastest. Clearly, the overall response time is very long, with high response latency, which is unacceptable for intelligent voice services and seriously affects the user experience.

[0088] In order to improve the above situation, this application proposes an optimization strategy: connect to TTS cosyvoice, store the response of the intelligent dialogue module in the buffer queue of the voice duplex module after each round of dialogue, and then call TTS cosyvoice in a streaming manner. Figure 3 As shown in Figure 1, this optimization strategy involves integrating a TTS service such as CosyVoice (a specialized TTS service that supports real-time text-to-speech conversion) and combining it with the buffer queue in the voice duplex module to implement streaming storage and processing of conversations. The voice duplex module includes an ASR unit and a TTS unit (specifically implemented based on a TTS service such as CosyVoice).

[0089] The following combination Figure 3 , the above steps 101 to 104 are described in detail.

[0090] After the client initiates the user's voice conversation stream to the server, it is first received by the voice duplex module on the server. The voice duplex module uses its ASR unit to convert the voice conversation stream into the corresponding conversation text. Then, based on the conversation text, it generates a request and sends it to the intelligent conversation module. After receiving the request, the intelligent conversation module begins to analyze and process the conversation text carried in the request and responds to the request. Unlike traditional methods, the intelligent conversation module performs a streaming response to the request. During the streaming response process, it gradually generates reply text segments and outputs each segment immediately after generating a part of the reply text, rather than waiting for the entire reply text content to be generated before outputting it. This streaming response method allows the client to receive partial response content as early as possible, reducing waiting time.

[0091] Furthermore, when the intelligent conversation module executes the streaming response process, each conversation text segment it outputs step by step will be sequentially stored in the first cache queue in the voice duplex module. This not only helps to smooth the data flow, but also allows the processing speed and order to be flexibly adjusted according to needs.

[0092] Furthermore, TTS cosyvoice will be called in a streaming manner. Specifically, once it is detected that a new reply text segment is stored in the first cache queue, TTS cosyvoice will be started for processing, and the newly stored reply text segment will be converted into a corresponding reply voice segment, and the reply voice segment will be stored in the second cache queue in the voice duplex module to wait for the playback opportunity to be played through the client. Since TTS CosyVoice supports streaming input, it can return the first reply voice segment (the first packet audio) for playback within approximately 500-600 milliseconds after receiving the first reply text segment, significantly reducing the waiting time for the first response.

[0093] For ease of understanding, a specific scenario example is given here for the above steps 101 to 104.

[0094] Suppose the user's voice conversation flow corresponds to the text "I want to check the status of my latest order." The intelligent conversation module gradually generates responses to this text. For example, the first response text segment generated may be "Your order is being processed." After a period of time, a second response text segment is generated, "Expected to ship within three days." Each of these response text segments is immediately stored in the first cache queue. After reading the first response text segment "Your order is being processed" from the first cache queue, TTS cosyvoice immediately begins synthesizing speech and outputs the first response audio segment approximately 500-600 milliseconds later. The client then plays the first response audio segment "Your order is being processed." Furthermore, TTS cosyvoice continues to process the second response text segment "Expected to ship within three days" stored in the first cache queue. " and outputs the corresponding second reply voice segment. The second reply voice segment will be appended to the above-mentioned first reply voice segment to wait until the first reply voice segment is played before playing the second reply voice segment, thereby forming a reply voice stream playback. It can be seen that with this method, when the user hears the first voice response, the subsequent voice has been synthesized in the background, and the user's overall perceived delay is reduced.

[0095] In summary, this application introduces streaming-supported TTS cosyvoice and the corresponding caching mechanism, and enables the intelligent dialogue module to adopt streaming responses, which can achieve simultaneous generation and playback, greatly shortening the overall delay from reply text generation to voice playback, especially the first response time is significantly optimized.

[0096] The above mainly describes the optimization and improvement of the TTS unit in the voice duplex module. In addition, this application also makes other optimizations for the voice duplex module, such as the optimization of the ASR unit and the timeout exception protection.

[0097] The following details the optimization of the ASR unit and the timeout exception protection.

[0098] 1. Optimization of ASR Unit

[0099] The ASR unit's recognition needs to return the recognized text content after detecting that the user has finished speaking, which relies on voice endpoint detection. Voice endpoint detection is also called Voice Activity Detection (VAD). Traditionally, when the ASR unit performs voice endpoint detection on the user's voice conversation flow through VAD, it does so by configuring a fixed silence duration. This results in the duration configuration for VAD having a significant impact on the real-time conversation effect. If the setting is too small, it can easily lead to misinterpretation (i.e., the user is mistakenly considered to have finished speaking before the sentence is finished). If the setting is too large, the recognition result will be returned long after the user finishes speaking.

[0100] In order to solve the problems caused by VAD duration configuration, this application integrates an additional second preset model (this model has a relatively small number of parameters) to continuously detect whether the user's current voice conversation flow has reached the end of a sentence, and immediately sends an instruction to ASR to return the recognized text content when the end of the sentence is detected, instead of waiting for the configured silence duration requirement to be met. This can reduce the ASR recognition time and significantly improve the fluency of real-time conversations and user experience.

[0101] Based on this content, the provided method may further include the following steps:

[0102] 100b. Perform sentence end detection on the voice dialogue stream using a second preset model;

[0103] 100c. When detecting that the voice dialogue stream reaches the end of a sentence, sending a return instruction to the voice recognition unit; the voice recognition unit is used to convert the voice dialogue stream into dialogue text;

[0104] 100d. Based on the conversation text returned by the speech recognition unit, trigger the agent to execute corresponding steps.

[0105] In the above description, the speech recognition unit is an ASR unit. The second preset model can be specifically used to monitor the user's voice conversation flow in real time and determine whether the user is at the end of a sentence based on the characteristics of the voice conversation flow and / or in combination with the intermediate output results of the ASR unit as additional context information. For example, certain specific words or phrases (such as thank you and goodbye) often appear at the end of a sentence. These words can help the second preset model more accurately determine the end of a sentence.

[0106] Furthermore, the second preset model is a lightweight model obtained through training. For example, a lightweight deep learning model (such as a model based on an LSTM (Long Short-Term Memory) network, a GRU (Gated Recurrent Unit), or a Transformer architecture) can be trained using a speech sample dataset with annotated sentence boundaries to obtain the second preset model.

[0107] The present application uses the second preset model to determine the end of a sentence, thereby replacing the fixed VAD duration configuration of traditional ASR, thereby reducing the ASR recognition waiting time and thereby reducing the response time (RT).

[0108] 2. Timeout exception protection

[0109] In the process of human-computer voice conversation, it is inevitable that there will be RT spikes in the model call of the intelligent dialogue module at a certain stage, which will cause the intelligent dialogue module to be unresponsive for a long time in a round of dialogue. Therefore, this application also plays a fallback speech. Specifically, the timing starts from the end of the user's speech. If the intelligent dialogue module has not started to output the reply after the set time (such as 3s), the preset fallback voice will be broadcast immediately, such as "The network is not very good, please say it again." Through this fallback speech, the fluency of the conversation is guaranteed, and the overall RT improvement of the conversation caused by the RT spike of the model call in the intelligent dialogue module is reduced.

[0110] Among them, RT spike refers to the phenomenon that the response time (RT) of the intelligent dialogue module suddenly increases significantly at a specific time point or time period.

[0111] In addition to the aforementioned TTS and ASR affecting response time (RT), Figure 3 The intelligent dialogue module shown in Figure 1 is also a key component that affects response time (RT). To further reduce response time, this application has also optimized the intelligent dialogue module. This optimization has been implemented in three versions. The following details the optimization of the three versions of the intelligent dialogue module.

[0112] 1. The first optimized version of the intelligent dialogue module

[0113] In order to continue the traditional rule engine (for SDP systems)'s ability to strictly control the dialogue service process, while being able to utilize the capabilities of models such as those used for intelligent dialogue, this application embeds a preset model for intelligent dialogue into a node in the SPD canvas, and this preset model is responsible for all dialogue functions, including policy rules, intent recognition, action planning, and response speech generation. To this end, this application uses a general language model with a large number of parameters (such as a parameter volume of up to 100 billion). In the field of natural language processing, the size of a model is usually measured by the number of its parameters. The more parameters, the more complex the language features that the model can theoretically learn, which may provide more accurate and richer language understanding and generation capabilities.

[0114] In this application, the preset model embedded in a working node in the SPD canvas is referred to as a third preset model. The third preset model serves as an intelligent dialogue module.

[0115] Based on this, in one possible technical solution, the above step 102 of "streaming response to the voice dialogue stream" may include the following steps:

[0116] S11, performing speech recognition on the user's voice dialogue flow to obtain the corresponding dialogue text;

[0117] S12. Use the conversation text to trigger a third preset model to execute the streaming response.

[0118] For the implementation of the above S11, please refer to the relevant content in other embodiments above.

[0119] In S12 above, the third preset model is a trained language model responsible for all conversational functions. In specific implementations, the third preset model can be, for example, a language model containing approximately 72 billion parameters. This third preset model is deployed in a node within the rule engine. The rule engine is used to control user-machine voice conversations according to preset conversational service rule logic. Each node in the rule engine can be an independent unit or step in the rule flow, and each stage represents a point in time where a specific task is processed or specific logic is executed.

[0120] This solution uses the third preset model to enable the intelligent dialogue module to have language model capabilities, but it also has many obvious problems, including the following:

[0121] 1) Prompt information is difficult to maintain: Because the third preset model mentioned above needs to complete all conversational functions, prompt information contains a lot of conversational service rules, knowledge, and conversational service process jumps. It combines natural language and code, making it complex to modify. Furthermore, online anomalies (bad cases) are difficult to fix, as fixing one problem may affect other conversations.

[0122] 2) The dialogue service effect is not good enough: the prompt information contains too much content, which may make the third preset model processing mentioned above prone to hallucinations, fabrication of facts or excessive promises.

[0123] 3. Slow response speed: In order to handle all conversational functions, the third preset model needs to be a language model with a large number of parameters. However, the more parameters and prompt information the third preset model has, the longer the response time (RT) tends to be. As a result, in this technical solution, the third preset model takes more than 800 milliseconds to output the first word (that is, the RT corresponding to the first word is more than 800 milliseconds). Combined with the time required for ASR and TTS processing, the time required for a single round of conversation reaches about 3.2 seconds. Obviously, the overall response time is very long, and the user perceives the conversation as very slow.

[0124] 2. The second optimized version of the intelligent dialogue module

[0125] In order to solve the problems existing in the first optimized version mentioned above, this application splits all dialogue functions and implements all dialogue functions through multiple functional components. Each functional component is responsible for a part of the dialogue function. To this end, this application adopts an agent architecture for the optimized design of the intelligent dialogue module.

[0126] However, considering some specific application scenarios, such as human-computer voice dialogue scenarios, a user inbound call / intelligent outbound call itself is a clear and basic task unit (such as user query order status, notification of user order status, marketing promotion, etc.), its goal is relatively simple and the process is controllable, so it does not require complex task decomposition and multi-step planning capabilities. Based on this, this application simplifies the traditional intelligent agent architecture, especially weakening or even omitting the planner, to form a lightweight intelligent agent architecture.

[0127] Figure 4a The lightweight intelligent agent architecture provided by this application is exemplarily shown. Figure 4a The architecture of the agent mainly consists of three components: Planner, Reasoner and Executor.

[0128] The aforementioned planner is responsible for breaking down user goals into specific tasks. In practice, this application weakens the planner by setting it to Empty, meaning that the task planning function is not enabled and the system proceeds directly to the next step. By weakening the planner, this application can reduce the complexity of the agent architecture, improve the agent's response efficiency, and avoid increased response delays caused by over-planning.

[0129] The reasoner, as described above, is the core of the entire agent, responsible for executing specific tasks and generating action commands. During a conversation, the reasoner needs to understand the user's intent based on their input and decide what action to take to answer the user's question (for example, deciding which pre-defined model to invoke to generate the response text). Furthermore, the reasoner maintains the conversation state.

[0130] The executor is primarily responsible for executing specific operations based on the reasoner's action instructions to generate a response text. For example, it generates a response text by calling the Explain model, or by calling the Rag model to retrieve and generate a response text, or by directly querying the speech configuration to determine the response text. The Explain model and Rag model described above belong to the first preset model. The Explain model is used to generate a more detailed and personalized response text; the Rag model is used to retrieve relevant information from the knowledge base and generate a response text accordingly. The knowledge base includes some pre-configured response speech.

[0131] Furthermore, the actuator can provide feedback to the reasoner (corresponding to the reasoning step shown in FIG4 ), so that the reasoner can adjust its decision and action plan based on the feedback.

[0132] In addition to the above, the intelligent agent can also include other components, such as tool modules (Tools), text modules (Text), search modules (Search), memory modules, etc. The tool module can provide the executor and reasoner with necessary auxiliary functions (such as API interface calling tools, text processing tools, etc.), and the text module is used to process text-related tasks, such as parsing instructions, etc. The search module is used to perform search operations, such as knowledge base queries, network searches, etc. The memory module is used to cache data, such as human-computer voice dialogue status, queries (State), queries (Qurey, such as the dialogue text corresponding to the user input voice dialogue flow), user intentions (Intend), actions decided by the reasoner (Acttion), and the reply text (Answer) output by the model.

[0133] Based on the above content, in another possible technical solution, the above 102 “streaming response to the voice dialogue stream” may include the following steps:

[0134] S21, performing speech recognition on the user's voice dialogue flow to obtain the corresponding dialogue text;

[0135] S22. Based on the conversation text, trigger the agent to execute the following steps in sequence:

[0136] S221: Determine a reply generation method based on the conversation text and construct a first prompt message;

[0137] S222: Select a first preset model from a plurality of first preset models based on the conversation generation method;

[0138] S223: Use the selected first preset model to execute the streaming response based on the first prompt information.

[0139] Regarding the implementation of the above S21, please refer to the relevant content in other embodiments above.

[0140] In the above S22, step S221 is implemented by a reasoner. For example, the reasoner can identify the user's intent based on the conversation text. Based on the user's intent and combined with predefined policy rules (used to guide the decision-making process), it can determine the method for generating a response. Furthermore, based on the user's intent, it can construct first prompt information as input for the subsequently invoked first preset model. In the above steps S222-S223, the multiple first preset models include, but are not limited to, the Explain model and the Rag model.

[0141] In the intelligent dialogue module, under the above-mentioned intelligent agent architecture, the response time of the intelligent dialogue module is mainly concentrated on the three parts of the reasoner, multiple first preset models (such as the Explain model and the Rag model). Since the functions of these three parts are relatively simple and concentrated, this application uses related models with relatively small parameter amounts (such as about 7 billion parameters) for these three parts, and when constructing the first prompt information, the prompt information can be shorter. This solution reduces the overall response time (RT) to a certain extent. Among them, the reasoner is mainly responsible for intention recognition and action decision-making, and its response time takes about 200-300ms; the Explain model is responsible for generating explanatory replies, and the Rag model is responsible for knowledge retrieval and generating replies. The reasoner decides whether to use the Explain model or the Rag model to generate a reply. Therefore, the first word response of the reply takes about 300-400ms. As a result, the response time (RT) of the intelligent agent is about 500-700ms, which is 100-300ms lower than the previous version.

[0142] 3. The third optimized version of the intelligent dialogue module:

[0143] Although the second optimized version of the intelligent dialogue module mentioned above significantly improves the dialogue effect, some illusions and answer quality issues still exist, and the response time (RT) improvement is not significant. One reason for the lack of significant improvement in response time (RT) is that if the reasoner is responsible for intent recognition and action decision-making while also generating prompt information for the first preset model, this will undoubtedly increase processing time. This is because generating high-quality prompt information usually requires an in-depth understanding of the user's input and customizing the prompts according to specific task requirements to ensure that high-quality responses can be obtained through the first preset model.

[0144] Through the above analysis, the present application optimizes and converges the function of the reasoner in the agent, and realizes concurrent execution of the reasoner and multiple first preset models. Figure 4b As shown:

[0145] For the reasoner, the convergence goal is to separate its responsibilities from multiple tasks such as awareness recognition, prompt generation, and initial model selection, focusing on basic dialogue strategy decisions. This improves the reasoner's lightweight and real-time performance. These basic dialogue strategy decisions include selecting the first question, deciding whether to terminate the conversation, and deciding which initial model to invoke (e.g., the Explain model or the Rag model).

[0146] Furthermore, the Explain model's prompts break down the conversation flow into detailed thought processes, improving response quality. Specifically, the Explain model is separated and, by breaking down the conversation flow into a visual "thought process," it guides the model to generate higher-quality responses. This can be achieved, for example, by using structured prompts to simulate human thought processes and encapsulating user input and conversation context into a fixed template format. The Explain model then processes this information to produce a response.

[0147] For example, assuming that the dialogue text corresponding to the voice dialogue flow input by the user is "Why is my credit card so high this month?", the structured prompt information design method may include but is not limited to the following steps:

[0148] Step 1: Identify possible causes

[0149] Tip: Considering the abnormally high credit card bill, please first check the following: whether there are new high-value purchases, subscription services or automatic renewal items.

[0150] Step 2: Further analysis

[0151] Tip: For each of the above points, please explain in detail how you verified it and how it affected your bill.

[0152] The Rag model improves knowledge selection accuracy through knowledge internalization pre-training, eliminating the need for knowledge retrieval. During the knowledge internalization process, a large amount of background knowledge is pre-integrated into the Rag model's parameters. Then, through large-scale pre-training, the Rag model can extract relevant information directly from internally stored knowledge, rather than relying on real-time retrieval at runtime. This knowledge internalization pre-training approach allows the Rag model to learn a wide range of knowledge during the pre-training phase and encode this knowledge in some form within its model structure. This avoids the latency associated with online retrieval and enables asynchronous concurrent processing similar to the Explain model.

[0153] Through the above optimization design, this application makes the Explain model and the Rag model independent concurrent execution models, and the processing of these two models no longer depends on the reasoner, so the calls of these two models can be executed concurrently with the agent in advance, and the reply text segments output by these two models are stored in their respective corresponding second cache queues for waiting to be pushed. For example, see Figure 4b or Figure 4c As shown in the figure, the segmented output of the Explain model is stored in the cache queue Q1, and the segmented output of the Rag model is stored in the cache queue Q2. The cache queue Q1 and the cache queue Q2 are different second cache queues. After the reasoner is executed, the result determines which segment of the second cache queue to use. Specifically, Figure 4c As shown, the reply text segments stored in one of the second cache queues are pushed in a streaming manner to be pushed to the first cache queue for waiting for processing. Wherein, the push is generally based on the first-in-first-out principle.

[0154] In summary, through the above optimization design, this application upgrades the intelligent body to Figure 4b The parallel execution architecture shown in the figure shows that the response time (RT) of a conversation depends on the faster response time of the reasoner or multiple first preset models (Explain model, Rag model). After optimization, the RT of the reasoner is approximately 150ms, and the first packet RT of the Explain model or Rag model is approximately 300ms, resulting in a RT of 300ms for a single conversation round.

[0155] Based on the above, in another possible technical solution, the above-mentioned 102 "streaming response to the voice dialogue stream, and storing multiple reply text segments output in the streaming response process in a first cache queue in real time" may include the following steps:

[0156] S31, performing speech recognition on the user's voice dialogue flow to obtain the corresponding dialogue text;

[0157] S32: Based on the conversation text, trigger the agent to execute the following steps:

[0158] S321, while determining a conversation generation method according to the conversation text, respectively executing the streaming response for the conversation text using a plurality of first preset models;

[0159] S322: According to the conversation generation method, multiple conversation texts output in segments by one of the first preset models during the streaming response are pushed to the first cache queue in real time for storage.

[0160] Regarding the implementation of the above S31, please refer to the relevant content in the above other embodiments.

[0161] In S321-S322 above, when streaming responses are executed for the conversation text using multiple first preset models, the multiple conversation text segments output by different first preset models are stored in different second cache queues. After determining a conversation generation method, the conversation text segment stored in the second cache queue corresponding to one of the first preset models is pushed to the first cache queue in real time for storage based on the conversation generation method.

[0162] Furthermore, the plurality of first preset models include a retrieval enhancement model. The retrieval enhancement model retrieves relevant information from its own internally stored knowledge based on the conversation text during the streaming response process to generate a reply text segment.

[0163] It is necessary to further explain the above-mentioned multiple first preset models: during the execution of the streaming response, the first preset model can output at least one reply text segment in segments. When only one reply text segment needs to be output, the first preset model can be understood as executing a one-time reply.

[0164] In the above-mentioned intelligent dialogue module (agent), the first preset model calls involved are almost all implemented directly through the corresponding call service platform. The call service platform has certain optimizations for model deployment and calling, and some require the caller to obtain acceleration support based on the parameter format or content. In order to more effectively provide response quality and response speed, this application also makes some optimizations for calling the corresponding first preset model through the call service platform. The optimization items include but are not limited to KVCache reuse optimization and OpenAI format call optimization, specifically open intelligent model (such as the first preset model) call input optimization, dialogue warm-up, etc.

[0165] The following describes each of the above optimization items in detail.

[0166] 1. KVCache reuse optimization (specifically, request modification)

[0167] At present, the framework of the calling service platform supports KVCache reuse between multiple rounds of conversations. In the actual model calling process, this application will convert the conversation identifier of the corresponding human-computer voice conversation (such as the conversation ID (chat_id) into, for example, a hexadecimal string, and then in the human-computer voice conversation process, each round of request (generated based on the conversation text corresponding to each round of users) will carry the field "chat_id": "xxxx", where "xxxx" is a hexadecimal string. After the request is sent to the calling service platform, the calling service platform can use, for example, a consistent hashing algorithm to perform a hash calculation on the value of the field "chat_id" carried in the request to assign the request to the corresponding agent based on the calculation result. This method of assigning requests based on the value of "chat_id" can effectively ensure that all requests for the same human-computer voice conversation fall on the same agent for processing, which can reduce the agent's reasoning time and improve the speed and quality of the response.

[0168] Based on this, the above-mentioned steps S22 and / or S32 in this application, "based on the dialogue text, triggering the agent to execute corresponding steps" may include:

[0169] A11. Generate a conversation request based on the conversation text and a conversation identifier of the human-computer voice conversation with the user;

[0170] A12. Send the dialogue request to the agent, triggering the agent to execute corresponding steps; wherein, the agent calls the same first preset model for dialogue requests carrying the same session identifier.

[0171] In specific implementations, an agent can integrate multiple types of first preset models, and each type of first preset model can contain multiple models, such as multiple Explain models and multiple Rag models. After receiving a conversation request, the agent sends the conversation request to the corresponding invocation service platform. The invocation service platform then assigns the conversation request to the first preset model to be invoked based on the session identifier carried in the conversation request. Specifically, the invocation service platform assigns conversation requests carrying the same session identifier to the same first preset model.

[0172] For example, combined with Figure 4aAs shown, in the second optimized version of the intelligent dialogue module, the intelligent dialogue module is an intelligent agent architecture of a non-parallel architecture and is non-parallel. Then: when the intelligent agent determines that the response generation method for the received dialogue request is to generate a response through the Explain model, it can regenerate a call request based on this response generation method and the session identifier carried in the dialogue request and send the call request to the call service platform; after that, the call service platform will use, for example, a consistent hashing algorithm, based on the session identifier carried in the call request (specifically in the form of a hexadecimal string), to select an Explain model from multiple Explain models for calling, so that all dialogue requests carrying the same session identifier can call the same Explain model to generate responses.

[0173] For example, combining Figure 4b As shown, in the third optimized version of the intelligent dialogue module, the intelligent dialogue module is an agent with a parallel architecture. When the agent receives a dialogue request, it invokes the service module, using a consistent hashing algorithm, for example, to select an Explain model from multiple Explain models and a Rag model from multiple Rag models based on the session identifier carried in the dialogue request. The agent then invokes the selected Explain model and Rag model. This ensures that the same Explain model and Rag model are invoked for all dialogue requests carrying the same session identifier.

[0174] 2. OpenAI format call optimization

[0175] When the service platform determines to call the corresponding first preset model, the present application also optimizes the call input parameters. Specifically, the prompt information is separated by separating the conversation content according to the role, that is, when constructing the prompt information input to the first preset model, the conversation content is separated according to the role. Among them, the following three roles are mainly included: the control role (also known as the system role, system), the user role (user), and the model role (also known as the assistant role, assistant). The control role (system) is used to store the prompt text with unchanged content, which contains the dialogue service rules and dialogue process constraints. The user user is used to pass the dialogue text content corresponding to the user voice dialogue flow. The model role (assistant) is used to pass the response content of the first preset model (or it can also be understood as an intelligent agent). Since the longest match cache is used during model inference, such a call will enable multiple conversations to have the largest longest match content, the overall response time (RT) will be lower, and the quality of the model answer will be higher than the request of splicing the prompt information and the dialogue history together.

[0176] For example, the prompt information constructed and input to the first preset model is as follows:

[0177] {

[0178] {Control role: system

[0179] Content for the management role: "You are a professional customer service assistant who helps users with questions about their order status. Always be friendly and detailed."

[0180] {Model role: assistant

[0181] Content of the model role: "Hello, I'm your account manager. You've only ever applied for a refund for a coconut order on the platform, correct?"

[0182] {User role: user

[0183] User role content (comtent): "yes"}

[0184] {Model role: assistant

[0185] Content of the model character: "Regarding this order, our salesperson completely understands your feelings. We have also negotiated with the merchant, but because they believe your request was made in error, we are unable to support your request. The only solution we can offer you is a 20 yuan refund with no need to return the product. Is this an acceptable solution?"}

[0186] {User role: user

[0187] User role content (comtent): "No, too little"}

[0188] }.

[0189] The above “No, it’s too little” is the dialogue text corresponding to the current voice dialogue flow of the user during the human-computer voice conversation.

[0190] As can be seen from the above, the prompt information constructed above records the user's conversation text and the reply text of the first preset model in sequence. Each time the user's new conversation text is added to the end of the list in the prompt information, there is no need to re-splice the entire conversation history. This not only ensures the continuity of the context, but also improves the reasoning efficiency.

[0191] Based on the above content, the “calling the first preset model” executed in the agent may include the following steps:

[0192] A21. Constructing a second prompt message based on the conversation text;

[0193] A22. Input the second prompt information into the first preset model to trigger the first preset model to perform a streaming response.

[0194] Among them, the second prompt information includes the content of multiple roles, and the multiple roles include management roles, user roles, and model roles. The content of the management role includes setting prompt text, and the setting prompt text includes constraint information for the first preset model. For example, the setting prompt text can be "You are a professional customer service assistant who helps users solve problems about order status. Please always provide friendly and detailed information" as given in the above example. The content of the user role includes the dialogue text corresponding to at least one voice dialogue flow input by the user during the human-computer voice conversation. The dialogue text includes the dialogue text corresponding to the user's current voice dialogue flow (such as "No, too little" given in the above example) and the dialogue text corresponding to the user's historical voice dialogue flow (such as "Yes" given in the above example). The content of the model role includes the reply text output by the first preset model, such as "Hello, I am your account manager. You have only applied for a refund for a coconut order on the platform, right?" given in the above example.

[0195] It should be noted here that the construction of the first prompt information involved in step S221 given in the other embodiments mentioned above can also be constructed by separating the dialogue content according to roles. In this case, the first prompt information constructed in step S221 can be the same as the second prompt information described here.

[0196] 3. Dialogue Warm-up

[0197] Currently, to improve model inference efficiency, PD splitting is often used. PD splitting involves splitting the prefill and decode phases of the model inference process onto different work nodes (such as different servers). This split is particularly suitable for traffic-intensive scenarios, as it prevents prefill tasks from negatively impacting the decode phase, thereby ensuring smoother and more responsive conversations.

[0198] Specifically, the above-mentioned Prefill phase is mainly responsible for initializing the model state, processing the input sequence and preparing the information required for subsequent decoding. This phase is usually time-consuming, especially for long sequences or complex tasks. And, the above-mentioned Decode phase is based on the state information generated in the Prefill phase. It performs step-by-step decoding to generate output sequences (such as replies, etc.). This phase is relatively lightweight, but requires a fast response to maintain the continuity of real-time conversations. In high-concurrency scenarios, if Prefill and Decode are executed on the same work node, it may cause resource contention and affect the response time of the Decode phase. By separating these two phases onto different work nodes, this bottleneck can be effectively alleviated and the efficiency of model inference can be significantly improved.

[0199] Based on this, this application will call the Explain and Rag models twice each before the actual conversation begins (such as a call before an outbound call) and / or after the first interaction (the first round of call after an outbound call). These calls do not directly serve user requests, but are used to warm up the conversation so that the model can prepare the necessary context information and status, thereby ensuring that requests in subsequent rounds of the conversation use the PD separation function, which has a significant effect on reducing the RT of non-first round conversations.

[0200] For example, after this "conversation warm-up", the conversation can enter the Decode phase faster in subsequent rounds because the relevant resources have been initialized or loaded into memory.

[0201] Based on the above content, the method provided in this application may further include the following steps:

[0202] A31. Obtaining conversation warm-up strategies

[0203] A32. When it is determined based on the conversation warm-up strategy that the warm-up time has arrived, the plurality of second preset models are called multiple times to perform a conversation warm-up operation.

[0204] The warm-up period includes: preparing for a human-machine voice conversation with the user and conducting the first round of conversation with the user. The warm-up period includes at least one of the following: initializing the model state of the first preset model and loading data related to the operation of the first preset model; the conversation inference process of the first preset model includes at least two processing stages (such as the aforementioned prefill stage and decode stage), and the at least two processing stages are distributed and executed on different work nodes.

[0205] In the above, the human-computer voice dialogue preparation period includes the period before the intelligent agent makes an outbound call. Specifically, it may include but is not limited to at least one of the following periods: a user screening period, a human-computer voice conversation establishment period (such as a human-computer voice conversation establishment period for screened users, or a human-computer voice conversation establishment period in response to a voice call request initiated by the user). And, the first round of dialogue period with the user includes but is not limited to: the period during which the user speaks for the first time in the first round of dialogue, or the period during which the intelligent agent outputs a reply for the first time in the first round of dialogue (such as outputting a preset reply such as "Hello, how can I help you?") and waits for the user's response.

[0206] In summary, this application solution has made engineering-side optimizations for the human-computer voice dialogue system, and the engineering-side optimization covers every module in the dialogue link (such as ASR, TTS, intelligent dialogue module, etc.). In addition to the above-mentioned engineering-side optimizations, many model optimizations have also been made on the algorithm side. Figure 4c The human-computer voice dialogue system architecture preferably adopted in this application is shown. With this architecture, the response time (RT) can be reduced to about 1.3-1.7s (see Figure 5 As shown in Figure 2, it can make human-computer dialogue smoother and more human-like.

[0207] Another embodiment of the present application provides a voice conversation method. The method includes the following steps:

[0208] 201. Determine a response generation method based on the conversation text corresponding to the user's voice conversation flow;

[0209] 202. While determining the conversation generation method, call a plurality of first preset models, and respectively use the plurality of first preset models to perform a streaming response on the conversation text;

[0210] 203. According to the conversation generation method, convert the conversation text segments output in segments by one of the first preset models during the streaming response process into conversation voice segments;

[0211] 204. Play the reply voice clip.

[0212] The above steps 201 to 202 and step 203 involving "selecting one of the first preset models to output the segmented reply text segments during the streaming response execution according to the reply generation method" are implemented through an intelligent agent.

[0213] For the specific implementation of the above 201 to 204, please refer to the relevant content in other embodiments.

[0214] In addition, the method provided in this embodiment may include other steps in addition to the steps given above. For details on the specific steps that may be included in this embodiment and the implementation of each step, please refer to the relevant content in other embodiments.

[0215] Another embodiment of the present application provides a voice conversation method. The method includes the following steps:

[0216] 301. Determine a response generation method based on the conversation text corresponding to the user's voice conversation flow;

[0217] 302. According to the conversation generation method, call one of the plurality of first preset models to perform a stream response on the conversation text.

[0218] 303. Play the reply voice segment based on the reply text segment output in the streaming response process.

[0219] The above steps 301 to 302 are implemented by an intelligent agent.

[0220] For the specific implementation of the above 301 to 203, please refer to the relevant content in other embodiments.

[0221] In addition, the method provided in this embodiment may include other steps in addition to the steps given above. For details on the specific steps that may be included in this embodiment and the implementation of each step, please refer to the relevant content in other embodiments.

[0222] Another embodiment of the present application provides a voice conversation method. The method includes the following steps:

[0223] 401. Receive the user's voice conversation flow;

[0224] 402. Based on the dialogue text corresponding to the voice dialogue flow, trigger a third preset model to perform a streaming response;

[0225] 403. Convert the reply text segments output in segments by the third preset model when executing the streaming response into reply voice segments;

[0226] 404. Play the reply voice clip;

[0227] The third preset model is deployed on a node in the rule engine, and the rule engine controls the human-computer voice dialogue according to the set dialogue service rule logic.

[0228] For the specific implementation of the above 401 to 404, please refer to the relevant content in other embodiments.

[0229] In addition, the method provided in this embodiment may include other steps in addition to the steps given above. For details on the specific steps that may be included in this embodiment and the implementation of each step, please refer to the relevant content in other embodiments.

[0230] Another embodiment of the present application provides a voice conversation method. The method includes the following steps:

[0231] 501. Using a second preset model, monitor the user's voice conversation flow;

[0232] 502. When the voice dialogue stream reaches the end of a sentence, trigger the agent to perform a streaming response based on the dialogue text corresponding to the voice dialogue stream;

[0233] 503. Play the reply voice segment based on the reply text segment output by the agent in the process of executing the streaming response.

[0234] The second preset model is a trained lightweight model. A lightweight model usually has fewer parameters.

[0235] For the specific implementation of the above 501 to 503, please refer to the relevant content in other embodiments.

[0236] In addition, the method provided in this embodiment may include other steps in addition to the steps given above. For details on the specific steps that may be included in this embodiment and the implementation of each step, please refer to the relevant content in other embodiments.

[0237] This application provides a voice dialogue solution that can be applied to various service scenarios, such as the intelligent question-and-answer scenario of an e-commerce platform, the intelligent question-and-answer scenario within an enterprise website, etc. For the intelligent question-and-answer scenario of an e-commerce platform, this application also provides a voice dialogue method, which includes the following steps:

[0238] 601. Obtaining a voice stream of a user's inquiry regarding a target product;

[0239] 602. Triggering the agent to execute a streaming response based on the text information corresponding to the query voice stream;

[0240] 603. Based on the reply text segments output by the agent in segments during the streaming response process, play the reply voice segments to respond to the inquiry voice stream.

[0241] Among them, the intelligent agent performs streaming response including: while determining the reply generation method based on the text information, using multiple first preset models to perform streaming response for the text information respectively; the reply text segment is output in segments by one of the first preset models during the execution of the streaming response.

[0242] The target product can be any product sold on the e-commerce platform, including clothes, shoes, suitcases, beauty products, instructional videos, software, etc. In addition, the text information corresponding to the query voice stream can also be called conversation text.

[0243] For the specific implementation of the above 601 to 603, please refer to the relevant content in other embodiments.

[0244] In addition, the method provided in this embodiment may include other steps in addition to the steps given above. For details on the specific steps that may be included in this embodiment and the implementation of each step, please refer to the relevant content in other embodiments.

[0245] It should be noted that in some of the processes described in the various embodiments and drawings of the present application, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to be different types.

[0246] This application also provides a voice dialogue system, specifically, an intelligent voice dialogue system with online human-computer voice dialogue function, such as a customer service system. Figure 1 The voice dialogue system may include a client 200 and a server 100. The client 200 is configured to collect the user's voice dialogue stream during a human-computer voice conversation and send the voice dialogue stream to the server 100. The voice dialogue stream may be, but is not limited to, a user's voice inquiry stream regarding a target product. The server is configured to execute the steps described in the various method embodiments of the present application. When the server plays a reply voice segment, it does so by sending the reply voice segment to the client, which then plays the segment.

[0247] Further, see Figure 1 And see again Figure 3 、 Figures 4a to 4c As shown, the server includes a voice duplex module and an intelligent dialogue module.

[0248] The Voice Duplex module includes a speech recognition unit (also known as an automatic speech recognition unit, ASR unit), which performs real-time text recognition on the user's voice conversation stream to generate conversation text. In other words, it converts the user's voice conversation stream into conversation text. The Voice Duplex module then generates a corresponding conversation request based on the conversation text obtained by the speech recognition unit and sends the conversation request to the intelligent conversation module.

[0249] The intelligent dialogue module can be understood as an intelligent robot, which can be constructed based on a third preset model (such as a language module) and deployed in a node in the rule engine. Or, in some other embodiments, such as Figure 4a to Figure 4c As shown, the intelligent dialogue module can also be constructed based on an intelligent agent. In the case where the intelligent dialogue module is constructed based on an intelligent agent, it can be a non-parallel multi-tasking processing architecture or a parallel multi-tasking processing architecture.

[0250] See Figure 4a As shown, when the intelligent dialogue module is designed as a non-parallel multi-tasking intelligent agent architecture, the intelligent dialogue module is used to: determine the reply generation method based on the dialogue text through the reasoner and construct the first prompt information; select a first preset model from multiple first preset models according to the reply generation method through the executor, use the selected first preset model to execute the streaming response based on the first prompt information, and store the reply text segments output by this first preset model in the first cache queue in the voice duplex module during the execution of the streaming response.

[0251] And, in e.g. Figure 4b or Figure 4c As shown, when the intelligent dialogue module is designed as a parallel multi-tasking intelligent agent architecture, the intelligent dialogue module is used to: while determining the reply generation method based on the dialogue text through the reasoner, it will also call multiple first preset models, and use the multiple first preset models to perform streaming responses for the dialogue text respectively, wherein the reply text segments output by different first preset models in the process of executing the streaming response will be stored in different second cache queues to wait for pushing, and the second cache queue is in the intelligent dialogue module; after determining the reply generation method, the reply text segment stored in the second cache queue corresponding to one of the preset models will be pushed to the voice duplex module through the executor to be stored in the first cache queue in the voice duplex module.

[0252] Furthermore, the voice duplex module also includes a text-to-speech synthesis unit (TTS unit) and a third cache queue. The TTS unit is configured to convert each response text segment stored in the first cache queue into a corresponding response voice segment, and then store the response voice segment in the third cache queue. Specifically, the TTS unit is configured to, upon detecting a response text segment stored in the first cache queue, convert the stored response text segment into a response voice segment, and then store the response voice segment in the third cache queue. Each response voice segment stored in the third cache queue is streamed to the client on a first-in-first-out basis and played by the client, thereby responding to the user's voice conversation stream.

[0253] For detailed descriptions of the specific forms of the client and server, please refer to the relevant content in other embodiments.

[0254] In summary, the present application constructs an end-to-end system based on an intelligent voice dialogue module (such as a third preset model (such as a language model) built based on an intelligent agent), and realizes intelligent voice dialogue. This can avoid the multi-step process of converting speech into text and then generating speech from text in traditional methods, thereby obtaining better dialogue effects and dialogue response time (RT).

[0255] The following describes the device embodiments corresponding to each method provided in this application.

[0256] An exemplary embodiment of the present application provides a voice dialogue device, which includes: a receiving module, a response module, a conversion module, and a playback module. The receiving module is used to receive a user's voice dialogue stream. The response module is used to perform a streaming response to the voice dialogue stream, wherein multiple reply text segments are generated in segments during the streaming response process, and the reply text segments are output in real time and stored in a first cache queue after being generated. The conversion module is used to, when detecting that the reply text segment is stored in the first cache queue, convert the stored reply text segment into a reply voice segment in real time. The playback module is used to play the reply voice segment.

[0257] Optionally, the above-mentioned response module, when used to perform a streaming response to the voice dialogue stream and store the reply text segments output in the streaming response process into the first cache queue, is specifically used to: perform text recognition on the voice dialogue stream to obtain a dialogue text; based on the dialogue text, trigger the intelligent agent to execute the following steps: while determining the reply generation method according to the dialogue text, respectively use multiple first preset models to execute the streaming response for the dialogue text; according to the reply generation method, push the reply text segments output in the streaming response process by one of the first preset models into the first cache queue for storage.

[0258] Furthermore, the response module is further configured to: store the reply text segments output by the plurality of first preset models in their respective corresponding second cache queues for waiting to be pushed;

[0259] Furthermore, the above-mentioned response module, when used to push the reply text segments output by one of the first preset models in the process of executing the streaming response into the first cache queue for storage according to the reply generation method, is specifically used to: push the reply text segments stored in the second cache queue corresponding to one of the first preset models into the first cache queue for storage according to the reply generation method; wherein the multiple first preset models include a retrieval enhancement model, and the retrieval enhancement model retrieves relevant information from the knowledge stored internally based on the conversation text to generate the reply text segments.

[0260] Optionally, when the above-mentioned response module is used to perform a streaming response to the voice dialogue stream, it is specifically used to: perform text recognition on the voice dialogue stream to obtain a dialogue text; based on the dialogue text, trigger the intelligent agent to execute the following steps: determine a reply generation method and construct a first prompt message according to the dialogue text; based on the reply generation method, select a first preset model from multiple first preset models; and use the selected first preset model to perform the streaming response based on the first prompt message.

[0261] The apparatus further includes: an assignment module configured to assign a unique session identifier to the human-machine voice conversation with the user; the voice conversation stream is the current input by the user during the human-machine voice conversation. Furthermore, the response module, when configured to trigger the agent to execute corresponding steps based on the conversation text, is configured to: generate a conversation request based on the session identifier and the conversation text; and send the conversation request to the agent to trigger the agent to execute corresponding steps; wherein the agent invokes the same first preset model for each conversation request carrying the same session identifier.

[0262] Furthermore, when the above-mentioned intelligent agent is used to call the first preset model, it is specifically used to: construct a second prompt information based on the dialogue text; input the second prompt information into the first preset model to trigger the first preset model to perform a streaming response; wherein, the second prompt information contains the contents of multiple roles, and the multiple roles include a management role, a user role, and a model role; the content of the management role includes: setting prompt text, and the setting prompt text contains constraint information on the first preset model; the content of the user role includes the dialogue text corresponding to at least one voice dialogue flow input by the user during the human-computer voice conversation; the content of the model role includes the reply text output by the first preset module.

[0263] Furthermore, the device also includes: an acquisition module and a calling module. The acquisition module is used to acquire a dialogue preheating strategy. The calling module is used to call the multiple first preset models multiple times to perform dialogue preheating operations when it is determined that the preheating time has been reached based on the dialogue preheating strategy. Wherein, reaching the preheating time includes: a period of preparation for human-computer voice dialogue with the user, and a period of the first round of dialogue with the user; the dialogue preheating operation includes at least one of the following: initializing the model state of the first preset model, loading data related to the operation of the first preset model; the conversation reasoning process of the first preset model includes at least two processing stages, and the at least two processing stages are distributed and executed on different working nodes.

[0264] Furthermore, the device also includes: a detection trigger module, which is used to use a second preset model to perform sentence end detection on the voice dialogue flow; when it is detected that the voice dialogue flow reaches the end of the sentence, based on the dialogue text corresponding to the voice dialogue flow, triggering the intelligent agent to execute corresponding steps.

[0265] Another exemplary embodiment of the present application further provides a voice dialogue device, which includes: a determination module, a response module, a conversion module, and a playback module. The determination module is used to determine a reply generation method based on the dialogue text corresponding to the user's voice dialogue flow. The response module is used to call multiple first preset models while determining the reply generation method, and respectively use the multiple first preset models to perform streaming responses for the dialogue text. The conversion module is used to convert the reply text segments output in segments by one of the first preset models during the streaming response process into reply voice segments according to the reply generation method. The playback module is used to play the reply voice segment.

[0266] Another exemplary embodiment of the present application further provides a voice dialogue device, comprising: a determination module, a response module, and a playback module. The determination module is configured to determine a response generation method based on a conversation text corresponding to a user's voice dialogue flow. The response module is configured to invoke one of a plurality of first preset models to perform a streaming response to the conversation text based on the response generation method. The playback module is configured to play a response voice segment based on the segmented response text segments output during the streaming response process.

[0267] Another exemplary embodiment of the present application further provides a voice dialogue device, which includes: a receiving module, a triggering module, a conversion module, and a playing module. The receiving module is used to receive the user's voice dialogue stream. The triggering module is used to trigger the third preset model to perform a streaming response based on the dialogue text corresponding to the voice dialogue stream. The conversion module is used to convert the reply text segments output in segments by the third preset model when performing the streaming response into reply voice segments. The playing module is used to play the reply voice segments. The third preset model is deployed on a node in the rule engine, and the rule engine controls the human-computer voice dialogue according to the set dialogue service rule logic.

[0268] Another exemplary embodiment of the present application further provides a voice dialogue device, comprising: a monitoring module, a triggering module, and a playback module. The monitoring module is configured to monitor the user's voice dialogue flow using a second preset model. The triggering module is configured to trigger an intelligent agent to perform a streaming response based on the dialogue text corresponding to the voice dialogue flow when the voice dialogue flow reaches the end of a sentence. The playback module is configured to play a response voice segment based on the response text segment output by the intelligent agent during the streaming response. The second preset model is a trained lightweight model.

[0269] Another exemplary embodiment of the present application also provides a voice dialogue device, which includes: an acquisition module, a trigger module, and a playback module. The acquisition module is used to acquire the inquiry voice stream input by the user for the target product. The trigger module triggers the intelligent agent to perform a streaming response based on the text information corresponding to the inquiry voice stream. The playback module is used to play the reply voice segment based on the reply text segment output by the intelligent agent in the process of performing the streaming response, so as to answer the inquiry voice stream. Wherein, the intelligent agent performs the streaming response including: while determining the reply generation method based on the text information, respectively using multiple first preset models to perform a streaming response for the text information; the reply text segment is output in segments by one of the first preset models in the process of performing the streaming response.

[0270] What needs to be explained here about the above-mentioned devices is that the above-mentioned devices can implement the technical solutions described in the above-mentioned corresponding method embodiments. The specific implementation principles of the above-mentioned modules or units can be found in the relevant contents of the above-mentioned corresponding method embodiments, and will not be described in detail here.

[0271] Figure 6 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 6 As shown, the electronic device includes: a memory 71 and a processor 72.

[0272] The memory 71 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, data structures, contact data, phone book data, messages, images, videos, etc.

[0273] The processor 72 is coupled to the memory 71 and is used to execute the computer program in the memory 71 to implement the steps in the various method embodiments provided in this application.

[0274] Further, if Figure 6 As shown, the electronic device also includes: a communication component 73, a display 74, a power component 75, an audio component 76 and other components. Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 6 The components in the dotted box are optional components, not mandatory components, and the specific components may depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT device, or a server device such as a conventional server, a cloud server or a server array, or an integrated device of a terminal device and a server device. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 6 If the electronic device of this embodiment is implemented as a conventional server, cloud server or server array and other server-side devices, it may not include Figure 6 Components within the dotted box.

[0275] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0276] The communication component is configured to facilitate wired or wireless communication between the device in which the communication component resides and other devices. The device in which the communication component resides can access a wireless network based on a communication standard, such as a 2G, 3G, 4G / LTE, 5G, or other mobile communication network, or a combination thereof. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0277] The above-mentioned display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0278] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0279] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0280] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above method embodiment. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassette, tape disk storage or other magnetic storage device or any other non-transmission medium.

[0281] Accordingly, an embodiment of the present application further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the processor is enabled to implement the steps in the above-mentioned method embodiment. It should be understood that each process or a combination of multiple processes in the above-mentioned method flow can be implemented by a computer program or instruction. In addition, these computer programs or instructions can be applied to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device, so that the processor of the general-purpose computer, the special-purpose computer, the embedded processor or other programmable data processing device can be implemented as a device for implementing the corresponding functions in the above-mentioned method embodiment.

[0282] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0283] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A voice dialogue method, characterized in that: include: Receive the user's voice conversation stream; Performing a streaming response on the voice dialogue stream, wherein a plurality of conversation text segments are generated in segments during the streaming response process, and the conversation text segments are output in real time after being generated so as to be stored in a first cache queue; When detecting that the reply text segment is stored in the first cache queue, converting the stored reply text segment into a reply voice segment in real time; Play the reply voice clip.

2. The method according to claim 1, characterized in that Providing a streaming response to the voice dialogue stream, including: Performing text recognition on the voice dialogue stream to obtain a dialogue text; Based on the conversation text, the intelligent agent is triggered to execute the following steps: while determining the reply generation method according to the conversation text, the streaming response is executed for the conversation text using multiple first preset models respectively; according to the reply generation method, multiple reply text segments output in segments by one of the first preset models during the execution of the streaming response are pushed to the first cache queue in real time for storage.

3. The method according to claim 2, characterized in that Also includes: The plurality of reply text segments output by the plurality of first preset models are respectively stored in real time into the respective corresponding second cache queues to wait for being pushed; And, according to the conversation generation method, pushing multiple conversation text segments output by one of the first preset models in a streaming response process to the first cache queue in real time for storage, including: According to the conversation generation method, the conversation text segment stored in the second cache queue corresponding to one of the first preset models is pushed to the first cache queue for storage in real time; Among them, the multiple first preset models include a retrieval enhancement model, which retrieves relevant information from the knowledge stored internally based on the dialogue text to generate the reply text segment.

4. The method according to claim 1, wherein Providing a streaming response to the voice dialogue stream, including: Performing text recognition on the voice dialogue stream to obtain a dialogue text; Based on the dialogue text, the intelligent agent is triggered to execute the following steps: according to the dialogue text, a reply generation method is determined and a first prompt message is constructed; based on the reply generation method, a first preset model is selected from a plurality of first preset models; and using the selected first preset model, the streaming response is executed based on the first prompt message.

5. The method according to any one of claims 2 to 4, characterized in that Also includes: Assigning a unique session identifier to the human-computer voice conversation with the user; The voice dialogue flow is the current input of the user during the human-computer voice conversation; And, based on the dialogue text, triggering the agent to execute corresponding steps, including: generating a conversation request based on the conversation identifier and the conversation text; Sending the dialogue request to the agent to trigger the agent to execute corresponding steps; The intelligent agent calls the same first preset model for the dialogue request carrying the same session identifier.

6. The method according to claim 5, characterized in that Calling the first preset model includes: Constructing second prompt information according to the dialogue text; Inputting the second prompt information into the first preset model to trigger the first preset model to perform a streaming response; Among them, the second prompt information contains the contents of multiple roles, and the multiple roles include a control role, a user role, and a model role; the content of the control role includes: setting prompt text, and the setting prompt text contains constraint information for the first preset model; the content of the user role includes the dialogue text corresponding to at least one voice dialogue flow input by the user during the human-computer voice conversation; the content of the model role includes the reply text output by the first preset module.

7. The method according to any one of claims 2 to 4, characterized in that Also includes: Get conversation warm-up strategies; When it is determined based on the conversation warm-up strategy that the warm-up time has arrived, calling the plurality of first preset models multiple times to perform the conversation warm-up operation; Among them, reaching the preheating time includes: a preparation period for human-computer voice dialogue with the user, and a period for the first round of dialogue with the user; the dialogue preheating operation includes at least one of the following: initializing the model state of the first preset model, loading data related to the operation of the first preset model; the conversation reasoning process of the first preset model includes at least two processing stages, and the at least two processing stages are distributed and executed on different working nodes.

8. The method according to any one of claims 2 to 4, characterized in that Also includes: Performing sentence end detection on the voice dialogue flow using a second preset model; When it is detected that the voice dialogue flow reaches the end of a sentence, the intelligent agent is triggered to execute corresponding steps based on the dialogue text corresponding to the voice dialogue flow.

9. A voice dialogue method, characterized in that: include: Determine the method for generating a response based on the conversation text corresponding to the user's voice conversation flow; While determining the conversation generation method, calling a plurality of first preset models, and respectively using the plurality of first preset models to perform streaming responses for the conversation text; According to the reply generation method, converting the reply text segments output by one of the first preset models in the process of executing the streaming response into reply voice segments; Play the reply voice clip.

10. A voice dialogue method, characterized in that: include: Determine the method for generating a response based on the conversation text corresponding to the user's voice conversation flow; According to the conversation generation method, calling one of the plurality of first preset models to perform a streaming response for the conversation text; Play the response voice segment based on the segmented response text segment output during the streaming response process.

11. A voice dialogue method, characterized in that: include: Receive the user's voice conversation stream; Based on the dialogue text corresponding to the voice dialogue flow, triggering a third preset model to perform a streaming response; Converting the reply text segments output in segments by the third preset model when executing the streaming response into reply voice segments; Playing the reply voice clip; The third preset model is deployed on a node in the rule engine, and the rule engine controls the human-computer voice dialogue according to the set dialogue service rule logic.

12. A voice dialogue method, characterized in that: include: Using the second preset model, monitoring the user's voice conversation flow; When the voice dialogue flow is detected to have reached the end of a sentence, the agent is triggered to perform a streaming response based on the dialogue text corresponding to the voice dialogue flow; Playing a reply voice segment based on the reply text segment output by the agent in the process of executing the streaming response; The second preset model is a trained lightweight model.

13. A voice dialogue method, characterized in that: include: Obtain the user's voice query stream for the target product; Based on the text information corresponding to the query voice stream, trigger the intelligent agent to perform a streaming response; Playing a reply voice segment based on the reply text segment output by the agent in the process of executing the streaming response, so as to respond to the inquiry voice stream; Among them, the intelligent agent performs streaming response including: while determining the reply generation method based on the text information, using multiple first preset models to perform streaming response for the text information respectively; the reply text segment is output in segments by one of the first preset models during the execution of the streaming response.

14. A voice dialogue system, characterized in that: include: The client is used to collect the user's voice dialogue flow during the human-computer voice conversation and send the voice dialogue flow to the server; The server is configured to execute the steps of the voice dialogue method according to any one of claims 1 to 13.

15. An electronic device, characterized in that: including a memory and a processor; wherein, The memory is used to store executable instructions; The processor implements the steps in the voice dialogue method according to any one of claims 1 to 13 by running the executable instructions.

16. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which can implement the steps of the voice dialogue method according to any one of claims 1 to 13 when executed.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the voice dialogue method according to any one of claims 1 to 13 are implemented.

Images