Real-time voice conversation method, device and equipment and storage medium

Abstract

The invention discloses a real-time voice dialogue method and device, equipment and a storage medium, and relates to the field of voice dialogues, and the method comprises the steps: obtaining an initial audio clip inputted by a tax payment user side, and determining a dynamic amplitude threshold value based on the initial audio clip; if the audio amplitude of a preset tail segment in the initial audio segment is not greater than a dynamic amplitude threshold value, determining an occupation proportion of the audio segment of which the audio amplitude is greater than the dynamic amplitude threshold value in the initial audio segment; if the occupation proportion is greater than a preset proportion threshold, determining a voice endpoint in the initial audio clip, and processing the initial audio clip based on the voice endpoint to obtain a first target audio clip; converting the first target audio clip into a text clip, and determining a user intention based on the text clip; and determining a corresponding second target audio clip based on the user intention, and issuing the second target audio clip to the tax payment user side to complete a real-time voice conversation. According to the invention, the accuracy of real-time voice conversation can be improved.

Description

Technical Field

[0001] This invention relates to the field of voice dialogue, and in particular to a real-time voice dialogue method, apparatus, device, and storage medium. Background Technology

[0002] In the field of taxpayer services, remote taxpayer service personnel have long been responsible for answering a massive number of inquiry calls. As the workload increases, labor costs rise sharply, and response times are delayed during peak periods, making it difficult to meet the demand for efficient services. Real-time voice dialogue systems have thus become a solution, but existing technologies have obvious limitations.

[0003] Current mainstream systems suffer from significant shortcomings: Architecturally, ASR (Automatic Speech Recognition) and NLU (Natural Language Understanding) modules are often tightly coupled. While aiming for low latency, this leads to module failures that can cause complete process blockages, and performance cannot be independently scaled. Speech detection relies on single features such as short-term energy, and fixed thresholds lack adaptability, making detection prone to failure. Semantic segmentation relies entirely on VAD (Voice Activity Detection) silence detection, often misinterpreting user pauses as sentence endings, causing semantic fragmentation and degrading the service experience. Furthermore, conversation content is limited to pre-built templates, hindering personalized interaction. In complex noisy environments, ASR recognition accuracy is low, and it cannot dynamically collect samples based on voice timbre to filter out noise interference, further impacting service quality. These shortcomings make existing systems ill-suited for the high concurrency and complexity requirements of data service scenarios, failing to effectively reduce the workload of data service personnel.

[0004] Therefore, improving the accuracy and completeness of real-time voice dialogue when answering taxpayers' inquiries is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a real-time voice dialogue method, apparatus, device, and storage medium, which can improve the accuracy and completeness of real-time voice dialogue when answering taxpayers' inquiries. The specific solution is as follows:

[0006] Firstly, this application provides a real-time voice dialogue method, including:

[0007] Obtain an initial audio segment input by the taxpayer user, and determine a dynamic amplitude threshold based on the initial audio segment; the initial audio segment is an audio segment containing a taxpayer consultation question;

[0008] If the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold, then in the initial audio segment, the proportion of audio segments with audio amplitude greater than the dynamic amplitude threshold is determined.

[0009] If the occupancy ratio is greater than a preset ratio threshold, then the speech endpoints in the initial audio segment are determined using speech activity detection technology, and the initial audio segment is segmented accordingly based on the speech endpoints to obtain the first target audio segment; the speech endpoints include a start endpoint that identifies the start position of the speech segment and an end endpoint that identifies the end position of the speech segment;

[0010] The first target audio segment is converted into a text segment using automatic speech recognition technology, and the user intent of the taxpayer terminal is determined based on a preset large model and the text segment.

[0011] Based on the user intent, the preset WebSocket technology, and the preset distributed caching architecture, a corresponding second target audio segment is determined, and the second target audio segment is sent to the taxpayer user terminal to play the second target audio segment in the taxpayer user terminal to complete real-time voice dialogue.

[0012] Optionally, obtaining the initial audio segment input by the taxpayer user and determining the dynamic amplitude threshold based on the initial audio segment includes:

[0013] The initial audio segment input by the taxpayer's client is obtained through an audio acquisition device and based on a preset WebSocket technology;

[0014] The percentile, peak value, and rate of change of the audio energy of the initial audio segment are determined, and the sensitivity coefficient is determined based on the degree of abrupt change in the audio energy.

[0015] The dynamic amplitude threshold is determined based on the preset weight value, the percentile, the peak value, the rate of change, and the sensitivity coefficient.

[0016] Optionally, if the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold, the method further includes:

[0017] Construct a sliding time window of preset duration; the preset duration is longer than the duration corresponding to the preset tail segment.

[0018] Based on the sliding time window, detect whether there is an audio segment in the current initial audio segment whose audio amplitude is greater than the dynamic amplitude threshold, and obtain the corresponding detection result;

[0019] If the detection result indicates that such a result exists, then proceed to the step of obtaining the initial audio segment input by the taxpayer user.

[0020] Optionally, before converting the first target audio segment into a text segment using automatic speech recognition technology, the method further includes:

[0021] Determine the audio sampling rate of the first target audio segment and the current batch size;

[0022] If both the audio sampling rate and the batch size meet the preset audio recognition conditions, then the first target audio segment is converted into a text segment using automatic speech recognition technology.

[0023] Optionally, the step of determining the corresponding second target audio segment based on the user intent, preset WebSocket technology, and preset distributed caching architecture, and sending the second target audio segment to the taxpayer user terminal, includes:

[0024] When the user's intention is for human consultation, the current dialogue state in the preset distributed caching architecture is set to human consultation state, and the current session state is read using preset WebSocket technology to retrieve the corresponding second target audio segment from the preset audio library and send the second target audio segment to the taxpayer user terminal.

[0025] When the user's intention is to end the conversation, the current conversation state in the preset distributed caching architecture is set to the end conversation state, and the current session state is read using the preset WebSocket technology to retrieve the corresponding second target audio segment from the preset audio library, and the second target audio segment is sent to the taxpayer user terminal;

[0026] When the user's intent is to inquire about a problem, a hybrid recall strategy is used to retrieve a first target problem corresponding to the user's intent from a preset problem knowledge base. The current dialogue state in the preset distributed caching architecture is set to a problem inquiry state. The current session state is read using the preset WebSocket technology to retrieve a second target audio segment corresponding to the first target problem from the preset audio library. The second target audio segment is then sent to the taxpayer's client. The hybrid recall strategy includes a similarity retrieval tool and a re-ranking technique. The first target problem is a problem that meets the preset problem completeness condition.

[0027] Optionally, the step of determining the corresponding second target audio segment based on the user intent, preset WebSocket technology, and preset distributed caching architecture, and sending the second target audio segment to the taxpayer user terminal, further includes:

[0028] When the user intent indicates that there is a second target question on the taxpayer user terminal, the current dialogue state in the preset distributed caching architecture is set to a question supplementation state. The current session state is read using the preset WebSocket technology to retrieve the second target audio segment corresponding to the second target question from the preset audio library. The second target audio segment is then sent to the taxpayer user terminal. After the second target audio segment is played on the taxpayer user terminal, the process jumps to the step of obtaining the initial audio segment input by the taxpayer user terminal. The second target question is a question that does not meet the preset question integrity condition.

[0029] Optionally, the audio segments in the preset audio library are audio segments in a preset encoding format obtained by converting a preset dialogue text template offline.

[0030] Secondly, this application provides a real-time voice dialogue device, comprising:

[0031] The threshold determination module is used to obtain an initial audio segment input by the taxpayer user terminal and determine a dynamic amplitude threshold based on the initial audio segment; the initial audio segment is an audio segment containing a taxpayer consultation question;

[0032] The proportion determination module is used to determine the proportion of audio segments with audio amplitudes greater than the dynamic amplitude threshold in the initial audio segment if the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold.

[0033] The segment determination module is used to determine the speech endpoints in the initial audio segment using speech activity detection technology if the occupancy ratio is greater than a preset ratio threshold, and to perform corresponding segmentation processing on the initial audio segment based on the speech endpoints to obtain a first target audio segment; the speech endpoints include a start endpoint that identifies the start position of the speech segment and an end endpoint that identifies the end position of the speech segment;

[0034] The intent determination module is used to convert the first target audio segment into a text segment using automatic speech recognition technology, and to determine the user intent of the taxpayer terminal based on a preset large model and the text segment.

[0035] The voice dialogue module is used to determine the corresponding second target audio segment based on the user intent, preset WebSocket technology and preset distributed caching architecture, and send the second target audio segment to the taxpayer user terminal to play the second target audio segment on the taxpayer user terminal to complete real-time voice dialogue.

[0036] Thirdly, this application provides an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the aforementioned real-time voice dialogue method.

[0039] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned real-time voice dialogue method.

[0040] In this application, an initial audio segment input by a taxpayer user is obtained, and a dynamic amplitude threshold is determined based on the initial audio segment. The initial audio segment is an audio segment containing a taxpayer consultation question. If the audio amplitude of a preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold, the proportion of audio segments with audio amplitudes greater than the dynamic amplitude threshold in the initial audio segment is determined. If the proportion is greater than a preset proportion threshold, speech activity detection technology is used to determine the speech endpoints in the initial audio segment, and the initial audio segment is segmented accordingly based on the speech endpoints to obtain a first target audio segment. The speech endpoints include a start endpoint indicating the start position of the speech segment and an end endpoint indicating the end position. Automatic speech recognition technology is used to convert the first target audio segment into a text segment, and the user intent of the taxpayer user is determined based on a preset large model and the text segment. Based on the user intent, preset WebSocket technology, and preset distributed caching architecture, a corresponding second target audio segment is determined, and the second target audio segment is sent to the taxpayer user to play the second target audio segment on the taxpayer user to complete real-time voice dialogue. As can be seen from the above, this application first obtains the initial audio segment input by the taxpayer's client, and then sets a dynamic amplitude threshold based on the initial audio segment. If the audio amplitude of the preset tail segment in the initial audio segment does not exceed the dynamic amplitude threshold, the proportion of segments in the initial audio segment whose audio amplitude is greater than the dynamic amplitude threshold is calculated. When this proportion exceeds a preset percentage threshold, speech endpoints in the initial audio segment are located using speech activity detection technology, and the initial audio segment is segmented based on these speech endpoints to obtain the first target audio segment. Subsequently, automatic speech recognition technology is used to convert the first target audio segment into a text segment, and the user intent of the taxpayer's client is identified by combining a preset large model and the text segment. Finally, based on the user intent, preset WebSocket technology, and preset distributed caching architecture, the corresponding second target audio segment is determined and sent to the taxpayer's client, which plays the segment to complete the real-time voice dialogue. In this way, this application can improve the accuracy and completeness of real-time voice dialogue when answering taxpayers' consultation calls, significantly improving the user experience. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a flowchart of a real-time voice dialogue method disclosed in this application;

[0043] Figure 2 This is a flowchart of a specific real-time voice dialogue method disclosed in this application;

[0044] Figure 3 This is a schematic diagram of a process disclosed in this application, from dynamic amplitude threshold calculation to user intent recognition;

[0045] Figure 4 This is a schematic diagram of a process disclosed in this application for determining whether there is continuous silence by using a sliding time window;

[0046] Figure 5 This is a schematic diagram of a process for determining a second target audio segment based on user intent, as disclosed in this application;

[0047] Figure 6 This is a schematic diagram of an audio segment in a preset audio library disclosed in this application;

[0048] Figure 7 This is a schematic diagram of the structure of a real-time voice dialogue device disclosed in this application;

[0049] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Current mainstream real-time voice dialogue systems suffer from significant drawbacks: Architecturally, ASR and NLU modules are often tightly coupled, which, while aiming for low latency, makes module failures prone to causing system-wide blockages, and performance cannot be independently scaled; speech detection relies on single features such as short-term energy, and fixed thresholds lack adaptability, making detection prone to failure; semantic segmentation relies entirely on VAD (Voice-Only Detection) silence judgment, often misinterpreting user pauses as sentence endings, causing semantic fragmentation and degrading the service experience. Furthermore, conversation content is limited to pre-built templates, failing to achieve personalized interaction; in complex noisy environments, ASR recognition accuracy is low, and it cannot dynamically collect samples based on voice timbre to filter out noise interference, further impacting service quality. These shortcomings make existing systems ill-suited for the high concurrency and complexity requirements of taxpayer service scenarios, and fail to effectively reduce the workload of taxpayer service personnel. Therefore, this application provides a real-time voice dialogue method, apparatus, device, and storage medium that can improve the accuracy and completeness of real-time voice dialogue when answering taxpayer inquiries.

[0052] See Figure 1 As shown, an embodiment of the present invention discloses a real-time voice dialogue method, including:

[0053] Step S11: Obtain the initial audio segment input by the taxpayer user terminal, and determine the dynamic amplitude threshold based on the initial audio segment; the initial audio segment is an audio segment containing taxpayer consultation questions.

[0054] In this embodiment, an initial audio segment transmitted from the taxpayer's client is acquired using an audio acquisition device based on a preset WebSocket technology. After acquiring the initial audio segment, the percentile, peak value, and rate of change of the audio energy of the initial audio segment are determined, and a sensitivity coefficient is determined based on the degree of abrupt change in the audio energy. Furthermore, this embodiment also determines the sensitivity coefficient based on the degree of abrupt change in the audio energy; this coefficient is used to adjust the sensitivity of subsequent threshold calculations. Finally, a dynamic amplitude threshold is determined based on a preset weight value, the percentile, the peak value, the rate of change, and the sensitivity coefficient. This threshold will be used for audio processing and feature extraction in subsequent steps, thereby providing basic support for the real-time identification and response to taxpayer consultation questions.

[0055] Step S12: If the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold, then in the initial audio segment, determine the proportion of audio segments with audio amplitude greater than the dynamic amplitude threshold.

[0056] In this embodiment, after obtaining the dynamic amplitude threshold, the audio amplitude of the preset tail segment of the initial audio segment is analyzed. If the audio amplitude of the preset tail segment never exceeds the dynamic amplitude threshold, it is determined that the user may have completed the input of the current statement.

[0057] To verify the continuity of speech activity, within the complete initial audio segment, the total duration of all audio segments with amplitudes exceeding the dynamic amplitude threshold must be counted, and their proportion of the total duration of the initial audio segment must be calculated, yielding the occupancy ratio. This occupancy ratio effectively reflects the proportion of valid speech components in the audio segment, providing a quantitative basis for subsequent judgments on whether the user is in a sentence interval or has finished speaking. By calculating the occupancy ratio, misjudgments caused by brief silences or environmental noise can be avoided, thereby improving the accuracy of speech endpoint detection and ensuring that the system can accurately capture the complete inquiry questions of taxpayers.

[0058] Furthermore, to avoid the system mistakenly assuming the user has stopped speaking and automatically processing audio content that doesn't fully express the taxpayer's intent simply because of a brief pause or thought after the taxpayer has finished speaking, a continuously muted sliding window will be activated for a secondary assessment. Specifically, a sliding time window of preset duration is constructed; the preset duration is longer than the duration corresponding to the preset end segment; based on the sliding time window, it is detected whether there is an audio segment in the current initial audio segment with an audio amplitude greater than the dynamic amplitude threshold, and the corresponding detection result is obtained; if the detection result indicates that such a segment exists, the process jumps to the step of obtaining the initial audio segment input by the taxpayer's end.

[0059] Step S13: If the occupancy ratio is greater than a preset ratio threshold, the speech endpoints in the initial audio segment are determined using speech activity detection technology, and the initial audio segment is segmented accordingly based on the speech endpoints to obtain the first target audio segment; the speech endpoints include a start endpoint that identifies the start position of the speech segment and an end endpoint that identifies the end position of the speech segment.

[0060] In this embodiment, when the calculated occupancy ratio is greater than a preset threshold, it indicates that the initial audio segment contains a sufficient amount of valid speech data, thereby triggering the speech activity detection process. This technology analyzes features such as the short-time energy and zero-crossing rate of the audio signal to locate the actual start and end positions of the user's speech, i.e., the start and end endpoints. Subsequently, based on these two identified endpoints, the initial audio segment is segmented, removing any leading silence segments and trailing silence segments that may exist at both ends, thereby extracting the first target audio segment containing the core speech content. This operation effectively eliminates the interference of invalid audio segments on subsequent processing, improving the processing efficiency and accuracy of the speech recognition engine.

[0061] Step S14: Use automatic speech recognition technology to convert the first target audio segment into a text segment, and determine the user intent of the taxpayer terminal based on the preset large model and the text segment.

[0062] In this embodiment, before performing speech recognition conversion, it is necessary to confirm whether the audio sampling rate of the first target audio segment meets the input requirements of automatic speech recognition. Specifically, the audio sampling rate of the first target audio segment and the current batch size are determined; if both the audio sampling rate and the batch size meet the preset audio recognition conditions, then the first target audio segment is converted into a text segment using automatic speech recognition technology, that is, the automatic speech recognition processing flow is started.

[0063] The process involves inputting the first target audio segment into the recognition engine, which converts the contained speech signals into corresponding text segments. These text segments are then input into a pre-defined large language model for deep semantic analysis and understanding. This large model parses the context, keywords, and semantic structure of the text segments to infer and output the user intent implied in the taxpayer's inquiry, laying the foundation for generating subsequent responses.

[0064] Step S15: Based on the user intent, the preset WebSocket technology, and the preset distributed caching architecture, determine the corresponding second target audio segment, and send the second target audio segment to the taxpayer user terminal to play the second target audio segment on the taxpayer user terminal to complete real-time voice dialogue.

[0065] In this embodiment, the system uses preset WebSocket technology and preset distributed caching architecture in conjunction with the determined user intent to generate and transmit a response.

[0066] When the user's intent is for human consultation, the current dialogue state in the preset distributed caching architecture is set to human consultation state. A preset WebSocket technology is used to read the current session state to retrieve the corresponding second target audio segment from a preset audio library, and then the second target audio segment is sent to the taxpayer's client. Specifically, the audio segments in the preset audio library are audio segments in a preset encoding format obtained by converting a preset dialogue text template offline.

[0067] When the user intends to end the conversation, the current conversation state in the preset distributed caching architecture is set to the end conversation state, and the current session state is read using the preset WebSocket technology to retrieve the corresponding second target audio segment from the preset audio library, and the second target audio segment is sent to the taxpayer user terminal.

[0068] When the user's intent is to inquire about a problem, a hybrid recall strategy is used to retrieve a first target problem corresponding to the user's intent from a preset problem knowledge base. The current dialogue state in the preset distributed caching architecture is set to a problem inquiry state. The current session state is read using the preset WebSocket technology to retrieve a second target audio segment corresponding to the first target problem from the preset audio library. The second target audio segment is then sent to the taxpayer's client. The hybrid recall strategy includes a similarity retrieval tool and a re-ranking technique. The first target problem is a problem that meets the preset problem completeness condition.

[0069] Furthermore, in this embodiment, when the user's intent indicates that the taxpayer's client has a second target problem, the current dialogue state in the preset distributed caching architecture is set to a problem supplementation state, and the current session state is read using the preset WebSocket technology to retrieve the second target audio segment corresponding to the second target problem from the preset audio library. The second target audio segment is then sent to the taxpayer's client, and after the second target audio segment is played on the taxpayer's client, the process jumps to the step of obtaining the initial audio segment input by the taxpayer's client. The second target problem is a problem that does not meet the preset problem integrity condition.

[0070] Through this interactive guidance, the user's true needs are gradually clarified until the problem becomes complete and solvable, thereby building an intelligent real-time voice dialogue system that can autonomously handle incomplete inquiries and has the ability to continuously ask follow-up questions.

[0071] As can be seen from the above, this application first obtains the initial audio segment input by the taxpayer's client, and then sets a dynamic amplitude threshold based on the initial audio segment. If the audio amplitude of the preset tail segment in the initial audio segment does not exceed the dynamic amplitude threshold, the proportion of segments in the initial audio segment whose audio amplitude is greater than the dynamic amplitude threshold is calculated. When this proportion exceeds a preset percentage threshold, speech endpoints in the initial audio segment are located using speech activity detection technology, and the initial audio segment is segmented based on these speech endpoints to obtain the first target audio segment. Subsequently, automatic speech recognition technology is used to convert the first target audio segment into a text segment, and the user intent of the taxpayer's client is identified by combining a preset large model and the text segment. Finally, based on the user intent, preset WebSocket technology, and preset distributed caching architecture, the corresponding second target audio segment is determined and sent to the taxpayer's client, which plays the segment to complete the real-time voice dialogue. In this way, this application can improve the accuracy and completeness of real-time voice dialogue when answering taxpayers' consultation calls, significantly improving the user experience.

[0072] The following is combined with Figure 2The schematic diagram shown illustrates the technical solution of the embodiments of this application in detail.

[0073] Specifically, for Figure 2 In the context of "audio streaming," the primary responsibility of the audio acquisition device is to capture taxpayers' voice input and process this audio content in real time. This device divides continuous audio data into small segments every 100 milliseconds and continuously pushes these segments to the model processing service ws (i.e., WebSocket). Upon receiving this audio data, the model processing service ws performs further analysis and processing to extract useful information.

[0074] for Figure 2 The "multi-strategy dynamic VAD" module specifically pushes the acquired audio content to the multi-strategy dynamic VAD module. The multi-strategy threshold calculation includes historical statistics, percentiles, peak values, and energy change rates, and weighted averages are used to fuse the results of different strategies. The sensitivity coefficient is adaptively adjusted based on energy mutations, and the audio threshold is dynamically adjusted according to different input audio content. For audio segments with an audio length exceeding 500ms but below the adaptive dynamic threshold, the proportion of sampling points in this audio segment that exceed the dynamic threshold is calculated. If the proportion of sampling points is higher than 1%, a VAD point is detected and an audio segment is cut for subsequent ASR model recognition processing.

[0075] for Figure 2 In the context of "ASR inference audio text," specifically, during the audio text content recognition process, the first step is to ensure that the selected audio segments meet the sampling point ratio requirements outlined in the previous steps. Once the audio segments meet these criteria, the next step considers the audio sampling rate, which refers to the number of times the sound signal is sampled per second, directly affecting the clarity and quality of the sound. Furthermore, batch size is also a crucial factor, determining the amount of audio data the system processes at a time. By comprehensively considering these factors, ASR can be effectively performed, converting audio content into text form for further analysis and processing.

[0076] for Figure 2The "LLM-assisted semantic diagnosis" step utilizes a semantic understanding model built with the Qwen large model to perform an in-depth assessment of the speech integrity of the audio text extracted through speech recognition technology. This model goes beyond traditional silence detection techniques, leveraging the advanced capabilities of the large model to determine whether the semantic content is fully and accurately expressed. In this way, the model can effectively distinguish between the subtle differences between a "true sentence ending" and "mere pauses in tone." This improvement avoids unnecessary subsequent processing caused by multiple pauses for thought when inputting audio information, significantly reducing the response time of the real-time conversation system. This not only improves processing efficiency but also significantly enhances the overall user experience. With the introduction of the large model's semantic judgment function, the system now only initiates subsequent processing when the user clearly and completely expresses their semantic intent.

[0077] Furthermore, referring to Figure 3 The steps from "multi-strategy dynamic VAD" to "LLM-assisted semantic diagnosis" will be explained in detail.

[0078] Taxpayers input audio segments at 100-millisecond intervals via their input devices, which are then accumulated and processed through a stable long-lived WebSocket connection. The system employs a fusion of multiple strategies to dynamically calculate a threshold for each audio segment, determining whether the final 500-millisecond segment is silent. If, within this final 500-millisecond segment, all audio amplitudes fall below the calculated dynamic threshold, the system infers that the user may have completed the first paragraph. At this point, refer to... Figure 4 The system will initiate a continuous silent sliding window for secondary judgment. Specifically, this sliding window is 2000 milliseconds long. Within this time window, if any audio amplitude exceeding the dynamic threshold is detected in the audio segment, the system will restart the timing. This design is to prevent the system from mistakenly assuming the user has stopped speaking and automatically processing audio content that does not fully express the taxpayer's intention simply because of a brief pause or thought after the taxpayer has finished speaking. This processing method effectively increases the system's response time and ensures an accurate understanding of the taxpayer's intention.

[0079] Next, the selected audio segment will be analyzed to determine the proportion of sampling points exceeding the dynamic threshold. Specifically, the percentage of these sampling points out of the total number of sampling points will be calculated. Once this proportion exceeds 1%, a crucial step will be performed: VAD (Voice-Activated Disturbance) points will be detected, and the audio will be segmented accordingly. The segmented audio segment will then be fed into the ASR (Automatic Speech Recognition) model for processing, aiming to convert the speech content in the audio into text. The resulting text content will then be submitted to the language understanding model to determine its semantic completeness. If the LLM (Large Language Model) determines that the text content can fully express the taxpayer's intent or question, it will output a flag "True," indicating successful processing. If the LLM believes that the text content cannot fully express the taxpayer's semantics, it will output a flag "False," meaning that further audio input from the taxpayer is needed for further processing.

[0080] for Figure 2 In the "intent recognition" step mentioned above, the ASR module identifies audio text with complete taxpayer semantics. A large model is then used to determine the intent and route the message. There are three main routing paths: transfer to human assistance, end the session, and ask a question. When the taxpayer's intent is identified as transfer to human assistance, the "_human" field in the Redis service cluster (i.e., the default distributed cache architecture) is updated to 1. At this time, the synchronized TTS (Text-to-Speech) output websocket service reads the "_human" field content of 1 from the Redis service cluster and then retrieves the TTS audio (i.e.,...)... Figure 5 The system retrieves the audio from the Redis service cluster (for TTS), retrieves the TTS audio message for transferring to a human operator, and simultaneously clears the content flag of the "_human" field. This audio message is then played to the taxpayer via an audio device. When the system detects that the taxpayer's intent is to end the session, it updates the session state in the Redis service cluster to "_XWWTJJQK". Similarly, the websocket service for TTS output reads the session state, retrieves the TTS audio, and simultaneously clears the content flag of the "_XWWTJJQK" field to obtain the ending audio message, which is then played to the taxpayer. When the system detects that the taxpayer's intent is to inquire about a problem, it employs a hybrid recall strategy based on Faiss (a vector retrieval tool library) vector retrieval and Rerank (a secondary ranking technique) to retrieve the taxpayer's question from the knowledge base. Simultaneously, it updates the "_confirm_type" field in the Redis service cluster to "knowledge". The websocket service for synchronous TTS output reads the "knowledge" field in the Redis service cluster to obtain the TTS audio message requiring user confirmation, and waits for user confirmation to confirm whether it is the question the user intended to ask.

[0081] for Figure 2 In the "Q&A Information Supplementation" section, specifically, during the aforementioned steps, if the taxpayer has clearly stated their specific questions, the system will use specific flags in the knowledge base to determine if multiple rounds of information supplementation are necessary. Once it is determined that multiple rounds of information supplementation are required, the system will automatically update the session status flags in the Redis service cluster, setting the "_is_multi_dimension" flag to 1 and the "_dimension_flag" flag to 0. This update ensures that the system can recognize that the current session requires multi-dimensional information supplementation. After this status update, the synchronized TTS output websocket service will begin working, reading the current session's _is_multi_dimension and _dimension_flag status information from the Redis service cluster. Based on this status information, the websocket service will retrieve the corresponding supplementary question information and the relevant TTS audio files, which will then be transmitted to the taxpayer. By listening to these supplementary question TTS audios, taxpayers can further supplement and improve their question content, making the entire consultation process more efficient and accurate.

[0082] Furthermore, referring to Figure 6 As shown, the steps of "intent recognition" and "question and answer information supplementation" mentioned above are explained in detail.

[0083] By leveraging the semantic understanding capabilities of large-scale language models, the audio text content submitted by taxpayers can be analyzed in depth to accurately determine their intent. This intent can be categorized into three main types: inquiry, transfer to human assistance, or termination of the conversation. When the taxpayer's intent is identified as an inquiry, the system initiates a persistent WebSocket connection to maintain continuous communication. Simultaneously, the system utilizes a Redis service cluster as middleware for the state machine, ensuring that each state node has corresponding flags set and key information stored in the Redis cluster. Thus, when the state changes, another persistent connection established via the WebSocket service can obtain these changed flags in real time and retrieve the corresponding state information from the Redis cluster. Subsequently, the system can use this information to retrieve the corresponding TTS audio file, enabling voice interaction with the user and providing a more natural and fluid user experience.

[0084] Furthermore, offline TTS audio content creation allows for the pre-creation of dialogue node templates and their conversion into base64 encoded audio files. This method effectively avoids the substantial resource consumption required for online TTS generation, thus reducing system load during processing. Additionally, it significantly shortens system response time and improves overall efficiency. These audio files can be quickly used in various application scenarios without real-time online conversion, ensuring a smooth user experience.

[0085] for Figure 2 In the "knowledge retrieval" section, specifically, during the aforementioned steps, taxpayers need to provide all relevant dimensional information about the question. Once this information is fully provided, the system will retrieve the corresponding dimensional answers from its built-in knowledge base. Next, the system will use a large model to analyze the session history and the retrieved answer content to determine the exact dimensional answer the user needs. Finally, through a synchronized TTS (Text-to-Speech) output service, this precisely determined dimensional answer will be clearly conveyed to the taxpayer.

[0086] Therefore, it is easy to see that the embodiments of this application can support semantic-driven intelligent segmentation. This multi-strategy dynamic VAD technology can effectively reduce the call frequency of the ASR system, thereby reducing resource consumption during processing. At the same time, compared with those large end-to-end models, this application has a significant advantage in terms of computational cost; it is more economical and efficient, and particularly suitable for enterprise-level private deployment environments. This improvement in cost-effectiveness makes this invention not only technologically advanced but also highly practical and competitive in commercial applications.

[0087] Accordingly, see Figure 7 As shown, this application embodiment provides a real-time voice dialogue device, including:

[0088] The threshold determination module 11 is used to obtain an initial audio segment input by the taxpayer user terminal and determine a dynamic amplitude threshold based on the initial audio segment; the initial audio segment is an audio segment containing a taxpayer consultation question;

[0089] The proportion determination module 12 is used to determine the proportion of audio segments with an audio amplitude greater than the dynamic amplitude threshold in the initial audio segment if the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold.

[0090] The segment determination module 13 is used to determine the speech endpoints in the initial audio segment using speech activity detection technology if the occupancy ratio is greater than a preset ratio threshold, and to perform corresponding segmentation processing on the initial audio segment based on the speech endpoints to obtain a first target audio segment; the speech endpoints include a start endpoint that identifies the start position of the speech segment and an end endpoint that identifies the end position of the speech segment;

[0091] The intent determination module 14 is used to convert the first target audio segment into a text segment using automatic speech recognition technology, and to determine the user intent of the taxpayer terminal based on a preset large model and the text segment.

[0092] The voice dialogue module 15 is used to determine the corresponding second target audio segment based on the user intent, preset WebSocket technology and preset distributed caching architecture, and send the second target audio segment to the taxpayer user terminal to play the second target audio segment on the taxpayer user terminal to complete real-time voice dialogue.

[0093] In some specific embodiments, the threshold determination module 11 specifically includes:

[0094] The segment acquisition unit is used to acquire the initial audio segment input by the taxpayer client through the audio acquisition device and based on the preset websocket technology;

[0095] The coefficient determination unit is used to determine the percentile, peak value and rate of change of the audio energy of the initial audio segment, and to determine the sensitivity coefficient based on the degree of abrupt change of the audio energy.

[0096] The threshold determination unit is used to determine the dynamic amplitude threshold based on the preset weight value, the percentile, the peak value, the rate of change, and the sensitivity coefficient.

[0097] In some specific embodiments, the ratio determination module 12 further includes:

[0098] A window construction unit is used to construct a sliding time window of a preset duration; the preset duration is longer than the duration corresponding to the preset tail segment.

[0099] An audio detection unit is used to detect, based on the sliding time window, whether there is an audio segment in the current initial audio segment whose audio amplitude is greater than the dynamic amplitude threshold, and to obtain the corresponding detection result;

[0100] The first step jump unit is used to jump to the step of obtaining the initial audio segment input by the taxpayer user terminal if the detection result shows that it exists.

[0101] In some specific embodiments, the intent determination module 14 further includes:

[0102] An information determination unit is used to determine the audio sampling rate and the current batch size of the first target audio segment;

[0103] The segment conversion unit is used to convert the first target audio segment into a text segment using automatic speech recognition technology if both the audio sampling rate and the batch size meet preset audio recognition conditions.

[0104] In some specific embodiments, the voice dialogue module 15 specifically includes:

[0105] The first audio delivery unit is used to set the current conversation state in the preset distributed caching architecture to the manual consultation state when the user's intention is manual consultation, and to read the current session state using preset WebSocket technology in order to retrieve the corresponding second target audio segment from the preset audio library and deliver the second target audio segment to the taxpayer user terminal.

[0106] The second audio delivery unit is used to set the current conversation state in the preset distributed caching architecture to the conversation end state when the user intends to end the conversation, and to use the preset websocket technology to read the current session state in order to retrieve the corresponding second target audio segment from the preset audio library and deliver the second target audio segment to the taxpayer user terminal.

[0107] The third audio delivery unit is used to, when the user's intent is to inquire about a problem, retrieve a first target question corresponding to the user's intent from a preset question knowledge base using a hybrid recall strategy, set the current dialogue state in the preset distributed caching architecture to a question inquiry state, read the current session state using the preset WebSocket technology, retrieve a second target audio segment corresponding to the first target question from the preset audio library, and deliver the second target audio segment to the taxpayer user terminal; the hybrid recall strategy includes a similarity retrieval tool and a re-ranking technique; the first target question is a question that meets the preset question completeness condition.

[0108] In some specific embodiments, the voice dialogue module 15 further includes:

[0109] The second step jump unit is used to, when the user intent indicates that there is a second target question on the taxpayer user terminal, set the current dialogue state in the preset distributed caching architecture to a question supplementation state, and use the preset WebSocket technology to read the current session state to retrieve the second target audio segment corresponding to the second target question from the preset audio library, send the second target audio segment to the taxpayer user terminal, and after the taxpayer user terminal plays the second target audio segment, jump to the step of obtaining the initial audio segment input by the taxpayer user terminal; the second target question is a question that does not meet the preset question integrity condition.

[0110] In some specific embodiments, the audio segments in the preset audio library are audio segments in a preset encoding format obtained by converting a preset dialogue text template offline.

[0111] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the real-time voice dialogue method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0112] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0113] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0114] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the real-time voice dialogue method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0115] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned real-time voice dialogue method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0117] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A real-time voice dialogue method, characterized in that, include: Obtain an initial audio segment input by the taxpayer user, and determine a dynamic amplitude threshold based on the initial audio segment; the initial audio segment is an audio segment containing a taxpayer consultation question; If the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold, then in the initial audio segment, the proportion of audio segments with audio amplitude greater than the dynamic amplitude threshold is determined. If the occupancy ratio is greater than a preset ratio threshold, then the speech endpoints in the initial audio segment are determined using speech activity detection technology, and the initial audio segment is segmented accordingly based on the speech endpoints to obtain the first target audio segment; the speech endpoints include a start endpoint that identifies the start position of the speech segment and an end endpoint that identifies the end position of the speech segment; The first target audio segment is converted into a text segment using automatic speech recognition technology, and the user intent of the taxpayer terminal is determined based on a preset large model and the text segment. Based on the user intent, the preset WebSocket technology, and the preset distributed caching architecture, a corresponding second target audio segment is determined, and the second target audio segment is sent to the taxpayer user terminal to play the second target audio segment in the taxpayer user terminal to complete real-time voice dialogue.

2. The real-time voice dialogue method according to claim 1, characterized in that, The step of obtaining the initial audio segment input by the taxpayer user and determining the dynamic amplitude threshold based on the initial audio segment includes: The initial audio segment input by the taxpayer's client is obtained through an audio acquisition device and based on a preset WebSocket technology; The percentile, peak value, and rate of change of the audio energy of the initial audio segment are determined, and the sensitivity coefficient is determined based on the degree of abrupt change in the audio energy. The dynamic amplitude threshold is determined based on the preset weight value, the percentile, the peak value, the rate of change, and the sensitivity coefficient.

3. The real-time voice dialogue method according to claim 1, characterized in that, The step of if the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold further includes: Construct a sliding time window of preset duration; the preset duration is longer than the duration corresponding to the preset tail segment. Based on the sliding time window, detect whether there is an audio segment in the current initial audio segment whose audio amplitude is greater than the dynamic amplitude threshold, and obtain the corresponding detection result; If the detection result indicates that such a result exists, then proceed to the step of obtaining the initial audio segment input by the taxpayer user.

4. The real-time voice dialogue method according to claim 1, characterized in that, Before converting the first target audio segment into a text segment using automatic speech recognition technology, the method further includes: Determine the audio sampling rate of the first target audio segment and the current batch size; If both the audio sampling rate and the batch size meet the preset audio recognition conditions, then the first target audio segment is converted into a text segment using automatic speech recognition technology.

5. The real-time voice dialogue method according to any one of claims 1 to 4, characterized in that, The step of determining the corresponding second target audio segment based on the user intent, preset WebSocket technology, and preset distributed caching architecture, and sending the second target audio segment to the taxpayer user terminal, includes: When the user's intention is for human consultation, the current dialogue state in the preset distributed caching architecture is set to human consultation state, and the current session state is read using preset WebSocket technology to retrieve the corresponding second target audio segment from the preset audio library and send the second target audio segment to the taxpayer user terminal. When the user's intention is to end the conversation, the current conversation state in the preset distributed caching architecture is set to the end conversation state, and the current session state is read using the preset WebSocket technology to retrieve the corresponding second target audio segment from the preset audio library, and the second target audio segment is sent to the taxpayer user terminal; When the user's intent is to inquire about a problem, a hybrid recall strategy is used to retrieve a first target problem corresponding to the user's intent from a preset problem knowledge base. The current dialogue state in the preset distributed caching architecture is set to a problem inquiry state. The current session state is read using the preset WebSocket technology to retrieve a second target audio segment corresponding to the first target problem from the preset audio library. The second target audio segment is then sent to the taxpayer's client. The hybrid recall strategy includes a similarity retrieval tool and a re-ranking technique. The first target problem is a problem that meets the preset problem completeness condition.

6. The real-time voice dialogue method according to claim 5, characterized in that, The step of determining the corresponding second target audio segment based on the user intent, preset WebSocket technology, and preset distributed caching architecture, and sending the second target audio segment to the taxpayer user terminal, further includes: When the user intent indicates that there is a second target question on the taxpayer user terminal, the current dialogue state in the preset distributed caching architecture is set to a question supplementation state. The current session state is read using the preset WebSocket technology to retrieve the second target audio segment corresponding to the second target question from the preset audio library. The second target audio segment is then sent to the taxpayer user terminal. After the second target audio segment is played on the taxpayer user terminal, the process jumps to the step of obtaining the initial audio segment input by the taxpayer user terminal. The second target question is a question that does not meet the preset question integrity condition.

7. The real-time voice dialogue method according to claim 5, characterized in that, The audio segments in the preset audio library are audio segments in a preset encoding format obtained by converting preset dialogue text templates offline.

8. A real-time voice dialogue device, characterized in that, include: The threshold determination module is used to obtain an initial audio segment input by the taxpayer user terminal and determine a dynamic amplitude threshold based on the initial audio segment; the initial audio segment is an audio segment containing a taxpayer consultation question; The proportion determination module is used to determine the proportion of audio segments with audio amplitudes greater than the dynamic amplitude threshold in the initial audio segment if the audio amplitude of the preset tail segment in the initial audio segment is not greater than the dynamic amplitude threshold. The segment determination module is used to determine the speech endpoints in the initial audio segment using speech activity detection technology if the occupancy ratio is greater than a preset ratio threshold, and to perform corresponding segmentation processing on the initial audio segment based on the speech endpoints to obtain a first target audio segment; the speech endpoints include a start endpoint that identifies the start position of the speech segment and an end endpoint that identifies the end position of the speech segment; The intent determination module is used to convert the first target audio segment into a text segment using automatic speech recognition technology, and to determine the user intent of the taxpayer terminal based on a preset large model and the text segment. The voice dialogue module is used to determine the corresponding second target audio segment based on the user intent, preset WebSocket technology and preset distributed caching architecture, and send the second target audio segment to the taxpayer user terminal to play the second target audio segment on the taxpayer user terminal to complete real-time voice dialogue.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the real-time voice dialogue method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the real-time voice dialogue method as described in any one of claims 1 to 7.

Images