An intelligent customer service system fusing speech recognition, dialogue generation and digital human output

Abstract

The application discloses an intelligent customer service system fusing voice recognition, dialogue generation and digital person output, and relates to the technical field of computer information processing.The system comprises a unified protocol access module, a message distribution module, a session management module, a voice recognition module, a dialogue generation module, a voice synthesis module, a multi-modal output module and a result verification module; unified protocol messages are sent through a single WebSocket long connection; terminal connection, voice recognition connection, voice synthesis connection and digital person video task association information mapping are established based on session identification, and expired results are filtered through message serial numbers; audio load is cleaned, decoded and fragmented to obtain recognized text; streaming answer text is cached and segmented to obtain semantic segments; text segments, voice segments and digital person video stream addresses are generated based on the same semantic segments.The application can reduce terminal access complexity, improve session isolation of streaming interaction and multi-modal output synchronization.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and in particular to an intelligent customer service system that integrates speech recognition, dialogue generation and digital human output. Background Technology

[0002] With the application of artificial intelligence technology in business reception, store navigation, showroom explanation, front desk consultation and intelligent customer service terminals, terminal devices are gradually required to have interactive capabilities such as voice acquisition, voice recognition, question understanding, intelligent answering, voice broadcasting and digital human video presentation. Existing intelligent customer service systems usually need to access voice recognition services, dialogue generation services, voice synthesis services and digital human video-driven services separately. The communication protocols, authentication methods, data formats and return results of various services are different, which leads to the need to maintain multiple connections and multiple sets of adaptation logic on the terminal side, increasing the complexity of device access and subsequent maintenance costs.

[0003] In actual streaming interaction, there is a continuous temporal relationship between user voice input, speech recognition results, large-model streaming responses, speech synthesis results, and digital human video results. If the system lacks a unified session management and message orchestration mechanism, problems such as old request results being mistakenly sent to new sessions, crosstalk between different user sessions, and mismatch between recognition results and subsequent responses can easily occur when the terminal reconnects, the user is interrupted, the network fluctuates, or third-party services return asynchronously. At the same time, large-model responses are usually returned as continuous streaming text. If the complete response is waited for before being broadcast, it will increase the waiting time for the user to hear the reply for the first time. If it is broadcast word by word or in too short segments, it will cause fragmented speech expression, frequent audio switching, and unnatural presentation of digital human video. In addition, existing intelligent customer service systems often handle subtitle display, voice broadcasting, and video driving separately when outputting text, voice, and digital human video, lacking a unified segment boundary and segment sequence association mechanism, which can easily lead to problems such as out-of-order, misalignment, or asynchrony between subtitles, voice, and digital human video. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the existing technology, such as the complexity of terminal access to multiple heterogeneous artificial intelligence services, the easy mixing of conversation results in streaming interaction, and the difficulty in synchronizing text, voice and digital human video output. The invention proposes an intelligent customer service system that integrates speech recognition, dialogue generation and digital human output.

[0005] To address the problems existing in the prior art, the present invention adopts the following technical solution: An intelligent customer service system integrating speech recognition, dialogue generation, and digital human output includes: The unified protocol access module is used to receive unified protocol messages sent by the terminal through a single WebSocket long connection. The unified protocol message includes interface type, session identifier, message sequence number, payload data and video enable flag. The payload data includes audio payload or user text. The message distribution module is used to distribute the payload data to the speech recognition module, dialogue generation module, or speech synthesis module according to the interface type. The session management module is used to establish a mapping between terminal connection and speech recognition connection, speech synthesis connection and digital human video task association information based on the session identifier, and to save the current valid message sequence number; The speech recognition module is used to clean, decode, and segment the audio load to obtain audio segments, and to obtain recognized text based on the audio segments; The dialogue generation module is used to generate streaming response text based on the identified text or the user text, and to cache and segment the streaming response text to obtain semantic fragments carrying the message sequence number. A speech synthesis module is used to generate speech segments based on the semantic segments; The multimodal output module is used to generate text and speech segments based on the same semantic segment, and generate a digital human video stream address when the video enable flag indicates that the video is enabled, and bind the output results corresponding to the same semantic segment to the same segment number; The result verification module is used to verify whether the message sequence number carried by the output result is consistent with the current valid message sequence number before the output result is sent back, and to send back the output result if they are consistent.

[0006] Preferably, the session management module includes: A connection storage unit is used to save the terminal connection object according to the session identifier; The repeated access processing unit is used to close the saved terminal connection and save the new terminal connection as the current terminal connection when the same session identifier is detected to be accessed again. The message sequence number update unit is used to update the currently valid message sequence number when the terminal reconnects or the user interrupts receiving; The expired result filtering unit is used to discard recognition results, voice segments, or digital human video stream addresses that do not match the currently valid message sequence number.

[0007] Preferably, the speech recognition module includes: An audio cleaning unit is used to remove audio data prefixes, spaces, and newline characters from the audio load to obtain cleaned audio data. An audio decoding unit is used to decode the cleaned audio data to obtain an audio byte stream; An audio segmentation unit is used to determine the segment length based on the sampling rate, bit depth, number of channels, and segment duration, and to divide the audio byte stream into multiple audio segments according to the segment length; The segment identification unit is used to configure an incrementing sequence number for each audio segment and to configure a final packet identifier for the last audio segment. The recognition result conversion unit is used to convert the intermediate recognition result and the final recognition result obtained from the audio segment into a unified business protocol result. A recognition response receiving unit is used to receive a recognition response returned by the speech recognition service based on the audio segment; A text parsing unit is used to parse intermediate recognized text, final recognized text, and final packet status from the recognized response; The text recognition output unit is used to send the final recognized text as the recognized text to the dialogue generation module when the final packet state indicates that the speech recognition has ended.

[0008] Preferably, the message distribution module includes: The identification start distribution unit is used to trigger the speech recognition module to establish a speech recognition connection when the interface type is speech recognition start request; An audio data distribution unit is used to send the payload data to the speech recognition module when the interface type is an audio data upload request. The identification end distribution unit is used to trigger the speech recognition module to send an audio segment carrying the final packet identifier and release the corresponding speech recognition connection when the interface type is a speech recognition end request; A dialogue distribution unit is used to send the identified text or the user text to the dialogue generation module when the interface type is a dialogue request. The synthesis and distribution unit is used to send the text to be synthesized to the speech synthesis module when the interface type is a speech synthesis request.

[0009] Preferably, the dialogue generation module includes: The streaming event receiving unit is used to receive response-type events and tool response-type events continuously returned by the dialogue model; A fragment caching unit is used to write the text content of the answer-type event into a fragment caching queue; The segmentation determination unit is used to determine segmentation boundaries when segmentation punctuation, natural sentence break events, the duration of no content determined by the session configuration, the maximum cache length determined by the session configuration, or the end of response event are detected. A semantic fragment generation unit is used to extract the text before the segment boundary from the fragment cache queue as a semantic fragment, and to take the remaining text under the answer end event as the final semantic fragment. The knowledge content storage unit is used to store the knowledge hit content in the tool response events.

[0010] Preferably, the multimodal output module includes: A text output unit is used to generate text segments based on the semantic segments and to configure segment numbers for the text segments. A speech output unit is configured to receive a speech segment generated by the speech synthesis module based on the semantic segment, and to assign the speech segment the same segment number as the text segment. The video output unit is configured to send the audio segment to the digital human video driver service when the video enable flag indicates that video is enabled, obtain the digital human video stream address, and configure the digital human video stream address with the same segment number as the text segment; The return organization unit is used to organize the text segment, the audio segment, and the digital human video stream address at the time of generation according to the segment sequence number, and obtain the output result.

[0011] Preferably, the speech synthesis module includes: The parameter reading unit is used to read language, timbre, and emotion parameters according to the session configuration; The request encapsulation unit is used to encapsulate the semantic fragment and the language, timbre and emotion parameters into a speech synthesis request; An audio receiving unit is used to receive audio stream fragments returned by a speech synthesis service; The encoding processing unit is used to convert the audio stream segments into audio encoded data, and when the audio encoded data exceeds the transmission length determined by the session configuration or the system default configuration, divide the audio encoded data into multiple encoded segments.

[0012] Preferably, the result verification module includes: The result tag reading unit is used to read the session identifier, message sequence number, and fragment sequence number carried in the outgoing result to be transmitted back; The validity determination unit is used to read the current valid message sequence number according to the session identifier, and determine whether the message sequence number carried by the outgoing result to be transmitted back is consistent with the current valid message sequence number; The result sending unit is used to send the result to be returned to the terminal through the single WebSocket long connection when the judgment result is consistent. The result discarding unit is used to stop sending the outgoing result to be returned when the result is determined to be inconsistent, and to record the reason for discarding.

[0013] Preferably, it further includes a configuration and logging module, the configuration and logging module comprising: The configuration reading unit is used to read the speech recognition language, speech synthesis timbre, speech synthesis language type, and speech synthesis emotion parameters based on the session identifier, store identifier, or configuration number. The tracking identifier generation unit is used to generate a tracking identifier for a single dialogue link; The log recording unit is used to record session identifiers, tracking identifiers, message sequence numbers, segment sequence numbers, user questions, answer content, knowledge hit content, speech recognition time, speech synthesis time, video generation time, and abnormal information. The log association unit is used to associate the identification results, semantic segments, voice segments and digital human video stream addresses in the same dialogue link according to the tracking identifier; The access control unit is used to restrict access permissions to the log content recorded by the log recording unit in accordance with access control rules.

[0014] Preferably, it further includes a knowledge base invocation module, the knowledge base invocation module comprising: The structured input generation unit is used to generate structured input content based on user questions, tracking identifiers, store identifiers, and user identifiers. The knowledge base selection unit is used to select the corresponding knowledge base based on the store identifier or geographical intent in the structured input content. The retrieval result judgment unit is used to compare the retrieval results returned by the knowledge base with the minimum matching conditions determined by the session configuration; The response constraint unit is used to provide the search results to the dialogue generation module when the search results meet the minimum matching conditions, and to generate a prompt to switch to human assistance and provide it to the dialogue generation module when the search results do not meet the minimum matching conditions.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention enables terminals to access the server via a single WebSocket long connection through a unified protocol access module. The unified protocol message carries the interface type, session identifier, message sequence number, payload data, and video enable flag. The server-side message distribution module, speech recognition module, dialogue generation module, speech synthesis module, and multimodal output module unify the orchestration of different artificial intelligence capabilities. Terminals do not need to separately adapt to the heterogeneous protocols of speech recognition, dialogue generation, speech synthesis, and digital human video-driven services, reducing the access complexity, protocol adaptation costs, and subsequent maintenance difficulty of terminal devices. The session management module establishes a mapping between terminal connections and the associated information of speech recognition, speech synthesis, and digital human video tasks. It also performs pre-transmission verification based on the current valid message sequence number, discarding expired results when the terminal reconnects, the user interrupts, or asynchronous tasks return late. This prevents old request results from being mistakenly sent to new sessions, improving session isolation and result reliability in the streaming customer service interaction process.

[0016] 2. This invention generates streaming response text from speech recognition results or user text, and caches and segments the streaming response text to obtain semantic segments carrying message sequence numbers. This allows speech synthesis and digital human video output to be executed segment by segment based on the formed semantic segments without waiting for the complete response to finish, thereby shortening the user's waiting time for the first response and avoiding semantic fragmentation and frequent audio switching caused by word-by-word broadcasting. Furthermore, the multimodal output module generates text segments, speech segments, and digital human video stream addresses based on the same semantic segment, and binds the output results corresponding to the same semantic segment to the same segment sequence number. This enables the terminal to match subtitles, speech, and digital human video output according to the segment sequence number, reducing the problems of disorder, misalignment, and asynchrony between text display, speech broadcasting, and digital human video presentation, and improving the real-time interactive experience and multimodal output consistency of the intelligent customer service system. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is an architecture diagram of an intelligent customer service system that integrates speech recognition, dialogue generation, and digital human output, provided as an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] Example: Figure 1 As shown, this embodiment provides an intelligent customer service system that integrates speech recognition, dialogue generation, and digital human output. This system is applied to intelligent customer service terminals, store reception terminals, exhibition hall navigation terminals, front desk Q&A terminals, amusement park navigation equipment, or other terminal devices that require human-computer interaction through voice, text, and digital human video. The terminal device establishes a communication link with the server through a single WebSocket long connection. The server receives the audio payload or user text uploaded by the terminal through the single WebSocket long connection and returns the recognition result, text fragments, voice fragments, and digital human video stream address to the terminal through the single WebSocket long connection. In this embodiment, the intelligent customer service system includes a unified protocol access module, a message distribution module, a session management module, a speech recognition module, a dialogue generation module, a speech synthesis module, a multimodal output module, a result verification module, a configuration and logging module, and a knowledge base call module. Each module is deployed on the server side, and the terminal side does not need to access the speech recognition service, speech synthesis service, dialogue generation service, and digital human video-driven service separately. The server side completes the establishment of multiple capability connections, protocol adaptation, data distribution, result verification, and multimodal output organization. In this embodiment, the unified protocol access module receives unified protocol messages sent by the terminal through a single WebSocket long connection. The unified protocol message includes at least an interface type, a session identifier, a message sequence number, payload data, and a video enable flag. The interface type is used to characterize the business action corresponding to the current message, and includes voice recognition start request, audio data upload request, voice recognition end request, dialogue request, voice synthesis request, and heartbeat request. The session identifier is used to distinguish between different terminals or different sessions. The message sequence number is used to distinguish between different rounds of requests within the same session. The payload data is used to carry audio payload or user text. The video enable flag indicates whether the digital human video stream address needs to be output in this round of dialogue.

[0020] Furthermore, after receiving the unified protocol message, the server generates the current round request context based on the unified protocol message. The current round request context includes at least a session identifier, message sequence number, interface type, video enable flag, and tracking flag. The recognition results generated by the speech recognition module, the semantic segments generated by the dialogue generation module, the speech segments generated by the speech synthesis module, the digital human video stream address generated by the multimodal output module, and the outgoing results to be transmitted all inherit the message sequence number from the current round request context. Thus, the message sequence number carried by the outgoing results to be transmitted has a clear source, and the result verification module can determine whether the outgoing results to be transmitted belong to the current valid request based on the message sequence number. In one specific implementation, the interface type field in the unified protocol message is api, the session identifier field is sid, the message sequence number field is serial, the payload data field is payload, and the video enable flag field is useVideo. For ease of subsequent description, this embodiment will uniformly use "interface type", "session identifier", "message sequence number", "payload data" and "video enable flag" for description.

[0021] In this embodiment, the session identifier is used to identify the terminal connection and its associated service capability connection on the server side; the message sequence number is used to identify a round of requests under the same session identifier; the trace identifier is used to associate logs, asynchronous tasks and intermediate results in the same dialogue link; the fragment sequence number is used to identify the output order of semantic fragments in the same round of requests; the session identifier, message sequence number, trace identifier and fragment sequence number have different functions: the session identifier is used for connection mapping, the message sequence number is used for old result filtering, the trace identifier is used for log association, and the fragment sequence number is used for order matching between text fragments, voice fragments and digital human video stream addresses; In this embodiment, after receiving a terminal connection request, the unified protocol access module forwards the connection request to the session management module. The session management module saves the terminal connection object according to the session identifier and establishes a mapping relationship between the terminal connection and the speech recognition connection, speech synthesis connection, and digital human video task association information according to the session identifier. The terminal connection is a WebSocket connection between the terminal and the server. The speech recognition connection is a connection between the server and the speech recognition service. The speech synthesis connection is a connection between the server and the speech synthesis service. The digital human video task association information is the task identifier, task status, or return result association information saved by the server for the digital human video driving task.

[0022] The session management module saves the current valid message sequence number; the current valid message sequence number is used to determine whether the result to be returned belongs to a terminal request that is still valid; when a terminal connects for the first time, the session management module establishes a terminal connection object according to the session identifier in the unified protocol message; when the same session identifier is detected to be connected again, the session management module closes the saved terminal connection and saves the new terminal connection as the current terminal connection; at the same time, the session management module updates the current valid message sequence number so that the late result corresponding to the old terminal connection can no longer be returned to the current terminal connection. When the terminal actively interrupts receiving results, re-establishes the connection, or the server detects a change in the connection status, the session management module also updates the current valid message sequence number. For speech recognition results, speech segments, or digital human video stream addresses that have been initiated but not yet returned, the server verifies the message sequence number through the result verification module before sending. If the message sequence number carried by the result to be returned is inconsistent with the current valid message sequence number, it means that the result belongs to an old request, an old connection, or an interrupted task. The result verification module discards the result and no longer sends it to the terminal through a single WebSocket long connection. In this way, in the case of device reconnection, user interruption, or delayed return of asynchronous tasks, the old session results can be prevented from being inserted into the current session. In this embodiment, the message distribution module receives the unified protocol message forwarded by the unified protocol access module and distributes the payload data to the corresponding module according to the interface type. When the interface type is a speech recognition start request, the message distribution module triggers the speech recognition module to establish a speech recognition connection. When the interface type is an audio data upload request, the message distribution module sends the payload data to the speech recognition module. When the interface type is a speech recognition end request, the message distribution module triggers the speech recognition module to send an audio segment carrying the final packet identifier and release the corresponding speech recognition connection. When the interface type is a dialogue request, the message distribution module sends the recognized text or user text to the dialogue generation module. When the interface type is a speech synthesis request, the message distribution module sends the text to be synthesized to the speech synthesis module. When the interface type is a heartbeat request, the message distribution module triggers the unified protocol access module to return a heartbeat response.

[0023] In this embodiment, the speech recognition module is used to clean, decode, and segment the audio load to obtain audio segments, and obtain the recognized text based on the audio segments; the speech recognition module includes an audio cleaning unit, an audio decoding unit, an audio segmentation unit, a segmentation identification unit, and a recognition result conversion unit; When the terminal sends a speech recognition start request, the speech recognition module creates a speech recognition connection based on the session identifier and reads the initialization parameters required for speech recognition. The initialization parameters include audio format, sampling rate, bit depth, number of channels, and recognition language. Specifically, the audio format is WAV, the sampling rate is 16000 Hz, the bit depth is 16 bits, the number of channels is mono, and the recognition language is determined according to the session configuration as Chinese, English, Cantonese, or other languages. The above specific parameters are exemplary parameters, and in actual implementation, they are configured according to the terminal's acquisition format and speech recognition service requirements. When the terminal continuously sends audio data upload requests, the audio cleaning unit receives the audio payload in the payload data; if the audio payload contains the prefix data:audio / wav;base64, the audio cleaning unit removes the prefix; if the audio payload contains spaces, newline characters, or other invalid whitespace characters, the audio cleaning unit deletes the spaces, newline characters, or other invalid whitespace characters to obtain cleaned audio data; the audio decoding unit performs Base64 decoding on the cleaned audio data to obtain an audio byte stream. The audio segmentation unit determines the segment length based on the sampling rate, bit depth, number of channels, and segment duration, and divides the audio byte stream into multiple audio segments according to the segment length; the segment length is determined according to the following relationship: Segment length = Sampling rate × Bit depth / 8 × Number of channels × Segment duration / 1000; Here, the sampling rate represents the number of sampling points collected per second, the bit depth represents the number of bits corresponding to each sampling point, the number of channels represents the number of audio channels, and the segment duration represents the number of milliseconds corresponding to each audio segment. Taking a sampling rate of 16000 Hz, a bit depth of 16 bits, a number of channels of 1, and a segment duration of 200 milliseconds as an example, the number of audio bytes per second is 16000×16 / 8×1, which is 32000 bytes. The segment length corresponding to each 200-millisecond audio segment is 32000×200 / 1000, which is 6400 bytes.

[0024] The segment identification unit configures an incrementing sequence number for each audio segment and a final packet identifier for the last audio segment. In this embodiment, the final packet identifier is used to enable the speech recognition service to recognize that the current speech input has ended. The final packet identifier is implemented using a negative sequence number or by using the last packet field in the speech recognition protocol message. When the terminal sends a speech recognition end request, the speech recognition module configures the last audio segment to be sent as an audio segment carrying the final packet identifier. When there is no new audio byte stream to be sent, the speech recognition module generates an empty payload or an end payload and configures the final packet identifier for the empty payload or the end payload. After the speech recognition service returns the final recognition result, the speech recognition module releases the corresponding speech recognition connection according to the final recognition result. Specifically, the segmentation identification unit writes the incrementing sequence number into the sequence number field of the third-party speech recognition protocol message; when the audio segment is the last audio segment, the segmentation identification unit sets the sequence number corresponding to the audio segment to an identifier with final inclusion meaning; the speech recognition module encapsulates the audio segment, the incrementing sequence number, the final packet identifier, and the compressed audio data into a protocol message required by the speech recognition service, and sends it to the speech recognition service through the speech recognition connection.

[0025] The speech recognition module internally consists of a send buffer, a response buffer, a send thread, and a receive thread. The send thread reads audio segments from the send buffer and sends them to the speech recognition service. The receive thread reads the recognition response from the speech recognition service and parses the text content and final packet status in the recognition response. The send thread and the receive thread are independent of each other, so that the sending of audio segments and the receiving of recognition results do not block each other. The recognition result conversion unit converts the intermediate and final recognition results returned by the speech recognition service into unified business protocol results. The unified business protocol results carry session identifiers, message sequence numbers, recognized text, and recognition status. If the recognition result needs to be sent back to the terminal, the result verification module reads the message sequence number carried by the recognition result before sending it back and determines whether the message sequence number is consistent with the currently valid message sequence number. If they are consistent, the result is sent back to the terminal through the unified protocol access module. If they are inconsistent, the recognition result is discarded. If the recognition result needs to enter the dialogue generation process, the message distribution module sends the recognized text to the dialogue generation module.

[0026] In this embodiment, the dialogue generation module is used to generate streaming response text based on the recognized text or user text, and to cache and segment the streaming response text to obtain semantic fragments carrying message sequence numbers; user text is the text content directly sent by the terminal through the payload data in the unified protocol message; that is, this embodiment supports both user-triggered dialogue via voice input and terminal-triggered dialogue by directly uploading text. The dialogue generation module includes a streaming event receiving unit, a segment caching unit, a segmentation judgment unit, a semantic segment generation unit, and a knowledge content storage unit. After receiving the recognized text or user text, the dialogue generation module assigns a tracking identifier to the current dialogue link and reads the store identifier, user identifier, voice configuration, knowledge base configuration, and video enable flag according to the session configuration. The dialogue generation module sends the recognized text or user text to the dialogue model and receives the streaming events continuously returned by the dialogue model. Streaming events include answer-type events and tool response-type events. Answer-type events contain the text content output by the dialogue model; tool response-type events contain knowledge base retrieval results or tool call results. The streaming event receiving unit continuously receives response-type events; the fragment caching unit writes the text content in the response-type events into the fragment caching queue; the fragment caching queue is used to cache continuous text content that has not yet formed semantic fragments; for single characters, words, phrases or punctuation marks returned by streaming, the fragment caching unit writes them into the fragment caching queue in the order of reception to ensure that subsequent segmentation operations are executed based on the complete streaming text order; The segmentation determination unit determines the segmentation boundary based on segmentation punctuation, natural sentence break events, the duration of no content determined by the session configuration, the maximum cache length determined by the session configuration, or the end-of-response event; segmentation punctuation includes Chinese period, Chinese comma, Chinese exclamation mark, Chinese question mark, English period, English comma, English exclamation mark, and English question mark; Natural sentence segmentation events refer to segment output conditions that the server can detect based on the streaming return status; the natural sentence segmentation event includes at least one of the following: the last character of the segment cache queue is a segment punctuation mark; no new answer-type event is received within the content-free duration determined by the session configuration; the text length in the segment cache queue reaches the maximum cache length determined by the session configuration; or an answer end event is received; after the server detects the natural sentence segmentation event, it does not directly end the entire round of dialogue, but extracts the text in the segment cache queue that meets the segmentation boundary as a semantic segment, and continues to send the semantic segment to the speech synthesis module and the multimodal output module; The duration of no content, the maximum cache length, and the sending length are all read by the configuration reading unit from the session configuration or the system default configuration. The unit of the duration of no content is milliseconds, used to determine whether segmentation is triggered when the streaming response text has not been updated for a period of time. The unit of the maximum cache length is the number of characters, used to limit the maximum cumulative length of the segment cache queue when no segmentation punctuation is detected. The unit of the sending length is bytes, used to limit the length of audio encoded data sent in a single WebSocket connection. When the duration of no content, the maximum cache length, or the sending length is not configured in the session configuration, the configuration reading unit reads the system default configuration. Furthermore, the sending length does not exceed the message cache limit corresponding to a single WebSocket long connection. For example, the maximum text message cache limit for a single WebSocket long connection is 2MB, the maximum binary message cache limit is 2MB, and the maximum idle timeout is 600 seconds.

[0027] After the segmentation determination unit determines the segmentation boundary, the semantic fragment generation unit extracts the text before the segmentation boundary from the fragment cache queue as a semantic fragment and removes the extracted text from the fragment cache queue. If an answer end event is received, the semantic fragment generation unit uses the remaining text in the fragment cache queue as the final semantic fragment. If the fragment cache queue is empty, the semantic fragment generation unit does not generate a new semantic fragment. For example, when the dialogue model continuously returns "Hello", ",", "Welcome to", "store", ";", the segment cache unit sequentially writes the above text content into the segment cache queue; when the segmentation determination unit detects a Chinese comma, it determines whether to generate a shorter semantic segment according to the session configuration; when a Chinese period is detected, the segmentation determination unit determines the segmentation boundary, and the semantic segment generation unit extracts "Hello, welcome to the store;" as a semantic segment; this semantic segment carries the current session identifier, current message sequence number, tracking identifier and segment sequence number, and serves as a common input boundary for subsequent text output, speech synthesis and digital human video driving; The knowledge content storage unit stores the knowledge hit content in the tool response type event; the knowledge hit content is used for log recording, answer generation constraint or subsequent operation analysis; the tool response type event is not directly used as the speech synthesis input, unless the dialogue generation module generates the text content in the answer type event according to the knowledge hit content; In this embodiment, the speech synthesis module is configured to generate a speech segment according to the semantic segment; the speech synthesis module includes a parameter reading unit, a request encapsulation unit, an audio receiving unit and an encoding processing unit; the parameter reading unit reads the language, timbre and emotion parameters according to the session configuration; the session configuration is determined according to the session identifier, store identifier or configuration number; the language is used to determine the language of the synthesized speech; the timbre is used to determine the sounding role of the synthesized speech; the emotion parameter is used to determine the emotional expression of the synthesized speech.

[0028] The request encapsulation unit encapsulates the semantic segment as well as the language, timbre and emotion parameters into a speech synthesis request, and sends the request to the speech synthesis service through a speech synthesis connection; the audio receiving unit receives the audio stream fragments returned by the speech synthesis service; the encoding processing unit converts the audio stream fragments into audio encoded data; if the audio encoded data exceeds the transmission length determined by the session configuration or the system default configuration, the encoding processing unit divides the audio encoded data into a plurality of encoded segments; the encoded segments are still bound to the segment sequence number corresponding to the same semantic segment during subsequent return organization, so as to prevent the terminal from mixing and playing the audio content corresponding to different semantic segments; In this embodiment, the multimodal output module is configured to generate a text segment and a speech segment according to the same semantic segment, generate a digital human video stream address when the video enable flag indicates that video is enabled, and bind the output results corresponding to the same semantic segment with the same segment sequence number; the multimodal output module includes a text output unit, a speech output unit, a video output unit and a return organization unit; The text output unit generates text segments based on semantic segments and configures segment numbers for the text segments; the text segments are used for terminal subtitle display or text display; the speech output unit receives speech segments generated by the speech synthesis module based on the same semantic segments and configures the speech segments with the same segment numbers as the text segments; when the video enable flag indicates that video is enabled, the video output unit sends the speech segments to the digital human video driver service to obtain the digital human video stream address and configures the digital human video stream address with the same segment numbers as the text segments; The digital human video stream address is generated and returned by the digital human video driving service based on the voice segment; after receiving the digital human video stream address, the terminal establishes a playback connection based on the digital human video stream address, and subscribes to the playback connection through the player to obtain continuous video signals; the server configures the digital human video stream address with the same segment number as the text segment and voice segment corresponding to the same semantic segment, and verifies the message sequence number through the result verification module before transmission.

[0029] In this embodiment, the text segment, audio segment, and digital human video stream address corresponding to the same semantic segment use the same segment number; the segment number is assigned by the server according to the generation order of the semantic segments; for the first semantic segment, the server generates the first segment number; for the second semantic segment, the server generates the second segment number; the text segment, audio segment, and digital human video stream address all carry the corresponding segment number, enabling the terminal to match subtitles, audio, and video according to the segment number; In normal voice mode, the video enable flag indicates that video is not enabled. In this mode, the multimodal output module generates text and voice segments and uses them as output results. The return transmission organization unit organizes the text and voice segments according to their segment numbers to obtain the outgoing results to be transmitted back. In digital human video mode, the video enable flag indicates that video is enabled. In this mode, the multimodal output module generates text, voice, and digital human video stream addresses and uses them as output results under the same segment number. The return transmission organization unit organizes the text, voice, and digital human video stream addresses according to their segment numbers to obtain the outgoing results to be transmitted back. In this embodiment, the result verification module is used to verify whether the message sequence number carried by the output result is consistent with the current valid message sequence number before the output result is sent back, and to send back the output result if they are consistent. The result verification module includes a result tag reading unit, a validity judgment unit, a result sending unit, and a result discarding unit. The result tag reading unit reads the session identifier, message sequence number, and fragment sequence number carried by the output result to be sent back. The validity judgment unit reads the current valid message sequence number stored by the session management module according to the session identifier, and judges whether the message sequence number carried by the output result to be sent back is consistent with the current valid message sequence number. If the sequence number of the message to be transmitted is consistent with the current valid sequence number, the result sending unit will send the message to be transmitted to the terminal through a single WebSocket long connection; if the sequence number of the message to be transmitted is inconsistent with the current valid sequence number, the result discarding unit will stop sending the message to be transmitted and record the reason for discarding; the reasons for discarding include terminal reconnection, user interruption of reception, update of the current valid sequence number, or late return of the task.

[0030] Furthermore, the result verification module performs the same message sequence number verification on the recognition results, text segments, speech segments, and digital human video stream addresses. For speech recognition results, the result verification module reads the message sequence number inherited from the current request context. For text segments, speech segments, and digital human video stream addresses, the result verification module reads the message sequence number inherited from their respective semantic segments. Only when the message sequence number carried by the content to be returned matches the currently valid message sequence number stored by the session management module will the server send the content to be returned through a single WebSocket long connection. Thus, even when the speech synthesis service or the digital human video driving service returns results asynchronously, the server can still filter out late results before sending. Through the result verification module, even if the digital human video generation process has already started in the external digital human video driver service and cannot be directly interrupted by this server, when the digital human video driver service returns the digital human video stream address, the server can still determine whether the digital human video stream address belongs to the current valid request based on the message sequence number; if the message sequence number corresponding to the digital human video stream address has expired, the server discards the digital human video stream address, thereby avoiding the old video output from affecting the current session.

[0031] After receiving the output results, the terminal presents them in a multimodal manner according to the segment sequence number. If the terminal receives a text segment first, it displays the subtitles first. If it subsequently receives an audio segment with the same segment sequence number, it plays the audio segment. If the video enable flag indicates that video is enabled and the terminal receives a digital human video stream address with the same segment sequence number, it establishes a playback connection based on the digital human video stream address and plays the digital human video. If the results of different segment sequence numbers arrive in different orders, the terminal caches and sorts them according to the segment sequence number to avoid misalignment of subtitles, audio, and digital human video. In this embodiment, the configuration and logging module is used to read session configuration and record interaction link logs. The configuration and logging module includes a configuration reading unit, a tracking identifier generation unit, a log recording unit, a log association unit, and a permission control unit. The configuration reading unit reads the speech recognition language, speech synthesis timbre, speech synthesis language, and speech synthesis emotion parameters according to the session identifier, store identifier, or configuration number. The tracking identifier generation unit generates a tracking identifier for a dialogue link. The tracking identifier is used to associate the speech recognition result, semantic segment, speech segment, digital human video stream address, knowledge hit content, and abnormal information in the same dialogue link. The log recording unit records session identifier, tracking identifier, message sequence number, segment sequence number, user question, answer content, knowledge hit content, speech recognition time, speech synthesis time, video generation time, and exception information. Speech recognition time records the time from sending an audio segment to the speech recognition service to receiving the recognition result. Speech synthesis time records the time from submitting a semantic segment to the speech synthesis service to receiving audio stream fragments. Video generation time records the time from submitting a speech segment to the digital human video driver service to obtaining the digital human video stream address. Exception information records connection errors, speech recognition failures, speech synthesis failures, video generation failures, result discarding due to inconsistent message sequence numbers, and other processing anomalies. The log association unit associates the recognition results, semantic segments, speech segments, and digital human video stream addresses in the same dialogue link based on the tracking identifier. Through the above log association method, the server can track each processing stage in a complete dialogue, which makes it easier to locate the cause of speech recognition delay, response generation delay, speech synthesis delay, video generation delay, or result discarding.

[0032] The access control unit restricts access to the log content recorded by the log recording unit according to access control rules; real-time logs are only accessible to accounts with administrative privileges; for terminal devices permanently deployed in the site, the log recording unit does not collect user identity information; for terminals integrated into applications, the log recording unit records user identifiers, and access to detailed user information requires additional permission verification; the access control unit can prevent unauthorized personnel from accessing user question content, user identifiers, or dialogue logs. In this embodiment, the knowledge base invocation module is used to invoke the corresponding knowledge base according to the user's question during the dialogue generation process. The knowledge base invocation module includes a structured input generation unit, a knowledge base selection unit, a retrieval result judgment unit, and an answer constraint unit. The structured input generation unit generates structured input content based on the user's question, tracking identifier, store identifier, and user identifier. The structured input content includes the user's question text, tracking identifier, store identifier, and user identifier. The store identifier is used to determine the store knowledge base corresponding to the current user's question, and the user identifier is used to distinguish the user context in a business system with the appropriate permissions. The knowledge base selection unit selects the corresponding knowledge base based on the store identifier or geographic intent in the structured input content; the geographic intent is determined by the store name, city name, region name, or preset geographic location keywords in the user question; when the user question does not contain the store name, city name, region name, or preset geographic location keywords, the knowledge base selection unit selects the corresponding knowledge base based on the store identifier in the structured input content; when the user question contains the store name, city name, region name, or preset geographic location keywords, the knowledge base selection unit selects the corresponding knowledge base based on the store or region corresponding to that keyword; The retrieval result judgment unit compares the retrieval results returned by the knowledge base with the minimum matching conditions determined by the session configuration. The minimum matching conditions include the matching score returned by the knowledge base being no lower than the minimum matching score in the session configuration, or returning a valid knowledge entry when the knowledge base service does not return a matching score. The valid knowledge entry refers to the retrieval results returned by the knowledge base that contain text content that can be used to generate an answer. When the retrieval results meet the minimum matching conditions, the answer constraint unit provides the retrieval results to the dialogue generation module. When the retrieval results do not meet the minimum matching conditions, the answer constraint unit generates content for switching to human prompts and provides it to the dialogue generation module.

[0033] To more clearly illustrate the working process of this invention, the following is a complete explanation in conjunction with a voice question-and-answer interaction; After the terminal starts, it initiates a connection request to the server through a single WebSocket long connection, carrying a session identifier in the connection request. After receiving the connection request, the unified protocol access module submits the session identifier to the session management module. The session management module checks whether there is a saved terminal connection based on the session identifier. If not, it saves the current terminal connection as the terminal connection corresponding to the session identifier. If there is already a saved terminal connection, it closes the saved terminal connection and saves the current terminal connection as a new terminal connection. The session management module also initializes the current valid message sequence number. When the user begins to speak, the terminal sends a speech recognition start request through a single WebSocket long connection; the interface type in the unified protocol message indicates the speech recognition start request, and the unified protocol message carries a session identifier and message sequence number; the message distribution module distributes the request to the speech recognition module according to the interface type; the speech recognition module establishes a speech recognition connection according to the session identifier, and reads the audio format, sampling rate, bit depth, number of channels and recognition language according to the session configuration; During continuous user speech, the terminal encodes the collected audio data into an audio payload and continuously sends it to the server via an audio data upload request. The message distribution module sends the audio payload to the speech recognition module. The speech recognition module first cleans the audio data prefixes, spaces, and newline characters from the audio payload, then decodes it to obtain the audio byte stream. Subsequently, the speech recognition module calculates the segment length based on the sampling rate, bit depth, number of channels, and segment duration, and divides the audio byte stream into multiple audio segments according to the segment length. The speech recognition module configures an incrementing sequence number for each audio segment and encapsulates the audio segment into a protocol message that the speech recognition service can receive. When the terminal sends a request to end speech recognition, the message distribution module triggers the speech recognition module to send an audio segment carrying the final packet identifier; after the speech recognition service returns the intermediate and final recognition results, the speech recognition module converts the recognition results into a unified business protocol result; if the recognition result needs to be displayed to the terminal, the result verification module checks whether the message sequence number carried by the recognition result is consistent with the current valid message sequence number, and sends the recognition result to the terminal if they are consistent; if the recognition result is used as input for subsequent dialogue, the message distribution module sends the recognized text to the dialogue generation module. After receiving the recognized text, the dialogue generation module generates a tracking identifier and reads the dialogue configuration based on the session identifier, store identifier, or configuration number. The dialogue generation module sends the recognized text to the dialogue model and receives the response-type events continuously returned by the dialogue model. The text content in the response-type events is written to the segment cache queue in sequence. The segmentation determination unit detects the text content in the segment cache queue. When it detects segmentation punctuation, natural sentence break events, no content duration reaching the session configuration, cached text length reaching the session configuration, or receives a response end event, it determines the segmentation boundary. The semantic segment generation unit extracts the text before the segmentation boundary from the segment cache queue, generates a semantic segment, and carries the current message sequence number and segment sequence number for the semantic segment. After receiving a semantic segment, the speech synthesis module reads the language, timbre, and emotion parameters according to the session configuration and encapsulates the semantic segment into a speech synthesis request. After the speech synthesis service returns audio stream segments, the speech synthesis module converts the audio stream segments into audio encoded data and divides the audio encoded data into multiple encoded segments when the audio encoded data exceeds the sending length determined by the session configuration or the system default configuration. The multimodal output module generates text segments and speech segments based on the same semantic segment and configures the same segment sequence number for the text segments and speech segments. If the video enable flag indicates that video is not enabled, the multimodal output module organizes text and audio segments according to segment sequence numbers and generates a result to be transmitted back. Before transmitting the output result back, the result verification module reads the message sequence number carried in the output result and determines whether the message sequence number is consistent with the current valid message sequence number. If they are consistent, the result is transmitted back through a single WebSocket long connection. If they are inconsistent, the output result is discarded. If the video enable flag indicates that video is enabled, the multimodal output module sends the audio segment to the digital human video driver service; the digital human video driver service generates a digital human video stream address based on the audio segment; the video output unit configures the same segment sequence number as the text segment for the digital human video stream address; the return organization unit organizes the text segment, audio segment, and digital human video stream address according to the segment sequence number, and generates the result to be returned; the result verification module still verifies whether the message sequence number carried by the output result is consistent with the current valid message sequence number before the return, and only if they are consistent will the result be returned through a single WebSocket long connection. If a user actively interrupts the session before all responses are provided, the terminal sends an interruption request to the server or disconnects. The session management module updates the current valid message sequence number. Subsequently, if the speech synthesis service returns a speech segment corresponding to the old message sequence number, or the digital human video driving service returns a digital human video stream address corresponding to the old message sequence number, the result verification module will determine that the message sequence number is inconsistent with the current valid message sequence number and discard the corresponding result. Through the above processing, even if the speech synthesis or digital human video generation task continues to be executed in an external service, the old result will not be sent to the current session.

[0034] This embodiment also provides a direct text input processing method; the terminal sends a dialogue request through a unified protocol message, with the user text carried in the payload data; the message distribution module sends the user text to the dialogue generation module according to the interface type; the dialogue generation module generates a streaming response text based on the user text, and executes the process according to the aforementioned segment caching, segmentation judgment, semantic segment generation, speech synthesis, multimodal output, and result verification; thus, this system can simultaneously support voice input and text input, and both input methods reuse the same semantic segment generation mechanism, segment sequence number binding mechanism, and message sequence number verification mechanism; It should be noted that the dialogue model is used to generate streaming response text. The focus of this embodiment is not on the training process or model structure of the dialogue model itself, but on the server's reception, classification, caching, segmentation, and multimodal output organization of the streaming events returned by the dialogue model. After receiving a response-type event, the server writes the text content therein into the segment cache queue. After receiving a tool response-type event, the server saves the knowledge hit content therein to the knowledge content storage unit. The server generates semantic segments based on the segmentation boundaries and sends the semantic segments to the speech synthesis module and the multimodal output module. Thus, even if the dialogue model, knowledge base, or tool service is replaced, as long as it can return response-type events and tool response-type events, the streaming segmentation and multimodal synchronous output process of this embodiment can still be executed.

[0035] In this embodiment, the prompt word configuration is read by the configuration and log module as part of the session configuration; the prompt word configuration is used to limit the response role, response tone, response length, unit expression method and knowledge base call rules of the dialogue generation module; the prompt word configuration does not change the core processing link of this system, and the unified protocol access module, message distribution module, session management module, speech recognition module, dialogue generation module, speech synthesis module, multimodal output module and result verification module still complete speech recognition, dialogue generation, speech synthesis and digital human output; In this embodiment, each module is deployed as a software program on the same server, or on multiple servers according to the workload; the unified protocol access module and session management module are deployed in the access service, the speech recognition module, speech synthesis module and multimodal output module are deployed in the capability orchestration service, and the configuration and logging module is deployed in the business configuration and logging service; regardless of whether a single-machine deployment or a distributed deployment is adopted, as long as the data association relationship in the same session link is maintained through session identifier, message sequence number, fragment sequence number and tracking identifier, the intelligent customer service system described in this embodiment can be realized; In this embodiment, a single WebSocket long connection is not limited to transmitting only one type of message. Instead, different business actions are distinguished by the interface type in the unified protocol message. The terminal does not need to establish separate connections with the speech recognition service, dialogue generation service, speech synthesis service, and digital human video driving service. After receiving the terminal's request, the server completes internal message distribution according to the interface type, converts the results returned by each third-party service into a unified business protocol result, and then returns it to the terminal through a single WebSocket long connection. Thus, the terminal only needs to send and receive messages according to the unified protocol message, reducing the complexity of terminal access. In this embodiment, semantic segments serve as the common basis for generating text segments, speech segments, and digital human video stream addresses. Semantic segments are not merely used for text display but also serve as the input boundary for speech synthesis and digital human video driving. By generating text segments, speech segments, and digital human video stream addresses through the same semantic segment and binding the output results with the same segment sequence number, the terminal can maintain consistency in subtitles, speech playback, and digital human video presentation during streaming output. Compared to waiting for the entire response to be completed before unified playback, this embodiment can generate speech segments or digital human video stream addresses immediately after forming a semantic segment. Compared to word-by-word playback, this embodiment can ensure that each output segment has relatively complete semantics, thereby reducing the problems of speech fragmentation and unnatural lip movements in digital humans. In this embodiment, the message sequence number is used to determine whether the output result belongs to the current valid request, the fragment sequence number is used to determine the order and modal correspondence of different semantic fragments within the same request, and the tracking identifier is used to determine the association between different module logs and asynchronous tasks under the same dialogue link. The message sequence number, fragment sequence number and tracking identifier play different roles, and the three together ensure session isolation, fragment synchronization and link traceability.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An intelligent customer service system integrating speech recognition, dialogue generation, and digital human output, characterized in that, include: The unified protocol access module is used to receive unified protocol messages sent by the terminal through a single WebSocket long connection. The unified protocol message includes interface type, session identifier, message sequence number, payload data and video enable flag. The payload data includes audio payload or user text. The message distribution module is used to distribute the payload data to the speech recognition module, dialogue generation module, or speech synthesis module according to the interface type. The session management module is used to establish a mapping between terminal connection and speech recognition connection, speech synthesis connection and digital human video task association information based on the session identifier, and to save the current valid message sequence number; The speech recognition module is used to clean, decode, and segment the audio load to obtain audio segments, and to obtain recognized text based on the audio segments; The dialogue generation module is used to generate streaming response text based on the identified text or the user text, and to cache and segment the streaming response text to obtain semantic fragments carrying the message sequence number. A speech synthesis module is used to generate speech segments based on the semantic segments; The multimodal output module is used to generate text and speech segments based on the same semantic segment, and generate a digital human video stream address when the video enable flag indicates that the video is enabled, and bind the output results corresponding to the same semantic segment to the same segment number; The result verification module is used to verify whether the message sequence number carried by the output result is consistent with the current valid message sequence number before the output result is sent back, and to send back the output result if they are consistent.

2. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The session management module includes: A connection storage unit is used to save the terminal connection object according to the session identifier; The repeated access processing unit is used to close the saved terminal connection and save the new terminal connection as the current terminal connection when the same session identifier is detected to be accessed again. The message sequence number update unit is used to update the currently valid message sequence number when the terminal reconnects or the user interrupts receiving; The expired result filtering unit is used to discard recognition results, voice segments, or digital human video stream addresses that do not match the currently valid message sequence number.

3. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The speech recognition module includes: An audio cleaning unit is used to remove audio data prefixes, spaces, and newline characters from the audio load to obtain cleaned audio data. An audio decoding unit is used to decode the cleaned audio data to obtain an audio byte stream; An audio segmentation unit is used to determine the segment length based on the sampling rate, bit depth, number of channels, and segment duration, and to divide the audio byte stream into multiple audio segments according to the segment length; The segment identification unit is used to configure an incrementing sequence number for each audio segment and to configure a final packet identifier for the last audio segment. The recognition result conversion unit is used to convert the intermediate recognition result and the final recognition result obtained from the audio segment into a unified business protocol result. A recognition response receiving unit is used to receive a recognition response returned by the speech recognition service based on the audio segment; A text parsing unit is used to parse intermediate recognized text, final recognized text, and final packet status from the recognized response; The recognition text output unit is used to send the final recognized text as the recognized text to the dialogue generation module when the final packet state indicates that the speech recognition has ended.

4. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The message distribution module includes: The identification start distribution unit is used to trigger the speech recognition module to establish a speech recognition connection when the interface type is speech recognition start request; An audio data distribution unit is used to send the payload data to the speech recognition module when the interface type is an audio data upload request. The identification end distribution unit is used to trigger the speech recognition module to send an audio segment carrying the final packet identifier and release the corresponding speech recognition connection when the interface type is a speech recognition end request; A dialogue distribution unit is used to send the identified text or the user text to the dialogue generation module when the interface type is a dialogue request. The synthesis and distribution unit is used to send the text to be synthesized to the speech synthesis module when the interface type is a speech synthesis request.

5. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The dialogue generation module includes: The streaming event receiving unit is used to receive response-type events and tool response-type events continuously returned by the dialogue model; A fragment caching unit is used to write the text content of the answer-type event into a fragment caching queue; The segmentation determination unit is used to determine segmentation boundaries when segmentation punctuation, natural sentence break events, the duration of no content determined by the session configuration, the maximum cache length determined by the session configuration, or the end of response event are detected. A semantic fragment generation unit is used to extract the text before the segment boundary from the fragment cache queue as a semantic fragment, and to take the remaining text under the answer end event as the final semantic fragment. The knowledge content storage unit is used to store the knowledge hit content in the tool response events.

6. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The multimodal output module includes: A text output unit is used to generate text segments based on the semantic segments and to configure segment numbers for the text segments. A speech output unit is configured to receive a speech segment generated by the speech synthesis module based on the semantic segment, and to assign the speech segment the same segment number as the text segment. The video output unit is configured to send the audio segment to the digital human video driver service when the video enable flag indicates that video is enabled, obtain the digital human video stream address, and configure the digital human video stream address with the same segment number as the text segment; The return organization unit is used to organize the text segment, the audio segment, and the digital human video stream address at the time of generation according to the segment sequence number, and obtain the output result.

7. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The speech synthesis module includes: The parameter reading unit is used to read language, timbre, and emotion parameters according to the session configuration. A request encapsulation unit is used to encapsulate the semantic fragment and the language, timbre and emotion parameters into a speech synthesis request; An audio receiving unit is used to receive audio stream fragments returned by a speech synthesis service; The encoding processing unit is used to convert the audio stream segments into audio encoded data, and when the audio encoded data exceeds the transmission length determined by the session configuration or the system default configuration, divide the audio encoded data into multiple encoded segments.

8. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, The result verification module includes: The result tag reading unit is used to read the session identifier, message sequence number, and fragment sequence number carried in the outgoing result to be transmitted back; The validity determination unit is used to read the current valid message sequence number according to the session identifier, and determine whether the message sequence number carried by the outgoing result to be transmitted back is consistent with the current valid message sequence number; The result sending unit is used to send the result to be transmitted back to the terminal through the single WebSocket long connection when the judgment result is consistent; The result discarding unit is used to stop sending the outgoing result to be returned when the result is determined to be inconsistent, and to record the reason for discarding.

9. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 1, characterized in that, It also includes a configuration and logging module, which includes: The configuration reading unit is used to read the speech recognition language, speech synthesis timbre, speech synthesis language type, and speech synthesis emotion parameters based on the session identifier, store identifier, or configuration number. The tracking identifier generation unit is used to generate a tracking identifier for a single dialogue link; The log recording unit is used to record session identifiers, tracking identifiers, message sequence numbers, segment sequence numbers, user questions, answer content, knowledge hit content, speech recognition time, speech synthesis time, video generation time, and abnormal information. The log association unit is used to associate the identification results, semantic segments, voice segments and digital human video stream addresses in the same dialogue link according to the tracking identifier; The access control unit is used to restrict access permissions to the log content recorded by the log recording unit in accordance with access control rules.

10. The intelligent customer service system integrating speech recognition, dialogue generation, and digital human output according to claim 5, characterized in that, It also includes a knowledge base invocation module, which includes: The structured input generation unit is used to generate structured input content based on user questions, tracking identifiers, store identifiers, and user identifiers. The knowledge base selection unit is used to select the corresponding knowledge base based on the store identifier or geographical intent in the structured input content. The retrieval result judgment unit is used to compare the retrieval results returned by the knowledge base with the minimum matching conditions determined by the session configuration; The response constraint unit is used to provide the search results to the dialogue generation module when the search results meet the minimum matching conditions, and to generate a prompt to switch to human assistance and provide it to the dialogue generation module when the search results do not meet the minimum matching conditions.

Images