Digital human interaction method and system, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a digital human interaction method and system, a storage medium and a program product, which can associate a user with an interaction process by adding a target session identifier to a first voice request, ensure context coherence of multiple rounds of conversations, and realize full duplex interaction. Furthermore, the voice signal is converted into the processable target text, the command task in the target text is identified, and the first command task text is generated, so that the intention of the user can be accurately identified, the task type can be distinguished, and the semantic understanding efficiency is improved. Furthermore, through parallel processing of text sentence segmentation and speech synthesis, the mechanical feeling of single word output is avoided, the waiting time of the user is shortened, and the interaction naturalness is improved. Furthermore, the target command task voice and the second command task text are sent to the client, so that bimodal synchronous output is realized, the interaction naturalness is improved, multi-scene adaptation is realized, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a digital human interaction method, system, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence, digital humans have been widely used in customer service, education, healthcare, entertainment, and other fields. However, most digital humans still have the following problems in their interaction methods:

[0003] (1) Traditional digital human interaction mostly adopts a single modality (pure text or pure voice), which has poor scene adaptability, cannot adapt to the needs of users in multiple scenarios, and cannot dynamically switch or integrate multimodal input according to the scenario, resulting in a limited user experience.

[0004] (2) Traditional digital humans adopt a half-duplex mode, which cannot support bidirectional real-time parallel interaction; or they adopt a one-way streaming output, which cannot be interrupted or followed up before the output is completed, and the rounds need to be switched, resulting in low dialogue efficiency.

[0005] (3) Traditional digital humans have high fusion latency because the speech recognition, semantic understanding and speech synthesis modules are executed sequentially and cannot be processed in parallel in real time. This results in a large overall response delay.

[0006] Therefore, there is an urgent need to propose a digital human interaction method that can achieve parallel processing of text and speech modalities, support full-duplex interaction, and improve the naturalness of interaction and response efficiency. Summary of the Invention

[0007] In view of this, the present invention provides a digital human interaction method, system, storage medium and program product to solve the problems of limited user experience, low dialogue efficiency and large overall response delay in traditional digital human interaction.

[0008] In a first aspect, the present invention provides a digital human interaction method for a digital human interaction system connected to a client; the method includes:

[0009] Upon receiving the first voice request from the client, a target session identifier is added to the first voice request to obtain the target voice request; speech recognition and processing are performed on the target voice request to obtain the target text; the command task in the target text is recognized, and the first command task text is generated based on the recognition result; the first command task text is processed by text segmentation and speech synthesis to obtain the second command task text and the target command task speech; the target command task speech and the second command task text are sent to the client.

[0010] The digital human interaction method provided by this invention, by adding a target session identifier to the first voice request, can associate the user with the interaction process, ensuring the coherence of multi-turn dialogue context, realizing full-duplex interaction, and supporting personalized output, thus solving the problem of chaotic session management in traditional digital human systems. Furthermore, converting the voice signal into processable text provides a foundation for subsequent semantic understanding and improves the accuracy of voice interaction. Furthermore, identifying the command task in the target text and generating the first command task text accurately identifies user intent, distinguishes task types, provides a basis for targeted responses, and improves semantic understanding efficiency. Furthermore, text segmentation avoids lengthy output. Simultaneously, speech synthesis enables natural speech responses, solving the problems of mechanical or high-latency traditional output. Furthermore, parallel processing of text segmentation and speech synthesis avoids the mechanical feel of single-word output, shortens user waiting time, and improves the naturalness of interaction. Furthermore, sending the target command task voice and the second command task text to the client achieves bimodal synchronous output, improving the naturalness of interaction and conforming to user dialogue habits. Therefore, by implementing this invention, multi-scenario adaptation is achieved and the user experience is improved by supporting voice request input and bimodal output of text and voice.

[0011] In one optional implementation, the target voice request is identified and processed to obtain target text, including: identifying the target voice request to obtain initial text; and cleaning and denoising the initial text to obtain target text.

[0012] The digital human interaction method provided by this invention recognizes target voice requests, achieving speech-to-text conversion and providing raw text material for subsequent processing. Furthermore, by cleaning and denoising the initial text, irrelevant characters and noise are removed, and the text format is standardized, providing clean input for subsequent command task recognition and improving the accuracy of semantic understanding.

[0013] In one alternative implementation, the target voice request is recognized to obtain initial text, including:

[0014] The target speech request is denoised and framed to obtain multiple speech signals; the Mel frequency cepstral coefficient algorithm is used to extract features from the multiple speech signals to obtain multiple acoustic features; the FunASR model is used to process the multiple acoustic features to obtain the initial text.

[0015] The digital human interaction method provided by this invention improves the quality of the speech signal through noise reduction and frame segmentation, reducing the impact of environmental interference on the recognition results. Furthermore, it utilizes the Mel-frequency cepstral coefficient algorithm for feature extraction, converting the speech signal into machine-recognizable feature parameters. Moreover, it employs the FunASR model to process multiple acoustic features, achieving efficient speech recognition, improving recognition accuracy, and reducing latency.

[0016] In one optional implementation, the command task in the target text is identified, and a first command task text is generated based on the identification result, including:

[0017] The command task in the target text is identified by using a pre-set large model and retrieval enhancement generation technology. The command task is then obtained. The pre-set large model is used to determine whether the command task is a complete slot command task. When the command task is a complete slot command task, the pre-set large model is used to process the command task and generate the first command task text.

[0018] The digital human interaction method provided by this invention enhances semantic understanding and improves the accuracy of complex command recognition by combining a pre-set large model and retrieval-enhanced generation technology to identify command tasks in target text. Furthermore, by determining whether a command task is a complete slot command task, the completeness of task information can be distinguished, avoiding invalid interactions and improving dialogue efficiency. Moreover, if it is a complete slot command task, the first command task text is directly generated using the pre-set large model, shortening the processing flow and reducing user waiting time.

[0019] In one alternative implementation, the method further includes:

[0020] When the command task is a non-complete slot command task, the command task is processed using a preset large model to generate the second command task text; the first command task text is processed by text segmentation and speech synthesis to obtain the third command task text and the initial command task speech; the third command task text and the initial command task speech are sent to the client so that the client sends a second speech request to the digital human interaction system; when the second speech request is received, the process is iterated repeatedly until the first command task text is obtained.

[0021] The digital human interaction method provided by this invention generates a second command task text if the command task is an incomplete slot command task, and guides the user to supplement information through bimodal output. Furthermore, through repeated iterations, it supports multi-turn dialogue to complete information, ensuring the task is ultimately executed effectively and optimizing the processing experience of incomplete tasks.

[0022] In one optional implementation, the first command task text is processed by text segmentation and speech synthesis to obtain the second command task text and the target command task speech, including:

[0023] The first command task text is processed by sentence segmentation to obtain the fourth command task text; the fourth command task text is processed by grammatical structure parsing and prosodic prediction to obtain the fifth command task text containing multiple grammatical information and multiple intonation change parameters; the fifth command task text is processed using preset speech synthesis technology and preset speech model to generate multiple basic speech features; the multiple basic speech features are converted using a vocoder to obtain the target command task speech.

[0024] The digital human interaction method provided by this invention avoids lengthy output and balances fluency and response speed by segmenting the first command task text. Furthermore, by performing grammatical structure analysis and prosodic prediction on the fourth command task text, the speech synthesis conforms to language rules and emotional expression, improving the naturalness of the speech. Moreover, by generating high-quality speech waveforms through a vocoder, the mechanical and disjointed problems of traditional speech synthesis are solved.

[0025] In one alternative implementation, the method further includes:

[0026] When a text request is received from the client, the command task in the text request is identified, and a first command task text is generated based on the identification result; the first command task text is processed by text segmentation and speech synthesis to obtain a second command task text and a target command task speech; the target command task speech and the second command task text are sent to the client.

[0027] The digital human interaction method provided by this invention supports text input modality, overcoming the limitations of traditional single-modality digital humans and enhancing scene adaptability. Furthermore, by supplementing the processing flow with support for text requests, it solves the limitations of traditional single-modality digital human input, achieving unified processing of text and voice input, further enhancing scene adaptability.

[0028] In a second aspect, the present invention provides a digital human interaction system for executing the digital human interaction method described in the first aspect or any corresponding embodiment thereof, the system being connected to a client; the system comprising:

[0029] The system comprises a session management module, a speech recognition processing module, a command task recognition module, a streaming window dynamic control module, and a speech synthesis module. The session management module, upon receiving a first speech request from a client, adds a target session identifier to the first speech request to obtain the target speech request, and then sends the target speech request to the speech recognition module. The speech recognition module performs speech recognition and processing on the target speech request to obtain the target text, and then sends the target text to the command task recognition module. The command task recognition module recognizes the command tasks in the target text, generates the first command task text based on the recognition results, and sends the first command task text to both the streaming window dynamic control module and the speech synthesis module. The speech synthesis module performs speech synthesis processing on the first command task text to obtain the target command task speech, and then sends the target command task speech to the streaming window dynamic control module. The streaming window dynamic control module performs text segmentation processing on the first command task text to obtain the second command task text, and sends the second command task text to the client, as well as the target command task speech to the client.

[0030] The digital human interaction system provided by this invention, through a session management module, associates users and requests, ensuring interaction continuity and supporting personalized services. Furthermore, a speech recognition processing module achieves accurate speech-to-text conversion, providing reliable input for semantic understanding. Further, a command task recognition module accurately identifies user intent and task completeness, improving response targeting. Further, a speech synthesis module generates natural and fluent speech, enhancing the naturalness and expressiveness of voice interaction. Furthermore, a streaming window dynamic control module enables segmented streaming output of text and hidden text, achieving simultaneous dual-modal output, optimizing the reading experience, shortening waiting time, improving interaction naturalness, and conforming to user dialogue habits. Therefore, by implementing this invention, and by supporting voice request input and dual-modal output of text and speech, multi-scenario adaptation is achieved, improving the user experience.

[0031] Thirdly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the digital human interaction method described in the first aspect or any corresponding embodiment thereof.

[0032] Fourthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the digital human interaction method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0033] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a structural block diagram of a digital human interaction system according to an embodiment of the present invention;

[0035] Figure 2 This is a flowchart illustrating a digital human interaction method according to an embodiment of the present invention;

[0036] Figure 3 This is a flowchart illustrating another digital human interaction method according to an embodiment of the present invention;

[0037] Figure 4 This is a flowchart illustrating another digital human interaction method according to an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of a microservice system architecture according to an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of a text and voice dual-modal intelligent interaction process according to an embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] This invention provides a digital human interaction method that supports voice request input and bimodal output of text and voice, enabling multi-scenario adaptation and improving user experience.

[0043] According to an embodiment of the present invention, a digital human interaction method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] This embodiment provides a digital human interaction method, which can be used for, for example Figure 1 The digital human interaction system 1 shown is connected to the client 2 and includes: a session management module 11, a speech recognition processing module 12, a recognition command task module 13, a streaming window dynamic control module 14, and a speech synthesis module 15.

[0045] This digital human interaction system 1 employs WebSocket (a network communication protocol) full-duplex communication technology to replace SSE's unidirectional streaming output. WebSocket, based on TCP (Transmission Control Protocol) full-duplex communication, establishes a persistent connection with a single handshake, enabling real-time bidirectional data transmission between the client and server. Furthermore, users can ask questions via text or voice at any time while listening to voice or viewing text output, without waiting for the current output to complete. In contrast, SSE only allows unidirectional communication; if a user needs to interrupt the output, network communication must be terminated, resulting in high network overhead. Moreover, SSE cannot handle binary data, making its flexibility and efficiency far inferior to WebSocket.

[0046] Furthermore, this digital human interaction system 1 adopts the Asyncio asynchronous communication framework to handle four types of asynchronous concurrent network input and output operations: text input requests, voice input requests, text output, and voice output. The Asyncio asynchronous communication mechanism is based on an event-driven programming model. It solves the synchronous blocking problem by utilizing the Event Loop mechanism, avoids the overhead of multi-threaded switching by leveraging the Coroutine model, and solves high-latency I / O operations through system-level I / O multiplexing. This enables it to support tens of thousands of concurrent connection sets and I / O operations, significantly improving the system's concurrent processing capabilities and real-time response efficiency.

[0047] Figure 2 This is a flowchart of a digital human interaction method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0048] Step S201: When the first voice request sent by the client is received, a target session identifier is added to the first voice request to obtain the target voice request.

[0049] Specifically, the digital human interaction system 1 listens for the first voice request sent by the client via the WebSocket full-duplex communication protocol (port 10001). Furthermore, the first voice request is transmitted in the form of a binary voice stream, containing the user's voice content (such as "check tomorrow's weather").

[0050] Furthermore, the handle_client function of the digital human interaction system 1 asynchronously receives the voice request, parses the client identifier (such as the cliSocket connection object) in the request, and confirms the uniqueness of the request source.

[0051] Furthermore, when the session management module 11 receives the first voice request, it can generate a unique session identifier, namely the target session identifier, for the first voice request to distinguish between different users or different sessions of the same user.

[0052] Furthermore, client connections can be bound to session identifiers through mapping relationships:

[0053] (1) Record client2Id[cliSocket] = talkId(client connection → session identifier);

[0054] (2) Record clientMap[talkId] = cliSocket(session identifier → client connection) to ensure that subsequent messages can be accurately sent back to the corresponding client.

[0055] Furthermore, the generated talkId (target session identifier) ​​is embedded into the original voice request to form a corresponding target voice request. Moreover, this target voice request not only contains the user's voice content but also carries the session identifier, achieving a binding of "voice content + session context".

[0056] Step S202: Perform speech recognition and processing on the target speech request to obtain the target text.

[0057] Specifically, when the speech recognition processing module 12 receives the target speech request sent by the session management module 11, it can perform speech-to-text and text optimization processing on the target speech request carrying the target session identifier, and finally obtain the standardized text that can be used for subsequent semantic understanding, namely the target text.

[0058] Furthermore, by converting speech signals into processable text, a foundation is provided for subsequent semantic understanding, thus improving the accuracy of voice interaction.

[0059] Step S203: Identify the command task in the target text and generate the first command task text based on the identification result.

[0060] Specifically, when the command task recognition module 13 receives the target text sent by the speech recognition processing module 12, it can perform semantic understanding on the target text to identify whether it contains the user's command task (such as a query, reservation, or other request with a clear goal), and generate the corresponding response text, i.e., the first command task text, based on the completeness of the task information.

[0061] Furthermore, by identifying the command tasks in the target text and generating the first command task text, the user's intent can be accurately identified, the task type can be distinguished, and a basis for targeted response can be provided, thus improving the efficiency of semantic understanding.

[0062] Step S204: Perform text segmentation and speech synthesis processing on the first command task text to obtain the second command task text and the target command task speech.

[0063] Specifically, when the streaming window dynamic control module 14 receives the first command task text sent by the recognition command task module 13, it can perform text segmentation processing on the generated first command task text and generate segmented text suitable for streaming output, namely the second command task text. Text segmentation processing can avoid lengthy output.

[0064] Simultaneously, the first command task text can be sent to the speech synthesis module 15, where the generated first command task text is processed by speech synthesis to convert it into natural and fluent speech, i.e., the target command task speech. Speech synthesis can achieve natural speech response, solving the problems of mechanical output or high latency in traditional methods.

[0065] Step S205: Send the target command task voice and the second command task text to the client.

[0066] Specifically, the streaming window dynamic control module 14 can send the generated second command task text to the client 2. Simultaneously, when the streaming window dynamic control module 14 receives the target command task voice sent by the speech synthesis module 15, it synchronously sends the target command task voice to the client 2.

[0067] The streaming window dynamic control module 14 can extract the corresponding target session identifier (talkId) from the second command task text and the target command task voice, and locate the corresponding user-end WebSocket connection (cliSocket) through clientMap[talkId] to ensure that the information is accurately sent to the user who initiated the request.

[0068] Furthermore, during transmission, the event loop mechanism of the Asyncio asynchronous communication framework can be utilized to process text and voice data asynchronously, and push them to client 2 simultaneously via the WebSocket send method:

[0069] (1) Second command task text: The segmented text content is carried in structured data (such as JSON format), and the client 2 supports real-time display of streaming text;

[0070] (2) Target command task voice: transmitted in the form of binary audio stream, supporting real-time playback of voice segments by client 2.

[0071] Furthermore, since WebSocket maintains a persistent connection, users can send new voice or text requests at any time during the push process (such as interrupting the current output). At this time, the system identifies the new request through session state tracking, pauses the current push, and prioritizes the processing of the new request, thereby achieving full-duplex real-time interaction.

[0072] The digital human interaction method provided in this embodiment, by adding a target session identifier to the first voice request, can associate the user with the interaction process, ensuring the coherence of multi-turn dialogue context, realizing full-duplex interaction, and supporting personalized output, thus solving the problem of chaotic session management in traditional digital human systems. Furthermore, converting the voice signal into processable text provides a foundation for subsequent semantic understanding and improves the accuracy of voice interaction. Furthermore, recognizing the command task in the target text and generating the first command task text accurately identifies user intent, distinguishes task types, provides a basis for targeted responses, and improves semantic understanding efficiency. Furthermore, text segmentation avoids lengthy output. Simultaneously, speech synthesis enables natural speech responses, solving the problems of mechanical or high-latency traditional output. Furthermore, parallel processing of text segmentation and speech synthesis avoids the mechanical feel of single-word output, shortens user waiting time, and improves the naturalness of interaction. Furthermore, sending the target command task voice and the second command task text to the client achieves bimodal synchronous output, improving the naturalness of interaction and conforming to user dialogue habits. Therefore, by implementing this invention, multi-scenario adaptation is achieved and the user experience is improved by supporting voice request input and bimodal output of text and voice.

[0073] This embodiment provides a digital human interaction method, which can be used for, for example Figure 1 The digital human interaction system 1 shown is connected to the client 2, and includes: a session management module 11, a speech recognition processing module 12, a recognition command task module 13, a streaming window dynamic control module 14, and a speech synthesis module 15.

[0074] Figure 3 This is a flowchart of a digital human interaction method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0075] Step S301: Upon receiving the first voice request from the client, add a target session identifier to the first voice request to obtain the target voice request. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0076] Step S302: Perform speech recognition and processing on the target speech request to obtain the target text.

[0077] Specifically, step S302 includes:

[0078] Step S3021: Recognize the target voice request to obtain the initial text.

[0079] Specifically, by recognizing the target voice request, the speech-to-text conversion was achieved, providing raw text material for subsequent processing.

[0080] In some optional implementations, step S3021 above includes:

[0081] Step a1: Denoise reduction and frame segmentation are performed on the target voice request to obtain multiple voice signals.

[0082] Step a2: Use the Mel frequency cepstral coefficient algorithm to extract features from multiple speech signals to obtain multiple acoustic features.

[0083] Step a3: Use the FunASR model to process multiple acoustic features to obtain the initial text.

[0084] Among them, the Mel-Frequency Cepstral Coefficients (MFCC) algorithm represents an algorithm that converts the time-domain features of a speech signal into frequency-domain features based on the human ear's perception characteristics of sound frequencies (sensitive to low frequencies and insensitive to high frequencies).

[0085] Furthermore, the FunASR model represents an automatic speech recognition model based on end-to-end technology, used to convert speech signals into text.

[0086] Specifically, environmental noise (such as background noise, current noise, etc.) contained in the target voice request can be filtered out using digital signal processing techniques (such as spectral subtraction, adaptive filtering) to preserve a clear user voice signal.

[0087] Furthermore, the noise-reduced continuous speech signal is divided into multiple overlapping short speech frames at fixed time intervals (e.g., 20-30 milliseconds / frame). Each frame serves as an independent processing unit, which facilitates subsequent feature extraction (since the speech signal can be considered a stationary signal in the short term).

[0088] Furthermore, a Fourier transform is performed on each speech frame to convert the time-domain signal into a frequency-domain signal and obtain spectral features.

[0089] Furthermore, the spectrum can be filtered using the Mel-frequency cepstral coefficient (MFCC) algorithm to simulate the human ear's perception characteristics of different frequencies (sensitive to low frequencies and insensitive to high frequencies).

[0090] Furthermore, logarithmic operations and discrete cosine transform (DCT) are performed on the filtered results to extract Mel frequency cepstral coefficients, which are then used as acoustic features characterizing speech frames.

[0091] Furthermore, the extracted acoustic features are input into the Encoder module of the FunASR model, and the features are encoded by a deep neural network (such as Transformer) to capture the contextual dependencies of the speech sequence.

[0092] Furthermore, the encoded features are fed into the Decoder module and decoded into the corresponding text sequence through an attention mechanism. The recognition results are then optimized by combining the fusion language model, such as correcting homophones and supplementing commonly used words.

[0093] Furthermore, the post-processing module restores the punctuation marks in the text and generates the initial text containing the user's voice content.

[0094] Noise reduction and frame segmentation improved the quality of the speech signal, reducing the impact of environmental interference on the recognition results. Furthermore, the Mel-frequency cepstral coefficient algorithm was used for feature extraction, converting the speech signal into machine-recognizable feature parameters. Finally, the FunASR model was used to process multiple acoustic features, achieving efficient speech recognition, improving recognition accuracy, and reducing latency.

[0095] Step S3022: Clean and denoise the initial text to obtain the target text.

[0096] Specifically, by cleaning up the generated initial text, noise introduced during the recognition process (such as misidentified symbols and disordered formatting) can be eliminated, thereby ensuring that subsequent command task recognition can perform accurate reasoning based on clean and standardized text and improving the reliability of semantic understanding.

[0097] First, remove control characters (such as \r, \n, \t) and special characters (such as *, ") from the initial text, and filter out meaningless symbols.

[0098] Secondly, merge consecutive spaces into a single space and remove non-text elements such as emojis from the text.

[0099] Finally, the text format is standardized, such as unifying capitalization and correcting obvious recognition errors, to ultimately generate target text with a well-structured and clear content.

[0100] Step S303: Identify the command task in the target text and generate the first command task text based on the identification result.

[0101] Specifically, step S303 includes:

[0102] Step S3031: Use a preset large model and retrieval enhancement generation technology to identify the command task in the target text and obtain the command task.

[0103] Among them, the pre-defined large model represents a large language model with powerful language understanding and reasoning capabilities, low training cost, fast speed, and low resource requirements.

[0104] Furthermore, Retrieval-Augmented Generation (RAG) refers to a technique aimed at improving the performance of generative artificial intelligence models by incorporating external knowledge sources.

[0105] Specifically, target text can be vectorized using a pre-defined large model and retrieval enhancement generation techniques.

[0106] Furthermore, vectors can be used to retrieve the top 5 knowledge items related to the target text from the Milvus vector knowledge base.

[0107] The Milvus vector knowledge base can be generated by following these steps:

[0108] First, the content of knowledge documents in formats such as Word and PDF is parsed to extract text information. Then, the text is segmented according to semantic logic or fixed length, breaking long documents into several independent text fragments to facilitate subsequent vector conversion and retrieval.

[0109] Secondly, the bce-embedding-base_v1 model (text embedding model) is used to perform vector transformation on the segmented text fragments, mapping each text fragment to a high-dimensional vector (i.e., the embedding process). These vectors can represent the semantic information of the text, making semantically similar text fragments closer together in the vector space.

[0110] Finally, the generated text vectors are stored in the Milvus vector database, forming a structured vector knowledge base. This knowledge base can be quickly retrieved based on the semantic similarity of text vectors, providing a foundation for knowledge matching when processing user requests.

[0111] Furthermore, through the above process, the Milvus vector knowledge base achieves structured storage and semantic association of external knowledge, and supports the rapid retrieval of relevant knowledge during user interaction to assist in large model reasoning, thereby improving the accuracy of command task recognition.

[0112] Furthermore, by combining the retrieved knowledge, a pre-defined large model is used to perform semantic understanding on the target text, and to identify the user's explicitly stated commands and tasks (such as queries, reservations, settings, etc., which have specific objectives).

[0113] Step S3032: Use the preset large model to determine whether the command task is a complete slot command task.

[0114] Specifically, the pre-defined large model can analyze whether a command task contains all the key information required to complete the task (such as time and location for weather queries, and date and number of people for service bookings) based on the semantic structure of the command task, i.e. whether it is a complete slot command task.

[0115] Furthermore, if no key information is missing, it is determined to be a complete slot command task; otherwise, it is an incomplete slot command task.

[0116] Step S3033: When the command task is a complete slot command task, the command task is processed using a preset large model and the first command task text is generated.

[0117] Specifically, if the slot information of the command task is complete, that is, when the command task is a complete slot command task, the preset large model can generate response content that conforms to semantic logic based on the task type (such as query, execution) (such as "I have found out that the weather in Shanghai tomorrow is sunny"), forming the first command task text, which can be directly used for subsequent bimodal output processing.

[0118] In some optional implementations, step S303 above further includes:

[0119] Step b1: When the command task is a non-complete slot command task, the command task is processed using a preset large model and a second command task text is generated.

[0120] Step b2 involves performing text segmentation and speech synthesis on the first command task text to obtain the third command task text and the initial command task speech.

[0121] Step b3: Send the third command task text and the initial command task voice to the client so that the client can send a second voice request to the digital human interaction system.

[0122] Step b4: When the second voice request is received, iterate repeatedly until the first command task text is obtained.

[0123] Specifically, if the slot information of the command task is missing, that is, when the command task is an incomplete slot command task (such as "Help me book a hotel" without mentioning the date), the pre-set large model can be used to generate follow-up text that supplements the key information (such as "Which day of the hotel do you need to book?"), which is the second command task text.

[0124] Furthermore, the second command task text is segmented according to punctuation marks such as ".", "!", and "?", as well as the number of tokens (≥20), to obtain the segmented third command task text.

[0125] Furthermore, the text of the first command task can be converted into natural and fluent speech, i.e., the speech of the target command task, through speech synthesis processing.

[0126] Furthermore, the digital human interaction system 1 can push the third command task text (segmented follow-up question text) and the initial command task voice (follow-up question voice) to the client in real time via WebSocket. After receiving the text, the user supplements the information according to the follow-up question content and sends a second voice request (containing missing slot information) through voice input. This realizes the bimodal output of the follow-up question content, triggers the user to supplement the information, and promotes the progress of multi-turn dialogue.

[0127] Furthermore, when the digital human interaction system 1 receives a new second voice request, it can repeat steps S301 to S303 until the slot information of the command task is complete and the corresponding first command task text is generated. Through multiple rounds of dialogue iteration, the key information of the task can be supplemented, ensuring that the final generated command task text meets the execution conditions, thus guaranteeing the integrity and effectiveness of the interaction and avoiding task execution failure due to missing information.

[0128] Step S304 involves performing text segmentation and speech synthesis on the first command task text to obtain the second command task text and the target command task speech. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0129] Step S305: Send the target command task (voice) and the second command task (text) to the client. See details below. Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0130] The digital human interaction method provided in this embodiment recognizes the target voice request, realizing the conversion from speech to text and providing raw text material for subsequent processing. Furthermore, by cleaning and denoising the initial text, irrelevant characters and noise are removed, and the text format is standardized, providing clean input for subsequent command task recognition and improving the accuracy of semantic understanding. Furthermore, by combining a preset large model and retrieval-enhanced generation technology to recognize command tasks in the target text, semantic understanding capabilities are enhanced, improving the accuracy of recognizing complex commands. Furthermore, by determining whether the command task is a complete slot command task, the completeness of task information can be distinguished, avoiding invalid interactions and improving dialogue efficiency. Furthermore, if it is a complete slot command task, the first command task text is directly generated using the preset large model, shortening the processing flow and reducing user waiting time. Furthermore, if the command task is an incomplete slot command task, a second command task text is generated, and bimodal output guides the user to supplement information. Furthermore, through repeated iterations, multi-turn dialogue to complete information is supported, ensuring the final effective execution of the task and optimizing the processing experience of incomplete tasks.

[0131] This embodiment provides a digital human interaction method, which can be used for, for example Figure 1 The digital human interaction system 1 shown is connected to the client 2, and includes: a session management module 11, a speech recognition processing module 12, a recognition command task module 13, a streaming window dynamic control module 14, and a speech synthesis module 15.

[0132] Figure 4 This is a flowchart of a digital human interaction method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0133] Step S401: Upon receiving the first voice request from the client, add a target session identifier to the first voice request to obtain the target voice request. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0134] Step S402: Perform speech recognition and processing on the target speech request to obtain the target text. For details, please refer to [link to relevant documentation]. Figure 3 Step S302 of the illustrated embodiment will not be described again here.

[0135] Step S403: Identify the command task in the target text and generate the first command task text based on the identification result. For details, please refer to [link to relevant documentation]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.

[0136] Step S404: Perform text segmentation and speech synthesis processing on the first command task text to obtain the second command task text and the target command task speech.

[0137] Specifically, step S404 includes:

[0138] Step S4041: Perform text segmentation on the first command task text to obtain the fourth command task text.

[0139] Specifically, after the streaming window dynamic control module 14 receives the first command task text sent by the recognition command task module 13, it can use the dynamic streaming window mechanism to segment the first command task text according to the dual-condition triggering rule:

[0140] (1) Punctuation marks (such as “.”, “!”, “?”) are used as sentence end marks. When such punctuation marks appear in the text, it is determined to be a complete sentence;

[0141] (2) Simultaneously monitor the number of tokens in the text. When the cumulative number of tokens reaches 20, sentence segmentation is triggered even if no ending punctuation appears.

[0142] (3) Finally, the first command task text is split into multiple segmented texts, namely the fourth command task text.

[0143] Furthermore, by reasonably splitting long texts, the mechanical feeling of word-by-word output and the waiting delay of the overall output are avoided, which provides a foundation for the natural connection between subsequent streaming text display and speech synthesis, and balances the real-time performance and smoothness of the interaction.

[0144] Step S4042: Perform grammatical structure parsing and prosodic prediction on the fourth command task text to obtain the fifth command task text containing multiple grammatical information and multiple intonation change parameters.

[0145] Specifically, in the speech synthesis module 15, syntactic analysis can be performed on each segment of the text in the fourth command task text to identify grammatical information such as subject-verb-object and modification relationships, so as to determine the sentence components and logical structure.

[0146] Furthermore, based on grammatical information and text semantics (such as interrogative, declarative, and exclamatory tones), corresponding intonation change parameters (such as pitch, speech rate, and pause duration) can be predicted, enabling speech synthesis to match the emotion and tone of the text, and ultimately forming a structured text containing grammatical information and intonation parameters, namely the fifth command task text.

[0147] Furthermore, the above process provides a basis for linguistic logic and emotional expression for subsequent speech synthesis, ensuring that the synthesized speech conforms to the prosodic rules of natural language and improving the naturalness and expressiveness of the speech output.

[0148] Step S4043: Using preset speech synthesis technology and preset speech model, process the fifth command task text to generate multiple basic speech features.

[0149] Specifically, the speech synthesis module 15 can activate the preset EdgeTTS speech synthesis technology (a lightweight edge speech synthesis solution) and load the corresponding cloud speech synthesis model (adapting to the resource limitations of edge devices and balancing synthesis efficiency and quality).

[0150] Furthermore, the pre-defined EdgeTTS speech synthesis technology can convert the text content into a series of basic speech features based on the grammatical structure and prosodic prediction results of the fifth command task text, which may include:

[0151] (1) Phoneme sequence: the smallest unit of speech corresponding to the pronunciation of text;

[0152] (2) Fundamental frequency curve: Represents the pitch variation of speech, matching the tone of the text (such as the rising tone of interrogative tone and the level tone of declarative tone);

[0153] (3) Spectral envelope: reflects the frequency distribution characteristics of speech and determines the timbre of speech;

[0154] (4) Duration information: the duration of pronunciation of each phoneme, with pause duration determined based on grammatical structure and semantics.

[0155] Step S4044: Use a vocoder to convert multiple basic speech features to obtain the target command task speech.

[0156] Specifically, a vocoder can be used to convert the signal into a continuous speech waveform signal through a signal synthesis algorithm, ultimately generating a playable target command task speech (streamed speech segment).

[0157] Furthermore, by transforming abstract acoustic features into perceptible speech signals and combining them with prior prosodic prediction, the generated speech is ensured to be natural, fluent, and emotionally resonant, providing users with a clear and easy-to-understand speech response.

[0158] Step S405: Send the target command task (voice) and the second command task (text) to the client. See details below. Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0159] In some optional implementations, when a text request is received from a client, the command task in the text request is identified, and a first command task text is generated based on the identification result; the first command task text is processed by text segmentation and speech synthesis to obtain a second command task text and a target command task speech; the target command task speech and the second command task text are sent to the client.

[0160] Specifically, if the received request from the client is a text request, then steps S403 to S405 are executed directly.

[0161] The digital human interaction method provided in this embodiment avoids lengthy output and balances fluency and response speed by performing text segmentation on the first command task text. Furthermore, by performing grammatical structure analysis and prosodic prediction on the fourth command task text, the speech synthesis conforms to language rules and emotional expression, improving the naturalness of the speech. Furthermore, by generating high-quality speech waveforms through a vocoder, the mechanical and disjointed problems of traditional speech synthesis are solved. Furthermore, the support for text input modality overcomes the limitations of traditional single-modality digital humans and enhances scene adaptability. Furthermore, by supplementing the processing flow to support text requests, the limitations of traditional single-modality input in digital humans are overcome, achieving unified processing of text and speech input, further enhancing scene adaptability.

[0162] This embodiment provides a digital human interaction system, such as Figure 1 As shown, the digital human interaction system 1 is connected to the client 2, and the digital human interaction system 1 includes: a session management module 11, a speech recognition processing module 12, a recognition command task module 13, a streaming window dynamic control module 14, and a speech synthesis module 15.

[0163] Optionally, the session management module 11 is used to add a target session identifier to the first voice request when it receives the first voice request sent by the client, obtain the target voice request, and send the target voice request to the speech recognition module 12.

[0164] The specific process can be found in the description of step S201 above, and will not be repeated here.

[0165] Optionally, the speech recognition processing module 12 is used to perform speech recognition and processing on the target speech request, obtain the target text, and send the target text to the recognition command task module 13.

[0166] The specific process can be found in the description of step S302 above, and will not be repeated here.

[0167] Optionally, the command task recognition module 13 is used to recognize the command task in the target text, generate a first command task text based on the recognition result, and send the first command task text to the streaming window dynamic control module 14 and the speech synthesis module 15 respectively.

[0168] The specific process can be found in the description of step S303 above, and will not be repeated here.

[0169] Optionally, the speech synthesis module 15 is used to perform speech synthesis processing on the first command task text to obtain the target command task speech, and send the target command task speech to the streaming window dynamic control module 14.

[0170] The specific process can be found in step S404 above, which describes the acquisition of the target command task voice. It will not be repeated here.

[0171] Optionally, the streaming window dynamic control module 14 is used to perform text segmentation processing on the first command task text to obtain the second command task text, and send the second command task text to the client 2, as well as send the target command task voice to the client 2.

[0172] For the specific process, please refer to the process of obtaining the second command task text in step S404 above and the description of step S205 above, which will not be repeated here.

[0173] In some alternative implementations, if the session management module 11 receives a text request from the client 2, it can directly send the text request to the recognition command task module for processing.

[0174] The digital human interaction system provided in this embodiment associates users and requests through a session management module, ensuring interaction continuity and supporting personalized services. Furthermore, a speech recognition processing module achieves accurate speech-to-text conversion, providing reliable input for semantic understanding. Furthermore, a command task recognition module accurately identifies user intent and task completeness, improving response targeting. Furthermore, a speech synthesis module generates natural and fluent speech, enhancing the naturalness and expressiveness of voice interaction. Furthermore, a streaming window dynamic control module enables segmented streaming output of text and hidden text, achieving simultaneous dual-modal output, optimizing the reading experience, shortening waiting time, improving interaction naturalness, and conforming to user dialogue habits. Therefore, by implementing this invention, and by supporting voice request input and dual-modal output of text and speech, multi-scenario adaptation is achieved, improving the user experience.

[0175] In one example, a text-and-voice dual-modal full-duplex digital human interaction method and system is provided, which is applicable to scenarios requiring real-time natural interaction such as intelligent customer service, virtual assistant, educational tutoring, and virtual social interaction. It can enable users to interact with digital humans simultaneously in both directions without perceptible delay.

[0176] 1. Adopt a microservice system architecture, such as Figure 5 As shown, it consists of the following service modules: real-time speech recognition (ASR), real-time speech synthesis (TTS), digital human image driving module (controlling facial expressions, actions, and lip movements), modality fusion and semantic understanding, streaming window dynamic control module, session tracking message management module, asynchronous concurrency control module, and network protocol encapsulation communication module.

[0177] 2. WebSocket full-duplex communication technology is adopted to replace SSE (Server-Sent Events) unidirectional streaming output. WebSocket is based on TCP full-duplex communication, establishing a persistent connection through a single handshake, enabling real-time bidirectional data transmission between the client and server. Users can immediately send text or voice questions while listening to voice or viewing text output, without waiting. In contrast, SSE only allows unidirectional communication; if a user wants to interrupt, the network communication must be terminated, resulting in high network overhead. Furthermore, SSE cannot handle binary data.

[0178] 3. Employing the Asyncio asynchronous communication framework, it handles four types of asynchronous concurrent network input and output operations: text input requests, voice input requests, text output, and voice output. The Asyncio asynchronous communication mechanism is based on an event-driven programming model. The Event Loop mechanism solves the synchronous blocking problem. The Coroutine model avoids multi-threaded switching, and system-level I / O multiplexing solves high-latency I / O operations. It can handle I / O operations from tens of thousands of concurrent connections.

[0179] 4. Session interruption and multi-turn dialogues are achieved through session management, dialogue state tracking, and historical message saving. Session management allows for personalized output by user and session ID. Dialogue state tracking effectively tracks the state from creation, initial response output, intermediate streaming output, final output, and interruption by the client. Historical message saving provides context to the larger model, enabling better reasoning, and establishes relationships between messages, distinguishing between slot-filling dialogues and multi-turn dialogues between intents.

[0180] 5. Utilize a dynamic streaming window to achieve streaming output of text and speech. For text answers in large models, if output to the client word by word, the converted speech sounds mechanical and disjointed; if outputted as a whole, the client wait time is too long. Therefore, this solution uses a dual-condition triggering of sentence segmentation and token count to output to the client segment by segment, resulting in smooth and natural speech.

[0181] (1) Microservice system architecture.

[0182] (a) Real-time speech recognition service.

[0183] This system implements speech recognition to text conversion based on FunASR. The system initializes the model and then listens for requests. An interface is encapsulated using Flask to provide interface services to the main scheduler.

[0184] FunASR is an end-to-end automatic speech recognition technology characterized by high accuracy and low latency. Training utilizes a manually annotated Mandarin speech recognition dataset containing 60,000 hours of speech data. The speech signal undergoes preprocessing such as noise reduction and framing, followed by acoustic feature extraction using MFCC. The FunASR engine employs an Encoder-Decoder architecture for speech recognition, utilizes a fusion language model to improve accuracy, and a post-processing module recovers punctuation marks.

[0185] (b) Real-time speech synthesis service.

[0186] This system implements speech synthesis and text-to-speech based on EdgeTTS. The system initializes the model and then listens for requests. An interface is encapsulated using Flask to provide interface services to the main scheduler.

[0187] EdgeTTS is a lightweight speech synthesis technology designed specifically for edge computing devices, enabling efficient speech synthesis in resource-constrained environments. In digital human speech generation technology, the text analysis module first parses the grammatical structure, the prosody prediction module generates intonation variations, EdgeTTS processes the cloud-based speech synthesis, a vocoder generates high-quality waveforms, and dynamic adjustment of emotion parameters enhances expressiveness.

[0188] (c) Large model service.

[0189] A privately deployed large model (such as DeepSeek) is used with an external vector knowledge base to implement knowledge-based question answering. The system initializes the model and then listens for requests. An interface is encapsulated using Flask to provide interface services to the main scheduler.

[0190] Among them, DeepSeek has strong language understanding and reasoning capabilities, low training cost and fast speed, and low resource requirements.

[0191] Furthermore, the system first parses and segments the text of knowledge documents such as Word and PDF, and then embeds the text into the Milvus vector database using the bce-embedding-base_v1 model. During a query, the question is first vectorized, and then the top 5 matching results are retrieved from Milvus. This knowledge is then fed into the larger model for generalization and inference.

[0192] (d) Master control service.

[0193] The main control service, based on WebSocket and Asyncio, calls various services to complete requests. It includes session management, message management, dynamic control of streaming windows, encapsulation of network protocols with clients, internal service calls, and asynchronous communication.

[0194] Furthermore, WebSocket+Asyncio specifically performs the following steps:

[0195] (i) Start the WebSockets service on port 10001 and set up the listener handle_client using async. Here, async is used to define asynchronous functions;

[0196] (ii) handle_client: For cliSocket (client socket) connections, asynchronously read each message, parse the message, obtain the session ID, and put it into the connection pool (clientMap). Use await to identify the proc process for messages. Here, await is used to pause the execution of asynchronous functions to wait for asynchronous operations to complete, so that the program does not block when waiting for I / O operations, improving efficiency. In this embodiment, it is used to handle asynchronous tasks such as WebSocket connections and messages.

[0197] (iii)proc: Processes user voice question requests, calls the ASR service to recognize the text, then queries Milvus to obtain relevant knowledge, and calls the large model service in a streaming manner to obtain inference tokens, which are then placed in a dynamic window. For each text segment in the window, the TTS service is called to synthesize a speech segment. At this point, the text and speech segments are asynchronously sent to the cliSocket using await.

[0198] (2) Session management + historical messages include:

[0199] (a) A map linking client sockets and session IDs:

[0200] client2Id[cliSocket] = talkId

[0201] clientMap[talkId] = cliSocket

[0202] (b) Manage customer inquiries and responses. Session data structure: user ID, session ID, session status, creation time.

[0203] (c) Manage each customer message. Message data structure: message ID, session ID, whether it is voice, business operation code, text question, text answer, voice stream file path, creation time.

[0204] (d) Current sessions and messages are in memory. Once they become historical messages, they are stored in the database.

[0205] (e) Each time a question is asked to the large model, the seven most recent historical messages will be passed to the large model as context.

[0206] (3) Dynamic window mechanism.

[0207] Specifically, a dynamic streaming window is used to achieve streaming output of text and speech. The streaming window is triggered by two conditions: sentence segmentation and token count, and outputs the speech to the client segment by segment, resulting in smooth and natural speech transitions.

[0208] Furthermore, the paragraph is a dynamic window that stores data by sentence. If the number of sentences is greater than or equal to 2, or if the total number of tokens in the paragraph is greater than or equal to 20, then this window outputs to the cliSocket.

[0209] Furthermore, the sentence processing logic in the paragraph is as follows: characters are placed into the sentence, and if a character is one of ['。', '!', '?'], then it is a complete sentence, thus completing the sentence collection.

[0210] Furthermore, the character processing logic in the sentence is as follows: remove control characters from ['\r', '\n', '\t'], remove special characters from ['*', '"'], keep only one consecutive space, and remove emojis. The sentence is kept in two copies: one is output to the client as the original text, and the other is converted into a speech segment and output to the client.

[0211] Furthermore, the text and voice dual-modal intelligent interaction process, such as Figure 6 As shown.

[0212] The text-and-voice bimodal full-duplex digital human interaction method and system provided in this example have the following effects:

[0213] (1) By using dual-modal fusion and full-duplex mechanism, the interaction is closer to real human dialogue, which improves the naturalness of the interaction.

[0214] (2) Combining text and speech dual-modal information improves the accuracy of understanding fuzzy semantics.

[0215] (3) The parallel processing architecture significantly reduces the overall response latency, shortens the response time, and significantly improves the concurrency.

[0216] (4) Supports interruption and topic jumping in continuous dialogue, optimizing the interactive experience.

[0217] (5) The modular design allows for quick integration of new interaction modalities (such as gestures and eye tracking), and has strong scalability.

[0218] This invention also provides a computer device for performing the above-described... Figures 2 to 4 The digital human interaction method shown.

[0219] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0220] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0221] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0222] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0223] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0224] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0225] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0226] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0227] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A digital human interaction method, characterized in that, For a digital human interaction system, the system being connected to a client; the method includes: When a first voice request is received from the client, a target session identifier is added to the first voice request to obtain the target voice request; The target voice request is subjected to speech recognition and processing to obtain the target text; The command task in the target text is identified, and a first command task text is generated based on the identification result; The first command task text is processed by text segmentation and speech synthesis to obtain the second command task text and the target command task speech. The target command voice and the second command text are sent to the client.

2. The method according to claim 1, characterized in that, The target voice request is identified and processed to obtain the target text, including: The target voice request is recognized to obtain the initial text; The initial text is cleaned and denoised to obtain the target text.

3. The method according to claim 2, characterized in that, The target voice request is recognized to obtain initial text, including: The target voice request is subjected to noise reduction and frame segmentation to obtain multiple voice signals; The Mel frequency cepstral coefficient algorithm is used to extract features from the multiple speech signals to obtain multiple acoustic features; The FunASR model is used to process the multiple acoustic features to obtain the initial text.

4. The method according to claim 1, characterized in that, The command task in the target text is identified, and a first command task text is generated based on the identification result, including: The command task is identified in the target text using a pre-set large model and retrieval enhancement generation technology. The preset large model is used to determine whether the command task is a complete slot command task; When the command task is a complete slot command task, the command task is processed using the preset large model to generate the first command task text.

5. The method according to claim 4, characterized in that, The method further includes: When the command task is a non-complete slot command task, the command task is processed using the preset large model and a second command task text is generated; The first command task text is processed by text segmentation and speech synthesis to obtain the third command task text and the initial command task speech. The third command task text and the initial command task voice are sent to the client, so that the client sends a second voice request to the digital human interaction system; Upon receiving the second voice request, the process iterates repeatedly until the first command task text is obtained.

6. The method according to claim 1, characterized in that, The first command task text is processed by text segmentation and speech synthesis to obtain the second command task text and the target command task speech, including: The first command task text is processed by text segmentation to obtain the fourth command task text. The fourth command task text is parsed using grammatical structure analysis and prosodic prediction to obtain a fifth command task text containing multiple grammatical information and multiple intonation change parameters. Using preset speech synthesis technology and preset speech model, the text of the fifth command task is processed to generate multiple basic speech features; The target command task speech is obtained by converting the multiple basic speech features using a vocoder.

7. The method according to claim 1, characterized in that, The method further includes: When a text request is received from the client, the command task in the text request is identified, and the first command task text is generated based on the identification result; The first command task text is processed by text segmentation and speech synthesis to obtain the second command task text and the target command task speech. The target command voice and the second command text are sent to the client.

8. A digital human interaction system, characterized in that, The system is used to perform the digital human interaction method according to any one of claims 1 to 7, wherein the system is connected to a client; the system includes: a session management module, a speech recognition processing module, a speech recognition command task module, a streaming window dynamic control module, and a speech synthesis module; The session management module is used to add a target session identifier to the first voice request when it receives a first voice request sent by the client, obtain the target voice request, and send the target voice request to the voice recognition module; The speech recognition module is used to perform speech recognition and processing on the target speech request to obtain the target text, and then send the target text to the recognition command task module. The command task recognition module is used to recognize the command task in the target text, generate a first command task text based on the recognition result, and send the first command task text to the streaming window dynamic control module and the speech synthesis module respectively. The speech synthesis module is used to perform speech synthesis processing on the first command task text to obtain the target command task speech, and send the target command task speech to the streaming window dynamic control module. The streaming window dynamic control module is used to perform text segmentation processing on the first command task text to obtain the second command task text, and send the second command task text to the client, as well as send the target command task voice to the client.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the digital human interaction method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer instructions for causing a computer to perform the digital human interaction method according to any one of claims 1 to 7.

Images