A digital human voice sensing interaction system and method based on big data information

Abstract

The application relates to the technical field of digital human interaction, and discloses a digital human voice sensing interaction system and method based on big data information, which comprises the following steps: collecting user voice and preprocessing to obtain clear voice; identifying the clear voice to obtain user text and confidence information; calling a big data information library and extracting key facts, and writing the facts into a fact maintaining register; determining a response stage according to the user text and constructing a dynamic attention mask; detecting fact missing and conflicts during decoding and rechecking and supplementing facts; continuously decoding to generate a response text; and converting the response text into voice, lip shape, expression and action output. The application realizes digital human fact maintaining and rechecking generation, improves reply accuracy and interaction reliability.

Description

Technical Field

[0001] This invention relates to the field of digital human interaction technology, and in particular to a digital human voice sensing and interaction system and method based on big data information. Background Technology

[0002] With the development of speech recognition, natural language processing, and digital human rendering technologies, digital humans have been widely applied in intelligent customer service, government services, and business consultation scenarios. Existing digital human interaction solutions typically collect user speech, convert it into text, input it into a natural language generation model to generate response text, and then drive the digital human to complete speech, lip movements, facial expressions, and gestures. To improve response accuracy, some solutions integrate knowledge bases or business databases to provide external information for response generation.

[0003] Existing solutions typically perform an information retrieval before response generation, using the retrieval results as model input. This lacks continuous maintenance of key facts and management of the generation process. Generative models are prone to missing facts in longer responses or reading mismatched information at different response stages. Existing attention masks often employ fixed structures, making it difficult to adjust the information retrieval range according to the response stage.

[0004] Current response generation methods mostly employ continuous decoding. When facts are missing, conflicting, or applicable conditions are incomplete during the generation process, the model will continue to generate responses, which can easily lead to incomplete responses from the digital human or inconsistencies with business data.

[0005] Therefore, how to provide a digital human voice sensing and interaction system and method based on big data information is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a digital human voice sensing and interaction system and method based on big data information. This invention utilizes voice sensing, UniLM response generation, big data information retrieval, and digital human driving technology. It achieves digital human natural language interaction through fact-keeping registers, dynamic attention masks, and generation breakpoint back-checking, and has the advantages of high fact consistency, strong response integrity, and natural interactive expression.

[0007] A digital human voice sensing interaction method based on big data information according to an embodiment of the present invention includes the following steps: Acquire user voice signals collected by voice sensing devices, perform voice preprocessing on user voice signals, and generate clear voice signals; Input a clear speech signal into the speech recognition model to generate user-input text and recognition confidence information; Based on the user's input text, retrieve relevant data from the big data information database for the current interaction, extract key facts from the retrieval results, encode the key facts as fact hidden states, and write them into the fact holding register set between UniLM layers; Input the user's text into the UniLM target, determine the response generation stage based on the current hidden state of the target, construct a dynamic attention mask according to the response generation stage, and the UniLM target reads the corresponding fact hidden state and context hidden state. During the UniLM target decoding process, the interruption point is detected based on the matching relationship between the facts required in the response generation stage and the facts already stored in the fact holding register. Based on the interruption point, supplementary facts are called from the big data information database to update the fact holding register. Based on the updated fact-holding register and dynamic attention mask, UniLM target-side decoding continues to be performed to generate digital human response text; The digital human's expression control parameters are generated based on the digital human's response text, driving the digital human to output speech, lip movements, facial expressions, and actions.

[0008] Optionally, the speech preprocessing of the user's speech signal specifically includes: Control the voice sensing device to collect user voice signals and perform analog-to-digital conversion on the user voice signals; The user speech signal after analog-to-digital conversion is subjected to frame-by-frame windowing processing, and the effective speech interval is determined based on short-time energy and zero-crossing rate. Environmental noise estimation, echo cancellation, spectrum enhancement, and amplitude normalization are performed on user speech signals within the effective speech range. The processed user voice signal is spliced ​​frame by frame to generate a clear voice signal.

[0009] Optionally, inputting the clear speech signal into the speech recognition model specifically includes: Acoustic features are extracted from clear speech signals and input into the speech recognition model in chronological order. Speech coding features are obtained by contextually encoding acoustic features using the acoustic coding network of a speech recognition model. The speech coding features are decoded step by step through the decoding network of the speech recognition model to generate several candidate word sequences and the sequence probability corresponding to each candidate word sequence. The target word sequence is determined from several candidate word sequences based on sequence probability. Perform word merging, punctuation restoration, and text normalization on the target word sequence to generate user input text; Calculate the identification confidence information based on the decoding probability of each target word in the target word sequence.

[0010] Optionally, extracting key facts from the call result specifically includes: Semantically encode the user-input text, and retrieve data records consistent with the current interaction topic from the big data information database based on the semantic encoding results; The retrieved data records are sorted according to their source identifier, generation time, and semantic relevance, and the data records in the sorted results are selected as the call results. The call results are processed for text normalization, converting entity names, business objects, numerical content, time content, and status content in the call results into a unified expression form; Based on the query object and query relationship in the user's input text, extract factual content corresponding to the current interaction topic from the call results after text normalization processing, and generate key facts.

[0011] Optionally, the fact-keeping register for writing UniLM inter-layer settings specifically includes: Input key facts and user-input text into the UniLM encoder, and generate the source hidden state through the UniLM encoder; Extract fact register units corresponding to key facts from the hidden state at the source end. Each fact register unit stores an internal fact representation corresponding to a fact category. After each Transformer layer in UniLM, a register update unit is set up. The fact register unit is refreshed from the hidden state of the source end of the current layer through the register update unit. The refreshed fact register unit is then passed to the hidden state calculation process of the target end of the next layer. When generating lexical units at the UniLM target end, the self-attention calculation result at the target end and the current state of the fact register unit are jointly input into the feedforward calculation process at the target end; When the target has generated content that covers the key facts corresponding to the fact register unit, the fact register unit is marked as used. In the subsequent decoding on the target, the fact register units that are not marked as used are read.

[0012] Optionally, constructing the dynamic attention mask according to the response generation stage specifically includes: In each round of word generation at the UniLM target end, the hidden state of the current layer target end is obtained, and the hidden state of the current layer target end is input into the stage state unit to generate discrete stage states. The discrete stage states are mapped to the response generation stage, which is distinguished based on the semantic position of the generated content in the digital human response text. Based on the response generation stage, the set of visible positions generated by the current round word element is determined. The set of visible positions consists of the contextual hidden state corresponding to the user input text, the fact hidden state in the fact holding register, and the hidden state corresponding to the content already generated by the target end. A dynamic attention mask is constructed based on the set of visible locations, and the dynamic attention mask is fed into the UniLM target-side self-attention calculation process; The UniLM target reads the corresponding factual hidden state and contextual hidden state according to the dynamic attention mask, generates the current round word, and updates the target's hidden state based on the current round word.

[0013] Optionally, the step of retrieving supplementary facts from the big data information database based on the generated breakpoint specifically includes: After each round of decoding at the UniLM target end, the target fact category to be read at the current decoding position is determined based on the current response generation stage; Match the target fact category with the fact register cells in the fact holding register that are not marked as used; When there is no fact register unit in the fact holding register that matches the target fact category, or when there is a fact register unit that matches the target fact category but the fact state is conflicting, a corresponding generation interruption point is generated and the target decoding is paused. A re-inspection request is generated based on the target fact category corresponding to the generated interruption point. The re-inspection request is sent to the big data information database and supplementary facts are invoked. Input the supplementary facts into the UniLM encoder to generate the supplementary fact hidden state, write the supplementary fact hidden state into the fact holding register, and set the fact state to the valid state; Resume UniLM target-side decoding based on the updated fact-holding register.

[0014] Optionally, the execution of UniLM target-side decoding specifically includes: Read the updated fact-holding register and reconstruct the dynamic attention mask for the current decoding position; Based on the dynamic attention mask, continue to perform UniLM target-side self-attention calculation to generate subsequent lexical units round by round; The generated word sequence is merged and normalized to generate the digital human's response text.

[0015] Optionally, the process of driving the digital human to output speech, lip movements, facial expressions, and actions specifically includes: Input the digital human's response text into the speech synthesis model to generate the response speech and phoneme timestamps; Match lip shape parameters according to phoneme timestamps; Perform emotion classification on the digital human's response text and generate facial expression parameters; Map action trigger words in the digital human's response text to action parameters; The response voice, lip-reading parameters, facial expression parameters, and motion parameters are simultaneously sent to the digital human rendering terminal, driving the digital human to complete the output of voice, lip-reading, facial expression, and motion.

[0016] A digital human voice sensing and interaction system based on big data information according to an embodiment of the present invention includes the following modules: The voice access module is used to receive user voice signals collected by the voice sensing device and output clear voice signals after preprocessing. The interaction understanding module is used to convert clear speech signals into user-input text and determine the current interaction intent; The data retrieval module is used to access the big data information database according to the user's input text and the current interaction intent, and to obtain the data records corresponding to the current interaction scenario. The fact management module is used to extract key facts from data records, write key facts into the fact holding register, and update the fact holding register during the generation process; The response generation module is used to call UniLM to generate digital human response text, and adjusts the dynamic attention mask according to the response generation stage during the generation process; The back-check control module is used to detect the missing facts and conflicting facts during the digital human response text generation process, and call the supplementary facts to update the fact holding register; The digital human driver module is used to generate speech, lip-reading, facial expression, and motion control parameters based on the digital human's response text, driving the digital human to complete interactive output.

[0017] The beneficial effects of this invention are: (1) This invention encodes key facts as fact hidden states and writes them into fact holding registers set between UniLM layers, so that the digital human response generation process can continuously read fact content related to the current interaction, reduce the problems of missing key facts, mixed use of facts and inconsistency of facts in the long text generation process, and improve the fact consistency of digital human responses.

[0018] (2) The present invention determines the response generation stage through the stage state unit and constructs a dynamic attention mask according to the response generation stage, so that the UniLM target end reads the corresponding fact hidden state, context hidden state and generated content in different decoding stages, avoiding the mismatch of information reading range caused by the fixed attention structure, and improving the structural integrity and semantic coherence of the digital human response.

[0019] (3) This invention detects the generation interruption point during the decoding process of the UniLM target end, and updates the fact holding register after calling supplementary facts from the big data information database according to the generation interruption point. This enables the digital human response generation process to perform back-check and supplementation when facts are missing, facts are conflicting, or applicable conditions are incomplete, thereby reducing the probability of erroneous and incomplete responses and improving the reliability of digital human interactive output. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a digital human voice sensing and interaction method based on big data information proposed in this invention; Figure 2 This is a schematic diagram of a digital human voice sensing and interaction system based on big data information proposed in this invention. Figure 3 This is a schematic diagram illustrating the generation interruption point re-checking of a digital human voice sensing interaction method based on big data information proposed in this invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0022] refer to Figures 1-3 A digital human voice sensing and interaction system and method based on big data information includes the following steps: Acquire user voice signals collected by voice sensing devices, perform voice preprocessing on user voice signals, and generate clear voice signals; Input a clear speech signal into the speech recognition model to generate user-input text and recognition confidence information; Based on the user's input text, retrieve relevant data from the big data information database for the current interaction, extract key facts from the retrieval results, encode the key facts as fact hidden states, and write them into the fact holding register set between UniLM layers; Input the user's text into the UniLM target, determine the response generation stage based on the current hidden state of the target, construct a dynamic attention mask according to the response generation stage, and the UniLM target reads the corresponding fact hidden state and context hidden state. During the UniLM target decoding process, the interruption point is detected based on the matching relationship between the facts required in the response generation stage and the facts already stored in the fact holding register. Based on the interruption point, supplementary facts are called from the big data information database to update the fact holding register. Based on the updated fact-holding register and dynamic attention mask, UniLM target-side decoding continues to be performed to generate digital human response text; The digital human's expression control parameters are generated based on the digital human's response text, driving the digital human to output speech, lip movements, facial expressions, and actions.

[0023] In this embodiment, performing voice preprocessing on the user's voice signal specifically includes: The system controls the voice sensing device to collect user voice signals and performs analog-to-digital conversion on the user voice signals. The voice sensing device uses a single-channel microphone or microphone array, and the digital voice data in pulse code modulation format is obtained after analog-to-digital conversion. The user speech signal after analog-to-digital conversion is subjected to frame-by-frame windowing processing. The effective speech interval is determined based on short-time energy and zero-crossing rate. The frame length for frame-by-frame processing is set to 32ms, the frame shift is set to 16ms, and the windowing function is a Hamming window. Short-time energy is used to determine speech intensity, and zero-crossing rate is used to help distinguish speech segments from noise segments. The frame interval that continuously meets the speech determination conditions is determined as the effective speech interval. The short-time energy is compared with the energy threshold, and the zero-crossing rate is compared with the zero-crossing rate threshold. When the short-time energy of the speech frame meets the energy threshold condition and the zero-crossing rate meets the zero-crossing rate threshold condition, the speech frame is determined as a valid speech frame, and the time interval corresponding to the consecutive valid speech frames is determined as the effective speech interval. Environmental noise estimation, echo cancellation, spectrum enhancement and amplitude normalization are performed on the user speech signal within the effective speech interval. Environmental noise estimation is based on the noise spectrum calculated in the non-speech interval. Echo cancellation adopts an adaptive filtering method. Spectrum enhancement adopts spectral subtraction or Wiener filtering. Amplitude normalization adjusts the processed speech amplitude to a preset amplitude range. The processed user speech signal is spliced ​​between frames to generate a clear speech signal. The splicing process reconstructs the processed speech frames according to the original time sequence, and the overlapping areas of adjacent speech frames are smoothly connected by weighted superposition.

[0024] In this embodiment, inputting a clear speech signal into the speech recognition model specifically includes: Acoustic features are extracted from clear speech signals and input into the speech recognition model in chronological order; the acoustic features are Mel-frequency cepstral coefficients, filter bank features, or log-Mel-spectral features. The acoustic features are context-encoded using the acoustic coding network of the speech recognition model to obtain speech coding features; the acoustic coding network adopts the Transformer coding network, which encodes the temporal dependencies between adjacent speech frames; The speech coding features are decoded step by step through the decoding network of the speech recognition model to generate several candidate word sequences and the sequence probability corresponding to each candidate word sequence. The decoding network adopts CTC decoding. The number of candidate word sequences can be set to 5. The sequence probability is obtained by accumulating the conditional probabilities of each word in the candidate word sequence. The target word sequence is determined from several candidate word sequences according to the sequence probability; the candidate word sequence with the highest sequence probability is determined as the target word sequence; when the sequence probability difference between more than one candidate word sequence and the highest candidate word sequence is less than a preset probability difference threshold, the multiple candidate word sequences are reordered in combination with the language model score. The target word sequence is processed by word merging, punctuation restoration and text normalization to generate user input text; word merging is used to merge sub-word units, character units or pinyin units into word text, and text normalization is used to convert numbers, dates, amounts, units and business numbers into a unified writing format; The identification confidence information is calculated based on the decoding probability of each target word in the target word sequence. The identification confidence information is calculated from the mean, minimum or normalized cumulative probability of the decoding probability of each target word in the target word sequence, and a correspondence is established with the user input text.

[0025] In this embodiment, extracting key facts from the call result specifically includes: The system performs semantic encoding on the user input text and retrieves data records consistent with the current interaction topic from the big data information database based on the semantic encoding results. Semantic encoding can use a text vectorization model to convert the user input text into a fixed-dimensional semantic vector. The big data information database pre-stores the semantic vectors corresponding to the data records. The semantic relevance between the user input text and the data records is calculated using cosine similarity, and candidate data records are returned according to the semantic relevance. The retrieved data records are sorted according to their source identifier, generation time, and semantic relevance. The data records in the sorted results are selected as the call results. The source identifier is used to distinguish the database from which the data comes, and the semantic relevance is used to determine the degree of matching between the data record and the user's input text. The first N data records in the sorted results are used as the call results, where N is a positive integer. The call results are processed for text normalization, converting entity names, business objects, numerical content, time content, and status content into a unified expression format. Entity names are converted into standard entity names according to a preset entity dictionary, business objects are converted into standard business object names according to a preset business object dictionary, numerical content is converted into a numerical value plus standard unit format, time content is converted into year, month, and day format, and status content is converted into standard status names according to a preset status dictionary, such as "Processing", "Undergoing", and "Completed". Based on the query object and query relationship in the user input text, the system extracts factual content corresponding to the current interaction topic from the call results after text normalization and generates key facts; the query object is obtained through entity recognition and the query relationship is obtained through dependency parsing.

[0026] In this embodiment, writing the fact-keeping register for UniLM inter-layer settings specifically includes: Key facts and user input text are input into the UniLM encoder, which generates source hidden states. The user input text and key facts are combined into an encoding sequence in chronological order. The encoding sequence is then processed by word embedding, position embedding, and paragraph embedding before being input into the UniLM encoder. The UniLM encoder calculates layer by layer to obtain the source hidden states corresponding to each word in the encoding sequence. The fact storage units corresponding to the key facts are extracted from the hidden state at the source end. Each fact storage unit stores an internal fact representation corresponding to a fact category. Based on the start and end positions of the key facts in the coding sequence, the hidden state at the source end within the corresponding range is selected for pooling to obtain the fact storage units. The fact categories can be divided into business numerical facts, time facts, rule facts, and status facts. Rule facts refer to "rule-based evidence" extracted from business knowledge, policy clauses, service processes, billing rules, and processing conditions. After each Transformer layer in UniLM, a register update unit is set up. The register update unit refreshes the fact register unit from the hidden state of the current layer source end, and passes the refreshed fact register unit to the hidden state calculation process of the next layer target end. The register update unit receives the hidden state of the current layer source end and the fact register unit of the previous layer, performs pooling on the hidden state corresponding to the key facts in the hidden state of the current layer source end, updates the fact register unit of the previous layer with the pooling result, and uses the updated fact register unit as the additional hidden state for the calculation of the hidden state of the next layer target end. When generating lexical units at the UniLM target end, the self-attention calculation result at the target end and the current state of the fact register unit are jointly input into the target end feedforward calculation process; the self-attention calculation result at the target end and the current state of the fact register unit are obtained by concatenating and linearly mapping to obtain the fused hidden state, the fused hidden state is input into the target end feedforward network, and the target end feedforward network outputs the hidden state required for the next lexical unit prediction. When the content generated by the target end covers the key facts corresponding to the fact register unit, the fact register unit is marked as used. In the subsequent decoding on the target end, fact register units that are not marked as used are read. The entities, values, time and status in the generated content are matched with the key facts corresponding to the fact register unit. When the match is successful, the corresponding fact register unit is marked as used. In the subsequent decoding, fact register units that are not marked as used are read first.

[0027] In this embodiment, constructing a dynamic attention mask according to the response generation stage specifically includes: In each round of word generation at the UniLM target end, the hidden state of the target end at the current layer is obtained, and the hidden state of the target end at the current layer is input into the stage state unit to generate discrete stage states. The stage state unit is composed of a linear mapping layer and a normalization layer, which maps the hidden state of the target end at the current layer to a stage probability distribution, and determines the stage number with the highest stage probability as the discrete stage state. The discrete stage states are mapped to response generation stages. The response generation stages are distinguished based on the semantic position of the generated content in the digital human response text. The response generation stages can be divided into the opening stage, the factual explanation stage, the reason explanation stage, the processing suggestion stage, and the follow-up questioning guidance stage according to the generation order of the digital human response text. The stage state unit automatically switches the response generation stage according to the hidden state corresponding to the generated content. The set of visible positions for the current round of word generation is determined based on the response generation stage. The set of visible positions consists of the contextual hidden state corresponding to the user input text, the factual hidden state in the fact holding register, and the hidden state corresponding to the content already generated by the target end. In the fact explanation stage, the set of visible positions at least includes the numerical factual hidden state and the state factual hidden state in the fact holding register. In the cause explanation stage, the set of visible positions at least includes the business rule factual hidden state and the hidden state corresponding to the content already generated by the target end. In the follow-up questioning and guidance stage, the set of visible positions at least includes the contextual hidden state corresponding to the user input text and several recently generated hidden states by the target end. A dynamic attention mask is constructed based on the set of visible positions, and the dynamic attention mask is fed into the UniLM target-side self-attention calculation process. The dynamic attention mask is represented by a binary matrix. The position with a value of 1 in the matrix represents the hidden state position that is allowed to be accessed by the word in the current round, and the position with a value of 0 represents the hidden state position that is prohibited from being accessed by the word in the current round. The dynamic attention mask is updated based on the set of visible positions when each round of word generation is performed. The UniLM target reads the corresponding factual hidden state and contextual hidden state according to the dynamic attention mask, generates the current round word, and updates the target's hidden state based on the current round word. When the UniLM target performs self-attention calculation, it only performs attention calculation on the positions where the dynamic attention mask is 1, inputs the calculation result into the word prediction layer to obtain the current round word, and appends the hidden state corresponding to the current round word to the hidden state sequence corresponding to the content already generated by the target.

[0028] In this embodiment, retrieving supplementary facts from the big data information database based on the generated interruption point specifically includes: After each round of decoding at the UniLM target end, the target fact category to be read at the current decoding position is determined based on the current response generation stage. The current response generation stage is output by the stage status unit, and the target fact category is determined according to the preset stage fact mapping table. The preset stage fact mapping table records the numerical facts, time facts, business rule facts and status facts corresponding to different response generation stages. The target fact category is matched with fact registers in the fact holding register that are not marked as used. The matching process is performed in the order of fact category consistency, fact status validity, and read flag not being used. Fact registers that meet the above conditions are designated as readable fact registers at the current decoding position. When there is no fact register unit in the fact holding register that matches the target fact category, or when there is a fact register unit that matches the target fact category but the fact state is conflicting, a corresponding generation interruption point is generated and the target decoding is paused. The generation interruption point includes the interruption point number, the decoding position of the interruption point, the target fact category, and the interruption reason. The interruption reason is set to fact missing or fact conflict. When the target decoding is paused, the hidden state sequence corresponding to the generated content is retained. A re-examination request is generated based on the target fact category corresponding to the generated breakpoint. The re-examination request is sent to the big data information database and supplementary facts are retrieved. The re-examination request carries the user input text, the target fact category, the decoding position of the breakpoint, and the generated content summary. The big data information database retrieves the data record corresponding to the target fact category again based on the re-examination request and extracts supplementary facts from the data record. The supplementary facts are input into the UniLM encoding end to generate the supplementary fact hidden state. The supplementary fact hidden state is written into the fact holding register, and the fact state is set to the valid state. The generation method of the supplementary fact hidden state is the same as that of the critical fact hidden state. After the supplementary fact hidden state is written into the fact holding register, a correspondence is established with the generation interrupt point number, and the read flag of the corresponding fact register unit is set to the unused state. UniLM target decoding is resumed based on the updated fact-holding register; when resuming target decoding, the hidden state sequence corresponding to the generated content retained before the pause is called, the updated fact-holding register is added to the set of visible positions of the current round's dynamic attention mask, and subsequent tokens are generated from the decoding position where the interruption point is located.

[0029] In this embodiment, performing UniLM target-side decoding specifically includes: Read the updated fact holding register and reconstruct the dynamic attention mask for the current decoding position; if the fact state is valid and the fact register unit is marked as unused, add the hidden state position corresponding to the fact register unit to the visible position set for the current decoding position, and update the dynamic attention mask based on the visible position set; Based on the dynamic attention mask, continue to perform UniLM target-side self-attention calculation to generate subsequent tokens in round after round; continue to perform token generation from the decoding position where the generation interruption point is located. In each round of generation, read the hidden state in the set of visible positions based on the dynamic attention mask, select the token of the current round according to the token prediction probability, until the generation end marker is marked. The generated word sequence is merged and normalized to generate the digital human response text; the word sequences generated in each round are merged according to the generation order, duplicate words and invalid symbols are deleted, and the numbers, dates, amounts and units are converted to a uniform format to generate the digital human response text.

[0030] In this embodiment, driving the digital human to output speech, lip movements, facial expressions, and actions specifically includes: The digital human's response text is input into the speech synthesis model to generate the response speech and phoneme timestamps. The speech synthesis model can use an existing text-to-speech model. The generated response speech is in pulse code modulation audio format, and the phoneme timestamps are output synchronously by the text-to-speech model during the speech synthesis process. Match lip shape parameters according to phoneme timestamps; the lip shape parameters are generated using the existing phoneme-to-lip shape mapping method, mapping the phoneme corresponding to the phoneme timestamp to the preset lip shape category. The preset lip shape category can be set to 10 to 20 categories, and the lip shape parameters are arranged in chronological order with the phoneme timestamps. The digital human's response text is classified into emotions to generate facial expression parameters. The emotion classification adopts the existing text emotion classification model, and the emotion categories can be set to neutral, positive, questioning, prompting and soothing. The facial expression parameters adopt the existing digital human facial expression driving parameter format and establish a correspondence with the emotion classification results. Map action trigger words in the digital human's response text to action parameters; the action trigger word mapping adopts the existing keyword matching method, mapping greeting, confirmation, prompt and end trigger words to preset action parameters, and the action parameters adopt the existing digital human skeleton driving parameters or animation clip numbers; The response voice, lip-reading parameters, facial expression parameters, and motion parameters are sent synchronously to the digital human rendering end, driving the digital human to complete the output of voice, lip-reading, facial expression, and motion. The digital human rendering end uses an existing 3D digital human rendering engine or a 2D digital human rendering engine to synchronously play the response voice according to a unified timeline and load the lip-reading parameters, facial expression parameters, and motion parameters to complete the digital human interactive output.

[0031] A digital human voice sensing and interaction system based on big data information according to an embodiment of the present invention includes the following modules: The voice access module is used to receive user voice signals collected by the voice sensing device and output clear voice signals after preprocessing. The interaction understanding module is used to convert clear speech signals into user-input text and determine the current interaction intent; The data retrieval module is used to access the big data information database according to the user's input text and the current interaction intent, and to obtain the data records corresponding to the current interaction scenario. The fact management module is used to extract key facts from data records, write key facts into the fact holding register, and update the fact holding register during the generation process; The response generation module is used to call UniLM to generate digital human response text, and adjusts the dynamic attention mask according to the response generation stage during the generation process; The back-check control module is used to detect the missing facts and conflicting facts during the digital human response text generation process, and call the supplementary facts to update the fact holding register; The digital human driver module is used to generate speech, lip-reading, facial expression, and motion control parameters based on the digital human's response text, driving the digital human to complete interactive output.

[0032] Example 1: To verify the feasibility of this invention in practice, it was applied to a smart customer service digital human interaction scenario. This scenario targets business consultation services requiring continuous question-and-answer sessions. Users submit inquiries via voice, including fee inquiries, status confirmations, rule consultations, confirmation of processing conditions, and follow-up operation requests. Traditional digital human solutions typically perform a knowledge base retrieval after a user asks a question, then feed the retrieval results into a natural language generation model. This approach is prone to problems when user questions are lengthy, involve numerous business facts, or require explanations in the response. These issues include incomplete use of retrieved facts, inconsistencies in cited information before and after the response, and continued responses despite missing key conditions.

[0033] In this scenario, the user first inputs a consultation voice through a voice sensing device. The voice access end performs noise reduction, echo cancellation, and amplitude normalization on the voice to obtain a clear voice signal. The voice recognition model converts the clear voice into the user's input text and outputs recognition confidence information. The interaction understanding process determines the consultation object and relationship from the user's input text. The data retrieval process accesses a big data information database to obtain data records matching the current interaction scenario and extracts numerical facts, temporal facts, state facts, and business rule facts. The extracted key facts are not directly concatenated as ordinary text to the response front end, but are encoded as fact hidden states and written to the fact holding register set between UniLM layers.

[0034] When the digital human generates a response, the UniLM target reads the current hidden state during each generation round, and the stage of response generation is determined by the stage state unit. In the factual explanation stage, the target primarily reads the factual hidden state and the user contextual hidden state; in the cause explanation stage, the target reads the business rule facts and the already generated explanation fragments; in the processing suggestion stage, the target reads the state facts, rule facts, and the previously generated content. The dynamic attention mask changes with the response stage, restricting the current generation position to accessing only information relevant to that stage, preventing the model from mixing facts, rules, and contextual content across different stages.

[0035] When the target decoding process detects that the required fact category for the current stage is not present in the fact holding register, or that the corresponding fact unit is in a conflict state, the back-check control process generates an internal back-check request, temporarily stores the hidden state sequence corresponding to the generated content, and pauses the current decoding path. The big data information database, based on the back-check request, calls up business data or knowledge data again, fills in the missing facts, writes them into the fact holding register, and sets the corresponding fact state to a valid state. Then, UniLM continues decoding from the paused position to generate the complete digital human response text. After generation, the digital human driving process inputs the response text into the speech synthesis model to obtain the response speech and phoneme timestamps, and simultaneously generates lip-sync parameters, facial expression parameters, and motion parameters to drive the digital human to complete user-facing interactive output.

[0036] For comparison, traditional retrieval and splicing generation methods, fixed attention mask generation methods, and the method of this invention were selected for verification on similar consultation tasks. All methods used the same voice input, the same knowledge source, the same digital human rendering platform, and the same evaluation criteria. Evaluation metrics applicable to similar models included factual accuracy, response completeness, business consistency pass rate, multi-turn interaction completion rate, user repeat question rate, and average response time. The results are shown in the table below.

[0037] Table 1: Comparison of Application Effects of Digital Human Voice Sensing Interaction Methods

[0038] As shown in Table 1, the method of this invention outperforms the two comparative methods in terms of fact accuracy, response completeness, and business consistency pass rate. This indicates that the fact-holding register can continuously store key facts during the response generation process, reducing the occurrence of omitted or incorrectly cited facts when generating long texts. While the fixed attention mask generation method has improved upon traditional retrieval and splicing generation methods, the inability to adjust the visible information range with each response stage still leads to mismatches between rule content and user status in causal explanations and processing suggestions. This invention controls the information reading range at different decoding stages through stage state units and dynamic attention masks, making the boundaries between fact description, causal explanation, and suggestion generation clearer.

[0039] In terms of multi-round interaction completion rate, the method of this invention achieved 91.5%, higher than the comparative methods, indicating that the generation of interruption point review can promptly fill in information when facts are missing or conflicting, rather than continuing to generate incomplete responses. The user repeat question rate decreased to 8.2%, indicating that the facts, reasons, and handling suggestions given in the digital human's initial response are more complete, reducing the number of times users need to ask follow-up questions. The average response time was 1.47 seconds, slightly higher than the traditional method and the fixed attention mask method, mainly because the fact matching and necessary review process were added during the generation process. However, this time is still within the acceptable range for voice interaction, and it achieves higher response accuracy and service reliability. The comprehensive comparison results show that this invention is suitable for digital human interaction scenarios that require accurate interpretation of business facts, dynamic retrieval of big data information, and completion of natural expression output.

[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A digital human voice sensing and interaction method based on big data information, characterized in that, Includes the following steps: Acquire user voice signals collected by voice sensing devices, perform voice preprocessing on user voice signals, and generate clear voice signals; Input a clear speech signal into the speech recognition model to generate user-input text and recognition confidence information; Based on the user's input text, retrieve relevant data from the big data information database for the current interaction, extract key facts from the retrieval results, encode the key facts as fact hidden states, and write them into the fact holding register set between UniLM layers; Input the user's text into the UniLM target, determine the response generation stage based on the current hidden state of the target, construct a dynamic attention mask according to the response generation stage, and the UniLM target reads the corresponding fact hidden state and context hidden state. During the UniLM target decoding process, the interruption point is detected based on the matching relationship between the facts required in the response generation stage and the facts already stored in the fact holding register. Based on the interruption point, supplementary facts are called from the big data information database to update the fact holding register. Based on the updated fact-holding register and dynamic attention mask, UniLM target-side decoding continues to be performed to generate digital human response text; The digital human's expression control parameters are generated based on the digital human's response text, driving the digital human to output speech, lip movements, facial expressions, and actions.

2. The digital human voice sensing and interaction method based on big data information according to claim 1, characterized in that, The specific steps of performing voice preprocessing on the user's voice signal include: Control the voice sensing device to collect user voice signals and perform analog-to-digital conversion on the user voice signals; The user speech signal after analog-to-digital conversion is subjected to frame-by-frame windowing processing, and the effective speech interval is determined based on short-time energy and zero-crossing rate. Environmental noise estimation, echo cancellation, spectrum enhancement, and amplitude normalization are performed on user speech signals within the effective speech range. The processed user voice signal is spliced ​​frame by frame to generate a clear voice signal.

3. The digital human voice sensing and interaction method based on big data information according to claim 2, characterized in that, The specific steps of inputting clear speech signals into the speech recognition model include: Acoustic features are extracted from clear speech signals and input into the speech recognition model in chronological order. Speech coding features are obtained by contextually encoding acoustic features using the acoustic coding network of a speech recognition model. The speech coding features are decoded step by step through the decoding network of the speech recognition model to generate several candidate word sequences and the sequence probability corresponding to each candidate word sequence. The target word sequence is determined from several candidate word sequences based on sequence probability. Perform word merging, punctuation restoration, and text normalization on the target word sequence to generate user input text; Calculate the identification confidence information based on the decoding probability of each target word in the target word sequence.

4. The digital human voice sensing and interaction method based on big data information according to claim 3, characterized in that, The extraction of key facts from the call results specifically includes: Semantically encode the user-input text, and retrieve data records consistent with the current interaction topic from the big data information database based on the semantic encoding results; The retrieved data records are sorted according to their source identifier, generation time, and semantic relevance, and the data records in the sorted results are selected as the call results. The call results are processed for text normalization, converting entity names, business objects, numerical content, time content, and status content in the call results into a unified expression form; Based on the query object and query relationship in the user's input text, extract factual content corresponding to the current interaction topic from the call results after text normalization processing, and generate key facts.

5. The digital human voice sensing and interaction method based on big data information according to claim 4, characterized in that, The fact-keeping register for writing UniLM inter-layer settings specifically includes: Input key facts and user-input text into the UniLM encoder, and generate the source hidden state through the UniLM encoder; Extract fact register units corresponding to key facts from the hidden state at the source end. Each fact register unit stores an internal fact representation corresponding to a fact category. After each Transformer layer in UniLM, a register update unit is set up. The fact register unit is refreshed from the hidden state of the source end of the current layer through the register update unit. The refreshed fact register unit is then passed to the hidden state calculation process of the target end of the next layer. When generating lexical units at the UniLM target end, the self-attention calculation result at the target end and the current state of the fact register unit are jointly input into the feedforward calculation process at the target end; When the target has generated content that covers the key facts corresponding to the fact register unit, the fact register unit is marked as used. In the subsequent decoding on the target, the fact register units that are not marked as used are read.

6. The digital human voice sensing and interaction method based on big data information according to claim 5, characterized in that, The construction of the dynamic attention mask according to the response generation stage specifically includes: In each round of word generation at the UniLM target end, the hidden state of the current layer target end is obtained, and the hidden state of the current layer target end is input into the stage state unit to generate discrete stage states. The discrete stage states are mapped to the response generation stage, which is distinguished based on the semantic position of the generated content in the digital human response text. Based on the response generation stage, the set of visible positions generated by the current round word element is determined. The set of visible positions consists of the contextual hidden state corresponding to the user input text, the fact hidden state in the fact holding register, and the hidden state corresponding to the content already generated by the target end. A dynamic attention mask is constructed based on the set of visible locations, and the dynamic attention mask is fed into the UniLM target-side self-attention calculation process; The UniLM target reads the corresponding factual hidden state and contextual hidden state according to the dynamic attention mask, generates the current round word, and updates the target's hidden state based on the current round word.

7. The digital human voice sensing and interaction method based on big data information according to claim 6, characterized in that, The specific details of retrieving supplementary facts from the big data information database based on the generated breakpoint include: After each round of decoding at the UniLM target end, the target fact category to be read at the current decoding position is determined based on the current response generation stage; Match the target fact category with the fact register cells in the fact holding register that are not marked as used; When there is no fact register unit in the fact holding register that matches the target fact category, or when there is a fact register unit that matches the target fact category but the fact state is conflicting, a corresponding generation interruption point is generated and the target decoding is paused. A re-inspection request is generated based on the target fact category corresponding to the generated interruption point. The re-inspection request is sent to the big data information database and supplementary facts are invoked. Input the supplementary facts into the UniLM encoder to generate the supplementary fact hidden state, write the supplementary fact hidden state into the fact holding register, and set the fact state to the valid state; Resume UniLM target-side decoding based on the updated fact-holding register.

8. The digital human voice sensing and interaction method based on big data information according to claim 7, characterized in that, The specific steps of performing UniLM target-side decoding include: Read the updated fact-holding register and reconstruct the dynamic attention mask for the current decoding position; Based on the dynamic attention mask, continue to perform UniLM target-side self-attention calculation to generate subsequent lexical units round by round; The generated word sequence is merged and normalized to generate the digital human's response text.

9. A digital human voice sensing and interaction method based on big data information according to claim 8, characterized in that, The specific methods for driving the digital human to output speech, lip movements, facial expressions, and actions include: Input the digital human's response text into the speech synthesis model to generate the response speech and phoneme timestamps; Match lip shape parameters according to phoneme timestamps; Perform emotion classification on the digital human's response text and generate facial expression parameters; Map action trigger words in the digital human's response text to action parameters; The response voice, lip-reading parameters, facial expression parameters, and motion parameters are simultaneously sent to the digital human rendering terminal, driving the digital human to complete the output of voice, lip-reading, facial expression, and motion.

10. A digital human voice sensing and interaction system based on big data information, applied to the digital human voice sensing and interaction method based on big data information as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The voice access module is used to receive user voice signals collected by the voice sensing device and output clear voice signals after preprocessing. The interaction understanding module is used to convert clear speech signals into user-input text and determine the current interaction intent; The data retrieval module is used to access the big data information database according to the user's input text and the current interaction intent, and to obtain the data records corresponding to the current interaction scenario. The fact management module is used to extract key facts from data records, write key facts into the fact holding register, and update the fact holding register during the generation process; The response generation module is used to call UniLM to generate digital human response text, and adjusts the dynamic attention mask according to the response generation stage during the generation process; The back-check control module is used to detect the missing facts and conflicting facts during the digital human response text generation process, and call the supplementary facts to update the fact holding register; The digital human driver module is used to generate speech, lip-reading, facial expression, and motion control parameters based on the digital human's response text, driving the digital human to complete interactive output.

Images