A big data driven digital human intelligent interaction method and system

Abstract

The present invention relates to the field of intelligent interaction technology, and discloses a big data driven digital human intelligent interaction method and system, the method comprising: extracting interactive audio and video frame sets in user historical interactive videos, performing image smoothing and image equalization on each video frame in the video frame set to obtain enhanced video frames; identifying interactive texts corresponding to interactive audios; extracting feature time sequence information of enhanced video frames, interactive texts and interactive audios respectively, performing feature interaction on the feature time sequence information to obtain multimodal features; calculating target interactive texts and corresponding target interactive voices according to multimodal features, generating target interactive expressions according to target interactive texts and target interactive voices; constructing interactive videos of target digital humans according to target interactive texts, target interactive voices and target interactive expressions, and using interactive videos to interact with users. The present invention can improve the accuracy of intelligent interaction of digital humans.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction technology, and in particular to a big data driven digital human intelligent interaction method and system. Background Art

[0002] Digital human, also known as virtual digital human or digital virtual human, is a virtual image created using digital technology. It can interact with humans through technologies such as speech recognition and natural language processing. This interaction simulates the way people communicate with each other, allowing digital humans to understand and respond to human language, expressions and behaviors. With the rapid development of artificial intelligence technology, digital human technology has become an important bridge connecting the real and virtual worlds, and has been widely used in multiple industries and fields.

[0003] However, the existing digital human intelligent interaction mainly uses TTS (Text to Speech) technology to convert text into speech, and combines the corresponding lip animation algorithm and animation library to generate actions. The content production cost is low. However, the unmodified synthesized audio and video often have monotonous movements and flat intonation, which limits the application scenarios of digital humans. That is, TTS technology needs to rely on professional equipment and artists to adjust the output results to eliminate obvious electronic sounds. At the same time, such digital humans can only perform actions in the action library according to the time points preset by the user, and the interaction is relatively simple, resulting in poor accuracy of digital human interaction. Summary of the invention

[0004] The present invention provides a big data driven digital human intelligent interaction method and system, the main purpose of which is to solve the problem of poor accuracy of digital human interaction.

[0005] To achieve the above-mentioned purpose, the present invention provides a big data driven digital human intelligent interaction method, comprising:

[0006] Acquire a user's historical interaction video, extract the interactive audio and video frame set in the user's historical interaction video, perform image smoothing and image equalization on each video frame in the video frame set, and obtain an enhanced video frame;

[0007] Calculating a one-sided autocorrelation function sequence of the interactive audio, and identifying an interactive text of the interactive audio according to the one-sided autocorrelation function sequence;

[0008] Extracting feature time sequence information of the enhanced video frame, the interactive text, and the interactive audio respectively, performing feature interaction on the feature time sequence information, and obtaining multimodal features;

[0009] Calculate the target interactive text of the target digital human and the target interactive speech corresponding to the target interactive text according to the multimodal features, and generate the target interactive expression of the target digital human according to the target interactive text and the target interactive speech;

[0010] An interactive video of the target digital human is constructed according to the target interactive text, the target interactive voice and the target interactive expression, and the interactive video is used to interact with the user.

[0011] Optionally, performing image smoothing and image equalization on each video frame in the video frame set to obtain an enhanced video frame includes:

[0012] Performing Fourier transform on each video frame in the video frame set to obtain an image frequency domain;

[0013] Performing frequency domain filtering on the image frequency domain to obtain an image filtering frequency domain, and performing inverse Fourier transform on the image filtering frequency domain to obtain a frequency domain filtering image;

[0014] The frequency domain of the image is filtered using the following formula: in, Indicates that the image frequency domain is The image filtering frequency domain at Indicated in The image frequency domain at Indicates the preset cutoff frequency, Indicates the preset low-pass filter order;

[0015] Performing template smoothing on the frequency domain filtered image to obtain a smoothed image;

[0016] Histogram equalization is performed on the smoothed image to obtain an enhanced image frame corresponding to each of the video frames.

[0017] Optionally, the calculating the one-sided autocorrelation function sequence of the interactive audio includes:

[0018] Performing signal division on the interactive audio to obtain multiple voice signal frames;

[0019] Performing Hamming window processing on each of the speech signal frames to obtain a windowed speech signal;

[0020] An autocorrelation function of the windowed speech signal is calculated, and a unilateral autocorrelation function sequence is generated according to the autocorrelation function.

[0021] Optionally, the identifying the interactive text of the interactive audio according to the one-sided autocorrelation function sequence includes:

[0022] Smoothing the one-sided autocorrelation function sequence to obtain a target autocorrelation function sequence;

[0023] Calculating the autocorrelation function of the target autocorrelation function sequence, and calculating the autoregressive coefficient according to the autocorrelation function;

[0024] The autocorrelation function of the target autocorrelation function sequence is calculated using the following formula: in, represents the autocorrelation function, represents a first-order non-negative time delay, Represents the windowed speech signal corresponding to the target autocorrelation function in the target autocorrelation function sequence The signal length, represents the time delay of the second-order autocorrelation function, Indicates the windowed speech signal The value of samples;

[0025] The cepstral coefficients of the interactive audio are calculated according to the autoregressive coefficients, and the interactive text of the interactive audio is identified according to the cepstral coefficients.

[0026] Optionally, respectively extracting characteristic timing information of the enhanced video frame, the interactive text, and the interactive audio includes:

[0027] Extracting the enhanced video frame, the interactive text, and the interactive audio respectively for sample division to obtain a sample sequence;

[0028] Performing feature convolution on the sample sequences respectively to obtain convolution feature vectors;

[0029] Performing bidirectional gated loop processing on the convolution feature vector to obtain a bidirectional hidden feature;

[0030] Performing multi-head self-attention calculation on the bidirectional hidden features to obtain feature weights;

[0031] The bidirectional hidden features are weightedly calculated according to the feature weights to obtain feature timing information of the enhanced video frame, the interactive text, and the interactive audio.

[0032] Optionally, performing feature interaction on the feature time series information to obtain multimodal features includes:

[0033] Calculating the correlation scores between the feature time series information;

[0034] The correlation score between the characteristic time series information is calculated using the following formula: in, Indicates Feature time series information With Feature time series information The correlation score between represents the preset activation function, Indicates the preset relevant weight;

[0035] Performing correlation interaction on the feature time series information according to the correlation score to obtain an interaction feature;

[0036] The interactive features are fused to obtain multimodal features.

[0037] Optionally, the step of calculating a target interactive text of a target digital human and a target interactive voice corresponding to the target interactive text according to the multimodal features includes:

[0038] Performing context prediction based on the multimodal features to obtain a target interactive text of the target digital human;

[0039] Performing feature encoding on the target interactive text to obtain text features;

[0040] Performing sentiment classification on the target interactive text according to the text features to obtain text sentiment corresponding to the target interactive text;

[0041] The target interactive text is converted into speech according to the text emotion to obtain the target interactive speech.

[0042] Optionally, generating a target interactive expression of the target digital human according to the target interactive text and the target interactive voice includes:

[0043] Preprocessing the target interactive text and the target interactive speech respectively to obtain enhanced interactive text and enhanced interactive speech;

[0044] Extracting interactive features of the enhanced interactive text and the enhanced interactive speech;

[0045] Performing feature splicing on the interaction features to obtain fused interaction features;

[0046] The interactive emotion of the target digital human is identified according to the fused interactive feature, and the target interactive expression of the target digital human is determined according to the interactive emotion.

[0047] Optionally, constructing the target digital human's interactive video according to the target interactive text, the target interactive voice and the target interactive expression includes:

[0048] Dynamically rendering the target digital human according to the target interactive voice and the target interactive expression to obtain a digital human video;

[0049] Performing time calibration on the target interactive text and the digital human video to obtain text time;

[0050] The digital human video is annotated with text according to the text time to obtain the interactive video of the target digital human.

[0051] In order to solve the above problems, the present invention also provides a digital human intelligent interaction system driven by big data, the system comprising:

[0052] A video frame enhancement module is used to obtain a user's historical interaction video, extract the interactive audio and video frame set in the user's historical interaction video, and perform image smoothing and image equalization on each video frame in the video frame set to obtain an enhanced video frame;

[0053] An interactive text recognition module, used to calculate a one-sided autocorrelation function sequence of the interactive audio, and recognize the interactive text of the interactive audio according to the one-sided autocorrelation function sequence;

[0054] A multimodal feature calculation module, used to extract feature time sequence information of the enhanced video frame, the interactive text and the interactive audio respectively, perform feature interaction on the feature time sequence information, and obtain multimodal features;

[0055] A target interaction expression calculation module is used to calculate the target interaction text of the target digital human and the target interaction voice corresponding to the target interaction text according to the multimodal features, and generate the target interaction expression of the target digital human according to the target interaction text and the target interaction voice;

[0056] The user interaction module is used to construct an interactive video of the target digital human according to the target interactive text, the target interactive voice and the target interactive expression, and utilize the interactive video to interact with the user.

[0057] The embodiment of the present invention can improve the image quality of the video frame and obtain an enhanced video frame by extracting the interactive audio and video frame set in the user's historical interactive video, and performing image smoothing and image equalization on each video frame in the video frame set; identifying the interactive text of the interactive audio can more accurately obtain the user's language during the interaction, which is conducive to the subsequent understanding and response of the digital human and improves the accuracy of the digital human interaction; extracting multimodal feature information can more accurately calculate the target interactive text and the corresponding target interactive voice of the target digital human, which can improve the user interaction experience and achieve more accurate interaction; according to the target interactive text, target interactive voice and target interactive expression, the multimodal information in the user's historical interactive video can be used to flexibly generate the target interactive text and the emotional target interactive voice, so as to achieve accurate user interaction. Therefore, the big data-driven digital human intelligent interaction method and system proposed by the present invention can solve the problem of poor accuracy of digital human interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a big data driven digital human intelligent interaction method provided in one embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a process of performing image smoothing and image equalization on each video frame in a video frame set provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a flow chart of calculating a one-sided autocorrelation function sequence of interactive audio provided by an embodiment of the present invention;

[0061] Figure 4 A functional module diagram of a big data-driven digital human intelligent interaction system provided in one embodiment of the present invention.

[0062] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0064] The embodiment of the present application provides a big data-driven digital human intelligent interaction method. The execution subject of the big data-driven digital human intelligent interaction method includes but is not limited to at least one of the electronic devices such as the server, the terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the big data-driven digital human intelligent interaction method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0065] Reference Figure 1 FIG. 1 is a flow chart of a big data driven digital human intelligent interaction method provided by an embodiment of the present invention. In this embodiment, the big data driven digital human intelligent interaction method includes:

[0066] S1. Obtain a user's historical interaction video, extract the interactive audio and video frame set in the user's historical interaction video, perform image smoothing and image equalization on each video frame in the video frame set, and obtain an enhanced video frame.

[0067] In the embodiment of the present invention, the user history plus interactive video is an interactive experience between the user and the digital human through the video medium, including the voice when the digital human interacts with the user, the digital human giving corresponding facial expressions according to the user's status and other video information.

[0068] In an embodiment of the present invention, the audio track channel in the user's historical interaction video is extracted to obtain the interactive audio, and the video frames are directly extracted from the user's historical interaction video at preset time intervals, for example, one video frame is extracted from the user's historical interaction video every 0.5 seconds or every 1 second to obtain a video frame set, wherein the video frames can be extracted using the OpenCV library by writing a Python script, for example, using cv2.VideoCapture to read the video, and then looping through the video frame set to extract the interactive audio from the user's historical interaction video using online conversion tools such as Clipchamp and Zamzar.

[0069] Furthermore, multiple consecutive video frames in a video frame set may have problems such as low image resolution, insufficient contrast, blurred edges, etc., which may lead to poor image quality in the video frame set and require image enhancement to obtain enhanced video frames.

[0070] In the embodiment of the present invention, refer to Figure 2 As shown, performing image smoothing and image equalization on each video frame in the video frame set to obtain an enhanced video frame includes:

[0071] S21, performing Fourier transform on each video frame in the video frame set to obtain an image frequency domain;

[0072] S22, performing frequency domain filtering on the image frequency domain to obtain an image filtering frequency domain, and performing inverse Fourier transform on the image filtering frequency domain to obtain a frequency domain filtering image;

[0073] S23, performing template smoothing on the frequency domain filtered image to obtain a smoothed image;

[0074] S24, performing histogram equalization on the smoothed image to obtain an enhanced image frame corresponding to each of the video frames.

[0075] In the embodiment of the present invention, Fourier transform is to decompose a graphic into a superposition of sine and cosine functions of different frequencies, which means to decompose an image into grayscale changes of different frequencies.

[0076] Furthermore, frequency domain filtering uses a low-pass filter to smooth the image in the frequency domain, which can improve the problem of blurred edge information of video frames. The frequency domain representation can be converted back to the spatial domain through inverse Fourier transform to obtain a frequency domain filtered image.

[0077] Specifically, the frequency domain of the image is filtered using the following formula: in, Indicates that the image frequency domain is The image filtering frequency domain at Indicated in The image frequency domain at Indicates the preset cutoff frequency, Indicates the preset low-pass filter order.

[0078] Among them, an n-th order Butterworth low-pass filter can be used to perform frequency smoothing on the image in the frequency domain.

[0079] In the embodiment of the present invention, template smoothing is to use template operation to calculate the neighborhood of each pixel in the frequency domain filter image, and to obtain a smoothed image by agreeing the pixel value of the frequency domain filter image after smoothing by the value of the own pixel and the neighborhood point. For example, each pixel in the frequency domain filter image and its 8 neighboring pixels are averaged to eliminate variant pixels and reduce noise interference contained in the image. The Gaussian template can be used to perform convolution with the frequency domain filter image to achieve template smoothing of the frequency domain filter image and obtain a smoothed image.

[0080] Furthermore, since the image smoothing process requires averaging or integration operations, smoothing the frequency domain filtered image will cause the edge clarity of the frequency domain filtered image to decrease. In order to improve the edge clarity of the frequency domain filtered image, it is necessary to perform histogram equalization on the smoothed image.

[0081] In detail, histogram equalization is to transform the pixel values ​​in the smoothed image by calculating the cumulative distribution function of each pixel value in the grayscale histogram of the smoothed image to obtain an enhanced image frame after equalization. Histogram equalization can make the grayscale distribution of the enhanced image frame more uniform, thereby improving the contrast and visual effect of the enhanced image frame.

[0082] In the embodiment of the present invention, by performing image smoothing and image equalization on each video frame in the video frame set, problems such as low video frame image resolution, insufficient contrast, and blurred edges can be effectively solved, thereby improving the image quality of the video frame.

[0083] S2. Calculate a one-sided autocorrelation function sequence of the interactive audio, and identify an interactive text of the interactive audio according to the one-sided autocorrelation function sequence.

[0084] In the embodiment of the present invention, the one-sided autocorrelation function sequence is a function sequence formed by calculating the autocorrelation function of the interactive speech, and can be used to analyze the signal periodicity characteristics of the interactive speech.

[0085] In the embodiment of the present invention, refer to Figure 3 As shown, the calculating of the one-sided autocorrelation function sequence of the interactive audio includes:

[0086] S31, dividing the interactive audio signal to obtain multiple voice signal frames;

[0087] S32, performing Hamming window processing on each of the speech signal frames to obtain a windowed speech signal;

[0088] S33, calculating the autocorrelation function of the windowed speech signal, and generating a unilateral autocorrelation function sequence according to the autocorrelation function.

[0089] In an embodiment of the present invention, the audio signal of the interactive audio can be divided into multiple voice signal frames according to a preset voice signal frame length, for example, divided into M voice signal frames with a frame length of N, and then each voice signal frame is windowed, wherein the Hamming window can be pre-set. The spectral leakage of the voice signal frame can be reduced through the Hamming window processing, so that the spectral analysis result is more accurate, thereby improving the accuracy of subsequent interactive text recognition.

[0090] In detail, the autocorrelation function of each windowed speech signal can be calculated using a short-time autocorrelation function, which is obtained by intercepting a section of the signal with a short-time window near the sample point of the signal and calculating the autocorrelation. Since the autocorrelation function sequence is even-symmetric, only the autocorrelation function of the even-numbered sequence can be taken to form a unilateral autocorrelation function sequence of the interactive audio.

[0091] In the embodiment of the present invention, the step of identifying the interactive text of the interactive audio according to the one-sided autocorrelation function sequence includes:

[0092] Smoothing the one-sided autocorrelation function sequence to obtain a target autocorrelation function sequence;

[0093] Calculating the autocorrelation function of the target autocorrelation function sequence, and calculating the autoregressive coefficient according to the autocorrelation function;

[0094] The cepstral coefficients of the interactive audio are calculated according to the autoregressive coefficients, and the interactive text of the interactive audio is identified according to the cepstral coefficients.

[0095] In an embodiment of the present invention, smoothing the one-sided autocorrelation function sequence is filtering the one-sided autocorrelation function sequence to suppress noise information of the one-sided autocorrelation function sequence, wherein a FIR (finite impulse response) filter can be used to smooth the one-sided autocorrelation function sequence to obtain a target autocorrelation function sequence.

[0096] Furthermore, the target autocorrelation function sequence is a discrete signal, and the second-order autocorrelation function of the one-sided autocorrelation function sequence in the target autocorrelation function sequence is calculated for the discrete signal, and the autocorrelation of the autocorrelation function is further calculated.

[0097] In detail, the autocorrelation function of the target autocorrelation function sequence is calculated using the following formula: in, represents the autocorrelation function, represents a first-order non-negative time delay, Represents the windowed speech signal corresponding to the target autocorrelation function in the target autocorrelation function sequence The signal length, represents the time delay of the second-order autocorrelation function, Indicates the windowed speech signal The value of samples.

[0098] In an embodiment of the present invention, multiple autocorrelation functions can be obtained through a target autocorrelation function sequence, and then the coefficients of an autoregressive (AR) model can be calculated using a conventional Levinson-Durbin algorithm, the autoregressive coefficients of speech can be estimated, and then the interactive text of the interactive audio can be recognized.

[0099] Furthermore, the cepstral coefficient is a feature extracted from the spectrum of the speech signal, which can capture the important features of the speech signal, such as rhythm, pitch, timbre, etc.

[0100] Specifically, the power spectral density (PSD) of the interactive audio can be calculated through the autoregressive coefficients, the logarithm of the power spectral density can be obtained, and the inverse Fourier Transform (IFT) of the logarithmic power spectrum can be performed to obtain the cepstral coefficients. The cepstral coefficients can capture the periodic and non-periodic components of the speech signal of the interactive audio, which is conducive to identifying the interactive text of the interactive audio.

[0101] In the embodiment of the present invention, the step of identifying the interactive text of the interactive audio according to the cepstral coefficients includes:

[0102] Constructing a Mel-frequency cepstral coefficient feature of the interactive audio according to the cepstral coefficient;

[0103] The interactive text of the interactive audio is recognized using a pre-built speech recognition model according to the Mel-frequency cepstral coefficient features.

[0104] In an embodiment of the present invention, the cepstral coefficients corresponding to the windowed speech signals of adjacent frames are concatenated to obtain Mel-frequency cepstral coefficient features, and speech recognition can be performed through the Mel-frequency cepstral coefficient features. Among them, the speech recognition model can be a pre-trained deep learning model such as a neural network structure based on CNN, RNN and LSTM, which can realize accurate recognition and conversion of human interaction audio to obtain interactive text.

[0105] In the embodiment of the present invention, by identifying the interactive text of the interactive audio, the language of the user during the interaction can be obtained more accurately, thereby helping the digital human to perform subsequent understanding and response, thereby improving the accuracy of the digital human interaction.

[0106] S3. Extract feature timing information of the enhanced video frame, the interactive text, and the interactive audio respectively, perform feature interaction on the feature timing information, and obtain multimodal features.

[0107] In an embodiment of the present invention, the feature timing information is to obtain the feature information of the enhanced video frames, interactive text and interactive audio in timing respectively, and obtain the visual, text and audio feature vectors corresponding to the enhanced video frames, interactive text and interactive audio with contextual timing information.

[0108] In the embodiment of the present invention, respectively extracting the characteristic timing information of the enhanced video frame, the interactive text, and the interactive audio includes:

[0109] Extracting the enhanced video frame, the interactive text, and the interactive audio respectively for sample division to obtain a sample sequence;

[0110] Performing feature convolution on the sample sequences respectively to obtain convolution feature vectors;

[0111] Performing bidirectional gated loop processing on the convolution feature vector to obtain a bidirectional hidden feature;

[0112] Performing multi-head self-attention calculation on the bidirectional hidden features to obtain feature weights;

[0113] The bidirectional hidden features are weightedly calculated according to the feature weights to obtain feature timing information of the enhanced video frame, the interactive text, and the interactive audio.

[0114] In detail, the enhanced video frames, interactive texts and interactive audios can be sampled and divided according to different modalities into sample sequences corresponding to the same number of enhanced video frames, interactive texts and interactive audios, wherein the enhanced video frames, interactive texts and interactive audios can be divided according to different division methods selected. For example, enhanced video frames can be divided according to number, interactive texts can be divided according to pause symbols in the text, and interactive audio can be divided according to pauses in speech and switching of speakers.

[0115] Furthermore, a pre-built convolutional neural network can be used to perform feature convolution on the sample sequence to map the features of the sample sequence to the same dimension to obtain a convolution feature vector, wherein the convolutional neural network can be a one-dimensional convolution kernel; after processing through the one-dimensional convolution layer, the obtained features are input into the BiGRU (bidirectional gated recurrent unit) for bidirectional gated recurrent processing to obtain the bidirectional hidden state of the convolution feature vector at each moment, which can capture the long-term dependency of the context, and then use the weights of the context features in the multi-head self-attention mechanism to perform weighted calculation on the bidirectional hidden features according to the feature weights, thereby obtaining the importance of different bidirectional hidden features, and concatenating the weighted calculated features to improve the accuracy of the feature timing information.

[0116] In the embodiment of the present invention, performing feature interaction on the feature time series information to obtain multimodal features includes:

[0117] Calculating the correlation scores between the feature time series information;

[0118] Performing correlation interaction on the feature time series information according to the correlation score to obtain an interaction feature;

[0119] The interactive features are fused to obtain multimodal features.

[0120] Specifically, the correlation score between the characteristic time series information is calculated using the following formula: in, Indicates Feature time series information With Feature time series information The correlation score between represents the preset activation function, Indicates the preset correlation weight.

[0121] In detail, by superimposing and multiplying the relevant scores with the feature timing information, we obtain interactive features, such as the interactive features between enhanced video frames and interactive texts, the interactive features between enhanced video frames and interactive audios, the interactive features between interactive texts and interactive audios, etc., and then by feature splicing, we obtain the final fusion of the enhanced video frames, interactive texts, and multimodal feature information corresponding to the interactive audio, so as to obtain more accurate multimodal features.

[0122] S4. Calculate the target interaction text of the target digital human and the target interaction voice corresponding to the target interaction text according to the multimodal features, and generate the target interaction expression of the target digital human according to the target interaction text and the target interaction voice.

[0123] In the embodiment of the present invention, the target interaction text is the text output by the target digital human based on the multimodal features corresponding to the user's historical interaction video, and the target interaction voice is the voice signal corresponding to the target interaction text. Through the target interaction text and the target interaction voice, the digital human interaction is performed in different forms.

[0124] In the embodiment of the present invention, the step of calculating the target interactive text of the target digital human and the target interactive speech corresponding to the target interactive text according to the multimodal features includes:

[0125] Performing context prediction based on the multimodal features to obtain a target interactive text of the target digital human;

[0126] Performing feature encoding on the target interactive text to obtain text features;

[0127] Performing sentiment classification on the target interactive text according to the text features to obtain text sentiment corresponding to the target interactive text;

[0128] The target interactive text is converted into speech according to the text emotion to obtain the target interactive speech.

[0129] In an embodiment of the present invention, after obtaining the multimodal features, a full connection calculation may be performed on the multimodal features to predict the following content of the interactive text and obtain the target interactive text output by the target digital human.

[0130] In an embodiment of the present invention, the target interaction text can be feature encoded through a pre-trained language model to quickly extract the text features of the target interaction text and improve the efficiency of digital human interaction. The emotion labels are predicted by the fully connected layer of the emotion classification through the text features to obtain the emotions in the target interaction text. The target interaction text is converted into speech using the text emotion to obtain the target interaction speech with emotion. Specifically, the target interaction speech can be synthesized using the existing emotional speech synthesis technology.

[0131] In the embodiment of the present invention, the target interaction expression is the expression of the target digital person during interaction. The emotion of the target digital person can be further determined according to the target interaction voice and the target interaction text, and then the corresponding target interaction expression can be determined.

[0132] In the embodiment of the present invention, generating the target interactive expression of the target digital human according to the target interactive text and the target interactive voice includes:

[0133] Preprocessing the target interactive text and the target interactive speech respectively to obtain enhanced interactive text and enhanced interactive speech;

[0134] Extracting interactive features of the enhanced interactive text and the enhanced interactive speech;

[0135] Performing feature splicing on the interaction features to obtain fused interaction features;

[0136] The interactive emotion of the target digital human is identified according to the fused interactive feature, and the target interactive expression of the target digital human is determined according to the interactive emotion.

[0137] In an embodiment of the present invention, preprocessing is to perform text cleaning and voice signal filtering on the target interactive text to obtain more accurate enhanced interactive text and enhanced interactive voice, and use the above-mentioned method of extracting feature time series information to perform feature extraction respectively to obtain corresponding interactive features, and perform feature splicing on the interactive features to obtain fused interactive features.

[0138] Furthermore, the emotional labels of the fused interaction features are predicted to obtain the interaction emotions, and the interaction emotions are matched with preset expressions to obtain target interaction expressions, such as happy expressions, comforting expressions, sad expressions, etc. The target interaction expressions can make the target digital human more human, improve the user interaction experience, and achieve more accurate interaction.

[0139] S5. Construct an interactive video of the target digital human according to the target interactive text, the target interactive voice and the target interactive expression, and utilize the interactive video to interact with the user.

[0140] In the embodiment of the present invention, the interactive video is a video of the target digital human interacting with the user. Through the video, the target digital human can interact with the user from multiple angles in the form of text, voice, and video, thereby effectively improving the effect of user interaction.

[0141] In the embodiment of the present invention, the step of constructing the target digital human's interactive video according to the target interactive text, the target interactive voice and the target interactive expression includes:

[0142] Dynamically rendering the target digital human according to the target interactive voice and the target interactive expression to obtain a digital human video;

[0143] Performing time calibration on the target interactive text and the digital human video to obtain text time;

[0144] The digital human video is annotated with text according to the text time to obtain the interactive video of the target digital human.

[0145] In the embodiment of the present invention, the target digital human can be dynamically rendered through 3D modeling tools, etc., and a digital human video with realistic expressions and lip shapes can be generated. The target interactive text and the target interactive voice of the video human video can be aligned, and then text annotation can be performed on the digital human video to obtain an interactive video with a subtitle function, thereby providing a multi-visual interactive video.

[0146] In an embodiment of the present invention, the interactive video is pushed to the interactive platform corresponding to the target digital human to realize the interaction between the target digital human and the user. By interacting through the interactive video, the multimodal information in the user's historical interactive video can be used to flexibly generate target interactive text and emotional target interactive voice, thereby realizing accurate user interaction.

[0147] like Figure 4 , which is a functional module diagram of a big data driven digital human intelligent interaction system provided by an embodiment of the present invention.

[0148] The big data driven digital human intelligent interaction system 400 of the present invention can be installed in an electronic device. According to the functions implemented, the big data driven digital human intelligent interaction system 400 can include a video frame enhancement module 401, an interactive text recognition module 402, a multimodal feature calculation module 403, a target interactive expression calculation module 404 and a user interaction module 405. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0149] In this embodiment, the functions of each module / unit are as follows:

[0150] The video frame enhancement module 401 is used to obtain a user's historical interaction video, extract the interactive audio and video frame set in the user's historical interaction video, and perform image smoothing and image equalization on each video frame in the video frame set to obtain an enhanced video frame;

[0151] The interactive text recognition module 402 is used to calculate the one-sided autocorrelation function sequence of the interactive audio, and recognize the interactive text of the interactive audio according to the one-sided autocorrelation function sequence;

[0152] The multimodal feature calculation module 403 is used to extract feature time sequence information of the enhanced video frame, the interactive text and the interactive audio respectively, perform feature interaction on the feature time sequence information, and obtain multimodal features;

[0153] The target interaction expression calculation module 404 is used to calculate the target interaction text of the target digital human and the target interaction speech corresponding to the target interaction text according to the multimodal features, and generate the target interaction expression of the target digital human according to the target interaction text and the target interaction speech;

[0154] The user interaction module 405 is used to construct an interaction video of the target digital human according to the target interaction text, the target interaction voice and the target interaction expression, and interact with the user by using the interaction video.

[0155] In detail, each module described in the big data driven digital human intelligent interaction system 400 in the embodiment of the present invention is used in the same manner as described above. Figures 1 to 3 The technical means are the same as the big data driven digital human intelligent interaction method described in, and can produce the same technical effects, so I will not go into details here.

[0156] The present invention also provides an electronic device, which may include a processor, a memory, a communication bus and a communication interface, and may also include a computer program stored in the memory and executable on the processor, such as a big data-driven digital human intelligent interaction method program.

[0157] In some embodiments, the processor may be composed of an integrated circuit, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and a combination of various control chips.

[0158] The memory includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device.

[0159] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory and at least one processor, etc.

[0160] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface.

[0161] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0162] Specifically, the specific implementation method of the processor for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0164] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0166] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0167] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0168] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0169] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A big data driven digital human intelligent interaction method, characterized in that: The method comprises: Acquire a user's historical interaction video, extract the interactive audio and video frame set in the user's historical interaction video, perform image smoothing and image equalization on each video frame in the video frame set, and obtain an enhanced video frame; Calculate the one-sided autocorrelation function sequence of the interactive audio, and identify the interactive text of the interactive audio according to the one-sided autocorrelation function sequence; wherein, identifying the interactive text of the interactive audio according to the one-sided autocorrelation function sequence includes: smoothing the one-sided autocorrelation function sequence to obtain a target autocorrelation function sequence; calculate the autocorrelation function of the target autocorrelation function sequence, and calculate the autoregressive coefficient according to the autocorrelation function; The autocorrelation function of the target autocorrelation function sequence is calculated using the following formula: Wherein, R represents the autocorrelation function, k represents the first-order non-negative time delay, N represents the signal length of the windowed speech signal x(a) corresponding to the target autocorrelation function in the target autocorrelation function sequence, m represents the time delay of the second-order autocorrelation function, and a represents the value of the a-th sample in the windowed speech signal; Calculate the cepstral coefficients of the interactive audio according to the autoregressive coefficients, and identify the interactive text of the interactive audio according to the cepstral coefficients; Extracting feature time sequence information of the enhanced video frame, the interactive text, and the interactive audio respectively, performing feature interaction on the feature time sequence information, and obtaining multimodal features; wherein performing feature interaction on the feature time sequence information to obtain multimodal features includes: calculating correlation scores between the feature time sequence information; The correlation score between the characteristic time series information is calculated using the following formula: Among them, P it Represents the i-th feature time series information Z i and the tth feature time series information Z t The correlation score between them, f(·) represents the preset activation function, and W represents the preset correlation weight; Performing correlation interaction on the feature time series information according to the correlation score to obtain interaction features; performing feature fusion on the interaction features to obtain multimodal features; Calculate the target interactive text of the target digital human and the target interactive speech corresponding to the target interactive text according to the multimodal features, and generate the target interactive expression of the target digital human according to the target interactive text and the target interactive speech; An interactive video of the target digital human is constructed according to the target interactive text, the target interactive voice and the target interactive expression, and the interactive video is used to interact with the user.

2. The big data driven digital human intelligent interaction method according to claim 1, characterized in that: The performing image smoothing and image equalization on each video frame in the video frame set to obtain an enhanced video frame includes: Performing Fourier transform on each video frame in the video frame set to obtain an image frequency domain; Performing frequency domain filtering on the image frequency domain to obtain an image filtering frequency domain, and performing inverse Fourier transform on the image filtering frequency domain to obtain a frequency domain filtering image; The frequency domain of the image is filtered using the following formula: Wherein, H(u,v) represents the image filter frequency domain at (u,v), D(u,v) represents the image frequency domain at (u,v), D0 represents the preset cutoff frequency, and n represents the preset low-pass filter order; Performing template smoothing on the frequency domain filtered image to obtain a smoothed image; Histogram equalization is performed on the smoothed image to obtain an enhanced image frame corresponding to each of the video frames.

3. The big data driven digital human intelligent interaction method according to claim 1, characterized in that: The calculating of the one-sided autocorrelation function sequence of the interactive audio comprises: Performing signal division on the interactive audio to obtain multiple voice signal frames; Performing Hamming window processing on each of the speech signal frames to obtain a windowed speech signal; An autocorrelation function of the windowed speech signal is calculated, and a unilateral autocorrelation function sequence is generated according to the autocorrelation function.

4. The big data driven digital human intelligent interaction method according to claim 1, characterized in that: The extracting the characteristic timing information of the enhanced video frame, the interactive text, and the interactive audio respectively includes: Extracting the enhanced video frame, the interactive text, and the interactive audio respectively for sample division to obtain a sample sequence; Performing feature convolution on the sample sequences respectively to obtain convolution feature vectors; Performing bidirectional gated loop processing on the convolution feature vector to obtain a bidirectional hidden feature; Performing multi-head self-attention calculation on the bidirectional hidden features to obtain feature weights; The bidirectional hidden features are weightedly calculated according to the feature weights to obtain feature timing information of the enhanced video frame, the interactive text, and the interactive audio.

5. The big data driven digital human intelligent interaction method according to claim 1, characterized in that: The step of calculating the target interactive text of the target digital human and the target interactive speech corresponding to the target interactive text according to the multimodal features includes: Performing context prediction based on the multimodal features to obtain a target interactive text of the target digital human; Performing feature encoding on the target interactive text to obtain text features; Performing sentiment classification on the target interactive text according to the text features to obtain text sentiment corresponding to the target interactive text; The target interactive text is converted into speech according to the text emotion to obtain the target interactive speech.

6. The big data driven digital human intelligent interaction method according to claim 1, characterized in that: The generating the target interactive expression of the target digital human according to the target interactive text and the target interactive voice comprises: Preprocessing the target interactive text and the target interactive speech respectively to obtain enhanced interactive text and enhanced interactive speech; Extracting interactive features of the enhanced interactive text and the enhanced interactive speech; Performing feature splicing on the interaction features to obtain fused interaction features; The interactive emotion of the target digital human is identified according to the fused interactive feature, and the target interactive expression of the target digital human is determined according to the interactive emotion.

7. The big data driven digital human intelligent interaction method according to claim 1, characterized in that: The step of constructing the target digital human's interactive video according to the target interactive text, the target interactive voice and the target interactive expression includes: Dynamically rendering the target digital human according to the target interactive voice and the target interactive expression to obtain a digital human video; Performing time calibration on the target interactive text and the digital human video to obtain text time; The digital human video is annotated with text according to the text time to obtain the interactive video of the target digital human.

8. A big data driven digital human intelligent interaction system, used to implement the big data driven digital human intelligent interaction method according to any one of claims 1 to 7, characterized in that: The system comprises: A video frame enhancement module is used to obtain a user's historical interaction video, extract the interactive audio and video frame set in the user's historical interaction video, and perform image smoothing and image equalization on each video frame in the video frame set to obtain an enhanced video frame; An interactive text recognition module is used to calculate the one-sided autocorrelation function sequence of the interactive audio, and identify the interactive text of the interactive audio according to the one-sided autocorrelation function sequence; wherein, identifying the interactive text of the interactive audio according to the one-sided autocorrelation function sequence includes: smoothing the one-sided autocorrelation function sequence to obtain a target autocorrelation function sequence; calculating the autocorrelation function of the target autocorrelation function sequence, and calculating the autoregressive coefficient according to the autocorrelation function; The autocorrelation function of the target autocorrelation function sequence is calculated using the following formula: Wherein, R represents the autocorrelation function, k represents the first-order non-negative time delay, N represents the signal length of the windowed speech signal x(a) corresponding to the target autocorrelation function in the target autocorrelation function sequence, m represents the time delay of the second-order autocorrelation function, and a represents the value of the a-th sample in the windowed speech signal; Calculate the cepstral coefficients of the interactive audio according to the autoregressive coefficients, and identify the interactive text of the interactive audio according to the cepstral coefficients; A multimodal feature calculation module, used to extract the feature time sequence information of the enhanced video frame, the interactive text and the interactive audio respectively, perform feature interaction on the feature time sequence information, and obtain multimodal features; wherein, performing feature interaction on the feature time sequence information to obtain multimodal features includes: calculating the correlation scores between the feature time sequence information; The correlation score between the characteristic time series information is calculated using the following formula: Among them, P it Represents the i-th feature time series information Z i and the tth feature time series information Z t The correlation score between them, f(·) represents the preset activation function, and W represents the preset correlation weight; Performing correlation interaction on the feature time series information according to the correlation score to obtain interaction features; performing feature fusion on the interaction features to obtain multimodal features; A target interaction expression calculation module is used to calculate the target interaction text of the target digital human and the target interaction voice corresponding to the target interaction text according to the multimodal features, and generate the target interaction expression of the target digital human according to the target interaction text and the target interaction voice; The user interaction module is used to construct an interactive video of the target digital human according to the target interactive text, the target interactive voice and the target interactive expression, and utilize the interactive video to interact with the user.

Images