Digital human interaction system and method based on big data information

Through a digital human interaction system based on big data information, the voice, action and expression features in user interaction video are extracted, combined with speech analysis and feature fusion, more accurate and real digital human interaction is achieved, and the accuracy rate in the existing technology is solved Low problem.

CN118800277BActive Publication Date: 2024-12-06SHENZHEN SPACE DIGITAL TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411277574.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-12-06
Estimated Expiration
2044-09-12

AI Technical Summary

Technical Problem

The existing digital human interaction method is based on template indexing, which leads to poor context understanding ability and difficulty in understanding user emotional changes. The lack of personalization of interaction templates leads to low accuracy of digital human interaction.

Method used

Using a digital human interaction system based on big data information, the feature extraction module extracts speech, action and expression features from user interaction videos, combined with the speech analysis module for speech recognition and emotion analysis, and the feature fusion module performs emotional feature fusion and semantic analysis. The feature mapping module performs text replies and speech synthesis, and the video interaction module performs digital human replies video rendering.

Benefits of technology

The accuracy and authenticity of digital human interaction are improved, and the accuracy and authenticity of digital human replies are enhanced through multi-dimensional feature analysis and emotional semantic recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118800277B_ABST
    Figure CN118800277B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data interaction technology, and invents a digital human interaction system and method based on big data information, including extracting interactive voice features, interactive action features and interactive expression features from user interactive videos; extracting interactive voice text and voice emotion features from interactive voice features; extracting action emotion features from interactive action features and expression emotion features from interactive expression features respectively, performing emotion feature fusion and emotion semantic analysis on voice emotion features, action emotion features and expression emotion features to obtain interactive emotion semantics; generating emotion reply voice, emotion reply expression and emotion reply action according to interactive emotion semantics; performing digital human interactive animation rendering and video interaction according to emotion reply voice, emotion reply expression and emotion reply action. The present invention can improve the accuracy of digital human interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data interaction technology, and in particular to a digital human interaction system and method based on big data information. Background Art

[0002] "Digital human" refers to a virtual human image or digital avatar created through computer technology, artificial intelligence, virtual reality, 3D modeling and other technical means. Digital human can play a role in many fields, including entertainment, education, medical care, customer service, advertising, and social media.

[0003] Most of the existing digital human interaction methods are based on template indexing, that is, identifying the user's interaction semantics, retrieving the interaction template corresponding to the interaction semantics from a preset interaction template database, and implementing interaction with the user according to the interaction template. In actual application, the interaction method based on template indexing has poor context understanding ability and is difficult to understand the user's emotional changes. In addition, the interaction template lacks personalization and appears to be relatively stiff during interaction, which leads to low accuracy in digital human interaction. Summary of the invention

[0004] The present invention provides a digital human interaction system and method based on big data information, the main purpose of which is to solve the problem of low accuracy when performing digital human interaction.

[0005] To achieve the above-mentioned purpose, the present invention provides a digital human interaction system based on big data information, the system comprising a feature extraction module, a speech analysis module, a feature fusion module, a feature mapping module and a video interaction module, wherein:

[0006] The feature extraction module is used to collect user interaction data to obtain a user interaction video, and extract interactive voice features, interactive action features and interactive expression features from the user interaction video, respectively. When the feature extraction module extracts the interactive voice features, interactive action features and interactive expression features from the user interaction video, respectively, it specifically includes: performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequence; performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio; performing frequency domain transformation on the windowed voice audio to obtain interactive voice features; performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence; performing motion estimation on the compensated frame image sequence to obtain a user motion function; performing image dynamic denoising on the compensated frame image sequence according to the user motion function using the following dynamic denoising algorithm to obtain a denoised frame image sequence:

[0007]

[0008] in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, z is the translation vector between the pixel coordinate x and the pixel coordinate y, || is the absolute value symbol, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the user motion The mathematical symbol of the function, M(y+z,x+z) is the pixel coordinates corresponding to the motion compensation result of the pixel point with pixel coordinates (y+z) calculated by the user motion function relative to the pixel point with pixel coordinates (x+z), v(y+z) is the pixel value of the pixel point with pixel coordinates (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel point with pixel coordinates y in the compensated frame image sequence; perform person recognition and background separation operations on the denoised frame image sequence to obtain a person image sequence; extract interactive action features and interactive expression features from the person image sequence respectively;

[0009] The speech analysis module is used to perform speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features;

[0010] The feature fusion module is used to extract action emotion features from the interactive action features and to extract expression emotion features from the interactive expression features, and to perform emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features to obtain interactive emotion semantics;

[0011] The feature mapping module is used to perform text reply analysis and speech synthesis on the interactive voice text according to the interactive emotional semantics to obtain emotional reply speech, and perform expression mapping and action mapping on the emotional reply speech respectively to obtain emotional reply expression and emotional reply action. wherein, when the feature mapping module performs text reply analysis and speech synthesis on the interactive voice text according to the interactive emotional semantics to obtain emotional reply speech, it specifically includes: using the interactive emotional semantics to perform emotion annotation on the interactive voice text to obtain interactive emotional text; performing text segmentation and text vectorization operations on the interactive emotional text to obtain emotional text features; performing position encoding and attention encoding on the emotional text features to obtain emotional text attention features; performing attention decoding on the emotional text attention features to obtain emotional reply text; performing speech spectrum conversion and emotion enhancement on the emotional reply text to obtain a reply voice spectrum; The reply speech spectrum is subjected to speech waveform synthesis and emotion consistency check to obtain an emotional reply speech, wherein the feature mapping module, when performing expression mapping and action mapping on the emotional reply speech respectively to obtain an emotional reply expression and an emotional reply action, specifically includes: extracting an emotional reply text from the emotional reply speech, and extracting an emotional reply semantics from the emotional reply text; performing facial expression mapping on the emotional reply semantics to obtain a primary reply expression; performing phoneme segmentation and lip shape mapping on the emotional reply speech to obtain a reply lip shape action; performing expression update on the primary reply expression using the reply lip shape action to obtain an emotional reply expression; performing emotion action mapping on the emotional reply semantics to obtain a primary reply action; performing text semantic analysis and semantic action mapping on the emotional reply text to obtain a semantic reply action; performing action update on the primary reply action using the semantic reply action to obtain an emotional reply action;

[0012] The video interaction module is used to perform digital human interaction animation rendering according to the emotional response voice, the emotional response expression and the emotional response action, obtain the digital human response video, and use the digital human response video to perform video interaction with the user.

[0013] Optionally, when the feature extraction module extracts the interactive action features and the interactive expression features from the personnel image sequence respectively, it specifically includes:

[0014] Performing posture estimation on the personnel image sequence to obtain a posture key point sequence;

[0015] Extracting skeletons from the person image sequence according to the posture key point sequence to obtain a person skeleton sequence;

[0016] Extracting joint features from the personnel skeleton sequence to obtain a personnel joint feature sequence;

[0017] Performing motion analysis on the joint feature sequence of the personnel to obtain interactive motion features;

[0018] Performing face tracking and face extraction on the person image sequence to obtain a face image sequence;

[0019] Positioning facial key points on the facial image sequence to obtain a facial key point sequence;

[0020] Extracting a facial texture feature sequence and a facial geometric feature sequence from the facial image sequence according to the facial key point sequence;

[0021] Generate interactive expression features according to the facial texture feature sequence and the facial geometric feature sequence.

[0022] Optionally, when the speech analysis module performs speech recognition and speech emotion analysis on the interactive speech feature to obtain the interactive speech text and speech emotion feature, the method specifically includes:

[0023] Performing power spectrum calculation on the interactive speech feature to obtain an interactive speech power spectrum;

[0024] Performing triangular filtering on the interactive speech power spectrum to obtain a speech phoneme spectrum;

[0025] The speech phoneme features are extracted from the speech phoneme spectrum using the following discrete transformation algorithm:

[0026]

[0027] Among them, c n refers to the nth feature in the speech phoneme feature, n is the feature index, k is the frequency component index, K is the total number of frequency components of the speech phoneme feature, log is the logarithmic function symbol, i is the frame index, M is the total number of sub-frames of the speech phoneme frequency, || is the absolute value symbol, X i (f k ) is the i-th frame of the speech phoneme spectrum at frequency f k The spectrum value at k It refers to the kth frequency component of the speech phoneme spectrum after Fourier transformation, cos is the symbol of the cosine function, and π is the symbol of pi;

[0028] Performing phoneme mapping on the speech phoneme features to obtain a speech phoneme sequence;

[0029] Performing text mapping and text optimization on the speech phoneme sequence to obtain interactive speech text;

[0030] A speech emotion feature is generated according to the interactive speech feature and the speech phoneme feature.

[0031] Optionally, when the speech analysis module generates the speech emotion feature according to the interactive speech feature and the speech phoneme feature, the method specifically includes:

[0032] Performing an inverse frequency domain transform on the interactive speech feature to obtain a windowed speech audio;

[0033] The emotional information entropy characteristics of the windowed speech audio are calculated using the following information entropy algorithm:

[0034]

[0035] Among them, D j It refers to the emotional information entropy feature of the speech signal of the jth channel in the windowed speech audio, ∞ is the infinity symbol, π is the pi symbol, σ j It refers to the standard deviation of the speech signal of the jth channel in the windowed speech audio, e is the Euler number, u j It refers to the signal value of the speech signal of the jth channel in the windowed speech audio, μ j refers to the mean value of the speech signal of the jth channel in the windowed speech audio, log is the logarithmic sign, and d is the integral sign;

[0036] Calculating the zero-crossing rate of the windowed speech audio to obtain a speech zero-crossing feature;

[0037] A speech emotion feature is generated according to the speech phoneme feature, the emotion information entropy feature and the speech zero-crossing feature.

[0038] Optionally, when the feature fusion module performs emotion feature fusion and emotion semantic analysis on the voice emotion feature, the action emotion feature and the expression emotion feature to obtain the interactive emotion semantics, the feature fusion module specifically includes:

[0039] The speech emotion feature, the action emotion feature and the expression emotion feature are respectively standardized and matrixed to obtain a speech feature matrix, an action feature matrix and an expression feature matrix;

[0040] Performing emotion correlation analysis on the speech feature matrix, the action feature matrix and the expression feature matrix to obtain a correlation vector group and a correlation coefficient;

[0041] According to the association degree vector group and the association coefficient, the speech feature matrix, the action feature matrix and the expression feature matrix are merged into a primary emotion feature matrix;

[0042] Performing a covariance operation on the primary emotion feature matrix to obtain an emotion covariance matrix;

[0043] Performing eigenvalue decomposition on the emotion covariance matrix to obtain emotion eigenvalues ​​and emotion eigenvectors;

[0044] Filtering a standard emotion feature vector from the emotion feature vector according to the emotion feature value;

[0045] Performing feature mapping on the primary emotion feature matrix using the standard emotion feature vector to obtain interactive emotion features;

[0046] Performing sentiment semantic analysis on the interactive sentiment features to obtain interactive sentiment semantics.

[0047] Optionally, when the feature fusion module performs emotional semantic analysis on the interactive emotional features to obtain interactive emotional semantics, the feature fusion module specifically includes:

[0048] Extracting attention emotion features from the interactive emotion features using a self-attention mechanism;

[0049] Decoding the attention emotion feature residual into an attention emotion vector;

[0050] Performing pooling and dimensionality reduction on the interactive emotion features to obtain pooled emotion features;

[0051] Performing support vector activation on the pooled sentiment features to obtain a vector machine sentiment feature;

[0052] Mapping the vector machine emotion feature activation into a vector machine emotion vector;

[0053] A standard emotion vector is generated according to the attention emotion vector and the vector machine emotion vector, and the standard emotion vector is mapped into interactive emotion semantics.

[0054] Optionally, when the video interaction module performs digital human interactive animation rendering according to the emotional response voice, the emotional response expression and the emotional response action to obtain the digital human response video, the video interaction module specifically includes:

[0055] Acquire a digital human model, and use the emotion response action to bind the digital human model to body movements to obtain digital human motion data;

[0056] Using the emotional response expression to bind the facial expression of the digital human model and adjust the lip shape to obtain digital human expression data;

[0057] Performing motion rendering on the digital human model according to the digital human motion data and the digital human expression data to obtain a primary response video;

[0058] The primary response video is synchronized with the audio by using the emotional response voice to obtain a digital human response video.

[0059] In order to solve the above problems, the present invention also provides a digital human interaction method based on big data information, the method comprising:

[0060] Collect user interaction data to obtain a user interaction video, and extract interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively. The extracting interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively, includes: performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequence; performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio; performing frequency domain transformation on the windowed voice audio to obtain interactive voice features; performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence; performing motion estimation on the compensated frame image sequence to obtain a user motion function; performing image dynamic denoising on the compensated frame image sequence according to the user motion function using the following dynamic denoising algorithm to obtain a denoised frame image sequence:

[0061]

[0062] in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, z is the translation vector between the pixel coordinate x and the pixel coordinate y, || is the absolute value symbol, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the user motion The mathematical symbol of the function, M(y+z,x+z) is the pixel coordinates corresponding to the motion compensation result of the pixel point with pixel coordinates (y+z) calculated by the user motion function relative to the pixel point with pixel coordinates (x+z), v(y+z) is the pixel value of the pixel point with pixel coordinates (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel point with pixel coordinates y in the compensated frame image sequence; perform person recognition and background separation operations on the denoised frame image sequence to obtain a person image sequence; extract interactive action features and interactive expression features from the person image sequence respectively;

[0063] Performing speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features;

[0064] Extracting action emotion features from the interactive action features and extracting expression emotion features from the interactive expression features respectively, performing emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features, and obtaining interactive emotion semantics;

[0065] According to the interactive emotional semantics, the interactive voice text is subjected to text reply analysis and speech synthesis to obtain emotional reply speech, and expression mapping and action mapping are respectively performed on the emotional reply speech to obtain emotional reply expression and emotional reply action, wherein, according to the interactive emotional semantics, the interactive voice text is subjected to text reply analysis and speech synthesis to obtain emotional reply speech, including: using the interactive emotional semantics to perform emotion annotation on the interactive voice text to obtain interactive emotional text; performing text segmentation and text vectorization operations on the interactive emotional text to obtain emotional text features; performing position encoding and attention encoding on the emotional text features to obtain emotional text attention features; performing attention decoding on the emotional text attention features to obtain emotional reply text; performing speech spectrum conversion and emotion enhancement on the emotional reply text to obtain a reply voice spectrum; The spectrum is used to perform speech waveform synthesis and emotion consistency check to obtain an emotion response speech, wherein the emotion response speech is respectively subjected to expression mapping and action mapping to obtain an emotion response expression and an emotion response action, including: extracting an emotion response text from the emotion response speech, and extracting an emotion response semantics from the emotion response text; performing facial expression mapping on the emotion response semantics to obtain a primary response expression; performing phoneme segmentation and lip shape mapping on the emotion response speech to obtain a response lip shape action; performing expression update on the primary response expression using the response lip shape action to obtain an emotion response expression; performing emotion action mapping on the emotion response semantics to obtain a primary response action; performing text semantic analysis and semantic action mapping on the emotion response text to obtain a semantic response action; performing action update on the primary response action using the semantic response action to obtain an emotion response action;

[0066] Digital human interaction animation rendering is performed according to the emotional response voice, the emotional response expression and the emotional response action to obtain a digital human response video, and the digital human response video is used to perform video interaction with the user.

[0067] The embodiments of the present invention can extract features of three dimensions, namely, speech, action and expression, for user interactive behaviors by extracting interactive speech features, interactive action features and interactive expression features, thereby analyzing user interactive information from multiple aspects and improving the accuracy of digital human interaction. By performing speech recognition and speech emotion analysis, it is possible to combine multi-dimensional speech features to realize text analysis and emotion analysis of speech, thereby improving the accuracy of subsequent digital human-generated responses. By performing emotion feature fusion, it is possible to perform feature fusion and feature dimensionality reduction according to the correlation and importance between different types of emotion features, thereby improving the accuracy of emotion semantic recognition and the efficiency of calculation. By combining the self-attention mechanism for emotion semantic analysis, it is possible to discover the detailed features between emotion features and improve the accuracy of emotion semantic recognition.

[0068] By performing text reply analysis and speech synthesis, it is possible to combine machine learning methods to synthesize reply speech containing emotions, thereby improving the authenticity of digital human interaction. By performing expression mapping and action mapping, it is possible to achieve body movements of the digital human in combination with the content when replying, and to achieve alignment of lip shape and speech, thereby improving the authenticity and accuracy of digital human interaction. By performing digital human interaction animation rendering and video interaction, it is possible to combine body movements, speech and facial expressions to simulate the interaction of the digital human, thereby improving the authenticity and accuracy of the digital human interaction. Therefore, the digital human interaction system and method based on big data information provided by the present invention can improve the accuracy of digital human interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A functional module diagram of a digital human interaction system based on big data information provided by an embodiment of the present invention;

[0070] Figure 2 A flowchart of performing speech recognition and speech emotion analysis provided by an embodiment of the present invention;

[0071] Figure 3 A flow chart of generating speech emotion features provided by an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of a flow chart of a digital human interaction method based on big data information provided by an embodiment of the present invention;

[0073] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0074] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0075] The embodiment of the present application provides a digital human interaction system based on big data information. The execution subject of the digital human interaction system based on big data information includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the system provided by the embodiment of the present application. In other words, the digital human interaction system based on big data information can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0076] like Figure 1 , which is a functional module diagram of a digital human interaction system based on big data information provided by an embodiment of the present invention.

[0077] The digital human interaction system 100 based on big data information of the present invention can be installed in an electronic device. According to the functions to be implemented, the digital human interaction system 100 based on big data information can include a feature extraction module 101, a speech analysis module 102, a feature fusion module 103, a feature mapping module 104 and a video interaction module 105. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0078] In this embodiment, the functions of each module / unit are as follows:

[0079] The feature extraction module 101 is used to collect user interaction data to obtain user interaction videos, and to extract interaction voice features, interaction action features, and interaction expression features from the user interaction videos.

[0080] In detail, the user interaction video refers to a video obtained by capturing the user's interactive behavior using a camera device, and the interactive data can be captured using devices such as a camera or a head-mounted camera.

[0081] In the embodiment of the present invention, when the feature extraction module 101 extracts the interactive voice feature, the interactive action feature and the interactive expression feature from the user interaction video respectively, the feature extraction module 101 specifically includes:

[0082] Performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequences;

[0083] Performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio;

[0084] Performing frequency domain transformation on the windowed speech audio to obtain interactive speech features;

[0085] Performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence;

[0086] Performing motion estimation on the compensated frame image sequence to obtain a user motion function;

[0087] Performing image dynamic denoising on the compensated frame image sequence according to the user motion function to obtain a denoised frame image sequence;

[0088] Performing personnel recognition and background separation operations on the denoised frame image sequence to obtain a personnel image sequence;

[0089] Interaction action features and interaction expression features are extracted from the personnel image sequence respectively.

[0090] Specifically, audio separation can be performed using multimedia processing tools such as FFmpeg or HandBrake, and the audio separation can separate the interactive voice audio from the user interaction video. Video framing can be performed using FFmpeg or OpenCV, and the interactive voice audio is the audio file of the user interaction video. Each interactive frame image in the interactive frame image sequence is a screenshot of a frame in the user interaction video.

[0091] In detail, audio framing can be performed using adaptive framing or overlapping framing methods, the audio windowing method can be a Hamming Window, a Hanning Window or a Blackman Window, the frequency domain transformation can be performed using a wavelet transform or a Fourier transform method, and the illumination compensation method can be a histogram equalization or a gamma correction method.

[0092] Specifically, the following dynamic denoising algorithm is used to perform dynamic denoising on the compensated frame image sequence to obtain a denoised frame image sequence:

[0093]

[0094] in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, || is the absolute value symbol, z is the translation vector between the pixel coordinate x and the pixel coordinate y, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the mathematical symbol of the user motion function, M(y+z,x+z) is the pixel coordinate corresponding to the motion compensation result of the pixel with pixel coordinate (y+z) calculated by the user motion function relative to the pixel with pixel coordinate (x+z), v(y+z) is the pixel value of the pixel with pixel coordinate (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel with pixel coordinate y in the compensated frame image sequence.

[0095] In detail, the optical flow method or the block matching method can be used for motion estimation. The dynamic denoising algorithm removes noise by weighted averaging based on neighborhood similarity. It can retain important edge and detail information while removing noise and remove motion noise. Edge detection operators such as Sobel or Canny or pre-trained deep convolutional neural networks can be used for person recognition and background separation.

[0096] Specifically, when the feature extraction module 101 extracts the interactive action features and the interactive expression features from the personnel image sequence respectively, it specifically includes:

[0097] Performing posture estimation on the personnel image sequence to obtain a posture key point sequence;

[0098] Extracting skeletons from the person image sequence according to the posture key point sequence to obtain a person skeleton sequence;

[0099] Extracting joint features from the personnel skeleton sequence to obtain a personnel joint feature sequence;

[0100] Performing motion analysis on the joint feature sequence of the personnel to obtain interactive motion features;

[0101] Performing face tracking and face extraction on the person image sequence to obtain a face image sequence;

[0102] Positioning facial key points on the facial image sequence to obtain a facial key point sequence;

[0103] Extracting a facial texture feature sequence and a facial geometric feature sequence from the facial image sequence according to the facial key point sequence;

[0104] Generate interactive expression features according to the facial texture feature sequence and the facial geometric feature sequence.

[0105] In detail, OpenPose, AlphaPose, HRNet and other algorithms can be used for posture estimation, and each posture key point in the posture key point sequence corresponds to a key point at a key position such as a shoulder, elbow or knee of a person image in the person image sequence, and the skeleton extraction refers to connecting each posture key point in the posture key point sequence in the person image sequence into a person skeleton.

[0106] Specifically, the joint feature extraction refers to calculating the joint angles and joint distances of each person skeleton in the person skeleton sequence, and the motion analysis method may be an optical flow method or a motion estimation method.

[0107] In detail, the trained convolutional neural network model can be used for face tracking and face extraction, and algorithms such as Dlib, OpenFace, and Facial Landmark Detection can be used to locate facial key points.

[0108] Specifically, a facial texture feature sequence can be extracted using a local binary method or a gradient directional histogram, and a facial geometric feature sequence can be calculated by calculating the distance between each facial key point in the facial key point sequence and the facial key point angle. Generating interactive expression features based on the facial texture feature sequence and the facial geometric feature sequence refers to splicing and fusing the facial texture feature sequence and the facial geometric feature sequence according to the sequence numbers.

[0109] In the embodiment of the present invention, by extracting interactive voice features, interactive action features and interactive expression features, it is possible to extract features in three dimensions of voice, action and expression for user interactive behavior, thereby analyzing the user's interactive information in multiple aspects and improving the accuracy of digital human interaction.

[0110] The speech analysis module 102 is used to perform speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features.

[0111] In detail, the interactive voice text is the text corresponding to the user dialogue in the user's interactive behavior, and the voice emotion feature is the emotion feature of the user at each moment when speaking corresponding to the interactive voice feature.

[0112] In the embodiment of the present invention, refer to Figure 2As shown, when the speech analysis module 102 performs speech recognition and speech emotion analysis on the interactive speech features to obtain the interactive speech text and speech emotion features, the following steps are specifically performed:

[0113] S21, performing power spectrum calculation on the interactive speech feature to obtain an interactive speech power spectrum;

[0114] S22, performing triangular filtering on the interactive speech power spectrum to obtain a speech phoneme spectrum;

[0115] S23, extracting speech phoneme features from the speech phoneme spectrum;

[0116] S24, performing text mapping and text optimization on the speech phoneme sequence to obtain an interactive speech text;

[0117] S25. Generate speech emotion features according to the interactive speech features and the speech phoneme features.

[0118] Specifically, the power spectrum may be calculated using Fourier transform or autocorrelation function, and the triangular filtering may be performed using a Mel filter.

[0119] In detail, the speech phoneme features are extracted using the following discrete transformation algorithm:

[0120]

[0121] Among them, c n refers to the nth feature in the speech phoneme feature, n is the feature index, k is the frequency component index, K is the total number of frequency components of the speech phoneme feature, log is the logarithmic function symbol, i is the frame index, M is the total number of sub-frames of the speech phoneme frequency, || is the absolute value symbol, X i (f k ) is the i-th frame of the speech phoneme spectrum at frequency f k The spectrum value at k It refers to the kth frequency component of the speech phoneme spectrum after Fourier transformation, cos is the symbol of the cosine function, and π is the symbol of pi.

[0122] Specifically, the discrete transformation algorithm can cover the audio scale from low frequency to high frequency and is close to the perception frequency of the human ear, thereby improving the accuracy of subsequent phoneme mapping. The trained Bayesian network or hidden Markov model can be used for phoneme mapping, and the N-gram model or RNN language model can be used for text mapping and text optimization.

[0123] For details, refer to Figure 3 As shown, when the speech analysis module 102 generates the speech emotion feature according to the interactive speech feature and the speech phoneme feature, it specifically includes:

[0124] S31, performing inverse frequency domain transformation on the interactive speech feature to obtain windowed speech audio;

[0125] S32, calculating the emotional information entropy characteristics of the windowed speech audio;

[0126] S33, calculating the zero-crossing rate of the windowed speech audio to obtain a speech zero-crossing feature;

[0127] S34, generating a speech emotion feature according to the speech phoneme feature, the emotion information entropy feature and the speech zero-crossing feature.

[0128] In detail, the emotional information entropy features are calculated using the following information entropy algorithm:

[0129]

[0130] Among them, D j It refers to the emotional information entropy feature of the speech signal of the jth channel in the windowed speech audio, ∞ is the infinity symbol, π is the symbol of pi, σ j It refers to the standard deviation of the speech signal of the jth channel in the windowed speech audio, e is the Euler number, u j It refers to the signal value of the speech signal of the jth channel in the windowed speech audio, μ j It refers to the mean value of the speech signal of the jth channel in the windowed speech audio, log is the logarithmic sign, and d is the integral sign.

[0131] Specifically, the inverse Fourier transform can be used to perform inverse frequency domain transform. The information entropy algorithm can improve the sensitivity of recognizing characteristics such as continuity and rate of change of speech signals, thereby improving the accuracy of speech emotion features. A fully connected network can be used to generate speech emotion features based on the speech phoneme features, the emotion information entropy features and the speech zero-crossing features.

[0132] In the embodiment of the present invention, by performing speech recognition and speech sentiment analysis, it is possible to combine multi-dimensional speech features to implement text analysis and sentiment analysis of speech, thereby improving the accuracy of subsequent digital human-generated responses.

[0133] The feature fusion module 103 is used to extract action emotion features from the interactive action features and to extract expression emotion features from the interactive expression features, respectively, and to perform emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features to obtain interactive emotion semantics.

[0134] In detail, the action emotion characteristics are the relevant emotion characteristics revealed by the body movements of people during interactive communication, the expression emotion characteristics are the relevant emotion characteristics revealed by the facial expressions of people during interactive communication, and the interactive emotion semantics are the emotion semantics of each time period during the interaction process of people.

[0135] Specifically, when the feature fusion module 103 extracts action emotion features from the interactive action features and extracts expression emotion features from the interactive expression features, it specifically includes: performing time convolution on the interactive action features to obtain action time features; performing primary emotion feature screening on the action time features to obtain action emotion features; performing time convolution on the interactive expression features to obtain expression time features, and performing primary emotion feature screening on the expression time features to obtain expression emotion features.

[0136] In detail, a long short-term memory network or a RNN network can be used for temporal convolution, and a linear discriminant analysis (LDA) method can be used for primary sentiment feature screening.

[0137] Specifically, when the feature fusion module 103 performs emotion feature fusion and emotion semantic analysis on the voice emotion feature, the action emotion feature, and the expression emotion feature to obtain the interactive emotion semantics, it specifically includes:

[0138] The speech emotion feature, the action emotion feature and the expression emotion feature are respectively standardized and matrixed to obtain a speech feature matrix, an action feature matrix and an expression feature matrix;

[0139] Performing emotion correlation analysis on the speech feature matrix, the action feature matrix and the expression feature matrix to obtain a correlation vector group and a correlation coefficient;

[0140] According to the association degree vector group and the association coefficient, the speech feature matrix, the action feature matrix and the expression feature matrix are merged into a primary emotion feature matrix;

[0141] Performing a covariance operation on the primary emotion feature matrix to obtain an emotion covariance matrix;

[0142] Performing eigenvalue decomposition on the emotion covariance matrix to obtain emotion eigenvalues ​​and emotion eigenvectors;

[0143] Filtering a standard emotion feature vector from the emotion feature vector according to the emotion feature value;

[0144] Performing feature mapping on the primary emotion feature matrix using the standard emotion feature vector to obtain interactive emotion features;

[0145] Performing sentiment semantic analysis on the interactive sentiment features to obtain interactive sentiment semantics.

[0146] In detail, Linear Discriminant Analysis (LDA) can be used to perform sentiment correlation analysis and fuse a primary sentiment feature matrix. The method of selecting a standard sentiment feature vector from the sentiment feature vector based on the sentiment feature value refers to selecting a number of feature vectors with larger corresponding sentiment feature values ​​from the sentiment feature vector to form a standard sentiment feature vector.

[0147] Specifically, when the feature fusion module 103 performs emotional semantic analysis on the interactive emotional features to obtain interactive emotional semantics, the following steps are specifically performed:

[0148] Extracting attention emotion features from the interactive emotion features using a self-attention mechanism;

[0149] Decoding the attention emotion feature residual into an attention emotion vector;

[0150] Performing pooling and dimensionality reduction on the interactive emotion features to obtain pooled emotion features;

[0151] Performing support vector activation on the pooled sentiment features to obtain a vector machine sentiment feature;

[0152] Mapping the vector machine emotion feature activation into a vector machine emotion vector;

[0153] A standard emotion vector is generated according to the attention emotion vector and the vector machine emotion vector, and the standard emotion vector is mapped into interactive emotion semantics.

[0154] In detail, the Transformer model or the BERT model can be used to extract attention emotion features and perform residual decoding, the support vector machine model can be used to perform support vector activation, the fully connected method can be used to generate a standard emotion vector based on the attention emotion vector and the vector machine emotion vector, and the preset emotion semantic library can be used to map the standard emotion vector into interactive emotion semantics.

[0155] In the embodiment of the present invention, by performing emotional feature fusion, feature fusion and feature dimensionality reduction can be performed according to the correlation and importance between different types of emotional features, thereby improving the accuracy of emotional semantic recognition and the efficiency of calculation. By combining the self-attention mechanism for emotional semantic analysis, the detailed features between the emotional features can be discovered, thereby improving the accuracy of emotional semantic recognition.

[0156] The feature mapping module 104 is used to perform text response analysis and speech synthesis on the interactive voice text according to the interactive emotional semantics to obtain emotional response speech, and to perform expression mapping and action mapping on the emotional response speech respectively to obtain emotional response expression and emotional response action.

[0157] In detail, the emotional response voice is the voice uttered by the digital human in response to human interaction, the emotional response expression refers to the corresponding expression of the digital human when playing the emotional response voice, and the emotional response action is the body movement of the digital human when playing the emotional response voice.

[0158] In the embodiment of the present invention, when the feature mapping module 104 performs text reply analysis and speech synthesis on the interactive speech text according to the interactive emotional semantics to obtain the emotional reply speech, the following steps are specifically performed:

[0159] Using the interactive emotional semantics to emotionally annotate the interactive speech text to obtain an interactive emotional text;

[0160] Performing text segmentation and text vectorization operations on the interactive emotional text to obtain emotional text features;

[0161] Performing position encoding and attention encoding on the emotional text feature to obtain an emotional text attention feature;

[0162] Performing attention decoding on the attention features of the emotional text to obtain an emotional reply text;

[0163] Performing voice spectrum conversion and emotion enhancement on the emotional response text to obtain a response voice spectrum;

[0164] The response speech spectrum is subjected to speech waveform synthesis and emotion consistency check to obtain the emotion response speech.

[0165] In detail, the emotion annotation refers to annotating corresponding emotion semantics for each element of the interactive semantic text according to the temporal correspondence between the interactive emotion semantics and the interactive voice text. The emotion semantics may be "happy", "sad" and "calm" etc.

[0166] Specifically, the text segmentation method can be the jieba segmentation method, the Word2Vec or BERT model can be used for text vectorization, and the pre-trained Transformer model or BERT model whose input is the interactive text labeled with emotional semantics and whose output is the reply text labeled with emotional semantics can be used for attention encoding and the attention decoding.

[0167] In detail, models such as Tacotron 2, FastSpeech, and Glow-TTS may be used to perform speech spectrum conversion, and the emotion enhancement refers to adjusting the pitch, speed, volume, and other features of the response speech spectrum according to the emotional semantics corresponding to each word in the emotional response text.

[0168] Specifically, models such as WaveNet, WaveGlow, and HiFi-GAN can be used to synthesize speech waveforms, and sentiment classifiers can be used to check sentiment consistency.

[0169] In detail, when the feature mapping module 104 performs expression mapping and action mapping on the emotion reply speech respectively to obtain the emotion reply expression and the emotion reply action, it specifically includes:

[0170] Extracting emotional response text from the emotional response speech, and extracting emotional response semantics from the emotional response text;

[0171] Performing facial expression mapping on the emotional response semantics to obtain a primary response expression;

[0172] Performing phoneme segmentation and lip shape mapping on the emotional response speech to obtain a response lip shape action;

[0173] Using the reply lip movement to update the primary reply expression to obtain an emotional reply expression;

[0174] Performing emotion-action mapping on the emotion reply semantics to obtain a primary reply action;

[0175] Performing text semantic analysis and semantic action mapping on the emotional response text to obtain a semantic response action;

[0176] The semantic reply action is used to update the primary reply action to obtain an emotional reply action.

[0177] In detail, the emotional response semantics are the emotional semantics annotated on each word in the emotional response text. Facial expression mapping can be performed based on action units (AUs for short), phoneme segmentation can be performed using a threshold-based segmentation method or a model-based segmentation method, and lip shape mapping can be performed through a lip shape dictionary.

[0178] Specifically, emotion-action mapping can be performed using a pre-recorded emotion-action database, in which each emotion corresponds to an action. The text semantic analysis method can be a keyword analysis method or a semantic analysis method based on deep learning. Emotion-action mapping can be performed using a pre-recorded semantic-action database, in which each semantic keyword corresponds to an action semantic-action mapping.

[0179] In the embodiment of the present invention, by performing text reply analysis and speech synthesis, a reply speech containing emotion can be synthesized in combination with a machine learning method, thereby improving the authenticity of the digital human interaction. By performing expression mapping and action mapping, the digital human can perform body movements in combination with the content when replying, and the alignment of lip shape and speech can be achieved, thereby improving the authenticity and accuracy of the digital human interaction.

[0180] The video interaction module 105 is used to perform digital human interaction animation rendering according to the emotional response voice, the emotional response expression and the emotional response action, obtain a digital human response video, and use the digital human response video to perform video interaction with the user.

[0181] In detail, the digital human reply video refers to a reply video that needs to be played to the user. When interacting with the digital human, the user can have a conversation with the digital human on the screen, and then the system plays the reply video for the conversation.

[0182] In the embodiment of the present invention, when the video interaction module 105 performs digital human interactive animation rendering according to the emotional response voice, the emotional response expression and the emotional response action to obtain the digital human response video, it specifically includes:

[0183] Acquire a digital human model, and use the emotion response action to bind the digital human model to body movements to obtain digital human motion data;

[0184] Using the emotional response expression to bind the facial expression of the digital human model and adjust the lip shape to obtain digital human expression data;

[0185] Performing motion rendering on the digital human model according to the digital human motion data and the digital human expression data to obtain a primary response video;

[0186] The primary response video is synchronized with the audio by using the emotional response voice to obtain a digital human response video.

[0187] In detail, Maya, Blender, 3ds Max and other tools can be used for body movement binding, and Blendshape can be used for facial expression binding and lip shape adjustment. The body movement binding refers to binding the joints, bones and emotional response actions of the digital human model, and the facial expression binding refers to binding the facial vertices of the digital human model and the muscles and bones of the emotional response expressions.

[0188] Specifically, motion rendering may be performed using renderers such as V-Ray, Arnold, and Cycles, and performing video interaction with the user using the digital human reply video refers to playing the digital human reply video for the user.

[0189] In the embodiment of the present invention, by performing digital human interactive animation rendering and video interaction, the digital human can be interactively simulated in combination with body movements, voice and facial expressions, thereby improving the realism and accuracy of the digital human interaction.

[0190] The embodiments of the present invention can extract features of three dimensions, namely, speech, action and expression, for user interactive behaviors by extracting interactive speech features, interactive action features and interactive expression features, thereby analyzing user interactive information from multiple aspects and improving the accuracy of digital human interaction. By performing speech recognition and speech emotion analysis, it is possible to combine multi-dimensional speech features to realize text analysis and emotion analysis of speech, thereby improving the accuracy of subsequent digital human-generated responses. By performing emotion feature fusion, it is possible to perform feature fusion and feature dimensionality reduction according to the correlation and importance between different types of emotion features, thereby improving the accuracy of emotion semantic recognition and the efficiency of calculation. By combining the self-attention mechanism for emotion semantic analysis, it is possible to discover the detailed features between emotion features and improve the accuracy of emotion semantic recognition.

[0191] By performing text reply analysis and speech synthesis, it is possible to combine machine learning methods to synthesize reply speech containing emotions, thereby improving the authenticity of digital human interaction. By performing expression mapping and action mapping, it is possible to achieve body movements of the digital human in combination with the content when replying, and to achieve alignment of lip shape and speech, thereby improving the authenticity and accuracy of digital human interaction. By performing digital human interaction animation rendering and video interaction, it is possible to combine body movements, speech and facial expressions to simulate digital human interaction, thereby improving the authenticity and accuracy of digital human interaction. Therefore, the digital human interaction system based on big data information provided by the present invention can improve the accuracy of digital human interaction.

[0192] Reference Figure 4 FIG. 1 is a flow chart of a digital human interaction method based on big data information provided by an embodiment of the present invention. In this embodiment, the digital human interaction method based on big data information includes:

[0193] S1. Collecting user interaction data to obtain a user interaction video, and extracting interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively. The extracting interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively, includes: performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequence; performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio; performing frequency domain transformation on the windowed voice audio to obtain interactive voice features; performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence; performing motion estimation on the compensated frame image sequence to obtain a user motion function; performing image dynamic denoising on the compensated frame image sequence according to the user motion function using the following dynamic denoising algorithm to obtain a denoised frame image sequence:

[0194]

[0195] in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, || is the absolute value symbol, z is the translation vector between the pixel coordinate x and the pixel coordinate y, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the user motion The mathematical symbol of the function, M(y+z,x+z) is the pixel coordinates corresponding to the motion compensation result of the pixel point with pixel coordinates (y+z) calculated by the user motion function relative to the pixel point with pixel coordinates (x+z), v(y+z) is the pixel value of the pixel point with pixel coordinates (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel point with pixel coordinates y in the compensated frame image sequence; perform person recognition and background separation operations on the denoised frame image sequence to obtain a person image sequence; extract interactive action features and interactive expression features from the person image sequence respectively;

[0196] S2, performing speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features;

[0197] S3, respectively extracting action emotion features from the interactive action features and extracting expression emotion features from the interactive expression features, performing emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features, to obtain interactive emotion semantics;

[0198] S4, performing text reply analysis and speech synthesis on the interactive voice text according to the interactive emotional semantics to obtain emotional reply speech, and performing expression mapping and action mapping on the emotional reply speech respectively to obtain emotional reply expression and emotional reply action;

[0199] S5. Perform digital human interactive animation rendering according to the emotional response voice, the emotional response expression and the emotional response action to obtain a digital human response video, and use the digital human response video to perform video interaction with the user.

[0200] In detail, the digital human interaction method based on big data information in the embodiment of the present invention is used in the same manner as above. Figure 1 The same technical means are used as the digital human interaction system based on big data information described in the text, and can produce the same technical effects, so I will not go into details here.

[0201] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0202] Collect user interaction data to obtain a user interaction video, and extract interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively. The extracting interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively, includes: performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequence; performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio; performing frequency domain transformation on the windowed voice audio to obtain interactive voice features; performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence; performing motion estimation on the compensated frame image sequence to obtain a user motion function; performing image dynamic denoising on the compensated frame image sequence according to the user motion function using the following dynamic denoising algorithm to obtain a denoised frame image sequence:

[0203]

[0204] in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, || is the absolute value symbol, z is the translation vector between the pixel coordinate x and the pixel coordinate y, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the user motion The mathematical symbol of the function, M(y+z,x+z) is the pixel coordinates corresponding to the motion compensation result of the pixel point with pixel coordinates (y+z) calculated by the user motion function relative to the pixel point with pixel coordinates (x+z), v(y+z) is the pixel value of the pixel point with pixel coordinates (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel point with pixel coordinates y in the compensated frame image sequence; perform person recognition and background separation operations on the denoised frame image sequence to obtain a person image sequence; extract interactive action features and interactive expression features from the person image sequence respectively;

[0205] Performing speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features;

[0206] Extracting action emotion features from the interactive action features and extracting expression emotion features from the interactive expression features respectively, performing emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features, and obtaining interactive emotion semantics;

[0207] Perform text response analysis and speech synthesis on the interactive speech text according to the interactive emotional semantics to obtain emotional response speech, and respectively perform expression mapping and action mapping on the emotional response speech to obtain emotional response expression and emotional response action;

[0208] Digital human interaction animation rendering is performed according to the emotional response voice, the emotional response expression and the emotional response action to obtain a digital human response video, and the digital human response video is used to perform video interaction with the user.

[0209] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and systems can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0210] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0212] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0213] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0214] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, system, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0215] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or systems stated in the system embodiment can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A digital human interaction system based on big data information, characterized in that: The system includes a feature extraction module, a speech analysis module, a feature fusion module, a feature mapping module and a video interaction module, wherein: The feature extraction module is used to collect user interaction data to obtain a user interaction video, and extract interactive voice features, interactive action features and interactive expression features from the user interaction video, respectively. When the feature extraction module extracts the interactive voice features, interactive action features and interactive expression features from the user interaction video, respectively, it specifically includes: performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequence; performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio; performing frequency domain transformation on the windowed voice audio to obtain interactive voice features; performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence; performing motion estimation on the compensated frame image sequence to obtain a user motion function; performing image dynamic denoising on the compensated frame image sequence according to the user motion function using the following dynamic denoising algorithm to obtain a denoised frame image sequence: in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, z is the translation vector between the pixel coordinate x and the pixel coordinate y, || is the absolute value symbol, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the user motion The mathematical symbol of the function, M(y+z,x+z) is the pixel coordinates corresponding to the motion compensation result of the pixel point with pixel coordinates (y+z) calculated by the user motion function relative to the pixel point with pixel coordinates (x+z), v(y+z) is the pixel value of the pixel point with pixel coordinates (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel point with pixel coordinates y in the compensated frame image sequence; perform person recognition and background separation operations on the denoised frame image sequence to obtain a person image sequence; extract interactive action features and interactive expression features from the person image sequence respectively; The speech analysis module is used to perform speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features; The feature fusion module is used to extract action emotion features from the interactive action features and to extract expression emotion features from the interactive expression features, and to perform emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features to obtain interactive emotion semantics; The feature mapping module is used to perform text reply analysis and speech synthesis on the interactive voice text according to the interactive emotional semantics to obtain emotional reply speech, and perform expression mapping and action mapping on the emotional reply speech respectively to obtain emotional reply expression and emotional reply action. wherein, when the feature mapping module performs text reply analysis and speech synthesis on the interactive voice text according to the interactive emotional semantics to obtain emotional reply speech, it specifically includes: using the interactive emotional semantics to perform emotion annotation on the interactive voice text to obtain interactive emotional text; performing text segmentation and text vectorization operations on the interactive emotional text to obtain emotional text features; performing position encoding and attention encoding on the emotional text features to obtain emotional text attention features; performing attention decoding on the emotional text attention features to obtain emotional reply text; performing speech spectrum conversion and emotion enhancement on the emotional reply text to obtain a reply voice spectrum; The reply speech spectrum is subjected to speech waveform synthesis and emotion consistency check to obtain an emotional reply speech, wherein the feature mapping module, when performing expression mapping and action mapping on the emotional reply speech respectively to obtain an emotional reply expression and an emotional reply action, specifically includes: extracting an emotional reply text from the emotional reply speech, and extracting an emotional reply semantics from the emotional reply text; performing facial expression mapping on the emotional reply semantics to obtain a primary reply expression; performing phoneme segmentation and lip shape mapping on the emotional reply speech to obtain a reply lip shape action; performing expression update on the primary reply expression using the reply lip shape action to obtain an emotional reply expression; performing emotion action mapping on the emotional reply semantics to obtain a primary reply action; performing text semantic analysis and semantic action mapping on the emotional reply text to obtain a semantic reply action; performing action update on the primary reply action using the semantic reply action to obtain an emotional reply action; The video interaction module is used to perform digital human interaction animation rendering according to the emotional response voice, the emotional response expression and the emotional response action, obtain the digital human response video, and use the digital human response video to perform video interaction with the user.

2. The digital human interaction system based on big data information as claimed in claim 1, characterized in that: When the feature extraction module extracts the interactive action features and the interactive expression features from the personnel image sequence respectively, it specifically includes: Performing posture estimation on the personnel image sequence to obtain a posture key point sequence; Extracting skeletons from the person image sequence according to the posture key point sequence to obtain a person skeleton sequence; Extracting joint features from the personnel skeleton sequence to obtain a personnel joint feature sequence; Performing motion analysis on the joint feature sequence of the personnel to obtain interactive motion features; Performing face tracking and face extraction on the person image sequence to obtain a face image sequence; Positioning facial key points on the facial image sequence to obtain a facial key point sequence; Extracting a facial texture feature sequence and a facial geometric feature sequence from the facial image sequence according to the facial key point sequence; Generate interactive expression features according to the facial texture feature sequence and the facial geometric feature sequence.

3. The digital human interaction system based on big data information as claimed in claim 1, characterized in that: When the speech analysis module performs speech recognition and speech emotion analysis on the interactive speech features to obtain the interactive speech text and speech emotion features, the speech analysis module specifically includes: Performing power spectrum calculation on the interactive speech feature to obtain an interactive speech power spectrum; Performing triangular filtering on the interactive speech power spectrum to obtain a speech phoneme spectrum; The speech phoneme features are extracted from the speech phoneme spectrum using the following discrete transformation algorithm: Among them, c n refers to the nth feature in the speech phoneme feature, n is the feature index, k is the frequency component index, K is the total number of frequency components of the speech phoneme feature, log is the logarithmic function symbol, i is the frame index, M is the total number of sub-frames of the speech phoneme frequency, || is the absolute value symbol, X i (f k ) is the i-th frame of the speech phoneme spectrum at frequency f k The spectrum value at k It refers to the kth frequency component of the speech phoneme spectrum after Fourier transformation, cos is the symbol of the cosine function, and π is the symbol of pi; Performing phoneme mapping on the speech phoneme features to obtain a speech phoneme sequence; Performing text mapping and text optimization on the speech phoneme sequence to obtain interactive speech text; A speech emotion feature is generated according to the interactive speech feature and the speech phoneme feature.

4. The digital human interaction system based on big data information as claimed in claim 3, characterized in that: When the speech analysis module generates the speech emotion feature according to the interactive speech feature and the speech phoneme feature, the speech analysis module specifically includes: Performing an inverse frequency domain transform on the interactive speech feature to obtain a windowed speech audio; The emotional information entropy characteristics of the windowed speech audio are calculated using the following information entropy algorithm: Among them, D j It refers to the emotional information entropy feature of the speech signal of the jth channel in the windowed speech audio, ∞ is the infinity symbol, π is the pi symbol, σ j It refers to the standard deviation of the speech signal of the jth channel in the windowed speech audio, e is the Euler number, u j It refers to the signal value of the speech signal of the jth channel in the windowed speech audio, μ j refers to the mean value of the speech signal of the jth channel in the windowed speech audio, log is the logarithmic sign, and d is the integral sign; Calculating the zero-crossing rate of the windowed speech audio to obtain a speech zero-crossing feature; A speech emotion feature is generated according to the speech phoneme feature, the emotion information entropy feature and the speech zero-crossing feature.

5. The digital human interaction system based on big data information as claimed in claim 1, characterized in that: When the feature fusion module performs emotion feature fusion and emotion semantic analysis on the voice emotion feature, the action emotion feature and the expression emotion feature to obtain the interactive emotion semantics, it specifically includes: The speech emotion feature, the action emotion feature and the expression emotion feature are respectively standardized and matrixed to obtain a speech feature matrix, an action feature matrix and an expression feature matrix; Performing emotion correlation analysis on the speech feature matrix, the action feature matrix and the expression feature matrix to obtain a correlation vector group and a correlation coefficient; According to the association degree vector group and the association coefficient, the speech feature matrix, the action feature matrix and the expression feature matrix are merged into a primary emotion feature matrix; Performing a covariance operation on the primary emotion feature matrix to obtain an emotion covariance matrix; Performing eigenvalue decomposition on the emotion covariance matrix to obtain emotion eigenvalues ​​and emotion eigenvectors; Filtering a standard emotion feature vector from the emotion feature vector according to the emotion feature value; Performing feature mapping on the primary emotion feature matrix using the standard emotion feature vector to obtain interactive emotion features; Performing sentiment semantic analysis on the interactive sentiment features to obtain interactive sentiment semantics.

6. The digital human interaction system based on big data information as claimed in claim 5, characterized in that: When the feature fusion module performs emotional semantic analysis on the interactive emotional features to obtain interactive emotional semantics, the feature fusion module specifically includes: Extracting attention emotion features from the interactive emotion features using a self-attention mechanism; Decoding the attention emotion feature residual into an attention emotion vector; Performing pooling and dimensionality reduction on the interactive emotion features to obtain pooled emotion features; Performing support vector activation on the pooled sentiment features to obtain a vector machine sentiment feature; Mapping the vector machine emotion feature activation into a vector machine emotion vector; A standard emotion vector is generated according to the attention emotion vector and the vector machine emotion vector, and the standard emotion vector is mapped into interactive emotion semantics.

7. The digital human interaction system based on big data information as claimed in claim 1, characterized in that: When the video interaction module performs digital human interactive animation rendering according to the emotional response voice, the emotional response expression and the emotional response action to obtain the digital human response video, specifically includes: Acquire a digital human model, and use the emotion response action to bind the digital human model to body movements to obtain digital human motion data; Using the emotional response expression to bind the facial expression of the digital human model and adjust the lip shape to obtain digital human expression data; Performing motion rendering on the digital human model according to the digital human motion data and the digital human expression data to obtain a primary response video; The primary response video is synchronized with the audio by using the emotional response voice to obtain a digital human response video.

8. A digital human interaction method based on big data information, characterized in that: The method comprises: Collect user interaction data to obtain a user interaction video, and extract interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively. The extracting interactive voice features, interactive action features, and interactive expression features from the user interaction video, respectively, includes: performing audio separation and video framing on the user interaction video to obtain interactive voice audio and interactive frame image sequence; performing audio framing and audio windowing operations on the interactive voice audio to obtain windowed voice audio; performing frequency domain transformation on the windowed voice audio to obtain interactive voice features; performing illumination compensation on the interactive frame image sequence to obtain a compensated frame image sequence; performing motion estimation on the compensated frame image sequence to obtain a user motion function; performing image dynamic denoising on the compensated frame image sequence according to the user motion function using the following dynamic denoising algorithm to obtain a denoised frame image sequence: in, is the pixel value of the pixel with pixel coordinate x in the denoised frame image sequence, y is the pixel with pixel coordinate y in the compensated frame image sequence, I refers to the set of all pixel points in the compensated frame image sequence, exp is the exponential function symbol, h is the preset Gaussian smoothing parameter, d is the preset neighborhood radius, z is the translation vector between the pixel coordinate x and the pixel coordinate y, || is the absolute value symbol, v() is the pixel value symbol of the pixel in the compensated frame image sequence, v(x+z) is the pixel value of the pixel with pixel coordinate (x+z) in the compensated frame image sequence, M() is the user motion The mathematical symbol of the function, M(y+z,x+z) is the pixel coordinates corresponding to the motion compensation result of the pixel point with pixel coordinates (y+z) calculated by the user motion function relative to the pixel point with pixel coordinates (x+z), v(y+z) is the pixel value of the pixel point with pixel coordinates (y+z) in the compensated frame image sequence, and v(y) is the pixel value of the pixel point with pixel coordinates y in the compensated frame image sequence; perform person recognition and background separation operations on the denoised frame image sequence to obtain a person image sequence; extract interactive action features and interactive expression features from the person image sequence respectively; Performing speech recognition and speech emotion analysis on the interactive speech features to obtain interactive speech text and speech emotion features; Extracting action emotion features from the interactive action features and extracting expression emotion features from the interactive expression features respectively, performing emotion feature fusion and emotion semantic analysis on the voice emotion features, the action emotion features and the expression emotion features, and obtaining interactive emotion semantics; According to the interactive emotional semantics, the interactive voice text is subjected to text reply analysis and speech synthesis to obtain emotional reply speech, and expression mapping and action mapping are respectively performed on the emotional reply speech to obtain emotional reply expression and emotional reply action, wherein, according to the interactive emotional semantics, the interactive voice text is subjected to text reply analysis and speech synthesis to obtain emotional reply speech, including: using the interactive emotional semantics to perform emotion annotation on the interactive voice text to obtain interactive emotional text; performing text segmentation and text vectorization operations on the interactive emotional text to obtain emotional text features; performing position encoding and attention encoding on the emotional text features to obtain emotional text attention features; performing attention decoding on the emotional text attention features to obtain emotional reply text; performing speech spectrum conversion and emotion enhancement on the emotional reply text to obtain a reply voice spectrum; The spectrum is used to perform speech waveform synthesis and emotion consistency check to obtain an emotion response speech, wherein the emotion response speech is respectively subjected to expression mapping and action mapping to obtain an emotion response expression and an emotion response action, including: extracting an emotion response text from the emotion response speech, and extracting an emotion response semantics from the emotion response text; performing facial expression mapping on the emotion response semantics to obtain a primary response expression; performing phoneme segmentation and lip shape mapping on the emotion response speech to obtain a response lip shape action; performing expression update on the primary response expression using the response lip shape action to obtain an emotion response expression; performing emotion action mapping on the emotion response semantics to obtain a primary response action; performing text semantic analysis and semantic action mapping on the emotion response text to obtain a semantic response action; performing action update on the primary response action using the semantic response action to obtain an emotion response action; Digital human interaction animation rendering is performed according to the emotional response voice, the emotional response expression and the emotional response action to obtain a digital human response video, and the digital human response video is used to perform video interaction with the user.

Citation Information

Patent Citations

  • Video generation and interaction method and device based on digital human, storage medium and program product

    CN117710543A

  • Multi-modal perception fusion emotion recognition method and robot emotion interaction method

    CN117994622A

  • Virtual digital human interaction method, apparatus and device, and storage medium

    CN118312040A

  • Virtual digital human system and method for learning disorder group

    CN118519524A

  • One-click photorealistic video generation using ai and real-time cgi

    WO2024112994A1