Real-time analysis method and system for digital human interactive video service based on deep learning

Abstract

The present invention discloses a real-time analysis method and system for digital human interactive video services based on deep learning, which relates to the field of human-computer interaction; including: a data acquisition module, which is responsible for collecting interactive video data between users and digital humans, including multimodal data of users' voices, gestures and facial expressions; a preprocessing module, which preprocesses the collected video data, including noise removal, enhancement, feature extraction, voice emotion analysis for voice data, key frame extraction and feature point annotation for gesture and facial expression data; a deep learning analysis module, which establishes a deep learning model and uses the deep learning model to identify the user's multimodal data. The present invention collects multimodal data of users' voices, gestures, and facial expressions, performs multimodal fusion after feature processing, and then recognizes the intention after weighted processing, so as to directly identify the user's intention and solve the limitation problem.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a real-time analysis method and system for digital human interaction video services based on deep learning. Background Art

[0002] With the rapid development of virtual reality (VR), augmented reality (AR) and holographic communication technology, digital humans, as an important interactive medium, have been widely used in film and television, games, education, medical care and other fields.

[0003] After searching, the Chinese patent publication number CN116248812B discloses a business processing method, storage medium and system based on digital human interactive video. The method first generates multiple digital human videos with fixed speech and digital human videos with flexible speech, and then merges the digital human videos with flexible speech and the digital human videos with fixed speech corresponding to the business processing flow to form a digital human interactive video and displays it to the user. Then, according to the user's response information, other corresponding digital human videos with fixed speech are selected, merged with the digital human videos with flexible speech to form a new digital human interactive video and display it to the user. The method is repeated until the preset conditions for business processing are met and the business processing is carried out.

[0004] The above patent has the following shortcomings: it requires the user to perform resilient operations in order to utilize the responses given by the user, and is unable to analyze the user's intentions based on the multi-modal data used, so it still has certain limitations.

[0005] To this end, the present invention proposes a real-time analysis method and system for digital human interactive video services based on deep learning Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a real-time analysis method and system for digital human interactive video services based on deep learning.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0009] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0010] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0011] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0012] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0013] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0014] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0015] Preferably: the deep learning analysis module includes:

[0016] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0017] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0018] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0019] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0020] Preferably, in the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0021] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0022] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0023] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0024] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0025] Preferably, in the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0026] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0027] B2: Set the activation function to generate the hidden state of the current time step;

[0028] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0029] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0030] Preferably: in step A2 and step B2, the activation function is f(x)=max(0,x), which represents the maximum value between 0 and x, f(x)=max(ax,x) where a∈(0.01), Where λ = 1.0507, f(x) = x·tanh(ln(1+e x )).

[0031] Preferably, the method for fusing multimodal data at the fusion layer comprises the following steps:

[0032] C1: Feature concatenation, which directly concatenates features from different sources and levels in the channel dimension to form a wider or deeper feature map;

[0033] C2: Feature weighting: set a weight for each feature according to its importance and fuse the features by weighted average;

[0034] C3: Feature pooling, which performs pooling operations on the weighted feature maps to reduce the dimension of the feature maps and reduce the amount of calculation.

[0035] Preferably, in the step C2, the weighted average is in the form of double weighting, which specifically includes the following steps:

[0036] C21: One-level weight setting, setting weights for voice, gesture and facial expression, which are K1, K2 and K3 respectively;

[0037] C22: Double weight setting, for each extracted voice, gesture and facial expression feature type, the weight is set again, which is voice gesture Facial expressions

[0038] C23: Use double weight calculation method for weighted calculation.

[0039] Preferably: in the C21 step, K1∈(0.7-0.9), K2∈(0.05-0.15), and K3∈(0.05-0.15).

[0040] Preferably: in the step C23, the weighted calculation formula is: where Y i is the feature value of speech feature type i, S o is the feature value of gesture feature type o, M u is the feature value of the facial expression feature type u.

[0041] The real-time analysis method of digital human interaction video service based on deep learning includes the following steps:

[0042] S1: Data collection, using the data collection module to collect multimodal data of the user's voice, gestures and facial expressions;

[0043] S2: Data preprocessing, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0044] S3: Deep learning analysis, building a deep learning model, using the deep learning model to identify the user's multimodal data, and fusing information from different modalities through multimodal fusion technology to more comprehensively understand the user's intentions;

[0045] S4: Interaction action generation: according to the user's intention, matching interaction actions are retrieved from the database and fed back to the front end;

[0046] S5: Real-time feedback module, receives feedback from the interactive action generation module and displays feedback actions to the user.

[0047] The beneficial effects of the present invention are:

[0048] The present invention collects multimodal data of the user's voice, gestures, and facial expressions, performs multimodal fusion after feature processing, and then identifies the intention after weighted processing, thereby directly identifying the user's intention and solving the limitation problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the real-time analysis method of digital human interactive video services based on deep learning proposed by the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods.

[0051] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected or set, or detachably connected or set, or integrally connected or set. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] Embodiment 1:

[0053] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0054] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0055] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0056] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0057] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0058] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0059] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0060] The deep learning analysis module includes:

[0061] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0062] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0063] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0064] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0065] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0066] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0067] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0068] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0069] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0070] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0071] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0072] B2: Set the activation function to generate the hidden state of the current time step;

[0073] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0074] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0075] In the A2 step and the B2 step, the activation function is

[0076] Embodiment 2:

[0077] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0078] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0079] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0080] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0081] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0082] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0083] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0084] The deep learning analysis module includes:

[0085] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0086] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0087] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0088] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0089] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0090] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0091] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0092] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0093] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0094] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0095] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0096] B2: Set the activation function to generate the hidden state of the current time step;

[0097] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0098] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0099] In the A2 step and the B2 step, the activation function is

[0100] Embodiment 3:

[0101] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0102] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0103] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0104] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0105] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0106] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0107] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0108] The deep learning analysis module includes:

[0109] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0110] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0111] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0112] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0113] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0114] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0115] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0116] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0117] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0118] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0119] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0120] B2: Set the activation function to generate the hidden state of the current time step;

[0121] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0122] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0123] In step A2 and step B2, the activation function is f(x)=max(0,x).

[0124] Embodiment 4:

[0125] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0126] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0127] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0128] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0129] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0130] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0131] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0132] The deep learning analysis module includes:

[0133] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0134] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0135] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0136] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0137] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0138] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0139] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0140] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0141] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0142] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0143] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0144] B2: Set the activation function to generate the hidden state of the current time step;

[0145] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0146] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0147] In the step A2 and the step B2, the activation function is f(x)=max(ax,x) where a∈(0.01).

[0148] Embodiment 5:

[0149] The real-time analysis system for digital human interactive video services based on deep learning includes:

[0150] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0151] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0152] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0153] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0154] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0155] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0156] The deep learning analysis module includes:

[0157] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0158] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0159] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0160] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0161] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0162] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0163] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0164] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0165] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0166] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0167] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0168] B2: Set the activation function to generate the hidden state of the current time step;

[0169] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0170] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0171] In the A2 step and the B2 step, the activation function is

[0172] Embodiment 6:

[0173] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0174] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0175] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0176] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0177] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0178] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0179] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0180] The deep learning analysis module includes:

[0181] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0182] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0183] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0184] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0185] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0186] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0187] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0188] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0189] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0190] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0191] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0192] B2: Set the activation function to generate the hidden state of the current time step;

[0193] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0194] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0195] In the A2 step and the B2 step, the activation function is Where λ = 1.0507.

[0196] Embodiment 7:

[0197] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0198] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0199] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0200] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0201] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0202] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0203] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0204] The deep learning analysis module includes:

[0205] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0206] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0207] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0208] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0209] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0210] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0211] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0212] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0213] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0214] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0215] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0216] B2: Set the activation function to generate the hidden state of the current time step;

[0217] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0218] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0219] In the A2 step and the B2 step, the activation function is f(x)=x·tanh(ln(1+e x )).

[0220] Embodiment 8:

[0221] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0222] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0223] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0224] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0225] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0226] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0227] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0228] The deep learning analysis module includes:

[0229] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0230] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0231] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0232] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0233] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0234] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0235] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0236] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0237] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0238] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0239] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0240] B2: Set the activation function to generate the hidden state of the current time step;

[0241] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0242] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0243] In the A2 step and the B2 step, the activation function is f(x)=max(0,x), which represents the maximum value between 0 and x, f(x)=max(ax,x) where a∈(0.01), Where λ = 1.0507, f(x) = x·tanh(ln(1+e x )).

[0244] The method for fusing multimodal data at the fusion layer comprises the following steps:

[0245] C1: Feature concatenation, which directly concatenates features from different sources and levels in the channel dimension to form a wider or deeper feature map;

[0246] C2: Feature weighting: set a weight for each feature according to its importance and fuse the features by weighted average;

[0247] C3: Feature pooling, which performs pooling operations on the weighted feature maps to reduce the dimension of the feature maps and reduce the amount of calculation.

[0248] In the step C2, the weighted average is in the form of double weighting, which specifically includes the following steps:

[0249] C21: One-level weight setting, setting weights for voice, gesture and facial expression, which are K1, K2 and K3 respectively;

[0250] C22: Double weight setting, for each extracted voice, gesture and facial expression feature type, the weight is set again, which is voice gesture Facial expressions

[0251] C23: Use double weight calculation method for weighted calculation.

[0252] Embodiment 9:

[0253] The real-time analysis system of digital human interactive video services based on deep learning includes:

[0254] The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions;

[0255] The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0256] Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention;

[0257] The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types;

[0258] The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end;

[0259] Real-time feedback module,The real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user.

[0260] The deep learning analysis module includes:

[0261] The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input;

[0262] Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation;

[0263] The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features;

[0264] The analysis layer performs intent recognition on the fused multimodal data and then outputs the results.

[0265] In the feature extraction layer, the spatial feature extraction of the image includes the following steps:

[0266] A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel;

[0267] A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function;

[0268] A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation;

[0269] A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

[0270] In the feature extraction layer, the temporal feature extraction of speech includes the following steps:

[0271] B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step;

[0272] B2: Set the activation function to generate the hidden state of the current time step;

[0273] B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information;

[0274] B4: Map the state of the hidden layer to the output layer to generate the final output result.

[0275] In the A2 step and the B2 step, the activation function is f(x)=max(0,x), which represents the maximum value between 0 and x, f(x)=max(ax,x) where a∈(0.01), Where λ = 1.0507, f(x) = x·tanh(ln(1+e x )).

[0276] The method for fusing multimodal data at the fusion layer comprises the following steps:

[0277] C1: Feature concatenation, which directly concatenates features from different sources and levels in the channel dimension to form a wider or deeper feature map;

[0278] C2: Feature weighting: set a weight for each feature according to its importance and fuse the features by weighted average;

[0279] C3: Feature pooling, which performs pooling operations on the weighted feature maps to reduce the dimension of the feature maps and reduce the amount of calculation.

[0280] In the step C2, the weighted average is in the form of double weighting, which specifically includes the following steps:

[0281] C21: One-level weight setting, setting weights for voice, gesture and facial expression, which are K1, K2 and K3 respectively;

[0282] C22: Double weight setting, for each extracted voice, gesture and facial expression feature type, the weight is set again, which is voice gesture Facial expressions

[0283] C23: Use double weight calculation method for weighted calculation.

[0284] In the C21 step, K1∈(0.7-0.9), K2∈(0.05-0.15), and K3∈(0.05-0.15).

[0285] In step C23, the weighted calculation formula is: where Y i is the feature value of speech feature type i, S o is the feature value of gesture feature type o, M u is the feature value of the facial expression feature type u.

[0286] Embodiment 10:

[0287] The real-time analysis method of digital human interactive video service based on deep learning includes the following steps:

[0288] S1: Data collection, using the data collection module to collect multimodal data of the user's voice, gestures and facial expressions;

[0289] S2: Data preprocessing, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data;

[0290] S3: Deep learning analysis, building a deep learning model, using the deep learning model to identify the user's multimodal data, and fusing information from different modalities through multimodal fusion technology to more comprehensively understand the user's intentions;

[0291] S4: Interaction action generation: according to the user's intention, matching interaction actions are retrieved from the database and fed back to the front end;

[0292] S5: Real-time feedback module, receives feedback from the interactive action generation module and displays feedback actions to the user.

[0293] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A real-time analysis system for digital human interactive video services based on deep learning, characterized in that: include: The data collection module is responsible for collecting the interactive video data between the user and the digital human, including the multimodal data of the user's voice, gestures and facial expressions; The preprocessing module performs preprocessing on the collected video data, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data; Deep learning analysis module: builds a deep learning model, uses the deep learning model to identify the user's multimodal data, and fuses information from different modalities through multimodal fusion technology to more comprehensively understand the user's intention; The database collects and stores a variety of matching user intentions and digital human interaction action information for different business types; The interactive action generation module searches the database for matching interactive actions based on user intent and feeds them back to the front end; A real-time feedback module receives feedback from the interactive action generation module and displays feedback actions to the user; The deep learning analysis module includes: The input layer receives the multimodal data from the preprocessing module and converts the data into frame sequences and audio streams as input; Feature extraction layer: for image data, convolutional neural network is used to extract spatial features, where spatial features include edges, textures and shapes; for speech data, recurrent neural network is used to extract temporal features, where temporal features include pitch, speaking speed and intonation; The fusion layer performs multi-module data fusion on the extracted spatial features and temporal features; The analysis layer performs intent recognition on the fused multimodal data and then outputs the results; The method for fusing multimodal data at the fusion layer includes the following steps: C1: Feature concatenation, which directly concatenates features from different sources and levels in the channel dimension to form a wider or deeper feature map; C2: Feature weighting: set a weight for each feature according to its importance and fuse the features by weighted average; C3: Feature pooling, which performs pooling operations on the weighted feature maps to reduce the dimension of the feature maps and the amount of calculation; In C2, weighted averaging is performed in the form of double weighting, which specifically includes the following steps: C21: One-level weight setting, setting weights for voice, gesture and facial expression, which are K1, K2 and K3 respectively; C22: Double weight setting, for each extracted feature type of speech, gesture and facial expression, the weight is set again, which are C23: weighted calculation using double weights; In C21, K1∈(0.7-0.9), K2∈(0.05-0.15), K3∈(0.05-0.15); In C23, the formula for weighted calculation is: where Y i is the feature value of speech feature type i, S o is the feature value of gesture feature type o, M u is the feature value of the facial expression feature type u.

2. The real-time analysis system for digital human interactive video services based on deep learning according to claim 1 is characterized in that: In the feature extraction layer, the spatial feature extraction of the image includes the following steps: A1: Convolution kernel definition, define multiple convolution kernels in the convolution layer, and define the weight and bias term of each convolution kernel; A2: Convolution operation: slide each convolution kernel on the image, perform convolution operations on different areas of the image through local connection and weight sharing mechanism, extract local features, and increase the complexity of feature extraction through activation function; A3: Pooling operation, which downsamples the feature map output by the convolutional layer to reduce the size of the feature map and the amount of calculation; A4: Fully connected operation, flattens the high-level features extracted by the convolutional layer and the pooling layer, and then performs classification and regression. It stops after the regression converges and outputs the results.

3. The real-time analysis system for digital human interactive video services based on deep learning according to claim 2 is characterized in that: In the feature extraction layer, the temporal feature extraction of speech includes the following steps: B1: The hidden layer of each time step receives the input of the current time step and the hidden state of the previous time step; B2: Set the activation function to generate the hidden state of the current time step; B3: The state of the hidden layer is cyclically transmitted at each time step, so that the network can remember the previous information and infer the subsequent results based on the previous information; B4: Map the state of the hidden layer to the output layer to generate the final output result.

4. The real-time analysis system for digital human interactive video services based on deep learning according to claim 3 is characterized in that: In A2 and B2, the activation function is f(x)=max(0,x), which represents the maximum value between 0 and x, f(x)=max(ax,x) where a∈(0.01), Where λ = 1.0507, f(x) = x·tanh(ln(1+e x )).

5. A method for real-time analysis of digital human interactive video services based on deep learning, which is a real-time analysis system for digital human interactive video services based on deep learning as described in any one of claims 1 to 4, characterized in that: The following steps are involved: S1: Data collection, using the data collection module to collect multimodal data of the user's voice, gestures and facial expressions; S2: Data preprocessing, including noise removal, enhancement, feature extraction, speech emotion analysis for speech data, key frame extraction and feature point annotation for gesture and facial expression data; S3: Deep learning analysis, building a deep learning model, using the deep learning model to identify the user's multimodal data, and fusing information from different modalities through multimodal fusion technology to more comprehensively understand the user's intentions; S4: Interaction action generation: according to the user's intention, matching interaction actions are retrieved from the database and fed back to the front end; S5: Real-time feedback module, receives feedback from the interactive action generation module and displays feedback actions to the user.

Images