Digital human interaction method and system based on multiple modes

Abstract

The invention relates to the technical field of digital human interaction, and particularly discloses a multi-modal-based digital human interaction method and system, which is used for processing interactive voice information and interactive video stream information by adopting a text processing technology and a video analysis algorithm based on a deep learning model. And automatically judging whether the current interaction of the target user is finished or not according to the interaction response representation between the feature obtained by performing time sequence propagation reasoning on the local semantic association feature of each interaction action and the word granularity semantic association feature of each interaction voice text. Therefore, mutual influence and dependency relationship among different modal features can be caught more finely, so that information of different modals can be fused, dynamic change and deep association among the modals can be caught, more accurate feature fusion is realized, and the current interaction state of a user is judged more accurately.

Description

Technical Field

[0001] The present application relates to the field of digital human interaction technology, and more specifically, to a multimodal digital human interaction method and system. Background Art

[0002] As Internet software and hardware technologies gradually mature, the concept of "metaverse" has become a hot topic, and the virtual digital human industry has also ushered in a period of rapid development. These digital humans are increasingly appearing in various fields, realizing human-computer interaction and becoming a new trend in human-computer interaction. However, at the current stage, the interactive capabilities of virtual digital humans usually still rely on the interactive solutions of early human-computer dialogue products. In the process of interacting with users, they often cannot respond in real time, and it is difficult to achieve the natural experience of communicating with real people.

[0003] In response to the above problems, Chinese patent CN116798427B proposes a multimodal human-computer interaction method and digital human system, which comprehensively processes voice and video information, predicts user intentions and sentence endings in real time, and searches for corresponding content in the digital human content library and outputs it to the user to achieve a natural and smooth interactive experience.

[0004] However, in the above-mentioned patent, the processing of multimodal information is to use multiple encoders for encoding, splicing the encoding results, and making predictions based on the splicing results. However, although this method can achieve feature fusion, it may be insufficient in capturing the relationship between features. That is, feature splicing may not fully consider the mutual influence and dependency between features of different modalities, resulting in insufficiently refined information fusion. Specifically, simply splicing feature vectors of different modalities together ignores the complex interactions and dynamic changes between modalities, and it is difficult to capture the deep-level connections between modalities. For example, there may be subtle synchronization changes between the user's voice and actions, and simple splicing cannot capture these subtle synchronization relationships, thereby reducing the accuracy and sophistication of the model's understanding of user intent.

[0005] Therefore, an optimized multimodal digital human interaction solution is desired. Summary of the invention

[0006] The present application provides a digital human interaction method and system based on multimodality, which can more finely capture the mutual influence and dependency between different modal features, so that it can not only fuse the information of different modalities, but also capture the dynamic changes and deep-level associations between modalities, thereby achieving more accurate feature fusion, and then more accurately judging the user's current interaction status.

[0007] In a first aspect, a multimodal digital human interaction method is provided, comprising: in the process of interaction between a target user and a digital human, obtaining interactive video stream information of the target user during the interaction, and extracting interactive voice information from the interactive video stream information to obtain multimodal interaction information; performing multimodal interaction analysis and task prediction on the target user during the interaction to obtain the current user intention and prediction result of the target user; using the current user intention and prediction result of the target user as user interaction intention query features, searching for target undertaking content corresponding to the user interaction intention query features in a digital human undertaking content library, and outputting the target undertaking content to the target user;

[0008] The multimodal interaction analysis and task prediction of the target user during interaction include:

[0009] Performing text recognition and word-granularity semantic association on the interactive speech information in the multimodal interactive information to obtain a sequence of interactive speech text word-granularity semantic association features;

[0010] Extracting interactive action semantic features and performing spatiotemporal propagation reasoning of action semantic nodes on the interactive video stream information in the multimodal interactive information to obtain a forward propagation reasoning representation of the local temporal semantics of the interactive action;

[0011] Performing interactive response analysis on the sequence of word-granular semantic association features of the interactive speech text and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain the feature representation of the user intention, including: performing nonlinear transformation on the local temporal semantic forward propagation reasoning representation of the interactive action to obtain the local temporal semantic representation of the interactive action after nonlinear transformation; performing local-global collaborative response on the local temporal semantic representation of the interactive action after nonlinear transformation and the sequence of word-granular semantic association features of the interactive speech text to obtain the feature representation of the user intention;

[0012] Based on the feature representation of the user intention, the prediction result is obtained.

[0013] In a second aspect, a multimodal digital human interaction system is provided, comprising: a multimodal interaction information acquisition module, used to acquire the interactive video stream information of the target user during the interaction between the target user and the digital human, and extract the interactive voice information from the interactive video stream information to obtain the multimodal interaction information; a multimodal interaction analysis module, used to analyze the multimodal interaction and predict the task of the target user during the interaction to obtain the current user intention and prediction result of the target user; a target undertaking content output module, used to use the current user intention and prediction result of the target user as the user interaction intention query feature, search for the target undertaking content corresponding to the user interaction intention query feature in the digital human undertaking content library, and output the target undertaking content to the target user;

[0014] Wherein, the multimodal interaction analysis module includes:

[0015] A text word granularity semantic association feature extraction unit, used for performing text recognition and word granularity semantic association on the interactive speech information in the multimodal interactive information to obtain a sequence of interactive speech text word granularity semantic association features;

[0016] An interactive action semantic feature extraction unit, used for performing interactive action semantic feature extraction and action semantic node spatiotemporal propagation reasoning on the interactive video stream information in the multimodal interactive information to obtain a local temporal semantic forward propagation reasoning representation of the interactive action;

[0017] An interactive response analysis unit is used to perform interactive response analysis on the sequence of interactive speech text word granularity semantic association features and the interactive action local temporal semantic forward propagation reasoning representation to obtain the feature representation of the user intention, including: performing nonlinear transformation on the interactive action local temporal semantic forward propagation reasoning representation to obtain the interactive action local temporal semantic representation after nonlinear transformation; performing local-global collaborative response on the interactive action local temporal semantic representation after nonlinear transformation and the sequence of interactive speech text word granularity semantic association features to obtain the feature representation of the user intention;

[0018] A prediction result obtaining unit is used to obtain the prediction result based on the feature representation of the user intention.

[0019] The present application provides a multimodal digital human interaction method and system, which analyzes the multimodal interaction and predicts the tasks of the target user during the interaction to obtain the current user intention and prediction results of the target user. It uses text processing technology and video analysis algorithms based on deep learning models to perform text recognition, word segmentation and semantic encoding on the interactive voice information, and performs key frame sampling and local semantic association on the interactive video stream information. In this way, the interactive response representation between the interactive response features obtained by time-series propagation reasoning of the local semantic association features of each interactive action and the granular semantic association features of each interactive voice text word can automatically determine whether the current interaction of the target user has ended. In this way, the mutual influence and dependency between different modal features can be captured more finely, so that not only the information of different modalities can be integrated, but also the dynamic changes and deep-level associations between modalities can be captured, thereby achieving more accurate feature fusion, and then more accurately judging the current interactive state of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0021] Figure 1 This is a schematic flowchart of a multimodal digital human interaction method according to an embodiment of the present application.

[0022] Figure 2 This is a schematic flowchart of multimodal interaction analysis and task prediction of the target user during interaction in the multimodal digital human interaction method according to an embodiment of the present application.

[0023] Figure 3 A schematic diagram of data flow for multimodal interaction analysis and task prediction of the target user during interaction in a multimodal digital human interaction method according to an embodiment of the present application.

[0024] Figure 4 The present invention is a schematic flowchart of a method for digital human interaction based on multimodality in an embodiment of the present application, in which text recognition and word-granularity semantic association are performed on interactive speech information in the multimodal interactive information to obtain a sequence of word-granularity semantic association features of interactive speech text.

[0025] Figure 5 In the multimodal digital human interaction method of an embodiment of the present application, a schematic flowchart is provided for extracting interactive action semantic features and performing spatiotemporal propagation reasoning of action semantic nodes on the interactive video stream information in the multimodal interactive information to obtain a forward propagation reasoning representation of the local temporal semantics of the interactive action.

[0026] Figure 6In the multimodal digital human interaction method according to an embodiment of the present application, an interactive response analysis is performed on the sequence of word-granular semantic association features of the interactive speech text and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain a schematic flowchart of the feature representation of the user intention.

[0027] Figure 7 This is a schematic block diagram of a multimodal digital human interaction system according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0029] Based on this, this application proposes a multimodal digital human interaction method. Figure 1 FIG. 1 is a schematic flow chart of a multimodal digital human interaction method according to an embodiment of the present application. Figure 1 As shown, the multimodal digital human interaction method includes: S1, in the process of interaction between the target user and the digital human, obtaining the interactive video stream information of the target user during the interaction, and extracting the interactive voice information from the interactive video stream information to obtain the multimodal interaction information; S2, analyzing the multimodal interaction and predicting the task of the target user during the interaction to obtain the current user intention and prediction result of the target user; S3, using the current user intention and prediction result of the target user as the user interaction intention query feature, searching the digital human acceptance content library for the target acceptance content corresponding to the user interaction intention query feature, and outputting the target acceptance content to the target user.

[0030] In the above-mentioned multimodal digital human interaction method, S1, in the process of the target user interacting with the digital human, the interactive video stream information of the target user during the interaction is obtained, and the interactive voice information is extracted from the interactive video stream information to obtain the multimodal interactive information. It should be understood that in the process of the target user interacting with the digital human, the system will obtain the interactive video stream information generated by the target user during the interaction, and extract the interactive voice information from this part of the video stream information, thereby forming multimodal interactive information including visual and auditory information. Optionally, in an embodiment of the present application, in the process of the target user interacting with the digital human, the interactive video stream information of the target user during the interaction is obtained, and the interactive voice information is extracted from the interactive video stream information to obtain the multimodal interactive information, including: when the user interacts with the digital human, the system will record the video stream information from the user in real time. This may include visual data such as the user's facial expressions and body movements. The audio part, that is, the user's interactive voice information, is separated from the recorded video stream. This step ensures that the audio signal can be separated from the video data for separate processing. The extracted voice information is combined with other non-verbal visual clues in the video stream (such as body movements, facial expressions, etc.) to form a multimodal data set. Such multimodal information contains the language content and non-verbal behavior of users when communicating, providing a rich data basis for subsequent analysis. This way of acquiring multimodal interactive information enables the digital human system to not only hear what the user says, but also see how the user expresses it, which is crucial for more accurately understanding and responding to user needs. By integrating different types of sensory input, the system can more fully understand the user's intentions and emotional state, thereby providing a more natural and effective interactive experience.

[0031] In the above-mentioned multimodal digital human interaction method, S2 analyzes the multimodal interaction and predicts the task of the target user during the interaction to obtain the current user intention and prediction results of the target user. It should be understood that by analyzing the multimodal information of the user such as voice, text and vision, the system can understand the user's intention from different angles, which is more comprehensive and accurate than the information provided by a single modality. At the same time, accurate intention recognition and prediction results enable the digital human to provide more natural, timely and appropriate responses, improving the user's experience of interacting with the digital human. Quickly and accurately predicting user intentions and sentence endings allows the digital human to provide follow-up content in a timely manner during the gap between user exchanges, making the conversation smoother. By predicting user intentions, the digital human can make more intelligent decisions, such as proactively providing assistance when the user may need help.

[0032] Correspondingly, in the multimodal interaction analysis and task prediction of the target user during the interaction to obtain the target user's current user intention and prediction results, the present application adopts text processing technology and video analysis algorithms based on deep learning models to perform text recognition, word segmentation and semantic encoding on the interactive voice information, and performs key frame sampling and local semantic association on the interactive video stream information, so as to automatically determine whether the current interaction of the target user has ended based on the interactive response representation between the features obtained by time-series propagation reasoning of the local semantic association features of each interactive action and the granular semantic association features of each interactive voice text word. In this way, the mutual influence and dependency between different modal features can be captured more finely, so that not only the information of different modalities can be integrated, but also the dynamic changes and deep-level associations between modalities can be captured, thereby achieving more accurate feature fusion, and then more accurately judging the current interaction state of the user.

[0033] Figure 2 This is a schematic flowchart of multimodal interaction analysis and task prediction of the target user during interaction in the multimodal digital human interaction method according to an embodiment of the present application. Figure 3 The following is a data flow diagram of the multimodal interaction analysis and task prediction of the target user during the interaction according to the multimodal digital human interaction method of the embodiment of the present application. Figure 2 and Figure 3 As shown, the multimodal interaction analysis and task prediction of the target user during interaction include: S21, performing text recognition and word-granular semantic association on the interactive voice information in the multimodal interaction information to obtain a sequence of word-granular semantic association features of the interactive voice text; S22, performing interactive action semantic feature extraction and action semantic node spatiotemporal propagation reasoning on the interactive video stream information in the multimodal interaction information to obtain a local temporal semantic forward propagation reasoning representation of the interactive action; S23, performing interactive response analysis on the sequence of word-granular semantic association features of the interactive voice text and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain a feature representation of the user intention; S24, obtaining the prediction result based on the feature representation of the user intention.

[0034] In the above-mentioned multimodal digital human interaction method, the S21 performs text recognition and word-granular semantic association on the interactive voice information in the multimodal interactive information to obtain a sequence of word-granular semantic association features of the interactive voice text. It should be understood that, considering that the voice contains a large amount of text content, these text information contain the intention of the target user. Based on this, in order to more deeply understand the content said by the target user and the intention behind it, in the technical solution of the present application, the interactive voice information in the multimodal interactive information is subjected to text recognition to obtain the interactive voice text.

[0035] Next, the interactive voice text is composed of multiple interactive words or phrases, each word or phrase contains its own context information. Therefore, in order to analyze and understand the semantic meaning contained in each word in a more fine-grained manner, in the technical solution of the present application, the interactive voice text is segmented and then passed through a word-granularity semantic encoder to obtain a sequence of interactive voice text word-granularity semantic association feature vectors. In this way, the interactive text content can be understood more carefully and meaningful information can be extracted from it, providing data support for subsequent interactive responses.

[0036] Optionally, in one embodiment of the present application, Figure 4 This is a schematic flow chart of a sequence of word-granularity semantic association features of interactive speech information in the multimodal interactive information in a multimodal digital human interaction method according to an embodiment of the present application, wherein text recognition and word-granularity semantic association are performed on the interactive speech information in the multimodal interactive information to obtain the interactive speech text word-granularity semantic association features. Figure 4 As shown, the interactive speech information in the multimodal interactive information is subjected to text recognition and word-granularity semantic association to obtain a sequence of word-granularity semantic association features of the interactive speech text, including: S211, performing text recognition on the interactive speech information in the multimodal interactive information to obtain interactive speech text; S212, performing word segmentation processing on the interactive speech text and then passing it through a word-granularity semantic encoder to obtain a sequence of word-granularity semantic association feature vectors of the interactive speech text as a sequence of word-granularity semantic association features of the interactive speech text.

[0037] In the above-mentioned multimodal digital human interaction method, the S22 extracts interactive action semantic features and performs spatiotemporal propagation reasoning of action semantic nodes on the interactive video stream information in the multimodal interactive information to obtain a local temporal semantic forward propagation reasoning representation of the interactive action. It should be understood that, considering that the interactive video stream information in the multimodal interactive information is composed of a series of continuous frames, each frame contains a large amount of pixel information, which includes the key actions and events in the interactive process. Therefore, in order to retain the frames that best reflect the changes in video content and reduce data redundancy, in the technical solution of the present application, the interactive video stream information in the multimodal interactive information is key frame sampled. Furthermore, in order to further capture and mine the semantic association of interactive actions between different key time points, the image after key true sampling is passed through an interactive action semantic feature extractor based on the Vi T model to obtain a sequence of interactive action local semantic association feature vectors, so that the high-level semantic features in the interactive action key frame image can be effectively extracted to better understand the user's behavior.

[0038] Furthermore, considering that the sequence of local semantic association feature vectors of the interactive action contains temporal feature information, in order to capture the dynamic information in the interactive action video frame sequence and combine the feature changes in the time dimension to better understand the temporal changes of the user's behavioral actions, in the technical solution of the present application, the sequence of local semantic association feature vectors of the interactive action is subjected to potential spatiotemporal feature aggregation to obtain the local temporal semantic forward propagation reasoning representation vector of the interactive action as the local temporal semantic forward propagation reasoning representation of the interactive action.

[0039] Optionally, in one embodiment of the present application, Figure 5 This is a schematic flow chart of extracting interactive action semantic features and performing spatiotemporal propagation reasoning of action semantic nodes on interactive video stream information in the multimodal interactive information in the digital human interaction method based on multimodal interaction information to obtain a forward propagation reasoning representation of the local temporal semantics of the interactive action. Figure 5 As shown, the interactive video stream information in the multimodal interactive information is subjected to interactive action semantic feature extraction and action semantic node spatiotemporal propagation reasoning to obtain a local temporal semantic forward propagation reasoning representation of the interactive action, including: S221, after key frame sampling of the interactive video stream information in the multimodal interactive information, a sequence of interactive action local semantic association feature vectors is obtained through an interactive action semantic feature extractor based on a Vi T model as a sequence of interactive action local semantic association features; S222, potential spatiotemporal feature aggregation is performed on the sequence of interactive action local semantic association feature vectors to obtain a local temporal semantic forward propagation reasoning representation of the interactive action.

[0040] Optionally, in one embodiment of the present application, potential spatiotemporal feature aggregation is performed on the sequence of local semantic association feature vectors of the interactive action to obtain the local temporal semantic forward propagation inference representation of the interactive action, including: extracting the current interactive action local semantic association feature vector from the sequence of local semantic association feature vectors of the interactive action, and defining other local semantic association feature vectors of the interactive action in the sequence of local semantic association feature vectors of the interactive action as historical interactive action local semantic association feature vectors to obtain a sequence of historical interactive action local semantic association feature vectors; calculating the interactive action static energy value of each historical interactive action local semantic association feature vector in the sequence of historical interactive action local semantic association feature vectors; determining the interactive action energy spatial attenuation factor based on the spatial span coefficient between each historical interactive action local semantic association feature vector and the current interactive action local semantic association feature vector; determining the interactive action energy spatial attenuation factor based on the time span coefficient between each historical interactive action local semantic association feature vector and the current interactive action local semantic association feature vector. The method comprises the following steps: measuring the time decay factor of the interaction action energy space decay factor; based on the interaction action energy space decay factor and the interaction action energy time decay factor, performing spatiotemporal modulation on the interaction action static energy value of each historical interaction action local semantic association feature vector to obtain the interaction action dynamic energy spatiotemporal transfer value of each historical interaction action local semantic association feature vector relative to the current interaction action local semantic association feature vector; inputting the interaction action dynamic energy spatiotemporal transfer value of each historical interaction action local semantic association feature vector into the energy transfer gating module based on the mask function to obtain a sequence of interaction action dynamic energy spatiotemporal transfer weights; based on the sequence of interaction action dynamic energy spatiotemporal transfer weights, calculating the weighted sum of the sequence of historical interaction action local semantic association feature vectors to obtain the historical interaction action local semantic transfer aggregation modulation vector; calculating the positional sum of the historical interaction action local semantic transfer aggregation modulation vector and the current interaction action local semantic association feature vector to obtain the interaction action local temporal semantic forward propagation reasoning representation vector as the interaction action local temporal semantic forward propagation reasoning representation.

[0041] Optionally, in one embodiment of the present application, the interaction action static energy value of each historical interaction action local semantic association feature vector in the sequence of historical interaction action local semantic association feature vectors is calculated, including: extracting the maximum value of the historical interaction action local semantic association feature vector to obtain the maximum value of the historical interaction action local semantic association feature; respectively calculating the mean and variance of the historical interaction action local semantic association feature vector to obtain the mean value of the historical interaction action local semantic association feature and the variance of the historical interaction action local semantic association feature; subtracting the historical interaction action local semantic association feature vector from the mean value of the historical interaction action local semantic association feature by position, and calculating the fourth power of the feature value of each position in the obtained historical interaction action local semantic differential vector to obtain the historical interaction action local semantic differential modulation vector; calculating the expected value of the historical interaction action local semantic differential modulation vector, and dividing the obtained expected value by the square of the variance of the historical interaction action local semantic association feature to obtain the historical interaction action local semantic energy value; adding the maximum value of the historical interaction action local semantic association feature to the historical interaction action local semantic energy value to obtain the interaction action static energy value.

[0042] Optionally, in one embodiment of the present application, based on the interaction action energy spatial attenuation factor and the interaction action energy temporal attenuation factor, the interaction action static energy value of each historical interaction action local semantic association feature vector is temporally modulated to obtain the interaction action dynamic energy spatiotemporal transfer value of each historical interaction action local semantic association feature vector relative to the current interaction action local semantic association feature vector, including: multiplying the interaction action energy spatial attenuation factor and the interaction action energy temporal attenuation factor of the historical interaction action local semantic association feature vector with the first modulation parameter and the second modulation parameter respectively, and adding the obtained interaction action energy spatial attenuation modulation factor and interaction action energy temporal attenuation modulation factor to obtain the interaction action dynamic energy spatiotemporal transfer value.

[0043] Optionally, in one embodiment of the present application, based on the spatial span coefficient between the local semantic association feature vectors of each historical interaction action and the local semantic association feature vector of the current interaction action, the interaction action energy spatial attenuation factor is determined, including: calculating the square of the spatial span value between the local semantic association feature vectors of the historical interaction action and the local semantic association feature vector of the current interaction action to obtain the interaction action spatial span coefficient; dividing the interaction action static energy value corresponding to the local semantic association feature vectors of the historical interaction action by the interaction action spatial span coefficient to obtain the interaction action energy spatial attenuation factor.

[0044] Optionally, in one embodiment of the present application, based on the time span coefficient between the local semantic association feature vectors of each historical interaction action and the local semantic association feature vector of the current interaction action, an interaction action energy time decay factor is determined, including: subtracting the interaction action timestamp of the current interaction action local semantic association feature vector from the interaction action timestamp of the historical interaction action local semantic association feature vector and rounding down to obtain an interaction action time span value; dividing the interaction action time span value by a time decay parameter to obtain an interaction action timing span energy decay coefficient; using the interaction action timing span energy decay coefficient as an exponential power, calculating a natural exponential function value with a natural constant e as the base to obtain an interaction action time span coefficient; dividing the interaction action static energy value corresponding to the historical interaction action local semantic association feature vector by the interaction action time span coefficient to obtain the interaction action energy time decay factor.

[0045] Optionally, in one embodiment of the present application, the interaction action dynamic energy spatiotemporal transfer value of the local semantic association feature vector of each historical interaction action is input into an energy transfer gating module based on a mask function to obtain a sequence of interaction action dynamic energy spatiotemporal transfer weights, including: in response to the interaction action dynamic energy spatiotemporal transfer value being greater than a predetermined threshold, the interaction action dynamic energy spatiotemporal transfer value is input into a sigmoid function to obtain a value as the interaction action dynamic energy spatiotemporal transfer weight.

[0046] In summary, in the embodiment of the present application, the potential spatiotemporal feature aggregation is performed on the sequence of the local semantic association feature vectors of the interactive action, which can be expressed by the formula:

[0047] X={x1,x2,...,x i ,...,x p}

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] Where X is the sequence of local semantic association feature vectors of the interactive action, x1, x2, ..., x i , ..., x pare the first, second, ..., i-th, ..., p-th interactive action local semantic association feature vectors in the sequence of the interactive action local semantic association feature vectors, respectively, i (j) is the feature value of each position in the local semantic association feature vector of the i-th interactive action, E(·) is the expected value, μ i and σ i 2 x i The mean and variance of max(x i ) is to extract x i The maximum value in is x i The corresponding interactive action static energy value, Count(x i →x p ) represents x i With x p The space span value between is x i and x p The interaction energy space attenuation factor between them, t p and t i The distribution represents the timestamp of the local semantic association feature vector between the pth and ith interaction actions, is a floor operation, L is the time decay parameter, exp(·) represents the exponential function value with the natural constant e as the base, is x i and x p The interaction energy time decay factor between them, γ and δ are the corresponding modulation parameters, e (i→p) is x i and x p mask(·) is the masking process, sigmoid(·) is the sigmoid function, θ is the predetermined threshold, p is the number of vectors in the sequence of local semantic association feature vectors of the interaction action, and f1 is the local temporal semantic forward propagation inference representation vector of the interaction action.

[0055] Specifically, firstly, the current interactive action local semantic association feature vector is extracted from the sequence of the interactive action local semantic association feature vector as the current node feature vector, and the historical interactive action local semantic association feature vector is obtained to obtain the sequence of the historical interactive action local semantic association feature vector as the sequence of the historical node feature vector. Then, the static energy value of each historical node feature vector is calculated to obtain the sequence of the interactive action static energy value. That is, the static energy value is related to the maximum value, mean value and variance of each historical node feature vector. This static energy value provides a quantitative measure for the intrinsic characteristics of the feature vector in space, and it provides a reference standard for the spatiotemporal attenuation effect of potential energy, thereby helping the model to identify the static criticality and influence of each node in space and time. Then, based on the spatial span coefficient and the time span coefficient between each historical node and the current node feature vector, the energy space attenuation factor and the energy time attenuation factor are determined to obtain the energy space attenuation factor of each interactive action and the energy time attenuation factor of each interactive action. It should be understood that the spatial span coefficient and the time span coefficient capture the spatial relationship and the time series relationship between nodes respectively. Using these coefficients, the attenuation factor adjusts the weight of the information flow between nodes. That is, the energy time decay factor and the energy space decay factor allow the model to adjust the intensity of information interaction according to the spatiotemporal distance between nodes, so that the transmission of information is closer to the physical phenomena of the real world. Furthermore, based on the energy space decay factor and the energy time decay factor, the static energy value of each historical node feature vector is spatiotemporally modulated to enhance the model's ability to extract dynamic changes in time series and obtain the dynamic energy spatiotemporal transfer value of the interactive action. Subsequently, the dynamic energy spatiotemporal transfer of each interactive action is passed through an energy transfer gating module based on a mask function to screen and determine key features, and the weights are adaptively adjusted. Finally, each masked value is used as a weight to perform weighted fusion on the feature vectors of each historical node to synthesize the feature information of the entire historical node, obtain the local semantic transfer aggregation modulation vector of the historical interactive action, and fuse the historical aggregation modulation vector with the current node feature vector to obtain a local temporal semantic forward propagation reasoning representation vector of the interactive action that integrates historical information and the current state.

[0056] In the above-mentioned multimodal digital human interaction method, the S23 performs interactive response analysis on the sequence of word-granular semantic association features of the interactive speech text and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain the feature representation of the user's intention. It should be understood that the sequence of word-granular semantic association feature vectors of the interactive speech text and the local temporal semantic forward propagation reasoning representation vector of the interactive action respectively represent the interactive information of the target user in two different modes during the interaction process. Based on this, in order to be able to more comprehensively understand the user's intentions and behaviors, thereby providing a more accurate and personalized response, in the technical solution of the present application, the sequence of word-granular semantic association features of the interactive speech text and the local temporal semantic forward propagation reasoning representation of the interactive action are subjected to interactive response analysis to obtain the speech word granularity-action temporal propagation reasoning semantic interactive response vector.

[0057] Optionally, in one embodiment of the present application, Figure 6 The present invention is a schematic flow chart of performing interactive response analysis on the sequence of the interactive speech text word granularity semantic association features and the interactive action local temporal semantic forward propagation reasoning representation to obtain the feature representation of the user intention in the multimodal digital human interaction method of the embodiment of the present application. Figure 6 As shown, interactive response analysis is performed on the sequence of word-granular semantic association features of the interactive speech text and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain a feature representation of the user intention, including: S231, performing a nonlinear transformation on the local temporal semantic forward propagation reasoning representation of the interactive action to obtain a local temporal semantic representation of the interactive action after nonlinear transformation; S232, performing a local-global collaborative response on the local temporal semantic representation of the interactive action after nonlinear transformation and the sequence of word-granular semantic association features of the interactive speech text to obtain a feature representation of the user intention.

[0058] Optionally, in one embodiment of the present application, the local temporal semantics forward propagation reasoning representation of the interactive action is subjected to a nonlinear transformation to obtain the local temporal semantics representation of the interactive action after the nonlinear transformation, including: multiplying the local temporal semantics forward propagation reasoning representation vector of the interactive action with the interactive action transformation matrix, and then adding the obtained interactive action transformation vector and the interactive action bias vector by position to obtain the local temporal semantics forward propagation reasoning representation vector of the interactive action after the nonlinear transformation as the local temporal semantics representation of the interactive action after the nonlinear transformation.

[0059] Optionally, in one embodiment of the present application, a local-global collaborative response is performed on the sequence of the local temporal semantic representation of the interactive action after the nonlinear transformation and the word-granular semantic association features of the interactive speech text to obtain a feature representation of the user intention, including: calculating the associated interaction vector between the local temporal semantic forward propagation inference representation vector of the interactive action after the nonlinear transformation and each interactive speech text word-granular semantic association feature vector in the sequence of the interactive speech text word-granular semantic association feature vector to obtain a sequence of interactive action-interactive speech semantic association interaction vectors; calculating each interactive action-interactive speech semantic association interaction vector in the sequence of the interactive action-interactive speech semantic association interaction vectors. The characteristic interaction energy factor of the interaction action-interaction speech semantic association interaction vector is obtained to obtain a sequence of interaction action-interaction speech semantic feature interaction energy factors; the sequence of interaction action-interaction speech semantic feature interaction energy factors is input into the Sigmoid function to obtain a sequence of normalized interaction action-interaction speech semantic feature interaction energy factors; the sequence of normalized interaction action-interaction speech semantic feature interaction energy factors is used as a sequence of weights, and the position-weighted sum of the sequence of interaction action-interaction speech semantic association interaction vectors is calculated to obtain a speech word granularity-action temporal propagation reasoning semantic interaction response vector as a feature representation of the user intention.

[0060] Optionally, in one embodiment of the present application, the associated interaction vectors between the local temporal sequence semantic forward propagation reasoning representation vector of the interactive action after the nonlinear transformation and each interactive speech text word granularity semantic association feature vector in the sequence of the interactive speech text word granularity semantic association feature vector are calculated to obtain a sequence of interactive action-interactive speech semantic association interaction vectors, including: respectively calculating the positional subtraction, positional dot multiplication and positional addition between the local temporal sequence semantic forward propagation reasoning representation vector of the interactive action after the nonlinear transformation and the interactive speech text word granularity semantic association feature vector to obtain the interactive action-interactive speech semantic association interaction vector sequence. Interaction speech semantics difference vector, interaction action-interaction speech semantics dot product vector and interaction action-interaction speech semantics sum vector; cascade the interaction action-interaction speech semantics difference vector, the interaction action-interaction speech semantics dot product vector and the interaction action-interaction speech semantics sum vector to obtain an interaction action-interaction speech semantics cascade vector; after performing one-dimensional convolution coding on the interaction action-interaction speech semantics cascade vector, perform local window-based maximum pooling processing on the obtained interaction action-interaction speech semantics convolution coding vector to obtain the interaction action-interaction speech semantics associated interaction vector.

[0061] Optionally, in one embodiment of the present application, the characteristic interaction energy factor of each interactive action-interactive speech semantics associated interaction vector in the sequence of the interactive action-interactive speech semantics associated interaction vectors is calculated to obtain a sequence of interactive action-interactive speech semantics characteristic interaction energy factors, including: respectively calculating the mean and variance of the interactive action-interactive speech semantics associated interaction vector to obtain the interactive action-interactive speech semantics mean and the interactive action-interactive speech semantics variance; calculating the positional difference between the interactive action-interactive speech semantics associated interaction vector and the interactive action-interactive speech semantics mean to obtain the interactive action-interactive speech semantics differential vector; calculating the fourth power of each eigenvalue in the interactive action-interactive speech semantics differential vector to obtain the interactive action-interactive speech semantics modulation differential vector; calculating the expected value of the interactive action-interactive speech semantics modulation differential vector to obtain the interactive action-interactive speech semantics expected value; and The interactive action-interactive speech semantic expectation value is divided by the square of the interactive action-interactive speech semantic variance to obtain the interactive action-interactive speech semantic kurtosis value; the sum of the interactive action-interactive speech semantic variance and the hyperparameter is calculated to obtain the interactive action-interactive speech associated interaction first energy factor; the square of the difference between the interactive action-interactive speech semantic kurtosis value and the interactive action-interactive speech semantic mean is calculated to obtain the interactive action-interactive speech semantic difference value; the value obtained by multiplying the interactive action-interactive speech semantic variance by a constant two and the value obtained by multiplying the hyperparameter by a constant two are added to the interactive action-interactive speech semantic difference value to obtain the interactive action-interactive speech associated interaction second energy factor; the interactive action-interactive speech associated interaction first energy factor is divided by the interactive action-interactive speech associated interaction second energy factor to obtain the interactive action-interactive speech semantic feature interaction energy factor.

[0062] In summary, in the embodiment of the present application, the interactive response analysis is performed on the sequence of the interactive speech text word granularity semantic association features and the interactive action local temporal semantic forward propagation reasoning representation, which can be expressed by the formula:

[0063] f 1t =Wf1+b

[0064]

[0065]

[0066]

[0067] E norm,12i =sigmoid(E 12i )

[0068] v m =∑i f 12i ⊙E norm，12i

[0069] Wherein, f1 is the local temporal semantic forward propagation reasoning representation vector of the interactive action, W and b are the interactive action transformation matrix and the interactive action bias vector respectively, and f 1t is the forward propagation reasoning representation vector of the local temporal semantics of the interactive action after the nonlinear transformation, f 2i is the i-th interactive speech text word granularity semantic association feature vector in the sequence of the interactive speech text word granularity semantic association feature vectors, ⊙ and are position-wise subtraction, position-wise multiplication, and position-wise addition, respectively. [·,·,·] is a cascade process. conv1D(·) is a one-dimensional convolutional encoding operation. MaxPool(·) is a maximum pooling operation. 12i f 2i and f 1t The interaction vector between the interaction action and the interaction speech semantic association, f 12i (t) is f 12i The eigenvalues ​​at each position in 12i Yes 12i The mean of , E(·) is the expected value, σ 12i 2 Yes 12i The variance of k 12i f 12i The kurtosis value, λ is a hyperparameter, E 12i Yes 12i The corresponding interaction action-interaction speech semantic feature interaction energy factor, sigmoid(·) is the sigmoid function, E norm，12i It is M 12i The corresponding normalized interaction action-interaction speech semantic feature interaction energy factor, ⊙ is the point multiplication by position, v m It is the speech word granularity-action temporal propagation reasoning semantic interaction response vector.

[0070] Specifically, the sequence of the interactive speech text word granularity semantic association features and the interactive action local temporal semantic forward propagation reasoning representation are subjected to interactive response analysis to obtain the speech word granularity-action temporal propagation reasoning semantic interactive response vector. First, the interactive action local temporal semantic forward propagation reasoning representation is subjected to nonlinear transformation to capture and represent the complex relationship and pattern in the interactive action to obtain the interactive action local temporal semantic forward propagation reasoning representation vector after nonlinear transformation. Then, the interactive action local temporal semantic forward propagation reasoning representation vector and the sequence of the interactive speech text word granularity semantic association features are subjected to local-global collaborative response to comprehensively consider the information in the current moment or local time period and the information in the entire interactive process, so as to capture and mine the semantic association relationship between each interactive speech text word and the interactive action, and obtain the speech word granularity-action temporal propagation reasoning semantic interactive response vector. In this way, the obtained interactive response vector can be used as a label vector for query, which means that the corresponding response content can be found in the digital human undertaking content library according to this vector, that is, the generation of the label vector is to facilitate efficient matching and retrieval in the content library.

[0071] In the above-mentioned multimodal digital human interaction method, the S24 obtains the prediction result based on the feature representation of the user intention. Optionally, in one embodiment of the present application, the prediction result is obtained based on the feature representation of the user intention, including: passing the speech word granularity-action temporal propagation reasoning semantic interaction response vector through a classifier-based interaction end predictor to obtain the prediction result, and the prediction result is used to indicate whether the current interaction of the target user has ended. That is, the feature representation of the user intention obtained by performing interaction response analysis using the sequence of the interactive speech text word granularity semantic association feature vectors and the interactive action local temporal semantic forward propagation reasoning representation vector is used for classification processing, so as to automatically determine whether the current interaction of the target user has ended.

[0072] Preferably, in an example of the present application, passing the speech word granularity-action temporal propagation reasoning semantic interaction response vector through a classifier-based interaction end predictor to obtain the prediction result includes:

[0073] Calculate the sum of the absolute values ​​of each characteristic value of the speech word granularity-action timing propagation reasoning semantic interaction response vector to obtain a first speech word granularity-action timing propagation reasoning semantic interaction response space structure value, and calculate the square root of the sum of the squares of each characteristic value of the speech word granularity-action timing propagation reasoning semantic interaction response vector to obtain a second speech word granularity-action timing propagation reasoning semantic interaction response space structure value;

[0074] Multiply each feature value of the speech word granularity-action timing propagation reasoning semantic interaction response vector by the first speech word granularity-action timing propagation reasoning semantic interaction response space structure value and the second speech word granularity-action timing propagation reasoning semantic interaction response space structure value to obtain a first speech word granularity-action timing propagation reasoning semantic interaction response structure reference value and a second speech word granularity-action timing propagation reasoning semantic interaction response structure reference value corresponding to each feature value;

[0075] Multiply each feature value of the speech word granularity-action timing propagation reasoning semantic interaction response vector by the length of the speech word granularity-action timing propagation reasoning semantic interaction response vector and the square root of the length to obtain a first speech word granularity-action timing propagation reasoning semantic interaction response scale transformation value and a second speech word granularity-action timing propagation reasoning semantic interaction response scale transformation value corresponding to each feature value;

[0076] Dividing the first speech word granularity-action timing propagation reasoning semantic interaction response structure reference value by the difference between the first speech word granularity-action timing propagation reasoning semantic interaction response space structure value and the first speech word granularity-action timing propagation reasoning semantic interaction response scale transformation value to obtain the first speech word granularity-action timing propagation reasoning semantic interaction response transformation adjustment value;

[0077] Dividing the second speech word granularity-action timing propagation reasoning semantic interaction response structure reference value by the difference between the second speech word granularity-action timing propagation reasoning semantic interaction response space structure value and the second speech word granularity-action timing propagation reasoning semantic interaction response scale transformation value to obtain the second speech word granularity-action timing propagation reasoning semantic interaction response transformation adjustment value;

[0078] Calculate the weighted sum of the first speech word granularity-action timing propagation reasoning semantic interaction response transformation adjustment value and the second speech word granularity-action timing propagation reasoning semantic interaction response transformation adjustment value to obtain each eigenvalue of the optimized speech word granularity-action timing propagation reasoning semantic interaction response vector;

[0079] The optimized speech word granularity-action timing propagation reasoning semantic interaction response vector is passed through a classifier-based interaction end predictor to obtain the prediction result.

[0080] Here, the speech word granularity-action temporal propagation reasoning semantic interaction response vector, denoted as V, is optimized as follows:

[0081]

[0082] f 1i =(Δ×f i) / (Δ-n×f i )

[0083]

[0084]

[0085]

[0086] f i ∈V∈R n

[0087] f 1i ∈V1∈R n

[0088] f 2i ∈V2∈R n

[0089] Where V is the speech word granularity-action time sequence propagation reasoning semantic interaction response vector, R represents a real number set, L represents the length of the feature vector, and f i represents the feature value of the i-th position of the speech word granularity-action time sequence propagation reasoning semantic interaction response vector, Δ represents the first speech word granularity-action time sequence propagation reasoning semantic interaction response space structure value, β represents the second speech word granularity-action time sequence propagation reasoning semantic interaction response space structure value, ⊙ represents the position point multiplication, represents positional addition, V1 represents the first speech word granularity-action temporal propagation reasoning semantic interaction response transformation adjustment vector, and f 1i represents the first speech word granularity-action time sequence propagation reasoning semantic interaction response transformation adjustment value, V2 represents the second speech word granularity-action time sequence propagation reasoning semantic interaction response transformation adjustment vector, f 2i represents the second speech word granularity-action timing propagation reasoning semantic interaction response transformation adjustment value, ω represents the weighted hyperparameter, and V′ represents the optimized speech word granularity-action timing propagation reasoning semantic interaction response vector.

[0090] That is, the present application takes into account that the sequence of the interactive speech text word granularity semantic association feature vectors and the interactive action local temporal semantic forward propagation reasoning representation vector respectively represent the text encoding features of the interactive speech text and the time domain node potential energy transfer aggregation features of the time domain context-associated image semantic features of the key frames of the interactive video stream. When performing interactive response analysis on them, the difference in feature distribution dimensions of heteromodal data will also make the dimensional composite response semantic representation of the speech word granularity-action temporal propagation reasoning semantic interactive response vector have a complex feature space distribution structure based on different temporal feature distribution patterns, thereby expecting to improve its classification regression convergence and generalization effects under complex distribution structures.

[0091] Therefore, in the preferred example, with respect to the spatial structure information of the feature set of the speech word granularity-action timing propagation reasoning semantic interaction response vector in the high-dimensional space, the scale-based frame transformation of each feature value of the speech word granularity-action timing propagation reasoning semantic interaction response vector is performed by taking the class norm space structured representation of the speech word granularity-action timing propagation reasoning semantic interaction response vector as the reference window, and the spatial structure-based frame attention weight adjustment of each feature value of the speech word granularity-action timing propagation reasoning semantic interaction response vector is implemented to ensure the invariance of the spatial transformation (translation, scaling and rotation) of the speech word granularity-action timing propagation reasoning semantic interaction response vector under the feature space interaction, thereby improving the convergence and generalization effect of the classification regression of the feature set of the speech word granularity-action timing propagation reasoning semantic interaction response vector under the complex spatial structure representation, and improving the accuracy of the prediction result obtained by passing it through the classifier-based interaction end predictor. In this way, the mutual influence and dependency between different modal features can be captured more finely, so that not only the information of different modalities can be integrated, but also the dynamic changes and deep correlations between modalities can be captured, thereby achieving more accurate feature fusion and further more accurately judging the user's current interaction status.

[0092] In the above-mentioned multimodal digital human interaction method, S3 uses the current user intention and prediction result of the target user as the user interaction intention query feature, searches for the target undertaking content corresponding to the user interaction intention query feature in the digital human undertaking content library, and outputs the target undertaking content to the target user. In particular, the user interaction intention query feature is used as the intention query label, and label matching is performed based on the intention query label and the label in the digital human undertaking content library. It should be understood that labels, as a structured representation method, can quickly match the corresponding items in the content library, thereby improving the query efficiency. That is, labels usually contain key information of user intentions, and through label matching, target undertaking content matching the user intention can be found more accurately, thereby improving the user experience and satisfaction. In this way, the mutual influence and dependency between different modal features can be captured more finely, so that not only the information of different modalities can be integrated, but also the dynamic changes and deep-level associations between modalities can be captured, thereby achieving more accurate feature fusion, and then more accurately judging the current interaction state of the user.

[0093] In summary, the multimodal digital human interaction method according to the embodiment of the present application is explained, and in the multimodal interaction analysis and task prediction of the target user during the interaction to obtain the current user intention and prediction result of the target user, it adopts text processing technology and video analysis algorithm based on deep learning model to perform text recognition, word segmentation and semantic encoding on the interactive voice information, and performs key frame sampling and local semantic association on the interactive video stream information, so as to automatically judge whether the current interaction of the target user is over according to the interactive response representation between the features obtained by time-series propagation reasoning of the local semantic association features of each interactive action and the granular semantic association features of each interactive voice text word. In this way, the mutual influence and dependency between different modal features can be captured more finely, so that not only the information of different modalities can be integrated, but also the dynamic changes and deep-level associations between modalities can be captured, so as to achieve more accurate feature fusion, and then more accurately judge the current interaction state of the user.

[0094] Figure 7 Schematic block diagram of a multi-modal digital human interaction system according to an embodiment of the present application. Figure 7 As shown, the multimodal digital human interaction system 100 includes: a multimodal interaction information acquisition module 110, which is used to obtain the interactive video stream information of the target user during the interaction between the target user and the digital human, and extract the interactive voice information from the interactive video stream information to obtain multimodal interaction information; a multimodal interaction analysis module 120, which is used to analyze the multimodal interaction and predict the task of the target user during the interaction to obtain the current user intention and prediction result of the target user; a target acceptance content output module 130, which is used to use the current user intention and prediction result of the target user as the user interaction intention query feature, search for the target acceptance content corresponding to the user interaction intention query feature in the digital human acceptance content library, and output the target acceptance content to the target user.

[0095] Optionally, in one embodiment of the present application, the multimodal interaction analysis module includes: a text word granularity semantic association feature extraction unit, which is used to perform text recognition and word granularity semantic association on the interactive voice information in the multimodal interaction information to obtain a sequence of interactive voice text word granularity semantic association features; an interactive action semantic feature extraction unit, which is used to perform interactive action semantic feature extraction and action semantic node spatiotemporal propagation reasoning on the interactive video stream information in the multimodal interaction information to obtain a local temporal semantic forward propagation reasoning representation of the interactive action; an interactive response analysis unit, which is used to perform interactive response analysis on the sequence of interactive voice text word granularity semantic association features and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain a feature representation of the user intention, including: performing a nonlinear transformation on the local temporal semantic forward propagation reasoning representation of the interactive action to obtain a local temporal semantic representation of the interactive action after the nonlinear transformation; performing a local-global collaborative response on the local temporal semantic representation of the interactive action after the nonlinear transformation and the sequence of word granularity semantic association features of the interactive voice text to obtain a feature representation of the user intention; a prediction result acquisition unit, which is used to obtain the prediction result based on the feature representation of the user intention.

[0096] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

Claims

1. A multimodal digital human interaction method, characterized in that: include: During the process of interaction between the target user and the digital human, the interactive video stream information of the target user during the interaction is obtained, and the interactive voice information is extracted from the interactive video stream information to obtain multimodal interactive information; Analyze the multimodal interaction and predict the task of the target user during the interaction to obtain the current user intention and prediction result of the target user; use the current user intention and prediction result of the target user as the user interaction intention query feature, search for the target undertaking content corresponding to the user interaction intention query feature in the digital human undertaking content library, and output the target undertaking content to the target user; The multimodal interaction analysis and task prediction of the target user during interaction include: Performing text recognition and word-granularity semantic association on the interactive speech information in the multimodal interactive information to obtain a sequence of interactive speech text word-granularity semantic association features; Extracting interactive action semantic features and performing spatiotemporal propagation reasoning of action semantic nodes on the interactive video stream information in the multimodal interactive information to obtain a forward propagation reasoning representation of the local temporal semantics of the interactive action; Performing interactive response analysis on the sequence of word-granular semantic association features of the interactive speech text and the local temporal semantic forward propagation reasoning representation of the interactive action to obtain the feature representation of the user intention, including: performing nonlinear transformation on the local temporal semantic forward propagation reasoning representation of the interactive action to obtain the local temporal semantic representation of the interactive action after nonlinear transformation; performing local-global collaborative response on the local temporal semantic representation of the interactive action after nonlinear transformation and the sequence of word-granular semantic association features of the interactive speech text to obtain the feature representation of the user intention; Based on the feature representation of the user intention, the prediction result is obtained.

2. The multimodal digital human interaction method according to claim 1, characterized in that: The interactive speech information in the multimodal interactive information is subjected to text recognition and word-granular semantic association to obtain a sequence of interactive speech text word-granular semantic association features, including: Performing text recognition on the interactive voice information in the multimodal interactive information to obtain interactive voice text; The interactive speech text is word segmented and then passed through a word granularity semantic encoder to obtain a sequence of interactive speech text word granularity semantic association feature vectors as a sequence of interactive speech text word granularity semantic association features.

3. The multimodal digital human interaction method according to claim 2, characterized in that: Extracting interactive action semantic features and performing spatiotemporal propagation reasoning of action semantic nodes on the interactive video stream information in the multimodal interactive information to obtain a forward propagation reasoning representation of the local temporal semantics of the interactive action, including: After performing key frame sampling on the interactive video stream information in the multimodal interactive information, the interactive action semantic feature extractor based on the ViT model is used to obtain a sequence of interactive action local semantic association feature vectors as the sequence of interactive action local semantic association features; Potential spatiotemporal feature aggregation is performed on the sequence of local semantic association feature vectors of the interactive action to obtain the local temporal semantic forward propagation reasoning representation of the interactive action.

4. The multimodal digital human interaction method according to claim 3, characterized in that: Performing potential spatiotemporal feature aggregation on the sequence of local semantic association feature vectors of the interactive action to obtain a forward propagation reasoning representation of the local temporal semantics of the interactive action, including: Extracting a current interaction action local semantic association feature vector from the sequence of interaction action local semantic association feature vectors, and defining other interaction action local semantic association feature vectors in the sequence of interaction action local semantic association feature vectors as historical interaction action local semantic association feature vectors to obtain a sequence of historical interaction action local semantic association feature vectors; Calculating the interactive action static energy value of each historical interactive action local semantic association feature vector in the sequence of the historical interactive action local semantic association feature vectors; Determining an interaction action energy spatial attenuation factor based on a spatial span coefficient between the local semantic association feature vectors of each historical interaction action and the local semantic association feature vector of the current interaction action; Determining an interaction action energy time decay factor based on a time span coefficient between the local semantic association feature vectors of each historical interaction action and the local semantic association feature vector of the current interaction action; Based on the interaction action energy spatial attenuation factor and the interaction action energy temporal attenuation factor, the interaction action static energy value of each of the local semantic association feature vectors of the historical interaction actions is temporally modulated to obtain the interaction action dynamic energy spatiotemporal transfer value of each of the local semantic association feature vectors of the historical interaction actions relative to the local semantic association feature vector of the current interaction action; Inputting the interactive action dynamic energy spatiotemporal transfer value of each historical interactive action local semantic association feature vector into an energy transfer gating module based on a mask function to obtain a sequence of interactive action dynamic energy spatiotemporal transfer weights; Based on the sequence of the dynamic energy spatiotemporal transfer weights of the interactive action, calculating the weighted sum of the sequence of local semantic association feature vectors of the historical interactive action to obtain the local semantic transfer aggregation modulation vector of the historical interactive action; The positional sum of the historical interaction action local semantic transfer aggregation modulation vector and the current interaction action local semantic association feature vector is calculated to obtain an interaction action local temporal semantic forward propagation reasoning representation vector as the interaction action local temporal semantic forward propagation reasoning representation.

5. The multimodal digital human interaction method according to claim 4, characterized in that: Calculating the interactive action static energy value of each historical interactive action local semantic association feature vector in the sequence of the historical interactive action local semantic association feature vectors includes: Extracting the maximum value of the local semantic association feature vector of the historical interaction action to obtain the maximum value of the local semantic association feature of the historical interaction action; Respectively calculating the mean and variance of the local semantic association feature vector of the historical interaction action to obtain the mean of the local semantic association feature of the historical interaction action and the variance of the local semantic association feature of the historical interaction action; subtracting the local semantic association feature vector of the historical interaction action from the mean of the local semantic association feature of the historical interaction action by position, and calculating the fourth power of the feature value of each position in the local semantic differential vector of the historical interaction action to obtain the local semantic differential modulation vector of the historical interaction action; calculating the expected value of the local semantic differential modulation vector of the historical interaction action, and dividing the obtained expected value by the square of the variance of the local semantic association feature of the historical interaction action to obtain the local semantic energy value of the historical interaction action; The maximum value of the local semantic association feature of the historical interactive action is added to the local semantic energy value of the historical interactive action to obtain the static energy value of the interactive action.

6. The multimodal digital human interaction method according to claim 5, characterized in that: Based on the interaction action energy spatial attenuation factor and the interaction action energy temporal attenuation factor, the interaction action static energy value of each historical interaction action local semantic association feature vector is temporally modulated to obtain the interaction action dynamic energy spatiotemporal transfer value of each historical interaction action local semantic association feature vector relative to the current interaction action local semantic association feature vector, including: multiplying the interaction action energy spatial attenuation factor and the interaction action energy temporal attenuation factor of the historical interaction action local semantic association feature vector with the first modulation parameter and the second modulation parameter respectively, and adding the obtained interaction action energy spatial attenuation modulation factor and interaction action energy temporal attenuation modulation factor to obtain the interaction action dynamic energy spatiotemporal transfer value.

7. The multimodal digital human interaction method according to claim 6, characterized in that: A nonlinear transformation is performed on the local temporal semantics forward propagation reasoning representation of the interactive action to obtain the local temporal semantics representation of the interactive action after the nonlinear transformation, including: multiplying the local temporal semantics forward propagation reasoning representation vector of the interactive action with the interactive action transformation matrix, and then adding the obtained interactive action transformation vector and the interactive action bias vector by position to obtain the local temporal semantics forward propagation reasoning representation vector of the interactive action after the nonlinear transformation as the local temporal semantics representation of the interactive action after the nonlinear transformation.

8. The multimodal digital human interaction method according to claim 7, characterized in that: The local temporal semantic representation of the interactive action after the nonlinear transformation and the sequence of the word-granular semantic association features of the interactive speech text are subjected to local-global collaborative response to obtain the feature representation of the user intention, including: Calculate the associated interaction vector between the local temporal semantic forward propagation reasoning representation vector of the interactive action after the nonlinear transformation and each interactive speech text word granularity semantic association feature vector in the sequence of the interactive speech text word granularity semantic association feature vector to obtain a sequence of interactive action-interactive speech semantic association interaction vectors; Calculating the characteristic interaction energy factor of each interactive action-interactive speech semantics associated interaction vector in the sequence of interactive action-interactive speech semantics associated interaction vectors to obtain a sequence of interactive action-interactive speech semantics characteristic interaction energy factors; Inputting the sequence of interaction energy factors of interaction action-interaction speech semantic features into a Sigmoid function to obtain a sequence of normalized interaction energy factors of interaction action-interaction speech semantic features; Using the sequence of normalized interactive action-interactive speech semantic feature interaction energy factors as a sequence of weights, the position-weighted sum of the sequence of interactive action-interactive speech semantic association interaction vectors is calculated to obtain a speech word granularity-action temporal propagation reasoning semantic interaction response vector as a feature representation of the user intention.

9. The multimodal digital human interaction method according to claim 8, characterized in that: Based on the feature representation of the user intention, the prediction result is obtained, including: passing the speech word granularity-action timing propagation inference semantic interaction response vector through a classifier-based interaction end predictor to obtain the prediction result, and the prediction result is used to indicate whether the current interaction of the target user is ended.

10. A multimodal digital human interaction system, characterized in that: include: A multimodal interaction information acquisition module is used to acquire the interactive video stream information of the target user during the interaction between the target user and the digital human, and to extract the interactive voice information from the interactive video stream information to obtain the multimodal interaction information; A multimodal interaction analysis module, used for multimodal interaction analysis and task prediction of the target user during interaction to obtain the current user intention and prediction result of the target user; The target undertaking content is given to the output module, which is used to use the current user intention and prediction result of the target user as the user interaction intention query feature, search for the target undertaking content corresponding to the user interaction intention query feature in the digital human undertaking content library, and output the target undertaking content to the target user; Wherein, the multimodal interaction analysis module includes: A text word granularity semantic association feature extraction unit, used for performing text recognition and word granularity semantic association on the interactive speech information in the multimodal interactive information to obtain a sequence of interactive speech text word granularity semantic association features; An interactive action semantic feature extraction unit, used for performing interactive action semantic feature extraction and action semantic node spatiotemporal propagation reasoning on the interactive video stream information in the multimodal interactive information to obtain a local temporal semantic forward propagation reasoning representation of the interactive action; An interactive response analysis unit is used to perform interactive response analysis on the sequence of interactive speech text word granularity semantic association features and the interactive action local temporal semantic forward propagation reasoning representation to obtain the feature representation of the user intention, including: performing nonlinear transformation on the interactive action local temporal semantic forward propagation reasoning representation to obtain the interactive action local temporal semantic representation after nonlinear transformation; performing local-global collaborative response on the interactive action local temporal semantic representation after nonlinear transformation and the sequence of interactive speech text word granularity semantic association features to obtain the feature representation of the user intention; A prediction result obtaining unit is used to obtain the prediction result based on the feature representation of the user intention.

Images