Digital human interaction method and system based on multi-modal understanding

Abstract

The invention relates to a digital human interaction method and system based on multi-modal understanding, and belongs to the technical field of digital human intelligent interaction. The method comprises the following steps: acquiring multi-modal data, and obtaining multi-modal features through a feature extraction model according to the multi-modal data; obtaining enhanced modal features through a knowledge enhancement model according to the multi-modal features; according to the enhanced modal features, a modal fusion feature vector is obtained through a graph representation learning model; and the big language model learns modal fusion feature vectors through a multi-head attention mechanism to obtain natural language output. The digital human understanding of multi-modal data is realized, and the smoothness, naturalness and intellectualization in the digital human interaction process are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital human intelligent interaction, and specifically relates to a digital human interaction method and system based on multimodal understanding. Background Art

[0002] Digital human refers to a real and natural virtual human with cognitive and interactive capabilities, and simulating human thinking and behavioral logic. Digital human can not only interact with humans in a virtual environment, but also understand, analyze and respond to various information. In recent years, with the rapid development of cutting-edge technologies such as computer technology and artificial intelligence, the intelligent optimization of digital human has become an important research hotspot. The key to optimizing the intelligence of digital human lies in the processing and fusion of multimodal data. Multimodal data covers multiple forms such as text, audio, and video, which can provide a more comprehensive understanding of the environment.

[0003] However, existing technologies still mainly focus on single-modal data processing. Although some methods have the ability to process multimodal data, in actual applications, they often only conduct in-depth analysis on one of the modalities. Due to the significant differences in the requirements for data feature extraction of different modalities and the lack of a unified multimodal fusion framework, the performance of digital humans in real-time interaction is not satisfactory. This limitation restricts the digital human's ability to understand and respond to complex scenes, thereby affecting the user's interactive experience. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a digital human interaction method based on multimodal understanding.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A digital human interaction method based on multimodal understanding, comprising:

[0007] Acquire multimodal data, and obtain multimodal features through a feature extraction model according to the multimodal data, wherein the multimodal data includes text data, video data, and audio data;

[0008] Obtaining enhanced modal features through a knowledge enhancement model according to the multimodal features;

[0009] Obtaining a modality fusion feature vector through a graph representation learning model according to the enhanced modality features;

[0010] The large language model learns the modality fusion feature vector through a multi-head attention mechanism to obtain natural language output.

[0011] Preferably, obtaining the multimodal features through a feature extraction model according to the multimodal data includes:

[0012] Extracting features from the text data using a BERT model to obtain text features;

[0013] Extracting features from the video data using a FACET tool to obtain video features;

[0014] Extract features from the audio data using the COVAREP tool to obtain audio features;

[0015] The multimodal feature is obtained according to the text feature, the video feature and the audio feature.

[0016] Preferably, obtaining enhanced modal features through a knowledge enhancement model according to the multimodal features includes:

[0017] Acquire prior knowledge, and obtain a modal knowledge similarity matrix by similarity calculation based on the prior knowledge and the multimodal features;

[0018] Obtaining a modal knowledge similarity normalized matrix by normalizing the modal knowledge similarity matrix;

[0019] Obtaining a supplementary modal feature by weighted summation based on the normalized matrix of the prior knowledge and the modal knowledge similarity;

[0020] The enhanced modal feature is obtained by summing up the supplementary modal feature and the multimodal feature.

[0021] Preferably, obtaining a modality fusion feature vector through a graph representation learning model according to the enhanced modality feature includes:

[0022] Obtaining an intra-modal feature vector through a single-modal intra-interaction model according to the enhanced modal features;

[0023] Obtaining an inter-modal feature vector through a multi-modal interaction model according to the enhanced modal features;

[0024] The modality fusion feature vector is obtained by fusing the intra-modality feature vector and the inter-modality feature vector through a gating unit.

[0025] Preferably, obtaining the intra-modal feature vector according to the enhanced modal feature through a single-modal intra-interaction model comprises:

[0026] The enhanced modal features carry a modal type label, and enhanced modal features of the same type are obtained by classification according to the modal type label;

[0027] Obtaining similarity scores between enhanced modal features through similarity calculation according to the same type of enhanced modal features;

[0028] A dynamic neighbor node set is obtained by Gumbel Sigmoid calculation according to the similarity scores between the enhanced modal features, wherein the dynamic neighbor node set carries a selection probability label, and the selection probability label includes 0 and 1;

[0029] According to the single-mode node and the selection probability label, a single-mode enhanced modal graph is obtained by selecting the connecting edge, wherein the single-mode node is the same type of enhanced modal feature;

[0030] Obtaining an attention coefficient by attention calculation according to the dynamic neighbor node set;

[0031] The intra-modal feature vector is obtained by updating the node features according to the attention coefficient and the unimodal enhanced modal graph.

[0032] Preferably, the selection of the connection edge judgment includes judging whether the selection probability label is 1, and if so, connecting the two single-mode nodes;

[0033] If not, the connection between the two unimodal nodes is disconnected.

[0034] Preferably, obtaining the inter-modal feature vector through a multi-modal interaction model according to the enhanced modal feature includes:

[0035] The similarity score between multi-enhanced modal features is obtained by calculating the similarity between each node and its neighboring nodes in the multi-modal enhanced modal graph;

[0036] Obtaining a multi-dynamic neighbor node set by the Gumbel Sigmoid technology according to the similarity scores between the multi-enhanced modal features, wherein the multi-dynamic neighbor node set carries a multi-modal selection probability label;

[0037] According to the multimodal nodes and the multimodal selection probability labels, a multimodal connection enhanced modal graph is obtained by judging the selected connection edges, wherein the multimodal nodes are the enhanced modal features;

[0038] Obtaining a multimodal attention coefficient through the attention calculation according to the multiple dynamic neighbor node sets;

[0039] The inter-modal feature vector is obtained by updating the node features according to the multimodal attention coefficient and the multimodal connection enhanced modal graph.

[0040] A digital human interaction system based on multimodal understanding and a digital human interaction method based on multimodal understanding, comprising a data preprocessing module, a knowledge enhancement module, a fusion graph representation module, and a fusion output module, including:

[0041] The data preprocessing module is used to obtain multimodal data and obtain multimodal features through a feature extraction model according to the multimodal data, wherein the multimodal data includes text data, video data and audio data;

[0042] The knowledge enhancement module is used to obtain enhanced modal features through a knowledge enhancement model according to the multimodal features;

[0043] The fusion graph representation module is used to obtain a modality fusion feature vector through a graph representation learning model according to the enhanced modality features;

[0044] The fusion output module is used for a large language model to learn the modality fusion feature vector through a multi-head attention mechanism to obtain a natural language output.

[0045] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned digital human interaction method based on multimodal understanding when executing the program.

[0046] A storage medium containing computer executable instructions, which are used to perform the above-mentioned digital human interaction method based on multimodal understanding when executed by a computer processor.

[0047] The beneficial effects of the present invention are:

[0048] (1) By combining prior knowledge with extracted features through the knowledge enhancement model, we can supplement the missing information in multimodal data, improve the model's ability to understand complex tasks, improve the smoothness and naturalness of conversations, and thus improve the intelligence of digital humans.

[0049] (2) Inter-modal fusion is performed through the intra-modal interaction model and the inter-modal interaction model to capture the relationship between the modalities and the inter-modalities, which helps the digital human understand the multimodal data.

[0050] (3) By selecting the connection edge to adaptively select the interconnected nodes, the problem of noise and information redundancy easily introduced by the full connection method is avoided, and the efficiency and accuracy of subsequent data fusion are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0052] Figure 1 The present invention provides a digital human interaction method based on multimodal understanding. DETAILED DESCRIPTION

[0053] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0054] See also Figure 1 , a digital human interaction method based on multimodal understanding, comprising:

[0055] S1: Acquire multimodal data, the multimodal data including text data, video data and audio data, and obtain multimodal features through a feature extraction model according to the multimodal data, the multimodal features including text features, video features and audio features;

[0056] S2: Obtaining enhanced modal features through a knowledge enhancement model according to the multimodal features;

[0057] S3: Obtaining a modality fusion feature vector through a graph representation learning model according to the enhanced modality features;

[0058] S4: The large language model learns the modality fusion feature vector through a multi-head attention mechanism to obtain natural language output.

[0059] In step S1, obtaining multimodal features through a feature extraction model according to the multimodal data includes:

[0060] S101: Extracting features through a BERT model according to the text data to obtain the text features;

[0061] S102: Extracting features from the video data using a FACET tool to obtain the video features;

[0062] S103: Extracting features from the audio data using a COVAREP tool to obtain the audio features;

[0063] S104: Obtaining the multimodal feature according to the text feature, the video feature, and the audio feature;

[0064] The multimodal feature is expressed as:

[0065]

[0066] Among them, X m represents the multimodal feature, m represents the modal sequence feature, T m is the sequence length, d m is the feature dimension, represents the modal feature of the first sequence, t represents the text feature, ν represents the video feature, a represents the audio feature, and R represents a matrix.

[0067] It should be noted that, in step S101, the BERT model can extract text features; in step S102, the FACET tool can recognize and analyze facial expressions in videos, and extract facial expression, emotion and facial movement features; in step S103, the COVAREP tool is an open source acoustic feature extraction tool that can extract relevant acoustic features from audio data.

[0068] In step S2, obtaining enhanced modal features through a knowledge enhancement model according to the multimodal features includes:

[0069] S201: Acquire prior knowledge, and obtain a modal knowledge similarity matrix by similarity calculation based on the prior knowledge and the multimodal features;

[0070] The similarity calculation expression is:

[0071] A m =(W mx X m )(W me e m ) T ,

[0072] Among them, A m represents the modal knowledge similarity matrix, W mx is the parameter matrix, X m represents the multimodal features, W me is the parameter matrix, e m represents the prior knowledge, T represents the transposed matrix;

[0073] S202: obtaining a modal knowledge similarity normalized matrix by normalizing the modal knowledge similarity matrix;

[0074] The normalized expression is:

[0075]

[0076] in, represents the normalized matrix of modal knowledge similarity, softmax is the normalized exponential function, A m represents the modal knowledge similarity matrix, d m is the feature dimension;

[0077] S203: Obtaining a supplementary modal feature by weighted summation based on the prior knowledge and the modal knowledge similarity normalization matrix;

[0078] The weighted summation expression is:

[0079]

[0080] in, represents the supplementary modal features, represents the normalized matrix of modal knowledge similarity, e m represents the prior knowledge;

[0081] S204: Obtaining the enhanced modal feature by summing up the supplementary modal feature and the multimodal feature;

[0082] The summation calculation expression is:

[0083]

[0084] Among them, U m represents the enhanced modal feature, represents the supplementary modal feature, X m represents the multimodal feature.

[0085] Specifically, the graph representation learning model is based on the information aggregation mechanism of graph neural network to enhance the fusion of modal features.

[0086] Specifically, the prior knowledge is emotion information and rules, and the prior knowledge includes emotion vocabulary, emotion expression patterns, the relationship between emotion and context, and the expression forms of specific emotions in different cultures or scenarios.

[0087] In step S3, obtaining a modality fusion feature vector through a graph representation learning model according to the enhanced modality feature includes:

[0088] S301: Obtaining an intra-modal feature vector through a single-modal intra-interaction model according to the enhanced modal feature;

[0089] S302: Obtaining an inter-modal feature vector through a multi-modal interaction model according to the enhanced modal feature;

[0090] S303: Obtain the modality fusion feature vector by fusing the intra-modality feature vector and the inter-modality feature vector through a gating unit;

[0091] The gating unit fusion expression is:

[0092]

[0093] in, represents the gated output of the information interaction feature within the mth modality, represents the inter-modal eigenvector, represents the gating vector of the i-th node after mapping in the l+1 layer, represents the modality fusion feature vector, tanh represents the hyperbolic tangent function, Wl 1 , and is a trainable parameter, and is a trainable parameter, is the hidden state of the ith node in the lth layer, ⊙ is the Hadamard product calculation, represents the intra-modal eigenvector, represents the inter-modal feature vector, and sigmoid is the activation function.

[0094] Specifically, the single-modal intra-interaction model includes:

[0095] S301-1: The enhanced modal feature carries a modal type label, and the same type of enhanced modal features are obtained by classification according to the modal type label;

[0096] S301-2: obtaining similarity scores between enhanced modal features through similarity calculation according to the enhanced modal features of the same type;

[0097] The similarity calculation expression is:

[0098]

[0099] in, represents the similarity score between the enhanced modality features, W s is a trainable parameter, For Node The feature representation of W e is a trainable parameter, For Node Neighbor nodes The feature representation of

[0100] S301-3: obtaining a dynamic neighbor node set by Gumbel Sigmoid calculation according to the similarity scores between the enhanced modal features, wherein the dynamic neighbor node set carries a selection probability label, and the selection probability label includes 0 and 1;

[0101] The Gumbel Sigmoid calculation expression is:

[0102]

[0103] Among them, п(a) represents the selection probability label, e is the base function, log represents the logarithmic function, sigmoid is the activation function, represents the similarity score between the enhanced modal features, g0 represents Gumbel noise, τ represents the temperature parameter, and g1 represents Gumbel noise;

[0104] S301-4: obtaining a single-mode enhanced modal graph by selecting connecting edges according to the single-mode nodes and the selection probability labels, wherein the single-mode nodes are enhanced modal features of the same type;

[0105] The single-mode enhanced modal diagram is expressed as:

[0106]

[0107] Among them, G m represents the single-mode enhanced mode diagram, V m The set of unimodal nodes in the graph representing mode m, ε m represents an adjacency matrix in a graph of modality m, the adjacency matrix represents a connection relationship between the unimodal nodes and the unimodal nodes, m represents the mth modality, t represents the text feature, v represents the video feature, a represents the audio feature, The i-th node in the graph representing mode m;

[0108] S301-5: Obtaining an attention coefficient through attention calculation according to the dynamic neighbor node set;

[0109] The attention calculation expression is:

[0110]

[0111] in, represents the attention coefficient, Softmax is a normalized exponential function, intra represents intra-modality, l represents the lth hidden layer, j represents the jth dynamic neighbor node, is the dynamic neighbor node set, is a trainable parameter, Represents the lth hidden layer node The characteristic representation of || represents the connection operation. Representation Node Neighbor nodes The feature representation of , m represents the mth mode;

[0112] S301-6: Obtain the intra-modal feature vector by updating node features according to the attention coefficient and the unimodal enhanced modal graph;

[0113] The node feature update expression is:

[0114]

[0115] in, represents the intra-modal eigenvector, j represents the jth dynamic neighbor node, is the dynamic neighbor node set, represents the attention coefficient, Representation Node Neighbor nodes The feature representation of , m represents the mth mode.

[0116] In step S301-4, the selection of the connection edge includes determining whether the selection probability label is 1, and if so, connecting the two single-mode nodes;

[0117] If not, the connection between the two unimodal nodes is disconnected.

[0118] It should be noted that, in step S301 - 3 , the Gumbel-Sigmoid technology obtains the dynamic neighbor node set by selecting whether to connect between nodes.

[0119] Specifically, the multimodal interaction model includes:

[0120] S302-1: obtaining a similarity score between multi-enhanced modal features by calculating the similarity between each node and its neighboring nodes in the multi-modal enhanced modal graph, wherein the multi-modal enhanced modal graph includes a text enhanced modal graph, a video enhanced modal graph, and an audio enhanced modal graph;

[0121] S302-2: obtaining a multi-dynamic neighbor node set by using the Gumbel Sigmoid technology according to the similarity scores between the multi-enhanced modal features, wherein the multi-dynamic neighbor node set carries a multi-modal selection probability label;

[0122] S302-3: Obtaining a multimodal connection enhanced modal graph by judging the selected connection edge according to the multimodal node and the multimodal selection probability label, wherein the multimodal node is the enhanced modal feature;

[0123] S302-4: Obtaining a multimodal attention coefficient through the attention calculation according to the multiple dynamic neighbor node sets;

[0124] S302-5: Obtain the inter-modal feature vector through the node feature update according to the multimodal attention coefficient and the multimodal connection enhanced modal graph.

[0125] In this embodiment, the multi-modal interaction model is the information interaction between modal graphs, such as the information interaction between nodes in the text enhancement modal graph and nodes in the video enhancement modal graph and the audio enhancement modal graph; the single modal intra-interaction model is the information interaction between single modalities, including information interaction between text modalities, information interaction between video modalities, and information interaction between audio modalities.

[0126] A digital human interaction system based on multimodal understanding and a digital human interaction method based on multimodal understanding, comprising a data preprocessing module, a knowledge enhancement module, a fusion graph representation module, and a fusion output module, including:

[0127] The data preprocessing module is used to obtain multimodal data and obtain multimodal features through a feature extraction model according to the multimodal data, wherein the multimodal data includes text data, video data and audio data;

[0128] The knowledge enhancement module is used to obtain enhanced modal features through a knowledge enhancement model according to the multimodal features;

[0129] The fusion graph representation module is used to obtain a modality fusion feature vector through a graph representation learning model according to the enhanced modality features;

[0130] The fusion output module is used for a large language model to learn the modality fusion feature vector through a multi-head attention mechanism to obtain a natural language output.

[0131] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0132] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The program code contained on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0134] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A digital human interaction method based on multimodal understanding, characterized in that: The following steps are involved: Acquire multimodal data, and obtain multimodal features through a feature extraction model according to the multimodal data, wherein the multimodal data includes text data, video data, and audio data; Obtaining enhanced modal features through a knowledge enhancement model according to the multimodal features; Obtaining a modality fusion feature vector through a graph representation learning model according to the enhanced modality features; The large language model learns the modality fusion feature vector through a multi-head attention mechanism to obtain natural language output.

2. The digital human interaction method based on multimodal understanding according to claim 1 is characterized in that: The step of obtaining multimodal features through a feature extraction model according to the multimodal data includes: Extracting features from the text data using a BERT model to obtain text features; Extracting features from the video data using a FACET tool to obtain video features; Extract features from the audio data using the COVAREP tool to obtain audio features; The multimodal feature is obtained according to the text feature, the video feature and the audio feature.

3. The digital human interaction method based on multimodal understanding according to claim 1, characterized in that: The step of obtaining enhanced modal features through a knowledge enhancement model according to the multimodal features includes: Acquire prior knowledge, and obtain a modal knowledge similarity matrix by similarity calculation based on the prior knowledge and the multimodal features; Obtaining a modal knowledge similarity normalized matrix by normalizing the modal knowledge similarity matrix; Obtaining a supplementary modal feature by weighted summation based on the normalized matrix of the prior knowledge and the modal knowledge similarity; The enhanced modal feature is obtained by summing up the supplementary modal feature and the multimodal feature.

4. The digital human interaction method based on multimodal understanding according to claim 1, characterized in that: The step of obtaining a modality fusion feature vector through a graph representation learning model according to the enhanced modality feature includes: Obtaining an intra-modal feature vector through a single-modal intra-interaction model according to the enhanced modal features; Obtaining an inter-modal feature vector through a multi-modal interaction model according to the enhanced modal features; The modality fusion feature vector is obtained by fusing the intra-modality feature vector and the inter-modality feature vector through a gating unit.

5. The digital human interaction method based on multimodal understanding according to claim 4 is characterized in that: The step of obtaining an intra-modal feature vector according to the enhanced modal feature through a single-modal intra-interaction model comprises: The enhanced modal features carry a modal type label, and enhanced modal features of the same type are obtained by classification according to the modal type label; Obtaining similarity scores between enhanced modal features through similarity calculation according to the same type of enhanced modal features; A dynamic neighbor node set is obtained by GumbelSigmoid calculation according to the similarity score between the enhanced modal features, wherein the dynamic neighbor node set carries a selection probability label, and the selection probability label includes 0 and 1; According to the single-mode node and the selection probability label, a single-mode enhanced modal graph is obtained by selecting the connecting edge, wherein the single-mode node is the same type of enhanced modal feature; Obtaining an attention coefficient by attention calculation according to the dynamic neighbor node set; The intra-modal feature vector is obtained by updating the node features according to the attention coefficient and the unimodal enhanced modal graph.

6. The digital human interaction method based on multimodal understanding according to claim 5 is characterized in that: The selection of the connection edge judgment includes judging whether the selection probability label is 1, and if so, connecting the two single-mode nodes; If not, the connection between the two unimodal nodes is disconnected.

7. The digital human interaction method based on multimodal understanding according to claim 4 is characterized in that: The step of obtaining an inter-modal feature vector through a multi-modal interaction model according to the enhanced modal feature includes: The similarity score between multi-enhanced modal features is obtained by calculating the similarity between each node and its neighboring nodes in the multi-modal enhanced modal graph; Obtaining a multi-dynamic neighbor node set by the GumbelSigmoid technology according to the similarity scores between the multi-enhanced modal features, wherein the multi-dynamic neighbor node set carries a multi-modal selection probability label; According to the multimodal nodes and the multimodal selection probability labels, a multimodal connection enhanced modal graph is obtained by judging the selected connection edges, wherein the multimodal nodes are the enhanced modal features; Obtaining a multimodal attention coefficient through the attention calculation according to the multiple dynamic neighbor node sets; The inter-modal feature vector is obtained by updating the node features according to the multimodal attention coefficient and the multimodal connection enhanced modal graph.

8. A digital human interaction system based on multimodal understanding using the digital human interaction method based on multimodal understanding as claimed in claim 1, comprising a data preprocessing module, a knowledge enhancement module, a fusion graph representation module, and a fusion output module, characterized in that: include: The data preprocessing module is used to obtain multimodal data and obtain multimodal features through a feature extraction model according to the multimodal data, wherein the multimodal data includes text data, video data and audio data; The knowledge enhancement module is used to obtain enhanced modal features through a knowledge enhancement model according to the multimodal features; The fusion graph representation module is used to obtain a modality fusion feature vector through a graph representation learning model according to the enhanced modality features; The fusion output module is used for a large language model to learn the modality fusion feature vector through a multi-head attention mechanism to obtain a natural language output.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the digital human interaction method based on multimodal understanding as described in any one of claims 1-7 is implemented.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the digital human interaction method based on multimodal understanding as described in any one of claims 1-7 when executed by a computer processor.

Images