A digital human interaction method and system based on multimodal large model

Abstract

The present invention discloses a digital human interaction method and system based on a multimodal large model, which relates to the field of digital human interaction technology. The method includes: collecting multimodal input data, performing feature extraction and time synchronization; using a preset multimodal large model to encode the input data of each mode, mapping the encoding result to a high-dimensional space, forming multiple modal feature vectors, performing weight distribution, fusing to generate a first interaction vector, extracting the intention feature representation, calculating and analyzing to obtain the user intention, and constructing a semantic association network; performing a graph convolution operation on the semantic association network, extracting logical relationship features, generating digital human interaction content based on the intention feature representation and the logical relationship features, and delivering it to the user through an output interface. By extracting features from multimodal input data, generating multimodal feature vectors and using semantic mapping technology, the fusion accuracy of multimodal data, the semantic understanding ability, and the accuracy and diversity of interactive responses are improved.

Description

Technical Field

[0001] The present invention relates to the field of digital human interaction technology, and in particular to a digital human interaction method and system based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence and big data technology, digital human has gradually become an important research direction in the field of human-computer interaction. In the prior art, digital human interaction systems are mostly based on single-modal data input (such as voice, image or text), and they have the following deficiencies in semantic understanding and multimodal data fusion: the single modality leads to limited semantic understanding ability. In the prior art, digital human systems mainly rely on voice recognition or image recognition technology to achieve human-computer interaction. Due to the lack of fusion processing ability of multimodal data, the system cannot accurately understand complex semantic expressions, especially when it comes to context association or multi-dimensional information interaction, it shows obvious limitations; the collaborative processing efficiency between modal data is low. For the collaborative processing of multimodal input data (such as voice, image and text), the prior art usually adopts an independent processing mode, lacks a mechanism for unified modeling and synchronous processing of multimodal data, resulting in insufficient data processing delay and accuracy; lacks a flexible semantic relationship construction mechanism. The representation and mining of semantic relationships by existing digital human interaction systems are mostly based on predefined rules, which are difficult to dynamically adapt to user needs and context changes, and cannot meet the needs of high-degree-of-freedom and diversified interaction scenarios. Summary of the invention

[0003] Based on the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a digital human interaction method and system based on a multimodal large model to solve the above-mentioned technical problems.

[0004] To achieve the above object, the present invention provides the following technical solution: a digital human interaction method based on a multimodal large model, comprising:

[0005] Collect multimodal input data, perform denoising, normalization and feature extraction on the collected multimodal input data, and perform time synchronization processing on input data of different modes;

[0006] Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results into a high-dimensional space to form multiple modal feature vectors;

[0007] The modal feature vectors of each modality are weighted, and the multi-modal feature vectors are fused based on a preset collaborative perception projection algorithm to generate a unified first interaction vector;

[0008] Extracting an intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain a user intention, and constructing a semantic association network according to the user intention;

[0009] A graph convolution operation is performed on the semantic association network to extract logical relationship features, and digital human interaction content is generated based on the intention feature representation and logical relationship features, and is delivered to the user through an output interface.

[0010] The present invention is further configured such that the multimodal input data includes text data, voice data and image data; the input data of each modality is encoded using a preset multimodal macro model, and the encoding results are uniformly mapped to a high-dimensional space to form a modal feature vector, including:

[0011] Performing word segmentation processing on the text data to generate a word segmentation sequence, performing feature extraction on the word segmentation sequence to generate context features for each word, performing sequence integration on word-level features to generate a coding feature representation of the text modality;

[0012] Performing a short-time Fourier transform on the speech data to generate a spectrogram representation in the time-frequency domain, performing multi-layer feature extraction on the spectrogram representation using a two-dimensional convolutional neural network to generate a coded feature representation of the speech modality;

[0013] Performing standardization preprocessing on the image data to normalize pixel values, performing hierarchical feature extraction on the normalized image data using a convolutional neural network, extracting the global feature vector of the last layer, and generating a coded feature representation of the image modality;

[0014] The encoded feature representation of each modality is mapped to a unified high-dimensional feature space, and the mapped features are standardized to form multiple modality feature vectors.

[0015] The present invention is further configured to perform weight distribution on the modal feature vectors of each modality, fuse the multi-modal feature vectors based on a preset collaborative perception projection algorithm, and generate a unified first interaction vector, including:

[0016] Based on multiple modal feature vectors, calculate the attention weights between modalities;

[0017] According to the attention weights between the modalities, the multimodal feature vectors are fused based on a preset collaborative perception projection algorithm to generate a unified first interaction vector.

[0018] The present invention is further configured such that the calculation logic of the attention weight between modalities is: Among them, α N is the modal eigenvector F N The attention weights, N∈{T,I,S}, F T is the text modality feature vector, F I is the speech modality feature vector, F Sis the image modality feature vector, ψ(·) is the modality attention score function, and K is the key vector matrix in the high-dimensional feature space;

[0019] The generation logic of the first interaction vector is: V = ∑ N∈{T,I,S} α N ·F N +λ·Ω(F T ,F I ,F S ), where V is the first interaction vector, λ is the balance coefficient, Ω(F T ,F I ,F S ) is the collaborative correlation function between modal eigenvectors. The calculation logic includes: calculating any two modal eigenvectors F M and F N The synergistic correlation coefficient, Among them, FR M,N is the modal eigenvector F M and F N The synergistic correlation coefficient, W R is the linear transformation matrix of the modal eigenvector, which performs weighted aggregation of the synergistic correlation coefficients of all modal eigenvector pairs.

[0020] The present invention is further configured to extract an intention feature representation according to the first interaction vector, obtain a user intention according to calculation and analysis of the intention feature representation, and construct a semantic association network according to the user intention, including:

[0021] Performing nonlinear mapping on the first interaction vector to extract intention feature representation in a high-dimensional space;

[0022] Calculate the probability distribution of predefined intent categories according to the intent feature representation, and set the intent corresponding to the intent category with the largest probability distribution as the user intent;

[0023] Context features related to the user intention are extracted according to the user intention, a semantic association matrix of the context features is established, and a semantic association network is constructed.

[0024] The present invention is further configured such that the extraction logic of the intention feature representation is: Z = φ I (W I ·V+b I ), where Z is the feature representation of intent, φ I (·) is a nonlinear activation function, W I is the weight matrix of the intent feature representation mapping, which is used to project the first interaction vector into the intent space, V is the first interaction vector, and b I is a bias term used to increase the expressive power of nonlinear mapping;

[0025] The calculation logic of the probability distribution of intent categories is: k = 1, 2, ..., K, where K is the number of intent categories, P(C k |Z) is the probability of the intent feature being represented in the Kth intent category, W k is the classification weight of the Kth intent category, b k is the bias term for the Kth intent category;

[0026] The calculation logic of the semantic association matrix WR of context features is: Among them, WR ij is the context feature C i and C j The semantic relevance of W is the semantic association matrix;

[0027] The semantic association network G=(Node,E), Among them, Node is the set of context nodes, each node represents a context feature, E is the semantic association edge between context nodes, and δ is the threshold of semantic association, which is used to control the sparsity of the association network.

[0028] The present invention is further configured to perform a graph convolution operation on the semantic association network to extract logical relationship features, including:

[0029] The context feature nodes are updated according to the context features and the normalized semantic association matrix to obtain a node feature matrix;

[0030] By stacking graph convolutional layers, high-order semantic associations between nodes are embedded into the final feature matrix;

[0031] All context feature nodes in the final feature matrix are globally aggregated to generate global logical relationship features.

[0032] The present invention is further configured to construct a node feature matrix according to context features, the node feature matrix H = [C1, C2, ..., C L ], L is the number of contextual features;

[0033] Normalizing the semantic association matrix, in, is the normalized semantic association matrix, I is the identity matrix, D is the degree matrix, and D ii =∑ j WR ij ;

[0034] The logic for updating the context feature node is: Among them, H (l+1)is the node feature matrix of the l+1th layer, the initial H (0) =H, is the graph convolution weight matrix, is the bias term, φ G (·) is a nonlinear activation function;

[0035] By stacking l layers of graph convolution, the high-order semantic associations between nodes are embedded into the final feature matrix H (l+1) ;

[0036] The calculation logic of global aggregation is: Among them, L G is the global logical relationship feature, is the final feature matrix H (l+1) The i-th context feature in φ P (·) is the global activation function, α i is the attention weight of the i-th context feature, and the calculation logic is: W A is the attention weight matrix.

[0037] The present invention is further configured to generate digital human interaction content according to the intention feature representation and the logical relationship feature, including:

[0038] generating a second interaction vector according to the intention feature representation and the logical relationship feature;

[0039] The second interaction vector is semantically mapped to generate digital human interaction content, wherein the semantic mapping is to calculate the cosine similarity between the second interaction vector and the semantic interaction vector corresponding to the specific semantics, and set the semantics with the largest cosine similarity as the semantic mapping content.

[0040] The present invention also provides a digital human interaction system based on a multimodal large model, the system comprising:

[0041] Data acquisition module: collects multimodal input data, performs denoising, normalization and feature extraction on the collected multimodal input data, and performs time synchronization processing on input data of different modes;

[0042] Feature generation module: Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results into a high-dimensional space to form multiple modal feature vectors;

[0043] Interaction generation module: weights the modal feature vectors of each modality, fuses the multi-modal feature vectors based on the preset collaborative perception projection algorithm, and generates a unified first interaction vector;

[0044] Network construction module: extracting the intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain the user intention, and constructing a semantic association network according to the user intention;

[0045] Content generation module: Perform graph convolution operation on the semantic association network, extract logical relationship features, generate digital human interaction content based on the intention feature representation and logical relationship features, and transmit it to the user through the output interface.

[0046] The present invention provides a digital human interaction method and system based on a multimodal large model. The method collects multimodal input data, performs denoising, normalization and feature extraction on the collected multimodal input data, and performs time synchronization processing on input data of different modes; uses a preset multimodal large model to encode the input data of each mode, and uniformly maps the encoding results to a high-dimensional space to form multiple modal feature vectors; performs weight allocation on the modal feature vectors of each mode, fuses the multimodal feature vectors based on a preset collaborative perception projection algorithm, and generates a unified first interaction vector; extracts an intention feature representation according to the first interaction vector, calculates and analyzes the intention feature representation to obtain a user intention, and constructs a semantic association network according to the user intention; performs a graph convolution operation on the semantic association network to extract logical relationship features, generates digital human interaction content according to the intention feature representation and the logical relationship features, and transmits it to the user through an output interface, and the beneficial effects produced include:

[0047] 1. Improve the fusion processing capability of multimodal data: Through unified preprocessing and feature extraction of multimodal input data such as voice, image and text, combined with encoding mapping and feature vector generation technology, the synchronous processing and feature alignment of modal data are realized, the accuracy of multimodal data fusion is improved, and the real-time and consistency of interactive data processing are ensured;

[0048] 2. Optimize dynamic modeling and representation of semantic relationships: Through the semantic relationship network based on the attention mechanism, by weighting the multimodal features, dynamically build a semantic context association network, and support semantic feature screening and updating based on user intentions, it can effectively mine and express the semantic associations between multimodal data, significantly improving the system's semantic understanding ability;

[0049] 3. Improve the accuracy and diversity of digital human interaction responses: Generate a unified semantic interaction vector through multimodal feature vectors, and generate interactive content that is highly consistent with user needs through dynamic semantic mapping and context feature fusion technology, making the interactive response more accurate and supporting personalized and scenario-based interaction forms.

[0050] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0052] Figure 1 A flowchart of a digital human interaction method based on a multimodal large model is shown as an exemplary embodiment of the present invention;

[0053] Figure 2 The figure is a schematic structural diagram of a digital human interaction system based on a multimodal large model, showing an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0055] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0056] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0057] Embodiment 1

[0058] A digital human interaction method based on a multimodal large model, such as Figure 1 As shown, including:

[0059] Collect multimodal input data, perform denoising, normalization and feature extraction on the collected multimodal input data, and perform time synchronization processing on input data of different modes;

[0060] Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results into a high-dimensional space to form multiple modal feature vectors;

[0061] The modal feature vectors of each modality are weighted, and the multi-modal feature vectors are fused based on a preset collaborative perception projection algorithm to generate a unified first interaction vector;

[0062] Extracting an intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain a user intention, and constructing a semantic association network according to the user intention;

[0063] A graph convolution operation is performed on the semantic association network to extract logical relationship features, and digital human interaction content is generated based on the intention feature representation and logical relationship features, and is delivered to the user through an output interface.

[0064] The present invention is further configured such that the multimodal input data includes text data, voice data and image data; the input data of each modality is encoded using a preset multimodal macro model, and the encoding results are uniformly mapped to a high-dimensional space to form a modal feature vector, including:

[0065] The text data is segmented to generate a segmentation sequence, and features are extracted from the segmentation sequence to generate context features for each word. The word-level features are sequentially integrated to generate a coding feature representation of the text modality. Specifically, the segmentation process is implemented by using a forward maximum matching or dictionary-based statistical segmentation algorithm, which is a prior art and the specific implementation method is not limited here. A pre-trained language model (such as BERT) is used to generate a context feature representation for each segmentation in the segmentation sequence. The pre-trained language model is a deep neural network model pre-trained through self-supervised learning of massive text data, and provides a general semantic representation or generation capability in language understanding and generation tasks. BERT is a bidirectional language model based on the Transformer encoder architecture, which captures a deeper semantic relationship between words in a sentence by jointly modeling the context. The context features of all words are integrated using a sequence aggregation method (including self-attention or maximum pooling) to generate a global representation of the text modality.

[0066] The speech data is subjected to a short-time Fourier transform to generate a spectrogram representation in the time-frequency domain, and a two-dimensional convolutional neural network is used to perform multi-layer feature extraction on the spectrogram representation to generate a coding feature representation of the speech modality; specifically, the speech data is subjected to a short-time Fourier transform to generate a spectrogram representation in the time-frequency domain: S(t) is the time domain speech signal, f is the frequency, t is the time; the spectrogram X S (f, t) is input into a two-dimensional convolutional network, and the features of the speech modality are extracted layer by layer. This is a prior art and will not be described in detail here. The global features of the last layer of the convolutional network are extracted as the encoding feature representation of the speech modality.

[0067] The image data is subjected to standardization preprocessing to normalize the pixel values, and the normalized image data is subjected to hierarchical feature extraction using a convolutional neural network, and the global feature vector of the last layer is extracted to generate a coding feature representation of the image modality; specifically, the image data I is input, and the image data is standardized and normalized to adjust the pixel value to the range of [0, 1] to enhance the robustness of the model to the data, and the standardized image data is input to the convolutional neural network to extract spatial features layer by layer. This is a prior art and will not be described in detail here. The global feature vector is extracted from the last layer as the coding feature representation of the image modality;

[0068] The encoded feature representation of each modality is mapped to a unified high-dimensional feature space, and the mapped features are standardized to form multiple modal feature vectors. Specifically, a linear mapping is used to project the encoded feature representation of each modality to a unified high-dimensional feature space; the high-dimensional mapping unifies the feature dimensions of different modalities, and the standardization eliminates the scale differences between modalities, laying the foundation for multi-modal feature fusion.

[0069] The present invention is further configured to perform weight distribution on the modal feature vectors of each modality, fuse the multi-modal feature vectors based on a preset collaborative perception projection algorithm, and generate a unified first interaction vector, including:

[0070] Based on multiple modal feature vectors, the attention weights between modalities are calculated; the present invention is further configured such that the calculation logic of the attention weights between modalities is: Among them, α N is the modal eigenvector F N The attention weights, N∈{T,I,S}, F T is the text modality feature vector, F I is the speech modality feature vector, F Sis the image modal feature vector, ψ(·) is the modal attention score function, and K is the key vector matrix of the high-dimensional feature space; specifically, the above-mentioned calculation logic obtains the attention weight between the modalities by calculating the correlation between each modal feature vector and the high-dimensional key vector matrix K. The calculation logic of the modal attention score function ψ(·) is determined by the specific implementation. In a feasible embodiment of the present invention, it is a dot product or nonlinear transformation in the high-dimensional feature space; the key vector matrix K of the high-dimensional feature space is used to capture the correlation between the modal features and the task, and can be initialized by task-related prior knowledge, or generated by adaptive training;

[0071] According to the attention weights between the modalities, the multimodal feature vectors are fused based on a preset collaborative perception projection algorithm to generate a unified first interaction vector; the present invention is further configured such that the generation logic of the first interaction vector is: V = Σ N∈{T,I,S} α N ·F N +λ·Ω(F T ,F I ,F S ), where V is the first interaction vector, λ is the balance coefficient, Ω(F T ,F I ,F S ) is the collaborative correlation function between modal eigenvectors. The calculation logic includes: calculating any two modal eigenvectors F M and F N The synergistic correlation coefficient, Among them, FR M,N is the modal eigenvector F M and F N The synergistic correlation coefficient, W R is the linear transformation matrix of the modal eigenvector, which performs weighted aggregation of the synergistic correlation coefficients of all modal eigenvector pairs. Specifically, the above calculation logic generates a unified interaction vector by combining the attention weights of the modal features and the collaborative correlation between the modalities. The calculation logic adopts the modal attention mechanism and the collaborative perception projection method. Through linear transformation and correlation calculation, the collaborative relationship and semantic association between different modal features are captured. The balance coefficient λ is used to adjust the weights of the modal feature independence (the first item) and the inter-modal synergy (the second item). The value is greater than 0 and is determined by the hyperparameter optimization method. The linear transformation matrix W of the modal feature vector R The modal feature vector is projected into the same semantic space, with the same dimension as the modal feature vector. It is usually obtained through training and learning. The inter-modal collaborative correlation coefficient FR M,N Used to measure the mode F M and F NThe correlation in the high-dimensional feature space and the inter-modal collaborative correlation function represent the interaction and fusion characteristics between modal features. The collaborative correlation coefficient and collaborative correlation function effectively capture the interaction characteristics between modalities and improve the overall performance of multimodal fusion.

[0072] The present invention is further configured to extract an intention feature representation according to the first interaction vector, obtain a user intention according to calculation and analysis of the intention feature representation, and construct a semantic association network according to the user intention, including:

[0073] The first interaction vector is nonlinearly mapped to extract the intention feature representation in the high-dimensional space; the present invention is further configured such that the extraction logic of the intention feature representation is: Z = φ I (W I ·V+b I ), where Z is the feature representation of intent, φ I (·) is a nonlinear activation function, W I is the weight matrix of the intent feature representation mapping, which is used to project the first interaction vector into the intent space, V is the first interaction vector, and b I is a bias term used to increase the expressive power of nonlinear mapping; specifically, the above logic extracts the intent feature representation Z from the first interaction vector V through nonlinear mapping. Through linear transformation and nonlinear activation of high-dimensional space, the mapping from interaction features to intent space is realized, the purpose is to enhance the model's ability to understand the semantics of interaction features and extract intent features related to the task; first, the first interaction vector V is combined with the weight matrix W I Multiply, project to the intention space through linear transformation, and add bias b to the projection result I , bias b I The dimension is consistent with the intention space Z, and a nonlinear activation function is applied to the bias-adjusted features. The activation function φ I (·) can be ReLU, GELU or Sigmoid, etc. The specific choice depends on the task requirements and is not specifically restricted here;

[0074] The probability distribution of the predefined intent categories is calculated according to the intent feature representation, and the intent corresponding to the intent category with the largest probability distribution is set as the user intent; the present invention is further configured such that the calculation logic of the probability distribution of the intent category is: k = 1, 2, ..., K, where K is the number of intent categories, P(C k |Z) is the probability of the intent feature being represented in the Kth intent category, W k is the classification weight of the Kth intent category, b k is the bias term of the Kth intent category; specifically, by performing linear transformation and Softmax operation on the intent feature vector Z, each intent category C is obtained.k The probability distribution of the user intent is finally set as the category with the largest probability value. The goal is to map high-dimensional features to the predefined intent category space and use the probability distribution to quantify the possibility of each category, thereby realizing the recognition of user intent. The Softmax operation converts the score into a probability distribution, making the classification of user intent more intuitive and efficient.

[0075] According to the user intention, context features related to the intention are extracted, a semantic association matrix of the context features is established, and a semantic association network is constructed. The present invention is further configured such that the calculation logic of the semantic association matrix WR of the context features is: Among them, WR ij is the context feature C i and C j The semantic relevance of W is a semantic association matrix; the semantic association network Among them, Node is a set of context nodes, each node represents a context feature, E is the semantic association edge between context nodes, δ is the threshold of semantic association, which is used to control the sparsity of the association network; specifically, the above logic describes the process of building a semantic association network from context features. The core is to use the feature similarity matrix WR to generate a semantic association matrix and build a semantic association network G = (Node, E). By defining the relationship between nodes (context features), a network structure is further formed to capture the semantic connection between features. The semantic association matrix W W The weight matrix used to linearly transform the context features maps the context features to a unified semantic space and eliminates the scale differences between the original features. The threshold δ of semantic association is used to control the threshold of the sparsity of the semantic association network, adjust the number of edges in the network, and balance information richness and computational complexity. After constructing the semantic association network, the features can be further aggregated and extracted through the graph structure to improve the model's utilization of context information.

[0076] The present invention is further configured to perform a graph convolution operation on the semantic association network to extract logical relationship features, including:

[0077] The context feature nodes are updated according to the context features and the normalized semantic association matrix to obtain a node feature matrix;

[0078] By stacking graph convolution layers, the high-order semantic associations between nodes are embedded into the final feature matrix; the present invention is further configured to construct a node feature matrix according to context features, the node feature matrix H = [C1, C2, ..., C L ], L is the number of context features; the semantic association matrix is ​​normalized, in, is the normalized semantic association matrix, I is the identity matrix, D is the degree matrix, and D ii =∑ j WR ij ; The logic for updating the context feature node is: Among them, H (l+1) is the node feature matrix of the l+1th layer, the initial H (0) =H, is the graph convolution weight matrix, is the bias term, φ G (·) is a nonlinear activation function; by stacking l layers of graph convolution, the high-order semantic associations between nodes are embedded into the final feature matrix H (l+1) ; Specifically, the above logic describes the method of performing graph convolution operations on the semantic association network, by normalizing the semantic association matrix And stacked convolution operations to extract the high-order semantic relationship between context features and generate the final node feature matrix H (l+1) The core goal is to capture the global semantic information of nodes in the semantic association network through the graph convolution model; in each layer of graph convolution, the node feature matrix H (l) Through the normalized semantic association matrix Perform feature aggregation and update it to the next layer node feature matrix H through linear transformation and nonlinear activation (l+1) , graph convolution weight matrix It is the weight matrix of the graph convolution, which represents the feature transformation parameters of the current convolution layer. The output of the last layer is set to the final feature matrix H (l+1) ; Graph convolution can gradually aggregate high-order semantic relationships through multi-layer operations to achieve the expression of global semantic information.

[0079] All context feature nodes in the final feature matrix are globally aggregated to generate global logical relationship features. The present invention is further configured such that the calculation logic of global aggregation is: Among them, L G is the global logical relationship feature, is the final feature matrix H (l+1) The i-th context feature in φ P (·) is the global activation function, α i is the attention weight of the i-th context feature, and the calculation logic is: W A is the attention weight matrix; specifically, the above logic describes the method of globally aggregating all context feature nodes in the final feature matrix, with the goal of generating a global logical feature L G, which is used to represent the weighted integration result of all node features. Its core is based on the attention weight α i Node Features Perform weighted summation and pass the global activation function φ P (·) Introducing nonlinearity to improve the expression ability of global features, the attention weight matrix W A The matrix used to linearly transform node features and determine the distribution of attention weights is used to optimize the importance of node features in global semantic expression through learning. It is initialized to random values ​​and optimized through training; the global activation function φ P (·) Select according to task requirements, including ReLU: suitable for sparse representation; GELU: smooth and stable activation; Tanh: suitable for smooth transition. No specific restrictions are made here.

[0080] The present invention is further configured to generate digital human interaction content according to the intention feature representation and the logical relationship feature, including:

[0081] Generate a second interaction vector based on the intention feature representation and the logical relationship feature; specifically, V2=φ2(Q·Z+R·L G +b2), the second interaction vector, which integrates the information of intention features and logical relationship features, is used as the input of semantic mapping, and is represented by linear combination of intention feature representation Z and logical relationship feature L G , we introduce nonlinear activation to obtain, Q·Z is the linear transformation of the intent feature representation Z, which maps the intent feature into the semantic space to capture task-related information, R·L G is the logical relation feature L G The linear transformation of the logical feature is performed, the structural information of the logical feature is integrated with the intention feature to improve the expressiveness of the feature. The bias term b2 adjusts the distribution of the feature and enhances the nonlinear expressiveness of the model. The nonlinear activation function φ2(·) improves the semantic expressiveness of the vector through nonlinear mapping. By integrating the intention feature and the logical feature, the second interaction vector can comprehensively express the semantics and contextual logic of the user input.

[0082] The second interaction vector is semantically mapped to generate digital human interaction content, wherein the semantic mapping is to calculate the cosine similarity between the second interaction vector and the semantic interaction vector corresponding to the specific semantics, and set the semantic with the largest cosine similarity as the semantic mapping content; specifically, the above logic includes two main steps: generating the second interaction vector and semantically mapping the second interaction vector to generate digital human interaction content, the purpose is to dynamically generate the interaction content between the digital human and the user based on the intention feature and the logical relationship feature, quantify the matching degree between the second interaction vector and the specific semantics by calculating the cosine similarity between V2 and each semantic interaction vector, select the semantic corresponding to the semantic interaction vector with the largest cosine similarity as the final semantic mapping content, and the semantic mapping mechanism based on cosine similarity can dynamically adapt to user input and efficiently match it with predefined semantic content.

[0083] Embodiment 2

[0084] See also Figure 2 , the exemplary digital human interaction system based on multimodal large model includes:

[0085] Data acquisition module: collects multimodal input data, performs denoising, normalization and feature extraction on the collected multimodal input data, and performs time synchronization processing on input data of different modes;

[0086] Feature generation module: Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results into a high-dimensional space to form multiple modal feature vectors;

[0087] Interaction generation module: weights the modal feature vectors of each modality, fuses the multi-modal feature vectors based on the preset collaborative perception projection algorithm, and generates a unified first interaction vector;

[0088] Network construction module: extracting the intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain the user intention, and constructing a semantic association network according to the user intention;

[0089] Content generation module: Perform graph convolution operation on the semantic association network, extract logical relationship features, generate digital human interaction content based on the intention feature representation and logical relationship features, and transmit it to the user through the output interface.

[0090] It should be noted that the digital human interaction system based on a multimodal large model provided in the above embodiment and the digital human interaction method based on a multimodal large model provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In practical applications, the digital human interaction system based on a multimodal large model provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0091] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0092] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0093] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0094] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0101] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A digital human interaction method based on a multimodal large model, characterized in that: include: Collect multimodal input data, perform denoising, normalization and feature extraction on the collected multimodal input data, and perform time synchronization processing on input data of different modes; Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results into a high-dimensional space to form multiple modal feature vectors; The modal feature vectors of each modality are weighted, and the multi-modal feature vectors are fused based on a preset collaborative perception projection algorithm to generate a unified first interaction vector; Extracting an intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain a user intention, and constructing a semantic association network according to the user intention; A graph convolution operation is performed on the semantic association network to extract logical relationship features, and digital human interaction content is generated based on the intention feature representation and logical relationship features, and is delivered to the user through an output interface.

2. A digital human interaction method based on a multimodal large model according to claim 1, characterized in that: The multimodal input data includes text data, voice data and image data; Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results to a high-dimensional space to form a modal feature vector, including: Performing word segmentation processing on the text data to generate a word segmentation sequence, performing feature extraction on the word segmentation sequence to generate context features for each word, performing sequence integration on word-level features to generate a coding feature representation of the text modality; Performing a short-time Fourier transform on the speech data to generate a spectrogram representation in the time-frequency domain, performing multi-layer feature extraction on the spectrogram representation using a two-dimensional convolutional neural network to generate a coded feature representation of the speech modality; Performing standardization preprocessing on the image data to normalize pixel values, performing hierarchical feature extraction on the normalized image data using a convolutional neural network, extracting the global feature vector of the last layer, and generating a coded feature representation of the image modality; The encoded feature representation of each modality is mapped to a unified high-dimensional feature space, and the mapped features are standardized to form multiple modality feature vectors.

3. The digital human interaction method based on a multimodal large model according to claim 1, characterized in that: The modal feature vectors of each modality are weighted, and the multi-modal feature vectors are fused based on the preset collaborative perception projection algorithm to generate a unified first interaction vector, including: Based on multiple modal feature vectors, calculate the attention weights between modalities; According to the attention weights between the modalities, the multimodal feature vectors are fused based on a preset collaborative perception projection algorithm to generate a unified first interaction vector.

4. A digital human interaction method based on a multimodal large model according to claim 3, characterized in that: The calculation logic of attention weights between modalities is: Among them, α N is the modal eigenvector F N The attention weights, N∈{T,I,S}, F T is the text modality feature vector, F I is the speech modality feature vector, F S is the image modality feature vector, ψ(·) is the modality attention score function, and K is the key vector matrix in the high-dimensional feature space; The generation logic of the first interaction vector is: V = ∑ N∈{T,I,S} α N ·F N +λ·Ω(F T ,F I ,F S ), where V is the first interaction vector, λ is the balance coefficient, Ω(F T ,F I ,F S ) is the collaborative correlation function between modal eigenvectors. The calculation logic includes: calculating any two modal eigenvectors F M and F N The synergistic correlation coefficient, Among them, FR M,N is the modal eigenvector F M and F N The synergistic correlation coefficient, W R is the linear transformation matrix of the modal eigenvector, which performs weighted aggregation of the synergistic correlation coefficients of all modal eigenvector pairs.

5. The digital human interaction method based on a multimodal large model according to claim 1, characterized in that: Extracting an intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain a user intention, and constructing a semantic association network according to the user intention, including: Performing nonlinear mapping on the first interaction vector to extract intention feature representation in a high-dimensional space; Calculate the probability distribution of predefined intent categories according to the intent feature representation, and set the intent corresponding to the intent category with the largest probability distribution as the user intent; Context features related to the user intention are extracted according to the user intention, a semantic association matrix of the context features is established, and a semantic association network is constructed.

6. A digital human interaction method based on a multimodal large model according to claim 5, characterized in that: The extraction logic of intent feature representation is: Z = φ I (W I ·V+b I ), where Z is the feature representation of intent, φ I (·) is a nonlinear activation function, W I is the weight matrix of the intent feature representation mapping, which is used to project the first interaction vector into the intent space, V is the first interaction vector, and b I is a bias term used to increase the expressive power of nonlinear mapping; The calculation logic of the probability distribution of intent categories is: k = 1, 2, ..., K, where K is the number of intent categories, P(C k |Z) is the probability of the intent feature being represented in the Kth intent category, W k is the classification weight of the Kth intent category, b k is the bias term for the Kth intent category; The calculation logic of the semantic association matrix WR of context features is: Among them, WR ij is the context feature C i and C j The semantic relevance of W is the semantic association matrix; The semantic association network G=(Node,E), Among them, Node is the set of context nodes, each node represents a context feature, E is the semantic association edge between context nodes, and δ is the threshold of semantic association, which is used to control the sparsity of the association network.

7. The digital human interaction method based on a multimodal large model according to claim 5, characterized in that: Performing graph convolution operation on the semantic association network to extract logical relationship features includes: The context feature nodes are updated according to the context features and the normalized semantic association matrix to obtain a node feature matrix; By stacking graph convolutional layers, high-order semantic associations between nodes are embedded into the final feature matrix; All context feature nodes in the final feature matrix are globally aggregated to generate global logical relationship features.

8. The digital human interaction method based on a multimodal large model according to claim 7, characterized in that: Construct a node feature matrix based on context features, node feature matrix H = [C1, C2, ..., C L ], L is the number of contextual features; Normalizing the semantic association matrix, in, is the normalized semantic association matrix, I is the identity matrix, D is the degree matrix, and D ii =Σ j WR ij ; The logic for updating the context feature node is: Among them, H (l+1) is the node feature matrix of the l+1th layer, the initial H (0) =H, is the graph convolution weight matrix, is the bias term, φ G (·) is a nonlinear activation function; By stacking l layers of graph convolution, the high-order semantic associations between nodes are embedded into the final feature matrix H (l+1) ; The calculation logic of global aggregation is: Among them, L G is the global logical relationship feature, is the final feature matrix H (l+1) The i-th context feature in φ P (·) is the global activation function, α i is the attention weight of the i-th context feature, and the calculation logic is: W A is the attention weight matrix.

9. The digital human interaction method based on a multimodal large model according to claim 1, characterized in that: According to the intention feature representation and the logical relationship feature, the digital human interaction content is generated, including: generating a second interaction vector according to the intention feature representation and the logical relationship feature; The second interaction vector is semantically mapped to generate digital human interaction content, wherein the semantic mapping is to calculate the cosine similarity between the second interaction vector and the semantic interaction vector corresponding to the specific semantics, and set the semantics with the largest cosine similarity as the semantic mapping content.

10. A digital human interaction system based on a multimodal large model, used to implement a digital human interaction method based on a multimodal large model as described in any one of claims 1 to 9, characterized in that: include: Data acquisition module: collects multimodal input data, performs denoising, normalization and feature extraction on the collected multimodal input data, and performs time synchronization processing on input data of different modes; Feature generation module: Use the preset multi-modal large model to encode the input data of each modality, and uniformly map the encoding results into a high-dimensional space to form multiple modal feature vectors; Interaction generation module: weights the modal feature vectors of each modality, fuses the multi-modal feature vectors based on the preset collaborative perception projection algorithm, and generates a unified first interaction vector; Network construction module: extracting the intention feature representation according to the first interaction vector, calculating and analyzing the intention feature representation to obtain the user intention, and constructing a semantic association network according to the user intention; Content generation module: Perform graph convolution operation on the semantic association network, extract logical relationship features, generate digital human interaction content based on the intention feature representation and logical relationship features, and transmit it to the user through the output interface.

Images