Multi-modal digital human generation method and system based on AIGC and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a multi-modal digital human generation method and system based on AIGC and a storage medium. The method comprises the steps of performing modeling and dynamic weight adjustment processing on multi-modal feature data to obtain dynamic personalized knowledge data; performing intelligent matching processing on the target scene information and the dynamic personalized knowledge data to obtain situational behavior instruction data; performing hierarchical action sequence generation, adaptive resolution optimization and speech synthesis processing on the initial digital human to obtain multi-modal digital human performance data; and behavior deviation identification is carried out on the multi-modal digital human performance data to obtain behavior deviation data and performance feedback data, and adaptive optimization processing is carried out on the multi-modal digital human performance data and the performance feedback data to obtain a target digital human. According to the invention, the efficiency and accuracy of multi-modal digital human generation based on AIGC are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal digital human generation method, system and storage medium based on AIGC. Background Art

[0002] With the rapid development of artificial intelligence and computer graphics technology, digital humans have been widely used in entertainment, education, customer service and other fields. Traditional digital human generation methods mainly rely on pre-set animations and scripts, which makes it difficult to achieve natural and smooth multimodal interactions. In recent years, deep learning-based methods have made significant progress in the field of digital human generation, and can achieve the coordinated generation of vision, speech and action to a certain extent.

[0003] However, existing digital human generation technologies still have some significant limitations. First, the collaborative processing and fusion of multimodal data remains a huge challenge, especially in maintaining temporal consistency and semantic coordination between vision, speech, and action. For example, the generated digital human may have problems such as lip shape not matching speech, expression not coordinating with intonation, or action not matching context. Second, existing methods are difficult to effectively capture and express subtle personality characteristics of characters, such as specific facial expressions, unique voice rhythms, or personalized body language, resulting in the lack of uniqueness and realism of generated digital humans. In addition, for complex and changing scenes, the adaptability of existing technologies is insufficient, and it is difficult to dynamically adjust the behavior and performance of digital humans according to different social situations, cultural backgrounds, or emotional states, which limits the expressiveness and credibility of digital humans in real-world applications. Finally, when processing long-term multimodal data, existing technologies often have problems such as incoherent behavior and unstable emotional expression, which affects the overall performance quality of digital humans. Summary of the invention

[0004] The present application provides a method, system and storage medium for generating a multimodal digital human based on AIGC, which are used to improve the efficiency and accuracy of generating a multimodal digital human based on AIGC.

[0005] In a first aspect, the present application provides a method for generating a multimodal digital human based on AIGC, and the method for generating a multimodal digital human based on AIGC includes: performing data enhancement processing on pre-acquired multimodal character data to obtain an enhanced multimodal data set, and performing feature extraction on the enhanced multimodal data set to obtain multimodal feature data; performing modeling and dynamic weight adjustment processing based on a graph structure neural network on the multimodal feature data to obtain dynamic personality knowledge data; performing intelligent matching processing on preset target scene information and the dynamic personality knowledge data to obtain contextualized behavior instruction data; performing hierarchical action sequence generation, adaptive resolution optimization, and speech synthesis processing on a preset initial digital human according to the contextualized behavior instruction data to obtain multimodal digital human performance data; performing behavioral deviation identification on the multimodal digital human performance data to obtain behavioral deviation data and performance feedback data, and adaptively optimizing the multimodal digital human performance data and the performance feedback data through the behavioral deviation data to obtain a target digital human.

[0006] In combination with the first aspect, in a first implementation method of the first aspect of the present application, the pre-acquired multimodal character data is subjected to data enhancement processing to obtain an enhanced multimodal data set, and feature extraction is performed on the enhanced multimodal data set to obtain multimodal feature data, including: performing timestamp alignment processing on the multimodal character data including video data, audio data and motion capture data to obtain time-synchronized multimodal original data, and performing high dynamic range image synthesis processing on the synchronized video data in the time-synchronized multimodal original data to obtain enhanced video data; performing spectral enhancement processing on the synchronized audio data in the time-synchronized multimodal original data to obtain enhanced audio data, and performing spectral enhancement processing on the synchronized motion capture data in the time-synchronized multimodal original data Perform joint trajectory smoothing processing to obtain enhanced motion data; perform facial feature point extraction processing on the enhanced video data to obtain facial key point data; perform Mel-frequency cepstral coefficient extraction processing on the enhanced audio data to obtain an audio feature vector; perform bone angle and speed calculation processing on the enhanced motion data to obtain a motion feature matrix; perform time alignment processing on the facial key point data, the audio feature vector and the motion feature matrix to obtain aligned feature data, and perform feature importance evaluation based on random forest on the aligned feature data, and select the top K features sorted based on the importance score to obtain reduced dimension feature data, wherein K is a positive integer greater than 1; perform feature scaling processing on the reduced dimension feature data to obtain the multimodal feature data.

[0007] In combination with the first aspect, in a second implementation of the first aspect of the present application, the multimodal feature data is modeled based on a graph structure neural network and dynamically adjusted in weight to obtain dynamic personality knowledge data, including: performing personality feature node construction processing on the multimodal feature data to obtain an initial personality feature graph, and performing inter-modal correlation weight initialization processing on the initial personality feature graph to obtain a weighted personality feature graph; performing cross-modal feature aggregation processing on the weighted personality feature graph to obtain aggregated personality data, and performing nonlinear expression mapping processing on the aggregated personality data to obtain an expression feature vector; performing emotion residual connection processing on the expression feature vector to obtain enhanced emotion representation data, and performing multi-head personality attention processing on the enhanced emotion representation data to obtain attention-weighted personality features; performing scene-adaptive pooling processing on the attention-weighted personality features to obtain hierarchical personality representation data, and performing global personality extraction processing on the hierarchical personality representation data to obtain a comprehensive personality vector; performing dynamic behavior weight update processing on the comprehensive personality vector to obtain an optimized personality structure; performing iterative personality propagation processing on the optimized personality structure to obtain the dynamic personality knowledge data.

[0008] In combination with the first aspect, in a third implementation method of the first aspect of the present application, the preset target scene information and the dynamic personality knowledge data are intelligently matched to obtain contextualized behavior instruction data, including: performing multimodal scene feature extraction on the target scene information to obtain a scene feature vector; performing cross-attention processing on the scene feature vector and the dynamic personality knowledge data to obtain personality-scene association data; performing contextualized behavior pattern generation processing on the personality-scene association data to obtain a candidate behavior sequence; performing filtering processing on the candidate behavior sequence based on preset behavior rules to obtain a target behavior sequence that conforms to the preset behavior rules; performing emotion intensity parameterization processing on the target behavior sequence to obtain emotion parameterized behavior instructions; performing timestamp alignment and interpolation smoothing processing on the emotion parameterized behavior instructions to obtain the contextualized behavior instruction data.

[0009] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, the hierarchical action sequence generation, adaptive resolution optimization and speech synthesis processing are performed on the preset initial digital human according to the contextualized behavior instruction data to obtain multimodal digital human performance data, including: performing time-series decomposition processing on the contextualized behavior instruction data to obtain a single-frame behavior instruction sequence, and performing key frame extraction processing on the single-frame behavior instruction sequence to obtain key behavior node data; performing interpolation and expansion processing on the key behavior node data to obtain complete action sequence data; performing bone mapping processing on the complete action sequence data to obtain initial bone The method comprises the steps of: obtaining skeleton animation data, performing physical constraint-based animation correction on the initial skeleton animation data to obtain target skeleton animation data; performing skin weight calculation processing on the target skeleton animation data to obtain deformed mesh data, and performing texture mapping processing on the deformed mesh data to obtain initial rendering data; performing adaptive resolution adjustment processing on the initial rendering data to obtain optimized rendering data; performing phoneme segmentation processing on the speech part in the contextualized behavior instruction data to obtain phoneme sequence data, and performing acoustic parameter generation processing on the phoneme sequence data and the optimized rendering data to obtain the multimodal digital human performance data.

[0010] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, the multimodal digital human performance data is subjected to behavioral deviation identification to obtain behavioral deviation data and performance feedback data, and the multimodal digital human performance data and the performance feedback data are subjected to adaptive optimization processing through the behavioral deviation data to obtain a target digital human, including: performing modal separation processing on the multimodal digital human performance data to obtain visual component data, audio component data and action component data; performing time window segmentation processing on the visual component data, the audio component data and the action component data to obtain a multimodal behavior segment set, wherein the multimodal behavior segment set includes: visual behavior segments, audio behavior segments and action behavior segments; performing facial expression feature extraction processing on the visual behavior segments to obtain expression feature sequence; performing phoneme segmentation processing on the audio behavior segment to obtain phoneme time series data; performing joint angle calculation processing on the action behavior segment to obtain a skeletal movement sequence; performing time series alignment processing on the expression feature sequence, the phoneme time series data and the skeletal movement sequence to obtain an aligned multimodal sequence, and performing cross-modal consistency analysis on the aligned multimodal sequence to obtain a modal coordination index; performing threshold segmentation processing on the modal coordination index to obtain a preliminary behavior deviation mark; performing cluster analysis processing on the preliminary behavior deviation mark to obtain the behavior deviation data, and performing user experience mapping processing on the behavior deviation data to obtain the performance feedback data; performing adaptive optimization processing on the multimodal digital human performance data and the performance feedback data through the behavior deviation data to obtain a target digital human.

[0011] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, the adaptive optimization processing of the multimodal digital human performance data and the performance feedback data is performed through the behavioral deviation data to obtain a target digital human, including: performing importance scoring processing on the behavioral deviation data to obtain a deviation priority list; performing semantic analysis processing on the performance feedback data to obtain a user preference feature vector, and weighted fusion processing of the deviation priority list and the user preference feature vector to obtain an adjustment target matrix; performing expression correction processing on the visual data in the multimodal digital human performance data based on the adjustment target matrix to obtain a corrected expression sequence; The audio data in the modal digital human performance data is processed by tone adjustment based on the adjustment target matrix to obtain adjusted voice data; the action data in the multimodal digital human performance data is processed by posture fine-tuning based on the adjustment target matrix to obtain an adjusted action sequence; the corrected expression sequence, the adjusted voice data and the adjusted action sequence are processed by time alignment to obtain preliminary adjusted multimodal data, and the preliminary adjusted multimodal data is processed by inter-modal correlation analysis to obtain a correlation matrix; the correlation matrix is ​​processed by threshold screening to obtain modal coordination data, and the modal coordination data is processed by multimodal fusion to obtain the target digital human.

[0012] In a second aspect, the present application provides a multimodal digital human generation system based on AIGC, and the multimodal digital human generation system based on AIGC includes:

[0013] An enhancement module is used to perform data enhancement processing on the pre-acquired multimodal character data to obtain an enhanced multimodal data set, and perform feature extraction on the enhanced multimodal data set to obtain multimodal feature data;

[0014] A processing module, used for performing modeling and dynamic weight adjustment processing on the multimodal feature data based on a graph structure neural network to obtain dynamic personality knowledge data;

[0015] A matching module, used to perform intelligent matching processing on the preset target scene information and the dynamic personality knowledge data to obtain contextualized behavior instruction data;

[0016] A generation module, used to generate hierarchical action sequences, optimize adaptive resolution, and perform speech synthesis processing on a preset initial digital human according to the contextualized behavior instruction data, so as to obtain multimodal digital human performance data;

[0017] The identification module is used to identify behavioral deviations of the multimodal digital human performance data, obtain behavioral deviation data and performance feedback data, and adaptively optimize the multimodal digital human performance data and the performance feedback data through the behavioral deviation data to obtain a target digital human.

[0018] The third aspect of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the above-mentioned AIGC-based multimodal digital human generation method.

[0019] In the technical solution provided by the present application, by performing data enhancement processing and feature extraction on the pre-acquired multimodal character data, the quality and diversity of the original data are significantly improved, laying a solid foundation for subsequent personalized modeling, which not only enhances the realism of the generated digital human, but also greatly improves the generalization ability of the model; secondly, by adopting a modeling and dynamic weight adjustment processing method based on a graph structure neural network, the accurate capture and dynamic expression of complex personality characteristics are achieved, so that the generated digital human can show rich and diverse personalized characteristics, greatly enhancing the uniqueness and realism of the digital human; thirdly, by performing intelligent matching processing on the target scene information and dynamic personality knowledge data, the present method can dynamically generate adaptive behavior instructions according to different scene requirements, greatly improving the performance ability and naturalness of the digital human in a complex and changeable environment; in addition, when generating multimodal digital human performance data, the present method adopts advanced technologies such as hierarchical action sequence generation, adaptive resolution optimization and speech synthesis to ensure a high degree of coordination between the three modes of vision, action and speech, and effectively solve the common modal inconsistency problem in traditional methods; it is worth mentioning. It is worth noting that this method also introduces behavioral deviation identification and adaptive optimization mechanisms, which can detect and correct abnormal behaviors in the performance of digital humans in real time, and make dynamic adjustments based on user feedback, which not only improves the performance quality of digital humans, but also continuously optimizes and improves the behavior patterns of digital humans. At the same time, the modular design of this method makes the entire generation process highly flexible and scalable, and can be customized and optimized according to the needs of different application scenarios. In addition, through the deep fusion and collaborative processing of multimodal data, this method can generate more natural, smooth and expressive digital humans, greatly improving the user's interactive experience and immersion. It is worth mentioning that the adaptive optimization mechanism of this method can not only improve the performance quality of digital humans, but also continuously learn and evolve based on long-term interactive data, making the behavior and performance of digital humans closer and closer to real humans. In addition, this method performs well in processing long-term series of multimodal data, can maintain the consistency of behavior and the stability of emotional expression, solves the common problem of long-term performance instability in the existing technology, and improves the efficiency and accuracy of multimodal digital human generation based on AIGC. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A schematic diagram of an embodiment of a multimodal digital human generation method based on AIGC in an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of an embodiment of a multimodal digital human generation system based on AIGC in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiments of the present application provide a method, system and storage medium for generating a multimodal digital human based on AIGC. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the multimodal digital human generation method based on AIGC includes:

[0025] Step S101: performing data enhancement processing on the pre-acquired multimodal character data to obtain an enhanced multimodal data set, and performing feature extraction on the enhanced multimodal data set to obtain multimodal feature data;

[0026] It is understandable that the execution subject of the present application may be a multimodal digital human generation system based on AIGC, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0027] Specifically, multimodal character data includes video data, audio data and motion capture data, which are collected simultaneously by high-definition cameras, high-fidelity microphones and motion capture devices. In order to ensure the synchronization of data, these data are time-stamp aligned, and the data of different modalities are aligned to a unified time axis using a linear interpolation method. Data enhancement processing is performed on the time-synchronized multimodal raw data. For video data, high dynamic range image synthesis technology is used to merge images with different exposure times to expand the dynamic range of the image and improve the detail expression of the image. Specifically, a multiple exposure fusion algorithm is used to fuse short, medium and long exposure image sequences to obtain enhanced video data with a wider dynamic range. For audio data, spectral enhancement processing technology is used to improve the clarity and recognizability of the audio by adjusting the spectral characteristics of the audio signal. This process uses a Wiener filter to filter the original audio signal according to the estimated noise spectrum and signal spectrum to obtain enhanced audio data. For motion capture data, joint trajectory smoothing is used, and the Savitzky-Golay filter is used to smooth the original motion data to reduce the impact of noise and obtain more smooth and natural enhanced motion data.

[0028] After completing data enhancement, feature extraction is performed on the enhanced multimodal dataset. Facial feature points are extracted from the enhanced video data. The OpenFace algorithm is used to extract 68 facial key points and obtain facial key point data. Mel frequency cepstral coefficients (MFCC) are extracted from the enhanced audio data. A 13-dimensional audio feature vector is obtained through short-time Fourier transform, Mel filter bank and discrete cosine transform. Bone angles and velocities are calculated for the enhanced motion data. Quaternions are used to represent joint rotations, and the relative angles and angular velocities between adjacent joints are calculated to obtain the motion feature matrix. Subsequently, the facial key point data, audio feature vectors and motion feature matrices are time-series aligned to ensure the consistency of different modal features in the time dimension. The feature importance evaluation method based on random forest is used to select the top K features ranked by importance score to achieve feature dimensionality reduction. Specifically, by constructing multiple decision trees, the contribution of each feature to the prediction accuracy of the model is calculated, and the K features with the highest contribution are selected. The feature data after dimensionality reduction is subjected to feature scaling processing, and the feature values ​​are mapped to the [0, 1] interval using the Min-Max normalization method to obtain the final multimodal feature data.

[0029] For example, assume that the original captured video data is 1080p resolution at 30fps, the audio data is mono audio with a sampling rate of 44.1kHz, and the motion capture data is 42 key point data at 60fps. Through timestamp alignment processing, all data are unified to a sampling rate of 60fps. High dynamic range image synthesis is performed on the video data, and the original 8-bit color depth is expanded to 16-bit color depth, which effectively improves the detail performance of the image. Spectral enhancement processing is performed on the audio data. After using the Wiener filter, the signal-to-noise ratio of the speech is improved by about 5dB. Savitzky-Golay filtering is performed on the motion data, using a 5th-order polynomial with a window size of 11 frames, which effectively smoothes the motion trajectory. In the feature extraction stage, 68 facial key points are extracted from the video, 13-dimensional MFCC features are extracted from the audio, and 126-dimensional angle and speed features (42 key points, 3 dimensions per point) are calculated from the motion data. After random forest feature selection, the top 100 most important features are selected from the original 207-dimensional features (68+13+126). Finally, through Min-Max normalization, all eigenvalues ​​are mapped to the interval [0, 1] to obtain the final 100-dimensional multimodal feature data.

[0030] Step S102, performing modeling and dynamic weight adjustment processing based on a graph structure neural network on the multimodal feature data to obtain dynamic personality knowledge data;

[0031] Specifically, in the AIGC-based multimodal digital human generation method, modeling and dynamic weight adjustment of multimodal feature data based on graph structure neural network are key steps, aiming to capture and express the personality characteristics of digital people. First, the multimodal feature data is processed by personality feature node construction, each feature is regarded as a node in the graph, and the initial personality feature graph is constructed. Specifically, for 100-dimensional multimodal feature data, each dimension is regarded as a node, forming a graph structure containing 100 nodes. Then, the initial personality feature graph is initialized with inter-modal association weights, and the weights of the edges are set according to the correlation between the features to obtain a weighted personality feature graph. The initial weight is determined by calculating the Pearson correlation coefficient between the features. The larger the absolute value of the correlation coefficient, the larger the corresponding edge weight.

[0032] Subsequently, the weighted personality feature graph is subjected to cross-modal feature aggregation, and the node features are aggregated using a graph convolutional network (GCN). The aggregation operation of GCN can be expressed as:

[0033] ;

[0034] in, Is a node exist +1 layer features, Is a node The neighbor set of is the normalization constant, It is The weight matrix of the layer, is the activation function. Through the multi-layer GCN, the aggregated personality data is obtained. The aggregated personality data is processed by nonlinear expression mapping, and the aggregated features are mapped to the expression space using a multi-layer perceptron (MLP) to obtain the expression feature vector. Each layer of the MLP can be expressed as:

[0035] ;in is the weight matrix, is the bias vector, is the activation function. The emotion residual connection is performed on the expression feature vector, and the original features are added to the mapped features to obtain the enhanced emotion representation data. The residual connection can be expressed as:

[0036] F(x) = H(x) + x;

[0037] Where H(x) is the mapping function and x is the input feature. Multi-head personality attention processing is performed on the enhanced emotion representation data, and the multi-head self-attention mechanism is used to capture the long-distance dependency between features to obtain attention-weighted personality features. The calculation of multi-head attention can be expressed as:

[0038] ;

[0039] in

[0040] Among them, Q, K, and V represent query, key, and value matrices respectively. In this scheme, they all come from the same input sequence, namely the self-attention mechanism.

[0041] is the number of attention heads, Indicates The calculation of attention heads is as follows:

[0042] ;

[0043] Among them: Concat is a concatenation function used to concatenate the outputs of all attention heads together; is the output linear transformation matrix, which is used to map the concatenated vectors to the final output space; For the The query weight matrix of each head; For the The key weight matrix of each head; For the The multi-head attention mechanism allows the model to focus on information in different subspaces at the same time, thus enhancing the ability to capture complex personality characteristics.

[0044] Then, the attention-weighted personality features are subjected to scene-adaptive pooling, and the graph pooling operation is used to reduce the complexity of the graph to obtain hierarchical personality representation data. The hierarchical personality representation data is subjected to global personality extraction, and the graph-level features are converted into vector representations using the graph readout operation to obtain a comprehensive personality vector. The graph readout uses global average pooling:

[0045] ;

[0046] Where N is the number of nodes, is a node feature.

[0047] Finally, the comprehensive personality vector is dynamically updated with behavioral weights, and the importance of features is dynamically adjusted according to the current task and context to obtain an optimized personality structure. The weight update uses a gated update mechanism:

[0048] ;

[0049] in is the current feature, is context information, is the gate value, Represents element-wise multiplication. The optimized personality structure is iteratively propagated, and the final dynamic personality knowledge data is obtained by propagating personality features through multiple iterations.

[0050] For example, suppose the initial multimodal feature data is a 100-dimensional vector, including 30-dimensional visual features, 20-dimensional audio features, and 50-dimensional action features. First, construct an initial personality feature graph containing 100 nodes. By calculating the Pearson correlation coefficient between the features, a weighted personality feature graph is obtained. For example, the correlation coefficient between the visual feature "eyebrow height" and the audio feature "pitch" is 0.7, and the weight of the corresponding edge is set to 0.7. Use 3 layers of GCN for feature aggregation, and the output dimensions of each layer are 64, 32, and 16 respectively. In the first layer of GCN, for the node , assuming it has 5 neighbor nodes and the initial feature dimension is 100, then its update process is:

[0051] ;

[0052] in is a 64×100 matrix. Nonlinear expression mapping uses a 2-layer MLP with a 16-dimensional input and an 8-dimensional expression feature vector output. After the emotional residual connection, the feature dimension remains at 16. Multi-head attention uses 4 attention heads, each with a dimension of 4. Scene-adaptive pooling reduces the number of nodes from 100 to 50. Global personality extraction obtains a 16-dimensional comprehensive personality vector. Dynamic behavior weight update assigns a weight between 0 and 1 to each dimension in the 16-dimensional vector according to the current scene information. Finally, after 5 iterative propagations, stable 16-dimensional dynamic personality knowledge data is obtained. The whole process not only captures the complex relationship between multimodal features, but also realizes the dynamic adjustment of personality features, providing rich personalized information for subsequent digital human generation.

[0053] Step S103, performing intelligent matching processing on the preset target scene information and the dynamic personality knowledge data to obtain contextualized behavior instruction data;

[0054] Specifically, intelligent matching of preset target scene information and dynamic personality knowledge data to obtain contextualized behavior instruction data is a key step in the multimodal digital human generation method based on AIGC. Multimodal scene feature extraction is performed on the target scene information to obtain a scene feature vector. This process uses a pre-trained multimodal encoder, such as the CLIP model, to encode the image and text description of the scene into a unified feature space. Assume that the dimension of the scene feature vector is 128, expressed as .

[0055] Next, cross-attention processing is performed on the scene feature vector and dynamic personality knowledge data to obtain personality-scene association data. The calculation formula of the cross-attention mechanism is as follows:

[0056] ;

[0057] in, , , , is the scene feature vector, It is dynamic individual knowledge data. This is the value matrix described above, , , is a learnable weight matrix, is the dimension of the key vector. The output A of this step is the personality-scenario association data, which has the same dimension as the dynamic personality knowledge data.

[0058] Then, the personality-scenario association data is processed for situational behavior pattern generation to obtain candidate behavior sequences. This step uses a recurrent neural network (RNN) or a Transformer decoder to generate behavior sequences. The generation process can be expressed as:

[0059] ;

[0060] in, is the hidden state of the RNN, is the representation of the time step t of the personality-scene association data, is the state update function of RNN, g is the output function, is the generated candidate behavior. The candidate behavior sequence is filtered by the preset behavior rules to obtain the behavior data that meets the rules. This step uses the predefined rule set ;

[0061] Filter the generated behaviors. Each rule is a Boolean function, the input is the behavior y, and the output is whether it meets the rule. The filtering process can be expressed as:

[0062]

[0063] The behavior data that meets the rules is processed by sentiment intensity parameterization to obtain sentiment parameterized behavior instructions. This step uses the sentiment analysis model E to score each behavior y and obtain the sentiment intensity vector e:

[0064] ;

[0065] in is the number of affective dimensions (e.g., pleasure, activation, etc.).

[0066] Finally, the emotional parameterized behavior instructions are timestamp aligned and interpolated to obtain contextualized behavior instruction data. Timestamp alignment ensures that the behavior instructions are consistent with the time axis of the target scene, and interpolation smoothing uses a linear interpolation method to fill the gaps between behavior instructions to ensure the continuity of the behavior.

[0067] For example, suppose there is a social scene, and the goal is to generate a digital person's behavior in a party. The scene feature vector S is multimodally encoded to obtain a 128-dimensional vector. The dynamic personality knowledge data P is a 16-dimensional vector that represents the personality characteristics of the digital person. Through cross-attention processing, 16-dimensional personality-scene association data A is obtained. The Transformer decoder is used to generate candidate behavior sequences, such as "walk to the bar", "talk to others", "dance", etc. The preset behavior rules include "do not do illegal behavior", "be polite", etc., and behaviors that do not meet the rules are filtered out. The remaining behaviors are parameterized by emotional intensity. For example, the emotional vector of "dancing" may be [0.8, 0.7], indicating a high degree of pleasure and activation. Finally, these behavioral instructions are aligned to the timeline of the scene, such as "walk to the bar at 19:30:00", "talk to others at 19:35:00", "dance at 20:00:00", and linear interpolation is performed between these discrete time points to obtain continuous contextualized behavioral instruction data. This process ensures that the generated digital human behavior not only meets the scene requirements, but also reflects personality characteristics, while maintaining the continuity and naturalness of the behavior.

[0068] Step S104: generating hierarchical action sequences, optimizing adaptive resolution, and performing speech synthesis processing on the preset initial digital human according to the contextualized behavior instruction data, so as to obtain multimodal digital human performance data;

[0069] Step S105: identify behavioral deviations of the multimodal digital human performance data to obtain behavioral deviation data and performance feedback data, and adaptively optimize the multimodal digital human performance data and performance feedback data using the behavioral deviation data to obtain a target digital human.

[0070] Specifically, the key steps are to generate hierarchical action sequences, optimize adaptive resolution, and perform speech synthesis processing on the preset initial digital human according to the contextualized behavior instruction data. First, the contextualized behavior instruction data is decomposed in time sequence to obtain a single-frame behavior instruction sequence. This process discretizes the continuous behavior instructions into a series of specific instructions at time points. Then, the single-frame behavior instruction sequence is processed by key frame extraction to obtain key behavior node data. This step identifies the key moments of behavior changes and reduces redundant information. Next, the key behavior node data is interpolated and extended to obtain complete action sequence data. The actions between key frames are filled by interpolation algorithms to ensure the continuity and smoothness of the actions. The complete action sequence data is processed by bone mapping to obtain initial bone animation data. This step converts abstract action instructions into specific bone motion data. The initial bone animation data is corrected by animation based on physical constraints to obtain target bone animation data. This process introduces physical simulation to make the action more realistic and natural. Next, the target bone animation data is processed by skin weight calculation to obtain deformation mesh data. This step applies bone animation to the 3D model to achieve model deformation. Then, the deformed mesh data is texture mapped to obtain the initial rendering data, adding surface details and materials to the 3D model.

[0071] The initial rendering data is processed by adaptive resolution adjustment to obtain optimized rendering data. This step dynamically adjusts the rendering quality according to different display devices and network conditions. At the same time, the speech part in the contextualized behavior instruction data is processed by phoneme segmentation to obtain phoneme sequence data, in preparation for subsequent speech synthesis. Finally, the phoneme sequence data and the optimized rendering data are processed by acoustic parameter generation to obtain multimodal digital human performance data. This step generates voice data synchronized with the action and completes the multimodal performance of the digital human. After obtaining the multimodal digital human performance data, behavioral deviation identification is performed to obtain behavioral deviation data and performance feedback data. This process first performs modal separation on the multimodal digital human performance data to obtain visual component data, audio component data, and action component data. Then, these component data are processed by time window segmentation to obtain a multimodal behavior segment set. The visual behavior segment is processed by facial expression feature extraction to obtain an expression feature sequence; the audio behavior segment is processed by phoneme segmentation to obtain phoneme timing data; the action behavior segment is processed by joint angle calculation to obtain a skeletal motion sequence.

[0072] The expression feature sequence, phoneme time series data and skeletal motion sequence are aligned to obtain an aligned multimodal sequence, and the aligned multimodal sequence is analyzed for cross-modal consistency to obtain a modal coordination index. The modal coordination index is segmented by threshold to obtain a preliminary behavior deviation mark, and then the preliminary behavior deviation mark is clustered to obtain behavior deviation data. Finally, the behavior deviation data is processed by user experience mapping to obtain performance feedback data. The multimodal digital human performance data and performance feedback data are adaptively optimized through the behavior deviation data to obtain the target digital human. This process includes importance scoring of the behavior deviation data to obtain a deviation priority list; semantic analysis of the performance feedback data to obtain a user preference feature vector; and weighted fusion of the deviation priority list and the user preference feature vector to obtain an adjusted target matrix. Then, the visual data, audio data and action data in the multimodal digital human performance data are corrected according to the adjusted target matrix to obtain a corrected expression sequence, adjusted voice data and adjusted action sequence. Finally, these corrected data are time-aligned to obtain preliminary adjusted multimodal data, and inter-modal correlation analysis and threshold screening are performed to finally obtain modal coordination data. Through multimodal fusion processing, the target digital human is obtained.

[0073] For example, suppose a digital person's performance in a social scenario needs to be generated and optimized. The initial contextualized behavior instruction data includes actions such as "smile", "wave", and "speak", with a duration of 10 seconds. Through time sequence decomposition and key frame extraction, 5 key behavior nodes are obtained, such as 0 seconds "smile start", 2 seconds "wave start", 4 seconds "speak start", etc. After interpolation expansion, a complete action sequence of 300 frames (30fps) is generated. Bone mapping converts these actions into motion data of 20 key bone points. Physical constraint correction ensures that the actions conform to the physical laws of the real world, such as the inertial effect when the arm is waving. Skin weight calculation applies skeletal animation to a 3D model with 10,000 vertices. Texture mapping adds a 4K resolution skin texture. Adaptive resolution optimization adjusts the rendering resolution to 1080p according to the target device. Speech synthesis generates 10 seconds of audio data that matches the action.

[0074] In the behavioral deviation identification stage, the 10-second data was divided into 20 0.5-second time windows. The motion trajectories of 68 facial key points, the time series data of 200 phonemes, and the angle changes of 20 bone points were extracted. Cross-modal consistency analysis found that at the 3rd second, the facial expression did not match the voice emotion, and the modal coordination index was lower than the threshold of 0.7. Cluster analysis classified this deviation as the "emotional inconsistency" type. User experience mapping converted this deviation into a feedback score of -0.5 (range -1 to 1). In the adaptive optimization stage, a 16x16 adjustment target matrix was generated based on the deviation data and user feedback. The expression near the 3rd second was corrected, and the angle of the corner of the mouth was increased from 15 degrees to 20 degrees to match the pleasant emotion of the voice. At the same time, the voice pitch at the corresponding moment was adjusted, and the fundamental frequency was increased from 220Hz to 240Hz. These corrections, through time alignment and multimodal fusion, ultimately generated a target digital human performance with more coordinated behavior and in line with user expectations.

[0075] In the embodiment of the present application, by performing data enhancement processing and feature extraction on the pre-acquired multimodal character data, the quality and diversity of the original data are significantly improved, laying a solid foundation for subsequent personalized modeling, which not only enhances the realism of the generated digital human, but also greatly improves the generalization ability of the model; secondly, by adopting a modeling and dynamic weight adjustment processing method based on a graph structure neural network, the accurate capture and dynamic expression of complex personality characteristics are achieved, so that the generated digital human can show rich and diverse personalized characteristics, greatly enhancing the uniqueness and realism of the digital human; furthermore, by performing intelligent matching processing on the target scene information and dynamic personality knowledge data, the method can dynamically generate adaptive behavior instructions according to different scene requirements, greatly improving the performance ability and naturalness of the digital human in a complex and changeable environment; in addition, when generating multimodal digital human performance data, the method adopts advanced technologies such as hierarchical action sequence generation, adaptive resolution optimization and speech synthesis to ensure a high degree of coordination between the three modes of vision, action and speech, and effectively solve the common modality inconsistency problem in traditional methods; it is worth noting What is worth mentioning is that this method also introduces behavioral deviation recognition and adaptive optimization mechanisms, which can detect and correct abnormal behaviors in the performance of digital humans in real time, and make dynamic adjustments based on user feedback, which not only improves the performance quality of digital humans, but also continuously optimizes and improves the behavior patterns of digital humans. At the same time, the modular design of this method makes the entire generation process highly flexible and scalable, and can be customized and optimized according to the needs of different application scenarios. In addition, through the deep fusion and collaborative processing of multimodal data, this method can generate more natural, smooth and expressive digital humans, greatly improving the user's interactive experience and immersion. It is worth mentioning that the adaptive optimization mechanism of this method can not only improve the performance quality of digital humans, but also continuously learn and evolve based on long-term interactive data, making the behavior and performance of digital humans closer and closer to real humans. In addition, this method performs well in processing long-term series of multimodal data, can maintain the consistency of behavior and the stability of emotional expression, solves the common problem of long-term performance instability in the prior art, and improves the efficiency and accuracy of multimodal digital human generation based on AIGC.

[0076] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0077] (1) Performing timestamp alignment processing on multimodal character data including video data, audio data, and motion capture data to obtain time-synchronized multimodal raw data, and performing high dynamic range image synthesis processing on synchronized video data in the time-synchronized multimodal raw data to obtain enhanced video data;

[0078] (2) performing spectrum enhancement processing on the synchronized audio data in the time-synchronized multimodal raw data to obtain enhanced audio data, and performing joint trajectory smoothing processing on the synchronized motion capture data in the time-synchronized multimodal raw data to obtain enhanced motion data;

[0079] (3) Extract facial feature points from the enhanced video data to obtain facial key point data;

[0080] (4) Extracting the Mel-frequency cepstral coefficients of the enhanced audio data to obtain an audio feature vector;

[0081] (5) Calculate the bone angle and speed of the enhanced motion data to obtain the motion feature matrix;

[0082] (6) Performing temporal alignment processing on the facial key point data, audio feature vectors and action feature matrix to obtain aligned feature data, and performing feature importance evaluation based on random forest on the aligned feature data, and selecting the top K features sorted by importance scores to obtain reduced dimension feature data, where K is a positive integer greater than 1;

[0083] (7) Perform feature scaling processing on the dimension-reduced feature data to obtain multimodal feature data.

[0084] Specifically, processing multimodal human data including video data, audio data, and motion capture data is a key initial step. First, timestamp alignment is performed to synchronize data of different modalities to a unified timeline. This process uses a linear interpolation method to map each data point to a common time series based on its timestamp, thus obtaining time-synchronized multimodal raw data.

[0085] For synchronized video data, high dynamic range (HDR) image synthesis processing technology is used. This technology expands the dynamic range of the image and improves the expression of details by merging images with different exposure times. The specific implementation uses a multiple exposure fusion algorithm, which can be expressed as:

[0086] ;

[0087] in, is the final HDR image, It is images with different exposures, is the corresponding weight, is the number of images. Weight Usually calculated based on the brightness, contrast, and saturation of the image.

[0088] For synchronized audio data, spectrum enhancement processing is performed. This process uses the Wiener filter to filter the original audio signal according to the estimated noise spectrum and signal spectrum. The frequency domain expression of the Wiener filter is:

[0089] ;

[0090] in, is the frequency response of the filter, is the power spectrum of the signal, is the power spectrum of the noise. For the synchronous motion capture data, the Savitzky-Golay filter is used to smooth the joint trajectory. This filter smoothes the data by local polynomial fitting while retaining the high-order moments of the signal. The filtered data point can be expressed as:

[0091] ;

[0092] in, is the smoothed data point, is the original data point, is the filter coefficient, and m is the half-width of the window. Next, the facial feature points are extracted from the enhanced video data. This step uses the improved OpenFace algorithm to extract 68 standard facial key points to form facial key point data. The enhanced audio data is processed by Mel-frequency cepstral coefficient (MFCC) extraction. The MFCC calculation process includes short-time Fourier transform of the audio signal, mapping the frequency to the Mel scale, taking the logarithm, and performing discrete cosine transform, and finally obtaining a 13-dimensional audio feature vector.

[0093] The enhanced motion data is processed by calculating the bone angle and velocity. Quaternions are used to represent joint rotations, and the relative angles and angular velocities between adjacent joints are calculated to form a motion feature matrix. Then, the facial key point data, audio feature vectors, and motion feature matrix are aligned in time to ensure the consistency of different modal features in the time dimension and obtain aligned feature data. The aligned feature data is evaluated for feature importance based on random forests. The random forest algorithm calculates the contribution of each feature to the model prediction accuracy by constructing multiple decision trees. The feature importance score can be expressed as:

[0094] ;

[0095] in, It is a feature The importance score of , M is the number of decision trees, It is a feature In the decision tree The impurity reduction in . Select the top K features ranked by importance score to obtain the dimension-reduced feature data.

[0096] Finally, the reduced-dimensional feature data is scaled and the feature values ​​are mapped to the [0,1] interval using the Min-Max normalization method to obtain the final multimodal feature data. The Min-Max normalization formula is:

[0097] ;

[0098] in, is the normalized eigenvalue, X is the original eigenvalue, and are the minimum and maximum values ​​of the feature, respectively.

[0099] For example, suppose a 10-second multimodal character data is collected, including 30fps 1080p video, 44.1kHz audio, and 60fps motion capture data. First, timestamp alignment is performed to unify all data to a sampling rate of 60fps. HDR processing is performed on the video data to expand the original 8-bit color depth to 16-bit color depth, significantly improving the details of dark and bright areas. Spectral enhancement is performed on the audio data, and after using the Wiener filter, the signal-to-noise ratio of speech is improved from 15dB to 20dB. The motion data is smoothed using a Savitzky-Golay filter, and the window size is set to 11 frames, which effectively reduces jitter. In the feature extraction stage, 68 facial key points are extracted from the video to obtain 68x2x600 facial key point data (x coordinates and y coordinates, 600 frames). 13-dimensional MFCC features are extracted from the audio to obtain a 13x600 audio feature vector. The motion data calculates the angles and angular velocities of 20 key joints to obtain an 80x600 motion feature matrix (4 dimensions for each joint: 3 angle components and 1 angular velocity). The 161-dimensional features (68x2 + 13 + 80) are evaluated for random forest feature importance using 1,000 decision trees. Assuming K=100, the top 100 most important features are selected. Finally, through Min-Max normalization, all feature values ​​are mapped to the [0,1] interval to obtain the final 100x600 multimodal feature data. This series of processing not only improves data quality, but also significantly reduces data dimensions, laying the foundation for subsequent personalized modeling.

[0100] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0101] (1) constructing individual feature nodes for the multimodal feature data to obtain an initial individual feature graph, and initializing the inter-modal correlation weights for the initial individual feature graph to obtain a weighted individual feature graph;

[0102] (2) Performing cross-modal feature aggregation processing on the weighted personality feature map to obtain aggregated personality data, and performing nonlinear expression mapping processing on the aggregated personality data to obtain an expression feature vector;

[0103] (3) Performing emotional residual connection processing on the expression feature vector to obtain enhanced emotional representation data, and performing multi-head personality attention processing on the enhanced emotional representation data to obtain attention-weighted personality features;

[0104] (4) Performing scene-adaptive pooling processing on the attention-weighted personality features to obtain hierarchical personality representation data, and performing global personality extraction processing on the hierarchical personality representation data to obtain a comprehensive personality vector;

[0105] (5) Dynamically update the behavior weights of the comprehensive personality vector to obtain an optimized personality structure;

[0106] (6) Perform iterative personality propagation processing on the optimized personality structure to obtain dynamic personality knowledge data.

[0107] Specifically, personalized modeling of multimodal feature data is a key step. First, the multimodal feature data is processed by constructing personalized feature nodes, treating each feature as a node in the graph, and constructing an initial personalized feature graph. Assuming there are 100-dimensional feature data, a graph structure containing 100 nodes will be formed. Next, the initial personalized feature graph is initialized with inter-modal association weights, and the weights of the edges are set by calculating the Pearson correlation coefficient between the features to obtain a weighted personalized feature graph.

[0108] The weighted personality feature graph is subjected to cross-modal feature aggregation, and the node features are aggregated using a graph convolutional network (GCN). The aggregation operation of GCN can be expressed as:

[0109] = ( );

[0110] in, It is The node feature matrix of the layer, is the adjacency matrix with self-connection added, is the degree matrix, is a learnable weight matrix, is the activation function.

[0111] The aggregated personality data is processed by nonlinear expression mapping, and the aggregated features are mapped to the expression space using a multi-layer perceptron (MLP) to obtain the expression feature vector. Each layer of the MLP can be expressed as:

[0112] ;in is the weight matrix, is the bias vector, is the activation function. The emotion residual connection is performed on the expression feature vector, and the original features are added to the mapped features to obtain the enhanced emotion representation data. The residual connection can be expressed as:

[0113] F(x) = H(x) + x;

[0114] Where H(x) is the mapping function and x is the input feature. Multi-head personality attention processing is performed on the enhanced emotion representation data, and the multi-head self-attention mechanism is used to capture the long-distance dependency between features to obtain attention-weighted personality features. The calculation of multi-head attention can be expressed as:

[0115] ;

[0116] in ;

[0117] Among them, Q, K, and V represent query, key, and value matrices respectively. In this scheme, they all come from the same input sequence, namely the self-attention mechanism.

[0118] is the number of attention heads, Indicates The calculation of attention heads is as follows:

[0119] ;

[0120] Among them: Concat is a concatenation function used to concatenate the outputs of all attention heads together; is the output linear transformation matrix, which is used to map the concatenated vectors to the final output space; For the The query weight matrix of each head; For the The key weight matrix of each head; For the The multi-head attention mechanism allows the model to focus on information in different subspaces at the same time, thus enhancing the ability to capture complex personality characteristics.

[0121] Then, the attention-weighted personality features are subjected to scene-adaptive pooling, and the graph pooling operation is used to reduce the complexity of the graph to obtain hierarchical personality representation data. The hierarchical personality representation data is subjected to global personality extraction, and the graph-level features are converted into vector representations using the graph readout operation to obtain a comprehensive personality vector. The graph readout uses global average pooling:

[0122] ;

[0123] Where N is the number of nodes, is a node feature.

[0124] Finally, the comprehensive personality vector is dynamically updated with behavioral weights, and the importance of features is dynamically adjusted according to the current task and context to obtain an optimized personality structure. The weight update uses a gated update mechanism:

[0125] ;

[0126] in is the current feature, is context information, is the gate value, Represents element-wise multiplication. The optimized personality structure is iteratively propagated, and the final dynamic personality knowledge data is obtained by propagating personality features through multiple iterations.

[0127] For example, assume that the initial multimodal feature data is a 100-dimensional vector, including 30-dimensional visual features, 20-dimensional audio features, and 50-dimensional action features. First, construct an initial personality feature graph containing 100 nodes. By calculating the Pearson correlation coefficient between the features, a weighted personality feature graph is obtained. For example, the correlation coefficient between the visual feature "eyebrow height" and the audio feature "pitch" is 0.7, and the weight of the corresponding edge is set to 0.7. Use 3 layers of GCN for feature aggregation, and the output dimensions of each layer are 64, 32, and 16 respectively. In the first layer of GCN, for node i, assuming it has 5 neighbor nodes and the initial feature dimension is 100, its update process is:

[0128] ;

[0129] in is a 64×100 matrix. Nonlinear expression mapping uses a 2-layer MLP with a 16-dimensional input and an 8-dimensional expression feature vector output. After the emotional residual connection, the feature dimension remains at 16. Multi-head attention uses 4 attention heads, each with a dimension of 4. Scene-adaptive pooling reduces the number of nodes from 100 to 50. Global personality extraction obtains a 16-dimensional comprehensive personality vector. Dynamic behavior weight update assigns a weight between 0 and 1 to each dimension in the 16-dimensional vector according to the current scene information. Finally, after 5 iterative propagations, stable 16-dimensional dynamic personality knowledge data is obtained. The whole process not only captures the complex relationship between multimodal features, but also realizes the dynamic adjustment of personality features, providing rich personalized information for subsequent digital human generation.

[0130] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0131] (1) Perform multimodal scene feature extraction on the target scene information to obtain a scene feature vector;

[0132] (2) Perform cross-attention processing on the scene feature vector and dynamic personality knowledge data to obtain personality-scene association data;

[0133] (3) Generate contextualized behavior patterns based on the personality-scenario association data to obtain candidate behavior sequences;

[0134] (4) Filtering the candidate behavior sequences based on the preset behavior rules to obtain the target behavior sequences that meet the preset behavior rules;

[0135] (5) Parameterize the target behavior sequence with emotional intensity to obtain emotional parameterized behavior instructions;

[0136] (6) Perform timestamp alignment and interpolation smoothing on the emotional parameterized behavioral instructions to obtain contextualized behavioral instruction data.

[0137] Specifically, intelligent matching of preset target scene information and dynamic personality knowledge data to obtain contextualized behavior instruction data is a key step in the multimodal digital human generation method based on AIGC. Multimodal scene feature extraction is performed on the target scene information to obtain a scene feature vector. This process uses a pre-trained multimodal encoder, such as the CLIP model, to encode the image and text description of the scene into a unified feature space. Assume that the dimension of the scene feature vector is 128, expressed as .

[0138] Next, cross-attention processing is performed on the scene feature vector and dynamic personality knowledge data to obtain personality-scene association data. The calculation formula of the cross-attention mechanism is as follows:

[0139] ;

[0140] in, , , , is the scene feature vector, It is dynamic individual knowledge data. This is the value matrix described above, , , is a learnable weight matrix, is the dimension of the key vector. The output A of this step is the personality-scenario association data, which has the same dimension as the dynamic personality knowledge data.

[0141] Then, the personality-scenario association data is processed for situational behavior pattern generation to obtain candidate behavior sequences. This step uses a recurrent neural network (RNN) or a Transformer decoder to generate behavior sequences. The generation process can be expressed as:

[0142] ;

[0143] in, is the hidden state of the RNN, is the representation of the time step t of the personality-scene association data, is the state update function of RNN, g is the output function, is the generated candidate behavior. The candidate behavior sequence is filtered by the preset behavior rules to obtain the behavior data that meets the rules. This step uses the predefined rule set ;

[0144] Filter the generated behaviors. Each rule is a Boolean function, the input is the behavior y, and the output is whether it meets the rule. The filtering process can be expressed as:

[0145] ;

[0146] The behavior data that meets the rules is processed by sentiment intensity parameterization to obtain sentiment parameterized behavior instructions. This step uses the sentiment analysis model E to score each behavior y and obtain the sentiment intensity vector e:

[0147] ;

[0148] in is the number of affective dimensions (e.g., pleasure, activation, etc.).

[0149] Finally, the emotional parameterized behavior instructions are timestamp aligned and interpolated to obtain contextualized behavior instruction data. Timestamp alignment ensures that the behavior instructions are consistent with the time axis of the target scene, and interpolation smoothing uses a linear interpolation method to fill the gaps between behavior instructions to ensure the continuity of the behavior.

[0150] For example, suppose there is a social scene, and the goal is to generate a digital person's behavior in a party. The scene feature vector S is multimodally encoded to obtain a 128-dimensional vector. The dynamic personality knowledge data P is a 16-dimensional vector that represents the personality characteristics of the digital person. Through cross-attention processing, 16-dimensional personality-scene association data A is obtained. The Transformer decoder is used to generate candidate behavior sequences, such as "walk to the bar", "talk to others", "dance", etc. The preset behavior rules include "do not do illegal behavior", "be polite", etc., and behaviors that do not meet the rules are filtered out. The remaining behaviors are parameterized by emotional intensity. For example, the emotional vector of "dancing" may be [0.8, 0.7], indicating a high degree of pleasure and activation. Finally, these behavioral instructions are aligned to the timeline of the scene, such as "walk to the bar at 19:30:00", "talk to others at 19:35:00", "dance at 20:00:00", and linear interpolation is performed between these discrete time points to obtain continuous contextualized behavioral instruction data. This process ensures that the generated digital human behavior not only meets the scene requirements, but also reflects personality characteristics, while maintaining the continuity and naturalness of the behavior.

[0151] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0152] (1) Performing time-series decomposition processing on the contextualized behavior instruction data to obtain a single-frame behavior instruction sequence, and performing key frame extraction processing on the single-frame behavior instruction sequence to obtain key behavior node data;

[0153] (2) Perform interpolation and expansion processing on key behavior node data to obtain complete action sequence data;

[0154] (3) Performing skeletal mapping processing on the complete action sequence data to obtain initial skeletal animation data, and performing animation correction based on physical constraints on the initial skeletal animation data to obtain target skeletal animation data;

[0155] (4) Perform skin weight calculation on the target skeletal animation data to obtain deformed mesh data, and perform texture mapping on the deformed mesh data to obtain initial rendering data;

[0156] (5) Adaptively adjust the resolution of the initial rendering data to obtain optimized rendering data;

[0157] (6) Perform phoneme segmentation on the speech part of the contextualized behavior instruction data to obtain phoneme sequence data, and perform acoustic parameter generation processing on the phoneme sequence data and optimized rendering data to obtain multimodal digital human performance data.

[0158] Specifically, the contextualized behavior instruction data is decomposed in time series to discretize the continuous behavior instructions into a single-frame behavior instruction sequence. This process uses a time window sliding technique, where each window represents a time step and the window size is determined based on the target frame rate. For example, for a 30fps target animation, every 33.3 milliseconds of behavior instruction is extracted as a frame.

[0159] Next, perform key frame extraction on the single-frame action instruction sequence to obtain key action node data. This step uses the action saliency detection algorithm to calculate the difference between each frame and adjacent frames. Frames whose difference exceeds the preset threshold are marked as key frames. The difference calculation can use cosine similarity:

[0160] ;

[0161] Among them, A and B are the feature vectors of adjacent frames, and (n) is the feature dimension. The key behavior node data is interpolated and expanded to obtain the complete action sequence data. Spline interpolation method, such as cubic spline interpolation, is used here to ensure that the generated action curve is smooth and continuous. The cubic spline interpolation function can be expressed as:

[0162] ;

[0163] in, is the interpolation function of the ith segment, , , , is the coefficient to be determined.

[0164] The complete action sequence data is processed by bone mapping to obtain the initial bone animation data. This step uses the inverse kinematics (IK) algorithm to convert the abstract action instructions into specific bone rotation data. The IK solution can use the Jacobian matrix method, and its iterative formula is:

[0165] ;

[0166] in, is the pseudo-inverse of the Jacobian matrix, is the change in joint angle, is the position change of the end effector. The initial skeletal animation data is corrected based on physical constraints to obtain the target skeletal animation data. This step introduces physical simulation and uses a spring-mass system to simulate the physical properties of muscles and joints. The spring force can be expressed as:

[0167] ;in, is the spring constant, is the damping coefficient, is the spring deformation. The target skeleton animation data is processed by skin weight calculation to obtain the deformed mesh data. Here, the linear blend skinning (LBS) algorithm is used, and the position of each vertex is affected by multiple bones weighted:

[0168] ;in, is the vertex position after deformation, is the weight, is the bone transformation matrix, is the original vertex position.

[0169] The deformed mesh data is texture mapped to obtain the initial rendering data. This step uses UV mapping technology to map the 2D texture image to the 3D model surface. The initial rendering data is adaptively adjusted to obtain optimized rendering data. Here, a dynamic resolution scaling algorithm is used to dynamically adjust the rendering resolution according to device performance and network conditions.

[0170] Finally, the speech part of the contextualized behavior instruction data is processed by phoneme segmentation to obtain phoneme sequence data. This step uses the hidden Markov model (HMM) for speech recognition and phoneme segmentation. The acoustic parameter generation process is performed on the phoneme sequence data and the optimized rendering data to obtain multimodal digital human performance data. Here, a deep learning model, such as WaveNet, is used to generate speech data synchronized with the animation.

[0171] For example, suppose there is a 10-second long contextual behavior instruction data, and the goal is to generate a 30fps animation. First, decompose the 10-second instruction data into 300 frames (10 seconds The proposed method uses a single-frame action instruction sequence of 30fps. Through action saliency detection, 50 key frames are identified, with an average of one key frame every 0.2 seconds. Using cubic spline interpolation, the 50 key frames are expanded into a complete action sequence of 300 frames.

[0172] In the bone mapping stage, it is assumed that the digital human model has 20 key bone points. Using the IK algorithm, the abstract action instructions of each frame are converted into the rotation data of 20 bone points, forming 300203 initial bone animation data (300 frames, 20 bones, 3 rotation angles for each bone). The physical constraint correction uses a time step of 10ms for 1000 iterations to ensure that the action complies with the laws of physics.

[0173] In the skin weight calculation stage, assume that the 3D model has 10,000 vertices and each vertex is affected by the 4 nearest bones. Through the LBS algorithm, 300100003 deformation mesh data is generated (300 frames, 10,000 vertices, 3 coordinates per vertex). Texture mapping uses 2048 2048 resolution texture map.

[0174] Adaptive resolution adjustment dynamically adjusts the rendering resolution from 1080p to 720p according to the target device, reducing the rendering load by 30%. In terms of voice processing, 10 seconds of voice data is divided into about 100 phonemes. Using the WaveNet model, voice data with a sampling rate of 44.1kHz is generated, which is precisely synchronized with 300 frames of animation. The final multimodal digital human performance data includes 300 frames of visual data, 441,000 audio samples (10 seconds × 44.1kHz) and corresponding timestamp information, achieving highly realistic and synchronized digital human performance.

[0175] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0176] (1) Perform modal separation processing on the multimodal digital human performance data to obtain visual component data, audio component data, and action component data;

[0177] (2) performing time window segmentation processing on the visual component data, the audio component data, and the motion component data to obtain a multimodal behavior segment set, wherein the multimodal behavior segment set includes: a visual behavior segment, an audio behavior segment, and a motion behavior segment;

[0178] (3) Extract facial expression features from visual behavior segments to obtain expression feature sequences;

[0179] (4) Perform phoneme segmentation on the audio behavior segment to obtain phoneme time series data;

[0180] (5) Calculate the joint angles of the action segments to obtain the skeletal motion sequence;

[0181] (6) Perform temporal alignment processing on the expression feature sequence, phoneme time series data, and skeletal movement sequence to obtain an aligned multimodal sequence, and perform cross-modal consistency analysis on the aligned multimodal sequence to obtain a modal coordination index;

[0182] (7) Perform threshold segmentation on the modal coordination index to obtain preliminary behavioral deviation markers;

[0183] (8) Performing cluster analysis on the preliminary behavioral deviation marks to obtain behavioral deviation data, and performing user experience mapping on the behavioral deviation data to obtain performance feedback data;

[0184] (9) Adaptively optimize the multimodal digital human performance data and performance feedback data through behavioral deviation data to obtain the target digital human.

[0185] Specifically, the multimodal digital human performance data is subjected to modal separation processing to separate the visual, audio and motion data. This process uses data parsing technology to extract data of different modalities into their respective data streams based on predefined data structures and tags.

[0186] Next, the separated visual component data, audio component data, and action component data are segmented into time windows to obtain a set of multimodal behavior segments. The time window segmentation uses a sliding window technique, and the window size is usually set to 0.5 seconds to 2 seconds, with a step size of 25% to 50% of the window size. For example, for 30fps video data, a window size of 1 second corresponds to 30 frames, and the step size may be 8 frames or 15 frames.

[0187] The facial expression feature extraction process is performed on the visual behavior segment to obtain the expression feature sequence. This step uses a facial feature point detection algorithm, such as the improved Active Shape Model (ASM), to extract 68 facial key points. Each key point is represented by (x, y) coordinates, so the expression feature of each frame is a 136-dimensional vector. The audio behavior segment is processed by phoneme segmentation to obtain phoneme time series data. Phoneme segmentation uses the hidden Markov model (HMM) combined with the Mel-frequency cepstral coefficient (MFCC) feature to divide the continuous speech signal into discrete phoneme units. The action behavior segment is processed by joint angle calculation to obtain a skeletal motion sequence. Here, the quaternion representation is used to calculate the joint angle, and the rotation of each joint is represented by a four-dimensional vector.

[0188] The expression feature sequence, phoneme time series data and skeletal motion sequence are time-series aligned to obtain an aligned multimodal sequence. The dynamic time warping (DTW) algorithm is used for time series alignment. The distance calculation formula of DTW is:

[0189] , where X and Y are two time series, and is the eigenvalue at the corresponding time point.

[0190] Perform cross-modal consistency analysis on the aligned multimodal sequences to obtain the modal coordination index. The consistency analysis is calculated using mutual information (MI), and the formula is:

[0191] ;

[0192] Among them, p(x,y) is the joint probability distribution, and p(x) and p(y) are marginal probability distributions.

[0193] The modal coordination index is subjected to threshold segmentation to obtain preliminary behavioral deviation marks. The threshold is usually set to 80% of the average coordination. The preliminary behavioral deviation marks are subjected to cluster analysis to obtain behavioral deviation data. The K-means clustering algorithm is used here, and the number of clusters K is determined by the elbow rule. The behavioral deviation data is subjected to user experience mapping to obtain performance feedback data. The mapping function can be a simple linear mapping or a more complex nonlinear function, such as an S-type function:

[0194] ;

[0195] Among them, k controls the steepness of the curve, It's the midpoint.

[0196] Finally, the performance data and performance feedback data of the multimodal digital human are adaptively optimized through the behavioral deviation data to obtain the target digital human. The optimization process uses the gradient descent method, and the objective function is:

[0197] ;

[0198] in, , , , is the weight coefficient, , , They are the loss functions for vision, audio, and action respectively. is the feedback loss.

[0199] For example, suppose there is a 10-second multimodal digital human performance data, with a video frame rate of 30fps and an audio sampling rate of 16kHz. First, modal separation is performed to obtain 300 frames of video data, 160,000 audio samples, and 300 sets of action data. Using a 1-second time window and a 0.5-second step size for segmentation, 19 multimodal behavior segments are obtained. 68 facial feature points are extracted for each visual segment to form a 30×136 expression feature sequence. The audio segment is segmented by phonemes, and an average of 10 phonemes are recognized per second to obtain 10×19 phoneme time series data. The action segment calculates the angles of 20 key joints to form a 30×80 skeletal motion sequence. After time series alignment, the mutual information between modalities is calculated to obtain the coordination index of 19 time points. The threshold is set to 0.75, and 3 time periods with coordination below the threshold are identified. These deviations are divided into two categories: "expression-speech mismatch" and "action-speech asynchrony" through K-means clustering (K=2). User experience mapping converts the degree of deviation into a feedback score ranging from -0.8 to 0.2. Finally, the optimization algorithm adjusts the data of these three time periods, such as increasing the smile level of the expression from 0.3 to 0.6 and delaying the action by 100ms to match the voice, and finally generates a target digital human performance with more coordinated behavior.

[0200] In a specific embodiment, the process of performing adaptive optimization processing on multimodal digital human performance data and performance feedback data through behavioral deviation data may specifically include the following steps:

[0201] (1) Perform importance scoring on behavioral deviation data to obtain a deviation priority list;

[0202] (2) Perform semantic analysis on the performance feedback data to obtain the user preference feature vector, and perform weighted fusion processing on the deviation priority list and the user preference feature vector to obtain the adjustment target matrix;

[0203] (3) Performing expression correction processing on the visual data in the multimodal digital human performance data based on the adjustment target matrix to obtain a corrected expression sequence;

[0204] (4) performing pitch adjustment processing on the audio data in the multimodal digital human performance data based on the adjustment target matrix to obtain adjusted speech data;

[0205] (5) Performing posture fine-tuning processing on the action data in the multimodal digital human performance data based on the adjustment target matrix to obtain an adjusted action sequence;

[0206] (6) Performing time alignment processing on the corrected expression sequence, the adjusted voice data, and the adjusted action sequence to obtain preliminary adjusted multimodal data, and performing inter-modal correlation analysis processing on the preliminary adjusted multimodal data to obtain a correlation matrix;

[0207] (7) Perform threshold screening on the correlation matrix to obtain modal coordination data, and perform multimodal fusion processing on the modal coordination data to obtain the target digital human.

[0208] Specifically, the behavior deviation data is scored according to their importance to obtain a deviation priority list. This process uses a weighted scoring model that takes into account the magnitude, duration, and impact on user perception of the deviation. The scoring formula can be expressed as:

[0209] ;

[0210] in, is the score of the ith deviation, is the deviation amplitude, is the duration, is the degree of impact, , , is the corresponding weight.

[0211] Next, semantic analysis is performed on the performance feedback data to obtain the user preference feature vector. This step uses natural language processing techniques, such as word embedding and sentiment analysis, to convert user feedback into numerical feature vectors. Word embedding can use the Word2Vec model, and sentiment analysis uses a deep learning model based on LSTM. The user preference feature vector can be expressed as:

[0212] ;

[0213] in, Represents user preference scores in different dimensions, such as naturalness, emotional expression, and motion fluency. The deviation priority list is weightedly fused with the user preference feature vector to obtain the adjustment target matrix. The fusion process uses matrix operations, and the formula is as follows:

[0214] ;

[0215] in, is the adjusted target matrix, is the deviation priority list, is the user preference feature vector, and is the equilibrium parameter, is the identity matrix.

[0216] Based on the adjustment target matrix, the visual, audio and motion data in the multimodal digital human performance data are corrected. The facial action units (AUs) are used to adjust the expression correction of the visual data, and the activation intensity of each AU is adjusted according to the corresponding element value of the adjustment target matrix. The pitch adjustment of the audio data uses the pitch shifting algorithm, and the adjustment amplitude is determined by the adjustment target matrix. The posture fine-tuning of the motion data is achieved through the inverse kinematics (IK) algorithm, and the adjustment target is specified by the adjustment target matrix.

[0217] The corrected expression sequence, voice data and action sequence are time-aligned to obtain the preliminary adjusted multimodal data. The dynamic time warping (DTW) algorithm is used for time alignment to ensure the consistency of different modal data in the time dimension. The distance calculation formula of DTW is: ;

[0218] Among them, X and Y are two time series, is the distance between two data points.

[0219] The inter-modal correlation analysis is performed on the preliminarily adjusted multimodal data to obtain the correlation matrix. This step uses mutual information (MI) to calculate the correlation between different modes, and the formula is as follows:

[0220] ;

[0221] Among them, p(x,y) is the joint probability distribution, and p(x) and p(y) are marginal probability distributions.

[0222] The correlation matrix is ​​processed by threshold screening to obtain modal coordination data. The threshold selection uses the Otsu method to automatically determine the optimal segmentation threshold. Finally, the modal coordination data is processed by multimodal fusion to obtain the target digital human. The fusion process uses the attention mechanism to dynamically adjust the fusion weight according to the importance of different modalities.

[0223] For example, suppose there is a 10-second multimodal digital human performance data, including 300 frames of video (30fps), 160,000 audio samples (16kHz sampling rate) and 300 sets of action data. The behavioral deviation analysis found three major deviations:

[0224] 1) The expression is unnatural in the 2nd to 3rd seconds;

[0225] 2) The voice pitch in the 5th and 6th seconds is too high;

[0226] 3) The movements are stiff at 8-9 seconds.

[0227] These deviations were scored for importance, with scores of 0.8, 0.6, and 0.7, respectively. After semantic analysis of user feedback, a 5-dimensional preference feature vector [0.9, 0.7, 0.8, 0.6, 0.8] was obtained, representing preferences for naturalness, emotional expression, movement fluency, timbre, and synchronization, respectively. Through weighted fusion, a 3x5 adjustment target matrix was obtained. Based on this matrix, the expression at the 2nd and 3rd seconds was corrected, and the AU activation intensity of the smile was increased from 0.3 to 0.6; the pitch of the voice at the 5th and 6th seconds was adjusted, and the fundamental frequency was reduced from 280Hz to 240Hz; the action at the 8th and 9th seconds was fine-tuned, and the arm swing amplitude was increased by 10%. Timing alignment ensures that these adjustments are synchronized in time, such as adjusting the start time of the expression change from 1.95 seconds to 2.00 seconds to accurately match the audio changes.

[0228] The inter-modal correlation analysis obtains a 3x3 correlation matrix, which indicates the correlation between vision-audio, vision-action, and audio-action. Assume that the obtained matrix is:

[0229] ;

[0230] The threshold determined by the Otsu method is 0.75, and the modal combinations with high correlation are retained after screening. Finally, multimodal fusion is performed through the attention mechanism, and the weights of different modalities are dynamically adjusted according to the correlation matrix, such as increasing the weight of the visual modality during the expression change period and increasing the weight of the audio modality during the voice adjustment period, and finally generating a target digital human performance with more coordinated behavior and in line with user preferences. This optimization process not only corrects specific behavioral deviations, but also achieves an overall improvement in digital human performance by considering the correlation between modalities and user preferences.

[0231] The above describes the multimodal digital human generation method based on AIGC in the embodiment of the present application. The following describes the multimodal digital human generation system based on AIGC in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a multimodal digital human generation system based on AIGC includes:

[0232] The enhancement module 201 is used to perform data enhancement processing on the pre-acquired multimodal character data to obtain an enhanced multimodal data set, and perform feature extraction on the enhanced multimodal data set to obtain multimodal feature data;

[0233] The processing module 202 is used to perform modeling and dynamic weight adjustment processing on the multimodal feature data based on a graph structure neural network to obtain dynamic personality knowledge data;

[0234] The matching module 203 is used to perform intelligent matching processing on the preset target scene information and the dynamic personality knowledge data to obtain contextualized behavior instruction data;

[0235] The generation module 204 is used to generate hierarchical action sequences, optimize adaptive resolution, and perform speech synthesis processing on the preset initial digital human according to the contextualized behavior instruction data to obtain multimodal digital human performance data;

[0236] The identification module 205 is used to identify the behavior deviation of the multimodal digital human performance data, obtain the behavior deviation data and the performance feedback data, and perform adaptive optimization processing on the multimodal digital human performance data and the performance feedback data through the behavior deviation data to obtain the target digital human.

[0237] Through the collaborative cooperation of the above-mentioned components, by performing data enhancement processing and feature extraction on the pre-acquired multimodal character data, the quality and diversity of the original data are significantly improved, laying a solid foundation for subsequent personalized modeling, which not only enhances the realism of the generated digital human, but also greatly improves the generalization ability of the model; secondly, by adopting the modeling and dynamic weight adjustment processing method based on the graph structure neural network, the accurate capture and dynamic expression of complex personality characteristics are realized, so that the generated digital human can show rich and diverse personalized characteristics, greatly enhancing the uniqueness and realism of the digital human; thirdly, by performing intelligent matching processing on the target scene information and dynamic personality knowledge data, this method can dynamically generate adaptive behavior instructions according to different scene requirements, greatly improving the performance ability and naturalness of the digital human in a complex and changeable environment; in addition, when generating multimodal digital human performance data, this method adopts advanced technologies such as hierarchical action sequence generation, adaptive resolution optimization and speech synthesis to ensure the high coordination between the three modalities of vision, action and speech, and effectively solve the common modality inconsistency problem in traditional methods; It is worth noting that this method also introduces behavioral deviation identification and adaptive optimization mechanisms, which can detect and correct abnormal behaviors in the performance of digital humans in real time, and make dynamic adjustments based on user feedback, which not only improves the performance quality of digital humans, but also continuously optimizes and improves the behavior patterns of digital humans. At the same time, the modular design of this method makes the entire generation process highly flexible and scalable, and can be customized and optimized according to the needs of different application scenarios. In addition, through the deep fusion and collaborative processing of multimodal data, this method can generate more natural, smooth and expressive digital humans, greatly improving the user's interactive experience and immersion. It is worth mentioning that the adaptive optimization mechanism of this method can not only improve the performance quality of digital humans, but also continuously learn and evolve based on long-term interactive data, making the behavior and performance of digital humans closer and closer to real humans. In addition, this method performs well in processing long-term series of multimodal data, can maintain the consistency of behavior and the stability of emotional expression, solves the common problem of long-term performance instability in the existing technology, and improves the efficiency and accuracy of multimodal digital human generation based on AIGC.

[0238] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the AIGC-based multimodal digital human generation method.

[0239] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0240] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal digital human generation method based on AIGC, characterized in that: The AIGC-based multimodal digital human generation method comprises: Performing data enhancement processing on the pre-acquired multimodal character data to obtain an enhanced multimodal data set, and performing feature extraction on the enhanced multimodal data set to obtain multimodal feature data; The multimodal feature data is modeled based on a graph structure neural network and dynamically adjusted in weight to obtain dynamic individual knowledge data; Intelligently matching the preset target scene information and the dynamic personality knowledge data to obtain contextualized behavior instruction data; According to the contextualized behavior instruction data, hierarchical action sequence generation, adaptive resolution optimization, and speech synthesis processing are performed on the preset initial digital human to obtain multimodal digital human performance data; Behavioral deviation identification is performed on the multimodal digital human performance data to obtain behavioral deviation data and performance feedback data, and adaptive optimization processing is performed on the multimodal digital human performance data and the performance feedback data through the behavioral deviation data to obtain a target digital human.

2. The AIGC-based multimodal digital human generation method according to claim 1, characterized in that: The method of performing data enhancement processing on the pre-acquired multimodal character data to obtain an enhanced multimodal data set, and performing feature extraction on the enhanced multimodal data set to obtain multimodal feature data, includes: Performing timestamp alignment processing on multimodal character data including video data, audio data and motion capture data to obtain time-synchronized multimodal original data, and performing high dynamic range image synthesis processing on synchronized video data in the time-synchronized multimodal original data to obtain enhanced video data; Performing spectrum enhancement processing on the synchronized audio data in the time-synchronized multimodal original data to obtain enhanced audio data, and performing joint trajectory smoothing processing on the synchronized motion capture data in the time-synchronized multimodal original data to obtain enhanced motion data; Performing facial feature point extraction processing on the enhanced video data to obtain facial key point data; Performing Mel-frequency cepstral coefficient extraction processing on the enhanced audio data to obtain an audio feature vector; Performing bone angle and speed calculation processing on the enhanced motion data to obtain a motion feature matrix; Performing time-series alignment processing on the facial key point data, the audio feature vector and the action feature matrix to obtain aligned feature data, performing feature importance evaluation based on random forest on the aligned feature data, and selecting the top K features sorted based on importance scores to obtain reduced dimension feature data, where K is a positive integer greater than 1; Perform feature scaling processing on the dimension-reduced feature data to obtain the multimodal feature data.

3. The AIGC-based multimodal digital human generation method according to claim 1, characterized in that: The multimodal feature data is modeled based on a graph structure neural network and dynamically adjusted in weight to obtain dynamic individual knowledge data, including: Performing a personality feature node construction process on the multimodal feature data to obtain an initial personality feature graph, and performing an inter-modality association weight initialization process on the initial personality feature graph to obtain a weighted personality feature graph; Performing cross-modal feature aggregation processing on the weighted personality feature map to obtain aggregated personality data, and performing nonlinear expression mapping processing on the aggregated personality data to obtain an expression feature vector; Performing emotional residual connection processing on the expression feature vector to obtain enhanced emotional representation data, and performing multi-head personality attention processing on the enhanced emotional representation data to obtain attention-weighted personality features; Performing scene-adaptive pooling processing on the attention-weighted personality features to obtain hierarchical personality representation data, and performing global personality extraction processing on the hierarchical personality representation data to obtain a comprehensive personality vector; Performing dynamic behavior weight updating processing on the comprehensive personality vector to obtain an optimized personality structure; The optimized personality structure is iteratively processed for personality propagation to obtain the dynamic personality knowledge data.

4. The AIGC-based multimodal digital human generation method according to claim 1, characterized in that: The intelligent matching process of the preset target scene information and the dynamic personality knowledge data to obtain the contextualized behavior instruction data includes: Performing multimodal scene feature extraction processing on the target scene information to obtain a scene feature vector; Performing cross-attention processing on the scene feature vector and the dynamic personality knowledge data to obtain personality-scene association data; Performing situational behavior pattern generation processing on the personality-scenario association data to obtain a candidate behavior sequence; Performing filtering processing on the candidate behavior sequence based on preset behavior rules to obtain a target behavior sequence that conforms to the preset behavior rules; Performing parameterized emotional intensity processing on the target behavior sequence to obtain emotional parameterized behavior instructions; The emotional parameterized behavior instruction is subjected to timestamp alignment and interpolation smoothing processing to obtain the contextualized behavior instruction data.

5. The AIGC-based multimodal digital human generation method according to claim 1, characterized in that: The step of generating hierarchical action sequences, optimizing adaptive resolution, and performing speech synthesis processing on the preset initial digital human according to the contextualized behavior instruction data to obtain multimodal digital human performance data includes: Performing time-series decomposition processing on the contextualized behavior instruction data to obtain a single-frame behavior instruction sequence, and performing key frame extraction processing on the single-frame behavior instruction sequence to obtain key behavior node data; Performing interpolation and expansion processing on the key behavior node data to obtain complete action sequence data; Performing skeleton mapping processing on the complete action sequence data to obtain initial skeleton animation data, and performing animation correction based on physical constraints on the initial skeleton animation data to obtain target skeleton animation data; Performing skin weight calculation processing on the target skeletal animation data to obtain deformed mesh data, and performing texture mapping processing on the deformed mesh data to obtain initial rendering data; Performing adaptive resolution adjustment processing on the initial rendering data to obtain optimized rendering data; The speech part in the contextualized behavior instruction data is processed by phoneme segmentation to obtain phoneme sequence data, and the phoneme sequence data and the optimized rendering data are processed by acoustic parameter generation to obtain the multimodal digital human performance data.

6. The AIGC-based multimodal digital human generation method according to claim 1, characterized in that: The method of performing behavioral deviation identification on the multimodal digital human performance data to obtain behavioral deviation data and performance feedback data, and performing adaptive optimization processing on the multimodal digital human performance data and the performance feedback data through the behavioral deviation data to obtain a target digital human includes: Performing modal separation processing on the multimodal digital human performance data to obtain visual component data, audio component data and action component data; Performing time window segmentation processing on the visual component data, the audio component data and the action component data to obtain a multimodal action segment set, wherein the multimodal action segment set includes: a visual action segment, an audio action segment and an action action segment; Performing facial expression feature extraction processing on the visual behavior segment to obtain an expression feature sequence; Performing phoneme segmentation processing on the audio behavior segment to obtain phoneme time sequence data; Performing joint angle calculation processing on the action behavior fragment to obtain a skeletal motion sequence; Performing time alignment processing on the expression feature sequence, the phoneme time series data and the skeletal movement sequence to obtain an aligned multimodal sequence, and performing cross-modal consistency analysis on the aligned multimodal sequence to obtain a modal coordination index; Performing threshold segmentation processing on the modal coordination index to obtain a preliminary behavior deviation mark; Performing cluster analysis processing on the preliminary behavior deviation marks to obtain the behavior deviation data, and performing user experience mapping processing on the behavior deviation data to obtain the performance feedback data; The multimodal digital human performance data and the performance feedback data are adaptively optimized using the behavior deviation data to obtain a target digital human.

7. The AIGC-based multimodal digital human generation method according to claim 6, characterized in that: The method of adaptively optimizing the multimodal digital human performance data and the performance feedback data by using the behavioral deviation data to obtain a target digital human includes: Performing importance scoring processing on the behavioral deviation data to obtain a deviation priority list; Performing semantic analysis on the performance feedback data to obtain a user preference feature vector, and performing weighted fusion processing on the deviation priority list and the user preference feature vector to obtain an adjustment target matrix; Performing expression correction processing on visual data in the multimodal digital human performance data based on the adjustment target matrix to obtain a corrected expression sequence; Performing pitch adjustment processing on the audio data in the multimodal digital human performance data based on the adjustment target matrix to obtain adjusted voice data; Performing posture fine-tuning processing on the action data in the multimodal digital human performance data based on the adjustment target matrix to obtain an adjustment action sequence; Performing time alignment processing on the modified expression sequence, the adjusted voice data and the adjusted action sequence to obtain preliminary adjusted multimodal data, and performing inter-modal correlation analysis processing on the preliminary adjusted multimodal data to obtain a correlation matrix; Threshold screening is performed on the correlation matrix to obtain modal coordination data, and multimodal fusion is performed on the modal coordination data to obtain the target digital human.

8. A multimodal digital human generation system based on AIGC, characterized in that: Used to execute the AIGC-based multimodal digital human generation method as described in any one of claims 1 to 7, the AIGC-based multimodal digital human generation system comprises: An enhancement module is used to perform data enhancement processing on the pre-acquired multimodal character data to obtain an enhanced multimodal data set, and perform feature extraction on the enhanced multimodal data set to obtain multimodal feature data; A processing module, used for performing modeling and dynamic weight adjustment processing on the multimodal feature data based on a graph structure neural network to obtain dynamic personality knowledge data; A matching module, used to perform intelligent matching processing on the preset target scene information and the dynamic personality knowledge data to obtain contextualized behavior instruction data; A generation module, used to generate hierarchical action sequences, optimize adaptive resolution, and perform speech synthesis processing on a preset initial digital human according to the contextualized behavior instruction data, so as to obtain multimodal digital human performance data; The identification module is used to identify behavioral deviations of the multimodal digital human performance data, obtain behavioral deviation data and performance feedback data, and adaptively optimize the multimodal digital human performance data and the performance feedback data through the behavioral deviation data to obtain a target digital human.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the AIGC-based multimodal digital human generation method as described in any one of claims 1 to 7 is implemented.

Images