Training method, system and medium for generating 5D digital human based on AIGC

Abstract

The present application relates to the field of data processing technology, and discloses a training method, system and medium for generating 5D digital humans based on AIGC. The method includes: preprocessing real-person feature data to obtain multi-dimensional preprocessed data; performing multi-level processing on the multi-dimensional preprocessed data to obtain 5D digital human generated data; performing cross-modal coordination processing on the 5D digital human generated data through a progressive multi-task learning algorithm to obtain digital human data to be processed; performing dynamic knowledge integration processing on the digital human data to be processed to obtain adaptive digital human data; performing personalized fine-tuning and style transfer processing on the adaptive digital human data to obtain target digital human data; performing real-time rendering and intelligent interactive processing on the target digital human data to obtain target 5D digital human data and generate a digital human training plan based on the target 5D digital human data. The present application improves the efficiency and accuracy of training for generating 5D digital humans based on AIGC.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a training method, system and medium for generating 5D digital humans based on AIGC. Background Art

[0002] Digital human training technology is a key link in achieving high-quality, intelligent digital humans. Traditional digital human training methods mainly rely on a large amount of manually annotated data and rule design, such as collecting real-life motion data through motion capture technology, or training speech models through speech synthesis technology. Although these methods can generate basically usable digital humans, the training process is time-consuming, costly, and difficult to adapt to complex and changing application scenarios. With the development of deep learning technology, digital human training methods based on neural networks have gradually emerged, such as using convolutional neural networks (CNNs) to train facial expression generation models, or using recurrent neural networks (RNNs) to train speech synthesis models. These methods have improved the efficiency and performance of digital human training to a certain extent.

[0003] However, the existing digital human training technology still has significant shortcomings. First, most training methods tend to focus on a single dimension (such as vision or voice) and lack comprehensive training of multi-dimensional features, resulting in the overall performance of the generated digital humans being less coordinated and natural. Secondly, the existing training models are usually static, and once the training is completed, it is difficult to adapt to new data and scenarios, and lack the ability to continuously learn and dynamically adapt. Furthermore, in terms of training efficiency, existing technologies often require a lot of computing resources and time, and it is difficult to meet the needs of rapid iteration and personalized customization. In addition, existing training methods generally lack in-depth modeling of the personalized characteristics of digital humans, making it difficult to generate digital humans with unique personality and style. Finally, in terms of real-time rendering and interactive training, existing technologies often find it difficult to achieve a good balance between model complexity and operating efficiency, which restricts the performance of digital humans in practical applications. Summary of the invention

[0004] The present application provides a training method, system and medium for generating 5D digital humans based on AIGC, which are used to improve the efficiency and accuracy of training for generating 5D digital humans based on AIGC.

[0005] In a first aspect, the present application provides a training method for generating 5D digital humans based on AIGC, and the training method for generating 5D digital humans based on AIGC includes: preprocessing pre-collected real-person feature data to obtain multi-dimensional preprocessed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits; performing multi-level processing on the multi-dimensional preprocessed data to obtain 5D digital human generation data, wherein the 5D digital human generation data includes: 3D morphological data, action expression data, voice synthesis data, personalized behavior data and knowledge graph; performing cross-modal coordination processing on the 5D digital human generation data through a progressive multi-task learning algorithm to obtain digital human data to be processed; performing dynamic knowledge integration processing on the digital human data to be processed through an adaptive neural architecture search algorithm to obtain adaptive digital human data; performing personalized fine-tuning and style transfer processing on the adaptive digital human data to obtain target digital human data; performing real-time rendering and intelligent interaction processing on the target digital human data through an adaptive LOD algorithm to obtain target 5D digital human data and generating a digital human training plan based on the target 5D digital human data.

[0006] In combination with the first aspect, in a first implementation of the first aspect of the present application, the pre-processing of the pre-collected real-person feature data to obtain multi-dimensional pre-processed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits includes: performing multimodal separation processing on the pre-collected real-person feature data to obtain original visual data, audio data and semantic data, and performing 3D reconstruction processing on the original visual data to obtain an initial static three-dimensional model; performing mesh optimization and texture mapping processing on the initial static three-dimensional model to obtain the static three-dimensional data; performing time series analysis and skeleton extraction processing on the original visual data to obtain original dynamic skeleton data , and perform motion trajectory smoothing and joint constraint processing on the original dynamic skeleton data to obtain the dynamic skeleton data; perform acoustic feature extraction and phoneme segmentation processing on the audio data to obtain initial speech features, and perform rhythmic pattern analysis and emotion labeling processing on the initial speech features to obtain the speech samples; perform natural language processing and knowledge graph construction on the semantic data to obtain initial character features, and perform personalized feature extraction and behavior pattern analysis on the initial character features to obtain the character portrait; perform multimodal alignment and integration processing on the static three-dimensional data, dynamic skeleton data, speech samples and character portraits to obtain the multi-dimensional pre-processed data.

[0007] In combination with the first aspect, in a second implementation of the first aspect of the present application, the multi-dimensional pre-processed data is multi-level processed to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph, including: topological structure optimization processing of the static three-dimensional data to obtain a high-precision 3D mesh model, and parameterized reconstruction processing of the high-precision 3D mesh model to obtain the 3D morphological data; time series interpolation processing of the dynamic skeleton data to obtain a continuous motion sequence, and through depth The spatiotemporal convolutional network performs expression mapping processing on the continuous motion sequence to obtain the action expression data; performs acoustic feature analysis processing on the speech sample to obtain a phoneme unit library, and performs context-sensitive splicing processing on the phoneme unit library to obtain the speech synthesis data; performs feature vectorization processing on the character portrait to obtain multi-dimensional behavior characteristics, and performs probabilistic graph model construction processing on the multi-dimensional behavior characteristics to obtain the personalized behavior data; performs semantic network construction processing on the character portrait to obtain an initial knowledge structure, and performs relational reasoning and expansion processing on the initial knowledge structure to obtain the knowledge graph.

[0008] In combination with the first aspect, in a third implementation of the first aspect of the present application, the cross-modal coordination processing of the 5D digital human generated data by the progressive multi-task learning algorithm to obtain the digital human data to be processed includes: performing task decomposition processing on the 5D digital human generated data to obtain multiple sub-task data sets, and performing difficulty evaluation analysis on the multiple sub-task data sets to obtain a task difficulty sequence; sorting the sub-task data sets according to the task difficulty sequence to obtain a progressive learning sequence, and performing feature extraction processing on the primary task data in the progressive learning sequence to obtain basic feature representation data; and performing task decomposition processing on the 5D digital human generated data by the progressive multi-task learning algorithm to obtain the basic feature representation data. The progressive multi-task learning algorithm iteratively optimizes the basic feature representation data to obtain multi-level feature representation data, and performs cross-modal attention mechanism processing on the multi-level feature representation data to obtain inter-modal correlation features; performs adaptive weight distribution processing on the inter-modal correlation features to obtain balanced feature representation data, and performs cross-modal consistency constraint processing on the balanced feature representation data to obtain coordinated feature data; performs progressive knowledge distillation processing on the coordinated feature data to obtain compressed feature representation data, and performs multimodal fusion processing on the compressed feature representation data to obtain the digital human data to be processed.

[0009] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, the dynamic knowledge integration processing of the digital human data to be processed by an adaptive neural architecture search algorithm to obtain adaptive digital human data includes: performing feature decomposition processing on the digital human data to be processed to obtain a multi-dimensional feature subspace, and performing neural architecture encoding processing on the multi-dimensional feature subspace to obtain a candidate architecture pool; evaluating the candidate architecture pool by an adaptive neural architecture search algorithm to obtain a preferred architecture sequence, and dynamically combining the preferred architecture sequence to obtain an adaptive neural network structure; performing parameter initialization processing on the adaptive neural network structure to obtain an initialization model, and performing incremental learning processing on the initialization model to obtain a dynamic knowledge representation; performing cross-modal knowledge fusion processing on the dynamic knowledge representation to obtain a comprehensive knowledge graph, and performing reasoning optimization processing on the comprehensive knowledge graph to obtain a reasoning enhanced knowledge base; performing personalized knowledge tailoring processing on the reasoning enhanced knowledge base to obtain a customized knowledge structure, and performing multimodal alignment processing on the customized knowledge structure to obtain the adaptive digital human data.

[0010] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, the adaptive digital human data is subjected to personalized fine-tuning and style transfer processing to obtain target digital human data, including: performing feature decoupling processing on the adaptive digital human data to obtain a multi-dimensional personality feature vector, and performing principal component analysis processing on the multi-dimensional personality feature vector to obtain core personality features; enhancing the core personality features through a conditional generative adversarial network algorithm to obtain personalized feature data, and performing semantic mapping processing on the personalized feature data to obtain personalized semantic representation data; performing multimodal cross-attention processing on the personalized semantic representation data to obtain cross-modal personality features, and performing style encoding processing on the cross-modal personality features to obtain a style feature vector; performing adaptive interpolation processing on the style feature vector to obtain a continuous style space, and performing nonlinear transformation processing on the continuous style space to obtain a style transfer matrix; performing tensor multiplication processing on the style transfer matrix and the cross-modal personality features to obtain stylized personality data; performing multimodal fusion processing on the stylized personality data to obtain the target digital human data.

[0011] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, the target digital human data is subjected to real-time rendering and intelligent interactive processing by an adaptive LOD algorithm to obtain target 5D digital human data and a digital human training program is generated according to the target 5D digital human data, including: performing multi-resolution decomposition processing on the target digital human data to obtain a hierarchical 5D data structure, and dynamically adjusting the details of the hierarchical 5D data structure by an adaptive LOD algorithm to obtain an adaptive rendering model; performing real-time ray tracing processing on the adaptive rendering model to obtain high-quality rendering data, and generating a digital human training program based on the target 5D digital human data. The method further comprises: performing physical engine simulation processing on the real-life interaction data to obtain realistic interaction data; performing multimodal fusion processing on the realistic interaction data to obtain comprehensive perception data, and performing situational perception analysis processing on the comprehensive perception data to obtain context-related information; performing decision tree generation processing on the context-related information to obtain an intelligent response strategy, and performing feedback loop optimization processing on the intelligent response strategy to obtain the target 5D digital human data; performing performance indicator extraction processing on the target 5D digital human data to obtain multi-dimensional evaluation indicators, and performing adaptive learning curve generation processing based on the multi-dimensional evaluation indicators to obtain the digital human training plan.

[0012] In a second aspect, the present application provides a training system for generating a 5D digital human based on AIGC, and the training system for generating a 5D digital human based on AIGC includes:

[0013] The data processing module is used to pre-process the pre-collected real person feature data to obtain multi-dimensional pre-processed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits;

[0014] A multi-layer processing module, used for performing multi-level processing on the multi-dimensional pre-processed data to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph;

[0015] A coordination processing module is used to perform cross-modal coordination processing on the 5D digital human generated data through a progressive multi-task learning algorithm to obtain the digital human data to be processed;

[0016] An integrated processing module, used for performing dynamic knowledge integration processing on the digital human data to be processed by an adaptive neural architecture search algorithm to obtain adaptive digital human data;

[0017] A migration processing module, used to perform personalized fine-tuning and style migration processing on the adaptive digital human data to obtain target digital human data;

[0018] The real-time rendering module is used to perform real-time rendering and intelligent interactive processing on the target digital human data through an adaptive LOD algorithm to obtain target 5D digital human data and generate a digital human training plan based on the target 5D digital human data.

[0019] A third aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned training method for generating a 5D digital human based on AIGC.

[0020] In the technical solution provided by the present application, by multimodal separation and preprocessing of pre-collected real-person feature data, multidimensional preprocessed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits are obtained, which lays a comprehensive and rich data foundation for the subsequent generation of 5D digital humans and ensures the authenticity and consistency of the generation results in various dimensions. The multidimensional preprocessed data is processed at multiple levels to obtain 5D digital human generation data including 3D morphological data, action expression data, voice synthesis data, personalized behavior data and knowledge graphs. This comprehensive data structure makes the generated digital human not only highly realistic visually, but also has rich facial expressions, natural voice expressions, personalized behavior patterns and deep knowledge reserves. The cross-modal coordination processing of 5D digital human generation data by the progressive multi-task learning algorithm effectively solves the coordination and consistency problems between different modal data and improves the overall coordination and naturalness of the generation results. The use of the adaptive neural architecture search algorithm to process the digital human data for dynamic knowledge integration greatly enhances the adaptability and generalization ability of the model, so that the generated digital human can better cope with complex and changing scenes and tasks. Personalized fine-tuning and style transfer processing of adaptive digital human data not only improves the personalization of digital humans, but also realizes rapid style conversion, greatly enhancing the diversity and customizability of generated results. Real-time rendering and intelligent interactive processing of target digital human data through adaptive LOD algorithm significantly improves rendering efficiency and interactive fluency while ensuring high-quality visual effects, so that the generated 5D digital human can achieve smooth real-time performance in various hardware environments. Finally, a digital human training plan is generated based on the target 5D digital human data, realizing closed-loop optimization of the training process, continuously improving the overall quality and performance of digital humans, and improving the efficiency and accuracy of training 5D digital humans based on AIGC. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 This is a schematic diagram of an embodiment of a training method for generating a 5D digital human based on AIGC in an embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of an embodiment of a training system for generating a 5D digital human based on AIGC in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application provide a training method, system and medium for generating 5D digital humans based on AIGC. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the training method for generating a 5D digital human based on AIGC includes:

[0026] Step S101, pre-processing the pre-collected real person feature data to obtain multi-dimensional pre-processed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits;

[0027] It is understandable that the execution subject of the present application can be a training system for generating 5D digital human based on AIGC, or a terminal or a server, which is not limited here. The embodiment of the present application is described by taking the server as the execution subject as an example.

[0028] Specifically, the collected real-life 3D scanning data is subjected to noise removal and mesh optimization. This step uses algorithms such as Gaussian filtering to eliminate abnormal points and surface noise generated during the scanning process, and then optimizes the model topology through mesh simplification and reconstruction algorithms to obtain high-quality static 3D data.

[0029] Processing motion capture data to generate dynamic skeleton data. This includes filtering and smoothing the raw motion data to remove jitter and unnatural movements. Then, the captured marker data is converted into skeleton motion data through the inverse kinematics algorithm and time series interpolation is performed to generate a continuous and smooth motion sequence. The preprocessing of speech samples involves noise reduction, segmentation and feature extraction of the recorded data. Background noise is removed using techniques such as spectral subtraction, and then the continuous speech is segmented into independent speech segments using the voice activity detection algorithm. Acoustic features such as Mel-frequency cepstral coefficients (MFCC) are extracted for each speech segment to form structured speech sample data. The generation of character portrait data requires structuring and quantifying the collected personality traits, behavioral habits, and knowledge background information. This includes using natural language processing technology to perform semantic analysis on text descriptions and convert qualitative features into quantitative indicators. At the same time, clustering and classification algorithms are used to perform pattern recognition on behavioral data to construct a multi-dimensional personality feature vector.

[0030] Throughout the preprocessing process, data standardization and alignment are crucial. Data of different modalities need to be converted to a unified coordinate system and time scale to ensure consistency in subsequent processing. In addition, all data are normalized to make the scales of different features comparable, which is conducive to the subsequent application of machine learning algorithms.

[0031] For example, when processing the real-person feature data of a virtual anchor, the static three-dimensional data may contain a high-precision facial model with more than 1 million vertices. Through the mesh simplification algorithm, the number of vertices can be reduced to about 200,000 while maintaining the accuracy of key facial features. Dynamic skeleton data may contain facial expression capture data at 60 frames per second. Through data smoothing and key frame extraction, the amount of data can be reduced by 50% without affecting the naturalness of the expression. Voice samples may contain several hours of raw recordings. After preprocessing, a structured voice library containing thousands of phoneme units can be extracted. Character portrait data may extract a personality feature vector containing 50 dimensions from hundreds of descriptive texts and behavior records.

[0032] Step S102: multi-level processing is performed on the multi-dimensional pre-processed data to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph;

[0033] Specifically, the static three-dimensional data is deeply processed to generate high-quality 3D morphological data. This includes using subdivision surface algorithms to refine the mesh model and increase the level of detail of the model. At the same time, parametric modeling technology is applied to convert the 3D model into an editable parametric representation to facilitate subsequent personalized adjustment and animation control. Dynamic bone data is combined with 3D morphological data to generate action expression data. This step uses a bone binding algorithm to map the bone motion data to the 3D model, and uses blend shapes technology (Blend Shapes) to achieve rich facial expressions. Through the interpolation algorithm, smooth transition animations can be generated between key frames to ensure the natural and smooth movements and expressions.

[0034] The generation of speech synthesis data is based on preprocessed speech samples, using deep learning models such as WaveNet or Tacotron to train personalized speech synthesizers. This process involves learning the mapping of phonemes to acoustic features and training the waveform generation network, which can ultimately generate natural and fluent speech based on the input text. The generation of personalized behavior data is based on character portrait data, and a behavior prediction model is built through machine learning algorithms. This includes using technologies such as decision trees, random forests, or neural networks to learn the complex relationship between personality traits and behavior patterns, so that personalized behavior responses can be generated in different scenarios. The construction of a knowledge graph organizes the knowledge background and context information of a person into a structured network. This involves natural language processing technologies such as entity recognition, relationship extraction, and ontology construction, which transform unstructured knowledge into a queryable and reasonable graph database. The consistency and coordination of data are crucial. It is necessary to ensure that data of different dimensions are synchronized in time and space, such as ensuring the precise correspondence between facial expressions and speech synthesis, and the reasonable matching of behavioral responses with knowledge background.

[0035] For example, when processing the 5D digital human generation data of a virtual news anchor, the 3D morphological data may contain a high-precision facial model consisting of 1 million polygons and 200 adjustable facial parameters. The motion expression data may include motion data of 50 facial bones and 100 mixed shapes for subtle expressions. The speech synthesis data may be based on 10 hours of recording samples to train a model that can generate high-quality speech at a 16kHz sampling rate and 16-bit depth. Personalized behavior data may include a behavior prediction model trained based on a 50-dimensional personality feature vector that can generate appropriate responses in 1,000 different scenarios. The knowledge graph may contain 10,000 entities and 50,000 relationships related to news reports, supporting real-time knowledge retrieval and reasoning.

[0036] Step S103, performing cross-modal coordination processing on the 5D digital human generated data through a progressive multi-task learning algorithm to obtain digital human data to be processed;

[0037] It should be noted that the concept of progressive learning is applied to the organization and arrangement of tasks. Starting from the most basic unimodal tasks, such as the reconstruction of 3D morphological data, gradually transition to more complex cross-modal tasks, such as the synchronization of action expressions and speech. This approach allows the model to build basic capabilities on simple tasks and then gradually adapt to more challenging tasks. The multi-task learning framework provides a way to handle multiple related tasks simultaneously. In this framework, data processing tasks of different modalities are regarded as interrelated learning objectives. For example, facial expression generation and speech synthesis are regarded as two related but independent tasks, and knowledge transfer and complementary learning are achieved by sharing the underlying feature representation.

[0038] First, the 5D digital human generation data is task-decomposed. 3D morphological data, action expression data, speech synthesis data, personalized behavior data, and knowledge graphs are divided into a series of subtasks. These subtasks are sorted according to difficulty and relevance to form a progressive learning sequence. Next, a multi-task neural network architecture is designed, which contains shared underlying feature extraction layers and task-specific output layers. The shared layers are responsible for learning common representations across modalities, while the task-specific layers focus on the unique features of each modality.

[0039] During the training process, a dynamic task scheduling strategy is adopted. In the initial stage, the model is only trained on simple single-modal tasks, such as static 3D morphology reconstruction. As the training progresses, more complex tasks are gradually introduced, such as the combination of action expression and 3D morphology, and then to the synchronization of voice and expression. This strategy ensures that the model can gradually adapt to more and more complex cross-modal coordination tasks. In order to deal with the imbalance problem between different modal data, a dynamic weight adjustment mechanism is introduced. By monitoring the learning progress and performance of each task, the loss weights of different tasks are dynamically adjusted to ensure that all modalities are fully learned and optimized.

[0040] In addition, a cross-modal attention mechanism is introduced, which enables the model to automatically pay attention to information from other related modalities when processing tasks in one modality. For example, when generating facial expressions, the model automatically pays attention to the speech content and emotional characteristics, thereby achieving more natural expression-speech synchronization.

[0041] For example, when processing the 5D digital human data of a virtual anchor, the application process of the progressive multi-task learning algorithm is as follows: First, the model trains a static reconstruction task on a 3D facial model with 200,000 vertices to learn basic geometric features. Then, the action data of 50 facial bones is introduced to train the model to learn facial animation. Next, a speech synthesis task is added, and the model is trained using 10 hours of speech samples to generate speech synchronized with facial movements. On this basis, a personalized behavior prediction task is introduced, and the model is trained using behavior data from 1,000 annotated scenes to predict appropriate reactions. Finally, the knowledge graph task is integrated to enable the model to generate rich content dialogues using a knowledge base of 10,000 entities and 50,000 relationships.

[0042] Step S104, performing dynamic knowledge integration processing on the digital human data to be processed by an adaptive neural architecture search algorithm to obtain adaptive digital human data;

[0043] Specifically, the adaptive neural architecture search algorithm analyzes the digital human data to be processed and determines the key tasks and performance indicators that need to be optimized. These tasks include but are not limited to 3D morphology generation, action expression synthesis, speech generation, personalized behavior prediction, and knowledge reasoning. Based on these tasks, the algorithm defines a search space containing different network layer types, connection methods, and hyperparameter selections. The algorithm starts an iterative search process. In each iteration, it generates multiple candidate neural network architectures and quickly evaluates them on a subset of the digital human data to be processed. Evaluation indicators include model performance, computational complexity, and memory usage. Through reinforcement learning or evolutionary algorithms, the search process gradually optimizes the architecture selection, tending to select architectures that perform balanced and efficiently on multiple tasks.

[0044] While searching for the architecture, the algorithm also performs dynamic knowledge integration. This process involves integrating knowledge from different sources (such as pre-trained models, expert rules, and real-time feedback) into the evolving neural network. Specifically, the algorithm uses knowledge distillation techniques to transfer the knowledge of the pre-trained model to the new architecture, while enabling the model to quickly adapt to new tasks and data distributions through meta-learning methods. In order to handle the multimodal characteristics of 5D digital human data, the algorithm pays special attention to the design of cross-modal architectures. It explores different multimodal fusion strategies, such as early fusion, mid-term fusion, and late fusion, to find the architecture that best suits multiple data types such as 3D morphology, motion expression, speech, and knowledge graphs.

[0045] In terms of knowledge graph integration, the algorithm explores graph neural network (GNN) structures to effectively process structured knowledge. It tries different types of graph convolutional layers and attention mechanisms to optimize the efficiency of knowledge reasoning and querying. As the search proceeds, the algorithm continuously updates and refines the neural network architecture. It adopts a progressive search strategy, first determining the macro architecture and then optimizing the micro structure, thereby striking a balance between search efficiency and architecture quality. At the same time, through online learning technology, the model is able to learn from continuously input data and continuously adjust its internal parameters and structure.

[0046] The adaptive neural architecture search algorithm outputs an optimized, adaptive neural network model that can effectively process the multimodal data of 5D digital humans and has the ability to dynamically integrate knowledge. This model not only performs well on predefined tasks, but also can flexibly respond to new scenarios and requirements.

[0047] For example, when dealing with a 5D digital human teaching assistant for the education field, the application process of the adaptive neural architecture search algorithm is as follows: the algorithm first analyzes 3D morphological data containing 1 million data points, 50 hours of voice data, 10,000 action expression sequences, and a knowledge graph containing 50,000 concepts in the education field. 20 different network layers and 100 connection modes are defined in the initial search space. After 1,000 rounds of iterative search, the algorithm finally determined a hybrid architecture containing a 3D convolutional neural network, a Transformer speech synthesis module, a temporal graph convolutional network, and a multi-head attention mechanism. This architecture has achieved excellent performance in multiple indicators such as 3D rendering quality, naturalness of voice, fluency of action, and accuracy of knowledge application. For example, in the knowledge question-answering task, the model can retrieve and synthesize relevant information from the knowledge graph within 0.1 seconds to generate accurate answers with an accuracy rate of 95%. Through dynamic knowledge integration, the model can also absorb new teaching content in real time, increasing the application accuracy of new knowledge points from 60% to 90% within 24 hours.

[0048] Step S105, performing personalized fine-tuning and style transfer processing on the adaptive digital human data to obtain target digital human data;

[0049] Specifically, the adaptive digital human data is feature decoupled and decomposed into different dimensions such as performance, behavior, voice, and knowledge. These dimensions are then adjusted based on predefined personality parameters such as extroversion, openness, and conscientiousness. For example, for a digital human with high extroversion, the richness of its facial expressions, the range of volume variation of its voice, and the tendency to actively provide information in conversations will be increased. This process uses conditional generative adversarial network (cGAN) technology to take personality parameters as conditional inputs to guide the generation process. Through adversarial training, the generator learns to produce data that conforms to a specific personality, while the discriminator learns to distinguish between real personalized performance and generated performance. This method is able to give the digital human a unique personality performance while maintaining its basic characteristics.

[0050] The style transfer process focuses on adjusting the visual appearance and speech characteristics of the digital human to match a specific artistic style or real-life reference. This process uses style transfer techniques from deep learning, such as the AdaIN (Adaptive InstanceNormalization) algorithm. For visual style, the statistical features of the target style image (such as color distribution and texture features) are applied to the 3D model and texture of the digital human. For speech style, a vocoder model is used to transfer the target speaker's voice characteristics (such as timbre and rhythm) to the digital human's speech synthesis model.

[0051] During the process, multimodal consistency constraints are also introduced to ensure that the style changes of different modalities (such as vision and speech) remain coordinated. For example, when the visual style becomes cartoonish, the speech is also adjusted accordingly to a more exaggerated and expressive style. In order to maintain the identity consistency of the digital human, an identity preservation loss function is used in the style transfer process. This ensures that the core features and recognizability of the digital human are retained even after applying a new visual or speech style. The whole process is iterative, and through multiple fine-tuning and style adjustments, it gradually approaches the target effect. After each iteration, a quality assessment is performed, including the evaluation of indicators such as visual consistency, behavioral naturalness, and speech fluency to guide the next round of adjustments.

[0052] For example, when creating a virtual customer service digital human for an e-commerce platform, the application of personalized fine-tuning and style transfer processing is as follows: First, based on the platform's brand image and target customer base, a set of personality parameters are set, including high affinity (0.8 / 1.0), moderate extroversion (0.6 / 1.0), and high conscientiousness (0.9 / 1.0). Using these parameters, the adaptive digital human data is fine-tuned through the cGAN model. The results show that the fine-tuned digital human shows more smiling expressions in customer interactions (from 3 times per minute to 5 times per minute), uses more positive vocabulary (the frequency of positive vocabulary usage increased by 20%), and can provide product information more proactively (the rate of proactive information provision increased from 50% to 75%).

[0053] In terms of style transfer, a popular real customer service representative was selected as a reference. Using the AdaIN algorithm, the visual style of the real customer service representative (such as makeup and hairstyle features) was applied to the 3D model of the digital human. This process involved processing a 3D model with 1 million vertices and a texture map with 4K resolution, and ultimately made the appearance of the digital human highly consistent with the brand image (the visual similarity score increased from 0.6 to 0.85). At the same time, the voice features of the real customer service representative were transferred to the voice model of the digital human using a vocoder model, and parameters such as the base frequency, resonance peak, and speech rate were adjusted to make the digital human's voice more friendly and brand-specific (the voice naturalness score increased from 3.5 / 5 to 4.2 / 5). The target digital human data obtained in the end is not only highly consistent with the brand image in appearance and voice, but also shows personality characteristics that match the customer service role, enabling more effective interaction with customers and providing a personalized shopping experience. This example demonstrates the important role of personalized fine-tuning and style transfer in creating unique and targeted 5D digital humans, as well as the potential of AIGC technology in achieving highly customized digital humans.

[0054] Step S106: Real-time rendering and intelligent interactive processing are performed on the target digital human data through an adaptive LOD algorithm to obtain the target 5D digital human data and generate a digital human training plan based on the target 5D digital human data.

[0055] Specifically, the core idea of ​​the adaptive LOD algorithm is to dynamically adjust the level of detail of the digital human model according to the viewing distance, hardware performance and scene complexity. For the target digital human data, multi-resolution decomposition is first performed to create a hierarchical 5D data structure with different levels of detail. This structure includes 3D geometric models, texture maps, animation bone data and expression blending shapes from high precision to low precision. During real-time rendering, the adaptive LOD algorithm dynamically selects the most appropriate level of detail according to the current viewing angle and computing resources. For example, when the digital human appears at a close distance, a high-precision model and high-resolution texture are used; while at a long distance, a low-precision model and low-resolution texture are switched. This dynamic adjustment not only ensures visual quality, but also optimizes rendering performance. In order to achieve smooth LOD transition, the algorithm uses progressive mesh simplification technology and texture mapping processing. In the process of geometric simplification, the edge collapse operation is used to gradually reduce the number of polygons while maintaining the overall shape of the model. Texture mapping pre-generates textures of different resolutions, and selects the appropriate texture mapping level according to the distance during rendering.

[0056] In terms of intelligent interaction processing, the algorithm integrates technologies such as natural language processing, emotion recognition, and context understanding. First, semantic analysis is performed on the input voice or text to extract key information and intent. Then, appropriate responses are generated by combining the digital human's knowledge graph and personalized behavior model. This process also takes into account the current context and historical interaction records to ensure the consistency and personalization of the response. In order to enhance the realism of the interaction, the algorithm also implements real-time expression and body movement generation. By analyzing the emotional characteristics and semantic content of the voice, facial expressions and body postures that match it are automatically generated. These actions are applied to the 3D model in real time through skeletal animation and blended shape technology, making the digital human's reactions more natural and vivid.

[0057] During the processing, performance optimization technologies such as GPU acceleration and multi-threaded processing are also introduced to ensure a smooth interactive experience even in complex scenarios. At the same time, adaptive streaming technology is used to dynamically adjust the quality of data transmission according to network conditions and optimize performance in online application scenarios. Finally, a digital human training program is generated based on the data collected during real-time rendering and interaction. This includes analyzing user interaction data, identifying areas where digital humans perform poorly, such as misunderstandings, inappropriate emotional expressions, etc., and automatically generating targeted training tasks and data sets for further optimizing digital human models.

[0058] For example, when developing a 5D digital human doctor for telemedicine consultation, the application of adaptive LOD algorithm and intelligent interaction processing is as follows: The 3D model of the digital human doctor originally contained 2 million polygons and 8K resolution texture. Through the adaptive LOD algorithm, 5 levels of detail were created, which were gradually simplified from 2 million polygons to 50,000 polygons, and the texture resolution was correspondingly reduced from 8K to 512x512. In actual application, when the user performs a full-body examination, the algorithm selects a 1 million polygon model and 4K texture; when interacting with a close-up of the face, a full-precision model is used. This dynamic adjustment increases the rendering frame rate from the original 25fps to a stable 60fps, while maintaining visual quality at different interaction distances.

[0059] In terms of intelligent interaction, the system integrates a knowledge graph containing 100,000 medical terms and 5,000 common diseases. When the patient describes the symptoms, the natural language processing module first converts the input into structured data, and then infers through the knowledge graph to generate diagnostic suggestions and consultation plans. For example, when the patient mentions "headache, fever and cough", the system can analyze the symptom combination within 0.5 seconds, generate 10 possible diagnostic hypotheses, and formulate an interaction plan containing 20 follow-up questions. At the same time, based on the recognition of the patient's anxiety through voice analysis, the digital human doctor automatically adjusts the facial expression to a gentler and more compassionate state, slows down the speech rate, and shows 15% more soothing body language such as smiles and nods. The target 5D digital human data not only achieves high-quality real-time rendering visually, but also demonstrates intelligent and personalized interaction capabilities in terms of function. Based on the interaction data of each consultation, the system automatically generates training plans, such as designing new training samples for misidentified symptom combinations, or creating a special facial animation data set to improve emotional expression in specific situations. This example demonstrates the key role of adaptive LOD algorithms and intelligent interaction processing in creating highly realistic, feature-rich 5D digital humans, as well as the potential of AIGC technology in realizing complex, professional digital human applications.

[0060] In the embodiment of the present application, by performing multimodal separation and preprocessing on the pre-collected real-person feature data, multi-dimensional preprocessed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits are obtained, which lays a comprehensive and rich data foundation for the subsequent 5D digital human generation, ensuring the authenticity and consistency of the generation results in various dimensions. The multi-dimensional preprocessed data is processed at multiple levels to obtain 5D digital human generation data including 3D morphological data, action expression data, voice synthesis data, personalized behavior data and knowledge graph. This comprehensive data structure makes the generated digital human not only highly realistic visually, but also has rich facial expressions, natural voice expressions, personalized behavior patterns and deep knowledge reserves. The cross-modal coordination processing of 5D digital human generation data by the progressive multi-task learning algorithm effectively solves the coordination and consistency problem between different modal data, and improves the overall coordination and naturalness of the generation results. The use of the adaptive neural architecture search algorithm to process the digital human data for dynamic knowledge integration processing greatly enhances the adaptability and generalization ability of the model, so that the generated digital human can better cope with complex and changeable scenes and tasks. Personalized fine-tuning and style transfer processing of adaptive digital human data not only improves the personalization of digital humans, but also realizes rapid style conversion, greatly enhancing the diversity and customizability of generated results. Real-time rendering and intelligent interactive processing of target digital human data through adaptive LOD algorithm significantly improves rendering efficiency and interactive fluency while ensuring high-quality visual effects, so that the generated 5D digital human can achieve smooth real-time performance in various hardware environments. Finally, a digital human training plan is generated based on the target 5D digital human data, realizing closed-loop optimization of the training process, continuously improving the overall quality and performance of digital humans, and improving the efficiency and accuracy of training 5D digital humans based on AIGC.

[0061] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0062] (1) Perform multimodal separation processing on the pre-collected real-person feature data to obtain original visual data, audio data, and semantic data, and perform 3D reconstruction processing on the original visual data to obtain an initial static 3D model;

[0063] (2) Perform mesh optimization and texture mapping on the initial static 3D model to obtain static 3D data;

[0064] (3) Performing time series analysis and skeleton extraction processing on the original visual data to obtain original dynamic skeleton data, and performing motion trajectory smoothing and joint constraint processing on the original dynamic skeleton data to obtain dynamic skeleton data;

[0065] (4) Perform acoustic feature extraction and phoneme segmentation on the audio data to obtain initial speech features, and perform prosodic pattern analysis and emotion labeling on the initial speech features to obtain speech samples;

[0066] (5) Perform natural language processing and knowledge graph construction on the semantic data to obtain initial character features, and then perform personalized feature extraction and behavior pattern analysis on the initial character features to obtain a character portrait;

[0067] (6) Perform multimodal alignment and integration processing on static three-dimensional data, dynamic skeleton data, voice samples and character portraits to obtain multi-dimensional pre-processed data.

[0068] Specifically, in the training method for generating 5D digital humans based on AIGC, multimodal separation and processing of pre-collected real-person feature data is a complex and critical step. First, the multimodal separation process separates the raw data into three main modalities: visual, audio, and semantic. Visual data includes high-resolution images and video sequences, audio data contains voice recordings, and semantic data is personality descriptions and background information in text form. For visual data, 3D reconstruction processing is first performed to obtain an initial static 3D model. This process uses multi-view stereo vision technology to infer 3D structures by analyzing 2D images from different angles. Specifically, structured light scanning or photogrammetry is used to capture images of the target from multiple angles, and then feature point matching and depth estimation algorithms are used to reconstruct 3D point clouds. Subsequently, the point cloud is converted into a mesh model through a Poisson surface reconstruction algorithm to obtain an initial static 3D model. The initial static 3D model is meshed and texture mapped. Mesh optimization includes simplification and subdivision operations, and quadrilateral retopology technology is used to optimize the mesh structure to improve rendering efficiency and animation deformation quality. Texture mapping uses UV unfolding technology to map the surface of a 3D model to a 2D plane, and then applies a high-resolution texture image to the model to ensure accurate restoration of details.

[0069] For dynamic data processing, the original visual data is analyzed in time series, and the movement of key points is tracked using the optical flow estimation algorithm. Then, the original dynamic skeleton data is obtained through the skeleton extraction algorithm, such as centerline extraction or learning-based posture estimation method. The motion trajectory of these data is smoothed, and the Kalman filter and other technologies are used to eliminate jitter and noise. At the same time, joint constraint processing is applied to ensure that the skeleton movement conforms to the physiological characteristics of the human body and avoids unnatural distortion or interlacing.

[0070] Audio data processing first extracts acoustic features, including calculating Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), and energy. Then, the dynamic time warping (DTW) algorithm is used to perform phoneme segmentation, dividing continuous speech into discrete phoneme units. The initial speech features obtained are analyzed for prosodic patterns to identify stress, pauses, and intonation changes, and sentiment is annotated using support vector machines (SVM) or deep learning models to add sentiment attribute labels to each speech segment. Semantic data processing involves natural language processing technology. First, the text is preprocessed using basic NLP technologies such as word segmentation, part-of-speech tagging, and named entity recognition. Then, the semantic features of the text are extracted using topic models (such as LDA) and word embedding technology. On this basis, a knowledge graph is constructed, and the relationship between entities is identified using a relationship extraction algorithm to form a structured knowledge representation. Personalized feature extraction is performed on the initial character features, and psychological theories such as the Big Five personality model are used to quantify personality characteristics. At the same time, behavioral data is analyzed using a sequence pattern mining algorithm to identify specific behavioral patterns and habits. Multimodal alignment and integration are performed on static 3D data, dynamic skeleton data, voice samples, and character portraits. This step uses timestamps to align data of different modalities to ensure their consistency in timing. Then, multimodal fusion techniques such as attention mechanisms or tensor decomposition methods are used to integrate features of different modalities into a unified representation space to form the final multi-dimensional pre-processed data.

[0071] For example, when creating a 5D digital human model for a virtual anchor, the preprocessing process is as follows: First, 500 high-resolution (4K) full-body photos and 10 minutes of 4K video data of the anchor were collected as visual input. Through multi-view 3D reconstruction, a high-precision static 3D model with 2 million vertices was generated. After mesh optimization, the number of vertices was reduced to 500,000 while maintaining the accuracy of key features. The use of 8K resolution texture maps ensures the realistic restoration of details such as skin and hair. For dynamic data, 60 frames per second of motion data were extracted from the video, and 206 key bone points were identified. Through motion smoothing, 95% of tiny jitters were eliminated, making the movements smoother. Joint constraint processing ensures that all joint movements are within the physiologically allowed range, enhancing the realism of the animation.

[0072] In terms of audio processing, 2 hours of speech samples were collected and 13-dimensional MFCC features and fundamental frequency contours were extracted. Phoneme segmentation obtained about 100,000 phoneme units, covering all the pronunciation characteristics of the anchor. Prosody analysis identified 20 common intonation patterns, and emotional annotation classified the speech samples into 6 basic emotional states. In semantic data processing, 500 program scripts of the anchor in the past year were analyzed, 5,000 key concepts and 3,000 entity relationships were extracted, and a graph containing the anchor's professional field knowledge was constructed. Personalized feature analysis obtained the anchor's specific scores in the Big Five personality model, such as extraversion 0.8, openness 0.7, etc. Behavioral pattern analysis identified 30 typical expressions and interactive habits. Finally, through multimodal alignment and integration, a unified 5D digital human data representation was generated, including a 3D model with 500,000 vertices, 206 dynamic skeleton points, 100,000 phoneme units with emotional annotations, and a personalized knowledge graph with 5,000 nodes. These data are time-aligned to an accuracy of 1 / 60 second, ensuring consistency in all dimensions during subsequent generation.

[0073] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0074] (1) Optimize the topological structure of static three-dimensional data to obtain a high-precision 3D mesh model, and perform parameterized reconstruction on the high-precision 3D mesh model to obtain 3D morphological data;

[0075] (2) Perform temporal interpolation processing on the dynamic skeleton data to obtain a continuous motion sequence, and perform expression mapping on the continuous motion sequence through a deep spatiotemporal convolutional network to obtain action expression data;

[0076] (3) Perform acoustic feature analysis on the speech sample to obtain a phoneme unit library, and perform context-sensitive splicing on the phoneme unit library to obtain speech synthesis data;

[0077] (4) Perform feature vectorization on the character portrait to obtain multi-dimensional behavioral features, and construct a probabilistic graphical model on the multi-dimensional behavioral features to obtain personalized behavioral data;

[0078] (5) Perform semantic network construction on the character portrait to obtain the initial knowledge structure, and then perform relational reasoning and expansion on the initial knowledge structure to obtain a knowledge graph.

[0079] Specifically, the static three-dimensional data is topologically optimized. This step uses a quadrilateral retopology algorithm to convert the original irregular triangular mesh into a more regular quadrilateral mesh structure. This optimization not only improves the geometric quality of the model, but also lays the foundation for subsequent animation deformation and texture mapping. Specifically, the algorithm first identifies the feature lines and key points of the model, then constructs quadrilateral patches based on these features, and finally adjusts the shape and distribution of the patches through iterative optimization to obtain a high-precision 3D mesh model that is uniform and conforms to the topological structure of the human body. The high-precision 3D mesh model is parametrically reconstructed. This step converts the 3D model into an editable parametric representation, making it easier to personalize and animate the model. The parametric reconstruction process includes UV unfolding, feature point annotation, and deformation operator definition. UV unfolding maps the 3D surface to a 2D plane for texture mapping; feature point annotation defines the control points of key areas such as the face and body; and deformation operators define how to change the shape of the model through parameter adjustment. These processing results together constitute the 3D morphological data.

[0080] For dynamic skeleton data, temporal interpolation is first performed. This step uses a spline interpolation algorithm, such as cubic Hermite spline interpolation, to generate smooth intermediate frames between known key frames to obtain a continuous motion sequence. This interpolation not only ensures the smoothness of the action, but also increases the frame rate while maintaining the original action characteristics, thereby improving the delicacy of the animation. Expression mapping is performed on the continuous motion sequence through a deep spatiotemporal convolutional network. This network structure is designed to capture the spatiotemporal features in the action sequence and map them to detailed facial expressions. The network contains 3D convolutional layers for processing information in spatial and temporal dimensions simultaneously; an attention mechanism for focusing on important facial areas; and a generative adversarial network (GAN) structure for generating realistic expression details. In this way, the motion data is converted into motion expression data containing rich expression information.

[0081] The processing of speech samples begins with acoustic feature analysis. This step extracts multi-dimensional acoustic features including Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy, etc. These features together describe the speaker's voice characteristics, such as timbre, rhythm, etc. Based on these features, a clustering algorithm is used to segment the speech samples into discrete phoneme units to form a phoneme unit library. The phoneme unit library is subjected to context-sensitive splicing. This step uses a unit selection algorithm to consider the context before and after the phoneme and select the most suitable unit from the phoneme unit library for splicing. At the same time, a prosodic model is used to adjust the duration, pitch, and energy of the phoneme to generate natural and fluent speech. This process also includes smoothing to ensure that the transition between phonemes is natural. The resulting speech synthesis data can reflect the speaker's unique voice characteristics and speaking style.

[0082] The first step in processing character portraits is feature vectorization. This step converts the character description in text form into a numerical feature vector. Natural language processing techniques, such as word embedding and sentence encoders, are used to extract the semantic features of the text. At the same time, psychological models, such as the Big Five personality model, are applied to quantify the character characteristics into specific numerical values. These processing results form multidimensional behavioral characteristics. Probabilistic graph models are constructed for multidimensional behavioral characteristics. This step uses probabilistic graph models such as Bayesian networks or Markov random fields to capture the interdependencies between behavioral characteristics. The model can predict the most likely behavioral response under given conditions by learning the statistical laws in historical data. This processing method makes the generated personalized behavioral data both conform to statistical laws and retain a certain degree of randomness, making the behavior of digital people more natural and diverse. Finally, the semantic network construction process is performed on the character portrait to obtain the initial knowledge structure. This step uses natural language processing and knowledge representation techniques, such as entity recognition and relationship extraction, to convert unstructured text information into a structured knowledge network. Next, the initial knowledge structure is processed for relational reasoning and expansion. This process uses the reasoning engine to perform logical deduction based on existing knowledge to discover potential relationships; at the same time, through knowledge graph completion technology, such as path sorting algorithms or neural network models, missing links in the knowledge graph are predicted and filled in. The resulting knowledge graph not only contains explicitly stated information, but also covers implicit knowledge associations.

[0083] For example, when creating a 5D digital human model for a virtual educational assistant, the process is as follows: First, the static three-dimensional data containing 500,000 vertices is topologically optimized to generate a high-precision 3D mesh model consisting of 200,000 quadrilateral faces. Through parametric reconstruction, 500 control points and 50 deformation operators are defined, enabling the model to accurately simulate various facial expressions and body postures. For dynamic skeletal data, the original 30fps motion capture data is upgraded to a 120fps continuous motion sequence through temporal interpolation processing, ensuring the ultimate smoothness of the movement. Using a deep neural network with 20 layers of spatiotemporal convolution, these action sequences are mapped to the fine control data of 52 facial muscles, generating rich action expression data including micro-expressions.

[0084] In terms of speech processing, about 100,000 phoneme units were extracted from the 10-hour original recording, each of which contained 39-dimensional MFCC features and fundamental frequency contours. Natural speech synthesis was achieved through a context-sensitive unit selection algorithm, and the synthesized speech achieved a naturalness score of 95% in a blind listening test.

[0085] For the character portrait, a 200-dimensional behavioral feature vector was extracted, covering personality traits, knowledge background, behavioral preferences, etc. A probabilistic graph model was constructed using a Bayesian network containing 1,000 nodes, which can generate behavioral responses that match the character characteristics in different scenarios, such as showing 90% patience and 85% explanation clarity in teaching scenarios. Finally, an initial knowledge structure containing 50,000 concept nodes and 100,000 relationship edges was constructed based on the character portrait. Through relational reasoning and expansion, the knowledge graph was expanded to 80,000 nodes and 200,000 relationship edges, covering a wide range of knowledge in the field of education.

[0086] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0087] (1) Perform task decomposition processing on the data generated by 5D digital humans to obtain multiple subtask data sets, and perform difficulty evaluation analysis on the multiple subtask data sets to obtain a task difficulty sequence;

[0088] (2) Sort the subtask data sets according to the task difficulty sequence to obtain a progressive learning sequence, and perform feature extraction on the primary task data in the progressive learning sequence to obtain basic feature representation data;

[0089] (3) The basic feature representation data is iteratively optimized through a progressive multi-task learning algorithm to obtain multi-level feature representation data, and the multi-level feature representation data is processed through a cross-modal attention mechanism to obtain inter-modal correlation features;

[0090] (4) Adaptively weight the inter-modal correlation features to obtain balanced feature representation data, and perform cross-modal consistency constraint processing on the balanced feature representation data to obtain coordinated feature data;

[0091] (5) Perform progressive knowledge distillation on the coordinated feature data to obtain compressed feature representation data, and perform multimodal fusion processing on the compressed feature representation data to obtain the digital human data to be processed.

[0092] Specifically, the 5D digital human generation data is task-decomposed and the overall task is decomposed into multiple subtasks, such as 3D shape generation, action expression synthesis, speech generation, personalized behavior prediction, and knowledge reasoning. Each subtask corresponds to a data subset. For example, the 3D shape generation task may contain 500,000 vertex mesh data and corresponding texture information, and the action expression synthesis task may contain 100,000 frames of action sequence data. The difficulty of these subtask data sets is evaluated and analyzed. The difficulty evaluation can be based on multiple indicators, such as data complexity, computing resource requirements, and expected accuracy. For example, information entropy is used to measure data complexity, and the average information entropy of each subtask data set is calculated. For the 3D shape generation task, the information entropy value that may be obtained is 8.5 bits / vertex, while for a simple speech unit recognition task, it may be only 3.2 bits / phoneme. Based on these indicators, a difficulty score is assigned to each subtask to form a task difficulty sequence.

[0093] According to the task difficulty sequence, the subtask data sets are sorted and processed to obtain a progressive learning sequence. This sequence gradually transitions from simple tasks to complex tasks. For example, the possible order is: basic speech unit recognition (difficulty score 2.5) → static 3D morphology reconstruction (difficulty score 4.0) → simple expression generation (difficulty score 5.5) → action sequence synthesis (difficulty score 7.0) → complex expression and speech synchronization (difficulty score 8.5) → personalized behavior generation (difficulty score 9.0).

[0094] The primary task data in the progressive learning sequence is processed for feature extraction to obtain basic feature representation data. For example, for speech data, 13-dimensional MFCC features are extracted; for 3D morphological data, geometric features such as curvature, normal, etc. are extracted; for expression data, facial key point positions and motion vectors are extracted. These basic features form the starting point for subsequent learning. The basic feature representation data is iteratively optimized through a progressive multi-task learning algorithm. This process involves designing a neural network architecture with a shared bottom layer and task-specific top layers. For example, a network bottom layer with 10 shared convolutional layers can be used, and then 2-3 specific fully connected layers are designed for each subtask. During the training process, only the shared layers are trained using data from the simplest task first, and then more complex tasks are gradually added, while fine-tuning the shared layers and training the corresponding task-specific layers. The resulting multi-level feature representation data can adapt to multiple related tasks at the same time.

[0095] The multi-level feature representation data is processed by a cross-modal attention mechanism to obtain inter-modal correlation features. This step uses a multi-head attention mechanism to allow the model to establish associations between different modalities. For example, when processing facial expressions and speech data, the attention mechanism can learn the correspondence between mouth shapes and phonemes, or the connection between expressions and intonation. Specifically, 8 attention heads can be set, each of which is responsible for capturing different types of cross-modal relationships.

[0096] The inter-modal correlation features are adaptively weighted to obtain balanced feature representation data. This step dynamically adjusts the weights of different modalities according to their importance in the current task. For example, when generating a speaking animation, a weight of 0.6 may be given to the speech feature, a weight of 0.3 to the facial action feature, and a weight of 0.1 to other features. These weights are automatically learned and adjusted through back-propagation. The balanced feature representation data is subjected to cross-modal consistency constraints to obtain coordinated feature data. This step introduces a consistency loss function to ensure that the outputs generated by different modalities are semantically consistent. For example, ensuring that the generated facial expressions match the speech content emotionally, or ensuring that the body movements are semantically coordinated with the conversation content. The coordinated feature data is then subjected to progressive knowledge distillation to obtain compressed feature representation data. This process involves training a smaller and more efficient student model to imitate the previously trained large teacher model. For example, a large model with 100 million parameters can be compressed to a small model with only 10 million parameters while maintaining more than 90% of the performance. This process is completed in multiple rounds of iterations, each focusing on a task of a specific difficulty level.

[0097] Finally, multimodal fusion processing is performed on the compressed feature representation data to obtain the digital human data to be processed. This step uses techniques such as tensor fusion or attention fusion to integrate features of different modalities into a unified representation. For example, a three-way tensor fusion layer can be used to process visual, speech, and semantic features at the same time to generate a 300-dimensional fusion vector as the final digital human representation.

[0098] For example, when creating a 5D digital human model for a virtual anchor, the task is first decomposed into five subtasks: static 3D modeling, facial expression generation, speech synthesis, body movement generation, and dialogue generation. After difficulty evaluation, the difficulty sequence is: static 3D modeling (3.5) → speech synthesis (5.0) → facial expression generation (6.5) → body movement generation (8.0) → dialogue generation (9.5). A progressive learning sequence is constructed in this order. Starting from static 3D modeling, geometric features of 500,000 vertices and texture features of 8K resolution are extracted to form a basic feature representation. Through 10 rounds of iterative training, more complex tasks are gradually introduced. For example, in the third round, the facial expression generation task is added, using the dynamic features of 52 facial key points; in the fifth round, speech synthesis is added, and 13-dimensional MFCC features and speech emotion annotation are introduced.

[0099] In cross-modal attention processing, it was found that there was a strong correlation between facial expressions and speech prosody, with a correlation coefficient of 0.85. The results of adaptive weight allocation showed that in the comprehensive performance, the weight of visual features was 0.4, the weight of speech features was 0.3, and the weight of semantic features was 0.3. The consistency constraint ensured the consistency of the generated expressions, speech, and dialogue content in emotional expression, with a consistency score of 95%.

[0100] Knowledge distillation compressed the original 500MB model to 50MB while maintaining 97% of the performance. The final multimodal fusion generated a 1024-dimensional vector that fully characterized the visual, auditory, and semantic features of the virtual anchor.

[0101] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0102] (1) Perform feature decomposition on the digital human data to be processed to obtain a multi-dimensional feature subspace, and perform neural architecture encoding on the multi-dimensional feature subspace to obtain a candidate architecture pool;

[0103] (2) The candidate architecture pool is evaluated by an adaptive neural architecture search algorithm to obtain a preferred architecture sequence, and the preferred architecture sequence is dynamically combined to obtain an adaptive neural network structure;

[0104] (3) Initializing the parameters of the adaptive neural network structure to obtain an initialization model, and performing incremental learning on the initialization model to obtain dynamic knowledge representation;

[0105] (4) Perform cross-modal knowledge fusion processing on the dynamic knowledge representation to obtain a comprehensive knowledge graph, and perform reasoning optimization processing on the comprehensive knowledge graph to obtain a reasoning enhanced knowledge base;

[0106] (5) Perform personalized knowledge tailoring on the reasoning enhanced knowledge base to obtain a customized knowledge structure, and perform multimodal alignment on the customized knowledge structure to obtain adaptive digital human data.

[0107] Specifically, the digital human data to be processed is subjected to feature decomposition. This step uses techniques such as principal component analysis (PCA) and independent component analysis (ICA) to decompose high-dimensional data into multiple low-dimensional feature subspaces. For example, for a 3D model data containing 1 million vertices, 500 principal components may be obtained through PCA, and each principal component represents a feature subspace. At the same time, for action data, 50 independent motion patterns may be obtained, and each pattern constitutes a feature subspace.

[0108] These multi-dimensional feature subspaces are processed by neural architecture encoding. This step uses encoding strategies to represent different neural network structures as vectors of fixed length. For example, a 100-bit binary string can be used to encode the number of layers in the network, the type of each layer (such as convolution, full connection, LSTM, etc.), activation function, and connection method. Through this encoding, 10,000 different candidate architectures may be generated to form a candidate architecture pool.

[0109] Then, the candidate architecture pool is evaluated by an adaptive neural architecture search algorithm. This step uses reinforcement learning or evolutionary algorithms to explore the architecture space. For example, the ENAS (Efficient Neural Architecture Search) algorithm is used to evaluate candidate architectures in parallel on 100 GPUs, each architecture is trained for 50 epochs, and its performance on the validation set is evaluated. After 1,000 rounds of search iterations, 100 best-performing architectures may be obtained to form a sequence of preferred architectures. The preferred architecture sequence is dynamically combined to obtain an adaptive neural network structure. This step uses a dynamic routing mechanism to allow flexible switching of different sub-networks under different tasks and input conditions. For example, a combined structure containing 5 sub-networks can be designed, each of which specializes in processing a modality (such as 3D morphology, expression, voice, etc.), and a controller network dynamically determines the degree of activation of each sub-network.

[0110] The parameters of the adaptive neural network structure are initialized to obtain the initialized model. Here, methods such as He initialization are used to set the initial weights according to the input and output dimensions of each layer. For example, for a network with 10 million parameters, it may be necessary to generate 10 million random numbers that conform to a specific distribution as initial weights. The initialized model is incrementally learned to obtain a dynamic knowledge representation. This step uses an online learning algorithm, such as a variant of stochastic gradient descent, to allow the model to be continuously updated when new data arrives. For example, a sliding window of 1000 samples can be set to update the model parameters whenever new training data arrives to maintain the real-time adaptability of the model.

[0111] Perform cross-modal knowledge fusion processing on the dynamic knowledge representation to obtain a comprehensive knowledge graph. This step uses graph neural network (GNN) technology to fuse knowledge representations of different modalities into a unified graph structure. For example, a knowledge graph containing 100,000 nodes and 500,000 edges can be constructed, where nodes represent concepts and edges represent relationships. Knowledge of different modalities is fused by sharing nodes and edges. Perform reasoning optimization processing on the comprehensive knowledge graph to obtain a reasoning enhanced knowledge base. This step uses a combination of symbolic reasoning and neural reasoning to expand and optimize the knowledge graph. For example, using knowledge graph embedding methods such as TransE to learn low-dimensional representations of entities and relationships, and then performing link prediction based on these representations, 10,000 new potential relationships may be discovered.

[0112] The reasoning-enhanced knowledge base is personalized and tailored to obtain a customized knowledge structure. This step selectively retains and organizes knowledge based on specific application scenarios and user needs. For example, for a digital human dedicated to the field of science and education, 50,000 concepts related to science and education may be retained, while knowledge in other fields may be appropriately simplified.

[0113] Finally, the customized knowledge structure is subjected to multimodal alignment to obtain adaptive digital human data. This step ensures that data of different modalities remain consistent in semantics and timing. For example, the attention mechanism is used to establish the correspondence between 3D motion data, voice data, and semantic knowledge, ensuring that facial expressions, voice content, and semantic understanding can be accurately synchronized during the generation process.

[0114] For example, when creating a 5D digital human model for a multilingual virtual tour guide, the collected data is first feature decomposed. The 3D model data is PCA to obtain 300 main morphological features, the motion data is ICA to obtain 100 basic motion patterns, the speech data extracts 50 timbre features and 20 rhythmic patterns, and the semantic data identifies 1000 core concepts. These features constitute a multi-dimensional feature subspace. During the neural architecture search process, 5000 candidate architectures are generated, each encoded with a 200-bit binary string. The ENAS algorithm is used to evaluate in parallel on 50 GPUs, and each architecture is trained for 100 epochs. Finally, 10 architectures with the best performance are selected to form a preferred architecture sequence. These architectures are dynamically combined into an adaptive network, which contains 5 specialized subnetworks: 3D morphology generation, motion synthesis, speech synthesis, language understanding, and multilingual translation. The initialized model contains 50 million parameters, and the initial weights are set using He initialization. Through incremental learning, the model can adapt to new language and cultural knowledge in real time. For example, 1,000 new user interaction data are processed every day to update model parameters. In the knowledge fusion stage, a comprehensive knowledge graph with 500,000 nodes and 2 million edges is constructed, covering multiple fields such as geography, history, and culture. Through knowledge graph embedding methods such as TransE, 50,000 new potential relationships were discovered, enriching the reasoning ability. Personalized tailoring retains 200,000 core concepts related to tourism, and customized language models for 10 different languages. Multimodal alignment ensures that when introducing attractions, 3D gestures, voice explanations, and displayed graphic information can be accurately synchronized, and the time error is controlled within 50 milliseconds.

[0115] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0116] (1) Perform feature decoupling processing on the adaptive digital human data to obtain a multi-dimensional personality feature vector, and perform principal component analysis on the multi-dimensional personality feature vector to obtain the core personality features;

[0117] (2) The core personality features are enhanced through the conditional generative adversarial network algorithm to obtain personalized feature data, and the personalized feature data is semantically mapped to obtain personalized semantic representation data;

[0118] (3) Perform multimodal cross-attention processing on the personalized semantic representation data to obtain cross-modal personality features, and perform style encoding processing on the cross-modal personality features to obtain a style feature vector;

[0119] (4) Adaptively interpolate the style feature vector to obtain a continuous style space, and perform nonlinear transformation on the continuous style space to obtain a style transfer matrix;

[0120] (5) Perform tensor multiplication on the style transfer matrix and the cross-modal personality features to obtain stylized personality data;

[0121] (6) Perform multimodal fusion processing on the stylized personality data to obtain the target digital human data.

[0122] Specifically, the adaptive digital human data is feature decoupled, and the variational autoencoder (VAE) technology is used to decompose the data into multiple independent feature dimensions. For example, for a digital human data containing 3D morphology, facial expression animation, voice and behavior pattern, VAE may decouple it into a 200-dimensional feature vector, of which 50 dimensions represent 3D morphological features, 50 dimensions represent facial expression dynamic features, 50 dimensions represent voice features, and 50 dimensions represent behavior features. The 200-dimensional personality feature vector is processed by principal component analysis (PCA) to identify the most representative core personality features. Through PCA, it may be found that the first 50 principal components can explain 90% of the data variance, and these 50 principal components are selected as core personality features. This step greatly reduces the complexity of subsequent processing while retaining the most critical personality information. The core personality features are enhanced by the conditional generative adversarial network (cGAN) algorithm. In this process, the generator network accepts 50-dimensional core features as input and tries to generate richer and more prominent personalized features. The discriminator network attempts to distinguish between generated features and real personality features. After 1,000 rounds of training, cGAN is able to generate 200-dimensional feature data with more distinct personalities. These data present richer and more delicate feature expressions while maintaining the original personality. The generated personalized feature data is semantically mapped. This step uses a deep neural network to convert numerical features into interpretable semantic representations. For example, the 200-dimensional features are mapped into a space containing 1,000 semantic labels, each of which represents a specific personality trait or behavioral tendency, such as "extroversion", "creativity", "caution", etc. This mapping network is trained with a large amount of labeled data and can accurately convert numerical features into human-understandable descriptions. Multimodal cross-attention processing is performed on the personalized semantic representation data. This step uses the multi-head attention mechanism of the transformer architecture to allow information interaction between different modalities (such as vision, hearing, and semantics). For example, 8 attention heads are set, each of which is responsible for capturing different types of cross-modal relationships. Through this processing, the model can learn complex cross-modal personality features such as "the association between smiling expressions and positive intonation" or "the correspondence between serious expressions and formal language".

[0123] The obtained cross-modal personality features are style encoded. This step uses an autoencoder network to compress complex cross-modal features into a concise style feature vector. For example, the previous 1000-dimensional semantic representation may be compressed into a 64-dimensional style feature vector, which highly summarizes the overall style characteristics of the digital human. The style feature vector is adaptively interpolated to obtain a continuous style space. This step uses the spherical linear interpolation (SLERP) algorithm to create a smooth transition between different style feature vectors. For example, 100 intermediate states are generated between the "formal" and "relaxed" styles to form a continuous style change spectrum.

[0124] The continuous style space is transformed nonlinearly to obtain the style transfer matrix. A multi-layer perceptron (MLP) network is used here to map the 64-dimensional style features to a 200x200 matrix space. This matrix can capture complex nonlinear style transformation relationships. The style transfer matrix and the cross-modal personality features are tensor multiplied. This step multiplies the 200-dimensional cross-modal personality features with the 200x200 style transfer matrix to obtain a new 200-dimensional stylized personality data. This operation achieves accurate style transfer while maintaining the original personality features. The stylized personality data is subjected to multimodal fusion processing to obtain the target digital human data. This step uses the attention mechanism and residual connection to reintegrate the stylized features into different modalities, including 3D morphology, expression animation, speech synthesis, and behavior generation modules.

[0125] For example, when creating a 5D digital human model for a virtual news anchor, the collected data is first feature decoupled. Through VAE processing, a 200-dimensional feature vector is obtained, which includes 50-dimensional facial features, 50-dimensional voice features, 50-dimensional body movement features, and 50-dimensional language expression features. After PCA analysis, it is found that the first 60 principal components can explain 95% of the variance, and these 60 principal components are selected as core personality features. The 60-dimensional core features are enhanced using cGAN. The generator and discriminator each contain 5 layers of neural networks. After 2,000 rounds of training, more distinct 200-dimensional personalized feature data is generated. These features are converted into 1,000 specific behavior and personality labels, such as "professionalism: 0.95" and "affinity: 0.75", through a pre-trained semantic mapping network. Multimodal cross-attention processing uses a 12-layer transformer structure with 8 attention heads per layer to learn complex cross-modal relationships, such as the strong correlation between serious expressions and formal language (correlation coefficient 0.92). These features are compressed into a 64-dimensional style feature vector through a 3-layer autoencoder. Through the SLERP algorithm, 100 transition states are generated between the two extreme styles of "serious news" and "relaxed interview". A 4-layer MLP network maps the 64-dimensional style features to a 200x200 style transfer matrix. This matrix is ​​tensor multiplied with the original 200-dimensional cross-modal features to obtain stylized personality data. The final multimodal fusion uses a residual attention network to reintegrate the stylized features into specific performances. For example, when reporting serious news, the system automatically adjusts the seriousness of facial expressions (from 0.6 to 0.85), the speech speed is slightly slowed down (from 180 words per minute to 160 words per minute), and the posture is more correct (the change in torso angle is reduced by 30%). At the same time, the language expression is more formal, and the frequency of professional terms used has increased by 40%.

[0126] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0127] (1) Perform multi-resolution decomposition processing on the target digital human data to obtain a hierarchical 5D data structure, and dynamically adjust the details of the hierarchical 5D data structure through an adaptive LOD algorithm to obtain an adaptive rendering model;

[0128] (2) Perform real-time ray tracing on the adaptive rendering model to obtain high-quality rendering data, and perform physical engine simulation on the high-quality rendering data to obtain realistic interactive data;

[0129] (3) Perform multimodal fusion processing on the realistic interaction data to obtain comprehensive perception data, and perform situational awareness analysis on the comprehensive perception data to obtain context-related information;

[0130] (4) Perform decision tree generation processing on context-related information to obtain an intelligent response strategy, and perform feedback loop optimization processing on the intelligent response strategy to obtain target 5D digital human data;

[0131] (5) The target 5D digital human data is processed for performance index extraction to obtain multi-dimensional evaluation indicators, and adaptive learning curve generation is performed based on the multi-dimensional evaluation indicators to obtain a digital human training plan.

[0132] Specifically, the target digital human data is subjected to multi-resolution decomposition processing. This step uses wavelet transform technology to decompose the data into a hierarchical structure with different resolution levels. For example, for a high-precision 3D model containing 1 million vertices, 5 resolution levels may be obtained through 4-level wavelet decomposition: 1 million vertices, 500,000 vertices, 250,000 vertices, 125,000 vertices, and 62,500 vertices. At the same time, for texture data, 4K, 2K, 1K, and 512x512 textures may be obtained by downsampling the original 8K resolution step by step. This hierarchical 5D data structure lays the foundation for subsequent adaptive rendering. The hierarchical 5D data structure is dynamically adjusted in detail through an adaptive LOD (Level of Detail) algorithm. This step uses an adaptive strategy based on viewpoint distance and hardware performance to select the most appropriate level of detail in real time. For example, when the digital human is very close to the camera, the highest precision model with 1 million vertices and 8K textures are used; when the distance increases to 10 meters, it may switch to a 250,000 vertex model and 2K textures; when the distance exceeds 50 meters, the lowest precision model with 62,500 vertices and 512x512 textures may be used. This dynamic adjustment ensures that rendering performance is optimized while maintaining visual quality. The adaptive rendering model is processed in real-time ray tracing, which uses the latest real-time ray tracing technology such as NVIDIA's RTX technology. The ray tracing algorithm simulates the propagation of light in the scene and calculates direct lighting, reflection, refraction, and global illumination effects. For example, for the skin of a digital human, 10 million rays may be emitted to calculate the subsurface scattering effect to make the skin appear naturally translucent. For the eyes, the corneal reflection and the fine structure of the iris may be calculated specifically, using 100,000 rays for each eye.

[0133] Perform physical engine simulation on high-quality rendering data. This step uses a physical engine such as PhysX or Bullet to simulate the physical interaction between the digital human and the environment. For example, 1,000 independent physical simulation bones are set for the digital human's hair, each with 10 nodes, to achieve a natural hair fluttering effect. For clothing, 100,000 particle points may be used for cloth simulation to achieve realistic wrinkles and fluttering effects. Perform multimodal fusion processing on realistic interaction data. This step integrates visual, auditory and tactile data into a unified perceptual representation. For example, visual rendering data (such as facial expressions), audio data (such as voice) and tactile feedback data (such as pressure when shaking hands) are fused into a 300-dimensional vector, with each dimension representing a specific perceptual feature.

[0134] Perform context-aware analysis on the comprehensive perception data. This step uses context-aware deep learning models, such as BERT or GPT, to analyze the current interaction context. The model input may include the last 10 sentences of dialogue history, the current environment description, and the user's emotional state, and outputs a context representation containing 100 semantic tags, each of which represents a specific context feature. Perform decision tree generation on context-related information. This step uses algorithms such as C4.5 or random forest to build a decision tree based on context information. The decision tree may contain 1,000 nodes, each of which represents a decision point, such as "Is the user's emotion positive?", "Is the current topic professional?", etc. The leaf nodes of the tree correspond to specific response strategies. Feedback loop optimization of intelligent response strategies. This step uses reinforcement learning algorithms, such as deep Q networks (DQN), to continuously adjust and optimize response strategies based on user feedback. For example, set up a Q table containing 1 million state-action pairs, and update the corresponding Q value based on the user's response after each interaction.

[0135] The target 5D digital human data is processed for performance index extraction. This step evaluates the performance of the digital human from multiple dimensions. 50 key indicators may be extracted, including visual realism (such as the naturalness score of skin rendering), motion fluency (such as the smoothness of motion transitions), speech naturalness (such as the richness of intonation changes), interactive intelligence (such as answer accuracy), etc. Adaptive learning curve generation is performed based on multi-dimensional evaluation indicators. This step uses the Bayesian optimization algorithm to dynamically adjust the learning strategy based on the current performance indicators. For example, if it is found that the speech synthesis performance lags behind other aspects, the algorithm may increase the weight of speech training and generate a new learning plan containing 1,000 training steps.

[0136] For example, when creating a 5D digital human model for a virtual medical consultation assistant, the 3D model data is first decomposed into five levels of precision: 2 million vertices, 1 million vertices, 500,000 vertices, 250,000 vertices, and 125,000 vertices. The texture data is downsampled from the original 16K resolution to 8K, 4K, 2K, and 1K. The adaptive LOD algorithm dynamically selects the appropriate model accuracy according to the complexity of the scene, using 2 million vertices and 16K textures for close-up conversations, and may drop to 500,000 vertices and 4K textures for full-body display, maintaining a stable frame rate of 60fps. Real-time ray tracing processing uses 5 million rays to simulate the reflection and wrinkles of the doctor's white coat, while a special subsurface scattering algorithm is used for the eyes, using 1 million rays to accurately simulate the iris structure, enhancing the realism of the eyes. In the physics engine simulation, 2,000 independent physical bones are set for the doctor's hair, each with 30 nodes, to achieve natural hair shaking. The white gown uses 200,000 particle points for fabric simulation, accurately reproducing the subtle deformation when wearing a stethoscope. Multimodal fusion processing integrates visual, speech and tactile data into a 500-dimensional vector, of which 300 dimensions represent visual features, 150 dimensions represent speech features, and 50 dimensions represent tactile features. Context-aware analysis uses a 12-layer transformer model, with input including the most recent 20 sentences of conversation history and the patient's expression data, and outputs 200 semantic tags to describe the current medical consultation situation. The decision tree generates a complex decision structure with 5,000 nodes, covering all aspects from symptom analysis to treatment recommendations. Feedback loop optimization uses the DQN algorithm to maintain a Q table containing 2 million state-action pairs, and updates the strategy after each consultation based on the patient's satisfaction score (1-5 points). Performance indicator extraction includes 100 detailed indicators, such as diagnostic accuracy, explanation clarity, empathy performance, etc. Based on these indicators, the adaptive learning curve generator developed an optimization plan consisting of 5,000 training steps, focusing on strengthening medical knowledge updating and patient communication skills.

[0137] The above describes the training method for generating a 5D digital human based on AIGC in the embodiment of the present application. The following describes the training system for generating a 5D digital human based on AIGC in the embodiment of the present application. Figure 2 In the embodiments of the present application, the training system for generating 5D digital human based on AIGC includes:

[0138] The data processing module 201 is used to pre-process the pre-collected real person feature data to obtain multi-dimensional pre-processed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits;

[0139] The multi-layer processing module 202 is used to perform multi-level processing on the multi-dimensional pre-processed data to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph;

[0140] The coordination processing module 203 is used to perform cross-modal coordination processing on the 5D digital human generated data through a progressive multi-task learning algorithm to obtain the digital human data to be processed;

[0141] An integration processing module 204 is used to perform dynamic knowledge integration processing on the digital human data to be processed by an adaptive neural architecture search algorithm to obtain adaptive digital human data;

[0142] A migration processing module 205 is used to perform personalized fine-tuning and style migration processing on the adaptive digital human data to obtain target digital human data;

[0143] The real-time rendering module 206 is used to perform real-time rendering and intelligent interactive processing on the target digital human data through an adaptive LOD algorithm to obtain target 5D digital human data and generate a digital human training plan according to the target 5D digital human data.

[0144] Through the collaboration of the above components, multi-modal separation and preprocessing of pre-collected real-person feature data are performed to obtain multi-dimensional pre-processed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits, laying a comprehensive and rich data foundation for the subsequent 5D digital human generation, ensuring the authenticity and consistency of the generation results in various dimensions. The multi-dimensional pre-processed data is processed at multiple levels to obtain 5D digital human generation data including 3D morphological data, action expression data, voice synthesis data, personalized behavior data and knowledge graph. This comprehensive data structure makes the generated digital human not only highly realistic visually, but also has rich facial expressions, natural voice expressions, personalized behavior patterns and deep knowledge reserves. The cross-modal coordination of 5D digital human generation data is carried out through the progressive multi-task learning algorithm, which effectively solves the coordination and consistency problems between different modal data and improves the overall coordination and naturalness of the generation results. The adaptive neural architecture search algorithm is used to perform dynamic knowledge integration processing on the processed digital human data, which greatly enhances the adaptability and generalization ability of the model, so that the generated digital human can better cope with complex and changing scenes and tasks. Personalized fine-tuning and style transfer processing of adaptive digital human data not only improves the personalization of digital humans, but also realizes rapid style conversion, greatly enhancing the diversity and customizability of generated results. Real-time rendering and intelligent interactive processing of target digital human data through adaptive LOD algorithm significantly improves rendering efficiency and interactive fluency while ensuring high-quality visual effects, so that the generated 5D digital human can achieve smooth real-time performance in various hardware environments. Finally, a digital human training plan is generated based on the target 5D digital human data, realizing closed-loop optimization of the training process, continuously improving the overall quality and performance of digital humans, and improving the efficiency and accuracy of training 5D digital humans based on AIGC.

[0145] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the training method for generating a 5D digital human based on AIGC.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for generating 5D digital human based on AIGC, characterized in that: The training method for generating 5D digital human based on AIGC includes: Preprocessing the pre-collected real person feature data to obtain multi-dimensional preprocessed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits; Performing multi-level processing on the multi-dimensional pre-processed data to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph; The 5D digital human generated data is processed cross-modally through a progressive multi-task learning algorithm to obtain the digital human data to be processed; The digital human data to be processed is subjected to dynamic knowledge integration processing by an adaptive neural architecture search algorithm to obtain adaptive digital human data, including: performing feature decomposition processing on the digital human data to be processed to obtain a multi-dimensional feature subspace, and performing neural architecture encoding processing on the multi-dimensional feature subspace to obtain a candidate architecture pool; evaluating the candidate architecture pool by an adaptive neural architecture search algorithm to obtain a preferred architecture sequence, and dynamically combining the preferred architecture sequence to obtain an adaptive neural network structure; performing parameter initialization processing on the adaptive neural network structure to obtain an initialization model, and performing incremental learning processing on the initialization model to obtain a dynamic knowledge representation; performing cross-modal knowledge fusion processing on the dynamic knowledge representation to obtain a comprehensive knowledge graph, and performing reasoning optimization processing on the comprehensive knowledge graph to obtain a reasoning enhanced knowledge base; performing personalized knowledge tailoring processing on the reasoning enhanced knowledge base to obtain a customized knowledge structure, and performing multi-modal alignment processing on the customized knowledge structure to obtain the adaptive digital human data; Performing personalized fine-tuning and style transfer processing on the adaptive digital human data to obtain target digital human data; The target digital human data is rendered in real time and intelligently interactively processed through an adaptive LOD algorithm to obtain target 5D digital human data and generate a digital human training plan based on the target 5D digital human data.

2. The training method for generating 5D digital human based on AIGC according to claim 1, characterized in that: The preprocessing of the pre-collected real person feature data to obtain multi-dimensional preprocessed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits includes: Performing multimodal separation processing on the pre-collected real person feature data to obtain original visual data, audio data and semantic data, and performing 3D reconstruction processing on the original visual data to obtain an initial static three-dimensional model; Performing mesh optimization and texture mapping processing on the initial static three-dimensional model to obtain the static three-dimensional data; Performing time series analysis and skeleton extraction processing on the original visual data to obtain original dynamic skeleton data, and performing motion trajectory smoothing and joint constraint processing on the original dynamic skeleton data to obtain the dynamic skeleton data; Performing acoustic feature extraction and phoneme segmentation processing on the audio data to obtain initial speech features, and performing prosodic pattern analysis and emotion labeling processing on the initial speech features to obtain the speech sample; Performing natural language processing and knowledge graph construction on the semantic data to obtain initial character features, and performing personalized feature extraction and behavior pattern analysis on the initial character features to obtain the character portrait; The static three-dimensional data, dynamic skeleton data, voice samples and character portraits are subjected to multi-modal alignment and integration processing to obtain the multi-dimensional pre-processed data.

3. The training method for generating 5D digital human based on AIGC according to claim 1, characterized in that: The multi-dimensional pre-processed data is processed at multiple levels to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph, including: Performing topological structure optimization processing on the static three-dimensional data to obtain a high-precision 3D mesh model, and performing parameterized reconstruction processing on the high-precision 3D mesh model to obtain the 3D morphological data; Performing time-series interpolation processing on the dynamic skeleton data to obtain a continuous motion sequence, and performing expression mapping processing on the continuous motion sequence through a deep spatiotemporal convolutional network to obtain the action expression data; Performing acoustic feature analysis processing on the speech sample to obtain a phoneme unit library, and performing context-sensitive splicing processing on the phoneme unit library to obtain the speech synthesis data; Performing feature vectorization processing on the character portrait to obtain multi-dimensional behavior features, and performing probabilistic graph model construction processing on the multi-dimensional behavior features to obtain the personalized behavior data; The character portrait is subjected to semantic network construction processing to obtain an initial knowledge structure, and the initial knowledge structure is subjected to relational reasoning and expansion processing to obtain the knowledge graph.

4. The training method for generating 5D digital human based on AIGC according to claim 1, characterized in that: The step of performing cross-modal coordinated processing on the 5D digital human generated data through a progressive multi-task learning algorithm to obtain the digital human data to be processed includes: Performing task decomposition processing on the 5D digital human generated data to obtain multiple subtask data sets, and performing difficulty evaluation analysis on the multiple subtask data sets to obtain a task difficulty sequence; Sorting the subtask data sets according to the task difficulty sequence to obtain a progressive learning sequence, and performing feature extraction on the primary task data in the progressive learning sequence to obtain basic feature representation data; Iteratively optimizing the basic feature representation data through a progressive multi-task learning algorithm to obtain multi-level feature representation data, and performing cross-modal attention mechanism processing on the multi-level feature representation data to obtain inter-modal correlation features; Performing adaptive weight distribution processing on the inter-modal correlation features to obtain balanced feature representation data, and performing cross-modal consistency constraint processing on the balanced feature representation data to obtain coordinated feature data; The coordinated feature data is subjected to progressive knowledge distillation processing to obtain compressed feature representation data, and the compressed feature representation data is subjected to multimodal fusion processing to obtain the digital human data to be processed.

5. The training method for generating 5D digital human based on AIGC according to claim 1, characterized in that: The step of performing personalized fine-tuning and style transfer processing on the adaptive digital human data to obtain target digital human data includes: Performing feature decoupling processing on the adaptive digital human data to obtain a multi-dimensional personality feature vector, and performing principal component analysis processing on the multi-dimensional personality feature vector to obtain core personality features; The core personality features are enhanced by a conditional generative adversarial network algorithm to obtain personalized feature data, and the personalized feature data are semantically mapped to obtain personalized semantic representation data; Performing multimodal cross-attention processing on the personalized semantic representation data to obtain cross-modal personality features, and performing style encoding processing on the cross-modal personality features to obtain a style feature vector; Performing adaptive interpolation processing on the style feature vector to obtain a continuous style space, and performing nonlinear transformation processing on the continuous style space to obtain a style transfer matrix; Performing tensor multiplication processing on the style transfer matrix and the cross-modal personality feature to obtain stylized personality data; The stylized personality data is subjected to multimodal fusion processing to obtain the target digital human data.

6. The training method for generating 5D digital human based on AIGC according to claim 5, characterized in that: The method of performing real-time rendering and intelligent interactive processing on the target digital human data by using an adaptive LOD algorithm to obtain target 5D digital human data and generating a digital human training program according to the target 5D digital human data includes: Performing multi-resolution decomposition processing on the target digital human data to obtain a hierarchical 5D data structure, and performing dynamic detail adjustment processing on the hierarchical 5D data structure through an adaptive LOD algorithm to obtain an adaptive rendering model; Performing real-time ray tracing processing on the adaptive rendering model to obtain high-quality rendering data, and performing physical engine simulation processing on the high-quality rendering data to obtain realistic interactive data; Performing multimodal fusion processing on the realistic interaction data to obtain comprehensive perception data, and performing situational awareness analysis processing on the comprehensive perception data to obtain context-related information; Performing decision tree generation processing on the context-related information to obtain an intelligent response strategy, and performing feedback loop optimization processing on the intelligent response strategy to obtain the target 5D digital human data; The target 5D digital human data is subjected to performance index extraction processing to obtain multi-dimensional evaluation indicators, and an adaptive learning curve generation processing is performed according to the multi-dimensional evaluation indicators to obtain the digital human training program.

7. A training system for generating 5D digital humans based on AIGC, characterized in that: Used to execute the training method for generating 5D digital human based on AIGC according to any one of claims 1 to 6, the training system for generating 5D digital human based on AIGC comprises: A data processing module is used to pre-process the pre-collected real person feature data to obtain multi-dimensional pre-processed data including static three-dimensional data, dynamic skeleton data, voice samples and character portraits; A multi-layer processing module, used for performing multi-level processing on the multi-dimensional pre-processed data to obtain 5D digital human generated data, wherein the 5D digital human generated data includes: 3D morphological data, action expression data, speech synthesis data, personalized behavior data and knowledge graph; A coordination processing module is used to perform cross-modal coordination processing on the 5D digital human generated data through a progressive multi-task learning algorithm to obtain the digital human data to be processed; An integrated processing module is used to perform dynamic knowledge integration processing on the digital human data to be processed through an adaptive neural architecture search algorithm to obtain adaptive digital human data, including: performing feature decomposition processing on the digital human data to be processed to obtain a multi-dimensional feature subspace, and performing neural architecture encoding processing on the multi-dimensional feature subspace to obtain a candidate architecture pool; evaluating the candidate architecture pool through an adaptive neural architecture search algorithm to obtain a preferred architecture sequence, and dynamically combining the preferred architecture sequence to obtain an adaptive neural network structure; performing parameter initialization processing on the adaptive neural network structure to obtain an initialization model, and performing incremental learning processing on the initialization model to obtain a dynamic knowledge representation; performing cross-modal knowledge fusion processing on the dynamic knowledge representation to obtain a comprehensive knowledge graph, and performing reasoning optimization processing on the comprehensive knowledge graph to obtain a reasoning enhanced knowledge base; performing personalized knowledge tailoring processing on the reasoning enhanced knowledge base to obtain a customized knowledge structure, and performing multi-modal alignment processing on the customized knowledge structure to obtain the adaptive digital human data; A migration processing module, used to perform personalized fine-tuning and style migration processing on the adaptive digital human data to obtain target digital human data; The real-time rendering module is used to perform real-time rendering and intelligent interactive processing on the target digital human data through an adaptive LOD algorithm to obtain target 5D digital human data and generate a digital human training plan based on the target 5D digital human data.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the training method for generating 5D digital humans based on AIGC as described in any one of claims 1 to 6 is implemented.

Images