3D digital virtual system based on large model fine tuning and TTS speech synthesis

Abstract

The invention provides a 3D digital virtual system based on large model fine tuning and TTS speech synthesis, which relates to the technical field of electric digital data processing, and comprises a processor, a user input module, a text preprocessing module, an M3E vector coding module, a Milvus vector database, a Prompt construction module, a LoRA fine tuning module, a Llama3-8B model reasoning module, a post-processing module and an output module, the user input module supports voice and text input and converts voice into text data through an ASR engine. According to the invention, the problems of huge computing resource consumption and low training efficiency in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, in particular to a 3D digital virtual system based on large model fine-tuning and TTS speech synthesis. Background Art

[0002] With the rapid development of generative artificial intelligence and real-time 3D rendering technology, the application of digital virtual systems in cultural and educational scenarios has gradually become a hot topic in the industry. However, when generating content, general large language models often lack the language style and knowledge system of specific figures, which can easily lead to historical errors or style inconsistencies. Moreover, speech synthesis technology usually requires a large amount of high-quality audio data when reproducing the timbre of specific historical figures. However, there are no direct recordings of historical figures such as Su Shi, making timbre restoration difficult. In response to the above problems, existing technologies have attempted to improve them through various methods, including: in the field of natural language processing, by fine-tuning large language models to enhance their adaptability to specific fields, but traditional fine-tuning methods require updating a large number of parameters, consume huge computing resources, and have low training efficiency. Summary of the Invention

[0003] The purpose of the present invention is to provide a 3D digital virtual system based on large model fine-tuning and TTS speech synthesis, which solves the problems of huge computing resource consumption and low training efficiency in the prior art.

[0004] The technical solution of the present invention: The present invention provides a 3D digital virtual system based on large model fine-tuning and TTS speech synthesis, including a processor, a user input module, a text preprocessing module, an M3E vector encoding module, a Milvus vector database, a Prompt construction module, a LoRA fine-tuning module, a Llama3-8B model inference module, a post-processing module and an output module, wherein the user input module supports voice and text input and converts voice into text data through an ASR engine.

[0005] In some embodiments, the ASR engine uses a Whisper model to perform real-time speech-to-text processing and transmits the text to a subsequent processing module via a Socket protocol.

[0006] In some embodiments, the Socket protocol defines a message header and message body structure for transmitting text, audio URL, and expression weight information.

[0007] In some embodiments, the Milvus vector database stores 768-dimensional question vectors encoded by the m3e-base model and performs similarity retrieval using Chebyshev distance.

[0008] In some embodiments, the M3E vector encoding module feeds the preprocessed text into the m3e-base model to generate a 768-dimensional question vector.

[0009] In some embodiments, the LoRA fine-tuning module performs adjustments to the q_proj and v_proj linear layer parameters in the Deepseek-r1:14b model.

[0010] In some embodiments, the speech synthesis module uses Bert-VITS2 sample timbre cloning technology to generate speech and supports three levels of emotional conversion.

[0011] In some embodiments, the Bert-VITS2 model is trained with at least 200 recitation audios to obtain target timbre data.

[0012] In some embodiments, the 3D modeling and animation module uses StableDiffusionLoRA raw images, DECA facial reconstruction, and UE5MetaHuman rendering to generate 3D images and drive expressions and body movements.

[0013] In some embodiments, StableDiffusionLoRA is trained on at least 20 target sample images.

[0014] According to the above technical features, the beneficial effects of the present invention are: the 3D digital virtual system based on large model fine-tuning and TTS speech synthesis provided by the present invention uses the Su Shi intelligent body in the three-in-one lightweight customized system that combines LoRA fine-tuning, RAG, and Prompt, taking into account the consistency of style, accuracy of answers and computing power friendliness, and combining AIGC+3D modeling technology to realize film and television-level three-dimensional images within the system, and also integrates speech recognition, natural language processing and other technologies to realize the digital human that can be presented by the lightweight customized system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural schematic diagram of the present invention; Figure 2 Schematic diagram of the working principle of the Bert-VITS2 model in the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be combined with the embodiments of the present invention. Figure 1-Figure 2 The technical solutions in the embodiments of the present invention are clearly and completely described together. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0017] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0018] It should be noted that the following are some brief descriptions of the terms in the embodiments of the present invention: 1.M3E is an important tool in the field of Chinese semantic retrieval.

[0019] 2.Milvus is an open source vector database.

[0020] 3. LoRA is an efficient large model parameter fine-tuning technology.

[0021] 4.Llama3-8B is a lightweight large model.

[0022] 5.Bert-VITS2 is a text-to-speech (TTS) model based on deep learning.

[0023] 6.StableDiffusionLoRA is a method that applies LoRA (low-rank adaptation) fine-tuning technology to the StableDiffusion text-to-image model.

[0024] 7.UE5MetaHuman is a toolset in Unreal Engine 5 that combines AI technology with real-time rendering capabilities to quickly create photorealistic digital humans.

[0025] Example An embodiment of the present invention provides a 3D digital virtual system based on large model fine-tuning and TTS speech synthesis, the system including a processor, a user input module, a text preprocessing module, an M3E vector encoding module, a Milvus vector database, a Prompt construction module, a LoRA fine-tuning module, a Llama3-8B model inference module, a post-processing module and an output module, wherein the user input module supports voice and text input and converts voice into text data through an ASR engine.

[0026] In some embodiments, the ASR engine uses the Whisper model to perform real-time speech-to-text conversion and transmits the results to subsequent processing modules via the Socket protocol. The Socket protocol defines a message header and body structure for transmitting text, audio URLs, and expression weight information. The Milvus vector database stores 768-dimensional question vectors encoded using the m3e-base model and performs similarity retrieval using the Chebyshev distance. The M3E vector encoding module feeds the pre-processed text into the m3e-base model to generate 768-dimensional question vectors. The LoRA fine-tuning module adjusts the q_proj and v_proj linear layer parameters in the Deepseek-r1:14b model. The speech synthesis module uses Bert-VITS2 sample timbre cloning technology to generate speech and supports three levels of emotional conversion. The Bert-VITS2 model is trained on at least 200 recitation audio recordings to obtain target timbre data. The 3D modeling and animation module uses StableDiffusion LoRA raw images, DECA facial reconstruction, and UE5 MetaHuman rendering to generate 3D characters and drive facial expressions and body movements. StableDiffusionLoRA is trained by performing training on at least 20 target sample images.

[0027] The specific embodiment of the present invention is combined with Figure 1-Figure 2 Provide detailed explanation. Figure 1 is a structural diagram, Figure 2 The following is a diagram of the working principle of the Bert-VITS2 model, which illustrates the complete processing flow from user questions to output answers. Figure 1 Each module and Figure 2 The model principle is described in detail.

[0028] After the system starts, users interact with the digital human through a client interface. The client interface supports voice and text input through a user input module. When the user clicks the voice input button, the client calls the operating system's microphone interface to collect audio data and converts the speech into text in real time using the Whisper model. The ASR engine utilizes streaming recognition technology, receiving audio data while simultaneously generating intermediate text output. Ultimately, the complete text output is transmitted to subsequent modules via the Socket protocol. The text preprocessing module cleans, segments, and normalizes the received text. The knowledge processing function combines M3E vector encoding, Milvus vector retrieval, and RAG technology to match relevant questions and answers from a pre-built knowledge base. For example, for polyphonetic characters in Chinese text, the system determines the correct pronunciation based on contextual semantics. For mixed-language text, the system converts it into a unified 768-dimensional vector representation using the M3E vector encoding module. This process ensures the standardization and consistency of text data during subsequent processing. Specifically, questions in the knowledge base are encoded into 768-dimensional vectors. Milvus searches the question vectors using Euclidean and Chebyshev distances. The five approximate answers obtained are fed into the fine-tuned model as references. M3E-Embedding is used to encode the question into a 768-dimensional vector, along with the original answer. The vectors are then written to Milvus 2.5 using PyMilvus to create an HNSW index. User questions are encoded using M3E, and the nearest k answers are retrieved using Euclidean and Chebyshev distances in Milvus. These are then used to generate prompts and fed into the LoRA fine-tuned model. The improvement process is as follows: first, the M3E-Embedding model with better performance is selected as the encoder. The M3E model performs well in multilingual and multi-domain tasks and can generate high-quality semantic vectors; the question is encoded into a 768-dimensional vector, and the original answer is saved together to ensure that complete context information can be obtained during retrieval; to improve retrieval efficiency, PyMilvus is used to write the encoded vector into the Milvus2.5 vector database, and an HNSW index is established. The HNSW index is an efficient approximate nearest neighbor search index that can quickly and accurately find the vector most similar to the query vector in large-scale data sets, greatly improving retrieval speed and accuracy.

[0029] The M3E vector encoding module feeds the preprocessed text into the m3e-base model to generate 768-dimensional question vectors. These vectors are then used for similarity searches in the Milvus vector database. The Milvus vector database stores knowledge base entries encoded by the m3e-base model. Each entry contains a question vector and its corresponding answer text. The system uses Chebyshev distance or Euclidean distance as a similarity calculation method to quickly find the knowledge base entry that most closely matches the user's question. The retrieved results are passed to the Prompt construction module, which combines the search results with a preset style template to generate a prompt. For example, for questions related to Su Shi, the system will add style constraints such as "Please answer in a bold and unrestrained style" to the prompt to ensure that the generated answer conforms to Su Shi's linguistic characteristics.

[0030] The LoRA fine-tuning module infers the prompt based on the Deepseek-r1:14b model to generate a preliminary response. During this process, the system only adjusts the parameters of the q_proj and v_proj linear layers in the model, reducing computational resource consumption. The Llama3-8B model inference module further optimizes the generated response, filtering out historical errors and optimizing classical expressions to maintain the consistency of Su Shi's language style. For example, when a user asks whether Su Shi met Yue Fei, the system automatically rejects fabricated content and returns accurate historical context. The post-processing module is responsible for annotating the generated response with sentiment labels, which are used to drive the 3D model's facial animation. In detail, the Lora low-rank matrix is ​​used to fine-tune the large model. In terms of training parameters, 8-bit quantization is used to freeze the base, which reduces the consumption of computing resources while ensuring model performance; LoRArank is set to 8, the learning rate is 5e-4, and CosineDecay is used to adjust the learning rate to make the training process smoother. At the same time, the early stopping strategy is used to monitor the perplexity (ppl) of the validation set to prevent overfitting and ensure the stability and accuracy of the model.

[0031] The speech synthesis module uses Bert-VITS2 small-sample voice cloning technology to generate high-quality speech. The module was trained on 200 audio recordings of recitations to emulate Su Shi's voice. It also supports three levels of emotional embedding for voice control. For example, the system adjusts the pitch, rate, and energy distribution of the speech based on different emotional labels, generating speech with a pleasant, contemplative, or emotional tone. The generated speech data is transmitted to the client via the Socket protocol and played synchronously with the 3D modeling and animation module.

[0032] The 3D modeling and animation module uses StableDiffusionLoRA to fine-tune raw images, DECA facial reconstruction, and UE5MetaHuman rendering technology to generate hyper-realistic 3D images and drive expressions and body movements. Specifically, the system trained 20 target sample portraits for 10 epochs, improving the CLIP-I score by 28% with only a 9MB increase in parameters. The DECA model predicts high-precision meshes and 100-dimensional expression BlendShapes at one time, taking into account both fine lines and animatability. UE5MetaHuman rendering technology uses Nanite and Lumen to ensure real-time rendering at 1080p60fps on the RTX4090 graphics card.

[0033] The communication and control module defines the message header and message body structure through the Socket protocol, which is used to transmit information such as text, audio URLs, and expression weights, to achieve efficient communication between the client and the server and ensure real-time interaction. After the client receives the data returned by the server, it sends the voice data to the audio playback system and applies the expression weights to the facial animation drive of the 3D model. For example, when the system detects that the emotion label is "joyful", it adjusts the 3D model's mouth corner upward deformation target weight to 0.8 and the eye smile deformation target weight to 0.6, thereby generating natural expression changes. In order to ensure the synchronization of sound and picture, the system uses an interpolation algorithm to smoothly transition the weight changes between different expressions to avoid abrupt switching effects.

[0034] It should be noted that in actual application scenarios, this system is suitable for a variety of scenarios such as museum exhibitions, poetry teaching, and cultural communication. For example, in a museum exhibition, users can use voice to ask, "What is the background of Su Shi's creation of "Red Cliff Fu"?" The system uses the above technical process to generate an answer that conforms to Su Shi's style and displays his facial expressions and body movements in a hyper-realistic 3D image. In poetry teaching scenarios, teachers can guide students to interact with digital virtual humans to explore Su Shi's literary style and philosophy of life. By integrating LoRA fine-tuning, RAG search enhancement, Prompt engineering, Bert-VITS2 speech synthesis, and hyper-realistic 3D modeling technology, the system realizes an immersive interactive experience and promotes technological progress in the digital transformation of traditional culture.

[0035] It should be noted that Bert-VITS2, as a text-to-speech system, cleverly combines the advantages of the pre-trained language model BERT and the end-to-end speech synthesis model VITS2. This system achieves breakthrough multilingual, highly natural, and highly controllable speech synthesis effects by deeply integrating contextual understanding capabilities in the field of natural language processing with high-fidelity synthesis technology in the field of acoustic modeling. In the Bert-VITS2 model, the text preprocessing layer performs multilingual word segmentation and phoneme conversion, text normalization, tone extraction, and language identification. The semantic understanding layer extracts multilingual BERT features, maps word-level to phoneme-level features, and encodes and integrates style text, enabling the system to understand contextual relationships, syntactic structure, and semantic emphasis, thereby more accurately expressing the actual meaning and emotion of the text when generating speech. The acoustic modeling layer employs a text encoder (fusion of phonemes, tones, and semantic features), duration prediction (stochastic and deterministic dual paths), and a stream transformation network (conditional variational autoencoder). Instead of treating phoneme duration as a deterministic problem, it establishes a probabilistic distribution model that generates naturally varying prosodic patterns, avoiding mechanical and monotonous speech output. The waveform generation layer employs the HiFi-GAN vocoder, optimized through adversarial training, and implements multi-speaker embedding. The output layer achieves high-quality multilingual speech output with controllable speech rate, emotion, and style parameters.

[0036] It should be noted that the system of the present invention adopts a multi-objective loss function design to balance the optimization objectives of different aspects: the reconstruction loss includes the Mel-spectrogram reconstruction loss (L1) and the waveform reconstruction loss to ensure the accuracy of the generated content; the adversarial loss improves the realism of the generated waveform through the adversarial signal provided by the multi-resolution discriminator; the KL divergence loss optimizes the latent space representation of the variational autoencoder to balance the reconstruction quality and sampling diversity; the duration loss supervises the accuracy of the duration prediction while allowing appropriate randomness; the feature matching loss ensures that the generated waveform is similar to the real waveform in the features of each layer of the discriminator; through comprehensive loss function design, it is ensured that the system is optimized simultaneously in multiple dimensions such as sound quality, naturalness and controllability.

[0037] It should be noted that the Bert-VITS2 model in the present system processes raw data through a series of steps: audio preprocessing, resampling to a uniform sampling rate (typically 22050Hz), silence removal and energy normalization, segmentation into training-suitable segments (typically 5-15 seconds), text preprocessing, transcript cleaning and standardization, language annotation and classification, phoneme annotation and alignment, dataset construction, generation of training and validation sets, unique assignment of each speaker, and generation of necessary metadata (such as language identifiers and phoneme sequences). This system provides a complete data preprocessing process, including text cleaning, phoneme conversion, and training set segmentation.

[0038] It should be noted that the Bert-VITS2 model in the system of the present invention adopts a number of advanced technologies to improve training efficiency: the gradient accumulation accumulates the gradients of multiple small batches and then updates them, effectively increasing the actual batch size and improving training stability; the mixed precision training combines FP16 and FP32 precision, significantly improving training speed and memory efficiency while maintaining numerical stability; the distributed training supports multi-GPU parallel training, adopts a strategy that combines data parallelism and model parallelism to accelerate large-scale model training; the dynamic batching dynamically adjusts the batch size according to the sequence length, optimizes memory usage, and improves training throughput; these efficient training technologies enable Bert-VITS2 to complete training under reasonable hardware conditions, lowering the resource threshold.

[0039] It should be noted that this invention combines LoRA fine-tuning, RAG, and Prompt to lightweight customize the dialogue model of the virtual character Su Shi within the system. Specifically, LoRA low-rank fine-tuning is used to lock in Su Shi's language style, supplemented by vector search enhancement and refined Prompt engineering, to achieve highly consistent dialogue on an 8B-scale model. It uses end-to-end local deployment, specifically, the entire chain is completed within a single RTX4090 machine, solving cloud dependency, privacy, and ongoing cost issues. It uses a multimodal real-time interaction pipeline, specifically, speech recognition, sentiment analysis, text generation, timbre synthesis, and 3D expression drive in parallel, with an end-to-end average latency of ≈1.7s, meeting the requirements of the real-time exhibition experience. Through small-sample high-fidelity timbre cloning, specifically, using only 200 recitation audios, Bert-VITS2 replicates the "Dongpo voice" with a subjective MOS score of 4.1 / 5; it also supports emotional embedding to switch tone. Automated data and toolchains are used, including Whisper automatic segmentation and annotation, automatic labeling of SD images with manual correction, and one-click training and deployment scripts; even non-deep learning developers can replicate these results. Through DECA + UE5 hyper-realistic reconstruction, DECA generates high-precision meshes and expression skeletons for standing figures, and UE5 MetaHuman refines materials and hair to achieve a cinematic 3D image. Specifically, Su Shi's image data collection collected approximately 300 images of Su Shi, including portraits and sculptures, from reliable online databases, museum digital collections, and historical document scans. These images cover Su Shi's image characteristics at different ages and in various scenarios, providing a rich and diverse data foundation for subsequent model training. Data cleaning and annotation meticulously cleans the collected image data to remove images that are severely damaged, blurry, or inconsistent with Su Shi's features. At the same time, professional historians and labelers annotated the images, including information such as Su Shi's facial features (such as the positions of the eyes, nose, mouth, etc.), expression features (such as smiling, serious, etc.), clothing features (such as official uniforms, casual clothes, etc.), and background scenes, so that the model can better learn the key features of Su Shi's image.

[0040] It should be noted that the entire system operates in strict accordance with end-to-end local deployment principles, eliminating cloud-based dependency and privacy risks. Automated data cleaning scripts, One-Click Notebook, and the Socket service framework significantly reduce environment setup and parameter tuning time. Field-tested results demonstrate a round-trip voice time of 2.7 seconds, a TTSMOS score of 4.1 / 5, and 91% consistency in answer style, all meeting expected performance indicators and meeting the requirements for instant interaction. Through these technical solutions, the system achieves an immersive interactive experience that delivers "sounds like, sees like, and answers accurately," demonstrating high practical value and promising market adoption.

[0041] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, from all points of view, the embodiments should be regarded as illustrative and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and range of equivalents of the claims are included in the present invention. Any reference signs in the claims should not be construed as limiting the claim to which they relate.

[0042] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A 3D digital virtual system based on large model fine-tuning and TTS speech synthesis, including a processor, characterized in that: Also includes: User input module, text preprocessing module, M3E vector encoding module, Milvus vector database, Prompt construction module, LoRA fine-tuning module, Llama3-8B model inference module, post-processing module and output module, among which: The user input module supports voice and text input and converts voice into text data through the ASR engine.

2. The system according to claim 1, wherein: The ASR engine uses the Whisper model to perform real-time speech-to-text processing and transmits it to the subsequent processing module via the Socket protocol.

3. The system according to claim 2, characterized in that The Socket protocol defines a message header and a message body structure for transmitting text, audio URL, and expression weight information.

4. The system according to claim 1, wherein: The Milvus vector database stores 768-dimensional question vectors encoded by the m3e-base model and performs similarity retrieval using Chebyshev distance.

5. The system according to claim 4, characterized in that The M3E vector encoding module feeds the preprocessed text into the m3e-base model to generate a 768-dimensional question vector.

6. The system according to claim 1, wherein: The LoRA fine-tuning module performs adjustments to the q_proj and v_proj linear layer parameters in the Deepseek-r1:14b model.

7. The system according to claim 1, wherein: The speech synthesis module uses Bert-VITS2 sample timbre cloning technology to generate speech and supports three levels of emotional conversion.

8. The system according to claim 7, characterized in that The Bert-VITS2 model obtains target timbre data by training at least 200 recitation audios.

9. The system according to claim 1, wherein: The 3D modeling and animation module uses StableDiffusionLoRA raw images, DECA facial reconstruction and UE5MetaHuman rendering to generate 3D images and drive expressions and body movements.

10. The system according to claim 9, characterized in that StableDiffusionLoRA is trained by performing training on at least 20 target sample images.

Images