Intelligent agent driven super-real digital body

Abstract

The neural network system based on the Transform model solves the problem of multi-modal interaction of a large model in the vertical field, and the multi-modal interaction comprises generative dialogue, voiceprint cloning, sentiment analysis and voice-driven facial animation generation. A generative adversarial network (GAN) is adopted and is composed of a generator and a discriminator, and the mapping relation between audio and facial features is learned through three modules including an Identity Encoder, a Speech Encoder and a Face Decoder. In the training process, the optimization generator generates a vivid facial animation, and the lip shape synchronization discriminator strengthens the accuracy of lip movement. A large model knowledge base is trained in combination with a data set, emotion voiceprint content is generated through a neural network, and face and lip shape synchronization is achieved through a deep learning algorithm.

Description

Technical Field

[0001] The present invention relates to human speech recognition and motion synchronization in audio and images, and particularly belongs to the fields of natural language processing and computer graphics. Background Art

[0002] With the continuous development of society and the rapid rise of the artificial intelligence industry, the application of digital humans has gradually become a new trend. Simultaneously, the demand for generative AI services is growing rapidly. Current AI services are largely limited to interacting with large public models built by leading companies through text. These services, which mostly rely solely on text-based human-computer interaction, are unable to accurately identify user intent and provide the vertical domain knowledge they require. They are also unable to disseminate and share a company's vertical domain knowledge through recognizable and well-known digital images and voiceprints, such as those of company leaders and public relations figures. This type of AI service is no longer able to meet the emerging needs of society and businesses. As AI services are evolving towards multimodal interaction, achieving accurate intent recognition, vertical domain knowledge bases, and customized voiceprints, images, movements, emotions, and gestures to match their content are key research and future development directions for this project. Summary of the Invention

[0003] The invention relates to a software system designed to address large-scale multimodal interaction in vertical domains. It provides a transformer-based neural network model to address generative dialogue, voiceprint cloning, sentiment analysis, and deep learning-based voice-driven facial animation generation algorithms in natural language processing. This model, designed based on a generative adversarial network (GAN), consists of two main components: a generator and a discriminator. The generator's task is to generate realistic facial animations based on the input audio waveform, while the discriminator's goal is to distinguish between the generated animations and real facial animations. The network is divided into three main modules: Identity Encoder, Speech Encoder, and Face Decoder. The Speech Encoder extracts audio features, and the Face Decoder extracts facial features. Together, these two modules learn the mapping relationship between audio and facial features. The Identity Encoder learns the basic features of the input image while preserving the spatial structure of the image. During training, the generator attempts to generate animations that are as close as possible to real facial animations, while the discriminator distinguishes these generated animations from real facial animations. In this way, the generator continuously learns and improves to increase the quality and realism of the facial animations it generates. In addition, a lip sync discriminator is included to enforce the generator to consistently produce accurate and realistic lip movements and to account for temporal correlations, thereby improving visual quality.

[0004] Based on the collected data sets and intent sets, the knowledge base of the large model is trained, and the feedback results of the knowledge base are used to generate emotional voiceprint content through a neural network model. The voiceprint content is then passed through a deep learning algorithm to infer the target person's face and lip synchronization to achieve the desired effect.

[0005] This invention can quickly build intent sets based on large vertical models for society and enterprises. It can conveniently and quickly infer voiceprints, images, movements, emotions, and gestures, producing rich digital avatars that meet the requirements of the large model knowledge base. Based on specific application scenarios, it provides multimodal interaction methods, including text, voice, vision, movement, and environmental human-computer interaction. This can significantly reduce the repetitiveness and complexity of tasks such as corporate business presentations, urban cultural tourism promotion, industrial process visualization, and rural live broadcast promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a list of Mathematicians; Figure 3 It is a digital human intention library; Figure 4It is the Dharma library of the wise man; Figure 5 It is the Mathematical Homo Sapiens corpus; Figure 6 It is through the customization of digital human images to form digital human images based on vertical fields; Figure 7 It is based on a large model in a vertical field, combined with the customized image of a digital intelligent person, to form an intelligent body-driven ultra-realistic digital avatar. DETAILED DESCRIPTION By initially collecting the target person's voiceprint, image, action, emotion, and gesture data, the system conducts voice cloning training, character image training, action combination training, emotion recognition training, and gesture combination training to complete the person's image model. Simultaneously, based on a given set of intents, the system trains a large model's knowledge base to pair behaviors and intents. In actual applications, end users can ask questions of the Digital Intelligence Agent based on their needs. The Digital Intelligence Agent searches through its trained vertical domain knowledge base to generate content answers and external presentations for the end user. For example, in the manufacturing industry, the Digital Intelligence Agent provides guidance and explanations to users based on various aspects of the product, including product introductions, usage, shelf life, and efficacy.

Claims

1. An intelligent agent-driven hyper-realistic digital avatar system, characterized in that: This includes training a large model knowledge base for a specific vertical domain, voiceprint training and inference based on a Transformer model, deep learning-based audio conversion into facial and body animation algorithms, and corresponding presentation methods. The large model training for a specific vertical domain involves training a private large model using collected intent sets, performing voiceprint cloning and facial animation inference using collected character datasets, and then inferring digital avatar expression content based on the content of the large model knowledge base.

2. As claimed in claim 1: train a vertical domain large model knowledge base, use a given intent set, and form a knowledge base unique to the intent set through private large model training.

3. As claimed in claim 1: Based on the voiceprint training and reasoning of the tranformer model, the target person's voiceprint and emotion dataset is used to train the target person's corresponding voiceprint and emotion model to achieve the effect of cloning the target person's voiceprint and emotion.

4. As claimed in claim 1: A deep learning-based audio-to-facial and body animation algorithm uses a dataset of the target person’s image, action, and gesture to train a model of the target person’s corresponding image, action, and gesture, so as to achieve the effect of being able to infer facial animation and body movements based on sound.

5. As claimed in claim 1: The intelligent body drives the corresponding display mode of the ultra-realistic digital avatar, and conducts user inquiries through apps, web pages, mini-programs, etc., and the target person's external image and voice respond to the knowledge base content.

Images