Multi-modal fusion-based AI digital human broadcast method and application platform

Abstract

The invention provides an AI digital human broadcast method based on multi-modal fusion and an application platform, and the method comprises the steps: 1, collecting multi-modal original data, and carrying out the preprocessing of the multi-modal original data, wherein the multi-modal original data comprise multi-modal original data composed of voices, texts and images; step 2, carrying out multi-modal feature extraction and fusion on the preprocessed multi-modal data; 3, based on the fused representation vector, generating voice broadcast content of the AI digital human, synchronously generating a dynamic image and a text prompt matched with the voice broadcast content, and ensuring multi-dimensional display of information; and step 4, optimizing and outputting the generated voice, text and image broadcast content. According to the invention, by optimizing the speech speed, intonation, expression and action details of the AI digital person and providing multiple sensory stimuli such as vision and hearing, the watching experience and interaction willingness of the user are remarkably improved, the brand image and memory points can be enhanced, and the conversion rate and application value of an AI digital person broadcasting platform are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, multimedia processing, natural language processing and computer graphics, and in particular to an AI digital human broadcasting method and application platform based on multimodal fusion. Background Art

[0002] With the rapid development of artificial intelligence technology, AI digital humans have become an important tool for information display and interaction in various industries. In existing technologies, AI digital humans can already realize basic voice broadcasting, text display and simple image display functions, but most of them are limited to a single mode or simple combination, lacking deep integration and interaction.

[0003] In the prior art, AI digital human broadcasting mainly includes two categories: one is the AI ​​digital human broadcasting system based on a single mode or simple dual mode (such as voice + text). Although such systems can realize basic functions, the interactive experience is single and it is difficult to attract user attention; the second is the AI ​​application that attempts to perform multimodal fusion, but it is often not widely used due to the high complexity of technical implementation and poor fusion effect. For example, Chinese patent CN 118052918A discloses a multimodal interaction system, which involves a variety of information such as voice and images, but has deficiencies in deep fusion and collaborative display between modalities. The existing AI digital human broadcasting platform has the following technical shortcomings in modal fusion: first, there is a lack of effective coordination between modalities, resulting in fragmented information display and incoherent user experience; second, the fusion algorithm is highly complex, making it difficult to achieve real-time, high-quality broadcasting effects; third, the application scenario is single, making it difficult to meet the diversified needs of bank brand promotion and marketing activities. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides an AI digital human broadcasting method and application platform based on multimodal fusion. The present invention aims to innovate a multimodal fusion technology to solve the above problems and improve the comprehensive performance and application value of the AI ​​digital human broadcasting platform.

[0005] According to a first aspect of the present invention, the present invention provides an AI digital human broadcasting method based on multimodal fusion, comprising the following steps:

[0006] Step 1: Collect and preprocess multimodal raw data, where the multimodal data includes raw data of multiple modes including speech, text, and image;

[0007] Step 2: Extract and fuse multimodal features of the preprocessed multimodal data;

[0008] Step 3: Generate the voice broadcast content of the AI ​​digital human based on the fused representation vector, and simultaneously generate dynamic images and text prompts that match the voice broadcast content to ensure the multi-dimensional display of information;

[0009] Step 4: Optimize and output the generated voice, text and image broadcast content.

[0010] Based on the above technical solution, the present invention can also make the following improvements.

[0011] Optionally, the preprocessing includes: cleaning and denoising the collected voice, text and image data, converting the format of the image data, and marking objects or areas in the image.

[0012] Optionally, the extracting and fusing multimodal features of the preprocessed multimodal data includes:

[0013] The deep learning model is used to extract the feature vectors of speech, text and image respectively. The deep learning model includes RNN and CNN models. RNN is used to extract speech feature vectors, and CNN is used to extract text feature vectors and image feature vectors.

[0014] Optionally, it also includes: designing a multimodal fusion algorithm to fuse the extracted feature vectors to form a unified representation vector; the multimodal fusion algorithm includes feature-level fusion, hybrid-level fusion and model-level fusion.

[0015] Optionally, the multimodal fusion algorithm may adopt a fusion model architecture including Transformer, hybrid neural network and an attention-based fusion method and a tensor-based fusion method.

[0016] Optionally, the AI ​​digital human is a 3D digital human model that can adapt to a variety of broadcast styles and has text input and preprocessing, speech synthesis, digital human animation generation, multi-mode content synthesis, output and display functions.

[0017] Optionally, the optimizing and outputting the generated voice, text and image broadcast contents includes:

[0018] Use natural language processing and computer graphics technology to optimize the generated broadcast content, such as adjusting speech speed, tone, expression, action, etc., to enhance the realism and attractiveness of the broadcast;

[0019] The optimized multimodal broadcast content is output to the target platform for users to watch and interact.

[0020] According to a second aspect of the present invention, there is provided an AI digital human broadcasting platform based on multimodal fusion, comprising:

[0021] A data acquisition unit, used to acquire and pre-process multimodal raw data, wherein the multimodal data includes raw data of multiple modes consisting of voice, text, and image;

[0022] A data processing unit, used for extracting and fusing multimodal features of the preprocessed multimodal data;

[0023] The AI ​​digital broadcast unit is used to generate the AI ​​digital human’s voice broadcast content based on the fused representation vector, and simultaneously generate dynamic images and text prompts that match the voice broadcast content to ensure the multi-dimensional display of information;

[0024] The broadcast content optimization unit is used to optimize and output the generated voice, text and image broadcast content.

[0025] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, an AI digital human broadcasting method based on multimodal fusion as described above is implemented.

[0026] Technical effects and advantages of the present invention:

[0027] The present invention provides an AI digital human broadcasting method and application platform based on multimodal fusion, which realizes the deep fusion and collaborative display of multimodal information through innovative fusion algorithms, and improves the consistency of information transmission and user experience. The use of efficient deep learning models and optimization algorithms ensures the real-time generation and high-quality output of multimodal broadcast content, meeting the timeliness requirements of bank brand promotion and marketing activities. Through modular design and configurable interfaces, the AI ​​digital human broadcasting platform proposed in this application can easily adapt to a variety of application scenarios, providing banks with more flexible and efficient brand promotion and marketing tools. By optimizing the speech speed, tone, expression, movement and other details of the AI ​​digital human, and providing multi-sensory stimulation (such as vision and hearing), the application proposal significantly improves the user's viewing experience and willingness to interact, helps to enhance the brand image and memory points, and improves the conversion rate.

[0028] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the AI ​​digital human broadcasting method based on multimodal fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It is understandable that, based on the defects in the background technology, the embodiment of the present invention proposes an AI digital human broadcasting method based on multimodal fusion, specifically as follows: Figure 1 As shown, the following steps are included:

[0032] Step 1: Collect and preprocess multimodal raw data, where the multimodal data includes raw data of multiple modes including speech, text, and image;

[0033] In this embodiment, raw data of multiple modalities such as voice, text, and images are collected; for scenarios where multiple modal data need to be collected simultaneously, the following methods can be considered: First, multimodal data collection platforms. Some platforms provide multimodal data collection and processing services. Second, custom data collection systems. According to specific needs, custom data collection systems are built using programming languages ​​(such as Python, Java, etc.) and database technologies (such as MySQL, MongoDB, etc.).

[0034] The collected data is cleaned, denoised, and formatted in a unified preprocessing operation to ensure data quality. Specifically, the voice data cleaning and denoising adopts a self-supervised cleaning method. That is, the initial speaker recognition model is first trained using the data from which the first two errors have been removed, and then the model is used to calculate a center point for each speaker in the data. The main purpose of text data cleaning and denoising is to remove irrelevant characters, normalize text, and segment text. Image data cleaning and denoising converts image data into a format suitable for subsequent processing, and objects or areas in the image are annotated, such as using annotation boxes, annotation points, etc.

[0035] Step 2: Extract and fuse multimodal features of the preprocessed multimodal data;

[0036] In an embodiment, a deep learning model (such as RNN, CNN, etc.) is used to extract feature vectors of speech, text, and image respectively.

[0037] RNN extracts speech feature vectors. RNN performs well in processing sequence data, so it is often used to extract speech feature vectors. Specific methods include: Preprocessing: Preprocess the speech signal, such as framing, windowing, pre-emphasis, etc., to extract the short-term features of the speech signal. Feature extraction: Using the cyclic structure of RNN, the temporal features of the speech signal are input into the network frame by frame, and the temporal dependencies in the speech signal are captured through the state transfer of the hidden layer. Common feature extraction methods include linear prediction coefficient (LPC), perceptual linear prediction (PLP), Mel-frequency cepstral coefficient (MFCC), etc. Feature vector generation: In the output layer of RNN, the state of the hidden layer is converted into a feature vector through a fully connected layer or a softmax layer. These feature vectors can be used for subsequent tasks such as speech recognition and speech synthesis.

[0038] CNN extracts text feature vectors. CNN performs well in processing image data, but because text data can also be regarded as a special "image" (i.e., a matrix composed of word vectors), CNN is also widely used in text feature extraction. Specific methods include: Word embedding: Convert each word in the text into a word vector, usually using a pre-trained word embedding model (such as Word2Vec, GloVe, etc.) to obtain the word vector. Convolution operation: Use multiple convolution kernels of different sizes to perform sliding convolution on the word vector matrix to extract local features in the text (such as n-gram features). These convolution kernels can be regarded as different filters for capturing different patterns in the text. Pooling operation: Pool the output of the convolution layer (such as maximum pooling, average pooling, etc.) to reduce the size of the feature map and retain the most important features. Fully connected layer: Connect the output of the pooling layer to the fully connected layer, and combine the local features into a global feature vector through the transformation of the weight matrix.

[0039] CNN extracts image feature vectors. CNN has a wide range of applications in image feature extraction. Specific methods include: Convolution layer: Use multiple convolution kernels to perform convolution operations on the input image to extract low-level features such as edges and textures in the image. These convolution kernels can be regarded as different filters used to capture different patterns in the image. Pooling layer: Pooling operations are performed on the output of the convolution layer to reduce the size of the feature map and retain the most important features. Pooling operations help reduce the amount of computation, reduce overfitting, and improve the robustness of the model. Fully connected layer: Connect the output of the pooling layer to the fully connected layer, and combine low-level features into high-level feature vectors through the transformation of the weight matrix. These high-level feature vectors can be used for subsequent image classification, detection and other tasks.

[0040] Furthermore, a multimodal fusion algorithm is designed to fuse the extracted feature vectors to form a unified representation vector;

[0041] It should be noted that the multimodal fusion algorithm refers to the combination of data from different modalities (such as vision, hearing, text, etc.). It includes: feature-level fusion (early fusion), hybrid-level fusion and model-level fusion; among them,

[0042] Feature-level fusion (early fusion): A method that concatenates features extracted from different modalities into a single high-dimensional feature vector immediately after feature extraction. Often combined with feature extraction methods to remove redundant information, such as principal component analysis (PCA), maximum relevance minimum redundancy algorithm (mRMR), autoencoders, etc. Decision-level fusion (late fusion): After obtaining the decisions based on each modality, an integration step is performed on these decisions by applying algebraic combination rules (such as maximum, minimum, sum, average, etc.) of multiple predicted class labels. It is unable to capture the correlation between different modalities because these modalities are assumed to be independent.

[0043] Hybrid-level fusion: A combination of early fusion and late fusion, which improves the limitations of feature-level fusion and decision-level fusion by combining early fusion with the output of single modality prediction. However, the problem has not been fundamentally solved, but it is just a compromise between the two solutions.

[0044] Model-level fusion: Aims to obtain a joint feature representation of the three modalities. The implementation depends mainly on the fusion model used. It is a deeper fusion method that produces a more optimized joint discriminative feature representation for classification and regression tasks.

[0045] The development trend of multimodal fusion algorithms is to adopt more efficient fusion model architectures: such as Transformer, hybrid neural networks, etc., which can more effectively process and combine data of different modalities. More advanced fusion methods: such as attention-based fusion methods, tensor-based fusion methods, etc., can more accurately capture the association and interaction between different modalities. Wider application scenarios: With the continuous development of technology, multimodal fusion algorithms will be applied in more fields, such as autonomous driving, smart homes, telemedicine, etc.

[0046] Step 3: Generate the voice broadcast content of the AI ​​digital human based on the fused representation vector, and simultaneously generate dynamic images and text prompts that match the voice broadcast content to ensure the multi-dimensional display of information;

[0047] In this embodiment, a 3D digital human model is required, which should be highly customizable and flexible to adapt to different broadcasting needs and scenarios. The digital human model should include details of face, body and expression to achieve a more natural broadcasting effect.

[0048] The digital human model should have: Data resources: A large amount of voice and text data is required to train and optimize the AI ​​model. This data should cover different languages, accents, speaking speeds and intonations, etc., to ensure that the AI ​​digital human can adapt to a variety of broadcasting styles.

[0049] Text input and preprocessing: The user inputs the text content to be broadcast. The text is preprocessed, including word segmentation, part-of-speech tagging, syntactic analysis, etc., for subsequent natural language processing.

[0050] Speech synthesis: Using speech synthesis technology, the pre-processed text content is converted into speech signals. In this process, parameters such as speech speed, tone, and volume can be adjusted to meet different broadcasting needs.

[0051] Digital human animation generation: Generate digital human lip shape, facial expression, body movement and other animation effects based on voice signals and text content. This requires the use of knowledge and technology in multiple fields such as computer graphics, animation technology and AI algorithms;

[0052] Multi-mode content synthesis: Synchronize and merge the generated voice signal and digital human animation effects to form a complete AI digital human voice broadcast multi-mode content. In this process, it is necessary to ensure the synchronization and coordination between voice and animation to achieve a more natural broadcast effect.

[0053] Output and display: The generated AI digital human voice broadcast multi-mode content is output to the designated platform or device, such as TV, computer, mobile phone, etc. If needed, the output content can be further edited and optimized to improve its quality and effect.

[0054] Step 4: Optimize and output the generated voice, text and image broadcast content.

[0055] The optimization and output of the generated voice, text and image broadcast content comprises the following steps:

[0056] Step 4.1: Use natural language processing and computer graphics technology to optimize the generated broadcast content, such as adjusting the speech speed, tone, expression, action, etc., to enhance the realism and attractiveness of the broadcast;

[0057] Step 4.2: Output the optimized multimodal broadcast content to the target platform (such as the bank’s official website, APP, social media, etc.) for users to watch and interact.

[0058] This technical proposal has the following features:

[0059] Deep fusion: Through innovative fusion algorithms, deep fusion of multiple modal information such as voice, text, and images is achieved to improve the consistency and richness of information display.

[0060] Real-time and efficient: Adopt efficient deep learning models and optimization algorithms to ensure real-time generation and high-quality output of multimodal broadcast content.

[0061] Flexible application: Supports a variety of application scenarios, meets the needs of different bank brand promotion and marketing activities, and improves user experience and conversion rate.

[0062] Multimodal fusion algorithm: especially the deep fusion algorithm for modal information such as speech, text, and image, is one of the core technologies of the present invention.

[0063] AI digital human broadcast content generation and optimization technology: including broadcast content generation based on fusion representation vectors, and the application of natural language processing and computer graphics technology in broadcast content optimization.

[0064] Flexible application scenario adaptability: Through modular design and configurable interfaces, the platform can easily adapt to the needs of different bank brand promotion and marketing activities.

[0065] Deep fusion and collaborative display: Compared with the simple combination or single modal display in the existing technology, this application proposal achieves deep fusion and collaborative display of multimodal information through an innovative fusion algorithm, thereby improving the consistency of information transmission and user experience.

[0066] Efficient and real-time: The use of efficient deep learning models and optimization algorithms ensures the real-time generation and high-quality output of multimodal broadcast content, meeting the timeliness requirements of bank brand promotion and marketing activities.

[0067] Wide applicability: Through modular design and configurable interfaces, the AI ​​digital human broadcasting platform proposed in this application can easily adapt to a variety of application scenarios, providing banks with more flexible and efficient brand promotion and marketing tools.

[0068] Enhanced user experience: By optimizing the AI ​​digital person's speech speed, tone, expression, movement and other details, and providing multi-sensory stimulation (such as vision and hearing), this application proposal significantly improves the user's viewing experience and willingness to interact, helps to enhance the brand image and memory points, and improves conversion rate.

[0069] According to a second aspect of the present invention, there is provided an AI digital human broadcasting platform based on multimodal fusion, comprising:

[0070] A data acquisition unit, used to acquire and pre-process multimodal raw data, wherein the multimodal data includes raw data of multiple modes consisting of voice, text, and image;

[0071] A data processing unit, used for extracting and fusing multimodal features of the preprocessed multimodal data;

[0072] The AI ​​digital broadcast unit is used to generate the AI ​​digital human’s voice broadcast content based on the fused representation vector, and simultaneously generate dynamic images and text prompts that match the voice broadcast content to ensure the multi-dimensional display of information;

[0073] The broadcast content optimization unit is used to optimize and output the generated voice, text and image broadcast content.

[0074] In addition, an embodiment of the present invention further provides an electronic device, which includes: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other through the communication bus. The processor can call the logic instructions in the memory to execute the steps of the AI ​​digital human broadcasting method based on multimodal fusion.

[0075] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI digital human broadcasting method based on multimodal fusion, characterized in that: The following steps are involved: Step 1: Collect and preprocess multimodal raw data, where the multimodal data includes raw data of multiple modes including speech, text, and image; Step 2: Extract and fuse multimodal features of the preprocessed multimodal data; Step 3: Generate the voice broadcast content of the AI ​​digital human based on the fused representation vector, and simultaneously generate dynamic images and text prompts that match the voice broadcast content; Step 4: Optimize and output the generated voice, text and image broadcast content.

2. According to the AI ​​digital human broadcasting method based on multimodal fusion according to claim 1, it is characterized in that: The preprocessing includes: cleaning and denoising the collected voice, text and picture data, converting the format of the image data, and marking the objects or areas in the image.

3. The AI ​​digital human broadcasting method based on multimodal fusion according to claim 1 is characterized in that: The multimodal feature extraction and fusion of the preprocessed multimodal data includes: The deep learning model is used to extract the feature vectors of speech, text, and image respectively. The deep learning model includes an RNN model and a CNN model. The RNN model is used to extract the speech feature vector, and the CNN is used to extract the text feature vector and the image feature vector.

4. The AI ​​digital human broadcasting method based on multimodal fusion according to claim 3 is characterized in that: Also includes: Design a multimodal fusion algorithm to fuse the extracted feature vectors to form a unified representation vector; The multimodal fusion algorithm includes feature-level fusion, hybrid-level fusion and model-level fusion.

5. The AI ​​digital human broadcasting method based on multimodal fusion according to claim 1 is characterized in that: The multimodal fusion algorithm may adopt a fusion model architecture including Transformer, hybrid neural network, and an attention-based fusion method and a tensor-based fusion method.

6. The AI ​​digital human broadcasting method based on multimodal fusion according to claim 1 is characterized in that: The AI ​​digital human is a 3D digital human model, which can adapt to a variety of broadcast styles and has text input and preprocessing, speech synthesis, digital human animation generation, multi-mode content synthesis, output and display functions.

7. The AI ​​digital human broadcasting method based on multimodal fusion according to claim 1 is characterized in that: The optimization and output of the generated voice, text and image broadcast content includes: Use natural language processing and computer graphics technology to optimize the generated broadcast content, including adjusting speech speed, tone, expression, and movement to enhance the realism and appeal of the broadcast; The optimized multimodal broadcast content is output to the target platform for users to watch and interact.

8. An AI digital human broadcasting platform based on multimodal fusion, characterized in that: include: A data acquisition unit, used to acquire and pre-process multimodal raw data, wherein the multimodal data includes raw data of multiple modes consisting of voice, text, and image; A data processing unit, used for extracting and fusing multimodal features of the preprocessed multimodal data; The AI ​​digital broadcast unit is used to generate the voice broadcast content of the AI ​​digital person based on the fused representation vector, and simultaneously generate dynamic images and text prompts that match the voice broadcast content; The broadcast content optimization unit is used to optimize and output the generated 3D digital human model broadcast content.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, an AI digital human broadcasting method based on multimodal fusion as described in any one of claims 1 to 6 is implemented.

Images