A digital human live broadcast method based on GAN neural network system

Abstract

The present invention discloses a digital human live broadcast method based on a GAN neural network system, and relates to the technical field of network live broadcast. The present invention adopts natural language processing (NLP) technology to cluster and merge user questions, and quickly identify and screen similar questions; uses a machine learning algorithm to prioritize questions, and automatically generates partial replies in combination with a preset answer database, so that a virtual human only needs to answer core questions, greatly improving interaction efficiency and user experience; uses a conditional generative adversarial network (cGAN) to generate images in combination with environmental audio and real-time text information, and a multi-scale generator architecture generates a high-resolution local detail image while generating a low-resolution global image, and adopts a multi-discriminator structure to perform global image discrimination, local detail discrimination and facial expression discrimination respectively; each discriminator works independently and outputs results according to its own discrimination criteria.

Description

Technical Field

[0001] The present invention relates to the technical field of network live broadcasting, and in particular to a digital human live broadcasting method based on a GAN neural network system. Background Art

[0002] Digital human live broadcast is a form of live broadcast using virtual characters using artificial intelligence technology. Digital human is a virtual digital model that can simulate human appearance and behavior, and perform corresponding actions and expressions based on the audience's input information. Digital human live broadcast can reduce labor costs and venue costs, improve work efficiency, achieve full-time exposure, and increase audience interactivity and experience.

[0003] For example, the patent publication number CN117119207A discloses a digital human live broadcast method, which is applied to a digital human short video production platform. The platform is provided with a model unit, a template unit, a sound unit, a script unit, a management unit and a live broadcast unit. The method includes: obtaining a target digital human model, a target template, a target sound and a target script, and uniformly displaying the target digital human model, the target template, the target sound and the target script to the user; managing and editing the target digital human model, the target template, the target sound and the target script to produce and process the digital human live broadcast; and performing a digital human live broadcast based on the target digital human model and the corresponding target template, the target sound and the target script through the digital human short video production platform.

[0004] However, traditional digital human live broadcast methods rely on preset responses and scripts, and cannot effectively handle a large number of questions raised by users in real time, resulting in low interaction efficiency and poor user experience. At the same time, when training the generator and discriminator, there is a lack of dynamic adjustment mechanism, and the model cannot be adjusted in time according to the loss changes during the training process, resulting in poor generation and discrimination effects. Therefore, a digital human live broadcast method based on the GAN neural network system is urgently needed to solve such problems. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a digital human live broadcast method based on the GAN neural network system, which solves the problems in the prior art that it relies on preset responses and scripts, cannot effectively process questions raised by a large number of users in real time, resulting in low interaction efficiency and poor user experience. At the same time, when training the generator and the discriminator, there is a lack of a dynamic adjustment mechanism, and the model cannot be adjusted in time according to the loss changes during the training process.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] The present invention provides a digital human live broadcast method based on a GAN neural network system. The GAN neural network system includes: an input module, a data preprocessing module, a generation module, a discrimination module, a rendering module and an output module. The digital human live broadcast method based on the GAN neural network system includes:

[0008] Step 1. The input module uses a microphone to collect audio and text instructions input by the user, and uses the instructions to control the behavior and conversation of the virtual person based on natural language processing technology; batch problem processing is also performed at the same time;

[0009] The input module has a built-in batch question processing module, which uses natural language processing (NLP) technology to cluster questions raised by users, identify similar questions and merge them; quickly cluster and filter user questions to improve response efficiency, and prioritize questions according to their importance and frequency, automatically generate replies to some questions, and the virtual person only needs to answer core questions; use machine learning algorithms to prioritize questions, combine with a preset answer database, and automatically generate some replies to improve the efficiency of live interaction;

[0010] Step 2. The data preprocessing module preprocesses the collected audio and text data, including denoising and format conversion;

[0011] Step 3. The generation module uses conditional GAN ​​to take the preprocessed audio and text data as conditional input to generate a virtual human image that is semantically consistent with the input data; a conditional input module is added to the generator to generate images in combination with environmental audio and real-time text information; the semantic consistency and expressiveness of the generated images are improved; the generation module adopts a multi-scale generator architecture to simultaneously generate low-resolution global images and high-resolution local detail images, and fuses the images, using an adaptive weight mechanism to dynamically adjust the fusion ratio of features at each scale to improve the overall quality and detail performance of the image;

[0012] Step 4. The discriminator module adopts a multi-discriminator structure. Different discriminators perform global image discrimination, local detail discrimination, and facial expression discrimination respectively. The multi-discriminator structure improves the stability and evaluation accuracy of the discriminator.

[0013] Step 5. Dynamic weight adjustment of adversarial training: dynamically adjust the weight of the loss function according to the loss changes of the generator and discriminator;

[0014] Step 6. Image post-processing and rendering: Post-process the generated virtual human image based on the rendering module, including detail enhancement and color adjustment, and fuse the processed image with the background video to generate the final live broadcast screen; use image enhancement technology based on the attention mechanism to improve the detail expression of the image;

[0015] Step 7. Live output: the output module encodes and compresses the final live picture and pushes it in real time through the live broadcast platform.

[0016] The present invention is further configured as follows: the problem identification and merging method in the batch problem processing module is:

[0017] s' i =clean(s i ), where s' i is the cleaned text, clean(s i ) is the text after the cleanup function is applied;

[0018] Perform feature extraction, use word embedding technology to convert text into vector representation, and weight word embedding by combining word frequency TF and inverse document frequency IDF:

[0019] Among them, v i is the weighted word embedding vector of question i, TF-IDF(t,doc,D) is the TF-IDF value of word t in document doc, and Embedding(t) is the word embedding vector of word t;

[0020] Calculate the cosine similarity matrix between all question vectors:

[0021] Among them, v i and v j is the weighted word embedding vector of the two questions;

[0022] The present invention is further configured as follows: the priority sorting method in the batch problem processing module is:

[0023] Use the similarity matrix to build a graph, and use the spectral clustering algorithm to cluster the questions. Construct the Laplace matrix L = D1-A, and use the spectral clustering algorithm to cluster the questions:

[0024] L sym =D1 -1 / 2 LD1 -1 / 2 , where L is the Laplace matrix, D1 is the degree matrix, and A is the similarity matrix;

[0025] Decompose the eigenvalues ​​of the Laplace matrix, select the eigenvectors corresponding to the first k smallest eigenvalues, and construct the eigenvector matrix:

[0026] U=[u1,u2,…,u k ],in, are the eigenvectors of the Laplacian matrix;

[0027] Perform Kmeans clustering on the eigenvector matrix U to obtain the final clustering result:

[0028] Among them, S is the clustering result, k is the number of clusters, is the centroid of the i1th cluster;

[0029] The present invention is further configured as follows: the priority sorting method in the batch question processing module also includes:

[0030] Based on the clustering results, similar questions are merged and sorted using a PageRank-based sorting algorithm based on the importance and frequency of the questions:

[0031] Among them, PR (p i ) is the problem p i PageRank value, d is the damping factor, N is the total number of questions, M(p i ) is pointing to the problem p i The problem set, L(p j ) is the problem p j The number of outgoing links;

[0032] Use a pre-trained natural language generation model to generate partial replies, based on a preset answer database and combined with the replies of the generation model; the batch question processing module uses NLP technology, graph clustering, and a PageRank-based sorting algorithm to perform complex clustering and priority sorting of user questions, improving the efficiency and accuracy of batch question processing, accurately identifying similar questions and merging them through feature extraction and similarity calculation, optimizing the interactive effect of virtual people during live broadcasts, improving user experience, and significantly reducing the burden of virtual people handling redundant questions;

[0033] The present invention is further configured as follows: in step 3, the image generation method based on the conditional input module is:

[0034] Perform environmental audio processing, extract Mel-spectrograms as audio features, and use word embedding models to convert text into high-dimensional vector representations; use audio features and text vectors as conditional inputs to construct a conditional generative adversarial network (cGAN);

[0035] The features of audio and text are fused to generate multimodal feature representation. The fusion function uses linear weighted summation and attention mechanism.

[0036] The generator accepts multimodal features as conditional input and generates images that are semantically consistent with the input:

[0037] I gen =G(z|f multi ), where I gen is the generated image, G is the generator, z is the random noise vector, f multiis the multimodal feature vector;

[0038] The generator network structure includes:

[0039] Input layer, input random noise and multimodal features, z input =[z,f multi ];

[0040] A multi-layer convolutional neural network (CNN) that uses convolutional and deconvolutional layers to generate images:

[0041] in, is the feature map of the i2th layer, is the convolution kernel of the i2th layer, is the bias of the i2th layer, * is the convolution operation, and σ is the activation function;

[0042] The present invention is further configured as follows: in step 3, the image generation method based on the conditional input module also includes:

[0043] Generate a low-resolution global image and a high-resolution local detail image, and fuse them:

[0044] I final =αI global +(1-α)I local , where I global is a low-resolution global image, I local is a high-resolution local detail image, α is an adaptive weight, I final To generate the final image, global and local details are fused;

[0045] The discriminator uses a multi-discriminator structure to discriminate global images, local details, and facial expressions respectively:

[0046] D total =D global (I input )+D local (I input )+D face (I input ), where I input is the input image, D global is the global discriminator, D local is the local discriminator, D face is the facial expression discriminator;

[0047] The loss function weights are dynamically adjusted according to the loss changes of the generator and discriminator; Generator loss:

[0048]

[0049] Discriminator loss:

[0050] Among them, L G is the loss function of the generator, L D is the loss function of the discriminator, L cond is the conditional loss, λ is the weight coefficient; use image post-processing technology to perform detail enhancement and color adjustment, and fuse the generated image with the background video; through multimodal feature fusion, generative adversarial network, multi-scale generation and discrimination technology, ensure the authenticity and detail richness of the generated image;

[0051] The present invention is further configured as follows: different discriminators of the discrimination module in step 4 respectively perform global image discrimination, local detail discrimination and facial expression discrimination, and the discrimination step comprises:

[0052] The global image discriminator is responsible for judging the overall structure and content consistency of the image;

[0053] Receive the generated image as input, use the convolutional neural network CNN to extract the global features of the image, input the extracted global features into the fully connected layer, and output a judgment result to determine whether the image is real;

[0054] Local detail discriminator, randomly crops multiple local regions from the generated image;

[0055] Use convolutional neural network (CNN) to extract detailed features for each local area;

[0056] Input the features of each local area into the fully connected layer and output the discrimination results respectively;

[0057] Combining the discrimination results of multiple local areas, the overall discrimination result is obtained;

[0058] Facial expression discriminator, which uses face detection algorithms to locate facial regions in images;

[0059] Use convolutional neural network (CNN) to extract facial features from the detected facial area;

[0060] Perform expression analysis on the extracted facial features to determine the authenticity of facial expressions;

[0061] The facial expression analysis results are input into the fully connected layer and the discrimination results are output;

[0062] The present invention is further configured as follows: the workflow of the multi-discriminator structure includes:

[0063] The global image discriminator, local detail discriminator and facial expression discriminator discriminate the same generated image simultaneously;

[0064] Each discriminator works independently and outputs the discrimination result according to its own discrimination criteria;

[0065] The final discrimination result is obtained by synthesizing the output results of each discriminator, and the output of each discriminator is synthesized by logistic regression method; the stability and accuracy of discrimination are improved; by combining the discrimination results of different discriminators, the overall discrimination accuracy can be improved, and the generated image has high quality and consistency in global structure, local details and facial expressions;

[0066] The present invention is further configured such that the dynamic weight adjustment of the adversarial training in step 5 maintains a balance between the generator and the discriminator, ensuring that the generator generates a real image while the discriminator effectively distinguishes between the real and generated images, and the dynamic weight adjustment method is:

[0067] Set initial weights for the loss functions of the generator and discriminator;

[0068] Use the initial weights to perform a certain number of adversarial training rounds to collect initial loss change data;

[0069] After each training round, monitor the loss values ​​of the generator and discriminator, and record the loss value of the generator (i.e., the difference between the generated image and the real image) and the loss value of the discriminator (the accuracy of distinguishing the real image from the generated image);

[0070] Analyze the loss trend of multiple rounds to determine the training status of the generator and discriminator;

[0071] If the loss value of the generator decreases slowly or tends to be stable, while the loss value of the discriminator decreases quickly, it means that the discriminator is too strong and the weight of the generator needs to be increased, and vice versa;

[0072] Dynamically adjust the weights of the generator and discriminator based on the loss trend analysis results;

[0073] If the loss value of the generator decreases too slowly, increase the weight of the generator; if the loss value of the discriminator decreases too slowly, increase the weight of the discriminator to improve the accuracy of the discrimination;

[0074] The present invention is further configured as follows: the weight updating method in step 5 is:

[0075] After each training round, the weights of the generator and discriminator are updated based on the loss monitoring and analysis results, and the weights are adjusted using a nonlinear method;

[0076] Through the feedback loop mechanism, the weights are continuously adjusted to keep the training of the generator and discriminator in a balanced state.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] The present invention uses natural language processing (NLP) technology to cluster and merge user questions, quickly identify and filter similar questions; uses machine learning algorithms to prioritize questions, and automatically generates partial replies in combination with a preset answer database, so that the virtual person only needs to answer core questions, greatly improving interaction efficiency and user experience;

[0079] The present invention uses conditional generative adversarial networks (cGANs) to generate images in combination with ambient audio and real-time text information, significantly improving the semantic consistency and expressiveness of generated images. The multi-scale generator architecture generates high-resolution local detail images while generating low-resolution global images.

[0080] The present invention adopts a multi-discriminator structure to perform global image discrimination, local detail discrimination and facial expression discrimination respectively; each discriminator works independently and outputs results according to its own discrimination criteria, and the outputs of each discriminator are integrated by a logistic regression method to improve the stability and evaluation accuracy of the discriminator, thereby ensuring that the generated image has high quality and consistency in global structure, local details and facial expressions;

[0081] The present invention introduces a dynamic weight adjustment mechanism for adversarial training, which dynamically adjusts the weight of the loss function by monitoring the loss value changes of the generator and the discriminator to ensure the balance between the generator and the discriminator.

[0082] The solution solves the problem that the existing technology relies on preset responses and scripts, cannot effectively handle a large number of questions raised by users in real time, resulting in low interaction efficiency and poor user experience. At the same time, when training the generator and discriminator, there is a lack of dynamic adjustment mechanism and the model cannot be adjusted in time according to the change of loss during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a flow chart of the digital human live broadcast method based on the GAN neural network system of the present invention. DETAILED DESCRIPTION

[0084] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0085] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0086] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0087] Example 1

[0088] See also Figure 1 The present invention provides a digital human live broadcast method based on a GAN neural network system. The GAN neural network system includes: an input module, a data preprocessing module, a generation module, a discrimination module, a rendering module and an output module. The digital human live broadcast method includes the following steps:

[0089] Step 1. The input module uses a microphone to collect audio and text instructions input by the user, and uses the instructions to control the behavior and conversation of the virtual person based on natural language processing technology; batch problem processing is also performed at the same time;

[0090] The input module has a built-in batch question processing module, which uses natural language processing (NLP) technology to cluster questions raised by users, identify similar questions and merge them; quickly cluster and filter user questions to improve response efficiency, and prioritize questions according to their importance and frequency, automatically generate replies to some questions, and the virtual person only needs to answer core questions; use machine learning algorithms to prioritize questions, combine with a preset answer database, automatically generate some replies, and improve the efficiency of live interaction; the question identification and merging methods in the batch question processing module are:

[0091] Data preprocessing is performed to remove stop words, punctuation marks, HTML tag noise data, and redundant whitespace characters. WordNetLemmatizer is used to restore words to their root form. The cleaning process is expressed as follows:

[0092] s' i =clean(s i ), where s' i is the cleaned text, clean(s i ) is the text after the cleanup function is applied;

[0093] Perform feature extraction, use word embedding technology to convert text into vector representation, and weight word embedding by combining word frequency TF and inverse document frequency IDF:

[0094] Among them, v i is the weighted word embedding vector of question i, TF-IDF(t,doc,D) is the TF-IDF value of word t in document doc, and Embedding(t) is the word embedding vector of word t;

[0095] Calculate the cosine similarity matrix between all question vectors:

[0096] Among them, v i and v j is the weighted word embedding vector of the two questions;

[0097] The priority sorting method in the batch problem processing module is:

[0098] Use the similarity matrix to build a graph, and use the spectral clustering algorithm to cluster the questions. Construct the Laplace matrix L = D1-A, and use the spectral clustering algorithm to cluster the questions:

[0099] L sym =D1 -1 / 2 LD1 -1 / 2 , where L is the Laplace matrix, D1 is the degree matrix, and A is the similarity matrix;

[0100] Decompose the eigenvalues ​​of the Laplace matrix, select the eigenvectors corresponding to the first k smallest eigenvalues, and construct the eigenvector matrix:

[0101] U=[u1,u2,…,u k ],in, are the eigenvectors of the Laplacian matrix;

[0102] Perform Kmeans clustering on the eigenvector matrix U to obtain the final clustering result:

[0103] Among them, S is the clustering result, k is the number of clusters, is the centroid of the i1th cluster;

[0104] Based on the clustering results, similar questions are merged and sorted using a PageRank-based sorting algorithm based on the importance and frequency of the questions:

[0105] Among them, PR (p i ) is the problem p iPageRank value, d is the damping factor, N is the total number of questions, M(p i ) is pointing to the problem p i The problem set, L(p j ) is the problem p j The number of outgoing links;

[0106] Use a pre-trained natural language generation model to generate partial replies, based on a preset answer database and combined with the replies of the generation model; the batch question processing module uses NLP technology, graph clustering, and a PageRank-based sorting algorithm to perform complex clustering and priority sorting of user questions, improving the efficiency and accuracy of batch question processing, accurately identifying similar questions and merging them through feature extraction and similarity calculation, optimizing the interactive effect of virtual people during live broadcasts, improving user experience, and significantly reducing the burden of virtual people handling redundant questions;

[0107] Step 2. The data preprocessing module preprocesses the collected audio and text data, including denoising and format conversion;

[0108] Step 3. The generation module uses conditional GAN ​​to take the preprocessed audio and text data as conditional input to generate a virtual human image that is semantically consistent with the input data; add a conditional input module to the generator to generate images in combination with ambient audio and real-time text information; improve the semantic consistency and expressiveness of the generated image;

[0109] The generation module adopts a multi-scale generator architecture to generate low-resolution global images and high-resolution local detail images at the same time, and fuses the images. It uses an adaptive weight mechanism to dynamically adjust the fusion ratio of each scale feature to improve the overall quality and detail performance of the image. The image generation method based on the conditional input module is as follows:

[0110] Perform environmental audio processing, extract Mel-spectrograms as audio features, and use word embedding models to convert text into high-dimensional vector representations; use audio features and text vectors as conditional inputs to construct a conditional generative adversarial network (cGAN);

[0111] The features of audio and text are fused to generate multimodal feature representation. The fusion function uses linear weighted summation and attention mechanism.

[0112] The generator accepts multimodal features as conditional input and generates images that are semantically consistent with the input:

[0113] I gen =G(z|f multi ), where I gen is the generated image, G is the generator, z is the random noise vector, f multi is the multimodal feature vector;

[0114] The generator network structure includes:

[0115] Input layer, input random noise and multimodal features, z input =[z,f multi ];

[0116] A multi-layer convolutional neural network (CNN) that uses convolutional and deconvolutional layers to generate images:

[0117] in, is the feature map of the i2th layer, is the convolution kernel of the i2th layer, is the bias of the i2th layer, * is the convolution operation, and σ is the activation function; a low-resolution global image and a high-resolution local detail image are generated and fused:

[0118] I final =αI global +(1-α)I local , where I global is a low-resolution global image, I local is a high-resolution local detail image, α is an adaptive weight, I final To generate the final image, global and local details are fused;

[0119] The discriminator uses a multi-discriminator structure to discriminate global images, local details, and facial expressions respectively:

[0120] D total =D global (I input )+D local (I input )+D face (I input ), where I input is the input image, D global is the global discriminator, D local is the local discriminator, D face is the facial expression discriminator;

[0121] The loss function weights are dynamically adjusted according to the loss changes of the generator and discriminator; Generator loss:

[0122]

[0123] Discriminator loss:

[0124] Among them, L G is the loss function of the generator, L D is the loss function of the discriminator, L condis the conditional loss, λ is the weight coefficient; image post-processing technology is used to perform detail enhancement and color adjustment, and the generated image is fused with the background video;

[0125] Through multimodal feature fusion, generative adversarial network, multi-scale generation and discrimination technology, the authenticity and detail richness of the generated images are guaranteed;

[0126] Step 4. The discriminant module adopts a multi-discriminator structure. Different discriminators perform global image discrimination, local detail discrimination, and facial expression discrimination respectively. The multi-discriminator structure improves the stability and evaluation accuracy of the discriminator. Different discriminators of the discriminant module perform global image discrimination, local detail discrimination, and facial expression discrimination respectively. The discrimination steps include:

[0127] The global image discriminator is responsible for judging the overall structure and content consistency of the image;

[0128] Receive the generated image as input, use the convolutional neural network CNN to extract the global features of the image, input the extracted global features into the fully connected layer, and output a judgment result to determine whether the image is real;

[0129] Local detail discriminator, randomly crops multiple local regions from the generated image;

[0130] Use convolutional neural network (CNN) to extract detailed features for each local area;

[0131] Input the features of each local area into the fully connected layer and output the discrimination results respectively;

[0132] Combining the discrimination results of multiple local areas, the overall discrimination result is obtained;

[0133] Facial expression discriminator, which uses face detection algorithms to locate facial regions in images;

[0134] Use convolutional neural network (CNN) to extract facial features from the detected facial area;

[0135] Perform expression analysis on the extracted facial features to determine the authenticity of facial expressions;

[0136] The facial expression analysis results are input into the fully connected layer and the discrimination results are output;

[0137] The workflow of the multi-discriminator structure includes:

[0138] The global image discriminator, local detail discriminator and facial expression discriminator discriminate the same generated image simultaneously;

[0139] Each discriminator works independently and outputs the discrimination result according to its own discrimination criteria;

[0140] The final discrimination result is obtained by synthesizing the output results of each discriminator, and the output of each discriminator is synthesized by logistic regression method; the stability and accuracy of discrimination are improved; by combining the discrimination results of different discriminators, the overall discrimination accuracy can be improved, and the generated image has high quality and consistency in global structure, local details and facial expressions;

[0141] Step 5. Dynamic weight adjustment of adversarial training. Dynamically adjust the weight of the loss function according to the loss changes of the generator and the discriminator. The dynamic weight adjustment of adversarial training maintains a balance between the generator and the discriminator, ensuring that the generator generates real images while the discriminator effectively distinguishes between real and generated images. The dynamic weight adjustment method is:

[0142] Set initial weights for the loss functions of the generator and discriminator;

[0143] Use the initial weights to perform a certain number of adversarial training rounds to collect initial loss change data;

[0144] After each training round, monitor the loss values ​​of the generator and discriminator, and record the loss value of the generator (i.e., the difference between the generated image and the real image) and the loss value of the discriminator (the accuracy of distinguishing the real image from the generated image);

[0145] Analyze the loss trend of multiple rounds to determine the training status of the generator and discriminator;

[0146] If the loss value of the generator decreases slowly or tends to be stable, while the loss value of the discriminator decreases quickly, it means that the discriminator is too strong and the weight of the generator needs to be increased, and vice versa;

[0147] Dynamically adjust the weights of the generator and discriminator based on the loss trend analysis results;

[0148] If the loss value of the generator decreases too slowly, increase the weight of the generator; if the loss value of the discriminator decreases too slowly, increase the weight of the discriminator to improve the accuracy of the discrimination;

[0149] After each training round, the weights of the generator and discriminator are updated based on the loss monitoring and analysis results, and the weights are adjusted using a nonlinear method;

[0150] Through the feedback loop mechanism, the weights are continuously adjusted to keep the training of the generator and the discriminator in a balanced state;

[0151] Step 6. Image post-processing and rendering: Post-process the generated virtual human image based on the rendering module, including detail enhancement and color adjustment, and fuse the processed image with the background video to generate the final live broadcast screen; use image enhancement technology based on the attention mechanism to improve the detail expression of the image;

[0152] Step 7. Live output: the output module encodes and compresses the final live picture and pushes it in real time through the live broadcast platform.

[0153] This application collects instructions based on natural language processing (NLP) technology to control the behavior and dialogue of virtual people. The input module has a built-in batch question processing module, which uses NLP technology to cluster questions raised by users, identify similar questions and merge them, quickly cluster and filter user questions, and improve response efficiency. Questions are prioritized according to importance and frequency, and partial replies are automatically generated using a preset answer database and machine learning algorithms, thereby improving the efficiency of live interaction; the data preprocessing module denoises and converts the format of the collected audio and text data;

[0154] The audio and text data preprocessed by the generation module are used as conditional inputs, and the conditional generative adversarial network cGAN is used to generate virtual human images that are semantically consistent with the input data. The conditional input module is added to the generator, and the image is generated by combining the environmental audio and real-time text information to improve the semantic consistency and expressiveness of the generated image; the multi-scale generator architecture enables the simultaneous generation of low-resolution global images and high-resolution local detail images, and dynamically adjusts the fusion ratio of each scale feature through an adaptive weight mechanism, significantly improving the overall quality and detail performance of the image;

[0155] The discriminator module adopts a multi-discriminator structure. Different discriminators perform global image discrimination, local detail discrimination, and facial expression discrimination respectively. The multi-discriminator structure improves the stability and evaluation accuracy of the discriminator. The global image discriminator is responsible for discriminating the overall structure and content consistency of the image, the local detail discriminator randomly crops multiple local areas from the generated image for detail discrimination, and the facial expression discriminator focuses on the authenticity and naturalness of the facial area. Each discriminator works independently and outputs the discrimination results according to its own discrimination criteria. Finally, the discrimination results are synthesized through the logistic regression method to improve the overall discrimination accuracy and stability.

[0156] The dynamic weight adjustment of adversarial training monitors the changes in the loss values ​​of the generator and the discriminator, dynamically adjusts the weight of the loss function to ensure the balance between the generator and the discriminator; after the initial setting of the weight, the trend of the loss value changes is monitored and analyzed after each training round; if the loss value of the generator decreases slowly or tends to be stable, while the loss value of the discriminator decreases rapidly, the weight of the generator is increased. Through this dynamic adjustment, the generator can generate higher quality images, and the discriminator can effectively distinguish between real and generated images.

[0157] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A digital human live broadcast method based on a GAN neural network system, characterized in that: The GAN neural network system includes: an input module, a data preprocessing module, a generation module, a discrimination module, a rendering module and an output module. The digital human live broadcast method based on the GAN neural network system includes: Step 1. The input module uses a microphone to collect audio and text instructions input by the user, and uses the instructions to control the behavior and conversation of the virtual person based on natural language processing technology; batch problem processing is also performed at the same time; The input module has a built-in batch question processing module, which uses natural language processing (NLP) technology to cluster questions raised by users, identify similar questions and merge them; it uses machine learning algorithms to prioritize questions and automatically generates partial replies based on the preset answer database; Step 2. The data preprocessing module preprocesses the collected audio and text data, including denoising and format conversion; Step 3. The generation module uses conditional GAN ​​to take the preprocessed audio and text data as conditional input to generate a virtual human image that is semantically consistent with the input data; a conditional input module is added to the generator to generate images in combination with environmental audio and real-time text information; the generation module adopts a multi-scale generator architecture to simultaneously generate low-resolution global images and high-resolution local detail images, and fuses the images, using an adaptive weight mechanism to dynamically adjust the fusion ratio of features at each scale; Step 4. The discrimination module adopts a multi-discriminator structure, and different discriminators perform global image discrimination, local detail discrimination, and facial expression discrimination respectively; Step 5. Dynamic weight adjustment of adversarial training: dynamically adjust the weight of the loss function according to the loss changes of the generator and discriminator; Step 6. Image post-processing and rendering: Post-process the generated virtual human image based on the rendering module, including detail enhancement and color adjustment, and fuse the processed image with the background video to generate the final live broadcast picture; Step 7. Live output: the output module encodes and compresses the final live picture and pushes it in real time through the live broadcast platform.

2. According to the method of claim 1, the digital human live broadcast method based on the GAN neural network system is characterized in that: The problem identification and merging method in the batch problem processing module is: Data preprocessing is performed to remove stop words, punctuation marks, HTML tag noise data, and redundant spaces. WordNetLemmatizer is used to restore words to their root form. The cleaning process is expressed as follows: s' i =clean(s i ), where s' i is the cleaned text, clean(s i ) is the text after the cleanup function is applied; Perform feature extraction, use word embedding technology to convert text into vector representation, and weight word embedding by combining word frequency TF and inverse document frequency IDF: Among them, v i is the weighted word embedding vector of question i, TF-IDF(t,doc,D) is the TF-IDF value of word t in document doc, and Embedding(t) is the word embedding vector of word t; Calculate the cosine similarity matrix between all question vectors: Among them, v i and v j are the weighted word embedding vectors of the two questions.

3. According to the method of claim 2, the digital human live broadcast method based on the GAN neural network system is characterized in that: The priority sorting method in the batch problem processing module is: Use the similarity matrix to build a graph, and use the spectral clustering algorithm to cluster the questions. Construct the Laplace matrix L = D1-A, and use the spectral clustering algorithm to cluster the questions: L sym =D1 -1 / 2 LD1 -1 / 2 , where L is the Laplace matrix, D1 is the degree matrix, and A is the similarity matrix; Decompose the eigenvalues ​​of the Laplace matrix, select the eigenvectors corresponding to the first k smallest eigenvalues, and construct the eigenvector matrix: U=[u1,u2,…,u k ],in, are the eigenvectors of the Laplacian matrix; Perform Kmeans clustering on the eigenvector matrix U to obtain the final clustering result: Among them, S is the clustering result, k is the number of clusters, is the centroid of the i1th cluster.

4. According to the method of claim 3, the digital human live broadcast method based on the GAN neural network system is characterized in that: The priority sorting methods in the batch problem processing module also include: Based on the clustering results, similar questions are merged and sorted using a PageRank-based sorting algorithm based on the importance and frequency of the questions: Among them, PR (p i ) is the problem p i PageRank value, d is the damping factor, N is the total number of questions, M(p i ) is pointing to the problem p i The problem set, L(p j ) is the problem p j The number of outgoing links; Use a pre-trained natural language generation model to generate partial responses, based on a preset answer database, combined with the response from the generation model.

5. A digital human live broadcast method based on a GAN neural network system according to claim 4, characterized in that: In step 3, the image generation method based on the conditional input module is: Perform environmental audio processing, extract Mel-spectrograms as audio features, and use word embedding models to convert text into high-dimensional vector representations; use audio features and text vectors as conditional inputs to construct a conditional generative adversarial network (cGAN); The features of audio and text are fused to generate multimodal feature representation. The fusion function uses linear weighted summation and attention mechanism. The generator accepts multimodal features as conditional input and generates images that are semantically consistent with the input: I gen =G(z|f multi ), where I gen is the generated image, G is the generator, z is the random noise vector, f multi is the multimodal feature vector; The generator network structure includes: Input layer, input random noise and multimodal features, z input =[z,f multi ]; A multi-layer convolutional neural network (CNN) that uses convolutional and deconvolutional layers to generate images: in, is the feature map of the i2th layer, is the convolution kernel of the i2th layer, is the bias of the i2th layer, * is the convolution operation, and σ is the activation function.

6. A digital human live broadcast method based on a GAN neural network system according to claim 5, characterized in that: In step 3, the image generation method based on the conditional input module also includes: Generate a low-resolution global image and a high-resolution local detail image, and fuse them: I final =αI global +(1-α)I local , where I global is a low-resolution global image, I local is a high-resolution local detail image, α is an adaptive weight, and is the final generated image, which combines global and local details; The discriminator uses a multi-discriminator structure to discriminate global images, local details, and facial expressions respectively: D total =D global (I input )+D local (I input )+D face (I input ), where I input is the input image, D global is the global discriminator, D local is the local discriminator, D face is the facial expression discriminator; The loss function weights are dynamically adjusted according to the loss changes of the generator and discriminator; Generator loss: Discriminator loss: Among them, L G is the loss function of the generator, L D is the loss function of the discriminator, L cond is the conditional loss, λ is the weight coefficient; image post-processing technology is used to perform detail enhancement and color adjustment, and the generated image is fused with the background video.

7. A digital human live broadcast method based on a GAN neural network system according to claim 6, characterized in that: In step 4, different discriminators of the discrimination module respectively perform global image discrimination, local detail discrimination and facial expression discrimination. The discrimination steps include: The global image discriminator is responsible for judging the overall structure and content consistency of the image; Receive the generated image as input, use the convolutional neural network CNN to extract the global features of the image, input the extracted global features into the fully connected layer, and output a judgment result to determine whether the image is real; Local detail discriminator, randomly crops multiple local regions from the generated image; Use convolutional neural network (CNN) to extract detailed features for each local area; Input the features of each local area into the fully connected layer and output the discrimination results respectively; Combining the discrimination results of multiple local areas, the overall discrimination result is obtained; Facial expression discriminator, which uses face detection algorithms to locate facial regions in images; Use convolutional neural network (CNN) to extract facial features from the detected facial area; Perform expression analysis on the extracted facial features to determine the authenticity of facial expressions; The facial expression analysis results are input into the fully connected layer and the discrimination results are output.

8. A digital human live broadcast method based on a GAN neural network system according to claim 7, characterized in that: The workflow of the multi-discriminator structure includes: The global image discriminator, local detail discriminator and facial expression discriminator discriminate the same generated image simultaneously; Each discriminator works independently and outputs the discrimination result according to its own discrimination criteria; The final discrimination result is obtained by combining the output results of each discriminator, and the output of each discriminator is synthesized using the logistic regression method.

9. A digital human live broadcast method based on a GAN neural network system according to claim 8, characterized in that: The dynamic weight adjustment of adversarial training in step 5 maintains a balance between the generator and the discriminator, ensuring that the generator generates realistic images while the discriminator effectively distinguishes between real and generated images. The dynamic weight adjustment method is: Set initial weights for the loss functions of the generator and discriminator; Use the initial weights to perform a certain number of adversarial training rounds to collect initial loss change data; After each training round, monitor the loss values ​​of the generator and discriminator, and record the loss values ​​of the generator and the discriminator; Analyze the loss trend of multiple rounds to determine the training status of the generator and discriminator; According to the loss trend analysis results, the weights of the generator and discriminator are dynamically adjusted.

10. A digital human live broadcast method based on a GAN neural network system according to claim 9, characterized in that: The weight update method in step 5 is: After each training round, the weights of the generator and discriminator are updated based on the loss monitoring and analysis results, and the weights are adjusted using a nonlinear method; Through the feedback loop mechanism, the weights are continuously adjusted to keep the training of the generator and discriminator in a balanced state.

Images