Interactive digital human generation method and system based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence digital persons, and particularly discloses an interactive digital person generation method and system based on artificial intelligence, and the method comprises the steps: collecting behavior data, voice data and image data of a real person; constructing a model for generating a digital human according to the collected data; a deep learning algorithm is adopted to analyze the model, initial digital human behavior characteristics, voice characteristics and appearance characteristics are generated, the initial digital human behavior characteristics, voice characteristics and appearance characteristics are input into a generative adversarial network (GAN), the behavior logic, voice logic and appearance logic of the digital human are optimized through the iterative training and optimization process of a generator and a discriminator, and the digital human behavior logic, voice logic and appearance logic are obtained. And finally, the interactive digital human with high authenticity is generated, the high authenticity of the digital human in behavior, appearance and sound is ensured, a personalized digital human image conforming to the expectation of the user is created, the user experience is further improved, the digital human image with high authenticity is generated, and the naturalness and interactivity of communication are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence digital human technology, and more specifically to an artificial intelligence-based interactive digital human generation method and system. Background Art

[0002] In recent years, with the continuous development of artificial intelligence technology, the metaverse has begun to attract widespread attention from scholars. Digital humans are the most important participants and creators in the metaverse, and are the most valuable and influential digital assets. At the same time, digital humans also have important applications in the fields of film, television and animation. With the development of metaverse technology, virtual digital humans will become the new "creative" subject and pose severe challenges to the traditional copyright system. In the metaverse, virtual digital humans will create a new digital civilization. With the continuous breakthroughs in various metaverse technologies, from the diversified construction and layout of digital scenes to the creation and enrichment of digital artworks such as NFTs, all will be within the scope of content generated by virtual digital humans.

[0003] Virtual digital people include service-oriented virtual digital people and identity-oriented virtual digital people. Service-oriented virtual digital people presented in virtual 3D images have been used in the fields of network anchors and news reports since 2017. In recent years, identity-oriented virtual digital people have gradually come into people's view. Identity-oriented virtual digital people have an identity correspondence with people in the real world, so they are more likely to be identified. Virtual digital people in the metaverse are also a kind of identity-oriented digital people. They are "digital residents" in the metaverse and "digitalized" people in this universe. They are twins with people in this universe. Virtual digital people in the metaverse have two main characteristics: first, digital memories will be uploaded. Through perception technology and emotional interaction technology, users can upload all kinds of senses, memories and even emotions to their "digital compatriots" synchronously; second, they will generate self-driving force under the support of various artificial intelligence technologies, become "clones" of people in the metaverse, and even achieve "interoperability" with people in the real world, realizing mutual replication between users in the real world and the virtual world.

[0004] However, the existing digital humans are generated through inefficient methods of manual modeling. With the development of computer vision and artificial intelligence technology, more and more technologies for automatically generating digital humans have emerged, but there are still problems with high technical means. The realism of digital humans is high, but there is a high delay in behavior and poor interaction. At the same time, the digital humans created for customer needs cannot form effective logical behaviors, have poor behavioral logic capabilities, low realism, high technical requirements, and poor experience when controlled by non-professionals. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides an interactive digital human generation method and system based on artificial intelligence to solve the problems existing in the above-mentioned background technology.

[0006] 1. The present invention provides the following technical solution: an interactive digital human generation method based on artificial intelligence, comprising the following steps:

[0007] S1. Collecting real-person behavior data, voice data, and image data;

[0008] S2, constructing a model for generating a digital human based on the collected data;

[0009] S3. Use deep learning algorithm to analyze the model, generate the initial digital human behavior characteristics, voice characteristics and appearance characteristics, input the initial digital human behavior characteristics, voice characteristics and appearance characteristics into the generative adversarial network (GAN), through the iterative training of the generator and the discriminator, optimize the process, optimize the digital human's behavior logic, voice logic and appearance logic, and finally generate an interactive digital human with a high degree of authenticity.

[0010] Furthermore, the behavioral data is collected by recording the three-dimensional motion data of the real person through motion capture technology, capturing the voice data of the real person, including characteristics such as pitch, speaking speed and accent, using a high-resolution microphone, synchronously capturing the facial expressions and body features of the real person from multiple angles using a high-resolution camera, and using biometric technology to collect the biometric data of the real person, such as fingerprints, voiceprints and retinal scans, for three-dimensional construction.

[0011] Furthermore, the process of the three-dimensional construction is as follows:

[0012] Construct a variety of lighting environments, collect human image samples and human feature points under various lighting environments from real human images or videos, and construct a two-dimensional or three-dimensional feature point cloud; use image processing and image combination technology to perform a three-dimensional reconstruction of each two-dimensional human image based on SFM's three-dimensional reconstruction technology to obtain various human models; among the various human models obtained, select a certain number of results with high credibility and combine them according to weights in a partitioned manner; calculate the subsequent models to produce a new model with coherence and consistency, so as to obtain the three-dimensional reconstruction result of the individual and generate a digital human with the same or similar appearance and movements; render the fused model, add some details and effects, and make the generated digital human more realistic and natural. The rendering methods include texture mapping, lighting model, shadow effect and reflection effect.

[0013] The process of SFM 3D reconstruction is as follows:

[0014] S21, detect the position of the face, the position and posture of the body, and some key points from the input two-dimensional image, such as eyes, nose, mouth, neck, arms, waist, hips, legs, etc., and then align the human bodies in different images into a unified coordinate system for subsequent feature extraction and matching;

[0015] S22, extracting some distinctive and stable feature points, such as corner points, edges, regions, etc., and corresponding feature descriptors, such as SIFT, SURF, ORB, etc., from the input two-dimensional image for subsequent feature matching, and then performing image and correspondence based on these features;

[0016] S23, according to the similarity of the feature descriptors, find the corresponding feature points in different images, perform feature matching, and use it for subsequent motion estimation;

[0017] S24, determining the intrinsic and extrinsic parameters of the camera according to some known camera parameters or through some calibration methods;

[0018] S25, estimating the relative motion relationship between the cameras, i.e., the rotation matrix and translation vector, based on feature matching;

[0019] S26, according to the camera's internal parameters, external parameters and feature matching, using the triangulation method, recovering the three-dimensional coordinates of the feature points in the scene to form a sparse three-dimensional point cloud. The sparse reconstruction can also use the bundle adjustment method to simultaneously optimize the camera parameters and the three-dimensional point coordinates to improve the accuracy and robustness of the reconstruction;

[0020] S27. According to the camera's intrinsic parameters, extrinsic parameters and sparse 3D point cloud, using stereo vision or multi-view stereo method, recover more 3D points or surfaces in the scene to form a dense 3D point cloud or grid. Dense reconstruction can be divided into pixel-based method and block-based method. The pixel-based method refers to directly performing depth estimation at the pixel level, and the block-based method refers to performing depth estimation on some predefined blocks or windows.

[0021] Furthermore, the model of the digital human adopts convolutional neural network (CNN), recurrent neural network (RNN) and speech recognition technology to extract facial and body features of image data and feature extraction of speech data respectively, applies sequence learning and behavior analysis algorithms to extract features of behavior data, and applies biometric recognition technology to analyze biometric data to enhance the security and personalization of the digital human. The model is analyzed using a deep learning algorithm to generate initial digital human behavior features, speech features and appearance features, which are input into a generative adversarial network (GAN). Through iterative training of the generator and the discriminator, the behavior logic, speech logic and appearance logic of the digital human are optimized. Through the adversarial training process, the parameters of the generator and the discriminator are continuously adjusted until the generated digital human reaches a predetermined level of authenticity.

[0022] The convolutional neural network (CNN) extracts facial and body features of the image data and is configured to automatically identify and extract facial and body features in the image data and compare the extracted feature points in the three-dimensional construction.

[0023] Furthermore, the recurrent neural network (RNN) and speech recognition technology are configured to automatically recognize and extract features such as tone, speaking speed and accent in speech data, the sequence learning and behavior analysis algorithm is configured to automatically recognize and extract features in behavior data, and the biometric recognition technology is configured to analyze biometric data and combine these data with the security and personalized characteristics of the digital person.

[0024] A recurrent neural network (RNN) is a neural network with the ability to process time series. It can process related data before and after. Its core feature is that it can remember previous information and use this information to influence subsequent output. This feature makes it very suitable for processing speech data.

[0025] The identification steps are:

[0026] S31, converting the voice signal into an electrical signal through a microphone or other device, and then converting it into a digital signal through analog-to-digital conversion (ADC);

[0027] S32, various noises in the speech are eliminated through signal processing technology, such as spectral subtraction, Wiener filtering, etc., to reduce the interference of environmental noise on the speech signal, thereby improving the accuracy of speech recognition;

[0028] S33, determine the starting and ending positions of the voice signal, so as to distinguish valid voice segments and silent segments, reduce invalid data, and improve recognition efficiency;

[0029] S34, using Mel frequency cepstral coefficient (MFCC) to extract speech features, Mel frequency cepstral coefficient (MFCC) extracts features by simulating the nonlinear perception characteristics of human ear to frequency, and the specific steps include pre-emphasis, framing, windowing, fast Fourier transform (FFT), Mel filter bank, logarithmic operation and discrete cosine transform (DCT), and performing speech data annotation;

[0030] S35. Construct an RNN model including an input layer, a hidden layer (LSTM or GRU) and an output layer, wherein the input layer receives a feature vector sequence, the hidden layer captures the temporal dependency through a recurrent connection, and the output layer generates a recognition result;

[0031] S36, using the labeled speech data to train the RNN model, and optimizing the model parameters through the back propagation algorithm (Backpropagation Through Time, BPTT);

[0032] S37. Further semantic understanding of the recognized text is performed, and the text is converted back into speech signals through speech synthesis technology to achieve natural human-computer interaction.

[0033] Furthermore, the adversarial training process is an iterative process in which the generator and the discriminator compete with each other to improve the authenticity of the generated digital human. The steps of the adversarial training process are:

[0034] Label the collected data and perform preliminary data preprocessing on the data;

[0035] The processing steps of the data preprocessing are:

[0036] S41, cleaning the labeled data to delete damaged, incomplete or irrelevant data, such as blurred, incomplete images and duplicate data;

[0037] S42. Convert all data to a unified format, such as unifying the date format to "YYYY-MM-DD" and the numeric type to floating point or integer. For missing data, you can choose to fill (using the mean, median or mode), delete or replace with a special marker; for example, in tabular data, replace null values ​​with "NaN" or a specific identifier;

[0038] S43, normalize the data and scale the data to a specific range, such as [0,1],

[0039] S44, flipping the normalized data horizontally or vertically, and randomly cropping them to increase data diversity and reduce over-simulation, and then randomly rotating the image by a certain angle, adjusting brightness, contrast, and saturation to simulate different environmental conditions;

[0040] S45. Reduce the dimension of the processed data through techniques such as principal component analysis (PCA) or linear discriminant analysis (LDA), while retaining important information, and extract the most relevant feature subsets to remove noise or redundant features to improve the performance of the model;

[0041] S46. Divide the data set into training set, validation set and test set, with a commonly used ratio of 70% / 15% / 15%. Ensure that the data after division maintains its original distribution characteristics, and then use k-fold cross-validation to evaluate the generalization ability of the model to avoid overfitting or underfitting problems caused by improper data division;

[0042] S47. Add tiny, carefully calculated perturbations to the input data to generate adversarial samples that can mislead the model. These adversarial samples are used to train the model to improve its robustness against malicious attacks. Multiple iterations of the projected gradient descent (PGD) method are used to generate adversarial samples to further enhance the model's defense capabilities.

[0043] Furthermore, the iterative training steps of the generator and the discriminator are:

[0044] S51. After the data is preprocessed for adversarial training, the parameters of the generator need to be randomly initialized. Its input is random noise, and the output is the generated sample. The parameters of the discriminator also need to be randomly initialized. It can be a neural network of any structure. The input is a real sample or a sample generated by the generator. The output is a probability value, indicating the probability that the input sample is a real sample.

[0045] S52. Randomly select a batch of data from the real data set and input it into the discriminator for training; at the same time, generate a batch of fake data (also called generated data) from the current state of the generator, and input this data into the discriminator for training; when training the discriminator, it is necessary to fix the parameters of the generator (using the .detach() function) and only update the parameters of the discriminator; the goal of the discriminator is to distinguish real data from fake data, so its loss function is usually defined as a binary cross entropy loss function, and the back propagation algorithm is used to update the parameters of the discriminator to minimize the loss function;

[0046] The binary cross entropy loss function formula is: Where: L represents the loss value, which is a scalar that measures the difference between the model prediction result and the true label; N represents the number of samples, that is, the total number of input data; y i represents the true label of the i-th sample, which takes a value of 0 or 1; p irepresents the probability that the model predicts that the i-th sample is a positive class (labeled as 1); log represents the natural logarithm, which is used to calculate the contribution of each sample to the total loss; when the true label is 1, it is hoped that the probability predicted by the model is also close to 1, at this time y i *log(p i ) will be very small; and when the true label is 0, we hope that the probability predicted by the model is close to 0, then (1-y i )*log(1-p i ) will also be small;

[0047] S53. The loss function of the generator is usually defined as the inverse of the cross entropy loss function of the discriminator's output result for fake data. In other words, the generator hopes that the discriminator's judgment result for fake data is as close to the real data as possible. The back propagation algorithm is also used to update the parameters of the generator to minimize its loss function.

[0048] S54, perform alternating training. In each round of training, first train the discriminator and then train the generator. This ensures that both networks can be sufficiently optimized. Repeat this process until the preset number of iterations is reached or a certain convergence condition is met (such as the gap between the fake data generated by the generator and the real data reaches a certain level).

[0049] Furthermore, the optimization process includes:

[0050] S61. After each GAN training, collect data on the interaction between the user and the digital human.

[0051] S62. Analyze user interaction data and extract key performance indicators required to improve digital humans;

[0052] S63. Adjust the parameters of the generative adversarial network to improve the behavior and interaction quality of the digital human in a targeted manner;

[0053] S64. Repeat the training and user feedback cycle until the digital human's performance cannot be significantly improved through further training.

[0054] An interactive digital human generation system based on artificial intelligence, which is used to implement the interactive digital human generation method based on artificial intelligence, comprises: a data acquisition module, which is used to collect behavior data, voice data, image data and biometric data of real people;

[0055] Data preprocessing and feature extraction module, used for data preprocessing and feature extraction through machine learning algorithms;

[0056] A model building module, used to build a model for generating digital humans;

[0057] Generative Adversarial Network (GAN) module, which is used to optimize the behavior logic, speech logic, and appearance logic of digital humans through iterative training of generators and discriminators;

[0058] An optimization module is used to adjust the parameters of the generative adversarial network and perform iterative optimization based on user feedback.

[0059] Technical effects and advantages of the present invention:

[0060] High-precision motion capture equipment, high-resolution cameras and other equipment are used to collect real human behavior data, voice data, and image data to ensure that the digital human is highly authentic in behavior, appearance, and sound. Deep learning algorithms are applied to conduct in-depth analysis of the collected data to generate a highly authentic digital human image, while ensuring that the digital human can interact with the user in a natural and fluent voice and text manner. With the help of generative adversarial networks (GANs) and other deep learning architectures, the response speed and interaction logic of the digital human are optimized so that it can give appropriate and timely feedback in real-time interaction, enhancing the naturalness and interactivity of communication. Machine learning algorithms are used to analyze user data, including user interaction history, preference settings, etc., to create a personalized digital human image that meets user expectations and further enhance user experience. SfM can recover the sparse three-dimensional point cloud of the target object from multiple images with high accuracy, greatly simplifying the operation process and reducing hardware requirements. SfM uses feature point detection algorithms such as SIFT, whose feature points are scale-invariant. Even if the image scale changes, the same feature points can be reliably extracted. The digital human is highly flexible and has good interactive effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of an interactive digital human generation method based on artificial intelligence according to the present invention.

[0062] Figure 2 The figure is a schematic diagram of an interactive digital human generation system based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. In addition, the forms of the various structures recorded in the following embodiments are merely illustrative. The artificial intelligence-based interactive digital human generation method and system involved in the present invention are not limited to the various structures recorded in the following embodiments. All other implementations obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0064] Example 1

[0065] The present invention provides an interactive digital human generation method based on artificial intelligence, comprising the following steps:

[0066] S1. Collecting real-person behavior data, voice data, and image data;

[0067] S2, constructing a model for generating a digital human based on the collected data;

[0068] S3. Use deep learning algorithm to analyze the model, generate initial digital human behavior features, voice features and appearance features, input the initial digital human behavior features, voice features and appearance features into the generative adversarial network (GAN), optimize the digital human behavior logic, voice logic and appearance logic through iterative training of the generator and discriminator, and finally generate an interactive digital human with high authenticity;

[0069] Furthermore, the collection of behavioral data uses motion capture technology to record the three-dimensional motion data of real people, uses high-resolution microphones to capture the voice data of real people, including characteristics such as pitch, speaking speed and accent, uses high-resolution cameras to simultaneously capture the facial expressions and body features of real people from multiple angles, and uses biometric technology to collect the biometric data of real people, such as fingerprints, voiceprints and retinal scans, for three-dimensional construction;

[0070] Motion capture technology includes the use of optical, inertial or hybrid motion capture systems to record three-dimensional motion data of a person, high-resolution microphones with a sampling rate of at least 44.1kHz to ensure high-quality capture of voice data, high-resolution cameras with at least 4k resolution and capable of capturing video streams of at least 60fps to ensure high-quality synchronous capture of facial expressions and body features, and biometric technology includes sensors and related software algorithms for collecting fingerprints, voiceprints and retinal scans to improve the security and personalization of digital people.

[0071] The process of 3D construction is as follows:

[0072] Construct a variety of lighting environments, collect human image samples and human feature points under various lighting environments from real human images or videos, and construct a two-dimensional or three-dimensional feature point cloud; use image processing and image combination technology to perform a three-dimensional reconstruction of each two-dimensional human image based on SFM's three-dimensional reconstruction technology to obtain various human models; among the various human models obtained, select a certain number of results with high credibility and combine them according to weights in a partitioned manner; calculate the subsequent models to produce a new model with coherence and consistency, so as to obtain the three-dimensional reconstruction result of the individual and generate a digital human with the same or similar appearance and movements; render the fused model, add some details and effects, and make the generated digital human more realistic and natural. The rendering methods include texture mapping, lighting model, shadow effect and reflection effect.

[0073] The process of SFM 3D reconstruction is as follows:

[0074] S21, detect the position of the face, the position and posture of the body, and some key points from the input two-dimensional image, such as eyes, nose, mouth, neck, arms, waist, hips, legs, etc., and then align the human bodies in different images into a unified coordinate system for subsequent feature extraction and matching;

[0075] S22, extracting some distinctive and stable feature points, such as corner points, edges, regions, etc., and corresponding feature descriptors, such as SIFT, SURF, ORB, etc., from the input two-dimensional image for subsequent feature matching, and then performing image and correspondence based on these features;

[0076] S23, according to the similarity of the feature descriptors, find the corresponding feature points in different images, perform feature matching, and use it for subsequent motion estimation;

[0077] S24, determining the intrinsic and extrinsic parameters of the camera according to some known camera parameters or through some calibration methods;

[0078] S25, estimating the relative motion relationship between the cameras, i.e., the rotation matrix and translation vector, based on feature matching;

[0079] S26, according to the camera's internal parameters, external parameters and feature matching, using the triangulation method, recovering the three-dimensional coordinates of the feature points in the scene to form a sparse three-dimensional point cloud. The sparse reconstruction can also use the bundle adjustment method to simultaneously optimize the camera parameters and the three-dimensional point coordinates to improve the accuracy and robustness of the reconstruction;

[0080] S27. Based on the camera's intrinsic parameters, extrinsic parameters and sparse 3D point cloud, stereo vision or multi-view stereo methods are used to restore more 3D points or surfaces in the scene to form a dense 3D point cloud or grid. Dense reconstruction can be divided into pixel-based methods and block-based methods. The pixel-based method refers to directly estimating the depth at the pixel level, and the block-based method refers to performing depth estimation on some predefined blocks or windows.

[0081] Then, convolutional neural network (CNN), recurrent neural network (RNN) and speech recognition technology are used to extract facial and body features of image data and feature extraction of speech data respectively, sequence learning and behavior analysis algorithms are used to extract features of behavior data, and biometric recognition technology is used to analyze biometric data to enhance the security and personalization of digital humans, and deep learning algorithms are used to analyze the model to generate initial digital human behavior features, speech features and appearance features, which are input into generative adversarial networks (GANs), and through iterative training of generators and discriminators, the behavior logic, speech logic and appearance logic of the digital human are optimized, and through the adversarial training process, the parameters of the generator and the discriminator are continuously adjusted until the generated digital human reaches a predetermined level of authenticity.

[0082] The convolutional neural network (CNN) extracts facial and body features from image data and is configured to automatically identify and extract facial and body features from image data and compare them with feature points extracted from three-dimensional construction. The double extraction and comparison makes the digital human model construction effect better and more realistic.

[0083] The dual extraction of 3D reconstruction and convolutional neural network (CNN) is used to better extract human features and reconstruct a better model.

[0084] Furthermore, the recurrent neural network (RNN) and speech recognition technology are configured to automatically recognize and extract features such as tone, speaking speed and accent in speech data, the sequence learning and behavior analysis algorithm is configured to automatically recognize and extract features in behavior data, and the biometric recognition technology is configured to analyze biometric data and combine these data with the security and personalized characteristics of the digital person.

[0085] A recurrent neural network (RNN) is a neural network with the ability to process time series. It can process related data before and after. Its core feature is that it can remember previous information and use this information to influence subsequent output. This feature makes it very suitable for processing speech data.

[0086] The identification steps are:

[0087] S31, converting the voice signal into an electrical signal through a microphone or other device, and then converting it into a digital signal through analog-to-digital conversion (ADC);

[0088] S32, various noises in the speech are eliminated through signal processing technology, such as spectral subtraction, Wiener filtering, etc., to reduce the interference of environmental noise on the speech signal, thereby improving the accuracy of speech recognition;

[0089] S33, determine the starting and ending positions of the voice signal, so as to distinguish valid voice segments and silent segments, reduce invalid data, and improve recognition efficiency;

[0090] S34, using Mel frequency cepstral coefficient (MFCC) to extract speech features, Mel frequency cepstral coefficient (MFCC) extracts features by simulating the nonlinear perception characteristics of human ear to frequency, and the specific steps include pre-emphasis, framing, windowing, fast Fourier transform (FFT), Mel filter bank, logarithmic operation and discrete cosine transform (DCT), and performing speech data annotation;

[0091] S35. Construct an RNN model including an input layer, a hidden layer (LSTM or GRU) and an output layer, wherein the input layer receives a feature vector sequence, the hidden layer captures the temporal dependency through a recurrent connection, and the output layer generates a recognition result;

[0092] S36, using the labeled speech data to train the RNN model, and optimizing the model parameters through the back propagation algorithm (Backpropagation Through Time, BPTT);

[0093] S37, further semantically understand the recognized text, and convert the text back into speech signals through speech synthesis technology to achieve natural human-computer interaction;

[0094] Furthermore, the adversarial training process is an iterative process in which the generator and the discriminator compete with each other to improve the authenticity of the generated digital human. The steps of the adversarial training process are:

[0095] Label the collected data and perform preliminary data preprocessing on the data;

[0096] The processing steps of the data preprocessing are:

[0097] S41, cleaning the labeled data to delete damaged, incomplete or irrelevant data, such as blurred, incomplete images and duplicate data;

[0098] S42. Convert all data to a unified format, such as unifying the date format to "YYYY-MM-DD" and the numeric type to floating point or integer. For missing data, you can choose to fill (using the mean, median or mode), delete or replace with a special marker; for example, in tabular data, replace null values ​​with "NaN" or a specific identifier;

[0099] S43, normalize the data and scale the data to a specific range, such as [0,1],

[0100] S44, flipping the normalized data horizontally or vertically, and randomly cropping them to increase data diversity and reduce over-simulation, and then randomly rotating the image by a certain angle, adjusting brightness, contrast, and saturation to simulate different environmental conditions;

[0101] S45. Reduce the dimension of the processed data through techniques such as principal component analysis (PCA) or linear discriminant analysis (LDA), while retaining important information, and extract the most relevant feature subsets to remove noise or redundant features to improve the performance of the model;

[0102] S46. Divide the data set into training set, validation set and test set, with a commonly used ratio of 70% / 15% / 15%. Ensure that the data after division maintains its original distribution characteristics, and then use k-fold cross-validation to evaluate the generalization ability of the model to avoid overfitting or underfitting problems caused by improper data division;

[0103] S47. Add tiny, carefully calculated perturbations to the input data to generate adversarial samples that can mislead the model. These adversarial samples are used to train the model to improve its robustness against malicious attacks. Multiple iterations of the projected gradient descent (PGD) method are used to generate adversarial samples to further enhance the model's defense capabilities.

[0104] Furthermore, the iterative training steps of the generator and the discriminator are:

[0105] S51. After the data is preprocessed for adversarial training, the parameters of the generator need to be randomly initialized. Its input is random noise, and the output is the generated sample. The parameters of the discriminator also need to be randomly initialized. It can be a neural network of any structure. The input is a real sample or a sample generated by the generator. The output is a probability value, indicating the probability that the input sample is a real sample.

[0106] S52. Randomly select a batch of data from the real data set and input it into the discriminator for training; at the same time, generate a batch of fake data (also called generated data) from the current state of the generator, and input this data into the discriminator for training; when training the discriminator, it is necessary to fix the parameters of the generator (using the .detach() function) and only update the parameters of the discriminator; the goal of the discriminator is to distinguish real data from fake data, so its loss function is usually defined as a binary cross entropy loss function, and the back propagation algorithm is used to update the parameters of the discriminator to minimize the loss function;

[0107] The binary cross entropy loss function formula is: Where: L represents the loss value, which is a scalar that measures the difference between the model prediction result and the true label; N represents the number of samples, that is, the total number of input data; y i represents the true label of the i-th sample, which takes a value of 0 or 1; p i represents the probability that the model predicts that the i-th sample is a positive class (labeled as 1); log represents the natural logarithm, which is used to calculate the contribution of each sample to the total loss; when the true label is 1, it is hoped that the probability predicted by the model is also close to 1, at this time y i *log(p i ) will be very small; and when the true label is 0, we hope that the probability predicted by the model is close to 0, then (1-y i )*log(1-p i ) will also be small;

[0108] S53. The loss function of the generator is usually defined as the inverse of the cross entropy loss function of the discriminator's output result for fake data. In other words, the generator hopes that the discriminator's judgment result for fake data is as close to the real data as possible. The back propagation algorithm is also used to update the parameters of the generator to minimize its loss function.

[0109] S54, perform alternating training. In each round of training, first train the discriminator and then train the generator. This ensures that both networks can be sufficiently optimized. Repeat this process until the preset number of iterations is reached or a certain convergence condition is met (such as the gap between the fake data generated by the generator and the real data reaches a certain level).

[0110] Furthermore, the optimization process includes:

[0111] S61. After each GAN training, collect data on the interaction between the user and the digital human.

[0112] S62. Analyze user interaction data and extract key performance indicators required to improve digital humans;

[0113] S63. Adjust the parameters of the generative adversarial network to improve the behavior and interaction quality of the digital human in a targeted manner;

[0114] S64. Repeat the training and user feedback cycle until the digital human's performance cannot be significantly improved through further training.

[0115] Example 2

[0116] For the parts not described in detail in this embodiment, please refer to the description of Embodiment 1. An interactive digital human generation system based on artificial intelligence is provided, including: a data acquisition module for collecting behavior data, voice data, image data and biometric data of real people;

[0117] Data preprocessing and feature extraction module, used for data preprocessing and feature extraction through machine learning algorithms;

[0118] A model building module, used to build a model for generating digital humans;

[0119] Generative Adversarial Network (GAN) module, which is used to optimize the behavior logic, speech logic, and appearance logic of digital humans through iterative training of generators and discriminators;

[0120] An optimization module is used to adjust the parameters of the generative adversarial network and perform iterative optimization based on user feedback.

[0121] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An interactive digital human generation method based on artificial intelligence, characterized in that: The following steps are involved: S1. Collecting real-person behavior data, voice data, and image data; S2, constructing a model for generating a digital human based on the collected data; S3. Use deep learning algorithm to analyze the model, generate the initial digital human behavior characteristics, voice characteristics and appearance characteristics, input the initial digital human behavior characteristics, voice characteristics and appearance characteristics into the generative adversarial network (GAN), through the iterative training of the generator and the discriminator, optimize the process, optimize the digital human's behavior logic, voice logic and appearance logic, and finally generate an interactive digital human with a high degree of authenticity.

2. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: The behavioral data is collected by recording the three-dimensional motion data of the real person through motion capture technology, capturing the voice data of the real person including features such as pitch, speaking speed and accent using a high-resolution microphone, capturing the facial expressions and body features of the real person from multiple angles synchronously using a high-resolution camera, and collecting the biometric data of the real person such as fingerprints, voiceprints and retinal scans using biometric recognition technology for three-dimensional construction.

3. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: The process of the three-dimensional construction is as follows: Construct a variety of lighting environments, collect human image samples and human feature points under various lighting environments from real human images or videos, and construct a two-dimensional or three-dimensional feature point cloud; use image processing and image combination technology to perform a three-dimensional reconstruction of each two-dimensional human image based on SFM's three-dimensional reconstruction technology to obtain various human models; among the various human models obtained, select a certain number of results with high credibility and combine them according to weights in a partitioned manner; calculate the subsequent models to produce a new model with coherence and consistency, so as to obtain the three-dimensional reconstruction result of the individual and generate a digital human with the same or similar appearance and movements; render the fused model, add some details and effects, and make the generated digital human more realistic and natural. The rendering methods include texture mapping, lighting model, shadow effect and reflection effect. The process of SFM 3D reconstruction is as follows: S21. Detect the position of the face, the position and posture of the body, and some key points from the input two-dimensional image. Such as eyes, nose, mouth, neck, arms, waist, hips, legs, etc., and then align the human body in different images into a unified coordinate system for subsequent feature extraction and matching; S22, extracting some distinctive and stable feature points, such as corner points, edges, regions, etc., and corresponding feature descriptors, such as SIFT, SURF, ORB, etc., from the input two-dimensional image for subsequent feature matching, and then performing image and correspondence based on these features; S23, according to the similarity of the feature descriptors, find the corresponding feature points in different images, perform feature matching, and use it for subsequent motion estimation; S24, determining the intrinsic and extrinsic parameters of the camera according to some known camera parameters or through some calibration methods; S25, estimating the relative motion relationship between the cameras, i.e., the rotation matrix and translation vector, based on feature matching; S26, according to the camera's internal parameters, external parameters and feature matching, using the triangulation method, recovering the three-dimensional coordinates of the feature points in the scene to form a sparse three-dimensional point cloud. The sparse reconstruction can also use the bundle adjustment method to simultaneously optimize the camera parameters and the three-dimensional point coordinates to improve the accuracy and robustness of the reconstruction; S27. Based on the camera's intrinsic parameters, extrinsic parameters and sparse 3D point cloud, stereo vision or multi-view stereo methods are used to restore more 3D points or surfaces in the scene to form a dense 3D point cloud or grid. Dense reconstruction can be divided into pixel-based methods and block-based methods. The pixel-based method refers to directly estimating the depth at the pixel level, and the block-based method refers to performing depth estimation on some predefined blocks or windows.

4. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: The model of the digital human adopts convolutional neural network (CNN), recurrent neural network (RNN) and speech recognition technology to extract facial and body features of image data and feature extraction of speech data respectively, applies sequence learning and behavior analysis algorithms to extract features of behavior data, and applies biometric recognition technology to analyze biometric data to enhance the security and personalization of the digital human. The model is analyzed by deep learning algorithm to generate initial digital human behavior features, speech features and appearance features, which are input into generative adversarial network (GAN). Through iterative training of generator and discriminator, the behavior logic, speech logic and appearance logic of the digital human are optimized. Through the adversarial training process, the parameters of the generator and discriminator are continuously adjusted until the generated digital human reaches a predetermined level of authenticity. The convolutional neural network (CNN) extracts facial and body features of the image data and is configured to automatically identify and extract facial and body features in the image data and compare the extracted feature points in the three-dimensional construction.

5. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: The recurrent neural network (RNN) and speech recognition technology are configured to automatically recognize and extract features such as tone, speaking speed and accent in speech data, the sequence learning and behavior analysis algorithm is configured to automatically recognize and extract features in behavior data, and the biometric recognition technology is configured to analyze biometric data and combine this data with the security and personalized features of the digital person. A recurrent neural network (RNN) is a neural network with time series processing capabilities. It can process related data before and after. Its core feature is that it can remember previous information and use this information to influence subsequent outputs. This feature makes it very suitable for processing speech data. The identification steps are: S31, converting the voice signal into an electrical signal through a microphone or other device, and then converting it into a digital signal through analog-to-digital conversion (ADC); S32, eliminating various noises in the speech through signal processing techniques, such as spectral subtraction, Wiener filtering, etc., to reduce the interference of environmental noise on the speech signal, thereby improving the accuracy of speech recognition; S33, determine the starting and ending positions of the voice signal, so as to distinguish valid voice segments and silent segments, reduce invalid data, and improve recognition efficiency; S34, using Mel frequency cepstral coefficient (MFCC) to extract speech features, Mel frequency cepstral coefficient (MFCC) extracts features by simulating the nonlinear perception characteristics of human ear to frequency, and the specific steps include pre-emphasis, framing, windowing, fast Fourier transform (FFT), Mel filter bank, logarithmic operation and discrete cosine transform (DCT), and performing speech data annotation; S35. Construct an RNN model including an input layer, a hidden layer (LSTM or GRU) and an output layer, wherein the input layer receives a feature vector sequence, the hidden layer captures the temporal dependency through a recurrent connection, and the output layer generates a recognition result; S36, using the labeled speech data to train the RNN model, and optimizing the model parameters through the back propagation algorithm (Backpropagation Through Time, BPTT); S37. Further semantic understanding of the recognized text is performed, and the text is converted back into speech signals through speech synthesis technology to achieve natural human-computer interaction.

6. The method for generating an interactive digital human based on artificial intelligence according to claim 4, characterized in that: The adversarial training process is an iterative process in which the generator and the discriminator compete with each other to improve the authenticity of the generated digital human. The steps of the adversarial training process are: Label the collected data and perform preliminary data preprocessing; The processing steps of the data preprocessing are: S41, cleaning the labeled data to delete damaged, incomplete or irrelevant data, such as blurred, incomplete images and duplicate data; S42. Convert all data to a unified format, such as unifying the date format to "YYYY-MM-DD" and the numeric type to floating point or integer. For missing data, you can choose to fill (using the mean, median or mode), delete or replace with a special marker; for example, in tabular data, replace null values ​​with "NaN" or a specific identifier; S43, normalize the data and scale the data to a specific range, such as [0,1], S44, flipping the normalized data horizontally or vertically, and randomly cropping them to increase data diversity and reduce over-simulation, and then randomly rotating the image by a certain angle, adjusting brightness, contrast, and saturation to simulate different environmental conditions; S45. Reduce the dimension of the processed data through techniques such as principal component analysis (PCA) or linear discriminant analysis (LDA), while retaining important information, and extract the most relevant feature subsets to remove noise or redundant features to improve the performance of the model; S46. Divide the data set into training set, validation set and test set, with a commonly used ratio of 70% / 15% / 15%. Ensure that the data after division maintains its original distribution characteristics, and then use k-fold cross-validation to evaluate the generalization ability of the model to avoid overfitting or underfitting problems caused by improper data division; S47. Add tiny, carefully calculated perturbations to the input data to generate adversarial samples that can mislead the model. These adversarial samples are used to train the model to improve its robustness against malicious attacks. Multiple iterations of the projected gradient descent (PGD) method are used to generate adversarial samples to further enhance the model's defense capabilities.

7. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: The iterative training steps of the generator and discriminator are: S51. After the data is preprocessed for adversarial training, the parameters of the generator need to be randomly initialized. Its input is random noise, and the output is the generated sample. The parameters of the discriminator also need to be randomly initialized. It can be a neural network of any structure. The input is a real sample or a sample generated by the generator. The output is a probability value, indicating the probability that the input sample is a real sample. S52. Randomly select a batch of data from the real data set and input it into the discriminator for training; at the same time, generate a batch of fake data (also called generated data) from the current state of the generator, and input this data into the discriminator for training; when training the discriminator, it is necessary to fix the parameters of the generator (using the .detach() function) and only update the parameters of the discriminator; the goal of the discriminator is to distinguish real data from fake data, so its loss function is usually defined as a binary cross entropy loss function, and the back propagation algorithm is used to update the parameters of the discriminator to minimize the loss function; The binary cross entropy loss function formula is: Where: L represents the loss value, which is a scalar that measures the difference between the model prediction result and the true label; N represents the number of samples, that is, the total number of input data; y i represents the true label of the i-th sample, which takes a value of 0 or 1; p i represents the probability that the model predicts that the i-th sample is a positive class (labeled as 1); log represents the natural logarithm, which is used to calculate the contribution of each sample to the total loss; when the true label is 1, it is hoped that the probability predicted by the model is also close to 1, at this time y i *log(p i ) will be very small; and when the true label is 0, we hope that the probability predicted by the model is close to 0, then (1-y i )*log(1-p i ) will also be small; S53. The loss function of the generator is usually defined as the inverse of the cross entropy loss function of the discriminator's output result for fake data. In other words, the generator hopes that the discriminator's judgment result for fake data is as close to the real data as possible. The back propagation algorithm is also used to update the parameters of the generator to minimize its loss function. S54, perform alternating training. In each round of training, first train the discriminator and then train the generator. This ensures that both networks can be sufficiently optimized. Repeat this process until the preset number of iterations is reached or a certain convergence condition is met (such as the gap between the fake data generated by the generator and the real data reaches a certain level).

8. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: The optimization process includes: S61. After each GAN training, collect data on the interaction between the user and the digital human. S62. Analyze user interaction data and extract key performance indicators required to improve digital humans; S63. Adjust the parameters of the generative adversarial network to improve the behavior and interaction quality of the digital human in a targeted manner; S64. Repeat the training and user feedback cycle until the digital human's performance cannot be significantly improved through further training.

9. An interactive digital human generation system based on artificial intelligence, which is used to implement an interactive digital human generation method based on artificial intelligence as claimed in any one of claims 1 to 8, characterized in that: include: A data collection module is used to collect behavior data, voice data, image data and biometric data of real people; Data preprocessing and feature extraction module, used for data preprocessing and feature extraction through machine learning algorithms; A model building module, used to build a model for generating digital humans; Generative Adversarial Network (GAN) module, which is used to optimize the behavior logic, speech logic, and appearance logic of digital humans through iterative training of generators and discriminators; An optimization module is used to adjust the parameters of the generative adversarial network and perform iterative optimization based on user feedback.

Images