Character-driven AIGC video generation method and device

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a character-driven AIGC video generation method, and the method comprises the steps: obtaining a driving character and a character image; generating a target voice feature according to the driving character; obtaining image features and facial features of the figure image according to the figure image; fusing the target voice features, the image features and the face features to obtain multiple frames of video images; according to the multiple frames of video images, a speaking video corresponding to the figure image is generated, the speaking video is an AIGC video, and the speaking content of the speaking video is the content of the driving characters. According to the method, the driving characters serve as input, so that the generation speed of the digital human speaking video is high, semantic information can be mined through the characters, the generation effect of the digital human speaking video is excellent, the digital human is vivid, the expression of the digital human is natural, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text-driven AIGC video generation method and device. Background Art

[0002] In recent years, with the development of deep learning technology, the technology of generating speaking videos by driving the face of a three-dimensional digital human through voice has developed rapidly and has been applied in many fields. The speaking video is an AIGC (Artificial Intelligence Generated Content) video. Existing digital human speaking videos usually use driving voice audio as input, which increases the cost of digital human speaking videos. In addition, the digital human speaking videos that use driving voice as input need to recognize the voice audio. In the process of recognizing voice audio, due to factors such as background noise, accent differences or fast speaking, recognition errors are generated in the voice audio. This leads to the problem that the generation speed of digital human speaking videos is slow and the generation effect is poor. Summary of the invention

[0003] The embodiment of the present application solves the technical problems of slow generation speed and poor generation effect of digital human speaking videos in the prior art by providing a text-driven AIGC video generation method and device. It uses driving text as input to increase the generation speed of digital human speaking videos, and can also mine semantic information through text, so that the digital human speaking video generation effect is excellent, the digital character is realistic, the digital human expression is natural, and the user experience is improved.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating an AIGC video driven by text, comprising: obtaining driving text and a character image; Generating target speech features according to the driving text; According to the character image, obtaining image features and facial features of the character image; fusing the target speech feature, the image feature and the facial feature to obtain multiple frames of video images; A speaking video corresponding to the character image is generated according to the multiple frames of the video image, wherein the speaking video is an AIGC video, and the speaking content of the speaking video is the content of the driving text.

[0005] Preferably, generating target speech features according to the driving text includes: According to the driving text, obtaining the voice and text features corresponding to the driving text; According to the speech, a speech feature is obtained, wherein the speech feature is a feature vector including a semantic feature; The target speech feature is obtained according to the speech feature and the text feature.

[0006] Preferably, obtaining the target speech feature according to the speech feature and the text feature includes: Extracting audio features from the speech features through an LSTM network, and extracting text features from the text features through a text extractor; Through a variance adapter, the audio feature and the text feature are residually connected to obtain a latent attribute feature; The latent attribute feature and the speech feature are concatenated through an encoder to obtain the target speech feature.

[0007] Preferably, obtaining the text feature according to the driving text includes: The driving text is encoded by a clip encoder to obtain the text features.

[0008] Preferably, obtaining speech features according to the speech includes: Mapping the speech to a latent space through a convolutional network to obtain latent features of the speech in the latent space; The latent features are encoded through a Transformer network to obtain the speech features.

[0009] Preferably, the step of fusing the target voice feature, the image feature and the facial feature to obtain multiple frames of video images includes: The target speech features, the image features and the facial features are fused through a diffusion model until the posture constraint condition of the motion estimation matrix is ​​reached, thereby obtaining a frame of the video image, and then obtaining multiple frames of the video image.

[0010] Preferably, the motion estimation matrix is: M = Mt,t,E[||E - Et(Gt,t,C)|| 2 ]; Among them, M is the motion estimation matrix, t is the time step, C is the speech feature, E is the multi-layer perceptron, Gt is Gaussian noise, Mt is the motion space matrix at t time step, and Et is the multi-layer perceptron linear operation.

[0011] Preferably, the posture constraint condition of the motion estimation matrix is ​​a target posture of a target feature obtained through the motion estimation matrix, and a condition for adjusting the target feature from a current posture to the target posture, wherein the target feature is a specified feature of the image feature and / or a specified feature of the facial feature.

[0012] Preferably, generating a speaking video corresponding to the character image according to the multiple frames of the video image includes: Repairing the multiple frames of the video image using a face repair model to obtain multiple frames of repaired video images; The multiple frames of the repaired video images are sequentially encoded to obtain the speaking video.

[0013] Based on the same inventive concept, in a second aspect, the present invention also provides an AIGC video generation device driven by text, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the AIGC video generation method driven by text of the first aspect are implemented.

[0014] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages: In the embodiment of the present invention, after obtaining the driving text and the character image, the target voice feature is obtained according to the driving text, and the image feature and the facial feature are obtained according to the character image. Here, the driving text is used as the input of the digital human speaking video, which greatly reduces the generation cost of the speaking video and improves the generation efficiency and generation speed. In addition, the target voice feature obtained by the driving text has high-level semantic information, which facilitates the generation of a realistic speaking video.

[0015] Then, the target voice features, image features and facial features are fused to obtain multiple frames of video images. Here, the target voice features are embedded and fused with the image features and facial features to generate video images frame by frame. Based on the target voice features with voice information, the digital person in the video image is realistic, and the digital person's speaking expression and lip shape are natural, so that the video image can reflect the emotions when speaking, and personalized video images and speaking videos can be formulated. Then, based on the multiple frames of video images, the speaking video corresponding to the character image is generated. In this way, the speaking video generation efficiency is high, the generation effect is excellent, and the user experience is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Also, throughout the accompanying drawings, the same reference figures are used to represent the same components. In the drawings: Figure 1 A schematic diagram showing the steps of a text-driven AIGC video generation method in an embodiment of the present invention is shown; Figure 2 A schematic diagram of a module for obtaining target speech features in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0017] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0018] Embodiment 1 The first embodiment of the present invention provides a method for generating AIGC video driven by text, such as Figure 1 As shown, including: S101, obtaining driving text and character images; S102, generating target speech features according to the driving text; S103, obtaining image features and facial features of the person image according to the person image; S104, fusing the target voice features, image features and facial features to obtain multiple frames of video images; S105, generating a speaking video corresponding to the character image according to the multiple frames of video images, wherein the speaking video is an AIGC video, and the speaking content of the speaking video is the content of the driving text.

[0019] It should be noted that the speaking video generated in this embodiment is an AIGC (Artificial Intelligence Generated Content) video.

[0020] In this embodiment, after obtaining the driving text and the character image, the target voice features are obtained according to the driving text, and the image features and facial features are obtained according to the character image. Here, the driving text is used as the input of the digital human speaking video, which greatly reduces the generation cost of the speaking video and improves the generation efficiency and speed. In addition, the target voice features obtained by the driving text have high-level semantic information, which facilitates the generation of a realistic speaking video.

[0021] Then, the target voice features, image features and facial features are fused to obtain multiple frames of video images. Here, the target voice features are embedded and fused with the image features and facial features to generate video images frame by frame. Based on the target voice features with voice information, the digital person in the video image is realistic, and the digital person's speaking expression and lip shape are natural, so that the video image can reflect the emotions when speaking, and personalized video images and speaking videos can be formulated. Then, based on the multiple frames of video images, the speaking video corresponding to the character image is generated. In this way, the speaking video generation efficiency is high, the generation effect is excellent, and the user experience is enhanced.

[0022] Next, combine Figure 1 The specific implementation steps of the text-driven AIGC video generation method provided in this embodiment are described in detail: First, execute step S101 to obtain the driving text and the character image. Specifically, the driving text is obtained through a device such as a keyboard. The character image is obtained through a camera, a mobile phone with a camera, a tablet computer, and other devices.

[0023] Next, step S102 is executed to generate target speech features according to the driving text.

[0024] Specifically, the process of generating the target speech feature is as follows: (1) based on the driving text, obtain the speech and text features corresponding to the driving text. (2) based on the speech, obtain the speech feature, where the speech feature is a feature vector containing semantic features. (3) based on the speech feature and the text feature, obtain the target speech feature.

[0025] In this embodiment, corresponding voice and text features are generated according to the driving text. The voice generated by the driving text can avoid the voice influence caused by factors such as background noise, accent differences or fast speaking, so that clear and accurate speaking content can be obtained in the process of generating the speaking video. Then, the voice features are obtained according to the voice, so that the voice features have semantic information, which is convenient for the speaking video to have semantic information, so that the generation effect of the speaking video is excellent. And the voice features and text features are combined to obtain the target voice features. In this way, the target voice features can further have more precise and accurate voice information, so that the speaking video can generate the dynamic speaking process of the digital person more efficiently and accurately during the generation process, thereby making the digital person realistic, the digital person's expression and lip shape natural, and improving the user experience.

[0026] The specific process of step (1) is to generate a speech of corresponding content from the driving text through TTS (Text To Speech) technology. Here, accurate speech is obtained through the driving text, so that the speaking video has accurate speech. In addition, the driving text is encoded through the clip (Contrastive Language-Image Pretraining) encoder to obtain text features. The text features are used to give the calibration speech features, so that the speech features have more precise and accurate semantic information, so that the speaking video has rich emotional information, and the digital person in the speaking video has real human emotional expression. The clip encoder includes an image encoder and a text encoder. The text encoder is used to convert the input driving text into a string of one-dimensional vectors based on the Transformer architecture, and then obtain text features. The text encoder of clip has great potential in the field of multimodal learning due to its efficient performance, strong generalization ability and unique cross-modal understanding ability. It can perform a variety of natural language processing and quickly and efficiently obtain accurate text features, laying a solid foundation for speaking videos.

[0027] The specific process of step (2) is to first map the speech to the latent space through a convolutional network to obtain the latent features of the speech in the latent space. Among them, the convolutional network includes convolutional networks such as AlexNet, VGG, and ResNet. Then, the latent features are encoded through the Transformer network to obtain speech features. Specifically, the local features are extracted from the input speech through the convolutional network to generate low-dimensional features Z. Then, the context information is extracted from the feature Z through the Transformer network to generate high-level features C, i.e., speech features. Speech features are feature vectors containing semantic features (i.e., semantic information). In the process of obtaining speech features based on speech, combining the local feature extraction capability of the convolutional network and the global context modeling capability of the Transformer network, the audio signal of the speech can be processed more effectively to obtain accurate speech features, so that the speech video has semantic information, and the generation effect of the speech video is excellent, and the speech video is vivid and flexible.

[0028] It should be noted that latent space refers to the feature representation space of one or more layers of the convolutional network. Each point in this space corresponds to an abstract representation of the input data, which usually contains the key features and structural information of the input data. The latent space of the convolutional network is a multi-dimensional feature representation space, which converts the input data into an abstract representation suitable for the task requirements through layer-by-layer feature extraction and transformation. This representation not only contains the structural information of the input data, but also has certain semantic meaning, which is the basis for the convolutional network to effectively handle complex tasks.

[0029] The specific process of step (3) is as follows: Figure 2 As shown in the figure, the audio features are first extracted from the speech features through the LSTM (Long Short-Term Memory) network, and the text features are extracted from the text features through the text extractor. The audio features and text features are concatenated to obtain joint features. The joint features are then adjusted through the variance adapter to obtain the adjusted joint features. Next, the adjusted joint features are encoded through the encoder to obtain the encoded joint features, and the encoded joint features are residually connected with the speech features to obtain the target speech features.

[0030] Specifically, audio features are extracted from speech features through the LSTM network. The LSTM network is a special recurrent neural network (RNN) that can effectively capture long dependencies in time series data. Through the LSTM network, time series features (i.e., audio features) can be extracted from speech signals. These features contain the prosody, rhythm, and emotional information of the speech, so that the speech features can be further reorganized into context-related features to achieve better audio features. In addition, text features are extracted from text features through a text extractor. Text features include semantic information. Text extractors include: Word2Vec model, BERT model. The audio features and text features are spliced ​​to obtain joint features. Among them, the splicing method is an additive splicing method.

[0031] Then, the joint features are adjusted through the variance adapter to obtain the adjusted joint features. The role of the variance adapter in adjusting the joint features is to adjust the variance value of the potential feature space, so that the features of different joint features are consistent through the variance value, and then in the diffusion model stage, the features of video images of different frames are kept consistent to avoid large changes in the generated video images. In this way, it can avoid the video image screen changing too much and too fast, and thus avoid the formation of abrupt images and videos, affecting the visual experience. Then, the adjusted joint features are encoded through the encoder to obtain the encoded joint features. Here, the encoder is used to adjust the number of channels of the joint features, align the encoded joint features and the speech features to the same dimension, so as to solve the problem of dimensional mismatch in the subsequent residual connection process. The principle of the encoder is to make a coding adjustment of the number of channels through a basic linear network and an activation function. Then, the encoded joint features are residually connected with the speech features to obtain the target speech features. The residual connection is used to retain the main information of the input features (i.e., the speech features and the encoded joint features), while introducing additional feature enhancement. In a deep network, as the number of layers increases, features will decay and lose their original semantic information. Adding the features before and after processing can effectively avoid feature degradation. Therefore, the speech features and the encoded joint features are connected by residuals to enhance the semantic information of the speech features and effectively avoid feature degradation. In addition, the speech features and the encoded joint features are fused to generate richer target speech features.

[0032] In the process of generating target speech features based on driving text, the semantic information of the features is enhanced, and the combination of text and audio makes the video playback effect better and the user experience better. Through the extraction and fusion of multimodal features, the quality and naturalness of the generated video are improved, making the digital human speaking video more realistic and vivid.

[0033] Next, step S103 is performed to obtain the image features and facial features of the character image according to the character image. Specifically, the image features and facial features are extracted from the character image through target detection and recognition models, such as yolov5, rt-detr, centernet and other models. Facial features can play a role in identifying the identity of the character and keeping the identity unchanged, avoiding the situation where the identity of the digital person changes during the generation of the speaking video. Among them, the image features are the features corresponding to all targets in the character image, and the facial features are the relevant features of the character's face. For example, the character image is a photo of the upper body of a girl. The image features of the character image include hair features, facial features, shoulder features, arm features, palm features, etc. The facial features of the character image are the facial features of the girl, including eyebrow features, eye features, nose features, mouth features, cheek features, forehead features, ear features, etc.

[0034] The facial feature detection process is to detect the five key points of the face and the face detection frame through the target detection model, and then crop the face area according to the face detection frame. The five key points of the face include: eye key points (left eye key points and right eye key points), nose key points, and mouth key points (left corner of mouth key points and right corner of mouth key points). The five key points of the face are used for face correction, and then a feature extraction model for face recognition is used to extract a rich facial feature vector, that is, facial features. In this way, even if the face of the person in the person image is in profile, a frontal face can be generated, the person image is optimized, and image features and facial features are efficiently extracted from the person image.

[0035] Then, step S104 is executed to fuse the target speech features, image features and facial features to obtain multiple frames of video images.

[0036] Specifically, the target speech features, image features and facial features are fused through the diffusion model until the posture constraint of the motion estimation matrix is ​​reached to obtain a frame of video image, and then multiple frames of video images are obtained. The diffusion model process has the effect of image denoising. In addition, in the fusion process of the diffusion model, the motion estimation matrix is ​​also required to perform posture constraints.

[0037] The motion estimation matrix is: M = Mt,t,E[||E - Et(Gt,t,C)|| 2 ]; Wherein, M is the motion estimation matrix, t is the time step, C is the speech feature, E is the multilayer perceptron, Gt is Gaussian noise, Mt is the motion space matrix at the t time step, and Et is the multilayer perceptron linear operation. Multilayer Perceptron (MLP) is a basic feedforward neural network. In this embodiment, it is composed of an input layer, a hidden layer (Liner linear layer + activation function Relu), and an output layer. M is the result of the operation of the motion space matrix Mt for each different time step. It should be noted that in the process of the diffusion model, the motion estimation matrix needs to be initialized before the motion estimation matrix can be used for subsequent operations.

[0038] The motion estimation matrix uses the potential semantics of the speech signal (i.e., speech features) to generate a matrix for controlling motion. The role of the motion estimation matrix is ​​to constrain the posture of each frame of the generated video image during the fusion process of the diffusion model. In addition, the motion estimation matrix can also estimate the motion relationship between each frame of the video image, generate a motion estimation trajectory, and then constrain the posture of the generated video image, ensuring a smooth transition between adjacent frames, effectively reducing jitter.

[0039] The working process of the diffusion model is: in the process of generating each frame of video image, the image features, facial features and target voice features are sampled once to obtain the sampled features. Then the sampled features are iterated once to generate a frame of video image. The diffusion model process is to sample the image features, facial features and target voice features multiple times, and denoise the character image multiple times, so that the image gradually fits the semantic expression of the feature description, that is, the semantics of the feature has a similar expression meaning to the semantics of the generated image, and the image gradually approaches the target image described by the feature, and then generates a video image. Among them, the role of sampling is that multiple sampling is better than single sampling, and the denoising process is gradual. In this way, it is ensured that the generated video image contains both text and voice information and retains the identity characteristics of the face. In addition, in the process of generating video images, the ControlNet module of the diffusion model runs the motion estimation matrix, so that the diffusion model fuses the target voice features, image features, facial features and motion estimation matrix to obtain a video image. And the ControlNet module constrains the posture of the generated video image through the motion estimation matrix to ensure a smooth transition and natural motion effect between adjacent frames, effectively reducing the jitter. In this way, the ControlNet module is used to precisely control facial landmarks (eyes, nose, lips), and assist in predicting and generating a motion expression matrix of facial expressions and lip shapes that are synchronized with speech, namely, a motion estimation matrix.

[0040] Each frame of video image executes the working process of the above diffusion model, and obtains video images frame by frame, and then obtains multiple frames of video images. The diffusion model is used to extract and fuse multimodal features, and combined with generative adversarial networks and motion estimation technology, high-quality digital human speaking videos are generated.

[0041] Among them, the posture constraint condition of the motion estimation matrix is ​​the target posture of the target feature obtained by the motion estimation matrix, and the condition for the target feature to be adjusted from the current posture to the target posture, wherein the target feature is a specified feature of the image feature and / or a specified feature of the facial feature. For example, the character image is a photo of the upper body of a girl, and in the first frame of the video image, the left pupil of the girl is located in the middle of the left eye. In the process of generating the second frame of the video image, the image features of the character image are sampled for multiple rounds and the target voice features and the facial features of the character image are embedded in the iterative process until the left pupil of the girl is located in the left position of the left eye according to the motion estimation matrix, the diffusion model iteration ends, and the second frame of the video image is generated.

[0042] In this embodiment, the video image is generated frame by frame by combining the diffusion model with the motion estimation matrix. In this way, each frame of the generated video image can retain the unique features of the input face, such as eyes, nose, mouth, etc., so as to ensure that the generated digital human is highly similar to the input face. The video image combines text features and voice features, so that the generated speaking video can not only accurately match the voice, but also better express emotions and intentions, improve the realism and naturalness of the video, and achieve the relevant effects of multimodal information fusion. The diffusion model generates a new frame of video image according to the current features and posture constraints in each round of sampling, and further optimizes it in the next round of sampling. This iterative process helps to gradually approach the ideal video image and improve the overall quality of the generated video. In the diffusion denoising process, the target image (i.e., video image) can be converged faster, the number of sampling rounds can be reduced, and the generation time can be accelerated.

[0043] Finally, step S105 is executed to generate a speaking video corresponding to the character image according to the multiple frames of video images, wherein the speaking video is an AIGC video, and the speaking content of the speaking video is the content of the driving text. Specifically, multiple frames of video images are repaired by a face repair model to obtain multiple frames of repaired video images. Among them, the face repair model includes models such as GFPGAN, GPEN, Restoreformer, and Codeformer. Each frame of the repaired video image is a video image with high-definition enhancement of the facial area and overall clarity. The multiple frames of repaired video images are sequentially encoded to obtain a speaking video. Among them, sequentially refers to the image sequence of the video image.

[0044] In this way, the face restoration model can significantly improve the resolution and clarity of the generated video image, eliminate the noise of face pixels, provide a better visual experience, make the generated face image more realistic and natural, and improve the quality of the speaking video. In addition, the visual effect of the generated speaking video can be significantly improved, the user experience can be enhanced, and the speaking video can be guaranteed.

[0045] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages: In this embodiment, after obtaining the driving text and the character image, the target voice features are obtained according to the driving text, and the image features and facial features are obtained according to the character image. Here, the driving text is used as the input of the digital human speaking video, which greatly reduces the generation cost of the speaking video and improves the generation efficiency and speed. In addition, the target voice features obtained by the driving text have high-level semantic information, which facilitates the generation of a realistic speaking video.

[0046] Then, the target voice features, image features and facial features are fused to obtain multiple frames of video images. Here, the target voice features are embedded and fused with the image features and facial features to generate video images frame by frame. Based on the target voice features with voice information, the digital person in the video image is realistic, and the digital person's speaking expression and lip shape are natural, so that the video image can reflect the emotions when speaking, and personalized video images and speaking videos can be formulated. Then, based on the multiple frames of video images, the speaking video corresponding to the character image is generated. In this way, the speaking video generation efficiency is high, the generation effect is excellent, and the user experience is enhanced.

[0047] Embodiment 2 Based on the same inventive concept, the second embodiment of the present invention also provides an AIGC video generation device driven by text, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned AIGC video generation methods driven by text are implemented.

[0048] It should be noted that this device includes but is not limited to virtual devices, electronic devices, electronic chips, etc.

[0049] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0051] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A text-driven AIGC video generation method, characterized in that: include: Get driving text and character images; Generating target speech features according to the driving text; According to the character image, obtaining image features and facial features of the character image; fusing the target speech feature, the image feature and the facial feature to obtain multiple frames of video images; A speaking video corresponding to the character image is generated according to the multiple frames of the video image, wherein the speaking video is an AIGC video, and the speaking content of the speaking video is the content of the driving text.

2. The method according to claim 1, characterized in that The step of generating target speech features according to the driving text includes: According to the driving text, obtaining voice and text features corresponding to the driving text; According to the speech, a speech feature is obtained, wherein the speech feature is a feature vector including a semantic feature; The target speech feature is obtained according to the speech feature and the text feature.

3. The method according to claim 2, characterized in that The step of obtaining the target speech feature according to the speech feature and the text feature includes: Extracting audio features from the speech features through an LSTM network, and extracting text features from the text features through a text extractor; Concatenating the audio feature and the text feature to obtain a joint feature; The joint feature is adjusted by a variance adapter to obtain an adjusted joint feature; The adjusted joint feature is encoded by an encoder to obtain an encoded joint feature, and the encoded joint feature is residually connected with the speech feature to obtain the target speech feature.

4. The method according to claim 2, characterized in that According to the driving text, the text feature is obtained, including: The driving text is encoded by a clip encoder to obtain the text features.

5. The method according to claim 2, characterized in that The obtaining of speech features according to the speech includes: Mapping the speech to a latent space through a convolutional network to obtain latent features of the speech in the latent space; The latent features are encoded through a Transformer network to obtain the speech features.

6. The method according to claim 1, characterized in that The step of fusing the target voice feature, the image feature and the facial feature to obtain multiple frames of video images includes: The target speech features, the image features and the facial features are fused through a diffusion model until the posture constraint condition of the motion estimation matrix is ​​reached, thereby obtaining a frame of the video image, and then obtaining multiple frames of the video image.

7. The method according to claim 6, characterized in that The motion estimation matrix is: M = Mt,t,E[||E - Et(Gt,t,C)|| 2 ]; Among them, M is the motion estimation matrix, t is the time step, C is the speech feature, E is the multi-layer perceptron, Gt is Gaussian noise, Mt is the motion space matrix at t time step, and Et is the multi-layer perceptron linear operation.

8. The method according to claim 6, characterized in that The posture constraint condition of the motion estimation matrix is ​​a target posture of a target feature obtained through the motion estimation matrix, and a condition for adjusting the target feature from a current posture to the target posture, wherein the target feature is a specified feature of the image feature and / or a specified feature of the facial feature.

9. The method according to claim 1, characterized in that Generating a speaking video corresponding to the character image according to the multiple frames of the video image, including: Repairing the multiple frames of the video image using a face repair model to obtain multiple frames of repaired video images; The multiple frames of the repaired video images are sequentially encoded to obtain the speaking video.

10. An AIGC video generation device driven by text, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the text-driven AIGC video generation method as described in any one of claims 1 to 9 are implemented.

Images