A method for generating a moving image of key points of a face and related equipment

Abstract

The present application discloses a method for generating a motion image of facial key points and related equipment, which can obtain a reference image frame, and call a facial key point prediction model to extract facial key point information corresponding to the face in the reference image frame; obtain a target audio, and call an audio feature model to extract the target audio feature of the target audio; based on a preset window size, intercept the audio feature to be processed from the target audio feature, and obtain at least one preceding image frame corresponding to the audio feature to be processed; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio feature to be processed, and the preceding image frame, call a diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes. The present application can generate a natural and accurate facial key point motion image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating a motion image of key points of a human face and related equipment. Background Art

[0002] Predicting facial key point motion images from audio is a key task involved in scenarios such as digital human speaking, audio image driving, and 3D facial animation. The input of this task is audio and a reference face image. For example, the audio can be a piece of speaking or singing audio. The goal of this task is to predict the motion sequence of the face under this audio.

[0003] Existing technologies are often based on a one-to-one approach, which extracts audio features and facial key point information, and then uses a generative model to make a one-to-one prediction. However, due to the diversity of facial key point motion images under the same audio, and people's sensitivity to head movements, expressions, and lip shapes, the generated results are not good. Summary of the invention

[0004] An embodiment of the present application provides a method for generating a facial key point motion image and related equipment. The related equipment may include a facial key point motion image generating device, an electronic device, a computer-readable storage medium and a computer program product, which can generate natural and accurate facial key point motion images.

[0005] The present application provides a method for generating a moving image of facial key points, including:

[0006] Obtain a reference image frame, and call a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame;

[0007] Acquire target audio, and call an audio feature model to extract target audio features of the target audio;

[0008] Based on a preset window size, intercepting an audio feature to be processed from the target audio feature, and acquiring at least one preceding image frame corresponding to the audio feature to be processed;

[0009] Based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame, the diffusion network model is called to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes.

[0010] Accordingly, the embodiment of the present application provides a device for generating a moving image of key points of a face, comprising:

[0011] A first acquisition unit is used to acquire a reference image frame and call a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame;

[0012] A second acquisition unit is used to acquire target audio and call an audio feature model to extract target audio features of the target audio;

[0013] A capture unit, configured to capture the audio feature to be processed from the target audio feature based on a preset window size, and obtain at least one preceding image frame corresponding to the audio feature to be processed;

[0014] The first prediction unit is used to call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame.

[0015] Optionally, in some embodiments of the present application, the interception unit may be specifically configured to segment the target audio feature into a plurality of adjacent audio features based on a preset window size; and determine an audio feature to be processed from the plurality of audio features.

[0016] Optionally, in some embodiments of the present application, the interception unit can be specifically used to determine a preceding audio feature that is located in front of the audio feature to be processed based on an arrangement order between the multiple audio features; and obtain at least one preceding image frame from a preceding facial key point motion image corresponding to the preceding audio feature.

[0017] Optionally, in some embodiments of the present application, the first prediction unit can be specifically used to obtain the noise to be added, the preset time step value, and the preset number of iterations; based on the noise to be added, the preset time step value, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the initial facial key point motion image corresponding to the first iteration process; perform noise processing on the initial facial key point motion image to obtain a noisy image after noise addition, and update the preset time step value to the time step value corresponding to the current iteration process; based on the noisy image after noise addition, the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through the iterative process of the preset number of iterations.

[0018] Optionally, in some embodiments of the present application, the facial key point motion image generation device may further include a rendering unit as follows:

[0019] The rendering unit is used to perform image rendering operation based on the facial key point motion image to obtain a target video.

[0020] Optionally, in some embodiments of the present application, the facial key point motion image generation device may further include a third acquisition unit, a key point first prediction unit, a noise adding unit, a second prediction unit, and a first training unit, as follows:

[0021] A third acquisition unit is used to acquire a sample video, and based on the preset window size, intercept a sample sub-video to be processed from the sample video, wherein the sample sub-video to be processed includes a plurality of sample image frames to be processed and a sample audio to be processed;

[0022] A first key point prediction unit, configured to call the face key point prediction model to predict the face key points corresponding to the faces in the plurality of sample image frames to be processed, and generate a target sample image frame;

[0023] A denoising unit, used to determine a training time step value, and perform denoising processing on the target sample image frame based on the training time step value to obtain a noisy sample image frame;

[0024] A second prediction unit is used to call the initial diffusion network model to predict and obtain a target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame;

[0025] The first training unit is used to train the initial diffusion network model based on the target prediction image frame and the target sample image frame to obtain a diffusion network model.

[0026] Optionally, in some embodiments of the present application, the second prediction unit can be specifically used to probabilistically discard the preceding sample image frame corresponding to the sample sub-video to be processed, and update the preceding sample image frame corresponding to the sample sub-video to be processed based on the result after probabilistic discarding; or to jointly probabilistically discard the preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed, and update the preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed based on the result after joint probabilistic discarding.

[0027] Optionally, in some embodiments of the present application, the second prediction unit may include a first calling subunit, a second calling subunit, a third calling subunit, a fourth calling subunit, a fifth calling subunit, and a sixth calling subunit, as follows:

[0028] The first calling subunit is used to call the audio encoder in the initial diffusion network model to process the sample audio to be processed, and obtain the encoded sample audio features;

[0029] A second calling subunit is used to call the input encoder in the initial diffusion network model to process the noisy sample image frame to obtain encoded sample image features;

[0030] The third calling subunit is used to call the time step encoder in the initial diffusion network model to process the training time step value to obtain the encoded sample time step feature;

[0031] A fourth calling subunit is used to call the preceding frame encoder in the initial diffusion network model to process the preceding sample image frame to obtain features of the encoded preceding sample image frame;

[0032] A fifth calling subunit is used to call the reference frame encoder in the initial diffusion network model to process the reference sample image frame to obtain features of the encoded reference sample image frame;

[0033] The sixth calling subunit is used to call the initial diffusion network model to predict and obtain a target predicted image frame based on the encoded sample image feature, the encoded sample time step feature, the encoded sample audio feature, the encoded previous sample image frame feature, and the encoded reference sample image frame feature.

[0034] Optionally, in some embodiments of the present application, the first calling subunit can be specifically used to call the audio feature model to extract sample audio features of the sample audio to be processed; input the sample audio features into the audio encoder in the initial diffusion network model for encoding, and obtain encoded audio features.

[0035] Optionally, in some embodiments of the present application, the sixth calling subunit can be specifically used to perform feature splicing on the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded preceding sample image frame features, and the encoded reference sample image frame features to obtain spliced ​​features; input the spliced ​​features into the self-attention subnetwork of the initial diffusion network model, and call the self-attention subnetwork to predict the target predicted image frame.

[0036] Optionally, in some embodiments of the present application, the first training unit may include a construction subunit and a second training subunit, as follows:

[0037] A construction subunit, configured to construct a target loss function based on a Euclidean distance between the target prediction image frame and the target sample image frame;

[0038] The second training subunit is used to train the initial diffusion network model by minimizing the target loss function to obtain a diffusion network model.

[0039] Optionally, in some embodiments of the present application, the construction subunit can be specifically used to construct a smoothing sub-loss function based on the difference between adjacent image frames of the target prediction image frame; construct a distance difference sub-loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame; and construct a target loss function based on the smoothing sub-loss function and the distance difference sub-loss function.

[0040] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the method for generating a motion image of facial key points provided in the embodiment of the present application.

[0041] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method for generating a motion image of facial key points provided in the embodiment of the present application are implemented.

[0042] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the method for generating a motion image of facial key points provided in an embodiment of the present application.

[0043] The embodiment of the present application provides a method for generating a facial key point motion image and related equipment, which can obtain a reference image frame, and call a facial key point prediction model to extract facial key point information corresponding to the face in the reference image frame; obtain a target audio, and call an audio feature model to extract the target audio feature of the target audio; based on a preset window size, intercept the audio feature to be processed from the target audio feature, and obtain at least one previous image frame corresponding to the audio feature to be processed; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio feature to be processed, and the previous image frame, call a diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterations. Since the diffusion network model has a many-to-many characteristic and has also shown significant generation capabilities in other fields, the present application can predict the facial key point motion image in each diffusion step by involving a lightweight diffusion network model, so as to generate a natural and accurate facial key point motion image, which includes reasonable head movement, timely blinking, expressions that fit the audio, and accurate lip shape, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a scene schematic diagram of the method for generating a moving image of facial key points provided in an embodiment of the present application;

[0046] Figure 2 is a first flow chart of a method for generating a moving image of facial key points provided in an embodiment of the present application;

[0047] Figure 3 is a second flow chart of the method for generating a moving image of facial key points provided in an embodiment of the present application;

[0048] Figure 4 is a third flow chart of the method for generating a moving image of facial key points provided in an embodiment of the present application;

[0049] Figure 5 is a fourth flow chart of the method for generating a moving image of facial key points provided in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of a moving image of key points of a face provided in an embodiment of the present application;

[0051] Figure 7 It is a structural schematic diagram of a device for generating a moving image of key points of a face provided in an embodiment of the present application;

[0052] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0054] The embodiment of the present application provides a method for generating a moving image of key points of a face and a related device, and the related device may include a moving image generating device of key points of a face, an electronic device, a computer-readable storage medium, and a computer program product. The moving image generating device of key points of a face may be integrated in an electronic device, and the electronic device may be a terminal or a server.

[0055] It is understandable that the method for generating a moving image of facial key points of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting the present application.

[0056] like Figure 1 As shown, a method for generating a moving image of key points of a face is performed by a terminal and a server together as an example. The system for generating a moving image of key points of a face provided in the embodiment of the present application includes a terminal and a server, etc. The terminal and the server are connected via a network, such as a wired or wireless network connection, etc., wherein the device for generating a moving image of key points of a face can be integrated in the server.

[0057] The server can be used to: obtain a reference image frame from a terminal, and call a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame; obtain target audio from a terminal, and call an audio feature model to extract target audio features of the target audio; based on a preset window size, intercept the audio features to be processed from the target audio features, and obtain at least one preceding image frame corresponding to the audio features to be processed; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the preceding image frame, call a diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes; perform image rendering operations based on the facial key point motion image to obtain a target video, and send the target video to the terminal for video display on the terminal. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The method or device for generating a motion image of facial key points disclosed in the present application, wherein multiple servers can be formed into a blockchain, and the servers are nodes on the blockchain.

[0058] The terminal can be used to obtain a reference image frame and a target audio, then input the reference image frame and the target audio into a server, and receive a target video from the server. The terminal can include a mobile phone, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, a tablet computer, a laptop computer, or a personal computer (PC). A client can also be set on the terminal, and the client can be an application client or a browser client, etc.

[0059] It should be noted that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0060] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in fields such as machine vision and natural language processing.

[0061] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding training sets, network models that can achieve different functions can be trained.

[0062] The present application relates to the field of computer vision of artificial intelligence technology, and provides a method for generating a moving image of key points of a face and a related device. The method for generating a moving image of key points of a face can be performed by a moving image generating device of key points of a face, or by an electronic device integrated with the moving image generating device of key points of a face.

[0063] It is understandable that in the specific implementation of the present application, related data such as user information (such as reference image frames and target audio) is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0064] The following are detailed descriptions. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. This embodiment can be applied to various scenarios such as cloud technology and artificial intelligence. Figure 2 As shown, the specific process of the method for generating a moving image of facial key points can be as follows:

[0065] S201, obtaining a reference image frame, and calling a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame.

[0066] Among them, audio prediction of facial key point motion images is a face generation task, that is, it can generate diversified dynamic faces that meet the audio characteristics based on the given audio. This task requires a reference face image, that is, a reference image frame, to be given in advance. For example, a photo of person A can be given in advance as a reference image frame, and a target audio can be used to generate a target video, which includes person A performing actions corresponding to the audio. This task needs to ensure that the generated facial key point motion images are natural and accurate. Specifically, the facial key point motion images need to include reasonable head movements, timely blinking, expressions that fit the audio, and accurate lip shapes, etc.

[0067] Among them, the facial key point prediction model is a neural network model that can realize the positioning of facial key points. Facial key point positioning is a technology in the field of computer vision, which aims to automatically locate key positions on facial images, such as eyes, nose, mouth and other feature points through algorithms.

[0068] For example, an image containing a face can be obtained as a reference image frame, and then the reference image frame can be input into a facial key point prediction model, and multiple facial key points of the face in the reference image frame can be extracted through the facial key point prediction model, and then the acquired information can be integrated to obtain facial key point information, wherein the facial key point information includes coordinate information of multiple facial key points.

[0069] In one embodiment, a model pre-training step is also required before the task is executed to obtain a facial key point prediction model that can accurately extract the coordinates of facial key points on a two-dimensional image.

[0070] S202: Obtain target audio, and call an audio feature model to extract target audio features of the target audio.

[0071] Among them, the audio feature model is a network model used to extract audio features from audio. For example, the HuBERT speech recognition pre-trained model can be used to extract audio features. This model is a deep learning framework of a pre-trained model. It is used to extract high-quality speech features from large-scale audio data. It brings new possibilities for speech recognition and conversion with its efficient unsupervised learning method and excellent performance.

[0072] For example, the audio of a song can be used as the target audio, and the target audio features in the target audio can be extracted through the HuBERT speech recognition pre-trained model.

[0073] In one embodiment, a model pre-training step is also required before the task is executed to obtain an audio feature model that can accurately extract audio features.

[0074] S203: based on a preset window size, extract the audio feature to be processed from the target audio feature, and obtain at least one preceding image frame corresponding to the audio feature to be processed.

[0075] Among them, the preceding image frame is an image frame used to provide preceding information for the generation of facial key point motion images. By introducing the preceding image frame, the coherence between the final generated video information can be greatly improved, thereby obtaining a better video generation effect.

[0076] For example, the preset window size may be 60, and then the audio features to be processed may be extracted from the target audio features according to the preset window size of 60. Then, at least one preceding image frame corresponding to the audio features to be processed is obtained so that the preceding information may be considered in subsequent processing.

[0077] Optionally, in one embodiment, the step of “cutting the audio feature to be processed from the target audio feature based on a preset window size” may include:

[0078] Based on a preset window size, the target audio feature is segmented into a plurality of adjacent audio features;

[0079] An audio feature to be processed is determined from a plurality of audio features.

[0080] For example, the preset window size may be 60, and then the target audio feature may be divided into multiple audio features of size 60 according to the preset window size of 60: audio feature 1, audio feature 2, audio feature 3, etc., and these audio features are adjacent to each other, that is, audio feature 1 is adjacent to audio feature 2, audio feature 2 is adjacent to audio feature 3, etc. If audio feature 1 has been processed and the corresponding preceding target video has been generated, then audio feature 2 may be determined as the audio feature to be processed.

[0081] Optionally, in one embodiment, the step of “obtaining at least one preceding image frame corresponding to the audio feature to be processed” may include:

[0082] Determine, according to the arrangement order of the multiple audio features, a preceding audio feature located before the audio feature to be processed;

[0083] At least one preceding image frame is obtained from a preceding facial key point motion image corresponding to a preceding audio feature.

[0084] For example, if audio feature 2 is determined as the audio feature to be processed, then audio feature 1 located in front of audio feature 2 can be determined as the previous audio feature. Since audio feature 1 has generated the corresponding previous target video, the last three frames of the previous facial key point motion image obtained during the generation of the previous target video can be used as the previous image frames.

[0085] In one embodiment, since one audio feature corresponds to at least one facial key point motion image, the purpose of obtaining the previous image frame is to consider the previous information. Therefore, the last three frames of the previous facial key point motion image obtained in the process of generating the previous target video can be used as the previous image. Other numbers of previous images can also be obtained, and the embodiment of the present application does not impose corresponding restrictions on this.

[0086] S204, based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes.

[0087] Among them, the facial key point motion image is an image composed of multiple facial key point motion sequences. The facial key point motion image can represent the distribution of facial key points at a certain moment. Through multiple facial key point motion images, the motion of each key point of the face over a period of time can be understood, so that the target video containing facial motion can be obtained by rendering. Input the reference image frame and the target audio, and after processing, it can be output as follows Figure 6 The multiple facial key point motion images shown in the figure can then be rendered to obtain the target video containing facial motion.

[0088] Among them, the diffusion model is a method that destroys the training data by continuously adding Gaussian noise, and then recovers the data by learning to flip the noise process. After training, the diffusion model can be used to generate data by passing randomly sampled noise through the learned denoising process. That is, the purpose of the diffusion model is to eliminate the continuous application of Gaussian noise to the training images, and it can be regarded as a series of denoising autoencoders.

[0089] In the embodiment of the present application, a potential diffusion model (Stable Diffusion) can be used. The potential diffusion model consists of three parts: a variational autoencoder (VAE), a U-Net, and a text encoder. The potential diffusion model can convert the image into a low-dimensional latent space by training the VAE, and the process of adding and removing Gaussian noise is applied to this latent space, and then the final denoised output is decoded into the pixel space. During the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent representation. Each denoising step is completed by a U-Net architecture containing a residual neural network (ResNet) in the middle, and the potential representation is obtained by denoising in the reverse direction from forward diffusion. Finally, the VAE decoder generates an output image by converting the representation back to the pixel space. The encoding of the adjustment data is exposed to the architecture of the denoising U-Net through a cross-attention mechanism. In order to adjust the text, a pre-trained fixed CLIP ViT-L / 14 text encoder is used to convert the prompt word into an embedding space.

[0090] In one embodiment, the Transformers structure can also be used to replace the U-Net backbone network in the diffusion model. This Diffusion Transformers is not only faster and has higher Gflops, but also achieves better results in the ImageNet 512×512 and 256×256 category conditional image generation tasks.

[0091] Among them, the noise to be added includes randomly initialized noise. Due to the randomness of noise initialization, the noise to be added obtained each time is different. So given the same target audio, multiple sets of motion sequences can be generated, which greatly improves the diversity of facial key point motion image generation technology.

[0092] The time step value corresponds to the iteration process. For example, if the number of iterations is set to 1000, the time step value t can be set to 999 for the first iteration. Then, as the iterations proceed, the time step value t also changes to obtain the final iteration result.

[0093] For example, the noise to be added can be obtained, and the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame can be obtained, and these contents can be input into the diffusion network model, and the diffusion network model can be called for multiple iterations, and finally the facial key point motion image of the reference image frame under the target audio can be predicted.

[0094] In one embodiment, since the target audio feature has been segmented into multiple audio features, they can be processed one sequence at a time in a sequential order. For example, the corresponding facial key point motion image 1 is first generated based on audio feature 1, and then the corresponding facial key point motion image 2 is generated based on audio feature 2, and so on.

[0095] Optionally, in one embodiment, the step of “calling the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterations based on the noise to be added, the time step value corresponding to the iteration process, the facial key point information, the audio feature to be processed, and the previous image frame” may include:

[0096] Obtain the noise to be added, the preset time step value, and the preset number of iterations;

[0097] Based on the noise to be added, the preset time step value, the facial key point information, the audio features to be processed, and the previous image frame, the diffusion network model is called to predict the initial facial key point motion image corresponding to the first iteration process;

[0098] The initial facial key point motion image is subjected to noise processing to obtain a noise image after noise addition, and the preset time step value is updated to the time step value corresponding to the current iteration process;

[0099] Based on the noisy image after adding noise, the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame, the diffusion network model is called to predict the facial key point motion image of the reference image frame under the target audio through an iterative process with a preset number of iterations.

[0100] For example, the preset number of iterations of the diffusion network model can be set to 1000, and the noise to be added can be obtained, the preset time step value can be set to 999, and then the noise to be added, the preset time step value 999, the facial key point information, the audio features to be processed, and the previous image frame can be input into the diffusion network model, and the diffusion network model can be called to predict the initial facial key point motion image X999 pred corresponding to the first iteration process. The initial facial key point motion image X999 pred is subjected to noise processing to obtain a noise image after noise addition, and the noise image after noise addition is used as the input of the next iteration, and the preset time step value 999 is updated to the time step value 998 corresponding to the current iteration process.

[0101] Then the noise image after adding noise, the time step value 998 corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame are input into the diffusion network model, and the diffusion network model is called to predict the initial facial key point motion image X998 pred corresponding to this iteration process. Then the initial facial key point motion image X998pred is subjected to noise processing to obtain the noise image after adding noise, and the noise image after adding noise is used as the input of the next iteration, and the preset time step value 998 is updated to the time step value 997 corresponding to the current iteration process, and so on, until the facial key point motion image X0 pred of the reference image frame under the target audio is predicted.

[0102] Optionally, in one embodiment, the method for generating a facial key point motion image may further include:

[0103] An image rendering operation is performed based on the motion image of facial key points to obtain a target video.

[0104] For example, after obtaining the motion image of the key points of the face through prediction, an image rendering operation can be performed on it to obtain the desired target video, which includes the face included in the reference image frame, and the movement of the face meets the requirements of the target audio. If the reference image frame includes person A, and the target audio is the audio of a song, the target video can be a video of person A singing the song. The embodiments of the present application can be used in the fields of digital human speech, audio image driving, and three-dimensional face animation.

[0105] Optionally, in one embodiment, as Figure 3 As shown, the method for generating a moving image of facial key points may also include a training process of a diffusion network model, as follows:

[0106] S301, obtaining a sample video, and based on a window size, extracting a sample sub-video to be processed from the sample video, wherein the sample sub-video to be processed includes a plurality of sample image frames to be processed and a sample audio to be processed.

[0107] For example, a sample video can be obtained as a sample, and the preset window size N is set to 60, and then a sample sub-video to be processed with a length of 60 is randomly extracted from the sample video, and the sample sub-video to be processed includes multiple sample image frames to be processed and sample audio to be processed.

[0108] S302: calling a facial key point prediction model to predict facial key points corresponding to faces in a plurality of sample image frames to be processed, and generating a target sample image frame.

[0109] For example, the facial key point prediction model can be called to predict the facial key points corresponding to the faces in multiple sample image frames to be processed, obtain the facial key point information (PTS), and perform image normalization on it to generate the target sample image frame X0, where the normalization formula can be as follows:

[0110] PTS=(PTS*2 / h)-1

[0111] Where h is the width of the image. Applying the above formula can make the normalized values ​​concentrated between (-1, 1), thereby achieving the purpose of normalization.

[0112] S303: Determine a training time step value, and perform noise processing on the target sample image frame based on the training time step value to obtain a noisy sample image frame.

[0113] For example, the total number of iterations of the initial diffusion network model can be determined to be 1000, and then the training time step value t is randomly selected within (0,1000). Since the diffusion network model needs to be predicted based on the noise map when predicting, the noise-free target sample image frame can be denoised, where the denoising formula can be as follows:

[0114]

[0115] Among them, ε represents Gaussian noise and t represents the training time step value.

[0116] S304, based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame, call the initial diffusion network model to predict and obtain the target predicted image frame.

[0117] Optionally, in one embodiment, before the step of “calling the initial diffusion network model to predict and obtain a target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame”, the following may also be included:

[0118] Determine a preset range of iteration times, and determine a training time step value from the preset range of iteration times;

[0119] Based on the time period corresponding to the sample sub-video to be processed, at least one preceding sample image frame is captured from the sample video.

[0120] For example, the total number of iterations of the initial diffusion network model can be determined to be 1000, and then the training time step value t is randomly selected in (0,1000). Then the time period corresponding to the sample sub-video to be processed is determined. In order to better maintain the continuity of the previous and next time series, it is also necessary to intercept 3 previous sample image frames before the time period from the sample video, where the previous sample image frames are in the form of PTS information.

[0121] Optionally, in one embodiment, the step of “calling the initial diffusion network model to predict and obtain a target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame” may include:

[0122] Calling the audio encoder in the initial diffusion network model to process the sample audio to be processed, and obtaining the encoded sample audio features;

[0123] Call the input encoder in the initial diffusion network model to process the noisy sample image frame to obtain the encoded sample image features;

[0124] Call the time step encoder in the initial diffusion network model to process the training time step value and obtain the encoded sample time step feature;

[0125] Calling the previous frame encoder in the initial diffusion network model to process the previous sample image frame, and obtaining the features of the encoded previous sample image frame;

[0126] Calling the reference frame encoder in the initial diffusion network model to process the reference sample image frame to obtain the encoded reference sample image frame features;

[0127] Based on the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded previous sample image frame features, and the encoded reference sample image frame features, the initial diffusion network model is called to predict the target predicted image frame.

[0128] For example, Figure 5As shown, the sample audio to be processed can be input into the audio encoder in the initial diffusion network model, and the encoded sample audio features can be obtained. The sample image frame after noise addition is input into the input encoder in the initial diffusion network model, wherein the input encoder includes a two-layer transformer network, and the encoded sample image features are obtained. The training time step value t is input into the time step encoder in the initial diffusion network model, wherein the time step encoder includes a fully connected network layer and an activation function network layer, and the encoded sample time step features are obtained. The previous sample image frame is input into the previous frame encoder in the initial diffusion network model, and the previous frame encoder includes a two-layer transformer network to obtain the encoded previous sample image frame features. In order to provide certain reference information to the network model, the embodiment of the present application also introduces a reference sample image frame, and then the reference sample image frame is input into the reference frame encoder in the initial diffusion network model, and the reference frame encoder includes a two-layer transformer network to obtain the encoded reference sample image frame features.

[0129] The embodiments of the present application do not limit the number of network layers in the network model, and the number of network layers in the network model can be increased or decreased according to actual needs.

[0130] Among them, transformer is a deep learning model architecture for natural language processing and other sequence-to-sequence tasks. It introduces a self-attention mechanism that enables the model to consider all positions in the input sequence at the same time. The self-attention mechanism allows the model to assign different attention weights according to different parts of the input sequence in order to better capture semantic relationships.

[0131] Optionally, in one embodiment, the step of “calling the audio encoder in the initial diffusion network model to process the sample audio to be processed to obtain the encoded sample audio features” may include:

[0132] Calling the audio feature model to extract sample audio features of the sample audio to be processed;

[0133] The sample audio features are input into the audio encoder in the initial diffusion network model for encoding to obtain encoded audio features.

[0134] For example, the audio feature model Hubert can be used to extract sample audio features of the sample audio to be processed, and then the sample audio features are input into the audio encoder in the initial diffusion network model for encoding, wherein the audio encoder includes a two-layer transformer network to obtain the encoded audio features.

[0135] Optionally, in one embodiment, the step of “calling the initial diffusion network model to predict and obtain a target predicted image frame based on the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded preceding sample image frame features, and the encoded reference sample image frame features” may include:

[0136] Performing feature splicing on the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded preceding sample image frame features, and the encoded reference sample image frame features to obtain spliced ​​features;

[0137] The concatenated features are input into the self-attention subnetwork of the initial diffusion network model, and the self-attention subnetwork is called to predict the target predicted image frame.

[0138] For example, the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded pre-order sample image frame features, and the encoded reference sample image frame features can be feature spliced ​​to obtain the spliced ​​features, and then the spliced ​​features are input into the self-attention sub-network (Transformer encoder) of the initial diffusion network model, wherein the self-attention sub-network of the embodiment of the present application may include eight layers of transformer layers for learning any pairwise time series information, that is, the self-attention sub-network can enable information interaction between any two frames of information. Then the result output by the self-attention sub-network is input into the linear layer to obtain the final target predicted image frame.

[0139] Optionally, in one embodiment, before the step of “calling the initial diffusion network model to predict and obtain a target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame”, at least one of the following two processing methods may be included:

[0140] Probabilistically discarding the preceding sample image frame corresponding to the sample sub-video to be processed, and updating the preceding sample image frame corresponding to the sample sub-video to be processed based on the result after probabilistic discarding;

[0141] The preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed are jointly probability discarded, and the preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed are updated based on the result after the joint probability discard.

[0142] In one embodiment, in order to realize the generation of many-to-many sequences, the diffusion model will randomly delete (drop) information during the training process. This step can be divided into two stages. In the first stage, the preceding sample image frame can be dropped. For example, the drop probability of the preceding sample image frame can be set to 0.36, that is, there is a 36% probability of setting all the inputs of the preceding sample image to 0; the second stage is the joint drop of the preceding sample image frame, the reference sample image frame, and the sample audio feature. For example, the drop probability can be set to 0.1, that is, there is a 10% probability of setting all the inputs of the preceding sample image frame, the reference sample image frame, and the sample audio feature to 0. Alternatively, only the step of dropping the preceding sample image frame can be implemented, or only the joint drop step of the preceding sample image frame, the reference sample image frame, and the sample audio feature can be implemented.

[0143] S305 , training an initial diffusion network model based on the target prediction image frame and the target sample image frame to obtain a diffusion network model.

[0144] For example, if the target prediction image frame obtained is the result after reducing noise, it can be compared with the noise-free target sample image frame, and the initial diffusion network model can be trained based on the difference results to obtain the trained diffusion network model, which can be applied to subsequent testing steps.

[0145] Optionally, in one embodiment, the step of “training an initial diffusion network model based on the target prediction image frame and the target sample image frame to obtain a diffusion network model” may include:

[0146] Constructing a target loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame;

[0147] The initial diffusion network model is trained by minimizing the objective loss function to obtain a diffusion network model.

[0148] Optionally, in one embodiment, the step of “constructing a target loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame” may include:

[0149] Based on the difference between adjacent image frames of the target prediction image frame, a smoothing sub-loss function is constructed;

[0150] Based on the Euclidean distance between the target prediction image frame and the target sample image frame, a distance difference sub-loss function is constructed;

[0151] Based on the smoothing sub-loss function and the distance difference sub-loss function, the target loss function is constructed.

[0152] Based on the Euclidean distance between the target prediction image frame and the target sample image frame, the distance difference sub-loss function formula can be constructed as follows:

[0153] L mse =||X0 pred , X0||2

[0154] Since the stability between adjacent frames is very important in the prediction of facial key point motion images, the stability between adjacent frames can be guaranteed by introducing the calculation of smoothness between adjacent frames. According to the difference between adjacent image frames of the target prediction image frame, the formula of the constructed smooth sub-loss function can be as follows:

[0155]

[0156] Among them, abs means taking the absolute value.

[0157] The final objective loss function formula can be as follows:

[0158]

[0159] In one embodiment, a pre-step can also be performed before model training to improve the effect of the overall model training. Since the face often occupies only a small position in the input video frame image, it is necessary to first perform face detection to obtain the target face area, then perform face registration on the target face area to obtain face key points, and finally perform image cropping based on the acquired face key points to obtain a cropped face image, and input this cropped face image as an input image into the diffusion network model.

[0160] As can be seen from the above, this embodiment can obtain a reference image frame, and call a facial key point prediction model to extract facial key point information corresponding to the face in the reference image frame; obtain target audio, and call an audio feature model to extract target audio features of the target audio; based on a preset window size, intercept the audio features to be processed from the target audio features, and obtain at least one previous image frame corresponding to the audio features to be processed; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterations. Since the diffusion network model has a many-to-many characteristic and has also demonstrated significant generation capabilities in other fields, the present application can predict the facial key point motion image in each diffusion step by involving a lightweight diffusion network model, so as to generate a natural and accurate facial key point motion image, which includes reasonable head movement, timely blinking, expressions that fit the audio, and accurate lip shape, etc.

[0161] According to the method described in the previous embodiment, this embodiment will be described from the perspective of a facial key point motion image generation device, which can be integrated in an electronic device, such as a server or a terminal. The present application embodiment provides a facial key point motion image generation method, such as Figure 4 As shown, the specific process of the method for generating a moving image of facial key points can be as follows:

[0162] S401: The electronic device performs pre-processing on a sample video.

[0163] For example, face detection can be first performed on the video frames in the initial sample video to obtain the target face area, and then face registration can be performed in the target face area to predict 256 facial key points in each face, and then image cropping can be performed based on the predicted facial key points to obtain a usable sample video.

[0164] In one embodiment, a model training process may be included before the task is executed. For example, the audio feature model Hubert and the facial key point prediction model may be trained so that the trained models can be directly applied.

[0165] S402: The electronic device obtains a sample video, and based on a preset window size, extracts a sample sub-video to be processed from the sample video.

[0166] For example, a sample video may be first obtained, and a preset window size N may be pre-set, such as N may be set to 60, and then a sample sub-video to be processed is randomly extracted from the sample video, the size of the sample sub-video to be processed is 60, and the sample sub-video to be processed includes a plurality of sample image frames (frames clip) to be processed, and a sample audio (audioclip) to be processed.

[0167] S403: The electronic device calls a facial key point prediction model to predict facial key points corresponding to faces in a plurality of sample image frames to be processed, and generates a target sample image frame.

[0168] For example, a facial key point prediction model can be applied to predict facial key points corresponding to faces in multiple sample image frames to be processed, and generate target sample image frames (PTS sequence).

[0169] In one embodiment, the image processing accuracy can be improved by normalization, and the final target sample image frame X0 can be obtained. The normalization formula can be as follows:

[0170] PTS=(PTS*2 / h)-1

[0171] Here, h represents the width of the image.

[0172] S404: The electronic device determines a training time step value, and performs noise processing on the target sample image frame based on the training time step value to obtain a noisy sample image frame.

[0173] For example, the total number of iterations of the diffusion network model can be determined to be 1000, and then the training time step value t is randomly selected from the range of (0,1000). Then, the noise-free target sample image frame X0 is subjected to noise processing to obtain the noisy sample image frame Xt, where the noise addition formula can be as follows:

[0174]

[0175] Here, ε represents Gaussian noise.

[0176] S405, the electronic device calls the initial diffusion network model to predict and obtain a target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame.

[0177] In one embodiment, in order to better maintain the continuity of the preceding and following time sequences, at least one preceding sample image frame may be introduced. For example, three preceding sample image frames may be introduced. The three preceding sample image frames are adjacent image frames located before the sample sub-video to be processed. That is, PTS information with a size of 3*256*2 is provided as the preceding sample image frame pre pts .

[0178] For example, Figure 5 As shown in the figure, the audio feature model Hubert can be called to predict the sample audio features of the sample audio to be processed, and then the sample audio features are input into the audio encoder in the initial diffusion network model for encoding to obtain the encoded audio features. encoded ), wherein the audio encoder includes a two-layer transformer network.

[0179] like Figure 5 As shown, the noisy sample image frame Xt can be input into the input encoder in the initial diffusion network model, where the input encoder includes a two-layer transformer network to obtain the encoded sample image feature (Xt encoded The training time step value t is input into the time step encoder (Timestepencoder) in the initial diffusion network model, where the time step encoder includes a fully connected network layer and an activation function network layer to obtain the encoded sample time step feature (tencoded ). The previous sample image frame pre pts Input the pre-order frame encoder (PrePts encoder) in the initial diffusion network model, which includes a two-layer transformer network to obtain the pre-order sample image frame features (prepts encoded ). In order to provide certain reference information to the network model, the present embodiment also introduces a reference sample image frame ref pts , the reference sample image frame is a PTS information with a size of 1*256*2, and then the reference sample image frame is input into the reference frame encoder (PefPts encoder) in the initial diffusion network model. The reference frame encoder includes a two-layer transformer network to obtain the encoded reference sample image frame feature (refpts encoded ).

[0180] In one embodiment, in order to achieve the generation of many-to-many sequences, the diffusion network model will randomly delete (drop) information during the training process. For example, the previous sample image frame can be dropped first, for example, the drop probability of the previous sample image frame can be set to 0.36, that is, there is a 36% probability of setting all the inputs of the sample previous image to 0; the second stage is the joint drop of the previous sample image frame, the reference sample image frame, and the sample audio feature, for example, the drop probability can be set to 0.1, that is, there is a 10% probability of setting all the inputs of the previous sample image frame, the reference sample image frame, and the sample audio feature to 0.

[0181] Then, the encoded sample image features, encoded sample time step features, encoded sample audio features, encoded previous sample image frame features, and encoded reference sample image frame features can be feature spliced ​​to obtain spliced ​​features, and then the spliced ​​features are input into the self-attention sub-network (Transformer encoder) of the initial diffusion network model, wherein the self-attention sub-network of the embodiment of the present application may include eight layers of transformer layers for learning any pairwise time series information, that is, the self-attention sub-network can enable information interaction between any two frames of information. Then the result output by the self-attention sub-network is input into the linear layer to obtain the final target predicted image frame X0. pred .

[0182] S406: The electronic device trains an initial diffusion network model based on the target prediction image frame and the target sample image frame to obtain a diffusion network model.

[0183] For example, a distance difference sub-loss function can be constructed according to the Euclidean distance between the target prediction image frame and the target sample image frame, where the distance difference sub-loss function formula can be as follows:

[0184] L mse =||X0 pred , X0||2

[0185] Among them, the loss function in the above formula is calculated using Euclidean distance.

[0186] Since the stability between adjacent frames is very important in the prediction of facial key point motion images, the stability between adjacent frames can be guaranteed by introducing the calculation of smoothness between adjacent frames. According to the difference between adjacent image frames of the target prediction image frame, the formula of the constructed smooth sub-loss function can be as follows:

[0187]

[0188] Among them, abs means taking the absolute value.

[0189] The final objective loss function formula can be as follows:

[0190]

[0191] S407: The electronic device obtains a reference image frame, and calls a facial key point prediction model to extract facial key point information corresponding to the face in the reference image frame.

[0192] For example, you can obtain a reference image frame and call the facial key point prediction model to extract the facial key point information corresponding to the face in the reference image frame. pts .

[0193] S408: The electronic device obtains the target audio, and calls the audio feature model to extract the target audio features of the target audio.

[0194] For example, the target audio may be obtained, and the audio feature model Hubert may be called to extract the target audio features of the target audio.

[0195] S409: The electronic device extracts the audio feature to be processed from the target audio feature based on a preset window size, and obtains at least one preceding image frame corresponding to the audio feature to be processed.

[0196] For example, the preset window size can be set to 60, and then the target audio feature can be segmented according to the size of 60 to obtain multiple adjacent audio features of size 60, and the currently processed audio feature to be processed can be determined from these audio features. After the audio feature to be processed is determined, the last three frames in the PTS sequence corresponding to the preceding audio feature that is located in front of the audio feature to be processed and adjacent to the audio feature to be processed can be obtained as the preceding image frames. For example, in the first prediction window, the preceding frame = [ref pts , ref pts , ref pts ], in the second prediction window, the preceding frame = the last three frames of the PTS sequence of the previous prediction window.

[0197] S410, the electronic device calls the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterations based on the noise to be added, the time step value corresponding to the iteration process, the facial key point information, the audio features to be processed, and the previous image frame.

[0198] For example, the preset number of iterations of the diffusion network model can be set to 1000, and the noise to be added can be obtained, the preset time step value can be set to 999, and then the noise to be added, the preset time step value 999, the facial key point information, the audio features to be processed, and the previous image frame can be input into the diffusion network model, and the diffusion network model can be called to predict the initial facial key point motion image X999 pred corresponding to the first iteration process. The initial facial key point motion image X999 pred is subjected to noise processing to obtain a noise image after noise addition, and the noise image after noise addition is used as the input of the next iteration, and the preset time step value 999 is updated to the time step value 998 corresponding to the current iteration process.

[0199] Then the noise image after adding noise, the time step value 998 corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame are input into the diffusion network model, and the diffusion network model is called to predict the initial facial key point motion image X998 pred corresponding to this iteration process. Then the initial facial key point motion image X998pred is subjected to noise processing to obtain the noise image after adding noise, and the noise image after adding noise is used as the input of the next iteration, and the preset time step value 998 is updated to the time step value 997 corresponding to the current iteration process, and so on, until the facial key point motion image X0 pred of the reference image frame under the target audio is predicted.

[0200] As can be seen from the above, the present embodiment can perform pre-processing on the sample video through the electronic device; obtain the sample video, and based on the preset window size, cut out the sample sub-video to be processed from the sample video; call the face key point prediction model to predict the face key points corresponding to the faces in multiple sample image frames to be processed, and generate a target sample image frame; determine the training time step value, and perform noise processing on the target sample image frame based on the training time step value to obtain the noisy sample image frame; based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame, call the initial diffusion network model to predict the target predicted image frame; based on the target prediction image frame, and target sample image frame, train the initial diffusion network model, and obtain the diffusion network model; obtain the reference image frame, and call the face key point prediction model to extract the face key point information corresponding to the face in the reference image frame; obtain the target audio, and call the audio feature model to extract the target audio feature of the target audio; based on the preset window size, intercept the audio feature to be processed from the target audio feature, and obtain at least one previous image frame corresponding to the audio feature to be processed; based on the noise to be added, the time step value corresponding to the iterative process, the face key point information, the audio feature to be processed, and the previous image frame, call the diffusion network model to predict the face key point motion image of the reference image frame under the target audio through multiple iterations. Since the diffusion network model has a many-to-many characteristic and has also shown significant generation capabilities in other fields, the present application can predict the face key point motion image in each diffusion step by involving a lightweight diffusion network model, so as to generate a natural and accurate face key point motion image, which includes reasonable head movement, timely blinking, expressions that fit the audio, and accurate lip shape, etc.

[0201] In order to better implement the above method, the embodiment of the present application also provides a device for generating a motion image of key points of a face, such as Figure 7 As shown, the facial key point motion image generation device may include a first acquisition unit 701, a second acquisition unit 702, a capture unit 703, and a first prediction unit 704, as follows:

[0202] The first acquisition unit 701 is used to acquire a reference image frame and call a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame;

[0203] The second acquisition unit 702 is used to acquire the target audio and call the audio feature model to extract the target audio feature of the target audio;

[0204] The interception unit 703 is used to intercept the audio feature to be processed from the target audio feature based on a preset window size, and obtain at least one preceding image frame corresponding to the audio feature to be processed;

[0205] The first prediction unit 704 is used to call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame.

[0206] Optionally, in some embodiments of the present application, the interception unit 703 may be specifically configured to segment the target audio feature into a plurality of adjacent audio features based on a preset window size; and determine an audio feature to be processed from the plurality of audio features.

[0207] Optionally, in some embodiments of the present application, the interception unit 703 can be specifically used to determine a preceding audio feature that is located in front of the audio feature to be processed based on the arrangement order between the multiple audio features; and obtain at least one preceding image frame from a preceding facial key point motion image corresponding to the preceding audio feature.

[0208] Optionally, in some embodiments of the present application, the first prediction unit 704 can be specifically used to obtain the noise to be added, the preset time step value, and the preset number of iterations; based on the noise to be added, the preset time step value, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the initial facial key point motion image corresponding to the first iteration process; perform noise processing on the initial facial key point motion image to obtain a noisy image after noise addition, and update the preset time step value to the time step value corresponding to the current iteration process; based on the noisy image after noise addition, the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through the iterative process of the preset number of iterations.

[0209] Optionally, in some embodiments of the present application, the facial key point motion image generation device may further include a rendering unit as follows:

[0210] The rendering unit is used to perform image rendering operation based on the facial key point motion image to obtain a target video.

[0211] Optionally, in some embodiments of the present application, the facial key point motion image generation device may further include a third acquisition unit, a key point first prediction unit, a noise adding unit, a second prediction unit, and a first training unit, as follows:

[0212] A third acquisition unit is used to acquire a sample video, and based on the preset window size, intercept a sample sub-video to be processed from the sample video, wherein the sample sub-video to be processed includes a plurality of sample image frames to be processed and a sample audio to be processed;

[0213] A first key point prediction unit, configured to call the face key point prediction model to predict the face key points corresponding to the faces in the plurality of sample image frames to be processed, and generate a target sample image frame;

[0214] A denoising unit, used to determine a training time step value, and perform denoising processing on the target sample image frame based on the training time step value to obtain a noisy sample image frame;

[0215] A second prediction unit is used to call the initial diffusion network model to predict and obtain a target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame;

[0216] The first training unit is used to train the initial diffusion network model based on the target prediction image frame and the target sample image frame to obtain a diffusion network model.

[0217] Optionally, in some embodiments of the present application, the second prediction unit can be specifically used to probabilistically discard the preceding sample image frame corresponding to the sample sub-video to be processed, and update the preceding sample image frame corresponding to the sample sub-video to be processed based on the result after probabilistic discarding; or to jointly probabilistically discard the preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed, and update the preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed based on the result after joint probabilistic discarding.

[0218] Optionally, in some embodiments of the present application, the second prediction unit may include a first calling subunit, a second calling subunit, a third calling subunit, a fourth calling subunit, a fifth calling subunit, and a sixth calling subunit, as follows:

[0219] The first calling subunit is used to call the audio encoder in the initial diffusion network model to process the sample audio to be processed, and obtain the encoded sample audio features;

[0220] A second calling subunit is used to call the input encoder in the initial diffusion network model to process the noisy sample image frame to obtain encoded sample image features;

[0221] The third calling subunit is used to call the time step encoder in the initial diffusion network model to process the training time step value to obtain the encoded sample time step feature;

[0222] A fourth calling subunit is used to call the preceding frame encoder in the initial diffusion network model to process the preceding sample image frame to obtain features of the encoded preceding sample image frame;

[0223] A fifth calling subunit is used to call the reference frame encoder in the initial diffusion network model to process the reference sample image frame to obtain features of the encoded reference sample image frame;

[0224] The sixth calling subunit is used to call the initial diffusion network model to predict and obtain a target predicted image frame based on the encoded sample image feature, the encoded sample time step feature, the encoded sample audio feature, the encoded previous sample image frame feature, and the encoded reference sample image frame feature.

[0225] Optionally, in some embodiments of the present application, the first calling subunit can be specifically used to call the audio feature model to extract sample audio features of the sample audio to be processed; input the sample audio features into the audio encoder in the initial diffusion network model for encoding, and obtain encoded audio features.

[0226] Optionally, in some embodiments of the present application, the sixth calling subunit can be specifically used to perform feature splicing on the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded preceding sample image frame features, and the encoded reference sample image frame features to obtain spliced ​​features; input the spliced ​​features into the self-attention subnetwork of the initial diffusion network model, and call the self-attention subnetwork to predict the target predicted image frame.

[0227] Optionally, in some embodiments of the present application, the first training unit may include a construction subunit and a second training subunit, as follows:

[0228] A construction subunit, configured to construct a target loss function based on a Euclidean distance between the target prediction image frame and the target sample image frame;

[0229] The second training subunit is used to train the initial diffusion network model by minimizing the target loss function to obtain a diffusion network model.

[0230] Optionally, in some embodiments of the present application, the construction subunit can be specifically used to construct a smoothing sub-loss function based on the difference between adjacent image frames of the target prediction image frame; construct a distance difference sub-loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame; and construct a target loss function based on the smoothing sub-loss function and the distance difference sub-loss function.

[0231] As can be seen from the above, in this embodiment, a reference image frame can be obtained through the first acquisition unit 701, and the facial key point prediction model can be called to extract the facial key point information corresponding to the face in the reference image frame; the target audio can be obtained through the second acquisition unit 702, and the audio feature model can be called to extract the target audio feature of the target audio; the audio feature to be processed can be intercepted from the target audio feature based on a preset window size through the interception unit 703, and at least one previous image frame corresponding to the audio feature to be processed can be obtained; the first prediction unit 704 can be used to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio feature to be processed, and the previous image frame. Since the diffusion network model has a many-to-many characteristic and has demonstrated significant generation capabilities in other fields, the present application can involve a lightweight diffusion network model to predict the motion image of facial key points in each diffusion step, so as to generate a natural and accurate motion image of facial key points, which includes reasonable head movement, timely blinking, expressions that fit the audio, accurate lip shape, etc.

[0232] The present application also provides an electronic device, such as Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, and the electronic device may be a terminal or a server, etc. Specifically:

[0233] The electronic device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will appreciate that Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0234] The processor 801 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 802, and calling data stored in the memory 802. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.

[0235] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0236] The electronic device also includes a power supply 803 for supplying power to each component. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 803 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0237] The electronic device may further include an input unit 804, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0238] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 801 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 802 according to the following instructions, and the processor 801 will run the application program stored in the memory 802, thereby realizing various functions, as follows:

[0239] A reference image frame is obtained, and a facial key point prediction model is called to extract facial key point information corresponding to the face in the reference image frame; a target audio is obtained, and an audio feature model is called to extract target audio features of the target audio; based on a preset window size, audio features to be processed are intercepted from the target audio features, and at least one previous image frame corresponding to the audio features to be processed is obtained; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame, a diffusion network model is called to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes.

[0240] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0241] As can be seen from the above, this embodiment can obtain a reference image frame, and call a facial key point prediction model to extract facial key point information corresponding to the face in the reference image frame; obtain target audio, and call an audio feature model to extract target audio features of the target audio; based on a preset window size, intercept the audio features to be processed from the target audio features, and obtain at least one previous image frame corresponding to the audio features to be processed; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame, call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterations. Since the diffusion network model has a many-to-many characteristic and has also demonstrated significant generation capabilities in other fields, the present application can predict the facial key point motion image in each diffusion step by involving a lightweight diffusion network model, so as to generate a natural and accurate facial key point motion image, which includes reasonable head movement, timely blinking, expressions that fit the audio, and accurate lip shape, etc.

[0242] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0243] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the methods for generating a moving image of facial key points provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0244] A reference image frame is obtained, and a facial key point prediction model is called to extract facial key point information corresponding to the face in the reference image frame; a target audio is obtained, and an audio feature model is called to extract target audio features of the target audio; based on a preset window size, audio features to be processed are intercepted from the target audio features, and at least one previous image frame corresponding to the audio features to be processed is obtained; based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio features to be processed, and the previous image frame, a diffusion network model is called to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes.

[0245] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0246] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0247] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the methods for generating facial key point motion images provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the methods for generating facial key point motion images provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0248] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above-mentioned face key point motion image generation aspect.

[0249] The above is a detailed introduction to a method for generating a motion image of facial key points and related equipment provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating a moving image of facial key points, characterized in that: include: Obtain a reference image frame, and call a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame; Acquire target audio, and call an audio feature model to extract target audio features of the target audio; Based on a preset window size, intercepting an audio feature to be processed from the target audio feature, and acquiring at least one preceding image frame corresponding to the audio feature to be processed, wherein the preceding image frame is an image frame used to provide preceding information for generating a motion image of facial key points; Based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio feature to be processed, and the previous image frame, calling the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes, including: Based on a current iteration process in the multiple iteration processes, obtaining randomly initialized noise to be added; Based on the randomly initialized noise to be added, the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame, the diffusion network model is called to predict the facial key point motion image corresponding to the audio features to be processed in the reference image frame under the current iteration process.

2. The method for generating a moving image of facial key points according to claim 1, characterized in that: Based on a preset window size, extracting the audio feature to be processed from the target audio feature, including: Based on a preset window size, segmenting the target audio feature into a plurality of adjacent audio features; An audio feature to be processed is determined from the plurality of audio features.

3. The method for generating a moving image of facial key points according to claim 1, characterized in that: Acquiring at least one preceding image frame corresponding to the audio feature to be processed, comprising: Determine, according to the arrangement order of the plurality of audio features, a preceding audio feature located before the audio feature to be processed; At least one preceding image frame is obtained from a preceding facial key point motion image corresponding to the preceding audio feature.

4. The method for generating a moving image of facial key points according to claim 1, characterized in that: Based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio feature to be processed, and the previous image frame, calling the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes, including: Obtain the noise to be added, the preset time step value, and the preset number of iterations; Based on the noise to be added, the preset time step value, the facial key point information, the audio feature to be processed, and the previous image frame, calling the diffusion network model to predict the initial facial key point motion image corresponding to the first iteration process; Performing noise processing on the initial facial key point motion image to obtain a noise image after noise addition, and updating the preset time step value to the time step value corresponding to the current iteration process; Based on the noisy image after adding noise, the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame, the diffusion network model is called to predict the facial key point motion image of the reference image frame under the target audio through the iteration process of the preset number of iterations.

5. The method for generating a moving image of facial key points according to claim 1, characterized in that: Based on the noise to be added, the time step value corresponding to the iterative process, the facial key point information, the audio feature to be processed, and the previous image frame, calling the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterative processes, the method further includes: An image rendering operation is performed based on the facial key point motion image to obtain a target video.

6. The method for generating a moving image of facial key points according to claim 1, characterized in that: The method further comprises: Acquire a sample video, and based on the preset window size, intercept a sample sub-video to be processed from the sample video, wherein the sample sub-video to be processed includes a plurality of sample image frames to be processed and a sample audio to be processed; Calling the facial key point prediction model to predict facial key points corresponding to faces in the multiple sample image frames to be processed, and generating target sample image frames; Determine a training time step value, and perform noise processing on the target sample image frame based on the training time step value to obtain a noisy sample image frame; Based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame, calling the initial diffusion network model to predict and obtain a target predicted image frame; Based on the target prediction image frame and the target sample image frame, the initial diffusion network model is trained to obtain a diffusion network model.

7. The method for generating a moving image of facial key points according to claim 6, characterized in that: Before the initial diffusion network model is called to predict and obtain the target predicted image frame based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame, at least one of the following two processing methods is also included: Probabilistically discarding the preceding sample image frame corresponding to the sample sub-video to be processed, and updating the preceding sample image frame corresponding to the sample sub-video to be processed based on the result after probabilistic discarding; The preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed are jointly probability discarded, and the preceding sample image frame, the reference sample image frame, and the sample audio features corresponding to the sample audio to be processed are updated based on the result after the joint probability discarding.

8. The method for generating a moving image of facial key points according to claim 6, characterized in that: Based on the noisy sample image frame, the training time step value, the sample audio to be processed, the previous sample image frame corresponding to the sample sub-video to be processed, and the reference sample image frame, the initial diffusion network model is called to predict and obtain a target predicted image frame, including: Calling the audio encoder in the initial diffusion network model to process the sample audio to be processed to obtain the encoded sample audio features; Calling the input encoder in the initial diffusion network model to process the noisy sample image frame to obtain encoded sample image features; Calling the time step encoder in the initial diffusion network model to process the training time step value to obtain the encoded sample time step feature; Calling the preceding frame encoder in the initial diffusion network model to process the preceding sample image frame to obtain features of the encoded preceding sample image frame; Calling the reference frame encoder in the initial diffusion network model to process the reference sample image frame to obtain encoded reference sample image frame features; Based on the encoded sample image features, the encoded sample time step features, the encoded sample audio features, the encoded previous sample image frame features, and the encoded reference sample image frame features, the initial diffusion network model is called to predict and obtain a target predicted image frame.

9. The method for generating a moving image of facial key points according to claim 8, characterized in that: Calling the audio encoder in the initial diffusion network model to process the sample audio to be processed to obtain the encoded sample audio features, including: Calling the audio feature model to extract sample audio features of the sample audio to be processed; The sample audio features are input into the audio encoder in the initial diffusion network model for encoding to obtain encoded audio features.

10. The method for generating a moving image of facial key points according to claim 8, characterized in that: Based on the encoded sample image feature, the encoded sample time step feature, the encoded sample audio feature, the encoded previous sample image frame feature, and the encoded reference sample image frame feature, calling the initial diffusion network model to predict and obtain a target predicted image frame, including: Performing feature splicing on the encoded sample image feature, the encoded sample time step feature, the encoded sample audio feature, the encoded preceding sample image frame feature, and the encoded reference sample image frame feature to obtain a spliced ​​feature; The spliced ​​features are input into the self-attention sub-network of the initial diffusion network model, and the self-attention sub-network is called to predict and obtain a target predicted image frame.

11. The method for generating a moving image of facial key points according to claim 6, characterized in that: Based on the target prediction image frame and the target sample image frame, the initial diffusion network model is trained to obtain a diffusion network model, including: Constructing a target loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame; The initial diffusion network model is trained by minimizing the objective loss function to obtain a diffusion network model.

12. The method for generating a moving image of facial key points according to claim 11, characterized in that: Constructing a target loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame, including: Constructing a smoothing sub-loss function based on the difference between adjacent image frames of the target prediction image frame; Constructing a distance difference sub-loss function based on the Euclidean distance between the target prediction image frame and the target sample image frame; Based on the smoothing sub-loss function and the distance difference sub-loss function, a target loss function is constructed.

13. A device for generating a moving image of facial key points, characterized in that: include: A first acquisition unit is used to acquire a reference image frame and call a facial key point prediction model to extract facial key point information corresponding to a face in the reference image frame; A second acquisition unit is used to acquire target audio and call an audio feature model to extract target audio features of the target audio; A capture unit, configured to capture the audio feature to be processed from the target audio feature based on a preset window size, and obtain at least one preceding image frame corresponding to the audio feature to be processed, wherein the preceding image frame is an image frame used to provide preceding information for generating a motion image of facial key points; The first prediction unit is used to call the diffusion network model to predict the facial key point motion image of the reference image frame under the target audio through multiple iterations based on the noise to be added, the time step value corresponding to the iteration process, the facial key point information, the audio feature to be processed, and the previous image frame, including: Based on a current iteration process in the multiple iteration processes, obtaining randomly initialized noise to be added; Based on the randomly initialized noise to be added, the time step value corresponding to the current iteration process, the facial key point information, the audio features to be processed, and the previous image frame, the diffusion network model is called to predict the facial key point motion image corresponding to the audio features to be processed in the reference image frame under the current iteration process.

14. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the operations in the method for generating a motion image of facial key points according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the method for generating a motion image of facial key points according to any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps in the method for generating a moving image of facial key points described in any one of claims 1 to 12 are implemented.

Images