Image generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide an image generation method and apparatus, an electronic device, and a storage medium. The image generation method relates to the field of image generation in the field of image processing. The image generation method comprises: upon acquiring an input image and a reference image, first encoding the input image and the reference image to obtain an input feature of the input image and a reference feature of the reference image; and then decoding the input feature and the reference feature to obtain a final output image. The image generation method can not only improve the image generation performance, but also improve the generalization ability of an image generation process and reduce the complexity and time cost of the image generation process.

Description

Image generation method, device, electronic device, and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 26, 2023, with application number 202311403638.5 and application name “Image Generation Method, Device, Electronic Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image generation in the field of image processing, and specifically, to an image generation method, apparatus, electronic device, and storage medium. Background Art

[0003] In the products of Intelligent Small Sample Digital Intelligence, the input audio provided by the user can be used to re-drive the lip shape of the target object (such as a person or animal) in the input video to the lip shape of the input audio, thereby obtaining the output video.

[0004] Typically, before using input audio to drive the lip movements of a target object in an input video, an image generation model must be trained based on the input video. During training, the image generation model uses the audio-driven rendering of the input video as input and produces a high-definition image of the target object. This allows the model to learn the features necessary to generate a high-definition image from the rendering. During prediction, the image generation model uses the audio-driven rendering as input and produces a high-definition image of the target object as output.

[0005] However, the image generation model trained using the input video is not a universal model, which reduces the generalization ability of the image generation process. In other words, after the image generation model is trained based on the input video, it can only generate renderings and high-definition images based on the input video. If the user changes the input video, the image generation model needs to be retrained based on the changed input video.

[0006] In addition, when training the image generation model, the model has requirements for the length of the input video, which is generally required to be more than 3 minutes and the input video must be a speaking video. In addition, when training the image generation model, the user needs to wait for 0.5 to 1 day, which increases the complexity and time cost of the image generation process.

[0007] Summary of the Invention

[0008] The present application provides an image generation method, device, electronic device and storage medium, which can improve the generalization ability of the image generation process and reduce the complexity and time cost of the image generation process on the basis of improving image generation performance.

[0009] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0010] Get input video and input audio;

[0011] Obtain a rendering that matches the facial contour of the target object in the input video;

[0012] Based on the audio unit in the input audio, adjusting the mouth shape of the target object in the rendering image to a mouth shape that matches the audio unit, to obtain an input image;

[0013] Sampling the input video to obtain an image sequence, and determining an image driven by the audio unit in the image sequence;

[0014] Determining a reference image of the input image based on the image driven by the audio unit;

[0015] Encoding the input image and the reference image to obtain input features of the input image and reference features of the reference image;

[0016] The input feature and the reference feature are decoded to obtain an output image.

[0017] In a second aspect, an embodiment of the present application provides an electronic device for executing the method of the first aspect or its respective implementations. Specifically, the electronic device includes a functional module for executing the method of the first aspect or its respective implementations.

[0018] In one implementation, the electronic device may include a processing unit configured to execute functions related to information processing. For example, the processing unit may be a processor.

[0019] In one implementation, the electronic device may include a transmitting unit and / or a receiving unit. The transmitting unit is configured to perform functions related to transmission, and the receiving unit is configured to perform functions related to reception. For example, the transmitting unit may be a transmitter or emitter, and the receiving unit may be a receiver or receiver. For another example, if the electronic device is a communication chip, the transmitting unit may be an input circuit or interface of the communication chip, and the transmitting unit may be an output circuit or interface of the communication chip.

[0020] In a third aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to perform the method of the first aspect or its respective implementations.

[0021] In one implementation, there are one or more processors and one or more memories.

[0022] In one implementation, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0023] In one implementation, the electronic device further includes a transmitter (transmitter) and a receiver (receiver).

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that it is used to store a computer program, which enables a computer to execute the method of the first aspect above.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, it implements the method of executing the above-mentioned first aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects above or in each of their implementations.

[0027] Based on the above technical solution, after obtaining the input image and the reference image based on the input video and the input audio, the embodiment of the present application can first encode the input image and the reference image, and obtain the input features of the input image and the reference features of the reference image, and then decode the input features and the reference features to obtain the final output image.

[0028] On the one hand, by introducing a reference image and decoding the input feature and the reference feature together during the decoding process, the decoded output image can have both the input feature and the reference feature. Furthermore, since the reference feature is a feature obtained by encoding the reference image, and the reference image is an image determined by the image driven by the audio unit in the image sequence obtained by sampling the input video, even if the input video is not used to train the encoding and decoding process, the reference feature can make the output image have the features of the image driven by the audio unit. This is equivalent to directly using the features of the image driven by the audio unit as the features required to generate the output image based on the input image in the image generation process, which can improve the image generation performance of the output image.

[0029] On the other hand, directly using the features of the image driven by the audio unit as the features required to generate the output image based on the input image can avoid using the input video to train the encoding and decoding process, and there is no need to have requirements on the length of the input video. A picture can also be used to generate a rendering image to an output image, which not only improves the generalization ability of the image generation process, but also avoids the image generation process's requirements for the input video and the training waiting time, thereby reducing the complexity and time cost of the image generation process.

[0030] In summary, the present application provides an image generation method, device, electronic device and storage medium, which, on the basis of improving image generation performance, can improve the generalization ability of the image generation process and reduce the complexity and time cost of the image generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is an example of a system framework provided by an embodiment of the present application.

[0032] FIG2 is a schematic flow chart of an image generation method provided in an embodiment of the present application.

[0033] FIG3 is an example of a process of generating an output image from an audio-driven rendering provided in an embodiment of the present application.

[0034] FIG4 is an example of the structure of the first encoder provided in an embodiment of the present application.

[0035] Figure 5 is an example of the structure of the first self-attention block provided in an embodiment of the present application.

[0036] FIG6 is another schematic flowchart of the image generation method provided in an embodiment of the present application.

[0037] FIG7 is an example of the structure of a decoder provided in an embodiment of the present application.

[0038] Figure 8 is an example of the structure of the first cross-attention block provided by an embodiment of the present application.

[0039] FIG9 is a schematic block diagram of an image generating device provided in an embodiment of the present application.

[0040] FIG10 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] The solution provided in this application may involve artificial intelligence technology, more specifically computer vision (CV) technology and machine learning (ML) technology in artificial intelligence technology.

[0043] To facilitate understanding of the technical solutions provided by this application, the following describes the terms related to the embodiments of this application.

[0044] Feature: refers to the information extracted from the data that is useful for predicting the results. It can be text or data.

[0045] Feature engineering refers to the process of transforming raw data into features that better represent the essence of the problem, so that applying these features to a predictive model can improve the model's prediction accuracy for unseen data. Simply put, features that significantly influence the dependent variable y are called independent variables x, and independent variables x are called features. The goal of feature engineering is to discover these features. Because good features offer greater flexibility, they can be trained with simpler models and produce superior results. The goal of feature engineering is to identify better features and obtain better training data. Feature engineering plays a crucial role in the success of machine learning. Many machine learning algorithms achieve success by building a learner that can understand engineered features.

[0046] Of course, the specific form of the raw data is not limited in the embodiments of the present application. As an example, the raw data can be constructed as tabular data to obtain a feature matrix, based on which Principal Component Analysis (PCA) can be used to extract features to create new features for characterizing the audio.

[0047] Feature engineering can include modules such as feature extraction, feature construction, and feature selection.

[0048] Feature extraction: Converting raw data into a set of features with distinct physical or statistical meaning or kernel. The object of feature extraction is the raw data, i.e., the original features, which are used to construct new features, i.e., converting the raw data into a set of features with distinct physical or statistical meaning.

[0049] Feature construction involves artificially constructing new features from raw data. Specifically, real data samples are used to determine the potential form and data structure of the new features from the perspective of the prediction target, allowing for better application in the prediction model. Feature construction requires strong insight and analytical skills, requiring the ability to identify physically meaningful features from the raw data. Assuming the raw data is tabular, new features can be created by using mixed or combined attributes, or by decomposing or segmenting existing features.

[0050] Feature selection: Select a set of statistically significant feature subsets from the feature set to achieve the effect of reducing dimensionality. For feature selection, you can rank based on feature importance, and then perform feature selection based on the feature ranking results. Assume that there is a standard tabular data, each row of the tabular data (i.e., category item) represents an observation sample data, and each column in the tabular data (feature item) is a feature. Among these features, some features carry a lot of information, while others carry very little information. Features that carry very little information belong to irrelevant data. The feature importance can be measured by the correlation between the feature item and the category item (feature importance), and then feature selection is performed based on the feature ranking results.

[0051] Derived features: These are new features derived from feature learning using raw data. Derived features generally arise from two factors: changes in the data itself, which lead to the emergence of features that were not originally present; or, during feature learning, the algorithm generates derived features based on certain relationships between features. Derived features can sometimes better reflect these relationships. Derived features also require machine learning and deep learning algorithms to possess stronger learning capabilities, namely, incremental learning, online learning, and transfer learning. In computer science, derived features are derived from feature learning using raw data. In machine learning, derived features arise from a variety of factors: data has temporal properties and strong temporal characteristics; data is dynamic, disordered, infinite, and emergent. Regarding feature transformation, data features are categorized as continuous and categorical, also known as discrete features. Different data types require different processing methods. For example, normalization is a common method for processing continuous data features. Its purpose is to quickly find the optimal solution through gradient descent. Feature normalization can be divided into linear and nonlinear normalization.

[0052] It should be understood that the embodiments of the present application do not limit the specific processes involved in feature engineering. As an example, the processes involved in feature engineering include but are not limited to: binning, one-hot encoding, feature hashing, embedding, log transformation, feature scaling, normalization, or feature interaction. Of course, other processes or steps may also be included, and this application does not limit them.

[0053] In addition, the embodiments of the present application do not specifically limit the model used by the encoder, the model used by the decoder, and the tooth matching model.

[0054] As an example, the prediction model or classification model includes, but is not limited to, a traditional learning model, an ensemble learning model, or a deep learning model. Optionally, the traditional learning model includes, but is not limited to, a tree model (regression tree) or a logistic regression (LR) model; the ensemble learning model includes, but is not limited to, an improved model of the gradient boosting algorithm (XGBoost) or a random forest model; and the deep learning model includes, but is not limited to, a long short-term memory (LSTM) network or a neural network.

[0055] It should be noted that the terms used in the implementation method part of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0056] For example, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" is only a description of the combination relationship of enumerated objects, indicating that one or more items may exist. For example, at least one of the following: A, B, C can mean the following combinations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time. The term "multiple" refers to two or more. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0057] For example, the term "corresponding" can indicate a direct or indirect correspondence between two items, an association between the two items, or a relationship between an indicator and the indicated, a configuration and the configured, and so on. The term "indication" can be a direct indication, an indirect indication, or an association. For example, "A indicates B" can mean that A directly indicates B, for example, B can obtain it through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can obtain it through C; it can also mean that A and B have an association. The terms "predefined" or "preconfigured" can mean that the corresponding code, table, or other relevant information that can be used for indication is pre-stored in the device, or it can refer to a protocol agreement. "Protocol" can refer to a standard protocol in this field. The term "when..." can be interpreted as "if," "if," "when," "in response to," and similar descriptions. Similarly, depending on the context, the phrase "if determined" or "if (stated condition or event) is detected" can be interpreted as "when determined" or "in response to determining" or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)" and similar descriptions. The terms "first", "second", "third", "fourth", "Ath", "Bth" and the like are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. Among them, digital video compression technology is mainly used to compress huge digital image video data for easy transmission and storage.

[0058] The following describes the application scenarios, technical problems to be solved, and inventive concepts of the embodiments of the present application.

[0059] FIG1 is an example of a system framework 100 provided in an embodiment of the present application.

[0060] As shown in FIG1 , the system framework 100 can be an application system. The specific type of application is not limited in this embodiment of the present application. The system framework 100 includes: a terminal 131, a terminal 132, and a server cluster 110. Both the terminal 131 and the terminal 132 can be connected to the server cluster 110 via a wireless or wired network 120.

[0061] Terminals 131 and 132 may be at least one of a smartphone, a game console, a desktop computer, a tablet computer, an e-book reader, an MP4 player, an MP4 player, and a laptop computer. Terminals 131 and 132 may be clients, each having an application installed and running. The application may be any intelligent small sample digital human product capable of image generation, for example, an online video program, a short video program, an offline video program, a picture sharing program, a voice social program, a comic program, a wallpaper program, a news push program, a supply and demand information push program, an academic exchange program, a technical exchange program, a policy exchange program, a program with a comment mechanism, a program with a point of view publishing mechanism, or a knowledge sharing program. Terminals 131 and 132 may be the terminals used by users 141 and 142, respectively, and registered accounts are logged into the applications running on terminals 131 and 132.

[0062] Server cluster 110 includes at least one of a single server, multiple servers, a cloud platform, and a virtualization center. Server cluster 110 provides backend services for applications (e.g., applications on terminals 131 and 132). Optionally, server cluster 110 performs primary computing tasks, while terminals 131 and 132 perform secondary computing tasks. Alternatively, server cluster 110 performs secondary computing tasks, while terminals 131 and 132 perform primary computing tasks. Alternatively, terminals 131 and 132 collaborate with server cluster 110 using a distributed computing architecture.

[0063] Taking the system framework 100 as a web browsing system as an example, the server cluster 110 includes: an access server 112, a web server 111 and a data server 113. There can be one or more access servers 112, and the access servers 112 can be deployed nearby in different cities. The access server 112 is used to receive service requests from terminals 131 and 132, and forward the service requests to the corresponding servers for processing. The web server 111 is a server for providing web pages to terminals 131 and 132, and the web pages are integrated with embedded code. The data server 113 is used to receive data reported by terminals 131 and 132 (such as business data, etc.).

[0064] It should be noted that Figure 1 shows an example of a system architecture used in the embodiment of the present application, and the system architecture applicable to the embodiment of the present application is not limited thereto. For example, the system architecture 100 may include one of the terminal 131 and the terminal 132.

[0065] In actual applications, taking terminal 131 as an example, the intelligent small sample digital human product in terminal 131 can send input audio and input video to the data server 113, and send a service request to the access server 112. The service request is used to request to use the input audio received by the data server 113 to re-drive the lip shape of the target object (such as a person or animal) in the input video to the lip shape of the input audio, thereby obtaining the output video, and sending the output video to the terminal 113 through the web server 111 so that the terminal 113 can display the output video.

[0066] Intelligent small-sample digital intelligence products can be applied to various fields. For example, in the short video field, a live streamer can use an audio clip of a product B to re-engineer the live streamer's lip movements in a video of a product A to match the audio clip, thereby improving work efficiency. Another example is that in the education field, a lecturer can use an audio clip of a lecture on knowledge point B to re-engineer a lecture video on knowledge point A and the lecturer's lip movements to match the audio clip, thereby improving work efficiency.

[0067] Typically, before using input audio to drive the lip movements of a target object in an input video, an image generation model must be trained based on the input video. During training, the image generation model uses the audio-driven rendering of the input video as input and produces a high-definition image of the target object. This allows the model to learn the features necessary to generate a high-definition image from the rendering. During prediction, the image generation model uses the audio-driven rendering as input and produces a high-definition image of the target object as output.

[0068] However, the image generation model trained using this input video is not a universal model, which reduces the generalization ability of the image generation process. Specifically, after the image generation model is trained on an input video, it can only generate high-definition images from a rendered image based on that input video. If the user changes the input video, the image generation model must be retrained based on the changed input video.

[0069] In addition, when training the image generation model, the model has requirements for the length of the input video, which is generally required to be more than 3 minutes and the input video must be a speaking video. In addition, when training the image generation model, the user needs to wait for 0.5 to 1 day, which increases the complexity and time cost of the image generation process.

[0070] In view of this, the present application provides an image generation method, apparatus, electronic device, and storage medium, which, on the basis of improving image generation performance, can enhance the generalization capability of the image generation process and reduce the complexity and time cost of the image generation process.

[0071] Figure 2 shows a schematic flow chart of an image generation method 200 according to an embodiment of the present application. This image generation method 200 can be executed by any electronic device with data processing capabilities. For example, the electronic device can be implemented as terminal 131 or terminal 132, or server cluster 110, or data server 113 within server cluster 110, as shown in Figure 1. This application does not impose any limitations here. For ease of description, the following description of image generation method 200 uses an image generation device as an example execution subject.

[0072] The image generation method 200 can be applied to digital human scenarios. For example, it can be applied to scenarios where digital humans are driven to speak based on text or audio. Specifically, the user provides a video of himself (he can speak or keep his mouth shut throughout, that is, the audio information of the video is not required), and there is no need to train the video provided by the user. Then, the user can provide any speaking audio, which can be synthesized through text or the user's own audio. Using the audio provided by the user, the lip shape and facial expression of the character in the video are re-driven, so that the lip shape and expression of the character in the generated video can correspond to the new audio provided by the user.

[0073] As shown in FIG2 , the image generation method 200 may include:

[0074] S210: Obtain input video and input audio.

[0075] For example, the image generation device can obtain input video and input audio. Exemplarily, the input video can be a video containing a target object, and the target object can be any object with a mouth. For example, the target object can be a person, an animal, a cartoon character, or even other objects, scenery, or buildings designed with mouth special effects. Exemplarily, the input video can be a video that includes audio or does not include audio. That is, the driving-in video can be a video of the target object speaking, or a video of the target object not speaking, and this application does not specifically limit this.

[0076] Exemplarily, the input video may be a video file, and this application does not specifically limit the format of the video file. For example, the video file may be MP4. MP4 format video has a smaller file size and a higher compression ratio, and is commonly used for sharing videos on the Internet and online video streaming. The video file may be AVI. AVI format video has wide compatibility and can be played on many different devices and operating systems. The video file may be MKV. MKV is an open multimedia container format that can contain multiple audio, video, and subtitle streams, and is commonly used to store high-definition video and Blu-ray movies. The video file may be MOV. MOV format video has higher quality and compatibility. The video file may be WMV. WMV format video is usually smaller, has a higher compression ratio, and lower quality. In addition, there are other formats such as AVCHD, BD, DivX, VCD, etc., and the specific choice can be made according to the usage scenario and device.

[0077] Exemplarily, the audio file may be audio generated based on text, or may be original sound recorded by a sound recording device.

[0078] Exemplarily, the input audio may be an audio file, and the present application does not limit the format of the audio file. For example, the audio file may be a WAVE file. A WAVE file is a digital sound format for recording sound waveforms, and high-quality sound files can be obtained through high sampling rates, high sampling bytes, and fast computers. The audio file may be a MOD file. A MOD file is a music file format, typically used to save music and sound effects, and can be used by MOD fans and some game programs. The audio file may be an RA file. An RA file is an audio stream file format, typically used for real-time transmission of audio information over a low-rate wide area network, such as in voice calls. There are also other audio file formats, such as AU, VOC, MPEG-3, and Real Audio, etc. Each format has its own unique characteristics and application scenarios, and the specific choice can be made according to the usage scenario and device.

[0079] S220: Obtain a rendering image that matches the facial contour of the target object in the input video.

[0080] For example, the image generation device can obtain a rendering that matches the facial contours of the target object in the input video. Exemplarily, this rendering refers to rendering the target object model into an image using rendering software to simulate the facial contours of the target object in the real world. Renderings typically use techniques such as light identification and radiosity to simulate real-world lighting effects and thus simulate the facial contours of the target object in the real world. Furthermore, renderings can also use techniques such as color correction and texture mapping to simulate real-world characteristics such as the target object's clothing color and texture.

[0081] For example, the rendering may be an image rendered by using 3D rendering software to render the target object model.

[0082] S230: Based on the audio unit in the input audio, adjust the mouth shape of the target object in the rendering image to a mouth shape that matches the audio unit to obtain an input image.

[0083] For example, the image generating device adjusts the mouth shape of the target object in the rendering image to a mouth shape that matches the audio unit in the input audio, and obtains the input image. Exemplarily, the audio unit can be a phoneme, which refers to the smallest unit of speech. The input audio is a continuous speech stream composed of a series of phonemes, which can be used to represent the speech characteristics of the target object. Phonemes can be used to represent various features in speech, such as the frequency, duration, pitch, volume, etc. of a sound. Phonemes are usually formed by a pronunciation action, and different pronunciation actions produce different phonemes.

[0084] Exemplarily, the audio unit may be a pronounced word, which refers to a specific language symbol consisting of a group of phonemes and can be used to express a specific language meaning.

[0085] Of course, in other alternative embodiments, the audio unit may be a speech unit of other types or forms, which is not specifically limited in this application.

[0086] It should be understood that the present application does not specifically limit the manner in which the image generation device adjusts the lip shape of the target object in the rendering image to a lip shape that matches the audio unit. For example, the lip shape of the target object in the rendering image can be adjusted to a lip shape that matches the audio unit using various rendering software. The lip shape adjustment of the target object can be achieved by setting the shape and texture of the model in the rendering software.

[0087] Exemplarily, when the rendering may be a 3D model of the facial contour of the target object, the image generating device may drive the 3D model according to the input audio to obtain a 3D rendering corresponding to the lip shape of the input audio, thereby obtaining the input image.

[0088] S240: Sampling the input video to obtain an image sequence, and determining an image driven by the audio unit in the image sequence.

[0089] For example, the image generation device samples the input video to obtain an image sequence, and determines the image driven by the audio unit in the image sequence. Exemplarily, the image generation device can sample the input video and obtain the image sequence by extracting image frames.

[0090] For example, the image generation device determines the image driven by the audio unit in the image sequence based on a predefined determination criterion. This application does not limit the specific implementation of the determination criterion. This determination criterion can be information pre-stored on the image generation device, information obtained from other devices, or information input by the user.

[0091] S250: Determine a reference image of the input image based on the image driven by the audio unit.

[0092] For example, the image generation device determines a reference image for the input image based on the image driven by the audio unit. Exemplarily, the image generation device may directly determine the image driven by the audio unit as the reference image, process the image driven by the audio unit and determine the processed image as the reference image, or even determine the reference image in the image sequence based on the image driven by the audio unit. This application does not specifically limit this.

[0093] S260: Encode the input image and the reference image to obtain input features of the input image and reference features of the reference image.

[0094] For example, the image generation device encodes the input image and the reference image to obtain input features of the input image and reference features of the reference image. Exemplarily, the image generation device may perform feature extraction, feature construction, feature selection, and other processing on the input image and the reference image, thereby achieving the purpose of encoding the input image and the reference image and obtaining the input features and the reference features. Feature extraction, feature construction, feature selection, and other processing may be performed by relevant technical means involved in feature engineering or neural networks, and this application does not specifically limit this.

[0095] Exemplarily, the image generating device can use an encoder to encode the input image and the reference image, and before encoding, the encoding process of the encoder can be pre-trained with a large amount of data, and then the input image and the reference image can be encoded using the trained encoder to obtain the output feature and the reference feature.

[0096] It is worth noting that after training based on a large amount of training data, for a new input video, there is no need to retrain the encoder. The input image and the reference image can be directly encoded to obtain the output feature and the reference feature.

[0097] S270: Decode the input feature and the reference feature to obtain an output image.

[0098] For example, the image generation device decodes the input features and the reference features to obtain an output image. Exemplarily, the image generation device may use a decoder to decode the input features and the reference features. Prior to decoding, the decoder may be pre-trained with a large amount of data to perform a decoding process. The decoder may then be pre-trained with the trained decoder to decode the input image and the reference image to obtain the output image.

[0099] It is worth noting that after training based on a large amount of training data, for a new input video, there is no need to retrain the decoding process. Decoding can be performed directly based on the input features and the reference features to obtain the output image. In addition, the encoder involved in step S260 and the decoder involved in step S270 can be trained jointly or separately, and this application does not make specific restrictions on this.

[0100] In this embodiment, the input image is an image obtained by adjusting the lip shape of the target object in the rendering image to a lip shape that matches the audio unit, and the reference image is an image determined based on the image driven by the audio unit, and the image driven by the audio unit is an image selected from the image sequence. Therefore, the decoding process involved can be understood as the image generating device obtains the output image by adjusting the lip shape of the target object in the reference feature based on the input feature, and makes the lip shape of the target object in the output image match the audio unit.

[0101] In this embodiment, after obtaining the input image and the reference image, the image generating device can first encode the input image and the reference image and obtain the input features of the input image and the reference features of the reference image, and then decode the input features and the reference features to obtain the final output image.

[0102] On the one hand, by introducing a reference image and decoding the input feature and the reference feature together during the decoding process, the decoded output image can have both the input feature and the reference feature. Furthermore, since the reference feature is a feature obtained by encoding the reference image, and the reference image is an image determined by the image driven by the audio unit in the image sequence obtained by sampling the input video, even if the input video is not used to train the encoding and decoding process, the reference feature can make the output image have the features of the image driven by the audio unit. This is equivalent to directly using the features of the image driven by the audio unit as the features required to generate the output image based on the input image in the image generation process, which can improve the image generation performance of the output image.

[0103] On the other hand, directly using the features of the image driven by the audio unit as the features required to generate the output image based on the input image can avoid using the input video to train the encoding and decoding process, and there is no need to have requirements on the length of the input video. A picture can also be used to generate a rendering image to an output image, which not only improves the generalization ability of the image generation process, but also avoids the image generation process's requirements for the input video and the training waiting time, thereby reducing the complexity and time cost of the image generation process.

[0104] In summary, the present application provides an image generation method, device, electronic device and storage medium, which, on the basis of improving image generation performance, can improve the generalization ability of the image generation process and reduce the complexity and time cost of the image generation process.

[0105] It is worth noting that when the image generation device only encodes the input image to obtain input features, and then directly decodes the input features to obtain the output image, its encoding and decoding processes can be implemented through an image generation model.

[0106] However, because the image generation model needs to learn the features required to generate a high-definition image based on the rendered image, it is necessary to train the model on the input video before using it to generate the output image based on the input image. Specifically, the image generation model needs to use the rendered image driven by the audio of the input video as input and generate a high-definition image of the target object. Therefore, during the prediction process, the image generation model can use the input image as input and generate and output the output image.

[0107] FIG3 is an example of a process of generating an output image from an audio-driven rendering provided in an embodiment of the present application.

[0108] As shown in Figure 3, after obtaining the audio-driven rendering, the data construction process must be performed on the audio-driven rendering to obtain a constructed image. The constructed image is then used as input to generate the output image using the image generation model. This is because it is difficult to generate a realistic high-definition portrait directly from the audio-driven rendering. Therefore, partial information from the original image (such as a reference image) can be used to assist in image generation.

[0109] For example, during the data construction process, facial key point technology can be used to first identify the target area in the audio-driven rendering image, which may include the nose and cheeks of the person in the image (taking the 256-point keyword as an example, the area below the line connecting the three points with indices 83, 177, and 211 is considered the lower half of the face). The target area is then used to replace the corresponding area in the original image to obtain a constructed image. Since the target area may not cover the entire cheeks and neck of the person in the original image, that is, some of the neck and cheek areas of the person in the original image may be exposed, affecting the learning of the image generation model, the dilation operation can be used. The dilation operation is a basic operation in morphological processing. The corresponding area in the original image can be expanded to form a black border, covering the original cheek and neck areas. When the constructed image obtained in this way is used as the input image of the image generation model, the result generated by the image generation model will not be affected by the image of the target person's cheeks and neck when speaking in the input video, thereby improving the stability of the effect.

[0110] This shows that the image generation model is only trained and fitted for a single video. If you change the video, even if it is a different video of the same person, but due to the different background, the image generation model cannot be used for inference. The image generation model needs to be retrained using the new video as training data.

[0111] However, in this embodiment, during the image generation process, the features of the image driven by the audio unit can be directly used as the features required to generate the output image based on the input image, which can improve the image generation performance of the output image. This avoids the problem of retraining the encoding and decoding processes based on the input video.

[0112] It is worth noting that in an embodiment of the present application, in step S230, the image generating device may also directly determine the image obtained by adjusting the lip shape of the target object in the rendering image to a lip shape that matches the audio unit based on the audio unit as the input image, or, the image obtained by adjusting the lip shape of the target object in the rendering image to a lip shape that matches the audio unit based on the audio unit may be used as the audio-driven rendering image as shown in Figure 3, and the data construction process as shown in Figure 3 may be performed on the audio-driven rendering image, and then the obtained constructed image may be used as the input image. This application does not make any specific limitations on this.

[0113] In some embodiments, S240 may include:

[0114] When the number of images in the image sequence is equal to the number of units included in the input audio, the image in the image sequence whose sampling number is the same as the position number of the audio unit in the input audio is determined as the image driven by the audio unit.

[0115] Exemplarily, if the number of images in the image sequence is equal to the number of units included in the input audio, the image generation device determines the image in the image sequence whose sampling number is the same as the position number of the audio unit in the input audio as the image driven by the audio unit. Furthermore, if the number of images in the image sequence is greater than the number of units included in the input audio, the image sequence is first downsampled to obtain a downsampled sequence, and then the image in the downsampled sequence whose sampling number is the same as the position number of the audio unit in the input audio is determined as the image driven by the audio unit. The number of images in the downsampled sequence is equal to the number of units included in the input audio. If the number of images in the image sequence is less than the number of units included in the input audio, the image sequence is first upsampled to obtain an upsampled sequence, and then the image generation device determines the image in the upsampled sequence whose sampling number is the same as the position number of the audio unit in the input audio as the image driven by the audio unit. The number of images in the upsampled sequence is equal to the number of units included in the input audio.

[0116] In this embodiment, the image whose sampling number in the image sequence is the same as the position number of the audio unit in the input audio is determined as the image driven by the audio unit, which can ensure that the video formed by the output image and the picture of the input video remain synchronized, thereby improving the picture effect of the video formed by the output image.

[0117] Of course, the sampling sequence number of the image in the image sequence is similar to the timestamp of the image, that is, the above-mentioned determination of the image driven by the audio unit is only a specific example of the image generating device determining the image determined by the audio unit according to the sequence number of the image in the image sequence (that is, the timestamp), and it should not be understood as a limitation to the present application.

[0118] For example, when the image generation device determines the image driven by the audio unit in the image sequence, as one possible implementation, the image generation device may select the image in the image sequence that best meets the conditions or requirements as the image driven by the audio unit. For example, the image in which the amplitude of the target object's motion is smaller than that of the baseline image may be selected as the image that best meets the conditions or requirements. Alternatively, the image generation device may select based on the attributes of the images in the image sequence. For example, the image generation device may select based on features such as color, shape, and texture, that is, images in the image sequence that meet preset attributes may be selected as the images driven by the audio unit. Alternatively, the image generation device may select based on the size and proportion of the target object in each image in the image sequence. For example, images in the image sequence in which the target object's size is larger than a preset size may be selected as the images driven by the audio unit. Alternatively, the image generation device may select based on the timestamps of the images in the image sequence. For example, images with timestamps matching the audio unit may be selected as the images driven by the audio unit. Alternatively, the image generation device may select based on the quality of the images in the image sequence. For example, images whose clarity and quality parameters are greater than a preset threshold may be selected as the images driven by the audio unit. In summary, the image generating device can determine the image driven by the audio unit in the image sequence based on specific needs and purposes.

[0119] In some embodiments, the S250 may include:

[0120] The image driven by the audio unit is determined as the reference image.

[0121] For example, the image generating device determines the image driven by the audio unit as the reference image. In this embodiment, determining the image driven by the audio unit as the reference image can ensure that the video formed by the output image and the picture of the input video remain synchronized, thereby improving the picture effect of the video formed by the output image. This is equivalent to when the image generating device generates an output image based on the input image and forms or generates an output video based on the output image, the features of the reference image can be aligned to the greatest extent with the features required for the output video and the input video to remain synchronized, thereby improving the picture effect of the output video.

[0122] In some embodiments, the S250 may include:

[0123] Taking the image driven by the audio unit as the center, a plurality of images are first selected forward and backward in the image sequence according to the sampling time sequence; then, the reference image is selected from the plurality of images.

[0124] For example, the image generating device takes the image driven by the audio unit as the center, firstly selects a plurality of images in the image sequence in a forward and backward order according to the sampling time; and then selects the reference image from the plurality of images.

[0125] Exemplarily, the plurality of images may include an image driven by the audio unit, or may not include an image driven by the audio unit.

[0126] Exemplarily, the number of the plurality of images may be a preset value, which may be information pre-stored in the image generating device, or information agreed upon through a protocol, or information input by a user.

[0127] For example, when the image generation device determines a reference image from the multiple images, as one possible implementation, the image generation device may select the image that best meets the conditions or requirements as the reference image. For example, the image in which the target object's motion amplitude is smaller than the reference image may be selected as the image that best meets the conditions or requirements. Alternatively, the image generation device may select based on image attributes from the multiple images. For example, the image generation device may select based on features such as color, shape, and texture, i.e., the image from the multiple images that meets preset attributes may be selected as the reference image. Alternatively, the image generation device may select based on the size and proportion of the target object in each of the multiple images. For example, the image in which the target object's size is larger than a preset size may be selected as the reference image. Alternatively, the image generation device may select based on timestamps, etc. from the multiple images. For example, the image whose timestamp matches the audio unit may be selected as the reference image. Alternatively, the image generation device may select based on image quality from the multiple images. For example, the image whose clarity or quality parameter exceeds a preset threshold may be selected as the reference image. In summary, the image generating device can determine a reference image from the multiple images based on specific needs and purposes.

[0128] In this embodiment, with the image driven by the audio unit as the center, the image generating device first selects multiple images in the image sequence forward and backward according to the sampling time order; then, the reference image is selected from the multiple images, which is beneficial for the image generating device to determine the reference image from the multiple images based on specific needs and purposes, and thus enable the output image to be an image that meets various needs or purposes, that is, the flexibility of the output image is improved.

[0129] In some embodiments, the input image is encoded by a first encoder, the first encoder including a residual block, a downsampling layer, and a first self-attention block; wherein S260 may include:

[0130] The image generating device first uses the residual block to extract features of the input image to obtain a first coding feature; then, uses the downsampling layer to downsample the first coding feature to obtain a second coding feature; based on the second coding feature, determines the third coding feature input to the first self-attention block; based on the third coding feature, determines the Q value, K value and V value of the first self-attention block, and based on the Q value, K value and V value of the first self-attention block, determines the fourth coding feature output by the first self-attention block; based on the fourth coding feature, determines the input feature.

[0131] For example, the residual block is a basic unit in a neural network. By introducing skip connections, it can effectively transfer gradients, alleviate the problems of vanishing and exploding gradients, and improve network training performance. A residual block typically consists of two or more convolutional layers. Each convolutional layer convolves the input with an appropriate convolution kernel and performs a nonlinear transformation using an activation function. A skip connection is added between convolutional layers, directly adding the input of that layer to the output. This skip connection can be a simple element-wise addition operation or a convolution operation that transforms the input before adding it. Finally, the result of the addition is transformed using an activation function to obtain the output of the residual block.

[0132] For example, in a convolutional neural network, the downsampling layer, also called the pooling layer, operates essentially the same as the convolution layer: the image within an s*s window is treated as a pixel, and the value of this pixel is the mean of all pixels within the window. However, the downsampling layer does not undergo backpropagation modifications. The pooling layer can also be viewed as a special convolution operation, where the step size setting corresponds to the distance. During a convolution operation, it is generally believed that closely spaced parts of an image are more correlated, while more distant parts are less correlated.

[0133] Exemplarily, when the image generating device determines the third coding feature input to the first self-attention block based on the second coding feature, the second coding feature can be directly determined as the third coding feature, or the third coding feature can be obtained after feature processing of the second coding feature. For example, the image generating device can perform feature extraction, feature construction, feature selection and other processing on the second coding feature to obtain the third coding feature. Among them, the feature extraction, feature construction, feature selection and other processing can be carried out by relevant technical means involved in feature engineering or neural networks, and this application does not make specific limitations on this. For example, the image generating device can use a residual block to extract the second coding feature, and use a downsampling layer to downsample the feature obtained by feature extraction, and then use a residual block to feature extract the feature obtained by downsampling again, and determine the feature obtained by feature extraction as the third coding feature.

[0134] Exemplarily, when the image generation device determines the Q value, K value, and V value of the first self-attention block based on the third coded feature, and determines the fourth coded feature output by the first self-attention block based on the Q value, K value, and V value of the first self-attention block, the first self-attention block can be used to multiply the weight of the third coded feature and the Q value of the first self-attention block to obtain the Q value of the first self-attention block; similarly, the third coded feature and the weight of the K value of the first self-attention block are multiplied to obtain the K value of the first self-attention block, and then the Q value of the first self-attention block and the transpose of the K value of the first self-attention block are multiplied to obtain the self-attention map. Subsequently, the first self-attention block is used to multiply the third coded feature and the weight of the V value of the first self-attention block to obtain the V value of the first self-attention block, and then the V value of the first self-attention block is multiplied by the self-attention map to obtain the fourth coded feature output by the first self-attention block.

[0135] Exemplarily, when the image generating device determines the input feature based on the fourth coding feature, it can directly determine the fourth coding feature as the input feature, or it can obtain the input feature after performing feature processing on the fourth coding feature. For example, the image generating device can perform feature extraction, feature construction, feature selection and other processing on the fourth coding feature to obtain the input feature. Among them, feature extraction, feature construction, feature selection and other processing can be carried out by relevant technical means involved in feature engineering or neural networks, and this application does not specifically limit this. For example, the image generating device can use a residual block to extract the fourth coding feature, and determine the feature obtained through feature extraction as the input feature.

[0136] FIG4 is an example of the structure of the first encoder provided in an embodiment of the present application.

[0137] As shown in FIG4 , the first encoder may include 4 residual blocks, 2 downsampling layers and a first self-attention block.

[0138] In actual processing, the image generation device first uses a residual block to extract features from the input image to obtain a first coding feature; then, it uses a downsampling layer to downsample the first coding feature to obtain a second coding feature; then, the image generation device can use a residual block to extract features from the second coding feature, and use a downsampling layer to downsample the features obtained through feature extraction, and then use a residual block to extract features from the downsampled features again, and determine the features obtained through feature extraction as the third coding feature. Then, the image generation device uses the first self-attention block to perform self-attention processing on the third coding feature to obtain the fourth coding feature. Finally, the image generation device can use a residual block to extract features from the fourth coding feature, and determine the features obtained through feature extraction as the input feature.

[0139] Figure 5 is an example of the structure of the first self-attention block provided in an embodiment of the present application.

[0140] As shown in Figure 5, the image generation device determines the Q value, K value, and V value of the first self-attention block based on the third coding feature, and based on the Q value and K value of the first self-attention block, uses a normalized exponential function to determine a self-attention map, and then multiplies the V value of the first self-attention block by the self-attention map to obtain the fourth coding feature output by the first self-attention block. Specifically, the image generation device can use the first self-attention block to multiply the third coding feature by the weight of the Q value of the first self-attention block to obtain the Q value of the first self-attention block; similarly, the third coding feature can be multiplied by the weight of the K value of the first self-attention block to obtain the K value of the first self-attention block, and then the Q value of the first self-attention block can be multiplied by the transpose of the K value of the first self-attention block to obtain the self-attention map. Next, using the first self-attention block, the third coding feature and the weight of the V value of the first self-attention block are multiplied to obtain the V value of the first self-attention block, and then the V value of the first self-attention block and the self-attention map are multiplied to obtain the fourth coding feature output by the first self-attention block.

[0141] It should be understood that FIG4 is only an example of the first encoder provided in this application, and FIG5 is only an example of the first self-attention block provided in this application, and should not be understood as limiting this application. For example, in other alternative embodiments, the second encoding feature can be directly determined as the third encoding feature, that is, the residual block and the downsampling layer between the first downsampling layer from left to right and the first self-attention block in the first encoder can be omitted. For another example, a new residual block or other module can be added.

[0142] Of course, in other alternative embodiments, the image generation device may also use other methods to determine the Q value, K value, and V value of the first self-attention block (as well as other self-attention blocks and cross-attention blocks mentioned below), and this application does not specifically limit this. For example, taking the determination of the Q value of the first self-attention block based on the third coding feature as an example, in other alternative embodiments, the third coding feature may be preprocessed first, and then the Q value of the first self-attention block is determined based on the features obtained after the preprocessing.

[0143] It should be understood that in the attention mechanism, Q stands for query, K stands for key, and V stands for value.

[0144] Wherein, query (Q) represents the current input or content requiring attention, which is used to guide the degree of attention to the key-value pair. In this embodiment, the Q value of the first self-attention block can be directly multiplied by the weight of the third encoding feature and the Q value of the first self-attention block to obtain the Q value of the first self-attention block.

[0145] Key (K): represents a feature or attribute related to the query. In this embodiment, the weight of the third encoded feature and the K value of the first self-attention block can be directly multiplied to obtain the K value of the first self-attention block.

[0146] Value (V): represents the content or information that needs attention. In this embodiment, the third encoding feature and the weight of the V value of the first self-attention block can be directly multiplied to obtain the V value of the first self-attention block.

[0147] Taking the first self-attention block as an example, after obtaining the Q value, K value and V value of the first self-attention block, the Q value is compared with the K value to calculate a self-attention map, which can be used to characterize the degree of attention to the V value or to measure the importance of the V value. Among them, the calculation of the attention map in the attention mechanism can be calculated using dot product or other similarity metrics (such as Euclidean distance, cosine similarity), and the output of the attention mechanism can be calculated by multiplying the calculated attention map and the V value. In this embodiment, the Q value of the first self-attention block and the transpose of the K value of the first self-attention block can be multiplied to obtain a self-attention map, and then the V value of the first self-attention block and the self-attention map are multiplied to obtain the fourth encoding feature output by the first self-attention block.

[0148] In some embodiments, the reference image is encoded by a second encoder, the first encoder and the second encoder have the same structure, and parameters of the first encoder and the second encoder are shared.

[0149] Exemplarily, the same structure of the first encoder and the second encoder can be understood as follows: the layers (and / or blocks) included in the first encoder are the same as the layers (and / or blocks) included in the second encoder, and the connection relationship between the layers (and / or blocks) in the first encoder is the same as the connection relationship between the layers (and / or blocks) in the second encoder.

[0150] Exemplarily, the parameter sharing of the first encoder and the parameter sharing of the second encoder can be understood as: the parameter type of the layer (and / or block) included in the first encoder is the same as the parameter type of the layer (and / or block) included in the second encoder, and the parameter value of the layer (and / or block) included in the first encoder is the same as the parameter value of the layer (and / or block) included in the second encoder.

[0151] FIG6 is another schematic flowchart of the image generation method provided in an embodiment of the present application.

[0152] As shown in Figure 6, the image generating device can use the first encoder to encode the input image to obtain the input feature, and use the second encoder to encode the reference image to obtain the reference feature; then, use the decoder to decode the input feature and the reference feature to obtain the output image and output the output image.

[0153] The first decoder and the second decoder have the same structure and share parameters. Sharing parameters can reduce the number of parameters, accelerate the inference speed, and ensure that the distribution of input features and reference features is basically consistent. Of course, in other alternative embodiments, the first decoder and the second decoder may not share parameters.

[0154] The first encoder, the second encoder and the decoder may be collectively referred to as an image generation model, the input of the image generation model being the input image and the reference image, and the output of the image generation model being the input image.

[0155] For the input image, the input image may be a rendering image, or an image constructed based on the rendering image. For example, the input image may be an image input or output based on the data construction process shown in FIG3 .

[0156] During training, a reference image is randomly selected from a certain frame of the input video. For example, if the image generation model is trained using the 100th frame of a video as the input image during the current training iteration, a frame between the 50th and 150th frames can be randomly selected as the reference image. This range is adjustable. During testing, the driving image is selected as the reference image. For example, if the driving image is the 100th frame of the test video, the reference image is the current 100th frame.

[0157] It's important to note that for image generation models, the training and test videos are different. The training videos consist of other people and diverse backgrounds. During testing, the model has never seen the same person or background before, so reference images are needed to supplement the person's facial details and background information.

[0158] For example, the training loss l of the image generation model is defined as follows: l pix =||x r -x h || 1 (1). l per =||φ(x r )-φ(x h )|| 1 (3). l=l pix +l mouth +l per +l adv (5).

[0159] Among them, x r The image output by the image generation model, x h is the real image (label) for training, Represents the mouth area corresponding to the image output by the image generation model, represents the mouth area corresponding to the real image, so l mouthis the loss for pixel-level enhancement of the mouth. φ is the VGG network, which is used to extract the deep features of the image. adv is the loss of the generative adversarial network, and D is the discriminator. This application does not limit the specific type of the discriminator (also known as the discriminant network), which can be varied, such as the common VGG network structure, or the structure of Patch GAN. The VGG network is a classic convolutional neural network. The design of the VGG network structure is very consistent. Its core is to use 3x3 convolution kernels and 2x2 maximum pooling operations. For example, the VGG16 network contains 13 convolutional layers and 3 fully connected layers, while the VGG19 network contains 16 convolutional layers and 3 fully connected layers. It should be noted that there is no essential difference between VGG16 and VGG19, and they only differ in the depth of the network. In general, the VGG network is a convolutional neural network structure with scalability in depth and breadth. It is characterized by the use of small convolution kernels and step sizes and fixed pooling layers to form the basic units of the network, which makes the network training easier and also gives the network stronger feature extraction capabilities. Generative Adversarial Networks (GANs) are deep learning models used to generate new data that resembles real data. A GAN consists of two parts: a generator and a discriminator. The generator's task is to generate data that is as realistic as possible, while the discriminator's task is to distinguish between generator-generated data and real data as accurately as possible. These two parts undergo adversarial training during training, hence the name "generative adversarial network." When training a GAN, the generator and discriminator are trained alternately. First, the discriminator is trained using real data and data generated by the generator to improve its ability to distinguish between real and generated data. The generator then uses the discriminator (the output of the feedback) to adjust how it generates data to try to deceive the discriminator as much as possible. This process is repeated until a certain effect is achieved.

[0160] Of course, FIG6 is merely an example of the present application and should not be construed as limiting the present application.

[0161] For example, in other alternative embodiments, the structures of the first encoder and the second encoder may be different, or the parameters of the first encoder and the second encoder may not be shared, which is not specifically limited in this application.

[0162] In some embodiments, the input feature and the reference feature are decoded by a decoder, the decoder including a first cross-attention block, a residual block, a second self-attention block, and an upsampling layer; wherein S270 may include:

[0163] The image generating device determines the Q value of the first cross-attention block based on the input feature, determines the K value and V value of the first cross-attention block based on the reference feature, and determines the first decoding feature based on the Q value, K value and V value of the first cross-attention block; then, based on the first decoding feature, determines the second decoding feature input to the second self-attention block; based on the second decoding feature, determines the Q value, K value and V value of the second self-attention block, and based on the Q value, K value and V value of the second self-attention block, determines the third decoding feature output by the second self-attention block; based on the third decoding feature, determines the fourth decoding feature for upsampling; upsamples the fourth decoding feature to obtain a fifth decoding feature; and determines the output image based on the fifth decoding feature.

[0164] Exemplarily, when the image generating device determines the Q value of the first cross-attention block based on the input feature, determines the K value and V value of the first cross-attention block based on the reference feature, and determines the first decoding feature based on the Q value, K value, and V value of the first cross-attention block, the first cross-attention block can be used to multiply the weight of the input feature and the Q value of the first cross-attention block to obtain the Q value of the first cross-attention block; similarly, the reference feature and the K value of the first cross-attention block are multiplied to obtain the K value of the first cross-attention block, and the Q value of the first cross-attention block and the transpose of the K value of the first cross-attention block are multiplied to obtain a cross-attention map. Then, the first cross-attention block is used to multiply the reference feature and the V value of the first cross-attention block to obtain the V value of the first cross-attention block, and the V value of the first cross-attention block and the cross-attention map are multiplied to obtain the first decoding feature output by the first cross-attention block.

[0165] It is worth noting that in the self-attention mechanism, the inputs used to calculate the Q value, K value, and V value are the same. For example, for the first self-attention block, the inputs used to calculate the Q value, K value, and V value are all the third encoded feature. In the cross-attention mechanism, the inputs used to calculate the Q value and K value are different, and the inputs used to calculate the K value and V value are the same. For example, for the first cross-attention block, the input used to calculate the Q value is the input feature, and the inputs used to calculate the K value and V value are all the reference feature.

[0166] Exemplarily, when the image generating device determines the second decoding feature input to the second self-attention block based on the first decoding feature, the first decoding feature can be directly determined as the second decoding feature, or the second decoding feature can be obtained after feature processing on the first decoding feature. For example, the image generating device can perform feature extraction, feature construction, feature selection and other processing on the first decoding feature to obtain the second decoding feature. Among them, feature extraction, feature construction, feature selection and other processing can be carried out by relevant technical means involved in feature engineering or neural networks, and this application does not make specific limitations on this. For example, the image generating device can use a residual block to extract features from the first decoding feature, and determine the features obtained through feature extraction as the second decoding feature.

[0167] Exemplarily, when the image generating device determines the Q value, K value, and V value of the second self-attention block based on the second decoding feature, and determines the third decoding feature output by the second self-attention block based on the Q value, K value, and V value of the second self-attention block, the specific process can refer to the processing process of the first self-attention block. To avoid repetition, it will not be repeated here.

[0168] Exemplarily, when the image generation device determines the fourth decoding feature for upsampling based on the third decoding feature, the third decoding feature may be directly determined as the fourth decoding feature, or the fourth decoding feature may be obtained after feature processing is performed on the third decoding feature. For example, the image generation device may perform feature extraction, feature construction, feature selection, and other processing on the third decoding feature to obtain the fourth decoding feature. Among them, the feature extraction, feature construction, feature selection, and other processing may be carried out by relevant technical means involved in feature engineering or neural networks, and this application does not specifically limit this. For example, the image generation device may use a residual block to extract features from the third decoding feature, and determine the features obtained through feature extraction as the fourth decoding feature.

[0169] Exemplarily, when determining the output image based on the fifth decoded feature, the image generation device may directly determine the fifth decoded feature as the output image, or may perform feature processing on the fifth decoded feature to obtain the output image. For example, the image generation device may perform feature extraction, feature construction, feature selection, and other processing on the fifth decoded feature to obtain the output image. The feature extraction, feature construction, feature selection, and other processing may be performed using relevant technical means involved in feature engineering or neural networks, and this application does not specifically limit this.

[0170] For example, the first cross-attention block differs from the second self-attention block in that:

[0171] The three parts Q / K / V of the second self-attention module are all obtained from the same input (i.e., the second decoding feature), while the Q of the first cross-attention block is obtained from the input feature, and the two parts K / V are obtained from the reference feature.

[0172] In this embodiment, by introducing the first cross-attention block, the input features can be fused with the reference features, and the input features can extract the required reference information from the reference features to supplement the missing details, such as the missing background of the black edges of the input image, and the missing facial details (the 3D rendering image has low accuracy), etc.

[0173] In some embodiments, the decoder further includes a tooth matching model; wherein upsampling the fourth decoded feature to obtain the fifth decoded feature may be implemented as follows:

[0174] When the mouth shape of the target object in the rendering image is open and the mouth shape of the target object in the reference image is closed, the image generating device uses the tooth matching model to obtain a tooth image that matches the mouth shape of the target object in the rendering image; then, based on the size of the rendering image and the size of the mouth shape of the target object in the rendering image, the tooth image is subjected to feature extraction to obtain tooth features; and based on the tooth features, the third decoding feature is upsampled to obtain the fifth decoding feature.

[0175] Exemplarily, the teeth matching model may be a pre-trained model.

[0176] Exemplarily, when the image generation device uses the tooth matching model to obtain a tooth image that matches the mouth shape of the target subject in the rendering, the device may use the mouth shape of the target subject in the rendering as input, predict the teeth of the target subject, and output the tooth image. Alternatively, based on the mouth shape of the target subject in the rendering, the device may select a tooth image that matches the mouth shape of the target subject in the rendering from a tooth library. The tooth library may be a database of one or more tooth images.

[0177] Exemplarily, the image generation device may extract features from the dental image based on the size of the rendering and the size of the mouth of the target object in the rendering to obtain dental features. The size of the dental features may be the same as the size of the rendering, and the size of valid features within the dental features may be the same as the size of the mouth of the target object in the rendering. Thus, when upsampling the third decoded feature based on the dental features, the image generation device may directly merge the dental features with the third decoded feature, and upsample the merged features to obtain the fifth decoded feature.

[0178] In this embodiment, by introducing a teeth matching model, the output image can be supplemented with teeth that match the mouth shape of the target object in the rendering image, thereby improving the realism and image effect of the output image.

[0179] In some embodiments, the decoder further includes a second cross-attention block; wherein, based on the fifth decoded feature, determining the output image may be implemented as follows:

[0180] The image generating device determines the sixth decoding feature input to the second cross-attention block based on the fifth decoding feature; then, determines the Q value of the second cross-attention block based on the sixth decoding feature, determines the K value and V value of the second cross-attention block based on the reference feature, and determines the seventh decoding feature based on the Q value, K value and V value of the second cross-attention block; and determines the output image based on the seventh decoding feature.

[0181] Exemplarily, when the image generating device determines the sixth decoding feature input to the second cross-attention block based on the fifth decoding feature, the fifth decoding feature can be directly determined as the sixth decoding feature, or the sixth decoding feature can be obtained after feature processing on the fifth decoding feature. For example, the image generating device can perform feature extraction, feature construction, feature selection and other processing on the fifth decoding feature to obtain the sixth decoding feature. Among them, feature extraction, feature construction, feature selection and other processing can be carried out by relevant technical means involved in feature engineering or neural networks, and this application does not specifically limit this. For example, the image generating device can directly determine the fifth decoding feature as the sixth decoding feature.

[0182] Exemplarily, when the image generating device determines the output image based on the seventh decoding feature, it can directly determine the seventh decoding feature as the output image, or it can obtain the output image after performing feature processing on the seventh decoding feature. For example, the image generating device can perform feature extraction, feature construction, feature selection and other processing on the seventh decoding feature to obtain the output image. Among them, the feature extraction, feature construction, feature selection and other processing can be carried out by relevant technical means involved in feature engineering or neural networks, and this application does not specifically limit this. For example, the image generating device can use a residual block to extract the seventh decoding feature, and use an upsampling layer to upsample the features obtained through feature extraction, and then use a residual block to extract the features obtained through upsampling, and determine the features obtained through feature extraction as the output image.

[0183] Exemplarily, the image generating device determines the Q value of the second cross-attention block based on the sixth decoding feature, determines the K value and V value of the second cross-attention block based on the reference feature, and determines the seventh decoding feature based on the Q value, K value and V value of the second cross-attention block. The specific process can refer to the processing process of the first cross-attention block (the only difference is that the input is different). To avoid repetition, it will not be repeated here.

[0184] FIG7 is an example of the structure of a decoder provided in an embodiment of the present application.

[0185] As shown in FIG7 , the first encoder may include 4 residual blocks, 2 upsampling layers, a second self-attention block, a first cross-attention block, and a second cross-attention block.

[0186] In actual processing, the image generation device first uses the first cross-attention block to perform cross-attention processing on the input feature to obtain the first decoded feature. Then, a residual block is used to perform feature extraction on the first decoded feature, and the feature obtained through feature extraction is determined as the second decoded feature. Next, the image generation device may perform self-attention processing on the second decoded feature using the second self-attention block to obtain a third decoded feature. A residual block is used to perform feature extraction on the third decoded feature to obtain a fourth decoded feature. An upsampling layer is used to upsample the fourth decoded feature to obtain a fifth decoded feature. Then, the image generation device may use the second cross-attention block to perform cross-attention processing on the fifth decoded feature to obtain a sixth decoded feature. A residual block is used to perform feature extraction on the sixth decoded feature to obtain a seventh decoded feature. Finally, the image generation device may use a residual block to perform feature extraction on the seventh decoded feature, upsample the feature obtained through feature extraction using an upsampling layer, and then use a residual block to perform feature extraction on the upsampled feature, and determine the feature obtained through feature extraction as the output image.

[0187] Figure 8 is an example of the structure of the first cross-attention block provided by an embodiment of the present application.

[0188] As shown in Figure 8, the image generating device determines the Q value of the first cross-attention block based on the input feature, determines the K value and V value of the first cross-attention block based on the reference feature, and determines the cross-attention map based on the Q value and K value of the first cross-attention block using a normalized exponential function, and then multiplies the V value of the first cross-attention block by the cross-attention map to obtain the first decoding feature output by the first cross-attention block. Specifically, the image generating device can use the first cross-attention block to multiply the weight of the input feature and the Q value of the first cross-attention block to obtain the Q value of the first cross-attention block; similarly, multiply the reference feature and the K value of the first cross-attention block to obtain the K value of the first cross-attention block, and then multiply the Q value of the first cross-attention block by the transpose of the K value of the first cross-attention block to obtain the cross-attention map. Next, using the first cross-attention block, the reference feature and the weight of the V value of the first cross-attention block are multiplied to obtain the V value of the first cross-attention block, and then the V value of the first cross-attention block and the cross-attention map are multiplied to obtain the first decoding feature output by the first cross-attention block.

[0189] It should be understood that Figure 7 is only an example of decoding provided by the present application, and Figure 8 is only an example of the first cross-attention block provided by the present application. It should not be understood as a limitation to the present application. For example, in other alternative embodiments, the seventh decoding feature can be directly determined as the output image, that is, the best residual block from left to right and the last upsampling layer in the decoder can be omitted. For another example, new residual blocks or other modules can be added. For another example, the first cross-attention block or the second cross-attention block involved in the figure can also be replaced by other modules for feature fusion. For example, the two can be directly merged and fused using 2D convolution. For another example, deformable convolution can also be used for fusion.

[0190] In some embodiments, S220 may include:

[0191] The image generation device samples the texture of the target object in the input video to obtain texture information of the target object; then, generates the rendering based on the texture information.

[0192] Exemplarily, the image generating device can utilize a neural network or other technology to perform feature extraction on the texture of the target object in the input video and obtain the texture information. The texture information can be a high-frequency signal of the texture feature that reflects the outline of the target object in the image in the input video. The outline is the line that constitutes the outer edge of a graphic or object, and is an important basis for technologies such as target detection, shape analysis, target recognition, and target detection. The high-frequency component in an image refers to the place where the image intensity (brightness / grayscale) changes dramatically, which is what we often call the edge (contour). For example, if there is a line of words on a piece of white paper, then we will definitely focus directly on the words and not pay too much attention to the white paper itself. Here, the words are high-frequency signals, and the white paper is a low-frequency signal.

[0193] In summary, the contour of the target object is reflected by the high-frequency components in the image.

[0194] Exemplarily, the image generating device may utilize a pre-trained model for generating images based on textures, and obtain and output the rendering image using the texture information as input.

[0195] In this embodiment, the image generation device samples the texture of the target object in the input video to obtain texture information of the target object; then, the rendering image is generated based on the texture information, which can ensure that the rendering image is a rendering image that matches the facial contour of the target object in the input video, and can ensure the realistic effect of the rendering image.

[0196] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the embodiments mentioned above. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the specific embodiments mentioned above can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0197] It should also be understood that in the various method embodiments of the present application, the size of the serial numbers of the processes involved above does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0198] The above describes the method provided in the embodiment of the present application, and the following describes the device provided in the embodiment of the present application.

[0199] FIG9 is a schematic block diagram of an image generating device 300 provided in an embodiment of the present application.

[0200] As shown in FIG9 , the image generating device 300 may include:

[0201] A first acquiring unit 310 is configured to acquire input video and input audio;

[0202] A second acquisition unit 320 is configured to acquire a rendering image that matches the facial contour of the target object in the input video;

[0203] An adjusting unit 330 is configured to adjust the lip shape of the target object in the rendering image to a lip shape that matches the audio unit in the input audio, thereby obtaining an input image.

[0204] A first determining unit 340 is configured to sample the input video to obtain an image sequence, and determine an image driven by the audio unit in the image sequence;

[0205] A second determining unit 350 is configured to determine a reference image of the input image based on the image driven by the audio unit;

[0206] an encoding unit 360, configured to encode the input image and the reference image to obtain input features of the input image and reference features of the reference image;

[0207] The decoding unit 370 is configured to decode the input feature and the reference feature to obtain an output image.

[0208] In some embodiments, the first determining unit 340 is specifically configured to:

[0209] When the number of images in the image sequence is equal to the number of units included in the input audio, the image in the image sequence whose sampling number is the same as the position number of the audio unit in the input audio is determined as the image driven by the audio unit.

[0210] In some embodiments, the second determining unit 350 is specifically configured to:

[0211] The image driven by the audio unit is determined as the reference image.

[0212] In some embodiments, the second determining unit 350 is specifically configured to:

[0213] Centering the image driven by the audio unit, multiple images are selected forward and backward in the image sequence according to the sampling time sequence;

[0214] The reference image is selected from the multiple images.

[0215] In some embodiments, the input image is encoded by a first encoder, the first encoder comprising a residual block, a downsampling layer, and a first self-attention block;

[0216] The encoding unit 360 is specifically configured to:

[0217] Performing feature extraction on the input image using the residual block to obtain a first coding feature;

[0218] Downsampling the first coding feature using the downsampling layer to obtain a second coding feature;

[0219] Determining a third encoded feature to be input to the first self-attention block based on the second encoded feature;

[0220] Determining a Q value, a K value, and a V value of the first self-attention block based on the third encoded feature, and determining a fourth encoded feature output by the first self-attention block based on the Q value, the K value, and the V value of the first self-attention block;

[0221] Based on the fourth encoding feature, the input feature is determined.

[0222] In some embodiments, the reference image is encoded by a second encoder, the first encoder and the second encoder have the same structure, and parameters of the first encoder and the second encoder are shared.

[0223] In some embodiments, the input feature and the reference feature are decoded by a decoder, the decoder comprising a first cross-attention block, a residual block, a second self-attention block, and an upsampling layer;

[0224] The decoding unit 370 is specifically configured to:

[0225] Determining a Q value of the first cross-attention block based on the input feature, determining a K value and a V value of the first cross-attention block based on the reference feature, and determining a first decoding feature based on the Q value, the K value, and the V value of the first cross-attention block;

[0226] Determining a second decoding feature input to the second self-attention block based on the first decoding feature;

[0227] Determining a Q value, a K value, and a V value of the second self-attention block based on the second decoding feature, and determining a third decoding feature output by the second self-attention block based on the Q value, the K value, and the V value of the second self-attention block;

[0228] Determining a fourth decoding feature for upsampling based on the third decoding feature;

[0229] Upsampling the fourth decoding feature to obtain a fifth decoding feature;

[0230] Based on the fifth decoded feature, the output image is determined.

[0231] In some embodiments, the decoder further comprises a second cross-attention block;

[0232] The decoding unit 370 is specifically configured to:

[0233] Determining a sixth decoded feature to be input to the second cross-attention block based on the fifth decoded feature;

[0234] Determining a Q value of the second cross-attention block based on the sixth decoding feature, determining a K value and a V value of the second cross-attention block based on the reference feature, and determining a seventh decoding feature based on the Q value, the K value, and the V value of the second cross-attention block;

[0235] Based on the seventh decoded feature, the output image is determined.

[0236] In some embodiments, the decoder further includes a teeth matching model;

[0237] The decoding unit 370 is specifically configured to:

[0238] When the mouth shape of the target object in the rendering image is open and the mouth shape of the target object in the reference image is closed, using the teeth matching model to obtain a teeth image that matches the mouth shape of the target object in the rendering image;

[0239] Based on the size of the rendering image and the size of the mouth of the target object in the rendering image, feature extraction is performed on the tooth image to obtain tooth features;

[0240] The third decoded feature is up-sampled based on the tooth feature to obtain the fifth decoded feature.

[0241] In some embodiments, the second acquiring unit 320 is specifically configured to:

[0242] Sampling the texture of the target object in the input video to obtain texture information of the target object;

[0243] The rendering is generated based on the texture information.

[0244] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they are not described here in detail. Specifically, the image generation device 300 may correspond to the corresponding subject in the method 200 of the embodiment of the present application, and the various units in the image generation device 300 are respectively for implementing the corresponding processes in the method 200. For the sake of brevity, they are not described here in detail.

[0245] It should also be understood that the various units in the image generating device 300 involved in the embodiment of the present application are divided based on logical functions. In actual applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit, and even these functions can be assisted by one or more other units. For example, part or all of the image generating device 300 is merged into one or several other units. For another example, a certain unit (or units) in the image generating device 300 can also be split into multiple smaller units in function to form a unit, which can achieve the same operation without affecting the realization of the technical effects of the embodiment of the present application. For another example, the image generating device 300 can also include other units. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units.

[0246] According to another embodiment of the present application, the image generation device 300 involved in the embodiment of the present application can be constructed by running a computer program (including program code) capable of executing each step involved in the corresponding method on a general-purpose computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory medium (RAM), and a read-only memory medium (ROM), and implementing the method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into an electronic device through the computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application. In other words, the units involved above can be implemented in the form of hardware, or can be implemented by instructions in the form of software, or can be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or performed by a combination of hardware and software in the decoding processor. Optionally, the software can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned method embodiment in combination with its hardware.

[0247] FIG10 is a schematic structural diagram of an electronic device 400 provided in an embodiment of the present application.

[0248] As shown in Figure 10, the electronic device 400 includes at least a processor 410 and a computer-readable storage medium 420. The processor 410 and the computer-readable storage medium 420 may be connected via a bus or other means. The computer-readable storage medium 420 is used to store a computer program 421, which includes computer instructions. The processor 410 is used to execute the computer instructions stored in the computer-readable storage medium 420. The processor 410 is the computing core and control core of the electronic device 400 and is suitable for implementing one or more computer instructions, specifically loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0249] As an example, the processor 410 may also be referred to as a central processing unit (CPU). The processor 410 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like.

[0250] As an example, the computer-readable storage medium 420 may be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 410. Specifically, the computer-readable storage medium 420 includes, but is not limited to: volatile memory and / or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0251] As shown in FIG. 10 , the electronic device 400 may further include a transceiver 430 .

[0252] The processor 410 may control the transceiver 430 to communicate with other devices. Specifically, the processor 410 may send information or data to other devices or receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include one or more antennas.

[0253] It should be understood that the various components in the electronic device 400 are connected via a bus system, wherein the bus system includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. It is worth noting that the electronic device 400 can be any electronic device with data processing capabilities; the computer-readable storage medium 420 stores a first computer instruction; the processor 410 loads and executes the first computer instruction stored in the computer-readable storage medium 420 to implement the corresponding steps in the method embodiment shown in Figure 1; in a specific implementation, the first computer instruction in the computer-readable storage medium 420 is loaded by the processor 410 and the corresponding steps are executed. To avoid repetition, this will not be repeated here.

[0254] According to another aspect of the present application, an embodiment of the present application provides a chip. The chip can be an integrated circuit chip with signal processing capabilities, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The chip can also be called a system-on-chip, a system-on-chip, a chip system, or a system-on-chip chip, etc. The chip can be applied to various electronic devices that can be installed with the chip, so that the device equipped with the chip can execute the corresponding steps in the various methods or logic block diagrams disclosed in the embodiments of the present application. For example, the chip can be suitable for implementing one or more computer instructions, specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0255] According to another aspect of the present application, an embodiment of the present application provides a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device of a computer for storing programs and data. It is understandable that the computer-readable storage medium herein may include both built-in storage media in the computer and, of course, extended storage media supported by the computer. The computer-readable storage medium provides a storage space that stores an operating system of an electronic device. The storage space stores computer instructions suitable for being loaded and executed by a processor. When the computer instructions are read and executed by the processor of the computer device, the computer device executes the corresponding steps in the various methods or logic block diagrams disclosed in the embodiments of the present application.

[0256] According to another aspect of the present application, an embodiment of the present application provides a computer program product or computer program. The computer program product or computer program includes computer instructions, which are stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the corresponding steps in the various methods or logic block diagrams disclosed in the embodiments of the present application. In other words, when software is used to implement the solution provided by the present application, it can be implemented in whole or in part in the form of a computer program product or computer program. The computer program product or computer program includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the functions of the embodiment of the present application are implemented.

[0257] It is worth noting that the computer involved in this application can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions involved in this application can be stored in a computer-readable storage medium, or can be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0258] Those skilled in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. In other words, professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of this application.

[0259] Finally, it should be noted that the above content is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the basic idea of ​​the present application, they should also be regarded as the content disclosed in the present application.

Claims

1. An image generation method, characterized in that: include: Get input video and input audio; Obtaining a rendering that matches the facial contour of the target object in the input video; Based on the audio unit in the input audio, the mouth shape of the target object in the rendering image is adjusted to a mouth shape matching the audio unit to obtain an input image; Sampling the input video to obtain an image sequence, and determining an image driven by the audio unit in the image sequence; Determining a reference image of the input image based on the image driven by the audio unit; Encoding the input image and the reference image to obtain input features of the input image and reference features of the reference image; The input features and the reference features are decoded to obtain an output image.

2. The method according to claim 1, characterized in that: The step of sampling the input video to obtain an image sequence and determining the image driven by the audio unit in the image sequence comprises: When the number of images in the image sequence is equal to the number of units included in the input audio, the image whose sampling number in the image sequence is the same as the position number of the audio unit in the input audio is determined as the image driven by the audio unit.

3. The method according to claim 1 or 2, characterized in that: The step of determining a reference image of the input image based on the image driven by the audio unit comprises: The image driven by the audio unit is determined as the reference image.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining a reference image of the input image based on an image driven by the audio unit in the input video comprises: Centering on the image driven by the audio unit, multiple images are selected forward and backward in the image sequence according to the sampling time sequence; The reference image is selected from the multiple images.

5. The method according to any one of claims 1 to 4, characterized in that: The input image is encoded by a first encoder, the first encoder comprising a residual block, a downsampling layer, and a first self-attention block; The step of encoding the input image and the reference image to obtain input features of the input image and reference features of the reference image includes: Using the residual block to extract features from the input image to obtain a first encoding feature; Downsampling the first coding feature using the downsampling layer to obtain a second coding feature; Based on the second encoding feature, determining a third encoding feature input to the first self-attention block; Determine a Q value, a K value, and a V value of the first self-attention block based on the third encoding feature, and determine a fourth encoding feature output by the first self-attention block based on the Q value, the K value, and the V value of the first self-attention block; Based on the fourth encoding feature, the input feature is determined.

6. The method according to claim 5, characterized in that The reference image is encoded by a second encoder, the first encoder and the second encoder have the same structure, and parameters of the first encoder and the second encoder are shared.

7. The method according to any one of claims 1 to 6, characterized in that: The input features and the reference features are decoded by a decoder, wherein the decoder includes a first cross attention block, a residual block, a second self-attention block, and an upsampling layer; The step of decoding the input feature and the reference feature to obtain an output image includes: Determine a Q value of the first cross-attention block based on the input feature, determine a K value and a V value of the first cross-attention block based on the reference feature, and determine a first decoding feature based on the Q value, K value, and V value of the first cross-attention block; Based on the first decoding feature, determining a second decoding feature input to the second self-attention block; Determine a Q value, a K value, and a V value of the second self-attention block based on the second decoding feature, and determine a third decoding feature output by the second self-attention block based on the Q value, the K value, and the V value of the second self-attention block; Based on the third decoding feature, determining a fourth decoding feature for upsampling; Upsampling the fourth decoding feature to obtain a fifth decoding feature; Based on the fifth decoded feature, the output image is determined.

8. The method according to claim 7, characterized in that The decoder also includes a second cross-attention block; Wherein, determining the output image based on the fifth decoding feature includes: Based on the fifth decoded feature, determining a sixth decoded feature to be input to the second cross-attention block; Determine the Q value of the second cross-attention block based on the sixth decoding feature, determine the K value and the V value of the second cross-attention block based on the reference feature, and determine the seventh decoding feature based on the Q value, K value and V value of the second cross-attention block. levy; Based on the seventh decoded feature, the output image is determined.

9. The method according to claim 7 or 8, characterized in that: The decoder also includes a teeth matching model; The upsampling of the fourth decoding feature to obtain the fifth decoding feature includes: When the mouth shape of the target object in the rendering image is open and the mouth shape of the target object in the reference image is closed, using the teeth matching model to obtain a teeth image that matches the mouth shape of the target object in the rendering image; Based on the size of the rendering image and the size of the mouth shape of the target object in the rendering image, feature extraction is performed on the tooth image to obtain tooth features; The third decoded feature is upsampled based on the tooth feature to obtain the fifth decoded feature.

10. The method according to any one of claims 1 to 9, characterized in that: The step of obtaining a rendering image that matches the facial contour of the target object in the input video includes: Sampling the texture of the target object in the input video to obtain texture information of the target object; The rendering is generated based on the texture information.

11. An image generating device, characterized in that: include: A first acquisition unit, used to acquire input video and input audio; A second acquisition unit, configured to acquire a rendering image matching the facial contour of the target object in the input video; an adjusting unit, configured to adjust the mouth shape of the target object in the rendering image to a mouth shape matching the audio unit based on the audio unit in the input audio, so as to obtain an input image; A first determining unit, configured to sample the input video to obtain an image sequence, and determine an image driven by the audio unit in the image sequence; A second determining unit, configured to determine a reference image of the input image based on the image driven by the audio unit; an encoding unit, configured to encode the input image and the reference image to obtain input features of the input image and reference features of the reference image; A decoding unit is used to decode the input feature and the reference feature to obtain an output image.

12. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program / instructions, characterized in that The computer program / instructions implement the method according to any one of claims 1 to 10 when executed by a processor.

Images