Virtual image driving method, device and equipment

Abstract

The invention provides a virtual image driving method, which comprises the steps of determining a target image corresponding to to-be-driven audio data from a target data set based on the to-be-driven audio data, the target data set comprising a plurality of reference images for the face of a target object, and the target image being at least one of the plurality of reference images; and generating a virtual image driving result corresponding to the to-be-driven audio data by using the target face three-dimensional model and the target image corresponding to the to-be-driven audio. The invention further provides a virtual image driving device and equipment.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of computer vision and human-computer interaction, and in particular to a virtual image driving method, device and equipment. Background Art

[0002] The speaker lip shape driving methods can generally be divided into universal lip shape driving and personalized lip shape driving. The universal lip shape driving model can drive facial animation based on standard lip shapes and phonemes using a set of universal lip shape states, while the personalized lip shape driving method can be customized based on the facial data and features of a specific speaker. However, since the universal lip shape driving model is not personalized based on the specific person's speech video, it cannot generate personalized lip shapes for a specific person; and although the personalized lip shape driving model can generate personalized lip shapes for a specific person, it usually requires recording the specific person's speech video in advance and performing personalized training on the model. The whole process is cumbersome and time-consuming. Summary of the invention

[0003] One aspect of the present disclosure provides a method for driving a virtual image, comprising: determining a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven, wherein the target data set includes multiple reference images for the face of the target object, and the target image is at least one of the multiple reference images; using a target facial three-dimensional model corresponding to the audio data to be driven and the target image to generate a virtual image driving result corresponding to the audio data to be driven.

[0004] According to an embodiment of the present disclosure, the target data set includes reference audio data corresponding to each reference image; determining the target image corresponding to the audio data to be driven from the target data set based on the audio data to be driven includes: determining the target image from multiple reference images based on the matching degree between the audio data to be driven and the reference audio data.

[0005] According to an embodiment of the present disclosure, based on the matching degree between the audio data to be driven and the reference audio data, a target image is determined from multiple reference images, including: determining first frequency characteristic data corresponding to the audio data to be driven, and second frequency characteristic data corresponding to the reference audio data; based on the similarity between the first frequency characteristic data and the second frequency characteristic data, determining the target image from multiple reference images.

[0006] According to an embodiment of the present disclosure, the target data set also includes second text data corresponding to the reference audio data, and the second text data is obtained by performing speech recognition on the reference audio data; based on the matching degree between the audio data to be driven and the reference audio data, determining the target image from multiple reference images, including: determining first text data corresponding to the audio data to be driven, and the first text data is obtained by performing speech recognition on the audio data to be driven; and determining the target image from multiple reference images based on the matching degree between the first text data and the second text data.

[0007] According to an embodiment of the present disclosure, a target facial three-dimensional model and a target image corresponding to the audio data to be driven are used to generate a virtual image driving result corresponding to the audio data to be driven, including at least one of the following: inputting the target facial three-dimensional model and the target image into a target rendering model to obtain a virtual image driving result; obtaining a target facial image through the target facial three-dimensional model, and superimposing at least part of the target image on the target facial image to obtain a virtual image driving result.

[0008] According to an embodiment of the present disclosure, there are multiple target images, and a target facial three-dimensional model and a target image corresponding to the audio data to be driven are used to generate a virtual image driving result corresponding to the audio data to be driven, including: determining weight information according to the degree of matching between each target image and the audio data to be driven; and generating a virtual image driving result corresponding to the audio data to be driven by using the target facial three-dimensional model, the target image and the weight information.

[0009] According to an embodiment of the present disclosure, a target data set is determined in the following manner: a target video corresponding to a target object is acquired; and multiple reference images and reference audio data corresponding to each reference image are determined based on a temporal correspondence between audio data and image data in the target video.

[0010] According to an embodiment of the present disclosure, the target facial 3D model is determined by the following operations: acquiring facial template data; and using a deformation control tool to adjust the facial template data based on the audio data to be driven to obtain the target facial 3D model.

[0011] Another aspect of the present disclosure provides a virtual image driving device, including: a target image determination module, used to determine a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven, wherein the target data set includes multiple reference images for the face of the target object, and the target image is at least one of the multiple reference images; and a driving result generation module, used to generate a virtual image driving result corresponding to the audio data to be driven using a target facial three-dimensional model and a target image corresponding to the audio data to be driven.

[0012] Another aspect of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the virtual image driving method as described above.

[0013] Another aspect of the present disclosure provides a non-volatile storage medium storing computer executable instructions, wherein the instructions are used to implement any of the above methods when executed.

[0014] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement any of the above methods when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 The application scenarios of the virtual image driving method and the virtual image driving device according to the embodiments of the present disclosure are shown;

[0017] Figure 2 A flowchart of a method for driving a virtual image according to an embodiment of the present disclosure is shown;

[0018] Figure 3 A flowchart of another virtual image driving method according to an embodiment of the present disclosure is shown;

[0019] Figure 4 A block diagram of a virtual image driving device according to an embodiment of the present disclosure is shown;

[0020] Figure 5 A schematic block diagram of an example electronic device that can be used to implement the method of the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0023] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0025] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0026] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.

[0027] Therefore, the technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable medium storing instructions, which can be used by an instruction execution system or in combination with an instruction execution system. In the context of the present disclosure, a computer-readable medium can be any medium that can contain, store, transmit, propagate, or transmit instructions. For example, a computer-readable medium can include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, device, or propagation medium. Specific examples of computer-readable media include: magnetic storage devices, such as magnetic tape or hard disk (HDD); optical storage devices, such as compact disk (CD-ROM); memory, such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0028] Figure 1The application scenarios of the virtual image driving method and the virtual image driving device according to the embodiments of the present disclosure are shown.

[0029] It should be noted that Figure 1 The examples shown are merely scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services. For example, the audio data to be driven may be processed in the first terminal device 101, the second terminal device 102, and the third terminal device 103 to generate a virtual image driving result. The background management server may analyze and process the received user request and other data, and feed back the processing result (virtual image driving result) to the terminal device.

[0034] It should be noted that the virtual image driving method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the virtual image driving device provided in the embodiment of the present disclosure can generally be set in the server 105. The virtual image driving method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the virtual image driving device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] In some examples, the terminal device collects video data and audio data of different speakers, and extracts the facial key points of the speakers (such as the trajectory of the lips) from the collected video data and audio data to obtain audio features; the audio features are then input into a universal lip shape drive model to output corresponding lip shape data, the lip shape data is mapped to a three-dimensional model or animation controller to generate a lip shape, and the generated lip shape is then combined with other facial expressions for rendering, and the lip shape drive result is output.

[0037] However, since the universal lip shape driven model is not individually trained based on a specific person's speaking video, it cannot generate personalized lip shapes for a specific person. The differences in lip shapes between different speakers may cause the universal model to be unable to adapt to each individual, limiting the universality of its application.

[0038] In other examples, the terminal device records a speaking video of the target individual, and by extracting key features of lip movement from the recorded video, the extracted features can be annotated with the corresponding audio or text data, and the annotated data set can be used to train a personalized lip shape driving model. Then, when the audio to be driven is input, the trained model can be used to generate the corresponding personalized lip shape animation.

[0039] However, in the process of generating personalized lip-shaped driving results using a personalized lip-shaped driving model, especially in application scenarios for high-quality and high-resolution data, it is necessary to record and process a large amount of video, audio, and image data, which requires special hardware support (such as GPU), resulting in a large consumption of time and resources. In addition, the training process of the personalized model is relatively complicated and requires multiple debugging and verification. From data collection to model training to application, the entire process takes a long time, affecting the actual usage efficiency of the application.

[0040] Based on the above problems, the present disclosure provides a virtual image driving method, including determining a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven, wherein the target data set includes multiple reference images for the face of the target object, and the target image is at least one of the multiple reference images; using the target facial three-dimensional model corresponding to the audio to be driven and the target image, a virtual image driving result corresponding to the audio data to be driven can be generated.

[0041] According to the embodiments of the present disclosure, by determining the target image from the target data set based on the audio data to be driven, the target facial three-dimensional model and the target image can be used to generate a virtual image driving result corresponding to the audio data to be driven. The embodiments of the present disclosure only use the target image to constrain the driven audio data, avoid personalized training for specific person data, realize the generation of personalized virtual image driving results for different specific people, simplify the system processing flow, save the system resource consumption, and further improve the efficiency of generating facial virtual images.

[0042] Figure 2 A flowchart of a virtual character driving method according to an embodiment of the present disclosure is shown.

[0043] like Figure 2 As shown, the virtual image driving method includes operations S201~S202.

[0044] In operation S201, a target image corresponding to the audio data to be driven is determined from a target data set based on the audio data to be driven, wherein the target data set includes multiple reference images for a target object's face, and the target image is at least one of the multiple reference images.

[0045] In an embodiment of the present disclosure, the target data set may include reference audio data, reference images and target images. The reference images may be different image frames corresponding to different time points in the reference audio data. The target images may be one or more images selected from different image frames according to needs, and are used to constrain the generation of virtual image driving results.

[0046] For example, based on the similarity comparison result between the frequency feature data corresponding to the current audio data to be driven and the frequency feature data corresponding to the reference audio data, the target image can be determined from multiple reference images. For example, based on the matching degree between the text data corresponding to the current audio data to be driven and the text data corresponding to the reference audio data, the target image can be determined.

[0047] In the embodiments of the present disclosure, the object corresponding to the audio data to be driven and the target object in the target data set may be the same object (person) or different objects. For example, the audio data to be driven may be audio data for object A, and the target data set may be a data set for target objects B, C, and D that has been acquired and stored in a specific system file. For example, the audio data to be driven may be audio data for object E acquired in real time, and the target data set may be a historical audio, video, or image data set for target object E acquired in a historical time period.

[0048] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user information (facial information) involved in the technical solution of this disclosure are in compliance with the relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0049] In operation S202, a virtual character driving result corresponding to the audio data to be driven is generated by using a target facial three-dimensional model and a target image corresponding to the audio data to be driven.

[0050] In the disclosed embodiment, the target facial 3D model (3D Mesh) can be a facial 3D model composed of multiple vertices, edges, faces and other factors (including texture, expression and shape), which can be generated in real time using different lip shape driving models, or obtained in real time from a 3D model library. The target image can be used to constrain the generation of the virtual image driving result.

[0051] For example, in the process of generating a virtual image driving result using a target rendering model or a related algorithm, the matching degree between multiple target images and the audio data to be driven is used to determine the allocation weights of multiple feature points of the target facial three-dimensional model, and then the weight value is used to set the degree to which the target image affects the target facial three-dimensional model during the driving process.

[0052] In an embodiment of the present disclosure, the target facial 3D model may be obtained by model rendering or from a system model library. For example, a deep audio-visual embedding model (DAVE) is used to combine the audio data to be driven and the visual information, and deep learning is used to learn the mapping relationship between audio and lip movements, so that the spatial representation of audio and lip shape is learned through a neural network to generate lip shape movements synchronized with the audio data, thereby generating a target facial 3D model.

[0053] For example, a MySQL (My Structured Query Language) database or file system can be created to store and manage target facial 3D models. An index can be created for each 3D model to store its feature information (including gender, age, emotion, etc.) for fast retrieval, or an application programming interface (Application Programming Interface, API) can be developed to meet application requests to retrieve specific facial models. A fast retrieval algorithm (such as feature-based similarity search) can be used to process model retrieval requests so that a matching target facial 3D model can be found within a preset time period.

[0054] In an embodiment of the present disclosure, a target rendering model or other processing methods (such as an image overlay algorithm) may be used to generate a virtual image driving result based on a target facial three-dimensional model and a target image.

[0055] For example, the target facial 3D model and the target image can be input into the target rendering model. The target rendering model can be a rendering model based on the cross-attention mechanism, which extracts features from the target image to obtain image features, and then uses the association fusion between the image features and the 3D Mesh features to achieve a weighted combination of features to generate a comprehensive feature representation, thereby achieving a dynamic generation of a virtual image driving result. It is understandable that the type of the target rendering model can be determined according to actual needs and is not specifically limited here.

[0056] For example, an image overlay algorithm can be used to process the target facial three-dimensional model to obtain a target facial image, and extract the facial features of the target image to obtain facial feature points and facial overlay areas, and then use image fusion technology to overlay part of the target image with the target facial image based on the facial feature points and facial overlay areas to generate a virtual image driving result.

[0057] According to the embodiments of the present disclosure, by determining the target image from the target data set based on the audio data to be driven, the target facial three-dimensional model and the target image can be used to generate a virtual image driving result corresponding to the audio data to be driven. Since only the target image is used to constrain the driven audio data, personalized training for specific person data is avoided, and personalized virtual image driving result generation for different specific people is achieved, which simplifies the system processing flow, saves system resource consumption, and further improves the efficiency of generating facial virtual images.

[0058] It can be understood that the above describes an example of how to generate a virtual image driving result, and the target image determination process of the present disclosure will be preliminarily described below.

[0059] According to an embodiment of the present disclosure, the target data set includes reference audio data corresponding to each reference image; determining the target image corresponding to the audio data to be driven from the target data set based on the audio data to be driven includes: determining the target image from multiple reference images based on the degree of matching between the audio data to be driven and the reference audio data.

[0060] In an embodiment of the present disclosure, the reference audio data may be pre-acquired audio data stored in a preset file system or database, and different frequency feature data may be obtained by extracting features from the reference audio data and the audio data to be driven, and text data corresponding to the reference audio data and the audio data to be driven may be obtained by identifying the reference audio data and the audio data to be driven. The matching degree between the reference audio data and the audio data to be driven may include the matching degree between different frequency feature data, and may also include the matching degree between the text data corresponding to the reference audio data and the audio data to be driven.

[0061] For example, the Mel-Frequency Cepstral Coefficients (MFCC) can be used to extract the frequency features of the reference audio data, and the Mel scale is used to process the frequency to achieve frequency transformation, and the Mel spectrum is processed through cepstrum analysis to obtain frequency feature data corresponding to the reference audio data.

[0062] For example, Automatic Speech Recognition (ASR) technology is used to preprocess and extract features of the audio data to be driven, and the spectral features of the audio data to be driven are obtained, and the extracted audio features are then input into the acoustic model and language model to obtain text data. The processed text data can be output in different formats according to actual needs.

[0063] In an embodiment of the present disclosure, a method for determining the degree of matching between different frequency feature data may include cosine similarity (Cosine Similarity) and dynamic time warping (Dynamic Time Warping, DTW). For example, dynamic time warping calculation can be used to process audio feature sequences of different lengths, and nonlinear alignment is used to calculate the similarity between different audio feature sequences, thereby obtaining the matching results between different frequency feature data. For example, different text data are preprocessed using cosine similarity, the processed text data is vectorized to generate word frequency vectors, and then the similarity of the word frequency vectors is calculated to obtain the matching results of different text data.

[0064] It can be understood that the above is a preliminary explanation of an example of how to determine a target image, and the method for determining the target image will be further explained below.

[0065] According to an embodiment of the present disclosure, based on the matching degree between the audio data to be driven and the reference audio data, a target image is determined from multiple reference images, including: determining first frequency characteristic data corresponding to the audio data to be driven, and second frequency characteristic data corresponding to the reference audio data; based on the similarity between the first frequency characteristic data and the second frequency characteristic data, determining the target image from multiple reference images.

[0066] In an embodiment of the present disclosure, audio data may correspond to different image frame data based on time features, and different image frame data may correspond to different frequency feature data. For example, in the audio data to be driven, there is a temporal correspondence between the audio data and the image frame data, the audio feature at time T1 corresponds to the image frame at time T1, and the feature data corresponding to the image frame at time T1 is the first frequency feature data. For example, for the reference audio data in the target data set, the audio feature at time T2 corresponds to the image frame at time T2, and the feature data corresponding to the image frame at time T2 is the second frequency feature data.

[0067] In an embodiment of the present disclosure, by associating the Mel-spectrogram cepstral coefficient features of the reference audio data with the corresponding image frames, a mapping relationship dictionary can be created to form a target data set for subsequent search and matching processing.

[0068] For example, based on a recorded video of a speaker, establishing a mapping relationship between the Mel-spectrogram cepstral coefficient features of the audio and the corresponding image frames may include: ensuring that the reference audio data is synchronized with the recorded video of the speaker and establishing a corresponding relationship between the two; extracting the MFCC features of the audio from the recorded video and simultaneously extracting a reference image for each frame; and recording the corresponding image frame for the MFCC features of each audio segment to form a dictionary structure of "MFCC features→image frame" key-value pairs.

[0069] For example, a method for calculating the similarity between first frequency feature data and second frequency feature data may include: representing the first frequency feature data and the second frequency feature data as vectors or matrices, and performing normalization processing so that different audios are comparable in the feature space; calculating the similarity value between the two using algorithms such as cosine similarity or Euclidean distance; comparing the similarity value with a similarity threshold, and when the similarity value is greater than or equal to the similarity threshold, determining the target image from multiple reference images. It is understandable that in the method for determining the similarity between frequency feature data, the audio data to be driven and the reference audio data may correspond to the same target object.

[0070] It can be understood that the above describes an example of how to determine the target image, and the target image determination method will be further described below.

[0071] According to an embodiment of the present disclosure, the target data set also includes second text data corresponding to the reference audio data, and the second text data is obtained by performing speech recognition on the reference audio data; based on the matching degree between the audio data to be driven and the reference audio data, determining the target image from multiple reference images, including: determining first text data corresponding to the audio data to be driven, and the first text data is obtained by performing speech recognition on the audio data to be driven; and determining the target image from multiple reference images based on the matching degree between the first text data and the second text data.

[0072] In the embodiments of the present disclosure, the audio data to be driven and the reference audio data are respectively identified by automatic recognition technology, and the first text data and the second text data can be obtained respectively. It can be understood that the method for identifying audio data can be determined according to actual needs, and is not specifically limited here. In the method for determining the matching degree between text data, the audio data to be driven and the reference audio data can correspond to different target objects.

[0073] For example, the acoustic model and language model can be combined to process audio data to generate text data. The audio data to be driven is preprocessed, and the sampling rate is adjusted, denoised, and framed so that the audio data meets the processing requirements of the model; spectral features and spectrograms are generated through feature extraction; and the spectral features are mapped to phonemes or acoustic units through the acoustic model. On this basis, the language model is used in combination with contextual information to output text data.

[0074] For example, in order to selectively focus on different parts of the audio features and effectively capture key information, the attention mechanism can be used to combine acoustic features and context information to generate text information based on reference audio data. For example, an audio processing library can be used to extract audio features of reference audio data and normalize them, so that the audio data can be encoded into a context vector using an encoder, and the weighted sum between the context vector output by the encoder and the current decoder input can be determined using the attention mechanism, and then the context vector can be combined with the previously generated word to generate the next word, and the text data can be gradually generated in combination with the decoding algorithm.

[0075] According to an embodiment of the present disclosure, the attention mechanism is used to perform recognition processing on audio data to generate text data, which can dynamically focus on important parts of the input sequence, so that the model can selectively focus on different parts of the audio features when generating each output, thereby more effectively capturing key information; at the same time, by introducing contextual information, the attention mechanism can better model the semantic and grammatical structures in the speech, making the generated text more accurate.

[0076] It should be noted that the source language of the audio data and the language of the text data in the present disclosure may be the same or different, and is not limited here. For example, if the audio to be driven is in Chinese and the reference audio is in English, a virtual image driving result in English or a virtual image driving result in Chinese may be generated according to actual needs.

[0077] In the embodiments of the present disclosure, Jaccard Similarity can be used to determine the matching degree between different text data. For example, text data A and text data B are processed to obtain processed text A' and text B', and then the processed text A' and text B' are converted into set a and set b, and the text matching degree between set a and set b is calculated using the Jaccard Similarity Algorithm to obtain a matching degree value, and the matching degree value range is between [0,1], and the larger the value, the higher the matching degree between the texts.

[0078] It can be understood that the above describes an example of how to determine the target image, and the following will provide a preliminary explanation of how the virtual image driving result is generated.

[0079] According to an embodiment of the present disclosure, a target facial three-dimensional model and a target image corresponding to the audio data to be driven are used to generate a virtual image driving result corresponding to the audio data to be driven, including at least one of the following: inputting the target facial three-dimensional model and the target image into a target rendering model to obtain a virtual image driving result; obtaining a target facial image through the target facial three-dimensional model, and superimposing at least part of the target image on the target facial image to obtain a virtual image driving result.

[0080] In an embodiment of the present disclosure, the target facial 3D model and the target image can be input into the target rendering model to output the virtual image driving result. The target rendering model can be a rendering model including an attention mechanism. Those skilled in the art can select a rendering model according to actual optimization needs, and the present disclosure does not limit the type of rendering model.

[0081] For example, the Transformer architecture is used to fuse the target facial 3D model and the target image. The geometric features of the target facial 3D model can be extracted using a graph neural network, the mesh point cloud is converted into voxels to obtain feature vectors, and the features of the target image are extracted using a trained convolutional neural network model; the extracted features are converted into the format of the Transformer input, and the 3D features and image features are converted into vectors of fixed dimensions respectively, each feature includes a position code to attach position information; then the Transformer encoder is used to fuse the 3D features and image features by calculating the attention weight matrix, and the different features are combined together by weighted sum, and the fused features are input into the generative network to generate the virtual image driving result.

[0082] According to the embodiments of the present disclosure, the target facial three-dimensional model and the target image are fused through a rendering model of the attention mechanism, so that the faces of different target objects can be generated using one model, avoiding the problems of low efficiency in personalized model training and application in traditional technologies, while supporting real-time interaction between users and the generation system, enhancing the user experience.

[0083] In the embodiment of the present disclosure, the target facial image may be a two-dimensional image obtained by processing the target facial three-dimensional model. The virtual image driving result may be generated by using an image superposition algorithm.

[0084] For example, according to actual needs, the viewing angle is preset, and the target facial 3D model is rendered into a 2D image using a graphics rendering engine to obtain the target facial image. The facial features (eyes, nose, mouth, etc.) are extracted from the target image using a deep learning algorithm, and the facial feature points are mapped to the corresponding points of the target facial 3D model. The extracted feature points are then used to define the area in the target image that needs to be superimposed on the target facial image, and the superimposed area is determined. The transformation strategy (affine transformation or perspective transformation) is then used to adjust the superimposed area to a size and angle that matches the target facial image, and by setting an appropriate transparency, part of the target image is fused with the target facial image, and weighted averaging or other image fusion techniques are used to superimpose part of the target image with the target facial image, and the final superimposed image is visualized to generate a virtual image driving result.

[0085] According to the embodiments of the present disclosure, by superimposing multiple images, the details and layering of the virtual image can be enhanced, making the virtual image more realistic and vivid; by superimposing target images of different frames, the animation effect of the virtual image can be achieved, making it more vivid visually.

[0086] It is understandable that the present disclosure can also integrate the target rendering model and the overlay algorithm to obtain the virtual image driving result. The integrated algorithm can combine the efficiency advantage of the target rendering model and the flexibility advantage of the overlay algorithm to make the generated virtual image driving result more vivid.

[0087] It can be understood that the above is a preliminary explanation of an example of how to generate a virtual image driving result, and the generation process of the virtual image driving result will be further explained below.

[0088] According to an embodiment of the present disclosure, there are multiple target images, and a target facial three-dimensional model and a target image corresponding to the audio data to be driven are used to generate a virtual image driving result corresponding to the audio data to be driven, including: determining weight information according to the degree of matching between each target image and the audio data to be driven; and generating a virtual image driving result corresponding to the audio data to be driven by using the target facial three-dimensional model, the target image and the weight information.

[0089] In an embodiment of the present disclosure, weight information can be determined based on the degree of match between the target image and the audio data to be driven, and weights can be assigned to different parts of the target facial three-dimensional model (such as facial feature points such as eyes, nose, and mouth) according to the matching results. The degree of match between the target image and the audio data to be driven can be directly proportional to the weight value. The higher the matching degree, the greater the weight value. The degree to which the target image responds to the target facial three-dimensional model during the driving process is set to constrain the generation of the virtual image driving result.

[0090] For example, the target facial 3D model, the target image and the weight information can be used as the input information of the target rendering model, and the target facial 3D model and the target image can be fused using the weight information to make the output virtual image driving result more realistic. For example, the weight information can also be added to the superposition algorithm to constrain the generation of the virtual image driving result using the weight information.

[0091] For example, based on the degree of match between the target image and the audio data to be driven, a weight value can be determined, and the target image can be mapped to each feature point of the target facial 3D model (such as eyes, mouth, eyebrows, etc.) according to the weight value; the feature points of the target facial 3D model can be linearly interpolated using the weight information to achieve smooth deformation; several expressions (such as smile, surprise, etc.) can be pre-defined, and then interpolation can be performed between these mixed shapes based on the weight information; the characteristics of the audio signal (such as pitch, loudness) can be analyzed, and the model's actions can be driven based on these characteristics (when the pitch changes, the degree of opening and closing of the mouth is adjusted; when the volume changes, the tension of the facial expression is adjusted); a timeline can be set to drive the image according to the rhythm and changes of the audio, key frames can be set at the required time points to indicate the state of a specific expression or action, and then intermediate frames can be generated based on the audio data; the driven 3D model can then be rendered in real time to ensure synchronization with the audio, and the final virtual image driving result can be output as needed and displayed in real time in the application.

[0092] According to an embodiment of the present disclosure, by applying weight information to the deformation and animation drive of the target facial three-dimensional model, the performance is kept consistent with the audio data, making the virtual image driving result more realistic and natural.

[0093] It can be understood that the above describes an example of how to generate a virtual image driving result, and the following will describe a method for determining a target data set.

[0094] According to an embodiment of the present disclosure, a target data set is determined in the following manner: a target video corresponding to a target object is acquired; and multiple reference images and reference audio data corresponding to each reference image are determined based on a temporal correspondence between audio data and image data in the target video.

[0095] In the embodiments of the present disclosure, the target object may include the same object as the object in the audio to be driven, or may be an object different from the object in the audio to be driven. The target video may be a video recorded in advance and stored in a file system or a database. There is a temporal correspondence between the audio data and the image data in the target video, and the reference image and the reference audio data may be determined based on the temporal correspondence.

[0096] For example, a common starting point can be selected from the audio data and the image data (the moment when the audio starts playing and the starting frame of the image display). The audio sampling rate and the image frame rate can be set; for reference audio data, the time can be calculated by dividing the sampling point by the sampling rate, and the time corresponding to a sampling point = the number of sampling points / sampling rate. For reference images, the time can be calculated by dividing the number of frames by the frame rate, and the time corresponding to a frame = the number of frames / frame rate; thus, the reference audio data can be analyzed, audio features (such as pitch, loudness, rhythm, etc.) can be extracted, and time tags can be created for these features, and then the time tags of the audio features can be matched with the time frames in the reference image, ensuring that the corresponding reference image is displayed when a specific reference audio data event occurs, thereby achieving synchronous mapping.

[0097] It can be understood that the above describes an example of how to determine a target data set, and the following will describe a method for determining a target facial three-dimensional model.

[0098] According to an embodiment of the present disclosure, the target facial 3D model is determined by the following operations: acquiring facial template data; and adjusting the facial template data based on the audio data to be driven using a deformation control tool to obtain the target facial 3D model.

[0099] In the embodiments of the present disclosure, the facial template data may be 3D template data of any lip shape stored in advance in a specific file system or database, or may be generated in real time using a facial template model. The deformation control tool may be a deformation control model or an animation controller, and the target facial 3D model may be obtained by adjusting the facial template data based on the audio data to be driven using the deformation control model or the animation controller.

[0100] For example, the audio data to be driven is analyzed to extract audio features (pitch, volume and spectral information), and speech recognition or sound classification is performed to obtain specific phonemes or words; the audio signal is then mapped to the corresponding lip shape state through the deformation model; the acquired facial template data is used to drive the lip shape animation to the 3D model using facial expression technology, and the shape interpolation method can be used to convert the lip shape state into the corresponding 3D mesh deformation; for each lip shape state, the feature point position of the 3D model is adjusted to achieve a dynamic effect, and a 3D graphics engine can be used for real-time rendering to ensure that the model changes are synchronized with the audio playback, and texture mapping and lighting effects can be added to improve the visual effect, thereby obtaining a three-dimensional model of the target face.

[0101] Figure 3 An example schematic diagram of a virtual image driving result generation process according to an embodiment of the present disclosure is shown.

[0102] like Figure 3As shown, in 300, a video of the target object can be recorded. The resolution and duration information of the video can be determined according to actual needs. The recorded video is defined as a reference video 310. Based on the time correspondence, a mapping relationship dictionary between the second frequency feature data 321 of the reference audio data in the reference video 310 and the reference image 322 can be constructed to obtain a target data set 320.

[0103] The acquired audio data to be driven 330 is subjected to feature extraction to obtain first frequency feature data 340, and the first frequency feature data 340 is input into the audio driven lip shape model 350 to generate the target facial three-dimensional model 360. At the same time, based on the matching degree between the first frequency feature data 340 of the current audio data to be driven 330 and the second frequency feature data 321 corresponding to the reference audio data, the reference image 322 having the greatest similarity to the frequency feature data of the audio data to be driven 330 is determined as the target image 370.

[0104] After obtaining the target facial 3D model 360 and the target image 370 , a target rendering model 380 including an attention mechanism can be used to generate a virtual image driving result 390 based on the target facial 3D model 360 and the target image 370 .

[0105] Based on the above virtual image driving method, the present disclosure also provides a virtual image driving device. Figure 4 The device is described in detail.

[0106] Figure 4 A block diagram of a virtual image driving device according to an embodiment of the present disclosure is shown.

[0107] like Figure 4 As shown, the virtual image driving device 400 includes a target image determination module 410 and a driving result generation module 420 .

[0108] According to some embodiments of the present disclosure, the virtual image driving device 400 may be used to implement the virtual image driving method of the embodiments of the present disclosure.

[0109] The target image determination module 410 may perform, for example, operation S210, for determining a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven, wherein the target data set includes a plurality of reference images for the face of the target object, and the target image is at least one of the plurality of reference images.

[0110] The driving result generating module 420 may perform, for example, operation S220 for generating a virtual image driving result corresponding to the audio data to be driven by using the target facial three-dimensional model and the target image corresponding to the audio data to be driven.

[0111] For example, the target image determination module 410 and the drive result generation module 420 can be combined in one module, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the target image determination module 410 and the drive result generation module 420 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in a proper combination of software, hardware and firmware. Alternatively, at least one of the target image determination module 410 and the drive result generation module 420 can be at least partially implemented as a computer program module, and when the program is run by a computer, the function of the corresponding module can be executed.

[0112] According to an embodiment of the present disclosure, the target data set includes reference audio data corresponding to each reference image; the target image determination module 410 includes: a target image determination submodule, which is used to determine the target image from multiple reference images based on the matching degree between the audio data to be driven and the reference audio data.

[0113] According to an embodiment of the present disclosure, the target image determination submodule includes: a feature data determination unit and a target image determination unit. The feature data determination unit is used to determine the first frequency feature data corresponding to the audio data to be driven and the second frequency feature data corresponding to the reference audio data; the target image determination unit is used to determine the target image from multiple reference images based on the similarity between the first frequency feature data and the second frequency feature data.

[0114] According to an embodiment of the present disclosure, the target data set also includes second text data corresponding to the reference audio data, and the second text data is obtained by performing speech recognition on the reference audio data; the target image determination submodule includes: a text data determination unit and an image determination unit. The text data determination unit is used to determine the first text data corresponding to the audio data to be driven, and the first text data is obtained by performing speech recognition on the audio data to be driven. The image determination unit is used to determine the target image from multiple reference images based on the matching degree between the first text data and the second text data.

[0115] According to an embodiment of the present disclosure, the driving result generation module 420 includes at least one of the following: an input submodule and an overlay submodule. The input submodule is used to input the target facial 3D model and the target image into the target rendering model to obtain the virtual image driving result; the overlay submodule is used to obtain the target facial image through the target facial 3D model, and overlay at least part of the target image on the target facial image to obtain the virtual image driving result.

[0116] According to an embodiment of the present disclosure, there are multiple target images, and the driving result generation module 420 includes: a weight information determination submodule and an image driving result generation submodule. The weight information determination submodule is used to determine the weight information according to the matching degree between each target image and the audio data to be driven; the image driving result generation submodule is used to generate a virtual image driving result corresponding to the audio data to be driven by using the target facial three-dimensional model, the target image and the weight information.

[0117] According to an embodiment of the present disclosure, a target data set is determined in the following manner: a target video corresponding to a target object is acquired; and multiple reference images and reference audio data corresponding to each reference image are determined based on a temporal correspondence between audio data and image data in the target video.

[0118] According to an embodiment of the present disclosure, the target facial 3D model is determined by the following operations: acquiring facial template data; and adjusting the facial template data based on the audio data to be driven using a deformation control tool to obtain the target facial 3D model.

[0119] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0121] Figure 5 A schematic block diagram of an example electronic device that can be used to implement the method of the embodiment of the present disclosure is shown.

[0122] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0124] A number of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the virtual image driving method. For example, in some embodiments, the virtual image driving method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the virtual image driving method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the virtual image driving method in any other appropriate manner (e.g., by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0132] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0133] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A method for driving a virtual image, comprising: Determining a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven, wherein the target data set includes a plurality of reference images for a face of a target object, and the target image is at least one of the plurality of reference images; A virtual image driving result corresponding to the audio data to be driven is generated by using the target facial three-dimensional model corresponding to the audio data to be driven and the target image.

2. The method according to claim 1, wherein: The target data set includes reference audio data corresponding to each of the reference images; The step of determining a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven comprises: The target image is determined from the plurality of reference images based on a matching degree between the to-be-driven audio data and the reference audio data.

3. The method according to claim 2, wherein: The determining the target image from the plurality of reference images based on the matching degree between the to-be-driven audio data and the reference audio data comprises: Determining first frequency characteristic data corresponding to the audio data to be driven, and second frequency characteristic data corresponding to the reference audio data; The target image is determined from the plurality of reference images based on the similarity between the first frequency feature data and the second frequency feature data.

4. The method according to claim 2, wherein: The target data set also includes second text data corresponding to the reference audio data, where the second text data is obtained by performing speech recognition on the reference audio data; The determining the target image from the plurality of reference images based on the matching degree between the to-be-driven audio data and the reference audio data comprises: Determining first text data corresponding to the audio data to be driven, wherein the first text data is obtained by performing speech recognition on the audio data to be driven; as well as The target image is determined from the plurality of reference images based on a degree of matching between the first text data and the second text data.

5. The method according to claim 1, wherein: The step of using the target facial three-dimensional model and the target image corresponding to the audio data to be driven to generate a virtual image driving result corresponding to the audio data to be driven includes at least one of the following: Inputting the target facial 3D model and the target image into a target rendering model to obtain the virtual image driving result; A target facial image is obtained through the target facial three-dimensional model, and at least a portion of the target image is superimposed on the target facial image to obtain the virtual image driving result.

6. The method according to claim 1, wherein: There are multiple target images, and using the target facial three-dimensional model corresponding to the audio data to be driven and the target image to generate a virtual image driving result corresponding to the audio data to be driven includes: Determining weight information according to the matching degree between each of the target images and the audio data to be driven; The target facial three-dimensional model, the target image and the weight information are used to generate a virtual image driving result corresponding to the audio data to be driven.

7. The method according to claim 1, wherein: The target data set is determined in the following manner: Acquire a target video corresponding to the target object; The multiple reference images and reference audio data corresponding to each of the reference images are determined based on a temporal correspondence between audio data and image data in the target video.

8. The method according to claim 1, wherein: The target facial 3D model is determined using the following operations: Get facial template data; as well as The facial template data is adjusted based on the audio data to be driven by using a deformation control tool to obtain the target facial three-dimensional model.

9. A virtual image driving device, comprising: a target image determination module, configured to determine a target image corresponding to the audio data to be driven from a target data set based on the audio data to be driven, wherein the target data set includes a plurality of reference images for a target object's face, and the target image is at least one of the plurality of reference images; and The driving result generating module is used to generate a virtual image driving result corresponding to the audio data to be driven by using the target facial three-dimensional model corresponding to the audio data to be driven and the target image.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images