Digital human video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention is suitable for the technical field of digital people, and provides a digital people video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the data features of multi-modal data; the data features of the multi-modal data comprise a first image feature of the image, a first audio feature of the audio and a first point cloud feature of the lip point cloud data; generating a second audio feature and a second point cloud feature fusing the correlation between the first audio feature and the first point cloud feature according to the first image feature; a digital human video is generated based on the first image feature, the second audio feature, and the second point cloud feature. By adopting the method, the visual quality and the audio and lip synchronism in the digital human video can be improved.

Description

Technical Field

[0001] The present application belongs to the field of digital human technology, and in particular, relates to a method, device, electronic device and storage medium for generating digital human videos. Background Art

[0002] Audio-driven digital human video is to generate a digital human video that matches the identity of the person in the original video and is synchronized with the driving audio given an original video and driving audio. With the development of computer vision and multimedia, audio-driven digital human video plays an increasingly important role in virtual reality, filmmaking, online conferencing and other fields.

[0003] Currently, Neural Radiance Field (NeRF) is mostly used to provide a new representation method for digital human reconstruction, so as to map spatial coordinates into color and density through neural network to render and generate high-precision digital human images.

[0004] However, due to the limited size of training data, the neural radiation field trained based on limited training data still has flaws in lip movement, facial details and stability in digital human videos. For example, it is impossible to improve visual quality and lip-sync (audio and lip movement). Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and storage medium for generating a digital human video, which can solve the problems of low visual quality and poor lip-audio (audio and lip movement) synchronization.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a digital human video, the method comprising:

[0007] Acquire data features of the multimodal data; the data features of the multimodal data include a first image feature of the image, a first audio feature of the audio, and a first point cloud feature of the lip point cloud data;

[0008] Generate, according to the first image feature, a second audio feature and a second point cloud feature that fuses the correlation between the first audio feature and the first point cloud feature;

[0009] A digital human video is generated based on the first image feature, the second audio feature, and the second point cloud feature.

[0010] In a second aspect, an embodiment of the present application provides a digital human video generation device, the device comprising:

[0011] An acquisition module, used to acquire data features of multimodal data; the data features of the multimodal data include a first image feature of an image, a first audio feature of an audio, and a first point cloud feature of lip point cloud data;

[0012] A first generating module, configured to generate, according to the first image feature, a second audio feature and a second point cloud feature by integrating the correlation between the first audio feature and the first point cloud feature;

[0013] The second generating module is used to generate a digital human video based on the first image feature, the second audio feature and the second point cloud feature.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the first aspect described above when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the first aspect described above is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the electronic device can obtain data features of multimodal data. For example, the first image feature of the image, the first audio feature of the audio, and the first point cloud feature of the lip point cloud data. Then, based on the first image feature, the second audio feature and the second point cloud feature that fuse the correlation between the first audio feature and the first point cloud feature are constructed. In this way, a digital human video can be generated based on the first image feature, the second audio feature, and the second point cloud feature with correlation. Using the above method, when generating a digital human video, the data features of the multimodal data including the lip point cloud data are collected as additional attachment conditions, and the audio corresponding to the first audio feature is synchronized with the lip point cloud data corresponding to the first point cloud feature through the same first image feature. Furthermore, the electronic device can understand the correlation between cross-modalities to generate the above-mentioned second audio feature and the second point cloud feature with correlation. Based on this, based on the first image feature with correlation, the second audio feature and the second point cloud feature corresponding to the lips, the image quality of the finally generated digital human image can be improved, and the lip synchronization can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a flow chart of a method for generating a digital human video according to an embodiment of the present application;

[0020] Figure 2 This is a flow chart of generating a second audio feature and a second point cloud feature in a digital human video generation method provided in an embodiment of the present application;

[0021] Figure 3 is a flow chart of generating a second audio feature and a second point cloud feature in a digital human video generation method provided by another embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of an application scenario of generating a digital human image in a digital human video generation method provided in an embodiment of the present application;

[0023] Figure 5 It is a structural schematic diagram of a digital human video generating device provided by an embodiment of the present application;

[0024] Figure 6 It is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0027] It should be noted that the information collection process (such as face image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is performed with the user's knowledge and permission, that is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0028] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0029] Audio-driven digital human video is to generate a digital human video that matches the identity of the person in the original video and is synchronized with the driving audio given an original video and driving audio. With the development of computer vision and multimedia, audio-driven digital human video plays an increasingly important role in virtual reality, filmmaking, online conferencing and other fields.

[0030] Currently, Neural Radiance Field (NeRF) is mostly used to provide a new representation method for digital human reconstruction, so as to map spatial coordinates into color and density through neural network to render and generate high-precision digital human images.

[0031] However, due to the limited size of training data, the neural radiation field trained based on limited training data still has flaws in lip movement, facial details and stability in digital human videos. For example, it is impossible to improve visual quality and lip-sync (audio and lip movement).

[0032] Based on this, in order to improve the visual quality and lip synchronization in digital human videos, an embodiment of the present application provides a method for generating digital human videos. The method can be applied to electronic devices such as tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), and netbooks. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0033] See also Figure 1 , Figure 1 The following is a flowchart of a method for generating a digital human video according to an embodiment of the present invention. The method comprises the following steps:

[0034] S101, obtaining data features of multimodal data; the data features of the multimodal data include a first image feature of an image, a first audio feature of an audio, and a first point cloud feature of lip point cloud data.

[0035] In one embodiment, the multimodal data includes but is not limited to two or more of text, image, audio, and lip point cloud data, and is not limited to this. In this embodiment, the multimodal data is explained as image, audio, and lip point cloud data.

[0036] Among them, the electronic device can obtain the above-mentioned multimodal data from the preset audio and video, and then extract the data features of the multimodal data.

[0037] For example, for audio, the electronic device may sample the audio data using a preset audio sampling rate to obtain the above audio. The audio sampling rate is the frequency at which the sound signal is sampled when collecting and processing the sound signal in digital audio, and may be expressed in Hertz (Hz).

[0038] The preset audio sampling rate can be set according to actual conditions and is not limited thereto. For example, the preset audio sampling rate can be 16000 Hz.

[0039] In one embodiment, after obtaining the audio, the electronic device may extract the first feature of the audio according to a preset sliding window strategy. Specifically, the electronic device may divide the signal corresponding to the audio into multiple short-time segments, each segment being a frame. Each frame is processed using a specific algorithm (e.g., Fourier transform), or the signal corresponding to the audio is encoded using an audio encoder to obtain the first audio feature corresponding to each frame segment, which is not limited to this.

[0040] The time length of each short segment can be set according to actual conditions. For example, the time length of each short segment can be 20 ms.

[0041] In another embodiment, after obtaining the first audio feature corresponding to each frame segment, the electronic device may also use a preset automatic speech recognition module (HuBert) to predict the classification label of each frame segment, and smooth the classification label using a one-dimensional convolution. The predicted classification label can be used to identify the segment so that it is synchronized with the image collected from the audio and video in a one-to-one correspondence. In addition, the lip point cloud data subsequently extracted from the audio corresponds to the image, thereby improving the fusion effect of the image features of the subsequent image and the first point cloud features of the lip point cloud data.

[0042] For images, the electronic device can extract frames of audio and video at a preset frame extraction frequency to obtain the above-mentioned images. Afterwards, for each frame of the image, the image can be semantically segmented and the background image can be extracted. Then, the facial key points in the image can be identified using a face recognition method to generate the head pose of the image.

[0043] In one embodiment, the preset frame frequency can be set according to actual conditions, and there is no limitation on this. Exemplarily, the preset frame frequency can be 25fps. In addition, the image can be semantically segmented using a model structure such as BiSeNet or U-Net. In addition, the background image can be extracted using a nearest neighbor method or a background difference method. At the same time, 68 facial key points in the image can be extracted using methods such as face alignment or local binary pattern histograms (LBHP) to estimate the head pose of the face in each frame of the image. It should be noted that the head pose can be used to generate digital human images later, or to supervise the generated image frames during the training process.

[0044] When generating the first image feature, a neural network model that can process images (for example, a convolutional neural network, a recurrent neural network, and other models) can be used to extract the first image feature.

[0045] However, in this embodiment, since the extracted first image features are used to generate a digital human video in a subsequent step, it is necessary to render the neural radiation field. Therefore, when generating the first image features, the preset ray sampling points can be used for feature extraction.

[0046] Exemplarily, according to the preset ray sampling points, the image can be projected on three planes to obtain projection values ​​corresponding to the three planes, and then the first image feature is generated based on the three projection values.

[0047] In one embodiment, the three-plane projections are projections on the X-axis, Y-axis, and Z-axis directions. For the preset ray sampling points, their coordinates can be pre-set, and there is no limitation on this. For example, the coordinate representation of the ray sampling point can be: S = (x, y, z) ∈ R XYZ .

[0048] Among them, the projection value of the image projected on any plane can be feature processed by the two-dimensional multi-resolution hash encoder corresponding to each plane to generate the output feature corresponding to the plane. Finally, the electronic device can splice the output features of the three planes to obtain the first image feature corresponding to the image.

[0049] Exemplarily, the first image feature can be expressed as follows:

[0050]

[0051] Among them, F c represents the first image feature, represents the output features projected onto the XY plane, represents the output features projected onto the YZ plane, Represents the output features projected onto the XZ plane.

[0052] It should be added that the use of three-plane projection to represent the first image feature can reduce the number of hash collisions when the multi-resolution hash encoder performs feature processing and improve data processing efficiency.

[0053] For the lip point cloud data, the electronic device can identify the lip key points from the image to generate the lip point cloud data based on the lip key points. In another embodiment, the electronic device can also generate mesh data using an audio generation mesh method. Then, based on the topological structure, the point cloud of the lip area is extracted separately from the mesh data to obtain the above lip point cloud data.

[0054] Based on the above description, it can be considered that when acquiring multimodal data, audio and image can be acquired first, and then lip point cloud data can be generated based on the audio. For example, corresponding lip point cloud data can be generated from each frame of audio.

[0055] After obtaining the lip point cloud data, the ISS (Intrinsic Shape Signature) model can be used for feature extraction, or a multi-resolution hash network can be used to encode the lip point cloud data to generate a first point cloud feature, which is not limited. In this embodiment, the electronic device can encode the point cloud using a multi-resolution hash network.

[0056] It should be noted that, during the encoding process, the encoding dimension of the multi-resolution hash network can be set according to actual conditions and is not limited thereto. However, in order to avoid a large amount of computational load, the above encoding dimensions can be limited to three.

[0057] In another embodiment, during the training process, due to the dynamics of the digital human video generation process, only capturing the first point cloud feature of each frame is not enough to effectively guide the generation process of each frame of the digital human image in the digital human video. Therefore, during the training process, after obtaining the first point cloud feature in the above manner, a dynamic differential encoder can also be used to take the difference between the first point cloud features of adjacent frames, and then splice all the differences to obtain the fifth point cloud feature (only used in the training process, when actually generating the digital human video, the first point cloud feature can be used to perform the subsequent steps) to participate in the training of the neural radiation field.

[0058] In the training process, the fifth point cloud feature corresponding to the lip point cloud data, the first audio feature corresponding to the audio, and the first image feature corresponding to the image need to be synchronized. At this time, the electronic device can establish the corresponding relationship between the above three based on a cross-modal contrast learning strategy, which is not described in detail. For example, the final representation of the fifth point cloud feature can be:

[0059]

[0060] Among them, F p represents the fifth point cloud feature, represents the first point cloud feature of the t+1th frame, and T represents the number of first point cloud features.

[0061] S102: Generate, based on the first image feature, a second audio feature and a second point cloud feature that fuse the correlation between the first audio feature and the first point cloud feature.

[0062] In one embodiment, the electronic device may concatenate or dot-product the first audio feature with the first image feature to obtain the second audio feature; and concatenate or dot-product the first point cloud feature with the first image to obtain the initial point cloud feature, and then concatenate or dot-product the initial point cloud feature with the second audio feature to obtain the second point cloud feature. In this way, a second audio feature and a second point cloud feature having correlation are generated.

[0063] In another embodiment, in order to be able to strengthen the audio feature (i.e., make the second audio feature more prominent in the audio-related part) and the point cloud feature (i.e., make the second point cloud feature more prominent in the lip-related part) respectively during the process of generating the second audio feature and the second point cloud feature, the electronic device may also perform the following steps: Figure 2 The steps S201-S203 shown are processed. The details are as follows:

[0064] S201: Generate a global image feature describing global information of an image based on a first image feature.

[0065] In one embodiment, the above-mentioned global image information mainly refers to the overall features that can describe the image or target (face), including features such as color, shape, texture, and spatial relationship. Therefore, it can be considered that the global image features can describe the global image information of the image or target.

[0066] Among them, the global information of the image usually comes from the deep network. As the network deepens, the receptive field becomes larger, making the global information of the feature map obtained by the deep network richer. Based on this, the electronic device can use a multi-layer network to capture the global information of the spatial geometry, or use global pooling to capture the global information of the image to generate global image features. In this embodiment, the method of generating global image features is not limited.

[0067] As an example, the electronic device may use a multi-layer network to capture global information of spatial geometry. Specifically, the multi-layer network may be an MLP (Multilayer Perceptron). For example, the multi-layer network may be a two-layer MLP.

[0068] It should be noted that when generating global image features that describe the global information of the image based on the first image features, different two-layer MLPs can be used to generate different global image features to perform dot products with the subsequent third audio features and the third point cloud features, respectively; or, the same two-layer MLP can be used to generate global image features, and there is no limitation on this.

[0069] S202: Generate a second audio feature according to the first audio feature and the global image feature.

[0070] In one embodiment, the above-mentioned method of generating the second audio feature includes but is not limited to concatenation or dot product, which is not limited to this. In this embodiment, the second audio feature is obtained by performing dot product between the first audio feature and the global image feature.

[0071] It should be noted that, in order to make the audio-related part of the second audio feature more prominent, the electronic device may first compress the first audio feature to obtain a third audio feature, and then perform a dot product between the third audio feature and the global image feature to generate the second audio feature.

[0072] Specifically, the electronic device may process the first audio feature using a self-attention mechanism to obtain a third audio feature. The self-attention mechanism is a mechanism for modeling sequence data, which allows the model to focus on relevant parts when processing input data (e.g., the second audio feature) to improve the performance and generalization ability of the model.

[0073] Based on this, it can be considered that the third audio feature generated after the self-attention mechanism is processed can better highlight the audio-related part. Furthermore, when the third audio is dot-producted with the global image feature to generate the second audio feature, the generated second audio feature can still highlight the audio-related features after fusing the image information.

[0074] As an example, the representation of the second audio feature may be generated as follows:

[0075]

[0076] in, Represents the second audio feature, MLP a represents a multi-layer MLP, F c represents the first image feature, SA represents the self-attention mechanism, and F a Represents a first audio feature.

[0077] S203: Generate a second point cloud feature according to the first point cloud feature, the global image feature and the second audio feature.

[0078] In one embodiment, when generating the second point cloud feature, the second audio feature generation method may also be referred to, that is, the first point cloud feature is first compressed to obtain the third point cloud feature, and then the third point cloud feature is dot-producted with the global image feature to obtain the second point cloud feature.

[0079] However, since the above-mentioned lip point cloud data is data generated based on audio. Therefore, in order to make the generated second point cloud features also include audio-related parts. The electronic device can compress the first point cloud features to obtain the third point cloud features, and then perform dot product between the third point cloud features and the global image features to generate intermediate point cloud features. At this time, the generated intermediate point cloud features can still highlight the audio-related features after fusing the image information. Then, the electronic device can generate a second point cloud feature based on the intermediate point cloud features and the second audio features, so that the final generated second point cloud features fuse the relevant parts of multimodal data such as lips, audio and images.

[0080] In another embodiment, in order to enable the second point cloud feature to also highlight the audio-related part, when generating the second point cloud feature according to the intermediate point cloud feature and the second audio feature, the electronic device may first generate a global audio feature describing the global audio information based on the second audio feature, so as to enhance the audio information in the second audio feature by capturing the global audio information. Finally, the intermediate point cloud feature is dot-producted with the global audio feature to obtain the second point cloud feature.

[0081] Among them, the above-mentioned method of compressing the first point cloud feature is similar to the method of compressing the first audio feature, and both can be obtained by processing with the self-attention mechanism. And the method of generating the global audio feature based on the second audio feature can be similar to the method of generating the global image feature based on the first image feature, and both can be obtained by processing with a multi-layer network (for example, a two-layer MLP), which will not be described in detail. In this embodiment, there is no limitation on the method of obtaining the third audio feature and the global audio feature.

[0082] As an example, see Figure 3 , Figure 3 This is a flowchart of generating a second audio feature and a second point cloud feature in a digital human video generation method provided by another embodiment of the present application. In order to establish the correlation between the corresponding first audio feature and the first point cloud feature, different MLPs can be used respectively. a and MLP b The first image feature is processed to obtain two global image features. For example, Figure 3The global image features a and b in .

[0083] Then, for the second audio feature, in order to make the second audio feature more prominent in the audio-related part, the self-attention mechanism can be used to process the first audio feature to obtain the third audio feature. Finally, the third audio feature is dot-producted with the global image feature a to generate the second audio feature fused with the image information.

[0084] And, for the third audio feature, the self-attention mechanism can also be used to process the first point cloud feature to obtain the third point cloud feature to highlight the lip-related part. Then, the third point cloud feature is dot-producted with the global image feature to generate an intermediate point cloud feature that incorporates image information. Afterwards, in order to fuse the features corresponding to the audio-related part again, MLPc can be used to extract the global audio features of the audio from the second audio feature. Finally, the intermediate point cloud feature is dot-producted with the global audio feature to obtain the second point cloud feature. In this way, the second point cloud feature finally generated can not only fuse the relevant parts of multimodal data such as lips, audio and images, but also highlight the parts related to the lips and audio.

[0085] S103: Generate a digital human video based on the first image feature, the second audio feature, and the second point cloud feature.

[0086] In one embodiment, after obtaining the first image feature, the second audio feature, and the second point cloud feature, the multiple features can be spliced ​​together, and a digital human video can be rendered using a preset neural radiation field. However, simple splicing alone is usually not enough to effectively fuse the above multiple cross-modal features. As a result, the digital human video rendered using the neural radiation field still has defects.

[0087] Based on this, in this embodiment, in order to generate a high-quality digital human video, the second audio feature and the second point cloud feature can be affine transformed to obtain a translation vector and a scaling vector. Then, the second audio feature and the second point cloud feature are transformed based on the translation vector and the scaling vector to obtain a fourth audio feature and a fourth point cloud feature. Finally, the first image feature, the fourth audio feature, and the fourth point cloud feature are input into the implicit function of the preset neural radiation field to generate a digital human video.

[0088] The purpose of performing affine transformation and other processing on the second audio feature and the second point cloud feature to obtain the fourth audio feature and the fourth point cloud feature is to help the model adjust the lip movements of the digital human through the calculated translation vector and scaling vector, so that the lip movements in the digital human video finally generated based on the fourth point cloud feature are synchronized with the audio signal corresponding to the fourth audio feature, thereby creating a more realistic virtual interactive experience. That is, to improve the quality of the digital human video and the lip synchronization.

[0089] In one embodiment, the electronic device may calculate the affine transformation according to the Adaptive Affine Transformation (AdaAT) operator to obtain the translation vector and the scaling vector, and apply the translation vector and the scaling vector to the second audio feature and the second point cloud feature to obtain the fourth audio feature and the fourth point cloud feature.

[0090] In another embodiment, in the neural radiant field technology, MLP can learn the audio-video cross-modal relationship. Based on this, in this embodiment, the cross-modal relationship between the lip points in the audio-image can also be learned through the MLP network to establish a connection between the audio features and the spatial area corresponding to the lips through the attention mechanism.

[0091] Furthermore, MLP can also be used to process and predict parameters such as the position, rotation, scaling, color, and translation of the digital human. Therefore, when generating a translation vector and a scaling vector, a preset MLP can also be used for processing to obtain the translation vector and the scaling vector. Then, the translation vector and the scaling vector are input into the adaptive MLP for processing to generate the fourth audio feature and the fourth point cloud feature.

[0092] When the translation vector and the scaling vector are input into the adaptive MLP, the first image feature can also be input into the adaptive MLP for synchronous processing to adjust the facial movements and expressions of the digital human in the first image feature. This ensures that the movements and expressions of the digital human in the final generated digital human video can be displayed naturally and realistically.

[0093] In one embodiment, the preset neural radiation field is a neural network that can reconstruct complex three-dimensional scenes from a partial two-dimensional image set to make digital interactions more realistic and accurate.

[0094] Specifically, NeRF is a neural network that uses MLP to construct a representation of a 3D scene, and is trained to map spatial coordinates and viewing directions to color and density values ​​to reconstruct a digital human image. Based on this, NeRF can determine the color and density value of each point in the digital human image based on a series of mathematical structures (implicit functions) output by the above adaptive MLP.

[0095] Specifically, the implicit function can be expressed as follows:

[0096]

[0097] Wherein, x represents the preset three-dimensional spatial coordinate of the ray sampling point, d represents the preset viewing direction of the ray sampling point, represents the fourth audio feature, Represents the fourth point cloud feature, Fc represents the first image feature, c represents the color information in the digital human image, and σ represents the density information in the digital human image.

[0098] In one embodiment, the density information may indicate the degree of absorption or scattering of light when passing through the standard point, and the color information may indicate the display color of the standard point, wherein the standard point includes but is not limited to key points of the head in areas such as eyes, nose, and lips.

[0099] It should be noted that, according to the above step S101, there will be multiple frames of images, audio and corresponding lip point cloud data. And, after processing each frame of image, audio and corresponding lip point cloud data according to the above steps, a frame of digital human image can be obtained. Based on this, the combination of multiple frames of digital human images and audio is the above digital human video.

[0100] In this embodiment, the electronic device can obtain data features of multimodal data. For example, the first image feature of the image, the first audio feature of the audio, and the first point cloud feature of the lip point cloud data. Then, based on the first image feature, the second audio feature and the second point cloud feature that fuse the correlation between the first audio feature and the first point cloud feature are constructed. In this way, a digital human video can be generated based on the first image feature, the second audio feature, and the second point cloud feature with correlation. Using the above method, when generating a digital human video, the data features of the multimodal data including the lip point cloud data are collected as additional attachment conditions, and the audio corresponding to the first audio feature is synchronized with the lip point cloud data corresponding to the first point cloud feature through the same first image feature. Furthermore, the electronic device can understand the correlation between cross-modalities to generate the above-mentioned second audio feature and the second point cloud feature with correlation. Based on this, based on the first image feature with correlation, the second audio feature and the second point cloud feature corresponding to the lips, the image quality of the finally generated digital human image can be improved, and the lip synchronization can be guaranteed.

[0101] In another embodiment, referring to Figure 4 , Figure 4 This is a schematic diagram of an application scenario for generating a digital human image in a digital human video generation method provided by an embodiment of the present application. First, for the image after frame extraction, the image can be projected into three planes according to the preset ray sampling points to obtain a three-plane representation of the image (i.e., the projection values ​​corresponding to the three planes). Then, the projection values ​​corresponding to the three planes are subjected to feature processing by a two-dimensional multi-resolution hash encoder to generate the output features corresponding to the plane, and the output features of the three planes are spliced ​​to obtain the first image features corresponding to the image. That is, Figure 4 F c .

[0102] in, Figure 4 The position of the camera can be considered as the position of the preset ray sampling point.

[0103] And, for audio, an audio encoder may be used to encode the audio signal to obtain the first audio feature. That is, Figure 4 F a .

[0104] And, for the lip point cloud data, the audio generation mesh method can be used to generate mesh data (ie, Figure 4 Then, based on the topological structure, the point cloud of the lip area can be extracted from the mesh data to obtain the above lip point cloud data. Finally, a multi-resolution hash network is used to encode the lip point cloud data to generate the first point cloud feature.

[0105] In the training process, due to the dynamics of the digital human video generation process, only capturing the first point cloud feature of each frame is not enough to effectively guide the generation process of each frame of the digital human image in the digital human video. Therefore, after obtaining the first point cloud feature, a dynamic differential encoder can also be used to take the difference between the first point cloud features of adjacent frames, and then splice all the differences to obtain the fifth point cloud feature. That is, Figure 4 F p It should be noted that since this embodiment does not describe the embodiment in the training process. Figure 5 F p Indicates that the first point cloud feature participates in the subsequent interpretation.

[0106] Afterwards, after obtaining the first image feature, the first audio feature, and the first point cloud feature, the audio point cloud enhancement process can be performed to obtain the second audio feature and the second point cloud feature. Specifically, the global image feature of the first image feature can be obtained first, and the first audio feature and the first point cloud feature can be processed by the self-attention mechanism respectively to obtain the third audio feature and the third point cloud feature. Then, the third audio feature is dot-producted with the global image feature to obtain the second audio feature. That is, Figure 4 In

[0107] And, dot product the third point cloud feature with the global image feature to obtain an intermediate point cloud feature. And, extract the global audio feature of the audio from the second audio feature. Finally, dot product the intermediate point cloud feature with the global audio feature to obtain a second point cloud feature. That is, Figure 4 In

[0108] Then, MLP is used to perform affine transformation calculation on the second audio feature and the second point cloud feature to generate a translation vector (i.e., Figure 4) and the scaling vector (i.e., Figure 4 After that, the translation vector and the scaling vector can be input into the adaptive MLP for transformation to generate the fourth audio feature (not shown in the figure) and the fourth point cloud feature (not shown in the figure).

[0109] Finally, the neural radiation field can be based on the preset three-dimensional spatial coordinates of the ray sampling point and the preset viewing direction, the fourth audio feature Fourth point cloud feature And the first image feature F c , generating a representation of the color information in the digital human image (i.e., Figure 4 c) and density information (i.e., Figure 4 σ in , and render based on the color information and density information to obtain a digital human image.

[0110] See also Figure 5 , Figure 5 1 is a structural block diagram of a digital human video generation device provided in an embodiment of the present application. The digital human video generation device in this embodiment includes modules for executing Figure 1 and Figure 2 For details, please refer to the steps in the corresponding embodiment. Figure 1 and Figure 2 as well as Figure 1 and Figure 2 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 5 , the digital human video generating device 500 may include: an acquisition module 510, a first generating module 520 and a second generating module 530, wherein:

[0111] The acquisition module 510 is used to acquire data features of multimodal data; the data features of the multimodal data include a first image feature of the image, a first audio feature of the audio, and a first point cloud feature of the lip point cloud data.

[0112] The first generating module 520 is used to generate, according to the first image feature, a second audio feature and a second point cloud feature by integrating the correlation between the first audio feature and the first point cloud feature.

[0113] The second generating module 530 is used to generate a digital human video based on the first image feature, the second audio feature and the second point cloud feature.

[0114] In one embodiment, the acquisition module 510 is further configured to:

[0115] Acquire audio and images, and generate lip point cloud data based on the audio; perform three-plane projection on the image according to preset ray sampling points, obtain the projection values ​​corresponding to the three planes, and generate a first image feature based on the three projection values; extract the first audio feature of the audio according to a preset sliding window strategy; use a multi-resolution hash network to encode the lip point cloud data to generate a first point cloud feature.

[0116] In one embodiment, the first generating module 520 is further configured to:

[0117] A global image feature describing global information of the image is generated based on the first image feature; a second audio feature is generated based on the first audio feature and the global image feature; and a second point cloud feature is generated based on the first point cloud feature, the global image feature and the second audio feature.

[0118] In one embodiment, the first generating module 520 is further configured to:

[0119] The first audio feature is compressed to obtain a third audio feature; and the third audio feature is dot-producted with the global image feature to generate a second audio feature.

[0120] In one embodiment, the first generating module 520 is further configured to:

[0121] The first point cloud feature is compressed to obtain a third point cloud feature; the third point cloud feature is dot-producted with the global image feature to generate an intermediate point cloud feature; and a second point cloud feature is generated according to the intermediate point cloud feature and the second audio feature.

[0122] In one embodiment, the first generating module 520 is further configured to:

[0123] A global audio feature describing the global audio information is generated based on the second audio feature; and a dot product is performed between the intermediate point cloud feature and the global audio feature to obtain a second point cloud feature.

[0124] In one embodiment, the second generating module 530 is further configured to:

[0125] Affine transformation is performed on the second audio feature and the second point cloud feature to obtain a translation vector and a scaling vector; the second audio feature and the second point cloud feature are transformed based on the translation vector and the scaling vector to obtain a fourth audio feature and a fourth point cloud feature; the first image feature, the fourth audio feature and the fourth point cloud feature are input into an implicit function of a preset neural radiation field to generate a digital human video; the implicit function is expressed as follows:

[0126]

[0127] Wherein, x represents the preset three-dimensional spatial coordinate of the ray sampling point, d represents the preset viewing direction of the ray sampling point, represents the fourth audio feature, Represents the fourth point cloud feature, F c represents the first image feature, c represents the color information in the digital human image, and σ represents the density information in the digital human image.

[0128] When it is understood that Figure 5 In the structural block diagram of the digital human video generation device shown in FIG. Figure 1 and Figure 2 The steps in the corresponding embodiments, and for Figure 1 and Figure 2 Each step in the corresponding embodiment has been explained in detail in the above embodiment. Figure 1 and Figure 2 as well as Figure 1 and Figure 2 The relevant descriptions in the corresponding embodiments are not repeated here.

[0129] Figure 6 1 is a structural block diagram of an electronic device provided by an embodiment of the present application. Figure 6 As shown, the electronic device 600 of this embodiment includes: a processor 610, a memory 620, and a computer program 630 stored in the memory 620 and executable on the processor 610, such as a program of a digital human video generation method. When the processor 610 executes the computer program 630, the steps in each embodiment of the digital human video generation method described above are implemented, such as Figure 1 Alternatively, the processor 610 implements the above when executing the computer program 630 Figure 5 The functions of each module in the corresponding embodiment are, for example, Figure 5 For details on the functions of each module shown, please refer to Figure 5 Related description in the corresponding embodiment.

[0130] Exemplarily, the computer program 630 may be divided into one or more modules, one or more modules are stored in the memory 620, and are executed by the processor 610 to implement the digital human video generation method provided in the embodiment of the present application. One or more modules may be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 630 in the electronic device 600. For example, the computer program 630 may implement the digital human video generation method provided in the embodiment of the present application.

[0131] The electronic device 600 may include, but is not limited to, a processor 610 and a memory 620. Those skilled in the art will appreciate that Figure 6It is only an example of the electronic device 600 and does not constitute a limitation of the electronic device 600. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0132] The processor 610 may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0133] The memory 620 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. The memory 620 may also be an external storage device of the electronic device 600, such as a plug-in hard disk, a smart memory card, a flash memory card, etc. equipped on the electronic device 600. Furthermore, the memory 620 may also include both an internal storage unit of the electronic device 600 and an external storage device.

[0134] An embodiment of the present application provides a computer-readable storage medium, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the digital human video generation method in the above-mentioned embodiments is implemented.

[0135] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the digital human video generation method in each of the above embodiments.

[0136] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a digital human video, characterized in that: The method comprises: Acquire data features of multimodal data; the data features of the multimodal data include a first image feature of the image, a first audio feature of the audio, and a first point cloud feature of the lip point cloud data; Generate, according to the first image feature, a second audio feature and a second point cloud feature that fuses the correlation between the first audio feature and the first point cloud feature; A digital human video is generated based on the first image feature, the second audio feature, and the second point cloud feature.

2. The method according to claim 1, characterized in that The data features of the multimodal data are obtained, including: Acquire the audio and the image, and generate the lip point cloud data based on the audio; According to preset ray sampling points, the image is projected on three planes to obtain projection values ​​corresponding to the three planes respectively, and the first image feature is generated based on the three projection values; Extracting a first audio feature of the audio according to a preset sliding window strategy; A multi-resolution hash network is used to encode the lip point cloud data to generate the first point cloud feature.

3. The method according to claim 1, characterized in that The step of generating, according to the first image feature, a second audio feature and a second point cloud feature that fuses the correlation between the first audio feature and the first point cloud feature comprises: generating a global image feature describing global information of the image based on the first image feature; generating the second audio feature according to the first audio feature and the global image feature; The second point cloud feature is generated according to the first point cloud feature, the global image feature and the second audio feature.

4. The method according to claim 3, characterized in that The generating the second audio feature according to the first audio feature and the global image feature includes: compressing the first audio feature to obtain a third audio feature; Perform a dot product between the third audio feature and the global image feature to generate the second audio feature.

5. The method according to claim 3, characterized in that: The step of generating the second point cloud feature according to the first point cloud feature, the global image feature and the second audio feature includes: Compressing the first point cloud feature to obtain a third point cloud feature; Performing a dot product between the third point cloud feature and the global image feature to generate an intermediate point cloud feature; The second point cloud feature is generated according to the intermediate point cloud feature and the second audio feature.

6. The method according to claim 5, characterized in that The step of generating the second point cloud feature according to the intermediate point cloud feature and the second audio feature includes: generating a global audio feature describing the audio global information based on the second audio feature; Perform a dot product between the intermediate point cloud feature and the global audio feature to obtain the second point cloud feature.

7. The method according to any one of claims 1 to 6, characterized in that: The step of generating a digital human video based on the first image feature, the second audio feature, and the second point cloud feature comprises: Performing affine transformation calculation on the second audio feature and the second point cloud feature to obtain a translation vector and a scaling vector; Transform the second audio feature and the second point cloud feature based on the translation vector and the scaling vector to obtain a fourth audio feature and a fourth point cloud feature; The first image feature, the fourth audio feature and the fourth point cloud feature are input into an implicit function of a preset neural radiation field to generate the digital human video; the implicit function is expressed as follows: Wherein, x represents the preset three-dimensional spatial coordinates of the ray sampling point, d represents the preset viewing direction of the ray sampling point, represents the fourth audio feature, represents the fourth point cloud feature, F c represents the first image feature, c represents the color information in the digital human image, and σ represents the density information in the digital human image.

8. A digital human video generation device, characterized in that: The device comprises: An acquisition module, used to acquire data features of multimodal data; the data features of the multimodal data include a first image feature of an image, a first audio feature of an audio, and a first point cloud feature of lip point cloud data; A first generating module, configured to generate, according to the first image feature, a second audio feature and a second point cloud feature by integrating the correlation between the first audio feature and the first point cloud feature; The second generating module is used to generate a digital human video based on the first image feature, the second audio feature and the second point cloud feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images