Method, device, equipment and medium for realizing 3D face and audio synchronization

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a method, device, equipment and medium for realizing synchronization of 3D face and audio. The method includes obtaining phonemes and 3D face data fragments in each frame segment in a first sampled video, and obtaining a correspondence table between phonemes and 3D face data fragments in each frame segment; inputting new audio data, and extracting new phonemes in each frame segment in the new audio data; corresponding the new phonemes in each frame segment to the correspondence table, and obtaining 3D face data fragments in the corresponding frame segment; reordering and combining the 3D face data fragments in each frame segment according to the time axis of the new audio data to obtain new 3D face data; performing image translation on the new 3D face data to obtain new 2D face data; merging the new 2D face data and the new audio data to obtain target data, and achieving the synchronization effect of 3D face and audio by corresponding phonemes and 3D faces in the same video frame segment one by one.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for realizing 3D face and audio synchronization. Background Art

[0002] Virtual digital people are based on computer vision and speech synthesis technologies. After model training of image, sound, action, etc., you can freely input any text in the background to generate a short video of "real people" explaining. At present, in the service, game or film and television industries, when virtual characters are involved, it is necessary to use various expressions according to the scene, which serves as the basis for the expressions of virtual characters, making the characters more and more vivid and realistic.

[0003] However, it is difficult for existing virtual characters to synchronize their lip movements with the output audio when speaking, resulting in a poor viewing experience for users. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that the existing audio track and the lip shape of the digital face are out of sync.

[0005] A first aspect of the present invention provides a method for realizing synchronization of 3D face and audio, comprising:

[0006] Acquire phonemes and 3D face data segments in each frame segment in the first sampled video, and obtain a correspondence table between the phonemes and the 3D face data segments in each frame segment;

[0007] Input new audio data, and extract new phonemes under each frame segment in the new audio data;

[0008] The new phonemes in each frame segment correspond to the corresponding relationship table respectively, to obtain the 3D face data fragments in the corresponding frame segment;

[0009] Rearrange and combine the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data;

[0010] Performing image translation on the new 3D face data to obtain new 2D face data;

[0011] The new 2D face data and the new audio data are combined to obtain target data.

[0012] Optionally, in a first implementation of the first aspect of the present invention, obtaining phoneme and 3D face data segments in each frame segment in the first sampled video includes:

[0013] Acquire audio data in the first sampled video;

[0014] Slice the audio data to obtain a plurality of phonemes and a time period in which each phoneme is located;

[0015] The frame segment corresponding to each phoneme is calculated according to the frame rate of the first sampling video and the time period in which the phoneme is located.

[0016] Optionally, in a second implementation of the first aspect of the present invention, obtaining phoneme and 3D face data segments in each frame segment in the first sampled video includes:

[0017] Obtain 2D face data in the first sampled video;

[0018] Slicing the 2D face data according to the frame segment corresponding to each of the phonemes to obtain a plurality of 2D face data segments;

[0019] Based on Deep3DFaceReconstruction technology, extracting vector information of each of the 2D face data fragments;

[0020] Based on the 3DMM technology, each of the vector information is 3D reconstructed to obtain a 3D face data segment.

[0021] Optionally, in a third implementation manner of the first aspect of the present invention, reordering and combining the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data includes:

[0022] A smoothing process is performed during the transition between every two frame segments in the new 3D face data.

[0023] Optionally, in a fourth implementation manner of the first aspect of the present invention, the performing smoothing during the transition between every two frame segments in the new 3D face data includes:

[0024] The current vector information is reduced according to a ratio of 1-n to obtain reduced vector information, where 0 <n<1;

[0025] Reduce two vector information whose distance from the current vector information is 1 by a ratio of n to obtain two distance vector information;

[0026] Adding the reduced vector information and the two distance vector information to obtain new vector information;

[0027] Based on the 3DMM technology, each of the new vector information is reconstructed in 3D to obtain multiple 3D face data segments with smooth transitions;

[0028] Combine a plurality of the 3D face data segments with smooth transitions to obtain new 3D face data.

[0029] Optionally, in a fifth implementation of the first aspect of the present invention, performing image translation on the new 3D face data to obtain new 2D face data includes:

[0030] Generate 2D image data based on the new 3D face data and process it through a preset generator;

[0031] Based on the adversarial network, obtaining 3D image data to which the 2D image data is mapped;

[0032] determining whether the 3D image data matches the object of the 2D image data;

[0033] If true, output true, and determine the 2D image data as new 2D face data;

[0034] If it is false, then output fake.

[0035] Optionally, in a sixth implementation of the first aspect of the present invention, if false, outputting fake includes:

[0036] Based on the VGG network, the error between the 3D image data and the new 3D face data is calculated and fed back to the adversarial network.

[0037] A second aspect of the present invention provides a device for realizing synchronization of 3D face and audio, comprising:

[0038] A first sampling module, used to obtain phonemes and 3D face data segments in each frame segment in the first sampled video, and obtain a correspondence table between the phonemes and the 3D face data segments in each frame segment;

[0039] A second sampling module, used for inputting new audio data and extracting new phonemes in each frame segment of the new audio data;

[0040] A mapping module, used for respectively corresponding the new phonemes in each frame segment to the corresponding relationship table, to obtain the 3D face data segment in the corresponding frame segment;

[0041] A reordering and combining module, used for reordering and combining the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data;

[0042] A conversion module, used for performing image translation on the new 3D face data to obtain new 2D face data;

[0043] The output module is used to merge the new 2D face data and the new audio data to obtain target data.

[0044] A third aspect of the present invention provides a device for realizing synchronization of 3D face and audio, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through a line; and the at least one processor calls the instructions in the memory so that the device for realizing synchronization of 3D face and audio executes the above-mentioned method for realizing synchronization of 3D face and audio.

[0045] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for realizing synchronization of 3D face and audio.

[0046] In an embodiment of the present invention, a method, device, equipment and medium for realizing synchronization of 3D face and audio are provided. The method includes obtaining phonemes and 3D face data segments in each frame segment in a first sampled video, and obtaining a correspondence table between phonemes and 3D face data segments in each frame segment; inputting new audio data, and extracting new phonemes in each frame segment in the new audio data; corresponding the new phonemes in each frame segment to the correspondence table, and obtaining 3D face data segments in the corresponding frame segment; reordering and combining the 3D face data segments in each frame segment according to the time axis of the new audio data, and obtaining new 3D face data; performing image translation on the new 3D face data, and obtaining new 2D face data; merging the new 2D face data and the new audio data, and obtaining target data, and achieving synchronization of 3D face and audio by corresponding phonemes and 3D faces in the same video frame segment one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of a first embodiment of a method for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a second embodiment of a method for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a third embodiment of a method for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a fourth embodiment of a method for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of a fifth embodiment of a method for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0052] Figure 6This is a schematic diagram of a sixth embodiment of a method for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of an embodiment of a device for realizing 3D face and audio synchronization in an embodiment of the present invention;

[0054] Figure 8 The figure is a schematic diagram of an embodiment of a device for realizing synchronization of 3D face and audio in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Embodiments of the present invention provide a method, apparatus, device and medium for achieving 3D face and audio synchronization.

[0056] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings, if present, are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 -Attached Figure 6 , an embodiment of the method for realizing 3D face and audio synchronization in an embodiment of the present invention includes:

[0058] 101. Obtain phonemes and 3D face data segments in each frame segment in the first sampled video, and obtain a correspondence table between the phonemes and 3D face data segments in each frame segment;

[0059] Furthermore, step 101 may further specifically perform the following steps:

[0060] 1011. Acquire audio data in a first sampled video;

[0061] 1012. Segment the audio data to obtain a plurality of phonemes and a time period of each phoneme;

[0062] 1013. Calculate the frame segment corresponding to each phoneme according to the frame rate of the first sampled video and the time period of the phoneme;

[0063] In steps 1011-1013, this embodiment uses pyfoal in the python library to obtain the audio data of the first sampled video and extract the phonemes in the audio data one by one, and each phoneme corresponds to a certain time period of the audio data, as shown in the following phoneme time period correspondence table:

[0064] Table 1. Phoneme time period correspondence table

[0065] Time period phoneme 0-1.619s sp 1.619-1.649s W 1.649-1.679s OW1 1.679-1.719s HH 1.719-1.908s AY1 1.908-2.008s sp 2.008-2.038s Z

[0066] From the above, we can know that the corresponding phoneme obtained in the audio data in the time period of 1.619-1.649s is 'W'. Since it needs to be matched with the image data later, and the segmentation unit of each segment of the image data is a video frame segment, the time period needs to be converted into a video frame segment. By multiplying it with the frame rate of the video, the corresponding video frame segment can be obtained. For example, in the time period of 1.619-1.649s, and the frame rate of the video is 25 frames per second, the corresponding video frame segment can be obtained as 40.475 frames-41.225 frames. Since the number of frames is generally taken as an integer, the video frame segment corresponding to the time period of 1.619-1.649s is 40 frames-41 frames.

[0067] 1014. Obtain 2D face data in the first sample video;

[0068] 1015. Slice the 2D face data according to the frame segment corresponding to each phoneme to obtain multiple 2D face data segments;

[0069] 1016. Based on Deep3DFaceReconstruction technology, extract the vector information of each 2D face data segment;

[0070] 1017. Based on the 3DMM technology, each vector information is 3D reconstructed to obtain a 3D face data segment.

[0071] In steps 1014-1017, since the characters in the video recordings are all two-dimensional images, the sampled face materials are all 2D face data, and the 2D face data is segmented according to the frame segment corresponding to each phoneme to obtain the 2D face data segments of each frame segment, and the vector information of each 2D face data segment is extracted based on the Deep3DFaceReconstruction technology, and then based on the 3DMM technology, the 3D face model is reconstructed according to each vector information. The specific formula is:

[0072]

[0073] in, is the average face shape, S i is the shape PCA principal component, α i is the shape factor, E i PCA principal components and β for face labels i is the facial expression coefficient, m and n are both 199.

[0074] 102. Input new audio data, and extract new phonemes in each frame segment of the new audio data; 103. Correspond the new phonemes in each frame segment to the corresponding relationship table, and obtain the 3D face data segment in the corresponding frame segment;

[0075] 104. Reorder and combine the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data;

[0076] Furthermore, step 104 specifically includes the following steps:

[0077] 1041. Perform smoothing processing during the transition between every two frame segments in the new 3D face data.

[0078] Furthermore, step 1041 may further specifically perform the following steps:

[0079] 10411. Reduce the current vector information according to the ratio of 1-n to obtain reduced vector information, where 0 <n<1;

[0080] 10412. Reduce two vector information whose distance from the current vector information is 1 by a ratio of n respectively to obtain two distance vector information;

[0081] 10413. Add the reduced vector information and the two distance vector information to obtain new vector information;

[0082] 10414. Based on the 3DMM technology, each new vector information is reconstructed into 3D to obtain multiple 3D face data segments with smooth transitions;

[0083] 10415. Combining multiple 3D face data segments with smooth transitions to obtain new 3D face data;

[0084] In steps 10411-10415, since the multiple 3D face data segments brought by the new phoneme are not continuous, after the multiple 3D face data segments are combined, smoothing processing needs to be performed between the two segments to ensure a natural overall transition effect. For example, in this embodiment, the following method is used:

[0085] smooth[i,:]=0.8*data[i,:]+0.1*data[i-1,:]+0.1*data[i+1,:];

[0086] The new vector is equal to the current vector * 0.8 plus the two vectors with a distance of 1 * 0.1.

[0087] 105. Perform image translation on the new 3D face data to obtain new 2D face data;

[0088] Furthermore, step 105 may specifically further perform the following steps:

[0089] 1051. Generate 2D image data based on the new 3D face data and by processing it through a preset generator;

[0090] 1052. Based on the adversarial network, obtain 3D image data mapped to the 2D image data;

[0091] 1053. Determine whether the 3D image data matches the object of the 2D image data;

[0092] 1054. If true, output true, and determine the 2D image data as new 2D face data;

[0093] 1055. If it is false, output fake.

[0094] Furthermore, step 1055 specifically includes the following steps:

[0095] 10551. Based on the VGG network, calculate the error between the 3D image data and the new 3D face data, and feed it back to the adversarial network.

[0096] In steps 1051-1055 and step 10551, pose2vid in the pix2pix algorithm is used to extract 2D image data from 3D face data, and the extracted 2D image data is not necessarily true. The discriminator in pix2pix is ​​needed to discriminate the 2D image data and the 3D image data generated by the generator. If the 2D image data and the 3D image data generated by the generator are true, then the 3D image data is 3D face data and can be output, otherwise it is false, and based on the VGG network, the error calculation is performed between the 3D image data and the new 3D face data. The VGG network uses VGG16, and only 3*3 convolution is used, and only 2*2 pooling is used. The structure is simple, and while reducing parameter input, more nonlinear mapping can be performed to increase the fitting ability of the network. Finally, the softmax output can obtain 1000 prediction results, thereby improving the accuracy of error calculation.

[0097] 106. Merge the new 2D face data and the new audio data to obtain target data.

[0098] In this embodiment, the obtained 2D face data is combined into a video form through opencv, and then the video is synthesized with new audio data through ffmpeg to obtain target data, that is, the dynamic expression of the virtual digital character's facial expression. Since the dynamic changes of the digital character's face are obtained by one-to-one correspondence and recombination of the phonemes and 3D face data of each video frame segment, the synchronization effect of 3D face and audio can be achieved, and the 3D face data gradually achieves high-definition effect under the training of the VGG network that continuously calculates the reconstruction error.

[0099] In an embodiment of the present invention, the method for synchronizing 3D face and audio adopts a one-to-one correspondence between phonemes and 3D faces in the same video frame segment to achieve the synchronization effect of 3D face and audio, and the 3D face data is trained by the VGG network to continuously calculate the reconstruction error, thereby reducing the blurring of the digital face when it is mapped to the real face, and gradually achieving a high-definition effect.

[0100] The above describes the method for realizing 3D face and audio synchronization in the embodiment of the present invention. The following describes the device for realizing 3D face and audio synchronization in the embodiment of the present invention. Figure 7 In one embodiment of the present invention, a device for realizing synchronization of 3D face and audio includes:

[0101] 201. A first sampling module, configured to obtain a phoneme and a 3D face data segment in each frame segment in a first sampled video, and obtain a correspondence table between the phoneme and the 3D face data segment in each frame segment;

[0102] 202. A second sampling module, configured to input new audio data and extract new phonemes in each frame segment of the new audio data;

[0103] 203. A mapping module, used to correspond the new phonemes in each frame segment to the corresponding relationship table, and obtain the 3D face data fragments in the corresponding frame segment;

[0104] 204. A reordering and combining module, configured to reorder and combine the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data;

[0105] 205. A conversion module, configured to perform image translation on the new 3D face data to obtain new 2D face data;

[0106] 206. An output module is used to merge the new 2D face data and the new audio data to obtain target data.

[0107] In an embodiment of the present invention, the method for synchronizing 3D face and audio adopts a one-to-one correspondence between phonemes and 3D faces in the same video frame segment to achieve the synchronization effect of 3D face and audio, and the 3D face data gradually achieves a high-definition effect under the training of the VGG network that continuously calculates the reconstruction error.

[0108] Another embodiment of the device for realizing 3D face and audio synchronization in the embodiment of the present invention includes:

[0109] 201. A first sampling module, configured to obtain a phoneme and a 3D face data segment in each frame segment in a first sampled video, and obtain a correspondence table between the phoneme and the 3D face data segment in each frame segment;

[0110] 202. A second sampling module, configured to input new audio data and extract new phonemes in each frame segment of the new audio data;

[0111] 203. A mapping module, used to correspond the new phonemes in each frame segment to the corresponding relationship table, and obtain the 3D face data fragments in the corresponding frame segment;

[0112] 204. A reordering and combining module, configured to reorder and combine the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data;

[0113] 205. A conversion module, configured to perform image translation on the new 3D face data to obtain new 2D face data;

[0114] 206. An output module is used to merge the new 2D face data and the new audio data to obtain target data.

[0115] Optionally, the first sampling module 201 may further specifically perform:

[0116] Acquire audio data in the first sampled video;

[0117] Slice the audio data to obtain multiple phonemes and the time period of each phoneme;

[0118] Calculate the frame segment corresponding to each phoneme according to the frame rate of the first sampled video and the time period of the phoneme;

[0119] Obtain 2D face data in the first sampled video;

[0120] According to the frame segment corresponding to each phoneme, the 2D face data is segmented to obtain multiple 2D face data segments;

[0121] Based on Deep3DFaceReconstruction technology, the vector information of each 2D face data segment is extracted;

[0122] Based on the 3DMM technology, each vector information is reconstructed in 3D to obtain a 3D face data segment.

[0123] Optionally, the reorder combination module 202 may also be specifically used for:

[0124] Smoothing is performed during the transition between every two frame segments in the new 3D face data.

[0125] Optionally, “performing smoothing during the transition between every two frame segments in the new 3D face data” can also be specifically used for:

[0126] Reduce the current vector information in a 1-n ratio to obtain reduced vector information, where 0 <n<1;

[0127] Reduce the two vector information whose distance from the current vector information is 1 by a ratio of n to obtain two distance vector information;

[0128] Add the reduced vector information and the two distance vector information to obtain new vector information;

[0129] Based on 3DMM technology, each new vector information is reconstructed into 3D to obtain multiple 3D face data segments with smooth transitions;

[0130] Combine multiple 3D face data segments with smooth transitions to obtain new 3D face data.

[0131] Optionally, the conversion module 205 may further specifically execute:

[0132] Generate 2D image data based on the new 3D face data and process it through a preset generator;

[0133] Based on the adversarial network, the 3D image data mapped to the 2D image data is obtained;

[0134] Determine whether the 3D image data matches the object of the 2D image data;

[0135] If true, output true, and determine the 2D image data as new 2D face data;

[0136] If it is false, then output fake.

[0137] Optionally, "if false, output fake" can also be used specifically for:

[0138] Based on the VGG network, the error between the 3D image data and the new 3D face data is calculated and fed back to the adversarial network.

[0139] In an embodiment of the present invention, the method for synchronizing 3D face and audio adopts a one-to-one correspondence between phonemes and 3D faces in the same video frame segment to achieve the synchronization effect of 3D face and audio, and the 3D face data is trained by the VGG network to continuously calculate the reconstruction error, thereby reducing the blurring of the digital face when it is mapped to the real face, and gradually achieving a high-definition effect.

[0140] above Figure 7 From the perspective of modular functional entities, the apparatus for realizing 3D face and audio synchronization in an embodiment of the present invention is described in detail. The following describes in detail the device for realizing 3D face and audio synchronization in an embodiment of the present invention from the perspective of hardware processing.

[0141] Figure 8 3 is a schematic diagram of a structure of a device for implementing 3D face and audio synchronization provided by an embodiment of the present invention. The device 300 for implementing 3D face and audio synchronization may have relatively large differences due to different configurations or performances, and may include one or more processors central processing units, CPU310, for example, one or more processors and memory 320, one or more storage media 330 storing application programs 333 or data 332, for example, one or more mass storage devices. Among them, the memory 320 and the storage medium 330 may be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules not shown in the diagram, and each module may include a series of instruction operations in the device 300 for implementing 3D face and audio synchronization. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the device 300 for implementing 3D face and audio synchronization.

[0142] The device 300 for realizing synchronization of 3D face and audio may further include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 330, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 4 The device structure for implementing 3D face and audio synchronization shown does not constitute a limitation on the device based on implementing 3D face and audio synchronization, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0143] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for realizing synchronization of 3D face and audio.

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device, such as a personal computer, a server, or a network device, to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory, ROM, random access memory, RAM, disk or optical disk, and other media that can store program codes.

[0146] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for realizing 3D face and audio synchronization, characterized in that: include: Acquire phonemes and 3D face data segments in each frame segment in the first sampled video, and obtain a correspondence table between the phonemes and the 3D face data segments in each frame segment; Input new audio data, and extract new phonemes under each frame segment in the new audio data; The new phonemes in each frame segment correspond to the corresponding relationship table respectively, to obtain the 3D face data fragments in the corresponding frame segment; Rearrange and combine the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data; Performing image translation on the new 3D face data to obtain new 2D face data; Merging the new 2D face data and the new audio data to obtain target data; The step of obtaining the phoneme and 3D face data segments in each frame segment of the first sampled video includes: Acquire audio data in the first sampled video; Slice the audio data to obtain a plurality of phonemes and a time period in which each phoneme is located; Calculating the frame segment corresponding to each of the phonemes according to the frame rate of the first sampled video and the time period in which the phonemes are located; The step of obtaining the phoneme and 3D face data segments in each frame segment of the first sampled video includes: Obtain 2D face data in the first sampled video; Slicing the 2D face data according to the frame segment corresponding to each of the phonemes to obtain a plurality of 2D face data segments; Based on Deep3DFaceReconstruction technology, extracting vector information of each of the 2D face data fragments; Based on the 3DMM technology, each of the vector information is 3D reconstructed to obtain a 3D face data segment; The step of reordering and combining the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data includes: Performing smoothing during the transition between every two frame segments in the new 3D face data; The step of performing smoothing during the transition between every two frame segments in the new 3D face data comprises: The current vector information is reduced according to a ratio of 1-n to obtain reduced vector information, where 0 <n<1; Reduce two vector information whose distance from the current vector information is 1 by a ratio of n to obtain two distance vector information; Adding the reduced vector information and the two distance vector information to obtain new vector information; Based on the 3DMM technology, each of the new vector information is reconstructed in 3D to obtain multiple 3D face data segments with smooth transitions; Combine a plurality of the 3D face data segments with smooth transitions to obtain new 3D face data.

2. The method for realizing 3D face and audio synchronization according to claim 1, characterized in that: The performing image translation on the new 3D face data to obtain new 2D face data comprises: Generate 2D image data based on the new 3D face data and process it through a preset generator; Based on the adversarial network, obtaining 3D image data to which the 2D image data is mapped; determining whether the 3D image data matches the object of the 2D image data; If true, output true, and determine the 2D image data as new 2D face data; If it is false, then output fake.

3. The method for realizing 3D face and audio synchronization according to claim 2, characterized in that: If it is false, the output fake includes: Based on the VGG network, the error between the 3D image data and the new 3D face data is calculated and fed back to the adversarial network.

4. A device for realizing synchronization of 3D face and audio, characterized in that: include: A first sampling module, used to obtain phonemes and 3D face data segments in each frame segment in the first sampled video, and obtain a correspondence table between the phonemes and the 3D face data segments in each frame segment; A second sampling module, used for inputting new audio data and extracting new phonemes in each frame segment of the new audio data; A mapping module, used for respectively corresponding the new phonemes in each frame segment to the corresponding relationship table, to obtain the 3D face data segment in the corresponding frame segment; A reordering and combining module, used for reordering and combining the 3D face data segments under each frame segment according to the time axis of the new audio data to obtain new 3D face data; A conversion module, used for performing image translation on the new 3D face data to obtain new 2D face data; An output module, used for merging the new 2D face data and the new audio data to obtain target data; Wherein, the first sampling module is further specifically used for: Acquire audio data in the first sampled video; Slice the audio data to obtain multiple phonemes and the time period of each phoneme; Calculate the frame segment corresponding to each phoneme according to the frame rate of the first sampled video and the time period of the phoneme; Obtain 2D face data in the first sampled video; According to the frame segment corresponding to each phoneme, the 2D face data is segmented to obtain multiple 2D face data segments; Based on Deep3DFaceReconstruction technology, the vector information of each 2D face data segment is extracted; Based on 3DMM technology, each vector information is reconstructed into 3D to obtain 3D face data fragments; Wherein, the reorder combination module is also specifically used for: Perform smoothing during the transition between every two frame segments in the new 3D face data; The smoothing process of the transition between every two frame segments in the new 3D face data includes: The current vector information is reduced according to a ratio of 1-n to obtain reduced vector information, where 0 <n<1; Reduce two vector information whose distance from the current vector information is 1 by a ratio of n to obtain two distance vector information; Adding the reduced vector information and the two distance vector information to obtain new vector information; Based on the 3DMM technology, each of the new vector information is reconstructed in 3D to obtain multiple 3D face data segments with smooth transitions; Combine a plurality of the 3D face data segments with smooth transitions to obtain new 3D face data.

5. A device for realizing synchronization of 3D face and audio, characterized in that: The device for realizing synchronization of 3D face and audio comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instruction in the memory to enable the device to execute the method for realizing synchronization of 3D face and audio according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for realizing synchronization of 3D face and audio according to any one of claims 1 to 3 is realized.

Images