Voice-driven digital human construction method and device based on enhanced deformable convolution and space-time motion compensation

Abstract

The invention discloses a voice-driven digital human construction method and device based on enhanced deformable convolution and space-time motion compensation. Firstly, Speech2Lip is used to generate a preliminary mouth shape synchronization video according to an input audio. Then, mouth shape features in the generated video and phoneme features of the audio are extracted, and two time sequences are formed; next, DTW (Dynamic Time Warping) is applied to perform time alignment on the audio and the mouth shape sequence, an optimal time path is found, and the mouth shape action is corrected, so that the mouth shape action is more accurately matched with the audio pronunciation time sequence; and adjusting the mouth shape animation according to the alignment result, generating a corrected mouth shape synchronous video, and ensuring that the mouth shape action is consistent with the pronunciation height of the audio. And finally, synthesizing the optimized mouth shape synchronous video with the audio to generate a final high-quality digital human video. Compared with the prior art, the method has the advantages that the generated digital human video driven by voice can be optimized, and the alignment accuracy of the mouth shape and the audio in the video is improved.

Description

Technical Field

[0001] The present invention belongs to the field of digital humans, and in particular relates to a method and device for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation. Background Art

[0002] With the rapid development of artificial intelligence technology, video production has stopped at relying solely on manual CG animation production. More and more technologies and models have made significant progress in the expression generation and voice drive of real-life videos. Unlike the previous use of CG animation to modify real-life videos, it is now possible to automatically generate the required digital human videos by providing model input and modifying model parameters. The digital human generated by the model can simulate the facial movements and mouth shape changes of real people in virtual scenes, which has brought wide applications in film and television production, virtual anchors, distance education and other fields.

[0003] However, although there are many models that can realize the generation and production of digital human videos, there are still many problems such as video frame skipping, insufficient matching between mouth shape and voice, and image artifacts. Since mouth shape movements are often modified based on training sets, mouth shape movements often cannot accurately follow the rhythm and pronunciation details of the voice, resulting in the generated video not matching the actual voice effect naturally and lacking synchronization. This problem limits the performance of digital human technology in high-demand scenarios.

[0004] Based on this, there is an urgent need for a voice-driven digital human generation method that can reduce the frequency of problems such as frame skipping and lip shape mismatch and improve the synchronization between video lip shape and audio. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the present invention discloses a method and device for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation. By modifying some of the original convolution layers in the model to enhanced deformable convolution, adding video alignment technology, and modifying the optimal alignment path addressing algorithm of the DTW algorithm, the authenticity of the digital human video generated by the model and the synchronization of the lip shape and audio are improved.

[0006] Technical solution: The present invention proposes a method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation, comprising the following steps:

[0007] Step 1: Prepare the original video V1 with clear face and no obstruction, and prepare the audio A1 with clear pronunciation and no noise;

[0008] Step 2: Integrate the enhanced variability convolutional network DCN into the Speech2Lip model, initialize DCN, take the original video V1 and audio A1 prepared in step 1 as input, and use the Speech2Lip model integrated with DCN to preliminarily generate a speech-driven digital human video V2;

[0009] Step 3: Use the Dilb facial key point detection algorithm to extract the mouth area features of the initially generated video V2, and the extracted features are recorded as LM; modify and store the video features LM and save them as the time series V_S of the mouth shape in the video;

[0010] Step 4: Use short-time Fourier transform to map the audio A1 onto the Mel-spectrogram, denoted as MS, and then transpose MS to obtain the audio sequence, denoted as A_S;

[0011] Step 5: Calculate the Euclidean distance between V_S and A_S at each time step and save it as M disEu , use the DTW algorithm to calculate the path P of time series alignment best ;

[0012] Step 6: Alignment path P calculated by DTW algorithm best , reorder the audio A1 and the initially generated video V2 so that the mouth shape and speech are more matched, and the final video V3 is generated.

[0013] Furthermore, the specific method of step 2 is:

[0014] Step 2.1: In the mouth shape generation layer of Speech2Lip, introduce three layers of DCN to replace the original convolutional layer. The specific introduction steps are as follows:

[0015] Step 2.1.1: Based on the 67-channel audio feature output in the Speech2Lip model, the first layer of DCN receives the audio features for preliminary mouth shape processing. The number of input channels of the first layer of DCN is set to 67 and the number of output channels is set to 128. This layer is denoted as D conv1 ;

[0016] Step 2.1.2: The second DCN receives the output of the first layer DCN and performs further lip processing. The second layer DCN input channel is set to 128 and the output channel is set to 256. This layer is called D conv2 ;

[0017] Step 2.1.2: The third DCN receives the output of the second 2-layer DCN and performs more complex and detailed lip processing. Set the third layer DCN input channel to 256 and the output channel to 512. This layer is called D conv3 ;

[0018] Step 2.2: Initialize DCN, including He initialization and offset layer zero initialization. The specific steps are as follows:

[0019] Step 2.2.1: After replacing the standard convolutional layer in the output layer of the Speech2Lip model to generate the mouth shape with DCN, first initialize the weights of each layer of DCN. To prevent the gradient from disappearing, select He initialization. The formula is as follows:

[0020]

[0021] W is the initialized weight matrix; N(0,σ2) means it obeys a normal distribution with a mean of 0 and a variance of σ2. in is the number of neurons input to the current layer;

[0022] Step 2.2.2: Zero-initialize the offset generation layer of DCN;

[0023] Step 2.3: Take the video V1 and audio A1 prepared in step 1 as input, and use the Speech2Lip model integrated into DCN to preliminarily generate a speech-driven digital human video V2;

[0024] Furthermore, the specific method of step 3 is:

[0025] Step 3.1: Load the Dlib facial key point detector model, input the generated speech-driven video V2 into the Dlib facial key point detector model for face detection, and perform facial key point detection on each frame. Each frame detection result contains 68 pairs of two-dimensional coordinates, such as [(x i0 ,y i0 ),(x i1 ,y i1 )…,(x i67 ,y i67 )], where x ij ,y ij is the coordinates of the jth pair of key points in the i-th frame, and the detection result is recorded as LM;

[0026] Step 3.2: For the 68 key points recorded in each frame of the LM data, extract the 49th to 68th key points that record the mouth shape as the mouth key point data. The data format is as follows: i48 ,y i48 ),(x i49 ,y i49 )…,(x i67 ,y i67 )], denoted as LMM;

[0027] Step 3.3: Concatenate the key point data of each frame in LMM into a 1D vector, such as; [x i48,y i48 ,x i49 ,y i49 …,x i67 ,y i67 ], at this time the length of each frame is 40, and the key point data at this time is recorded as the time series V_S used later.

[0028] Furthermore, the specific method of step 4 is:

[0029] Step 4.1: Use short-time Fourier transform to convert A1 from time domain to frequency domain. The time window size of Fourier transform is set to 2048, the moving step is 512, and the result of short-time Fourier transform is recorded as SR;

[0030] Step 4.2: Take the absolute value of each element of SR to obtain the amplitude spectrum of A1. The amplitude spectrum of A1 is recorded as SM.

[0031] Step 4.3: Convert SM to Mel spectrum according to the Hertz to Mel spectrum conversion formula Redistribute to Mel frequency, and record the converted Mel spectrum as MS;

[0032] Step 4.4: Convert the converted Mel-spectrogram MS into decibel units, and record the converted decibel graph as MS_C;

[0033] Step 4.5: Extract audio time features from MS_C, transpose MS_C, and convert it into the time series of audio A1. The audio time series of A1 is recorded as A_S.

[0034] Furthermore, the specific method of step 5 is:

[0035] Step 5.1: Through interpolation, the audio time steps with different numbers are matched with the number of video frames to achieve downsampling of the audio time series and synchronization with the video frames. The new audio time series after downsampling is recorded as A_SN.

[0036] Step 5.2: Construct the distance matrix of the new audio time series A_SN and the video time series V_S, that is, calculate the Euclidean distance between each time step of A_SN and V_S, and record this distance matrix as M disEu ;

[0037] Step 5.3: Construct a cumulative distance matrix to store the cost of the minimum path to the current point. The modified recursive formula is as follows:

[0038] C(i,j)=D(i,j)+min(C(i,j-1),C(i-1,j-1))

[0039] Where C(i,j) is the minimum total path cost from the starting point to the (i,j)th point; D(i,j) represents M disEu The distance from the i-th time step of the recorded audio time sequence A_SN to the j-th frame of the video time sequence V_S; this cumulative distance matrix is ​​recorded as M disCu ;

[0040] Step 5.4: Backtrack and accumulate the distance matrix M disCu , find the best path, that is, from the optimal alignment path, record the path as P best , where P best The elements are stored as a set of time step index pairs (i, j), where i is the time step index of the audio and j is the time step index of the video.

[0041] Furthermore, the specific method of step 6 is:

[0042] Step 6.1: Traverse the alignment path P best ;

[0043] Step 6.2: According to the voice-driven principle, align the time step index pairs in the path and reorder the audio and video. The order of the audio time steps remains unchanged. The order of the audio time steps is recorded as AS aligned ; Reselect video frames according to the index pairs and construct a new video frame sequence, including the following steps:

[0044] Step 6.2.1: Process the video duplicate frames, and repeat the video frames when the video frames correspond to multiple audio time steps to ensure that the video frames can be aligned with the audio time steps;

[0045] Step 6.2.2: Process video frame jumps. The jump is defined as the absolute value of the difference between the indexes of two frames being greater than 25. When a video frame jump occurs, the difference between the indexes of the current frame and the next frame is calculated, and a transition frame is generated by linear interpolation of the current frame and the next frame. This transition frame is inserted into the video frame sequence as the next frame. The order of the reordered video frames is recorded as VS aligned ;

[0046] Step 6.3: Regenerate the video according to the sorted video frame sequence VS, and record the newly generated video as V2 reordered ;

[0047] Step 6.4: Merge the original speech A1 and V2 reordered Combine them as the final aligned video V3.

[0048] The present invention also discloses a method and device for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the steps of the method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation are implemented as described above.

[0049] Beneficial Effects

[0050] 1. The method of the present invention is based on the existing voice-driven digital human model. By modifying the basic convolution layer for generating mouth shapes in the model, the offset generation module of the enhanced deformable convolutional network is used to offset the convolution kernel position to the mouth, and the focus is more on the generation of the mouth shape movement, so as to improve the accuracy of the mouth shape movement when the model initially generates voice-driven digital human videos;

[0051] 2. After obtaining the video generated by the improved model, the present invention manually adds video alignment technology to further process the video. By adding phase-based spatiotemporal motion compensation technology, in order to ensure that the audio effect is not affected during video alignment, the optimal alignment path addressing algorithm is modified to ensure that the aligned audio will not have problems caused by stretching, and improves the synchronization between the mouth shape and the audio and the mouth shape alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The overall flow chart of the method for constructing speech-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation;

[0053] Figure 2 To preliminarily generate a speech-driven digital human video flow chart using the improved Speech2Lip model;

[0054] Figure 3 Schematic diagram of using DCN to replace the original convolutional layer in the Speech2Lip mouth shape generation module;

[0055] Figure 4 Flowchart for audio and video feature extraction and generation of cumulative distance matrix;

[0056] Figure 5 To reorder the audio and video according to the optimal alignment path, the final video flow chart is obtained. DETAILED DESCRIPTION

[0057] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0058] The present invention discloses a method and device for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation. First, Speech2Lip is used to generate a preliminary lip-sync video based on the input audio. Then, the lip shape features in the generated video and the phoneme features of the audio are extracted to form two time series. Next, DTW (dynamic time warping) is applied to time-align the audio and lip sequence, find the optimal time path, and correct the lip movement to make it more accurately match the audio pronunciation timing. According to the alignment results, the lip animation is adjusted to generate a corrected lip-sync video to ensure that the lip movement is highly consistent with the pronunciation of the audio. Finally, the optimized lip-sync video is synthesized with the audio to generate the final high-quality digital human video. Specifically comprising the following steps:

[0059] Step 1: Prepare a clear and noise-free audio file in .wav format. Define the audio as A1 to drive the overall video. Prepare a base video for generating mouth animation. The face must be clear and unobstructed, and the camera must be stable. Define the video as V1.

[0060] Step 2: Integrate the enhanced variability convolutional network DCN into the Speech2Lip model, initialize DCN, take the original video V1 and audio A1 prepared in step 1 as input, and use the Speech2Lip model integrated with DCN to preliminarily generate a speech-driven digital human video V2.

[0061] Step 2.1: In the mouth shape generation layer of Speech2Lip, introduce three layers of DCN to replace the original convolutional layer. The specific introduction steps are as follows:

[0062] Step 2.1.1: Based on the 64-channel audio feature output in the Speech2Lip model, the first layer of DCN receives the audio features for preliminary mouth shape processing. The number of input channels of the first layer of DCN is set to 64 and the number of output channels is set to 128. This layer is denoted as D conv1 .

[0063] Step 2.1.2: The second DCN receives the output of the first layer DCN and performs further lip processing. The second layer DCN input channel is set to 128 and the output channel is set to 256. This layer is called D conv2 .

[0064] Step 2.1.2: The third DCN receives the output of the second 2-layer DCN and performs more complex and detailed lip processing. Set the third layer DCN input channel to 256 and the output channel to 512. This layer is called D conv3 .

[0065] Step 2.2: Initialize DCN, including He initialization and offset layer zero initialization. The specific steps are as follows:

[0066] Step 2.2.1: After replacing the standard convolutional layer in the output layer of the Speech2Lip model to generate the mouth shape with DCN, first initialize the weights of each layer of DCN. To prevent the gradient from disappearing, select He initialization. The formula is as follows:

[0067]

[0068] W is the initialized weight matrix; N(0,σ2) means it obeys a normal distribution with a mean of 0 and a variance of σ2. in is the number of neurons in the current layer input.

[0069] Step 2.2.2: Zero-initialize the offset generation layer of DCN.

[0070] Step 2.3: Take the video V1 and audio A1 prepared in step 1 as input, and use the improved Speech2Lip model to preliminarily generate a speech-driven digital human video V2.

[0071] Step 3: Use the Dilb facial key point detection algorithm to extract the mouth area features of the initially generated video V2, and the extracted features are recorded as LM; modify and store the video features LM and save them as the time series V_S of the mouth shape in the video.

[0072] Step 3.1: Load the Dlib facial key point detector model, input the generated voice-driven video V2 into the Dlib model for face detection, and perform facial key point detection on each frame. Each frame detection result contains 68 pairs of two-dimensional coordinates, such as [(x i0 ,y i0 ),(x i1 ,y i1 )…,(x i67 ,y i67 )], where x ij ,y ij is the jth pair of key point coordinates in the i-th frame, and the detection result is recorded as LM.

[0073] Step 3.2: For the 68 key points recorded in each frame of the LM data, extract the 49th to 68th key points that record the mouth shape as the mouth key point data. The data format is as follows: i48 ,y i48 ),(x i49 ,y i49 )…,(x i67 ,y i67 )], denoted as LMM.

[0074] Step 3.3: Concatenate the key point data of each frame in LMM into a 1D vector, such as; [x i48 ,y i48 ,x i49 ,y i49 …,x i67 ,y i67 ], at this time the length of each frame is 40, and the key point data at this time is recorded as the time series V_S used later.

[0075] Step 4: Use short-time Fourier transform to map the audio A1 onto the Mel-spectrogram, denoted as MS, and then transpose MS to obtain the audio sequence, denoted as A_S.

[0076] Step 4.1: Use short-time Fourier transform to convert A1 from time domain to frequency domain. Set the time window size of Fourier transform to 2048 and the moving step size to 512. The result of short-time Fourier transform is recorded as SR.

[0077] Step 4.2: Take the absolute value of each element of SR to obtain the amplitude spectrum of A1. The amplitude spectrum result of A1 is recorded as SM.

[0078] Step 4.3: Convert SM to Mel spectrum according to the Hertz to Mel spectrum conversion formula The spectral images are redistributed to the Mel frequency, and the converted Mel spectrum is recorded as MS.

[0079] Step 4.4: Convert the above converted Mel-spectrogram MS into decibel (db) units, and record the converted decibel graph as MS_C.

[0080] Step 4.5: Extract audio time features from MS_C and perform a transposition operation on MS_C to convert it into the time series of audio A1. The audio time series of A1 is recorded as A_S.

[0081] Step 5: Calculate the Euclidean distance between V_S and A_S at each time step and save it as M disEu , use the DTW algorithm to calculate the path P of time series alignment best .

[0082] Step 5.1: Through interpolation, the audio time steps with different numbers are matched with the number of video frames to achieve downsampling of the audio time series and synchronization with the video frames. The new audio time series after downsampling is recorded as A_SN.

[0083] Step 5.2: Construct the distance matrix of the new audio time series A_SN and the video time series V_S, that is, calculate the Euclidean distance between each time step of A_SN and V_S, and record this distance matrix as M disEu .

[0084] Step 5.3: Construct a cumulative distance matrix to store the cost of the minimum path to the current point. The modified recursive formula is as follows:

[0085] C(i,j)=D(i,j)+min(C(i,j-1),C(i-1,j-1))

[0086] Where C(i,j) is the minimum total path cost from the starting point to the (i,j)th point; D(i,j) represents M disEu The distance from the i-th time step of the recorded audio time series A_SN to the j-th frame of the video time series V_S, and this cumulative distance matrix is ​​recorded as M disCu .

[0087] Step 5.4: Backtrack and accumulate the distance matrix M disCu , find the best path, that is, from the optimal alignment path, record this path as P best , where P best The elements are stored as a set of time step index pairs (i, j), where i is the time step index of the audio and j is the time step index of the video.

[0088] Step 6: Alignment path P calculated by DTW algorithm best , reorder the audio A1 and the initially generated video V2 so that the mouth shape and speech are more matched, and the final video V3 is generated.

[0089] Step 6.1: Traverse the alignment path P best .

[0090] Step 6.2: According to the voice-driven principle, reorder the audio and video according to the time step index pairs in the alignment path. The order of the audio time steps remains unchanged. The order of the audio time steps is recorded as AS aligned ; Reselect video frames according to the index pair and construct a new video frame sequence, including the following steps:

[0091] Step 6.3.1: Process the repeated video frames. When the video frames correspond to multiple audio time steps, the video frames are repeated to ensure that the video frames can be aligned with the audio time steps.

[0092] Step 6.3.2: Process video frame jumps. The jump is defined as the absolute value of the index difference between two frames greater than 25. When a video frame jump occurs, the interpolation of the current frame and the next frame index is calculated, and the current frame and the next frame are linearly interpolated to generate a transition frame. This transition frame is inserted into the video frame sequence as the next frame. The order of the reordered video frames is recorded as VS aligned .

[0093] Step 6.4: Regenerate the video according to the sorted video frame sequence VS, and record the newly generated video as V2 reordered .

[0094] Step 6.5: Merge the original speech A1 and V2 reordered Combine them as the final aligned video V3.

[0095] The present invention can be combined with a computer system to form a voice-driven digital human construction device based on enhanced deformable convolution and spatiotemporal motion compensation. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the voice-driven digital human construction method based on enhanced deformable convolution and spatiotemporal motion compensation can be automatically executed.

[0096] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a speech-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation, characterized in that: The steps include: Step 1: Prepare the original video V1 with clear face and no obstruction, and prepare the audio A1 with clear pronunciation and no noise; Step 2: Integrate the enhanced variability convolutional network DCN into the Speech2Lip model, initialize DCN, take the original video V1 and audio A1 prepared in step 1 as input, and use the Speech2Lip model integrated with DCN to preliminarily generate a speech-driven digital human video V2; Step 3: Use the Dilb facial key point detection algorithm to extract the mouth area features of the initially generated video V2, and the extracted features are recorded as LM; modify and store the video features LM and save them as the time series V_S of the mouth shape in the video; Step 4: Use short-time Fourier transform to map the audio A1 onto the Mel-spectrogram, denoted as MS, and then transpose MS to obtain the audio sequence, denoted as A_S; Step 5: Calculate the Euclidean distance between V_S and A_S at each time step and save it as M disEu , use the DTW algorithm to calculate the path P of time series alignment best ; Step 6: Alignment path P calculated by DTW algorithm best , reorder the audio A1 and the initially generated video V2 so that the mouth shape and speech are more matched, and the final video V3 is generated.

2. The method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1: In the mouth shape generation layer of Speech2Lip, introduce three layers of DCN to replace the original convolutional layer. The specific introduction steps are as follows: Step 2.1.1: Based on the 67-channel audio feature output in the Speech2Lip model, the first layer of DCN receives the audio features for preliminary mouth shape processing. The number of input channels of the first layer of DCN is set to 67 and the number of output channels is set to 128. This layer is called D conv1 ; Step 2.1.2: The second DCN receives the output of the first layer DCN and performs further lip processing. The second layer DCN input channel is set to 128 and the output channel is set to 256. This layer is called D conv2 ; Step 2.1.2: The third DCN receives the output of the second 2-layer DCN and performs more complex and detailed lip processing. Set the third layer DCN input channel to 256 and the output channel to 512. This layer is called D conv3 ; Step 2.2: Initialize DCN, including He initialization and offset layer zero initialization. The specific steps are as follows: Step 2.2.1: After replacing the standard convolutional layer in the output layer of the Speech2Lip model to generate the mouth shape with DCN, first initialize the weights of each layer of DCN. To prevent the gradient from disappearing, select He initialization. The formula is as follows: W is the initialized weight matrix; N(0,σ2) means it obeys a normal distribution with a mean of 0 and a variance of σ2. in is the number of neurons input to the current layer; Step 2.2.2: Zero-initialize the offset generation layer of DCN; Step 2.3: Take the video V1 and audio A1 prepared in step 1 as input, and use the Speech2Lip model integrated into DCN to preliminarily generate a speech-driven digital human video V2.

3. The method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1: Load the Dlib facial key point detector model, input the generated speech-driven video V2 into the Dlib facial key point detector model for face detection, and perform facial key point detection on each frame. Each frame detection result contains 68 pairs of two-dimensional coordinates, such as [(x i0 ,y i0 ),(x i1 ,y i1 )…,(x i67 ,y i67 )], where x ij ,y ij is the coordinates of the jth pair of key points in the i-th frame, and the detection result is recorded as LM; Step 3.2: For the 68 key points recorded in each frame of the LM data, extract the 49th to 68th key points that record the mouth shape as the mouth key point data. The data format is as follows: i48 ,y i48 ),(x i49 ,y i49 )…,(x i67 ,y i67 )], denoted as LMM; Step 3.3: Concatenate the key point data of each frame in LMM into a 1D vector, such as; [x i48 ,y i48 ,x i49 ,y i49 …,x i67 ,y i67 ], at this time the length of each frame is 40, and the key point data at this time is recorded as the time series V_S used later.

4. The method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation according to claim 1, characterized in that: The specific method of step 4 is: Step 4.1: Use short-time Fourier transform to convert A1 from time domain to frequency domain. The time window size of Fourier transform is set to 2048, the moving step is 512, and the result of short-time Fourier transform is recorded as SR; Step 4.2: Take the absolute value of each element of SR to obtain the amplitude spectrum of A1. The amplitude spectrum of A1 is recorded as SM. Step 4.3: Convert SM to Mel spectrum according to the Hertz to Mel spectrum conversion formula Redistribute to Mel frequency, and record the converted Mel spectrum as MS; Step 4.4: Convert the converted Mel-spectrogram MS into decibel units, and record the converted decibel graph as MS_C; Step 4.5: Extract audio time features from MS_C, transpose MS_C, and convert it into the time series of audio A1. The audio time series of A1 is recorded as A_S.

5. The method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation according to claim 1, characterized in that: The specific method of step 5 is: Step 5.1: Through interpolation, the audio time steps with different numbers are matched with the number of video frames to achieve downsampling of the audio time series and synchronization with the video frames. The new audio time series after downsampling is recorded as A_SN. Step 5.2: Construct the distance matrix of the new audio time series A_SN and the video time series V_S, that is, calculate the Euclidean distance between each time step of A_SN and V_S, and record this distance matrix as M disEu ; Step 5.3: Construct a cumulative distance matrix to store the cost of the minimum path to the current point. The modified recursive formula is as follows: C(i,j)=D(i,j)+min(C(i,j-1),C(i-1,j-1)) Where C(i,j) is the minimum total path cost from the starting point to the (i,j)th point; D(i,j) represents M disEu The distance from the i-th time step of the recorded audio time sequence A_SN to the j-th frame of the video time sequence V_S; this cumulative distance matrix is ​​recorded as M disCu ; Step 5.4: Backtrack and accumulate the distance matrix M disCu , find the best path, that is, from the optimal alignment path, record the path as P best , where P best The elements are stored as a set of time step index pairs (i, j), where i is the time step index of the audio and j is the time step index of the video.

6. The method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation according to claim 1, characterized in that: The specific method of step 6 is: Step 6.1: Traverse the alignment path P best ; Step 6.2: According to the voice-driven principle, align the time step index pairs in the path and reorder the audio and video. The order of the audio time steps remains unchanged. The order of the audio time steps is recorded as AS aligned ; Reselect video frames according to the index pairs and construct a new video frame sequence, including the following steps: Step 6.2.1: Process the video duplicate frames, and repeat the video frames when the video frames correspond to multiple audio time steps to ensure that the video frames can be aligned with the audio time steps; Step 6.2.2: Process video frame jumps. The jump is defined as the absolute value of the difference between the two frame indexes being greater than 25. When a video frame jump occurs, the difference between the current frame and the next frame index is calculated, and the current frame and the next frame are linearly interpolated to generate a transition frame. This transition frame is inserted into the video frame sequence as the next frame. The order of the reordered video frames is recorded as VS aligned ; Step 6.3: Regenerate the video according to the sorted video frame sequence VS, and record the newly generated video as V2 reordered ; Step 6.4: Merge the original speech A1 and V2 reordered Combine them as the final aligned video V3.

7. A speech-driven digital human construction device based on enhanced deformable convolution and spatiotemporal motion compensation, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the steps of the method for constructing a voice-driven digital human based on enhanced deformable convolution and spatiotemporal motion compensation are implemented as described in any one of claims 1 to 6.

Images