A speech video generation method based on audio and video structure alignment

Abstract

The application discloses a speech video generation method based on audio and video structure alignment and belongs to the virtual digital person field.The application comprises an audio segmentation module, an audio conversion module, an audio coding module, a video coding module and a video fusion decoding module.The audio conversion module is used for converting segmented phonemes into a mel-frequency spectrogram which is more in line with the frequency range of human ears according to Fourier transform.In the audio coding process, the frames of the same phonemes are taken as a continuous time module, and the time module is taken as time consistency to constrain the change of the lips, so that the fine-grained control of the lips is realized at the phoneme level through the time consistency constraint.In the video coding, the part region of the multi-pose change face in the input video is set as a mask region, the mask region is used as spatial consistency to accurately control the change amplitude of the lips, the position of the speaker's lips is aligned, the visual artifacts of the video are reduced, the facial details are optimized, and a high-quality speech video with audio-visual synchronization is generated.

Description

Technical Field

[0001] This invention relates to an audio-driven video method, and more particularly to a speech video generation method based on audio-video structure alignment, belonging to the field of virtual digital humans. Background Technology

[0002] The goal of audio-driven speech video generation is to synthesize a natural-speaking video from arbitrary audio and video inputs. It has been widely used in many human-centered applications, such as virtual assistants, animated characters, and digital humans. The generation task can be transformed into a cross-modal mapping problem between audio and video. Ambiguity arising from its one-to-many nature—that a single audio speech may have multiple lip appearances—is a major obstacle to generating high-quality speech videos.

[0003] To mitigate ambiguity, existing methods typically employ direct or indirect mapping. Direct mapping establishes a direct correspondence between audio and video frames by aligning input audio features with identity features extracted from the input video portrait (Prajwal KR, Mukhopadhyay R, Namboodiri VP, et al. A lip sync expert is all you need for speech to lip generation in the wild[C] / / Proceedings of the 28th ACM International Conference on Multimedia.2020:484-492.). This method usually generates well-synchronized lip movements through one-to-one alignment, but it suffers from a lack of spatial constraints in the mapping, especially when head pose changes significantly.

[0004] Indirect mapping uses intermediate models, such as facial landmarks or neural radiation fields, to connect two consecutive mappings: audio to intermediate model and intermediate model to facial motion (Zhou Y, Han X, Shechtman E, et al. Makelttalk: speaker-aware talking-head animation[J]. ACM Transactions On Graphics(TOG),2020,39(6):1-15.). This approach can generate good facial details by using aligned intermediate models to drive the face, but this two-step operation may lead to the accumulation of temporal errors, thereby reducing the accuracy of lip synchronization.

[0005] Besides the methods based on direct and indirect mapping mentioned above, there are other methods that utilize priors such as audio memory, video motion, and pose sources to constrain the mapping from audio to video frames (Yu L, Yu J, Li M, et al. Multimodal inputs driven talking face generation with spatial–temporal dependency[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 31(1):203-216.). In fact, these methods attempt to combine the principles of direct and indirect mapping, leveraging their relative advantages to obtain high-quality speech videos with good lip synchronization and facial detail. The method in this paper also operates in this way, exploring the spatial and temporal consistency of the audio and video portraits themselves as constraints for speech video generation. Summary of the Invention

[0006] To address the limitations of current audio-driven speech video generation methods in accurately depicting lip-sync and facial details, this invention aims to provide a speech video generation method and system based on audio-video structural alignment. During speech video generation, spatiotemporal consistency is used to optimize the process. Temporal consistency improves the accuracy of audio-to-lip translation by constraining lip shape changes for the same phoneme, while spatial consistency reduces visual artifacts by aligning facial positions. This results in high-quality, audio-visual synchronized speech videos with accurate lip-sync and realistic facial details.

[0007] The objective of this invention is achieved through the following technical solution.

[0008] This invention discloses a method for generating speech videos based on audio-video structural alignment. In the audio encoding process, frames with the same phoneme are treated as a continuous temporal module, and this temporal module is used to constrain lip movements for temporal consistency. This temporal consistency constraint enables fine-grained control of the lips at the phoneme level. In the video encoding process, certain regions of the face exhibiting multiple poses in the input video are set as mask regions. These mask regions are used to precisely control the amplitude of lip movements for spatial consistency, aligning the speaker's lip position, reducing visual artifacts in the video, optimizing facial details, and generating high-quality speech videos with synchronized audio and video.

[0009] This invention discloses a method for generating speech videos based on audio-visual structure alignment, comprising the following steps:

[0010] Step 1: Audio Segmentation. The input audio is segmented into phonemes. By extracting audio feature information from the fine-grained phoneme segments, the extracted audio feature information has richer audio detail features.

[0011] Input audio is divided according to phonemes, as follows:

[0012] P = {p i |1≤i≤M}

[0013] P represents the set of phonemes, M represents the total number of phonemes, p i represents a phoneme, and i represents the current number of phonemes.

[0014] Step 2: Phoneme Conversion. The phonemes are converted into Mel spectrograms using Fourier transform, which better matches the frequency range of the human ear.

[0015] The formula for converting Fourier transform to Mel spectrum is as follows:

[0016]

[0017] F(ω) represents the Mel spectrum, f(t) represents the phoneme, and e -iωt Represents a complex function;

[0018] The Mel spectrogram of each phoneme obtained above is divided into its own time module through temporal sequence. Each time module contains the Mel spectrogram of one phoneme. By dividing the time module, the mapping relationship between phonemes and video is accurately established, and the accuracy of audio to speaker's lips is improved through the mapping relationship.

[0019] Each time module and its contained Mel spectrogram are represented as follows:

[0020] B = {b} j |1≤j≤N}

[0021] B represents the set of time modules, M represents the total number of frames in the Mel spectrogram, b j represents the Mel spectrogram, and j represents the current frame number of the Mel spectrogram.

[0022] Step 3: Audio Encoding. Audio encoding is performed based on the mapping relationship established in Step 2. During the audio encoding process, frames with the same phoneme are treated as a continuous time module, and the time module is used as a temporal consistency constraint to constrain the changes in the lips. Through the temporal consistency constraint, fine-grained control of the lips is achieved at the phoneme level.

[0023] Step 4: Video Encoding. Video encoding is performed by analyzing facial key points of the face in various poses within the input video. During video encoding, specific areas of the face in different poses are set as mask regions. These mask regions are used to precisely control the amplitude of lip movements to ensure spatial consistency, aligning the speaker's lips with their position, reducing visual artifacts in the video, and optimizing facial details.

[0024] Preferably, the width and height of the mask area are the lower half of the face, i.e., the width of the face is w and the height is h, and the width of the mask area is w and the height is h.

[0025] Step 5: Video Decoding. The time-consistent audio code obtained in Step 3 and the spatially consistent video code obtained in Step 4 are used for video fusion decoding. In video decoding, centrally symmetric interpolation is used to fill the lips into the mask area. Based on accurate lip synchronization and realistic facial details, a high-quality speech video with synchronized audio and video is generated.

[0026] Preferably, the lip filling is calculated using centrally symmetric interpolation. First, interpolation is performed in the x-direction, as follows:

[0027]

[0028]

[0029] Then, interpolation is performed in the y-direction:

[0030]

[0031] (x1,y1) and (x2,y2) represent the pixels in the source image, and (x,y) represent the pixels in the target image.

[0032] This invention also discloses a speech video generation system based on audio-video structure alignment, implemented based on the aforementioned speech video generation method based on audio-video structure alignment. The speech video generation system based on audio-video structure alignment includes an audio segmentation module, an audio conversion module, an audio encoding module, a video encoding module, and a video fusion decoding module.

[0033] The audio segmentation module is used to segment the input audio into phonemes. By extracting audio feature information from the fine-grained phonemes of the audio segmentation, the extracted audio feature information has richer audio detail features.

[0034] The audio conversion module is used to convert the phonemes segmented by the audio segmentation module into Mel spectrograms that are more in line with the frequency range of the human ear, based on Fourier transform.

[0035] The audio encoding module is used to constrain the changes in the lips by using the time module formed by the Mel spectrogram as a time consistency constraint, thereby achieving fine-grained control of the lips at the phoneme level through time consistency constraint.

[0036] The video encoding module is used to set a portion of the face with multiple pose changes in the input video as a mask area. The mask area is used to precisely control the range of lip changes by maintaining spatial consistency, aligning the speaker's lip position, reducing visual artifacts in the video, and optimizing facial details.

[0037] The video fusion decoding module is used to perform video fusion decoding on the temporally consistent audio encoding and spatially consistent video encoding. It uses centrally symmetrical interpolation calculation to fill the lips into the mask area, generating a high-quality speech video with synchronized audio and video based on accurate lip synchronization and realistic facial details.

[0038] Beneficial effects:

[0039] 1. The present invention discloses a method and system for generating speech videos based on audio-video structure alignment, which establishes temporal consistency between audio and video, uses consecutive frames with the same phoneme as time modules, captures lip movements based on audio, restricts lip movements with the same phoneme, and improves the synchronization between audio and lips.

[0040] 2. The present invention discloses a method and system for generating speech videos based on audio-visual structure alignment. By utilizing spatial consistency of faces in different poses, the method precisely controls the range of lip changes, aligns the position of the speaker's lips, reduces visual artifacts in the video, and optimizes facial details.

[0041] 3. The present invention discloses a method and system for generating speech videos based on audio-visual structure alignment. It performs video fusion decoding through temporally consistent audio coding and spatially consistent video coding. In the video decoding, centrally symmetrical interpolation calculation is used to fill the lips into the mask area. Based on accurate lip synchronization and realistic facial details, a high-quality speech video with synchronized audio and video is generated. Attached Figure Description

[0042] Figure 1 This is a flowchart of a speech video generation method based on audio-video structure alignment according to the present invention.

[0043] Figure 2 This is a diagram illustrating the audio-to-lip transmission process, where: Figure 2 (a) Input audio into the audio segmentation module. Figure 2 (b) represents the phonemes obtained from the input audio segmentation in the audio segmentation module. Figure 2 (c) is the phoneme-to-Mel spectrogram converted in the audio conversion module. Figure 2(d) is the lips encoded by the time module in the audio encoding module.

[0044] Figure 3 This is a schematic diagram of the mask area in the video encoding module.

[0045] Figure 4 This is a schematic diagram of filling in video fusion decoding. Detailed Implementation

[0046] The following is in conjunction with the appendix Figure 1 –4. A detailed description of the implementation of the method of the present invention is provided.

[0047] like Figure 1 As shown in the figure, the specific implementation steps of the speech video generation method based on audio and video structure alignment disclosed in this embodiment are as follows:

[0048] Step 1: Audio Segmentation. The input audio is segmented into phonemes. By extracting audio feature information from the fine-grained phoneme segments, the extracted audio feature information has richer audio detail features.

[0049] Input audio is divided according to phonemes, as follows:

[0050] P = {p i |1≤i≤M}

[0051] P represents the set of phonemes, M represents the total number of phonemes, p i This represents a phoneme, where 'i' represents the current number of phonemes. For example... Figure 2 As shown in (b).

[0052] Step 2: Phoneme Conversion. The phonemes are converted into Mel spectrograms using Fourier transform, which better matches the frequency range of the human ear.

[0053] The formula for converting Fourier transform to Mel spectrum is as follows:

[0054]

[0055] F(ω) represents the Mel spectrum, f(t) represents the phoneme, and e -iωt Represents a complex function;

[0056] The Mel spectrogram of each phoneme obtained above is divided into its own time modules based on temporal sequence. Each time module contains the Mel spectrogram of one phoneme. By dividing the time modules, a mapping relationship between phonemes and video is accurately established, thereby improving the accuracy of audio-to-speaker lip-sync. Specifically, as follows... Figure 2 As shown in (c).

[0057] Each time module and its contained Mel spectrogram are represented as follows:

[0058] B = {b} j |1≤j≤N}

[0059] B represents the set of time modules, M represents the total number of frames in the Mel spectrogram, b j represents the Mel spectrogram, and j represents the current frame number of the Mel spectrogram.

[0060] Step 3: Audio Encoding. Audio encoding is performed based on the mapping relationship established in Step 2. During the audio encoding process, frames with the same phoneme are treated as a continuous temporal module, and this temporal module is used to constrain lip movements through temporal consistency constraints. Fine-grained control of the lips is achieved at the phoneme level through these temporal consistency constraints. Specifically... Figure 2 As shown in (d).

[0061] Step 4: Video Encoding. Video encoding is performed by analyzing facial key points of the face in various poses within the input video. During video encoding, specific areas of the face in different poses are set as mask regions. These mask regions are used to precisely control the amplitude of lip movements to ensure spatial consistency, aligning the speaker's lips with their position, reducing visual artifacts in the video, and optimizing facial details.

[0062] Preferably, the width and height of the mask area are the lower half of the face, i.e., the width of the face is w and the height is h, and the width of the mask area is w and the height is h. Specifically, such as Figure 3 As shown.

[0063] Step 5: Video Decoding. The time-consistent audio code obtained in Step 3 and the spatially consistent video code obtained in Step 4 are used for video fusion decoding. In video decoding, centrally symmetric interpolation is used to fill the lips into the mask area. Based on accurate lip synchronization and realistic facial details, a high-quality speech video with synchronized audio and video is generated.

[0064] The lip filling is calculated using centrally symmetric interpolation. First, interpolation is performed in the x-direction, as follows:

[0065]

[0066]

[0067] Then, interpolation is performed in the y-direction:

[0068]

[0069] (x1, y1) and (x2, y2) represent the pixels in the source image, and (x, y) represent the pixels in the target image. Specifically... Figure 4 As shown.

[0070] To illustrate the content and implementation method of this invention, a specific embodiment is provided in this specification. The details introduced in the embodiment are not intended to limit the scope of the claims, but rather to aid in understanding the method described in this invention. Those skilled in the art will understand that various modifications, variations, or substitutions to the steps of the preferred embodiment are possible without departing from the spirit and scope of this invention and its appended claims. Therefore, this invention should not be limited to the content disclosed in the preferred embodiment and the accompanying drawings.

Claims

1. A speech video generation method based on audio-video structure alignment, characterized in that: The method comprises the following steps: Step one, audio segmentation; the input audio is segmented into phonemes, and the audio feature information is extracted by segmenting the audio into fine-grained phonemes, so that the extracted audio feature information has more rich audio detail features; Step two, phoneme conversion; the phonemes are converted into mel-spectrogram which is more consistent with the frequency range of human ear according to Fourier transform; The formula for converting Fourier transform into mel-spectrogram is as follows: F(ω) denotes a mel-spectrogram, f(t) denotes a phoneme, e -iωt denotes a complex-valued function; The mel-spectrogram of each phoneme obtained above is divided into respective time modules by time sequence, and each time module contains a mel-spectrogram of a phoneme, and the mapping relationship between the phonemes and the video is accurately established by dividing the time modules, and the accuracy of audio to the speaker's lips is improved through the mapping relationship; Each time module and the contained mel-spectrogram are represented as follows: B = {b j |1≤j≤N} B represents a set of time modules, M represents the total number of mel-spectrogram frames, b j represents a mel-spectrogram, and j represents the current frame number of the mel-spectrogram; Step three, audio encoding; audio encoding is performed based on the mapping relationship constructed in step two, in the audio encoding process, frames of the same phoneme are taken as a continuous time module, and the time module is taken as time consistency to constrain the change of the lips, and the fine-grained control of the lips is realized at the phoneme level through the time consistency constraint; Step four, video encoding; video encoding is performed by inputting the face key points of the multi-pose changing face in the video, in the video encoding, part of the region of the multi-pose changing face in the input video is set as a mask region, and the mask region is used for accurate control of the change amplitude of the lips as spatial consistency, aligning the position of the speaker's lips, reducing the visual artifacts of the video, and optimizing the facial details; Step five, video decoding; video fusion decoding is performed by time consistency audio encoding obtained in step three and spatial consistency video encoding obtained in step four, in the video decoding, the lips are filled into the mask region by using center-symmetric interpolation calculation, and on the basis of accurate lip synchronization and realistic facial details, a high-quality speech video with audio-visual synchronization is generated. 2.The speech video generation method based on audio-video structure alignment of claim 1, wherein: In step one, The input audio is divided according to phonemes, as follows: P = {p i |1≤i≤M} P represents a set of phonemes, M represents the total number of phonemes, p i represents a phoneme, and i represents the current number of phonemes. 3.The speech video generation method based on audio-video structure alignment of claim 2, wherein: The width and height of the mask region are the lower half of the face, i.e. the width of the face is w, the height of the face is h, the width of the mask region is w, and the height of the mask region is 4. The speech video generation method based on audio-video structure alignment according to claim 3, characterized in that: The center-symmetric interpolation calculation is used for the lip filling, first interpolation is performed in the x direction, as follows: Then, interpolation is performed in the y direction: (x1, y1) and (x2, y2) represent the pixel points of the source image, and (x, y) represents the pixel points of the target image.

5. A speech video generation system based on audio-video structure alignment, implemented based on a speech video generation method based on audio-video structure alignment as claimed in claim 1, 2, 3 or 4, characterized in that: The method comprises an audio segmentation module, an audio conversion module, an audio encoding module, a video encoding module, and a video fusion decoding module; The audio segmentation module is used to segment the input audio into phonemes, and the audio feature information is extracted by segmenting the audio into fine-grained phonemes, so that the extracted audio feature information has more rich audio detail features; The audio conversion module is used to convert the phonemes segmented by the audio segmentation module into mel-spectrogram which is more consistent with the frequency range of human ear according to Fourier transform; The audio encoding module is used to take the time module composed of the mel-spectrogram as time consistency to constrain the change of the lips, and realize fine-grained control of the lips at the phoneme level through time consistency constraint; The video coding module is configured to set a part region of a multi-pose changing face in the input video as a mask region, control a lip change range through the mask region as spatial consistency, align a speaker lip position, reduce visual artifacts of the video, and optimize facial details; The video fusion decoding module is configured to perform video fusion decoding on the time-consistent audio coding and the spatial-consistent video coding, fill the lip to the mask region through central symmetric interpolation calculation, and generate a high-quality speech video with audio-visual synchronization on the basis of accurate lip synchronization and realistic facial details.

Images