AIGC video creation method for cultural relic explanation

Abstract

The invention provides an AIGC video creation method for cultural relic explanation, and belongs to the technical field of AIGC video creation. S2, cultural relic video processing; s3, audio data processing; and S4, carrying out fusion synthesis treatment. According to the method, the high-precision video processing effect and the audio authenticity processing effect can be increased, meanwhile, the interestingness can be increased by adding 3D personnel, and cultural relics can be known from multiple angles through cultural relic 3D restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of AIGC video creation, and in particular to a method for creating AIGC (Artificial Intelligence Generated Content) videos for explaining cultural relics. Background Art

[0002] As an intuitive and vivid way of communication, cultural relic explanation videos can show the charm and connotation of cultural relics to the general audience. At present, the existing cultural relic explanation video creation solutions have the following problems: 1. Video processing: Complex scene segmentation accuracy: In complex backgrounds, segmentation models such as SAM may not be able to accurately distinguish between artifacts and backgrounds, resulting in unsatisfactory video generation effects.

[0003] Naturalness of enhancement and synthesis: When ProPainter performs image restoration and enhancement, problems such as over-restoration or inconsistency with the original style may occur, affecting the naturalness and consistency of the video.

[0004] 2. Audio processing: Accuracy of speech segmentation: When faced with long audio files with multiple people, the existing speech segmentation technology may not have sufficient accuracy and efficiency, affecting the subsequent processing effect.

[0005] Realism of voice cloning: When generating the voice of a specific historical figure, voice cloning technology may have problems such as unnatural pronunciation and insufficient emotional expression, affecting the audience's immersive experience.

[0006] 3. Reconstruction of cultural relics: Reconstruction accuracy and efficiency: The NeRF model may require a lot of computing resources and time when reconstructing highly complex or fine-textured artifacts, affecting efficiency and practicality.

[0007] Fidelity of lighting and materials: During the reconstruction of cultural relics, the NeRF model may not be able to perfectly restore the real material and reflection characteristics of the cultural relics under different lighting conditions, affecting the visual effect. Summary of the invention

[0008] The present invention provides an AIGC video creation method for cultural relics explanation to solve one or more of the above problems.

[0009] The embodiment of this specification discloses an AIGC video creation method for cultural relics explanation, including: S1. Virtual character video processing: Obtain real-life video, and use the interactive segmentation and tracking model, the video object elimination model, and the human body construction model to process the video to obtain a person video; S2. Cultural Relics Video Processing: Obtain real-life video of cultural relics, and reconstruct the cultural relics using 3D modeling technology based on pure computer vision to obtain a video of the cultural relics; S3. Audio data processing: Based on real-life video, the human voice track segmentation technology is used to remove the ambient sound of the video and separate the audio tracks of different characters. Then, the human voice text is recognized using voice text, and the text content of the human voice text is passed into the text generation voice model to obtain audio data. S4. Fusion synthesis processing: The character video, cultural relic video and audio data are synchronously fused to obtain a cultural relic explanation video based on AIGC.

[0010] In some embodiments of this specification, S1 includes: S11. Image segmentation and target tracking: Use the Segment and Track Anything model to perform image segmentation and target tracking on each frame of the real-life video to generate masks and image frames of the characters in the video; S12. Video object elimination: The generated image frame and mask are input into the ProPainter model to remove video objects and complete the background, thus generating a complete video frame after removing the person. S13. Generate character point cloud data and mesh: Use the SMPLer-X model to process the image frames containing human body data, generate point cloud data and mesh files for each frame of image, and obtain 3D data; S14. Use Pthon's PYrender library to render the 3D data generated by the SMPLer-X model, add a texture to the rendered 3D data for off-screen rendering, and superimpose pixels with the original video to obtain a virtual character rendering video, which is the character video.

[0011] In some embodiments of the present specification, in S2, a video of the explanation of cultural relics recorded on site and a video of cultural relics with surround camera angles are used as video sources, and the video source is a real-shot video of cultural relics. The real-shot video data of the cultural relics is separated, and after the image frames are extracted, they are input into the NeRF model for 3D reconstruction, and the 3D model of the cultural relics is derived to obtain a video of the cultural relics.

[0012] In some embodiments of this specification, S3 includes: S31. Speech separation and extraction: The MossFormer2 model is used to separate and extract the audio from the on-site recorded cultural relic explanation video to obtain the voice data of different characters; S32. Speech to text: Use speech-to-text technology to convert speech data into text data that can be recognized by the TTS model; S33.Text to Speech Synthesis: The recognized text data is input into the VITS2 model, the target timbre is selected, and high-quality speech data with different timbre characteristics is generated, that is, audio data is obtained.

[0013] In some embodiments of the present specification, the Segment and Track Anything model uses a fusion tracking algorithm mode, which includes an interactive tracking panel and an automatic tracking panel. The interactive tracking panel SAM is used for the first frame of the video, allowing the user to accurately mark the object of interest by clicking, drawing or text input; the automatic tracking panel is used to track new objects that appear in subsequent frames in the video; The algorithm structure of the interactive tracking section is: The user interacts with SAM by clicking the mouse to segment the target object in the keyframe; SAM uses interactive prompts from the user to generate masks of objects; Use Grounding-DINO for open-set object detection based on textual cues to enhance semantic understanding and assist SAM in generating more accurate masks; Receive the SAM-generated mask as a reference frame annotation through Deep Automatic Object Tracking and initialize tracking through the Gated Propagation Module; The algorithm structure of the automatic tracking section is: Full paragraph segmentation: SAM generates masks for each object in the key reference frame, and Deep Automatic Object Tracking merges annotations to track new and old objects; Object of Interest Segmentation: Grounding-DINO is used to detect new objects based on predefined textual cues, SAM generates masks, and Deep Automatic Object Tracking handles tracking; Among them, the calculation method of the mask of the new object involves comparing the segmentation result of SAM and the current tracking result of Deep Automatic Object Tracking, and determining the new object based on the difference between the two. The formula is as follows: ; N is the new object mask, T0 is the background of the Deep Automatic Object Tracking tracking result, and S is the SAM annotation result. Only where there is a difference between the SAM annotation and the Deep Automatic Object Tracking tracking result will it be marked in the new object mask. The criterion for determining new objects is based on the ratio between each object's annotation result S in SAM and the new object mask N. When the ratio is greater than a preset minimum threshold t, the object corresponding to the ratio is defined as a new object. The formula is as follows: ; in, represents the size of the object in S, represents the size of the object in N, t represents the minimum threshold for defining a new object, when and If the ratio is greater than t, the object will be defined as a new object.

[0014] In some embodiments of this specification, the algorithm flow of the ProPainter model is: 1. Cyclic optical flow completion: ProPainter uses a cyclic network to complete the optical flow field of the masked area, which helps to perform pixel-level propagation in subsequent steps; Use the RAFT algorithm to extract the forward and backward optical flow of the video; The damaged optical flow field is completed through an efficient recurrent network to generate a complete optical flow map; The network adopts deformable alignment and propagates optical flow information bidirectionally based on deformable convolution to complete the optical flow of the masked area. 2. Dual-domain propagation: The model performs global and local propagation in both the image and feature domains, using the completed optical flow information to fill in the missing areas; Image Propagation: Perform global propagation in the image domain, using optical flow-guided deformation operations to propagate pixels from unmasked areas to masked areas; Feature Propagation: Perform local propagation in the feature domain and use the deformable alignment module to guide feature propagation based on the completed optical flow map to improve the robustness to occlusion and inaccurate optical flow completion; 3. Mask-guided Sparse Video Transformer: Use a mask-guided sparse Transformer module to refine the propagated features and enhance the spatial and temporal coherence of the infill content through a self-attention mechanism; Use the self-attention mechanism in the Transformer architecture to capture long-range dependencies in the video; Introducing a mask-guided sparsity strategy to reduce computational complexity and memory consumption by selectively applying the attention mechanism; Sparse Query Space: Attention is applied only to query windows that intersect the masked area, thus reducing unnecessary computations. Sparse Key / Value Space: Reduces the size of the key / value space by selecting key / value frames at intervals of 2 time steps, thereby reducing computational and memory costs; The refined features extracted by the self-attention mechanism are used in subsequent modules to generate the final restored video sequence.

[0015] In some embodiments of this specification, the framework of the SMPLer-X model: 1. Backbone network: Vision Transformer: As an image feature extractor, it can process large-scale data and extract rich visual features; Embedding: The input image is segmented into patches of fixed size and converted into feature vectors through the embedding layer; Positional Encoding: Add positional encoding to maintain spatial information in the image; 2. Neck network: BoxNet: predicts bounding boxes of hands and faces using features extracted from a backbone network; Region of Interest Module: Crops the region of interest from the feature map based on the predicted bounding box; 3. Head network: Hand Head and Face Heads: For the ROIs of the hand and face, a deformable convolutional network is used for feature alignment, and then 3D key points and shape parameters are obtained through regression through a fully connected layer; Body Head: The body parts use additional task tokens combined with the features of the backbone network, and the posture and shape parameters of the body are obtained through full connection layer regression; The algorithm flow of the SMPLer-X model: 1. Input video; 2. Feature extraction: The input image is processed through the backbone network to extract deep visual features; 3. Keypoint and bounding box prediction: Use the neck network to predict the bounding boxes of the hands and face, and crop the ROI from the features of the spine network; 4. Feature alignment and parameter regression: For the ROIs of hands and faces, a deformable convolutional network is used for feature alignment to adapt to different human postures; The features after head network alignment are regressed to obtain 3D key points and shape parameters; 5. Virtual human rendering and synthesis: Rendering a 3D human body mesh using the SMPL model and predicted parameters; The rendered 3D mesh is added with a texture for off-screen rendering and pixel-wise superimposed with the original video to obtain a virtual character rendering video.

[0016] In some embodiments of this specification, the NeRF model includes: Scene representation: NeRF represents a scene as a continuous 5D function whose input is the coordinates in 3D space. and 2D viewing direction , the output is the RGB color and volume density at that location ; Neural Network: NeRF uses a multi-layer perceptron as a neural network to approximate 5D functions, and the input is 5D coordinates ,The output is volume density and RGB color; Volume rendering: NeRF samples 3D points along each camera ray and uses an MLP network to predict the color and density of the 3D points. It then accumulates the color and density of the 3D points using volume rendering techniques to generate a 2D image. The formula for volume rendering is expressed as: ; in, From the camera origin Departure direction , a ray with parameter t; Optimization process: NeRF optimizes model parameters by minimizing the error between the observed image and the corresponding view rendered from NeRF. This process is differentiable and uses gradient descent method for optimization. Positional encoding: To help MLP represent high-frequency functions, NeRF uses positional encoding to map the input coordinates to a higher-dimensional space as follows: ; function Applied to 3D coordinates respectively and the unit vector d of the viewing direction; Hierarchical Sampling: To improve efficiency, NeRF uses hierarchical sampling, first using a coarse network for sampling, and then performing more targeted sampling on the fine network based on the output of the coarse network.

[0017] In some embodiments of this specification, the MossFormer2 model includes: 1. Encoder and Decoder Encoder: Input: Mixed speech waveform ; Method: Use a one-dimensional convolutional layer (Cony1D) combined with a rectified linear unit (ReLU) to encode the input. Output: Encoded sequence ,in is the embedding dimension, is the coding sequence length, the coding sequence length Calculated as ,in is the convolution kernel size; Decoder: Input: masked encoded output; Method: Use transposed 1D convolutional layers with the same kernel size and stride as the encoder; Output: reconstructed original waveform; 2. Mixing MossFormer and Cycle Modules MossFormer Module: Core: Apply self-attention on the entire sequence; Strategy: Joint local-global self-attention strategy, performing full computational self-attention on non-overlapping local segments and using a linearized self-attention mechanism on the entire sequence; Goal: Capture long-range, coarse-scale dependencies; Loop Module: Goal: Model complex temporal dependencies in speech signals and capture local cyclic patterns; Method: Perform cyclic learning for each embedding dimension; 3. No RNN loop module Module composition: Bottleneck layer: Use a 1×11×1 convolutional layer to reduce the embedding dimension; GCU layer: built on the gated convolutional unit (GLU) and the expanded FSMN; Output layer: LayerNorm followed by a 1×11×1 convolutional layer; Extended FSMN blocks: Structure: It includes a feed-forward layer and a storage layer. The storage layer uses stacked two-dimensional dilated convolution blocks. Goal: Cover a wider receptive field and enhance information flow and gradient propagation; Conv-U Block: Structure: Contains LayerNorm layer, linear layer, SiLU activation and one-dimensional deep convolution layer; Goal: Assist the GCU layer to capture the local pattern of position; 4. Masking network: maps the encoded output to a set of masks that are used to separate different speech sources; The workflow of the MossFormer2 model: Input mixed speech waveform; Encoding: Convert the waveform into a coding sequence through Conv1D and ReLU; MossFormer module: applies a joint local-global self-attention strategy to capture long-range dependencies; Loop module: uses GCU layers and dilated FSMN modules to capture complex temporal dependencies and local loop patterns; Masking network: maps the encoded output to a set of masks; Decoding: Reconstruct the original waveform using a transposed 1D convolutional layer.

[0018] In some embodiments of this specification, the VITS2 model includes: 1. Random duration predictor: Input: Hidden representation of text and Gaussian noise ; Generator :use and As input, generate the predicted duration ; Discriminator :The discriminator receives and the logarithmic duration obtained from Monotonic Alignment Search or predicted duration As input; Training strategy: Using adversarial learning, the discriminator classifies according to the actual and predicted durations; Adversarial loss function: ; ; Mean square error loss function: ; 2. Monotonic alignment search: Find the alignment with the highest probability between text and audio among all possible monotonic alignments, and add Gaussian noise during initial training to increase the diversity of alignment search; algorithm: ; in is the noise sampled from a standard normal distribution and The product of the standard deviations of 3. Normalization flow: Use convolutional blocks to capture adjacent data and add a small Transformer block with residual connections to capture long-range dependencies; 4. Speaker-Conditioned Text Encoder: Generates speech with different characteristics based on speaker conditions in a multi-speaker model; Method: Introduce speaker vector in the third Transformer block of the text encoder to learn and express the unique speech characteristics of each speaker; Workflow diagram of the VITS2 model: Input text text; Text Encoder: Encodes input text into a hidden representation ; Duration prediction: Generate predicted duration through random duration predictor ; MAS alignment: Use MAS to align text and audio, and add Gaussian noise to enhance the diversity of alignment; Normalization flow: Use a combination of convolutional blocks and Transformer blocks to normalize the data and capture long-range and local dependencies; Speaker conditioning: adding speaker vectors to the text encoder to enhance the generation capabilities of multi-speaker models; Adversarial Learning: Use the generator G and the discriminator D for adversarial training to improve the naturalness of duration prediction and speech synthesis.

[0019] The embodiments of this specification can at least achieve the following beneficial effects: Accuracy and efficiency: Traditional video processing technologies are inaccurate or inefficient when it comes to object recognition, segmentation, and tracking. STA can automatically identify dynamic objects in videos and perform precise tracking and segmentation.

[0020] Reduce costs: Manual video editing and processing usually requires a lot of manpower and time investment, but the use of new technology ProPainter can reduce the need for manual operations and reduce the cost of editing and processing.

[0021] Increased creativity: Traditional videos have limited creativity, but by adding ProPainter technology and SMPler-X for three-dimensional creation, more realistic dynamic expression and analysis can be provided.

[0022] Flexible perspective generation: Traditional 3D modeling technology may be limited in perspective generation, while NeRF technology can generate images from nearly infinite perspectives, making it possible to display artifacts in all directions. Users can observe the model from any angle, increasing interactivity; it also overcomes occlusion problems.

[0023] The present invention can increase the high-precision video processing effect and the audio authenticity processing effect. At the same time, adding 3D personnel can increase the fun. The 3D restoration of cultural relics can understand the cultural relics from multiple angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0025] Figure 1 A schematic diagram of the AIGC video creation method for cultural relics explanation involved in some embodiments of the present invention.

[0026] Figure 2 The figure is a flowchart of the AIGC video creation method for cultural relics explanation involved in some embodiments of the present invention.

[0027] Figure 3 Schematic diagram of the SMPLer-X model framework involved in some embodiments of the present invention.

[0028] Figure 4 It is a flowchart of the MossFormer2 model involved in some embodiments of the present invention.

[0029] Figure 5 It is a flowchart of the VITS2 model involved in some embodiments of the present invention. DETAILED DESCRIPTION

[0030] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0031] The disclosure below provides many different embodiments or examples to implement different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the embodiments of the present invention. In addition, the embodiments of the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0032] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 and Figure 2 As shown, the embodiment of this specification discloses an AIGC video creation method for cultural relics explanation, including: S1. Virtual character video processing: Obtain real-life video, and use the interactive segmentation and tracking model (Segment and TrackAnything), video object elimination model (ProPainter) and human body construction model (SMPLer-X) to process the video and obtain the character video; S2. Cultural Relics Video Processing: Obtain the real video of the cultural relics, and use the 3D modeling technology (NeRF) based on pure computer vision to reconstruct the cultural relics and obtain the cultural relics video; S3. Audio data processing: Based on real-life video, the human voice track segmentation technology (MossFormer2) is used to remove the ambient sound of the video and separate the sound tracks of different characters. Then, the human voice text is recognized using voice text, and the text content of the human voice text is passed into the text-generated speech model (VITS2) to obtain the audio data of the intended timbre audio (such as the voice of Lazy Sheep). S4. Fusion synthesis processing: The character video, cultural relic video and audio data are synchronously fused to obtain a cultural relic explanation video based on AIGC.

[0034] Through the above steps, the present invention can generate high-quality, natural virtual character videos, cultural relic reconstruction videos and sound videos, and realize video creation in various scenarios.

[0035] In some embodiments of this specification, S1 includes: S11. Image segmentation and target tracking: Use the Segment and Track Anything model to perform image segmentation and target tracking on each frame of the real-life video to generate masks and image frames of the characters in the video; S12. Video object elimination: The generated image frame and mask are input into the ProPainter model to remove video objects and complete the background, thus generating a complete video frame after removing the person. S13. Generate character point cloud data and mesh: Use the SMPLer-X model to process the image frames containing human body data, generate point cloud data and mesh files for each frame of image, and obtain 3D data; S14. Use Pthon's PYrender library to render the 3D data generated by the SMPLer-X model, add a texture to the rendered 3D data for off-screen rendering, and superimpose pixels with the original video to obtain a virtual character rendering video, which is the character video.

[0036] In some embodiments of the present specification, in S2, a video of the explanation of cultural relics recorded on site and a video of cultural relics with surround camera angles are used as video sources, and the video source is a real-shot video of cultural relics. The real-shot video data of the cultural relics is separated, and after the image frames are extracted, they are input into the NeRF model for 3D reconstruction, and the 3D model of the cultural relics is derived to obtain a video of the cultural relics.

[0037] In some embodiments of this specification, S3 includes: S31. Speech separation and extraction: The MossFormer2 model is used to separate and extract the audio from the on-site recorded cultural relic explanation video to obtain the voice data of different characters; S32. Speech to text: Use speech-to-text technology to convert speech data into text data that can be recognized by the TTS (text-to-speech) model; S33.Text to Speech Synthesis: The recognized text data is input into the VITS2 model, the object timbre (such as Lazy Sheep's timbre) is selected, and high-quality speech data with different timbre characteristics is generated, that is, audio data is obtained.

[0038] In some embodiments of the present specification, the Segment and Track Anything model uses a fusion tracking algorithm mode, which includes an interactive tracking panel SAM (Segment Anything Model) and an automatic tracking panel (Interactive tracking mode), wherein the interactive tracking panel is used for the first frame of a video, allowing the user to accurately mark an object of interest by clicking, drawing or text input; the automatic tracking panel is used to track new objects that appear in subsequent frames of the video; The algorithm structure of the interactive tracking section is: The user interacts with SAM by clicking the mouse (e.g., clicking the body of a person to be segmented) to segment the target object in the keyframe; SAM uses interactive prompts from the user (e.g., clicking on object outlines) to generate masks of objects; Use Grounding-DINO for open-set object detection based on textual cues to enhance semantic understanding and assist SAM in generating more accurate masks; Receive the SAM-generated mask as a reference frame annotation through Deep Automatic Object Tracking and initialize tracking through the Gated Propagation Module; The algorithm structure of the automatic tracking section is: Segment Everything: SAM generates masks for each object in the key reference frame, and DeepAutomatic Object Tracking merges annotations to track new and old objects; Object of Interest Segmentation: Grounding-DINO is used to detect new objects based on predefined textual cues, SAM generates masks, and Deep Automatic Object Tracking handles tracking. Among them, the calculation method of the mask of the new object involves comparing the segmentation result of SAM and the current tracking result of Deep Automatic Object Tracking, and determining the new object based on the difference between the two. The formula is as follows: ; N is the new object mask, T0 is the background of the Deep Automatic Object Tracking tracking result, and S is the SAM annotation result. Only where there is a difference between the SAM annotation and the Deep Automatic Object Tracking tracking result will it be marked in the new object mask. The criterion for determining new objects is based on the ratio between each object's annotation result S in SAM and the new object mask N. When the ratio is greater than a preset minimum threshold t, the object corresponding to the ratio is defined as a new object. The formula is as follows: ; in, To determine the discriminant function for a new object, represents the size of the object in S, represents the size of the object in N, and t represents the minimum threshold for defining a new object. and If the ratio is greater than t, the object will be defined as a new object.

[0039] In some embodiments of this specification, the algorithm flow of the ProPainter model is: 1. Recurrent Flow Completion (RFC): ProPainter uses a recurrent network to complete the optical flow field of the masked area, which helps with pixel-level propagation in subsequent steps; Use the RAFT algorithm to extract the forward and backward optical flow of the video; The damaged optical flow field is completed through an efficient recurrent network to generate a complete optical flow map; The network uses deformable alignment and deformable convolution (DCN) to propagate optical flow information in both directions to complete the optical flow of the masked area. 2. Dual-Domain Propagation (DDP): The model performs global and local propagation in the image and feature domains simultaneously, using the completed optical flow information to fill in the missing areas; Image Propagation: Perform global propagation in the image domain, using optical flow-guided deformation operations to propagate pixels from unmasked areas to masked areas; Feature Propagation: Perform local propagation in the feature domain and use the deformable alignment module to guide feature propagation based on the completed optical flow map to improve the robustness to occlusion and inaccurate optical flow completion; Through this dual-domain propagation, the model can effectively utilize global and local information to improve the visual consistency and coherence of the filled content; 3. Mask-Guided Sparse Video Transformer (MSVT): Uses a mask-guided sparse Transformer module to refine propagated features and enhances the spatial and temporal coherence of infill content through a self-attention mechanism; Use the self-attention mechanism in the Transformer architecture to capture long-range dependencies in the video; Introducing a mask-guided sparsity strategy to reduce computational complexity and memory consumption by selectively applying the attention mechanism; Sparse Query Space: Attention is applied only to query windows that intersect the masked area, thus reducing unnecessary computations. Sparse Key / Value Space: Reduces the size of the key / value space by selecting key / value frames at intervals of 2 time steps, thereby reducing computational and memory costs; The refined features extracted by the self-attention mechanism are used in subsequent modules to generate the final restored video sequence.

[0040] In some embodiments of this specification, Figure 3 As shown, the framework of the SMPLer-X model: 1. Backbone Network: Vision Transformer (ViT): As an image feature extractor, it can process large-scale data and extract rich visual features; Embedding: The input image is segmented into patches of fixed size and converted into feature vectors through the embedding layer; Positional Encoding: Add positional encoding to maintain spatial information in the image; 2. Neck Network: BoxNet: predicts bounding boxes of hands and faces using features extracted from a backbone network; Region of Interest (ROI) Module: Crops the region of interest from the feature map based on the predicted bounding box; 3. Head Networks: Hand Head and Face Heads: For the ROIs of the hand and face, a deformable convolutional network (DCN) is used for feature alignment, and then 3D key points and shape parameters are obtained through regression through a fully connected layer; Body Head: The body parts use additional task tokens combined with the features of the backbone network, and the posture and shape parameters of the body are obtained through full connection layer regression; The algorithm flow of the SMPLer-X model: 1. Input video; 2. Feature extraction: The input image is processed through the backbone network to extract deep visual features; 3. Keypoint and bounding box prediction: Use the neck network to predict the bounding boxes of the hands and face, and crop the ROI from the features of the spine network; 4. Feature alignment and parameter regression: For the ROIs of hands and faces, a deformable convolutional network is used for feature alignment to adapt to different human postures; The features after head network alignment are regressed to obtain 3D key points and shape parameters; 5. Virtual human rendering and synthesis: Rendering a 3D human body mesh using the SMPL model and predicted parameters; The rendered 3D mesh is added with a texture for off-screen rendering and pixel-wise superimposed with the original video to obtain a virtual character rendering video.

[0041] In some embodiments of this specification, NeRF (Neural Radiance Fields) is a deep learning model for 3D scene representation and rendering. It can learn a continuous 3D representation of a scene from a set of sparse 2D images and can render new views.

[0042] The NeRF model includes: Scene representation: NeRF represents a scene as a continuous 5D function whose input is the coordinates in 3D space. and 2D viewing direction , the output is the RGB color and volume density at that location ; is the azimuth, is the elevation angle, is RGB color, is the volume density.

[0043] Neural Network: NeRF uses a multi-layer perceptron (MLP) as a neural network to approximate 5D functions, and the input is 5D coordinates ,The output is volume density and RGB color; Volume rendering: NeRF samples 3D points along each camera ray and uses an MLP network to predict the color and density of the 3D points. It then accumulates the color and density of the 3D points using volume rendering techniques to generate a 2D image. The formula for volume rendering is expressed as: ; in, Indicates along the ray The accumulated color value of is the volume density function, which means that The density at a point. is the color function, which means that The color at a certain point. From the camera origin Departure, along the direction The rays, is a parameter.

[0044] Optimization process: NeRF optimizes model parameters by minimizing the error between the observed image and the corresponding view rendered from NeRF. This process is differentiable and uses gradient descent method for optimization. Positional encoding: To help MLP represent high-frequency functions, NeRF uses positional encoding to map the input coordinates to a higher-dimensional space as follows: ; function Applied to 3D coordinates respectively and the unit vector d of the viewing direction; are the lowest frequency sine and cosine terms in the position encoding, p is the component of the input coordinate or direction vector, are the highest frequency sine and cosine terms in the position encoding, The number of frequencies to encode the position.

[0045] Hierarchical Sampling: To improve efficiency, NeRF uses hierarchical sampling, first using a coarse network for sampling, and then performing more targeted sampling on the fine network based on the output of the coarse network.

[0046] In some embodiments of the present specification, MossFormer2 is a deep learning model for speech separation and extraction, and its main algorithmic principles can be divided into several key steps: encoding, hybrid self-attention and recurrent modules, masking, and decoding.

[0047] The MossFormer2 model includes: 1. Encoder and Decoder Encoder: Input: Mixed speech waveform ; Method: Use a one-dimensional convolutional layer (Cony1D) combined with a rectified linear unit (ReLU) to encode the input. Output: Encoded sequence ,in is the embedding dimension, is the coding sequence length, the coding sequence length Calculated as ,in is the convolution kernel size; R is the number of repetitions, and T is the input sequence length.

[0048] Decoder: Input: masked encoded output; Method: Use transposed 1D convolutional layers with the same kernel size and stride as the encoder; Output: reconstructed original waveform; 2. Mixing MossFormer and Cycle Modules MossFormer Module: Core: Apply self-attention on the entire sequence; Strategy: Joint local-global self-attention strategy, performing full computational self-attention on non-overlapping local segments and using a linearized self-attention mechanism on the entire sequence; Goal: Capture long-range, coarse-scale dependencies; Loop Module: Goal: Model complex temporal dependencies in speech signals and capture local cyclic patterns; Method: Perform cyclic learning for each embedding dimension; 3. No RNN loop module Module composition: Bottleneck layer: Use a 1×11×1 convolutional layer to reduce the embedding dimension; GCU layer: built on the gated convolutional unit (GLU) and the expanded FSMN; Output layer: LayerNorm followed by a 1×11×1 convolutional layer; Extended FSMN blocks: Structure: It includes a feed-forward layer (FFN) and a storage layer. The storage layer uses stacked two-dimensional dilated convolution blocks. Goal: Cover a wider receptive field and enhance information flow and gradient propagation; Conv-U Block: Structure: Contains LayerNorm layer, linear layer, SiLU activation and one-dimensional deep convolution layer; Goal: Assist the GCU layer to capture the local pattern of position; 4. Masking network: maps the encoded output to a set of masks that are used to separate different speech sources; like Figure 4 As shown, the workflow of the MossFormer2 model is: Input mixed speech waveform x; Encoding: Convert the waveform to the encoding sequence X through Conv1D and ReLU; MossFormer module: applies a joint local-global self-attention strategy to capture long-range dependencies; Loop module: uses GCU layers and dilated FSMN modules to capture complex temporal dependencies and local loop patterns; Masking network: maps the encoded output to a set of masks; Decoding: Reconstruct the original waveform using a transposed 1D convolutional layer.

[0049] Through these steps, the MossFormer2 model is able to effectively separate and extract different speech sources, and is particularly suitable for processing mixed speech containing multiple characters' voices.

[0050] In some embodiments of the present specification, VITS2 (Variational Inference Text-to-Speech) is an improved text-to-speech synthesis model that combines variational inference, normalized flow, and self-attention mechanism to improve the quality and efficiency of speech generation.

[0051] The VITS2 model includes: 1. Random duration predictor: Input: Hidden representation of text and Gaussian noise ; Generator :use and As input, generate the predicted duration ; Discriminator :The discriminator receives and the logarithmic duration obtained from Monotonic Alignment Search (MAS) or predicted duration As input; Training strategy: Using adversarial learning, the discriminator classifies according to the actual and predicted durations; Adversarial loss function (least-squares loss): ; ; is the adversarial loss of the discriminator, is the adversarial loss of the generator.

[0052] Mean square error loss function (MSE Ioss): ; 2. Monotonic alignment search: Find the alignment with the highest probability between text and audio among all possible monotonic alignments, and add Gaussian noise during initial training to increase the diversity of alignment search; algorithm: ; in is the noise sampled from a standard normal distribution and The product of the standard deviations of In Location and The logarithmic Gaussian probability calculated between . is a normal distribution (also called a Gaussian distribution), is the latent variable after transformation from normal flow. is the mean of the normal distribution, is the standard deviation of the normal distribution, The log-maximum likelihood value to search for a monotonic alignment. and is the previously calculated alignment probability value.

[0053] 3. Normalization flow: Use convolution blocks to capture adjacent data and add a small Transformer block with residual connections to capture long-range dependencies; Effect: The Transformer block is able to collect information from different positions while transforming the distribution.

[0054] 4. Speaker-Conditioned Text Encoder: Generates speech with different characteristics based on speaker conditions in a multi-speaker model; Method: Introduce speaker vector in the third Transformer block of the text encoder to learn and express the unique speech characteristics of each speaker; like Figure 5 As shown, the workflow diagram of the VITS2 model is: Input text text; Text Encoder: Encodes input text into a hidden representation ; Duration prediction: Generate predicted duration through random duration predictor ; MAS alignment: Use MAS to align text and audio, and add Gaussian noise to enhance the diversity of alignment; Normalization flow: Use a combination of convolutional blocks and Transformer blocks to normalize the data and capture long-range and local dependencies; Speaker conditioning: adding speaker vectors to the text encoder to enhance the generation capabilities of multi-speaker models; Adversarial Learning: Use the generator G and the discriminator D for adversarial training to improve the naturalness of duration prediction and speech synthesis.

[0055] Through these steps, the VITS2 model is able to efficiently generate natural and high-quality speech in multi-speaker environments.

[0056] The above-described embodiments are used to illustrate the present invention, not to limit the present invention, so changes in the exemplified values ​​or replacement of equivalent elements should still fall within the scope of the present invention.

[0057] From the above detailed description, it can be understood by those skilled in the art that the present invention can indeed achieve the aforementioned objectives and is in compliance with the provisions of the Patent Law.

[0058] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0059] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0060] The basic concepts have been described above. Obviously, for those of ordinary skill in the art who have read this application, the above invention disclosure is only for example and does not constitute a limitation of this application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements and amendments to this application. Such modifications, improvements and amendments are suggested in this application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of this application.

[0061] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0062] In addition, it will be understood by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvement thereof. Therefore, various aspects of the present application may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as a "unit", "module" or "system". In addition, various aspects of the present application may take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0063] The computer program code required for the operation of each part of the present application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C programming language, VisualBasic, Fortran2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on the remote computer, or run entirely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0064] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this application, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a pure software solution, for example, installation on an existing server or mobile device.

[0065] Similarly, it should be noted that in order to simplify the description disclosed in this application and thus help understand one or more embodiments of the invention, in the above description of the embodiments of the application, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this method of the application should not be interpreted as reflecting the intention that the claimed object requires more features than those explicitly stated in each claim. On the contrary, the subject of the invention should have fewer features than the above single embodiment.

Claims

1. A method for creating AIGC videos of cultural relics explanations, characterized in that: include: S1. Virtual character video processing: Obtain real-life video, and use the interactive segmentation and tracking model, the video object elimination model, and the human body construction model to process the video to obtain a person video; S2. Cultural Relics Video Processing: Obtain real-life video of cultural relics, and reconstruct the cultural relics using 3D modeling technology based on pure computer vision to obtain a video of the cultural relics; S3. Audio data processing: Based on real-life video, the human voice track segmentation technology is used to remove the ambient sound of the video and separate the audio tracks of different characters. Then, the human voice text is recognized using voice text, and the text content of the human voice text is passed into the text generation voice model to obtain audio data. S4. Fusion synthesis processing: The character video, cultural relic video and audio data are synchronously fused to obtain a cultural relic explanation video based on AIGC.

2. The AIGC video creation method for cultural relics explanation according to claim 1 is characterized in that: S1 includes: S11. Image segmentation and target tracking: Use the Segment and Track Anything model to perform image segmentation and target tracking on each frame of the real-life video to generate masks and image frames of the characters in the video; S12. Video object elimination: The generated image frame and mask are input into the ProPainter model to remove video objects and complete the background, thus generating a complete video frame after removing the person. S13. Generate character point cloud data and mesh: Use the SMPLer-X model to process the image frames containing human body data, generate point cloud data and mesh files for each frame of image, and obtain 3D data; S14. Use Pthon's PYrender library to render the 3D data generated by the SMPLer-X model, add a texture to the rendered 3D data for off-screen rendering, and superimpose pixels with the original video to obtain a virtual character rendering video, which is the character video.

3. The AIGC video creation method for cultural relics explanation according to claim 1 is characterized in that: In S2, the on-site recorded cultural relic explanation video and the cultural relic video with surround camera angles are used as video sources. The video source is the real-shot video of the cultural relic. The real-shot video data of the cultural relic is separated, and after the image frames are extracted, they are input into the NeRF model for 3D reconstruction, and the 3D model of the cultural relic is derived to obtain the cultural relic video.

4. The AIGC video creation method for cultural relics explanation according to claim 1 is characterized in that: S3 includes: S31. Speech separation and extraction: The MossFormer2 model is used to separate and extract the audio from the on-site recorded cultural relic explanation video to obtain the voice data of different characters; S32. Speech to text: Use speech-to-text technology to convert speech data into text data that can be recognized by the TTS model; S33.Text to Speech Synthesis: The recognized text data is input into the VITS2 model, the target timbre is selected, and high-quality speech data with different timbre characteristics is generated, that is, audio data is obtained.

5. The AIGC video creation method for cultural relics explanation according to claim 2 is characterized in that: The Segment and Track Anything model uses a fusion tracking algorithm mode, which includes an interactive tracking section and an automatic tracking section. The interactive tracking section is used for the first frame of the video, allowing the user to accurately mark the object of interest by clicking, drawing or text input; the automatic tracking section is used to track new objects that appear in subsequent frames in the video; The algorithm structure of the interactive tracking section is: The user interacts with the interactive tracking panel by clicking the mouse to segment the target object in the keyframe; The interactive tracking section uses the user's interactive prompts to generate masks of objects; Use Grounding-DINO for open-set object detection based on textual cues to enhance semantic understanding and assist the interactive tracking module in generating more accurate masks; Receive the mask generated by the interactive tracking block as a reference frame annotation through Deep Automatic Object Tracking and initialize tracking through the Gated Propagation Module; The algorithm structure of the automatic tracking section is: Full-paragraph segmentation: The interactive tracking module generates masks for each object in the key reference frame, and Deep Automatic Object Tracking merges annotations to track new and old objects; Object of Interest Segmentation: Grounding-DINO is used to detect new objects based on predefined textual cues, interactive tracking panels generate masks, and Deep Automatic Object Tracking handles tracking; The calculation method of the mask of the new object involves comparing the segmentation result of the interactive tracking plate and the current tracking result of DeepAutomatic Object Tracking, and determining the new object based on the difference between the two. The formula is as follows: ； N is the new object mask, T0 is the background of the Deep Automatic Object Tracking tracking result, and S is the interactive tracking plate annotation result. Only where there is a difference between the annotation of the interactive tracking plate and the tracking result of Deep Automatic Object Tracking will it be marked in the new object mask. The criterion for determining new objects is based on the ratio between the annotation result S of each object in the interactive tracking plate and the new object mask N. When the ratio is greater than a preset minimum threshold t, the object corresponding to the ratio is defined as a new object. The formula is as follows: ； in, represents the size of the object in S, represents the size of the object in N, t represents the minimum threshold for defining a new object, when and If the ratio is greater than t, the object will be defined as a new object.

6. The AIGC video creation method for cultural relics explanation according to claim 2 is characterized in that: The algorithm flow of the ProPainter model is: Circular optical flow completion: ProPainter uses a cyclic network to complete the optical flow field of the masked area, which helps with pixel-level propagation in subsequent steps; Use the RAFT algorithm to extract the forward and backward optical flow of the video; The damaged optical flow field is completed through an efficient recurrent network to generate a complete optical flow map; The network adopts deformable alignment and propagates optical flow information bidirectionally based on deformable convolution to complete the optical flow of the masked area. Dual-domain propagation: The model performs global and local propagation in both the image and feature domains, using the completed optical flow information to fill in the missing areas; Image Propagation: Perform global propagation in the image domain, using optical flow-guided deformation operations to propagate pixels from unmasked areas to masked areas; Feature Propagation: Perform local propagation in the feature domain and use the deformable alignment module to guide feature propagation based on the completed optical flow map to improve the robustness to occlusion and inaccurate optical flow completion; Mask-guided Sparse Video Transformer: Uses a mask-guided sparse Transformer module to refine propagated features and enforces spatial and temporal coherence of infill content through a self-attention mechanism; Use the self-attention mechanism in the Transformer architecture to capture long-range dependencies in the video; Introducing a mask-guided sparsity strategy to reduce computational complexity and memory consumption by selectively applying the attention mechanism; Sparse Query Space: Attention is applied only to query windows that intersect the masked area, thus reducing unnecessary computations. Sparse Key / Value Space: Reduces the size of the key / value space by selecting key / value frames at intervals of 2 time steps, thereby reducing computational and memory costs; The refined features extracted by the self-attention mechanism are used in subsequent modules to generate the final restored video sequence.

7. The AIGC video creation method for cultural relics explanation according to claim 2 is characterized in that: The framework of the SMPLer-X model: Backbone Network: Vision Transformer: As an image feature extractor, it can process large-scale data and extract rich visual features; Embedding: The input image is segmented into patches of fixed size and converted into feature vectors through the embedding layer; Positional Encoding: Add positional encoding to maintain spatial information in the image; Neck Network: BoxNet: predicts bounding boxes of hands and faces using features extracted from the backbone network; Region of Interest Module: Crops the region of interest from the feature map based on the predicted bounding box; Head network: Hand Head and Face Heads: For the ROIs of the hand and face, a deformable convolutional network is used for feature alignment, and then 3D key points and shape parameters are obtained through regression through a fully connected layer; Body Head: The body parts are combined with the features of the backbone network using additional task tokens, and the posture and shape parameters of the body are obtained through full connection layer regression; The algorithm flow of the SMPLer-X model is as follows: Input view; Feature extraction: The input image is processed through the backbone network to extract deep visual features; Keypoint and bounding box prediction: Use the neck network to predict the bounding boxes of the hands and face, and crop the ROIs from the features of the spine network; Feature Alignment and Parameter Regression: For the ROIs of hands and faces, a deformable convolutional network is used for feature alignment to adapt to different human postures; The features after head network alignment are regressed to obtain 3D key points and shape parameters; Virtual Human Rendering and Synthesis: Rendering a 3D human body mesh using the SMPL model and predicted parameters; The rendered 3D mesh is added with a texture for off-screen rendering and pixel-wise superimposed with the original video to obtain a virtual character rendering video.

8. The AIGC video creation method for cultural relics explanation according to claim 3 is characterized in that: The NeRF model includes: Scene representation: NeRF represents a scene as a continuous 5D function whose input is the coordinates in 3D space. and 2D viewing direction , the output is the RGB color and volume density at that location ; Neural Network: NeRF uses a multi-layer perceptron as a neural network to approximate 5D functions, and the input is 5D coordinates ,The output is volume density and RGB color; Volume rendering: NeRF samples 3D points along each camera ray and uses an MLP network to predict the color and density of the 3D points. It then accumulates the color and density of the 3D points using volume rendering techniques to generate a 2D image. The formula for volume rendering is expressed as: ； in, From the camera origin Departure direction , a ray with parameter t; Optimization process: NeRF optimizes model parameters by minimizing the error between the observed image and the corresponding view rendered from NeRF. This process is differentiable and uses gradient descent method for optimization. Positional encoding: To help MLP represent high-frequency functions, NeRF uses positional encoding to map the input coordinates to a higher-dimensional space as follows: ； function Applied to 3D coordinates respectively and the unit vector d of the viewing direction; Hierarchical Sampling: To improve efficiency, NeRF uses hierarchical sampling, first using a coarse network for sampling, and then performing more targeted sampling on the fine network based on the output of the coarse network.

9. The AIGC video creation method for cultural relics explanation according to claim 4 is characterized in that: The MossFormer2 model includes: Encoder: Input: Mixed speech waveform ; Method: Use one-dimensional convolutional layer combined with trimmed linear unit to encode the input line; Output: Encoded sequence ,in is the embedding dimension, is the coding sequence length, the coding sequence length Calculated as ,in is the convolution kernel size; Decoder: Input: masked encoded output; Method: Use transposed 1D convolutional layers with the same kernel size and stride as the encoder; Output: reconstructed original waveform; MossFormer Module: Core: Apply self-attention on the entire sequence; Strategy: Joint local-global self-attention strategy, performing full computational self-attention on non-overlapping local segments and using a linearized self-attention mechanism on the entire sequence; Goal: Capture long-range, coarse-scale dependencies; Loop Module: Goal: Model complex temporal dependencies in speech signals and capture local cyclic patterns; Method: Perform cyclic learning for each embedding dimension; No RNN cycle module: Bottleneck layer: Use a 1×11×1 convolutional layer to reduce the embedding dimension; GCU layer: built based on gated convolutional units and dilated FSMN; Output layer: LayerNorm followed by a 1×11×1 convolutional layer; Extended FSMN blocks: Structure: It includes a feed-forward layer and a storage layer. The storage layer uses stacked two-dimensional dilated convolution blocks. Goal: Cover a wider receptive field and enhance information flow and gradient propagation; Conv-U Block: Structure: Contains LayerNorm layer, linear layer, SiLU activation and one-dimensional deep convolution layer; Goal: Assist the GCU layer to capture the local pattern of position; Masking network: maps the encoded output to a set of masks that are used to separate different speech sources; The workflow of the MossFormer2 model: Input mixed speech waveform; Encoding: Convert the waveform into a coding sequence through Conv1D and ReLU; MossFormer module: applies a joint local-global self-attention strategy to capture long-range dependencies; Loop module: uses GCU layers and dilated FSMN modules to capture complex temporal dependencies and local loop patterns; Masking network: maps the encoded output to a set of masks; Decoding: Reconstruct the original waveform using a transposed 1D convolutional layer.

10. The AIGC video creation method for cultural relics explanation according to claim 4 is characterized in that: The VITS2 model includes: Random duration predictor: Input: Hidden representation of text and Gaussian noise ; Generator :use and As input, generate the predicted duration ; Discriminator :The discriminator receives and the logarithmic duration obtained from Monotonic Alignment Search or predicted duration As input; Training strategy: Using adversarial learning, the discriminator classifies according to the actual and predicted durations; Adversarial loss function: ； ； Mean square error loss function: ； Monotonic alignment search: Find the alignment with the highest probability between text and audio among all possible monotonic alignments, and add Gaussian noise during initial training to increase the diversity of alignment search; algorithm: ； in is the noise sampled from a standard normal distribution and The product of the standard deviations of Normalization flow: Use convolutional blocks to capture adjacent data and add a small Transformer block with residual connections to capture long-range dependencies; Speaker-Conditioned Text Encoder: Generates speech with different characteristics based on speaker conditions in a multi-speaker model; Method: Introduce speaker vector in the third Transformer block of the text encoder to learn and express the unique speech characteristics of each speaker; Workflow diagram of the VITS2 model: Input text text; Text Encoder: Encodes input text into a hidden representation ; Duration prediction: Generate predicted duration through random duration predictor ; Monotone Alignment Search: Use monotone alignment search to align text and audio, adding Gaussian noise to enhance the diversity of alignment; Normalization flow: Use a combination of convolutional blocks and Transformer blocks to normalize the data and capture long-range and local dependencies; Speaker conditioning: adding speaker vectors to the text encoder to enhance the generation capabilities of multi-speaker models; Adversarial Learning: Use the generator G and the discriminator D for adversarial training to improve the naturalness of duration prediction and speech synthesis.

Images