A method and system for creating AIGC multimodal audiovisual content

Abstract

The present invention discloses an AIGC multimodal audio-visual content creation method and system, which belongs to the technical field of multimodal audio-visual content generation and automated creation, including an input module, in which a user inputs text descriptions and materials required for creation; an AIGC generation module, which converts the text descriptions and materials input by the user into video sequences and music clips; a multimodal fusion and alignment module, which fuses and aligns the generated video sequences and music clips to obtain audio and video; a video intelligent editing module, which splices audio and video clips to maintain visual coherence and a smooth connection between the two audio and video clips; and an output module, which outputs the generated audio and video into the media format required by the user. The present invention solves the problems existing in the prior art of multimodal audio-visual content creation, such as the lack of cross-modal collaborative working ability, the inconsistency in rhythm, semantics and emotion of the content generated when multimodal information is fused and aligned, the lack of support for the post-editing process, the unnaturalness of content connection, and the slow creation speed.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal audiovisual content generation and automated creation, and in particular to an AIGC multimodal audiovisual content creation method and system. Background Art

[0002] With the rapid development of generative artificial intelligence (AIGC) technology, content generation technology based on deep learning has made significant progress in many fields. Existing AIGC technologies, such as text-generated image, video generation, music generation, and speech synthesis, have been widely used. For example, text-generated image technology can convert natural language text into high-quality images through generative adversarial networks (GAN) and diffusion models; video generation technology uses deep learning models to generate short video clips; music generation technology can automatically generate music according to specified emotions or rhythms; speech synthesis technology (TTS) is also widely used in scenarios such as virtual assistants and video dubbing.

[0003] Although these single-modality generation technologies have made significant progress, there are still many deficiencies in the existing technologies for the creation of multimodal audiovisual content. First, most existing systems can only handle the generation of a single modality (such as images, videos, or audio), lack the ability to work across modalities, and the creation tools of different modalities are often independent, resulting in the need for manual intervention in the creation process to integrate and align multimodal information, which increases the cost and time of creation.

[0004] Existing technologies also face great technical challenges when dealing with the fusion and alignment of multimodal information. Audiovisual content creation usually requires the fusion of multiple modal information such as text, images, audio and video. However, due to the differences in semantics and timing between different modalities, it is difficult for existing systems to achieve automatic multimodal alignment, which can easily lead to inconsistent rhythm, semantics or emotions in the generated content, affecting the overall creative effect.

[0005] At the same time, the generation of video content not only involves the production of materials, but also requires complex editing operations, such as shot splicing, shot switching, transition effects, etc. Most of the current AIGC technologies focus on material generation and lack support for the post-editing process. Users need to complete these operations manually, which greatly reduces the degree of automation and efficiency of creation. Finally, when generating complex narrative content, existing generation systems often find it difficult to ensure the coherence and consistency of the content. Especially in the field of video generation, the clips generated by existing technologies usually lack a clear narrative structure, and the transitions and connections between shots are not natural enough, which affects the fluency and look of the final work.

[0006] Therefore, the existing technology has obvious deficiencies in multimodal content generation, fusion and editing. There is an urgent need for a system that can integrate multimodal generation technology, automatically realize information alignment and fusion, and support intelligent editing and splicing operations, so as to improve the efficiency and quality of multimodal audio-visual content creation. Summary of the invention

[0007] The purpose of the present invention is to provide an AIGC multimodal audio-visual content creation method and system to solve the problems existing in the prior art of multimodal audio-visual content creation, such as the lack of cross-modal collaborative working capability, the inconsistency of the rhythm, semantics and emotion of the content generated when multimodal information is fused and aligned, the lack of support for the post-editing process, the unnatural connection of the content, and the slow creation speed.

[0008] To achieve the above object, the present invention provides a method for creating AIGC multimodal audiovisual content, comprising the following steps:

[0009] S1, the user inputs text description and materials, including images, video clips and music;

[0010] S2, use GPT-4 to analyze and optimize the text description, and then convert the optimized and analyzed text description into several images through the stable diffusion XL model;

[0011] S3, using a stable video diffusion model to expand the images obtained in S2 and the images contained in the material into several video sequences;

[0012] S4, uses natural language processing technology to analyze the keywords and important information in the text description, and then inputs the keywords and important information into Suno, which parses the keywords and important information and then generates a music clip;

[0013] S5, analyzing and synchronizing the text description, video sequence and music clips to make them consistent in content and time, and obtaining several audio and video;

[0014] S6, using the short-to-long video diffusion model to connect several audio and video together to obtain a long audio and video;

[0015] S7, outputs the generated long audio and video into a media format desired by the user.

[0016] Preferably, in S1, the user must input a text description in the information input, and can optionally input materials.

[0017] Preferably, in S4, after Suno generates the music clip, the music clip generated by Suno is fed back to the user, and the user modifies the music clip by changing the text description according to actual needs.

[0018] Preferably, the process of S5 is to first perform multimodal feature alignment and then perform time alignment. The specific steps of S5 are:

[0019] S51, input the text description into the pre-trained language model, extract the semantic features in the text description, and capture the information about the emotion, theme and style in the text;

[0020] S52, using a residual convolutional neural network model to divide the video sequence generated in S3 and the video in the material into a number of video frames, and perform frame extraction and visual feature extraction;

[0021] S53, using a 3D convolutional neural network model and a long short-term memory model to extract dynamic features of the video sequence and the video in the material generated in S3, and using the dynamic features as the timing information of the video;

[0022] S54, segmenting the music in the material and the music clip generated in S4 into short clips synchronized with the video frames, and then using the Mel frequency spectrogram and Mel frequency cepstral coefficient extraction model to extract frequency, rhythm and pitch features of the music in the material and the music clip generated in S4;

[0023] S55, feature alignment using a shared embedding space to align features of text descriptions, videos, and music;

[0024] S56, using the audio-video converter model to time-align the features of the text description, video, and music to obtain several audio and video.

[0025] Preferably, in S55, the features of the text description, video and music are processed using a contrastive learning optimization model in a shared embedding space so that similar features are closer and dissimilar features are farther away.

[0026] Preferably, in S6, a natural language processing model is used to parse the text description, key information related to transition in the text description is extracted, and the key information is input into a short-to-long video diffusion model to generate transition frames that meet customer needs.

[0027] The present invention also provides an AIGC multimodal audiovisual content creation system, using the above-mentioned AIGC multimodal audiovisual content creation method, comprising:

[0028] Input module, used for users to input text descriptions and materials required for creation, including images, video clips and music;

[0029] The AIGC generation module is used to convert the text description and materials input by the user into video sequences and music clips that meet the requirements;

[0030] The multimodal fusion and alignment module is used to fuse and align the generated video sequences and music clips to make the semantics and timing between the video sequences and the music clips consistent, thus obtaining audio and video;

[0031] Video intelligent editing module, used to splice audio and video clips, adjust the transition frames between several audio and video, and maintain visual coherence and connection between two audio and video;

[0032] The output module is used to output the generated audio and video into the media format required by the user.

[0033] Therefore, the present invention adopts an AIGC multimodal audio-visual content creation method and system of the above structure, which has the following beneficial effects:

[0034] (1) The system of the present invention integrates multiple generative artificial intelligence technologies such as text generation, image generation, video generation, and music generation, and can generate audiovisual content of multiple modalities from a single input, greatly simplifying the creation process, reducing manual intervention, and realizing the automatic generation of audiovisual content;

[0035] (2) The system of the present invention ensures the semantic and temporal consistency and coherence of content in different modalities through the fusion and alignment technology of multimodal information, solves the problem of multimodal information being difficult to integrate naturally in the prior art, and makes the generated audio-visual content more fluent and coordinated;

[0036] (3) The present invention provides an intelligent video editing function, which automatically completes video splicing, lens switching and transition effect generation, thereby improving the efficiency and quality of content creation; the system automatically adjusts lens transitions and special effects generation according to the video plot, making the video more natural and smooth, reducing the workload of users in post-editing; through the intelligent editing process, the system can complete high-quality content creation in a very short time;

[0037] (3) The system of the present invention also has strong flexibility and customization capabilities. Users can adjust the style, format and duration of the generated content according to specific needs to meet the needs of different creation scenarios. It is applicable to multiple fields, improves the creation efficiency and quality of multimodal audio-visual content, lowers the creation threshold, and realizes efficient, low-cost and highly personalized audio-visual content creation.

[0038] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a structural schematic diagram of an AIGC multimodal audio-visual content creation method and system embodiment of the present invention;

[0040] Figure 2It is a flowchart of an AIGC multimodal audio-visual content creation method and system embodiment S5 of the present invention;

[0041] Figure 3 The present invention is a flowchart of an AIGC multimodal audio-visual content creation method and system embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0043] The components of embodiments of the present invention may be arranged and designed in a variety of different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0046] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0047] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "setting", "installation" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0049] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “left”, “right”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they cannot be understood as limitations on the present invention.

[0050] The specific model specifications need to be selected and determined based on the actual specifications of the device, and the specific selection calculation method adopts the existing technology in this field, so it will not be described in detail.

[0051] like Figure 1 As shown, the present invention provides an AIGC multimodal audio-visual content creation system, including an input module for users to input text descriptions and materials required for creation, the materials including images, video clips and music;

[0052] The AIGC generation module is used to convert the text description and materials input by the user into video sequences and music clips that meet the requirements;

[0053] The multimodal fusion and alignment module is used to fuse and align the generated video sequences and music clips to make the semantics and timing between the video sequences and the music clips consistent, thus obtaining audio and video;

[0054] The multimodal fusion and alignment module is responsible for fusing and aligning the generated multimodal content to ensure semantic and temporal consistency between different modal information. Through deep learning models such as Cross-modal Alignment Networks, the system automatically analyzes and synchronizes text, video, and audio to keep them consistent in content and synchronize them on the timeline. This module can also automatically adjust the style of music and the rhythm of dubbing according to the mood or rhythm of the video to ensure the natural combination of multimodal content.

[0055] Video intelligent editing module, used to splice audio and video clips, adjust the transition frames between several audio and video, maintain visual coherence and smooth connection between two audio and video;

[0056] The video intelligent editing module provides users with automated post-processing functions. The module can intelligently complete the splicing of audio and video clips, use deep learning models to analyze audio and video content, automatically select appropriate shot switching points, and generate natural transition effects. Specifically, the present invention adopts the short to long video diffusion model (SEINE) model, which can generate a natural transition effect between two audio and video clips, ensuring that the audio and video remain visually consistent and seamless. The implementation of this process relies on text descriptions or instructions provided by the user to guide the transition and conversion of the scene, so that the style, content and time between the two audio and video segments remain consistent.

[0057] The output module is used to output the generated audio and video into the media format required by the user.

[0058] like Figure 3 As shown, the present invention also provides a creation method of an AIGC multimodal audio-visual content creation system, comprising the following steps:

[0059] S1, the user inputs text description and materials, including images, video clips and music;

[0060] In S1, the information input by the user must include a text description, and the material can be input selectively.

[0061] S2, uses GPT-4 to analyze and optimize the text description to ensure that these prompts can provide enough key information for image generation.

[0062] The optimized and analyzed text description is then converted into several images through the Stable Diffusion XL (SDXL) model;

[0063] As a latent diffusion model, the SDXL model has the ability to generate high-resolution images. It uses an extended UNet architecture and combines a dual text encoder system to interpret and process complex text in detail. By understanding the input text at multiple levels, the SDXL model can generate rich and sophisticated images that meet user needs.

[0064] The working principle of the diffusion model is to gradually transform random noise samples into high-quality images through a gradual denoising process. The SDXL model denoises in the latent space and then maps the denoised results back to the pixel space through the decoder to finally generate the image. The model also includes an image quality improvement mechanism that can perform further refinement after the image is generated to improve the visual effect of the image.

[0065] S3, uses the Stable Video Diffusion (SVD) model to expand the images obtained in S2 and the images contained in the material into several video sequences;

[0066] The SVD model processes images in the temporal dimension to ensure that the generated video frames remain consistent and coherent in time. It processes spatial and temporal features through the 3DUNet structure to ensure smooth transitions in the temporal dimension of the video while retaining the details and style of the initial image. In addition, the SVD model is able to generate multi-view videos and achieve dynamic changes such as scene translation and push-pull shots by simulating the motion effects of the camera. The generation of each frame not only takes into account the content of the previous frame, but also generates appropriate dynamic effects based on the text description, such as the movement of characters and physical changes in the scene.

[0067] S4 uses natural language processing technology to analyze the keywords and important information in the text description, and extracts core words such as "relaxed", "jazz", and "early morning" to help better define the style, emotion, and atmosphere of the music. Then build a request and input the keywords and important information into the request. This request will be sent to Suno's server through the network to request the generation of music. Suno will parse these inputs and generate corresponding music clips. The generated music may be in jazz style, with a relaxing rhythm, which fits the emotional atmosphere of "early morning". After Suno generates the music clip, it feeds back the music clip generated by Suno to the user, and the user modifies the music clip by changing the text description according to actual needs. For example, if the music is required to be faster or slower, a new music clip will be generated again. At the same time, in S4, the above method can be used to synthesize a voice clip that meets the requirements.

[0068] like Figure 2 As shown, S5 analyzes and synchronizes the text description, video sequence, and music clips to make them consistent in content and time, and obtains several audio and video clips;

[0069] The process of S5 is to first perform multimodal feature alignment and then perform time alignment. The specific steps of S5 are:

[0070] S51, input the text description into the pre-trained language model, extract the semantic features in the text description, and capture the information about the emotion, theme and style in the text;

[0071] The text description is input into a pre-trained language model such as GPT or BERT to extract the semantic features of the text. Text features are usually represented as context-aware vector representations that can capture information such as sentiment, theme, style, etc. in the text description.

[0072] S52, using a residual convolutional neural network model (ResNet), divides the video sequence generated in S3 and the video in the material into several video frames, and performs frame extraction and visual feature extraction;

[0073] Using a ResNet pre-trained convolutional neural network (CNN), multiple frames are extracted from the video per second, and each frame is input into the CNN model to extract visual features. These visual features can be represented as vectors of fixed dimensions.

[0074] S53, uses a 3D convolutional neural network model (3D-CNN) and a long short-term memory model (LSTM) to extract dynamic features of the video sequences and videos in the material generated in S3, ensures the temporal consistency of the video, and uses the dynamic features as the timing information of the video;

[0075] S54, the music in the material and the music clip generated in S4 are divided into short clips synchronized with the video frames, and then the frequency, rhythm and pitch features of the music in the material and the music clip generated in S4 are extracted using a Mel spectrogram (Mel spectrogram) and Mel frequency cepstral coefficient (MFCC audio feature extraction model) extraction model. If the music clip and the music in the material here contain speech content, an automatic speech recognition system (Automatic Speech Recognition, ASR) can be used for speech-to-text conversion.

[0076] S55, feature alignment using a shared embedding space to align features of text descriptions, videos, and music;

[0077] The specific process of S55 is to use the contrastive learning optimization model in the shared embedding space to process the features of text descriptions, videos, and music, so that similar features are closer and dissimilar features are farther apart. The specific method is to use related audio, video, images, and text as positive samples and irrelevant ones as negative samples. The model learns cross-modal alignment by optimizing contrastive loss (such as NT-Xent loss function).

[0078] S56, using the audio-video converter model to time-align the features of the text description, video, and music to obtain several audio and video.

[0079] Specifically, video frames, audio clips, text descriptions, and images all need to be consistent at the same point in time. For example, a scene in a video should be played synchronously with the corresponding audio clip, and the corresponding text description or subtitles should be displayed when the scene occurs. Here, the deep learning time alignment model (Audio-Visual Transformer) architecture is used to achieve this. After the features of audio, video, and text descriptions are input into the time alignment model, the model can learn the relationship between time steps through the self-attention mechanism, capture the association between different modalities in the time dimension, and thus achieve multi-modal synchronization.

[0080] Through the cross-modal alignment network, the system can automatically analyze and synchronize text, video, audio, and images to keep them consistent in content and time. The key to achieving this goal is to use a shared embedding space, contrastive learning, cross-modal attention mechanisms, and time series alignment methods. Through these techniques, data from different modalities can be mapped to the same semantic space and kept synchronized in the time dimension, providing a strong foundation for applications such as multimodal interaction, search, and sentiment analysis.

[0081] S6, using the short-to-long video diffusion model to connect several audio and video together to obtain a long audio and video;

[0082] The specific process in S6 is to use the natural language processing model (NLP) to parse the text description, extract the key information related to the transition in the text description, input the key information into the short to long video diffusion model (SEINE), and generate transition frames that meet customer needs.

[0083] The core of SEINE is the diffusion model, which is a kind of generative model. It gradually generates realistic pictures or video frames by gradually adding the features of the image or video to the random noise and reverse denoising. The working principle of the diffusion model includes the forward process and the reverse process. The forward process is to gradually add noise to the image / video until they become completely random. The reverse process is that the SEINE model learns how to recover the video frames from the random noise, that is, gradually denoising. The denoising process is the process of generating new video frames, which can naturally connect two different audio and video clips.

[0084] The diffusion model generates a series of transition frames between two video clips, ensuring that the visual content, color, and style remain smoothly transitioned during this period. A key feature of the SEINE model is the ability to control the generation of transition frames through text descriptions. Users can enter text to describe the transition content between two scenes, such as "smooth transition from the first video scene to the second scene" or "gradually change from a quiet city night scene to a sunrise scene at the seaside." These text descriptions provide guidance information for the SEINE model to generate video transition frames. For example, if the user requires a sunrise effect in the transition, the SEINE model will generate video frames that match this scenario based on the user's description. Text-guided generation parses the text description entered by the user through a natural language processing (NLP) model to extract key information related to the transition (such as scene, style, color, emotion, etc.). This information will be used as input conditions for the diffusion model to generate video transition frames that match the user's description. The SEINE model not only generates simple visual transitions, but can also handle more complex scene transitions. For example, between two videos of completely different styles, SEINE generates transition frames and gradually adjusts the color, light, and scene elements (such as weather, time, object position, etc.) to make the scene smoothly transition from the first audio and video segment to the second audio and video segment.

[0085] Scene transition is achieved by controlling the parameters in the generation process. SEINE can adjust the generated transition frames according to the differences between the two audio and video clips. In addition to scene transition, the SEINE model can also handle visual style alignment between two audio and video clips. This is critical for video editing, especially when connecting two videos of different styles, such as transitioning from animation style to realistic style, or from warm tones to cold tones. Style transfer Through the style control mechanism in the diffusion model, SEINE is able to adjust the visual features of the two audio and video clips to maintain visual consistency during the transition process. After generating the transition frame, SEINE merges the generated transition video with the original two audio and video segments to generate a new complete audio and video. This new audio and video segment contains the previous and next audio and video segments and the transition audio and video in the middle, and is visually and temporally coherent.

[0086] S7, outputs the generated long audio and video into a media format desired by the user.

[0087] The system outputs the generated multimodal content into common media formats such as MP4, MP3, JPG, etc. through the output module, supporting users to adjust the format, length and style of the generated content according to their needs to achieve highly customized creative output.

[0088] Example 1: Automatically generate creative advertising videos

[0089] The user inputs the basic information of the advertising product and the promotional theme text, and the system generates product display images and animation clips through the text-to-image and text-to-video modules. Then, the system automatically generates background music and voice-over commentary consistent with the product style according to the scene requirements, and generates a complete advertising video through the intelligent editing module, which is output in high-definition MP4 format. The video can be used for social media promotion or advertising playback.

[0090] Example 2: Automatically generate music MV

[0091] The user provides an audio file of a song and a simple description of the music scene, and the system automatically generates a video that matches the mood of the song. Through multimodal alignment technology, the system synchronizes the video, lyrics and subtitles with the music, and generates camera switching and special effects with a sense of musical rhythm, and outputs it in the format of a music MV.

[0092] Example 3: Post-processing of movie special effects scenes

[0093] In the special effects scenes of film production, many scenes require complex post-processing. The user enters a preliminary description text or a low-resolution preview video of the special effects scene in the film. The system first automatically generates static background images related to the special effects scene through the Vincent Image module, and then uses the Vincent Video function to generate high-quality dynamic video clips, such as explosion effects, virtual scenes of science fiction scenes, etc. In post-production, the system will use intelligent editing technology to seamlessly splice the special effects scene with other shot videos. At the lens switching point, the system will automatically generate a smooth transition effect, so that the switching between special effects and real shots is a natural transition. In addition, the lens group stitching function will automatically splice the real shots and special effects shots together to ensure smooth videos and stunning visual effects.

[0094] Example 4: Intelligent processing of film and television drama editing

[0095] The user first uploads the original video material through the input module, including multiple footage and related script information. The system uses natural language processing technology to parse the script, extract key plot, scene and character information, and intelligently select the footage that best meets the narrative needs. Next, the system automatically splices the selected footage and designs appropriate transition effects to ensure the coherence and visual beauty of the editing.

[0096] Therefore, the present invention adopts the above-mentioned AIGC multimodal audio-visual content creation method and system, which integrates multiple generative artificial intelligence technologies such as text generation image, video generation, and music generation, and can generate multiple modal audio-visual content from a single input, greatly simplifying the creation process, reducing manual intervention, and realizing the automatic generation of audio-visual content;

[0097] Through the fusion and alignment technology of multimodal information, the semantic and temporal consistency and coherence of different modal contents are ensured, which solves the problem of multimodal information being difficult to integrate naturally in the existing technology, making the generated audio-visual content more fluent and coordinated;

[0098] It provides intelligent video editing functions, automatically completes video splicing, lens switching and transition effect generation, and improves the efficiency and quality of content creation; the system automatically adjusts lens transitions and special effects generation according to the video plot, making the video more natural and smooth, reducing the workload of users in post-editing; through the intelligent editing process, the system can complete high-quality content creation in a very short time;

[0099] It also has powerful flexibility and customization capabilities. Users can adjust the style, format and duration of generated content according to specific needs to meet the needs of different creation scenarios. It is suitable for multiple fields, improving the creation efficiency and quality of multimodal audio-visual content, lowering the creation threshold, and realizing efficient, low-cost and highly personalized audio-visual content creation.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for creating AIGC multimodal audiovisual content, characterized by: The following steps are involved: S1, the user inputs text description and materials, including images, video clips and music; S2, use GPT-4 to analyze and optimize the text description, and then convert the optimized and analyzed text description into several images through the stable diffusion XL model; S3, using a stable video diffusion model to expand the images obtained in S2 and the images contained in the material into several video sequences; S4, uses natural language processing technology to analyze the keywords and important information in the text description, and then inputs the keywords and important information into Suno, which parses the keywords and important information and then generates a music clip; S5, analyzing and synchronizing the text description, video sequence and music clips to make them consistent in content and time, and obtaining several audio and video; S6, using the short-to-long video diffusion model to connect several audio and video together to obtain a long audio and video; S7, outputting the generated long audio and video into a media format desired by the user; The process of S5 is to first perform multimodal feature alignment and then perform time alignment. The specific steps of S5 are: S51, input the text description into the pre-trained language model, extract the semantic features in the text description, and capture the information about the emotion, theme and style in the text; S52, using a residual convolutional neural network model to divide the video sequence generated in S3 and the video in the material into a number of video frames, and perform frame extraction and visual feature extraction; S53, using a 3D convolutional neural network model and a long short-term memory model to extract dynamic features of the video sequence and the video in the material generated in S3, and using the dynamic features as the timing information of the video; S54, segmenting the music in the material and the music clip generated in S4 into short clips synchronized with the video frames, and then using the Mel frequency spectrogram and Mel frequency cepstral coefficient extraction model to extract frequency, rhythm and pitch features of the music in the material and the music clip generated in S4; S55, uses a shared embedding space to align features of text descriptions, videos, and music; S56, using the audio-video converter model to time-align the features of the text description, video, and music to obtain several audio and video.

2. The AIGC multimodal audiovisual content creation method according to claim 1, characterized in that: In S1, the user must enter a text description in the information input, and can optionally enter the material.

3. The AIGC multimodal audiovisual content creation method according to claim 1, characterized in that: In S4, after Suno generates the music clip, it feeds back the music clip generated by Suno to the user, and the user modifies the music clip by changing the text description according to actual needs.

4. The AIGC multimodal audiovisual content creation method according to claim 1, characterized in that: In S55, a contrastive learning optimization model is used in a shared embedding space to process the features of text descriptions, videos, and music, so that similar features are closer and dissimilar features are farther apart.

5. The AIGC multimodal audiovisual content creation method according to claim 1, characterized in that: In S6, a natural language processing model is used to parse the text description, extract key information related to the transition in the text description, and input the key information into the short-to-long video diffusion model to generate transition frames that meet customer needs.

6. An AIGC multimodal audiovisual content creation system, characterized by: The method for creating AIGC multimodal audiovisual content according to any one of claims 1 to 5 comprises: Input module, used for users to input text descriptions and materials required for creation, including images, video clips and music; The AIGC generation module is used to convert the text description and materials input by the user into video sequences and music clips that meet the requirements; The multimodal fusion and alignment module is used to fuse and align the generated video sequences and music clips to make the semantics and timing between the video sequences and the music clips consistent, thus obtaining audio and video; Video intelligent editing module, used to splice audio and video clips, adjust the transition frames between several audio and video, and maintain visual coherence and connection between two audio and video; The output module is used to output the generated audio and video into the media format required by the user.

Images