AI intelligent short video generation method and system based on deep learning model

Abstract

The invention relates to the technical field of multimedia processing, and discloses an AI intelligent short video generation method and system based on a deep learning model, and the method comprises the steps: obtaining multi-modal information inputted by a user, extracting each modal feature through a Transform-XL model, an attention ResNet-50 model and a VGGish model, and generating a multi-modal demand feature matrix through a feature fusion gating unit; taking the multi-modal demand feature matrix as input, adopting a multi-modal fusion Transform model to output a video content planning vector, generating an initial frame sequence through a StyleGAN3 model, and correcting coherence through a bidirectional LSTM time sequence optimization module to obtain an audio planning feature vector; based on the video content planning vector and the audio planning feature vector, background music and voice commentary are matched through a PPO algorithm, and audio and video alignment is carried out; coding and synthesizing the video and the audio according to an output result of audio and video alignment, and outputting the video and the audio into a piece; according to the invention, the short video generation efficiency and the content quality are improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia processing technology, specifically to an AI-powered intelligent short video generation method and system based on a deep learning model. Background Technology

[0002] With the explosive growth in demand for short video content, traditional video production methods can no longer meet the needs for fast, low-cost, and high-quality content production. Current AI short video generation technologies have low multimodal data fusion, and most methods only achieve simple matching of text and images. They cannot deeply coordinate audio rhythm, visual style, and content semantics to generate personalized content. They rely on fixed templates or single style transfer models, making it difficult to adapt to the creative needs and scene characteristics of different users. Furthermore, their temporal logic is poor, making it difficult to meet the needs of real-time creation. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing an AI-powered intelligent short video generation method and system based on a deep learning model.

[0004] The first aspect of this invention provides an AI-powered intelligent short video generation method based on a deep learning model, the method comprising the following steps: The system acquires multimodal information from user input, extracts features for each modality using the Transformer-XL model, the attention ResNet-50 model, and the VGGish model, and generates a multimodal requirement feature matrix through a feature fusion gating unit. Using the multimodal demand feature matrix as input, the multimodal fusion Transformer model outputs the video content planning vector, the StyleGAN3 model generates the initial frame sequence, and the bidirectional LSTM temporal optimization module corrects the coherence to obtain the audio planning feature vector. Based on video content planning vectors and audio planning feature vectors, the PPO algorithm is used to match background music and voice narration, and perform audio-video alignment. Based on the output of audio-video alignment, the video and audio are encoded and synthesized, and then output as a finished product.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of acquiring multimodal information input by the user, extracting features of each modality using the Transformer-XL model, the attention ResNet-50 model, and the VGGish model respectively, and generating a multimodal requirement feature matrix through a feature fusion gating unit includes: The system acquires and preprocesses multimodal information from user input, uses the Transformer-XL model to process the preprocessed text description, and outputs a text semantic feature vector. The ResNet-50 model enhanced with attention mechanism is used to extract image features, and the visual feature vector is output after global average pooling. The VGGish model is used to process audio preference data. First, the audio is converted into a Mel spectrogram. Then, the spectral features are extracted layer by layer through multi-layer convolution and pooling operations to capture the emotional tone and rhythmic variation of the audio. After feature integration through a fully connected layer, the audio feature vector is output. Text semantic feature vectors, visual feature vectors, and audio feature vectors are input into the feature fusion gating unit. The GRU gating structure is used to first calculate the information entropy of each modality feature, and then the gating mechanism is used to dynamically weight and fuse each modality feature to finally generate a multimodal requirement feature matrix.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of acquiring and preprocessing the multimodal information input by the user, processing the preprocessed text description using a Transformer-XL model, and outputting a text semantic feature vector includes: The text in the multimodal information is segmented and stop words are removed to filter effective semantics. The images in the multimodal information are uniformly scaled to a preset resolution and color standardized. The audio files corresponding to the audio preferences are converted into mono audio with a fixed sampling rate. The Transformer-XL model first converts text vocabulary into initial vectors through a pre-defined short video domain-specific vocabulary embedding layer. Then, it uses a segmented loop mechanism to coherently capture the semantics of long texts and introduces a domain adaptation layer to enhance the semantic parsing of professional terms in the short video domain. Finally, it outputs text semantic feature vectors.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of using a multimodal demand feature matrix as input, employing a multimodal fusion Transformer model to output a video content planning vector, generating an initial frame sequence through a StyleGAN3 model, correcting coherence through a bidirectional LSTM temporal optimization module, and obtaining an audio planning feature vector includes: The multimodal requirement feature matrix is ​​input into the multimodal fusion Transformer model. The encoder calculates the association weights between text and image, image and audio, and text and audio features through a cross-modal attention mechanism, thereby achieving deep coupling of semantic information from different modalities. The decoder generates a video content planning vector that includes the number of shots, scene logic, and theme distribution based on the coupled features. The video content planning vector is input into the StyleGAN3 model to generate the initial video frame sequence. The initial frame sequence is then input into the bidirectional LSTM temporal optimization module. The temporal features from the first frame to the last frame are extracted by the forward LSTM, and the reverse temporal correlation is captured from the last frame to the first frame by the backward LSTM. The attention window mechanism is used to focus on the frame transition area between adjacent shots, calculate the differences in color and motion trajectory between frames and perform smooth correction, and adjust the frames in combination with the sharpness index. Finally, the coherence-corrected frame sequence and the corresponding audio planning feature vector are output.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of inputting the video content planning vector into the StyleGAN3 model to generate an initial video frame sequence includes: The StyleGAN3 model's content branch receives content information from the video content planning vector and generates a low-resolution frame structure containing the main body outline. The style branch receives style parameters from the video content planning vector and performs texture filling and style rendering on the low-resolution frame. The content branch and style branch interact with each other through the attention fusion module and then output the initial frame sequence.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of matching background music and voice narration using the PPO algorithm based on video content planning vectors and audio planning feature vectors, and performing audio-video alignment, includes: Using style information from the video content planning vector and emotion and rhythm features from the audio feature vector as input, a reward function is constructed. The reward function includes the matching degree between audio and video styles and the fit of users' historical feedback preferences. The PPO algorithm adjusts the feature matching weights through iterative training, selects the optimal background music from a pre-set audio material library, and determines the generation parameters for the voice narration. Based on the voice narration parameters output by the PPO algorithm, combined with the semantic information in the video content planning vector, a voice narration consistent with the theme of the video content is generated. The tone fluctuation of the narration is controlled according to the emotional features in the audio planning feature vector, and the narration speed is adjusted according to the rhythm features. Extract video temporal features including shot transition times and screen motion amplitude, and audio temporal features including beat points and pitch change points of background music and voice narration. Use dynamic time warping algorithm to perform alignment calculations on video and audio temporal features, and output audio-video alignment results.

[0010] Optionally, in the sixth implementation of the first aspect of the present invention, the extraction of video temporal features including shot switching times and image motion amplitude, and audio temporal features including beat points and pitch change points of background music and voice narration, and the use of a dynamic time warping algorithm to perform alignment calculations on the video temporal features and audio temporal features, and outputting audio-video alignment results, includes: The video temporal feature sequence and the audio temporal feature sequence are normalized respectively to construct a two-dimensional distance matrix, where the matrix elements are the feature distances at corresponding positions in the video temporal feature sequence and the audio temporal feature sequence; The optimal path from the starting point to the ending point of the matrix is ​​found through dynamic programming. Based on the optimal path, the correspondence between video temporal features and audio temporal features is established. The timestamps of the video frame sequence are associated and matched with the timestamps of the audio to generate an alignment table containing the correspondence of each feature point, which is output as the audio-video alignment result.

[0011] A second aspect of this invention provides an AI-powered intelligent short video generation system based on a deep learning model, the system comprising: The feature extraction module is used to acquire multimodal information from user input. It extracts features of each modality through the Transformer-XL model, the attention ResNet-50 model, and the VGGish model, and generates a multimodal requirement feature matrix through the feature fusion gating unit. The content planning module takes the multimodal demand feature matrix as input, uses the multimodal fusion Transformer model to output the video content planning vector, generates the initial frame sequence through the StyleGAN3 model, and corrects the coherence through the bidirectional LSTM temporal optimization module to obtain the audio planning feature vector. The audio-video alignment module is used to match background music and voice narration based on video content planning vectors and audio planning feature vectors, and to perform audio-video alignment using the PPO algorithm. The final output module is used to encode and synthesize video and audio based on the output results of audio-video alignment and output the final video.

[0012] A third aspect of the present invention provides an AI-powered intelligent short video generation device based on a deep learning model, the deep learning model-based AI intelligent short video generation device comprising a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the deep learning model-based AI intelligent short video generation device to perform the various steps of the deep learning model-based AI intelligent short video generation method as described in any of the preceding claims.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the AI-powered intelligent short video generation method based on a deep learning model as described in any of the preceding claims.

[0014] The technical solution provided by this invention acquires multimodal information input by the user, extracts features of each modality using a Transformer-XL model, an Attention ResNet-50 model, and a VGGish model respectively, and generates a multimodal demand feature matrix through a feature fusion gating unit; using the multimodal demand feature matrix as input, a multimodal fusion Transformer model outputs a video content planning vector, an initial frame sequence is generated using a StyleGAN3 model, and the coherence is corrected by a bidirectional LSTM temporal optimization module to obtain an audio planning feature vector; based on the video content planning vector and the audio planning feature vector, the PPO algorithm is used to match background music and voice narration, and audio-video alignment is performed; according to the output result of audio-video alignment, video and audio are encoded and synthesized, and the final product is output; this invention utilizes Transformer-XL to obtain a multimodal demand feature matrix. The Ormer-XL model, the Attention ResNet-50 model, and the VGGish model selectively extract features from each modality. After dynamic fusion via gating units, the semantic and style matching degree between the generated video and user needs is significantly improved, avoiding the problem of fragmented multimodal information. The content generation quality is excellent. The StyleGAN3 model ensures the image quality and details of the initial frame sequence, and the bidirectional LSTM temporal optimization module effectively corrects the problem of abrupt shot transitions, improving the coherence of video narrative and visual smoothness. The audio and video synchronization is accurate and efficient. The PPO algorithm can quickly match suitable audio and video resources based on feature vectors, and combined with the alignment mechanism, it achieves audio-visual rhythm coordination with small synchronization errors. The overall process is efficient and practical, forming a closed loop from feature extraction to final output, reducing manual intervention, improving the efficiency and quality of short video generation, and adapting to the needs of short video creation in multiple scenarios. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0016] Figure 1 A flowchart illustrating the AI-powered intelligent short video generation method based on a deep learning model provided in this embodiment of the invention; Figure 2 A schematic diagram of the structure of an AI intelligent short video generation system based on a deep learning model provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an AI-powered intelligent short video generation device based on a deep learning model, provided in an embodiment of the present invention. Detailed Implementation

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The flowchart of the AI ​​intelligent short video generation method based on a deep learning model provided in this embodiment of the invention includes the following steps: Step 101: Obtain multimodal information input by the user, extract features of each modality through Transformer-XL model, attention ResNet-50 model and VGGish model respectively, and generate multimodal requirement feature matrix through feature fusion gating unit; In this embodiment, multimodal information input by the user is acquired and preprocessed. The Transformer-XL model is used to process the preprocessed text description and output the text semantic feature vector. An attention-enhanced ResNet-50 model is used to extract image features, which are then output as a visual feature vector after global average pooling. The VGGish model is used to process audio preference data. The audio is first converted into a Mel spectrogram, and then spectral features are extracted layer by layer through multi-layer convolution and pooling operations to capture the emotional tone and rhythmic variation of the audio. After feature integration through a fully connected layer, an audio feature vector is output. The text semantic feature vector, visual feature vector, and audio feature vector are input into a feature fusion gating unit. A GRU gating structure is used to first calculate the information entropy of each modality feature, and then the gating mechanism is used to dynamically weight and fuse the features of each modality to finally generate a multimodal demand feature matrix.

[0019] In this embodiment, the text in the multimodal information is segmented and stop word removed to filter effective semantics. The images in the multimodal information are uniformly scaled to a preset resolution and color standardized. The audio files corresponding to the audio preferences are converted into mono audio with a fixed sampling rate. The Transformer-XL model first converts the text vocabulary into initial vectors through a preset short video domain-specific vocabulary embedding layer. Then, it uses a segmented loop mechanism to coherently capture the semantics of long texts and introduces a domain adaptation layer to enhance the semantic parsing of professional terms in the short video domain. Finally, it outputs the text semantic feature vector.

[0020] In this embodiment, when extracting image features using the ResNet-50 model enhanced by the attention mechanism, the user-input reference image is first standardized to a uniform size and pixel values ​​are normalized. Then, the image enters the convolutional layer and residual block structure of ResNet-50. Through multiple sets of convolutions, batch normalization, and activation operations, basic visual features from edges, textures to semantics are extracted layer by layer. An attention module is embedded in the key residual stage of the model. This module first calculates the importance weights of each feature channel through the channel attention branch, strengthening the response of effective feature channels such as color style and subject outline. Then, the spatial attention branch focuses on the feature mapping of the main area and detailed texture in the image, suppressing interference from irrelevant information such as background. Finally, a global average pooling operation is performed on the deep feature map enhanced by attention, transforming the two-dimensional feature map into a one-dimensional vector with a fixed dimension, that is, outputting a visual feature vector that can accurately represent the core visual information of the image.

[0021] Step 102: Using the multimodal demand feature matrix as input, the multimodal fusion Transformer model is used to output the video content planning vector. The initial frame sequence is generated through the StyleGAN3 model, and the coherence is corrected by the bidirectional LSTM temporal optimization module to obtain the audio planning feature vector. In this embodiment, the multimodal requirement feature matrix is ​​input into the multimodal fusion Transformer model. The encoder calculates the association weights between text and image, image and audio, and text and audio features through a cross-modal attention mechanism, and performs deep coupling of semantic information of different modalities. The decoder generates a video content planning vector containing the number of shots, scene logic, and theme distribution based on the coupled features. The video content planning vector is input into the StyleGAN3 model to generate an initial video frame sequence. The initial frame sequence is input into the bidirectional LSTM temporal optimization module. First, the temporal features of the forward LSTM from the first frame to the last frame are extracted. At the same time, the backward LSTM captures the reverse temporal correlation from the last frame to the first frame. The attention window mechanism is used to focus on the frame transition region of adjacent shots, calculate the differences in color and motion trajectory between frames and perform smoothing correction. The frames are adjusted in combination with the sharpness index. Finally, the coherence-corrected frame sequence and the corresponding audio planning feature vector are output.

[0022] In this embodiment, the content branch of the StyleGAN3 model receives the content information in the video content planning vector and generates a low-resolution frame structure containing the main body outline. The style branch receives the style parameters in the video content planning vector and performs texture filling and style rendering on the low-resolution frame. After the content branch and style branch interact with each other through the attention fusion module, the initial frame sequence is output.

[0023] In this embodiment, the content branch of the StyleGAN3 model receives the video content planning vector output by the multimodal fusion Transformer. It first parses the core content information contained in the vector, such as the main object, scene layout, and motion trajectory, through a fully connected layer, and maps this information into a content feature map. Then, the content branch generates a low-resolution frame structure with main outline and spatial relationship through a progressively increasing convolutional structure. The style branch takes the style parameters in the video content planning vector as input. It first parses the information such as color tone, texture, and artistic style contained in the parameters through the style feature extraction submodule and converts them into style feature vectors. For the low-resolution frame structure output by the content branch, the style branch performs texture filling on the low-resolution frame at different feature levels through an adaptive style injection unit. At the same time, it adjusts the color distribution, edge strokes, and other details of the frame according to the style vector to achieve the initial combination of style and content. During the process, the main outline and spatial relationship determined by the content branch are strictly preserved. The attention fusion module is responsible for achieving deep collaboration between content features and style features. It first applies channel attention weighting to the feature maps output from both branches, highlighting the feature channels corresponding to the main content region and key style details. Then, it constructs a cross-branch attention matrix, calculates the spatial correlation weights between content features and style features, and accurately matches style features to the corresponding regions of content features, completing feature interaction. Finally, a feature fusion convolutional layer integrates the interacted features, outputting an initial frame sequence that combines clear content structure with a unified style.

[0024] Step 103: Based on the video content planning vector and audio planning feature vector, the PPO algorithm is used to match the background music and voice narration, and audio-video alignment is performed; In this embodiment, a reward function is constructed using style information from the video content planning vector and emotion and rhythm features from the audio feature vector as input. The reward function includes the matching degree between audio and video styles and the fit of user historical feedback preferences. The PPO algorithm adjusts the feature matching weights through iterative training, selects the optimal background music from a preset audio material library, and determines the generation parameters of the voice narration. Based on the voice narration parameters output by the PPO algorithm, combined with the semantic information in the video content planning vector, a voice narration consistent with the theme of the video content is generated. The tone fluctuation of the narration is controlled according to the emotion features in the audio planning feature vector, and the narration speed is adjusted according to the rhythm features. Video temporal features, including shot switching moments and the amplitude of screen movement, and audio temporal features, including the beat points and pitch change points of the background music and voice narration, are extracted. The dynamic time warping algorithm is used to perform alignment calculations on the video temporal features and audio temporal features, and the audio-video alignment result is output.

[0025] In this embodiment, before starting iterative training, the PPO algorithm first sets initial matching weights for feature dimensions such as video style, audio emotion, and audio rhythm. In each iteration, the algorithm selects candidate background music from the audio material library based on the current weights, constructs multiple matching schemes by combining them with the voice narration generation parameters, calculates the reward value of each scheme through a reward function, compares the reward differences of different schemes, and uses a pruning objective function to adjust the feature matching weights, suppressing drastic fluctuations in weights and retaining high-quality adjustment directions. After multiple iterations until the reward value tends to stabilize, the background music with the highest reward value is selected, and the parameters of the corresponding iteration round are determined as the final generation parameters of the voice narration, including the speech rate matching the video rhythm and the tone matching the emotional characteristics.

[0026] In this embodiment, the video temporal feature sequence and the audio temporal feature sequence are normalized respectively to construct a two-dimensional distance matrix, where the matrix elements are the feature distances between corresponding positions in the video temporal feature sequence and the audio temporal feature sequence. The optimal path from the starting point to the ending point of the matrix is ​​found through dynamic programming. Based on the optimal path, the correspondence between the video temporal features and the audio temporal features is established. The timestamps of the video frame sequence and the timestamps of the audio are associated and matched to generate an alignment table containing the correspondence of each feature point, which is output as the audio-video alignment result.

[0027] Step 104: Based on the output results of audio and video alignment, encode and synthesize video and audio, and output the finished product.

[0028] Please see Figure 2 A schematic diagram of the structure of an AI intelligent short video generation system based on a deep learning model provided in this embodiment of the invention. The system includes: The feature extraction module is used to acquire multimodal information from user input. It extracts features of each modality through the Transformer-XL model, the attention ResNet-50 model, and the VGGish model, and generates a multimodal requirement feature matrix through the feature fusion gating unit. The content planning module takes the multimodal demand feature matrix as input, uses the multimodal fusion Transformer model to output the video content planning vector, generates the initial frame sequence through the StyleGAN3 model, and corrects the coherence through the bidirectional LSTM temporal optimization module to obtain the audio planning feature vector. The audio-video alignment module is used to match background music and voice narration based on video content planning vectors and audio planning feature vectors, and to perform audio-video alignment using the PPO algorithm. The final output module is used to encode and synthesize video and audio based on the output results of audio-video alignment and output the final video.

[0029] Figure 3 This is a schematic diagram of the structure of an AI-powered intelligent short video generation device based on a deep learning model, according to an embodiment of the present invention. This AI-powered intelligent short video generation device 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the AI-powered intelligent short video generation device 300 based on the deep learning model. Furthermore, the processor 310 may be configured to communicate with the storage media 330, executing the series of instruction operations in the storage media 330 on the AI-powered intelligent short video generation device 300 based on the deep learning model to implement the method provided in the above embodiment.

[0030] The AI-powered intelligent short video generation device 300 based on a deep learning model may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3The structure of the AI-powered short video generation device based on a deep learning model shown does not constitute a limitation on the computer device provided by this invention. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0031] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the various steps of the AI ​​intelligent short video generation method based on a deep learning model provided in the above embodiments.

[0032] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the above-described equipment, apparatus, or unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0033] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0034] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended technical solutions and their equivalents.

Claims

1. An AI-powered intelligent short video generation method based on a deep learning model, characterized in that, The method includes the following steps: The system acquires multimodal information from user input, extracts features for each modality using the Transformer-XL model, the attention ResNet-50 model, and the VGGish model, and generates a multimodal requirement feature matrix through a feature fusion gating unit. Using the multimodal demand feature matrix as input, the multimodal fusion Transformer model outputs the video content planning vector, the StyleGAN3 model generates the initial frame sequence, and the bidirectional LSTM temporal optimization module corrects the coherence to obtain the audio planning feature vector. Based on video content planning vectors and audio planning feature vectors, the PPO algorithm is used to match background music and voice narration, and perform audio-video alignment. Based on the output of audio-video alignment, the video and audio are encoded and synthesized, and then output as a finished product.

2. The AI-powered intelligent short video generation method based on a deep learning model as described in claim 1, characterized in that, The process of acquiring multimodal information from user input involves extracting features for each modality using the Transformer-XL model, the Attention ResNet-50 model, and the VGGish model, respectively. A multimodal requirement feature matrix is ​​then generated via a feature fusion gating unit, including: The system acquires and preprocesses multimodal information from user input, uses the Transformer-XL model to process the preprocessed text description, and outputs a text semantic feature vector. The ResNet-50 model enhanced with attention mechanism is used to extract image features, and the visual feature vector is output after global average pooling. The VGGish model is used to process audio preference data. First, the audio is converted into a Mel spectrogram. Then, the spectral features are extracted layer by layer through multi-layer convolution and pooling operations to capture the emotional tone and rhythmic variation of the audio. After feature integration through a fully connected layer, the audio feature vector is output. Text semantic feature vectors, visual feature vectors, and audio feature vectors are input into the feature fusion gating unit. The GRU gating structure is used to first calculate the information entropy of each modality feature, and then the gating mechanism is used to dynamically weight and fuse each modality feature to finally generate a multimodal requirement feature matrix.

3. The AI ​​intelligent short video generation method based on a deep learning model as described in claim 2, characterized in that, The process of acquiring and preprocessing multimodal information from user input involves using the Transformer-XL model to process the preprocessed text description and outputting a text semantic feature vector, including: The text in the multimodal information is segmented and stop words are removed to filter effective semantics. The images in the multimodal information are uniformly scaled to a preset resolution and color standardized. The audio files corresponding to the audio preferences are converted into mono audio with a fixed sampling rate. The Transformer-XL model first converts text vocabulary into initial vectors through a pre-defined short video domain-specific vocabulary embedding layer. Then, it uses a segmented loop mechanism to coherently capture the semantics of long texts and introduces a domain adaptation layer to enhance the semantic parsing of professional terms in the short video domain. Finally, it outputs text semantic feature vectors.

4. The AI ​​intelligent short video generation method based on a deep learning model as described in claim 1, characterized in that, The process involves taking a multimodal demand feature matrix as input, using a multimodal fusion Transformer model to output a video content planning vector, generating an initial frame sequence through a StyleGAN3 model, and correcting the coherence through a bidirectional LSTM temporal optimization module to obtain an audio planning feature vector, including: The multimodal requirement feature matrix is ​​input into the multimodal fusion Transformer model. The encoder calculates the association weights between text and image, image and audio, and text and audio features through a cross-modal attention mechanism, thereby achieving deep coupling of semantic information from different modalities. The decoder generates a video content planning vector that includes the number of shots, scene logic, and theme distribution based on the coupled features. The video content planning vector is input into the StyleGAN3 model to generate the initial video frame sequence. The initial frame sequence is then input into the bidirectional LSTM temporal optimization module. The temporal features from the first frame to the last frame are extracted by the forward LSTM, and the reverse temporal correlation is captured from the last frame to the first frame by the backward LSTM. The attention window mechanism is used to focus on the frame transition area between adjacent shots, calculate the differences in color and motion trajectory between frames and perform smooth correction, and adjust the frames in combination with the sharpness index. Finally, the coherence-corrected frame sequence and the corresponding audio planning feature vector are output.

5. The AI ​​intelligent short video generation method based on a deep learning model as described in claim 4, characterized in that, The step of inputting the video content planning vector into the StyleGAN3 model to generate the initial video frame sequence includes: The StyleGAN3 model's content branch receives content information from the video content planning vector and generates a low-resolution frame structure containing the main body outline. The style branch receives style parameters from the video content planning vector and performs texture filling and style rendering on the low-resolution frame. The content branch and style branch interact with each other through the attention fusion module and then output the initial frame sequence.

6. The AI ​​intelligent short video generation method based on a deep learning model as described in claim 1, characterized in that, The process of matching background music and voice narration using the PPO algorithm based on video content planning vectors and audio planning feature vectors, and performing audio-video alignment, includes: Using style information from the video content planning vector and emotion and rhythm features from the audio feature vector as input, a reward function is constructed. The reward function includes the matching degree between audio and video styles and the fit of users' historical feedback preferences. The PPO algorithm adjusts the feature matching weights through iterative training, selects the optimal background music from a pre-set audio material library, and determines the generation parameters for the voice narration. Based on the voice narration parameters output by the PPO algorithm, combined with the semantic information in the video content planning vector, a voice narration consistent with the theme of the video content is generated. The tone fluctuation of the narration is controlled according to the emotional features in the audio planning feature vector, and the narration speed is adjusted according to the rhythm features. Extract video temporal features including shot transition times and screen motion amplitude, and audio temporal features including beat points and pitch change points of background music and voice narration. Use dynamic time warping algorithm to perform alignment calculations on video and audio temporal features, and output audio-video alignment results.

7. The AI ​​intelligent short video generation method based on a deep learning model as described in claim 6, characterized in that, The extraction includes video temporal features such as shot transition times and image motion amplitude, and audio temporal features such as beat points and pitch change points of background music and voice narration. A dynamic time warping algorithm is used to align the video and audio temporal features, outputting an audio-video alignment result, including: The video temporal feature sequence and the audio temporal feature sequence are normalized respectively to construct a two-dimensional distance matrix, where the matrix elements are the feature distances at corresponding positions in the video temporal feature sequence and the audio temporal feature sequence; The optimal path from the starting point to the ending point of the matrix is ​​found through dynamic programming. Based on the optimal path, the correspondence between video temporal features and audio temporal features is established. The timestamps of the video frame sequence are associated and matched with the timestamps of the audio to generate an alignment table containing the correspondence of each feature point, which is output as the audio-video alignment result.

8. An AI-powered intelligent short video generation system based on a deep learning model, characterized in that, The system includes: The feature extraction module is used to acquire multimodal information from user input. It extracts features of each modality through the Transformer-XL model, the attention ResNet-50 model, and the VGGish model, and generates a multimodal requirement feature matrix through the feature fusion gating unit. The content planning module takes the multimodal demand feature matrix as input, uses the multimodal fusion Transformer model to output the video content planning vector, generates the initial frame sequence through the StyleGAN3 model, and corrects the coherence through the bidirectional LSTM temporal optimization module to obtain the audio planning feature vector. The audio-video alignment module is used to match background music and voice narration based on video content planning vectors and audio planning feature vectors, and to perform audio-video alignment using the PPO algorithm. The final output module is used to encode and synthesize video and audio based on the output results of audio-video alignment and output the final video.

9. An AI-powered intelligent short video generation device based on a deep learning model, characterized in that, The AI-powered short video generation device based on a deep learning model includes a memory and at least one processor. The memory stores instructions. The at least one processor invokes the instructions in the memory to cause the AI-powered short video generation device based on a deep learning model to perform each step of the AI-powered short video generation method based on a deep learning model as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the AI ​​intelligent short video generation method based on a deep learning model as described in any one of claims 1-7.

Images