Intelligent MV generation method, system and device based on AIGC and medium

Abstract

The invention relates to an intelligent MV generation method, system and device based on AIGC and a medium, and belongs to the technical field of artificial intelligence content generation. The method comprises the following steps: receiving an original music file input by a user and a user instruction; performing audio preprocessing on the original music file to generate a standardized audio signal, and mapping a user instruction into a semantic vector; rhythm features and emotion features are extracted, music feature vectors are generated and subjected to feature alignment with semantic vectors, and an initial key frame sequence is generated based on a diffusion model; performing particle effect density dynamic adjustment and shot switching point generation on the initial key frame sequence, outputting an enhanced key frame sequence, performing optical flow frame insertion and long time sequence prediction, and generating an optimized video frame sequence; and performing block parallel rendering on the optimized video frame sequence according to the target platform parameters to generate an MV file and metadata. According to the invention, deep fusion of audio and visual contents can be realized, and high-quality video contents can be generated according to individual demands of users.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence content generation, and in particular to an AIGC-based intelligent MV generation method, system, device and medium. Background Art

[0002] With the development of artificial intelligence technology, the artistic creation method based on AIGC (artificial intelligence generated content) has become an important breakthrough in the field of creation in recent years. Traditional content creation, especially the production of music videos (MVs), usually requires a lot of manual intervention, including audio editing, video editing, special effects design, and visual and emotional matching. These processes are not only cumbersome, but also require professional skills and rich experience. With the rise of digitalization, networking and platformization, the needs of creators have become more personalized and efficient, which has promoted the application of AIGC technology in video production.

[0003] At present, the automated creation of audio and video content is usually limited to some simple template processing, and it is impossible to generate personalized, high-quality content according to the specific needs and emotional instructions of users. In addition, this method often ignores the deep integration between audio and visual content, and is mostly based on fixed visual effects and backgrounds. It is difficult to truly understand and adjust the visual performance according to the emotional and rhythmic changes of the audio, and lacks the ability for dynamic interaction between audio and picture. Summary of the invention

[0004] In order to achieve a deep integration of audio and visual content and generate high-quality video content according to the personalized needs of users, the present application provides an AIGC-based intelligent MV generation method, system, device and medium.

[0005] In a first aspect, the present application provides an AIGC-based intelligent MV generation method, which adopts the following technical solution: An AIGC-based intelligent MV generation method, the generation method comprising: Receiving an original music file and a user instruction input by a user; wherein the user instruction includes text keywords, reference images and style parameters; Performing audio preprocessing on the original music file to generate a standardized audio signal, and mapping the user instruction into a semantic vector through a cross-modal coding model; Extracting rhythm features and emotional features from the standardized audio signal to generate a music feature vector; Aligning the music feature vector with the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; According to the spectrum energy characteristics in the music feature vector, dynamically adjust the particle special effect density and generate the shot switching points for the initial key frame sequence, and output an enhanced key frame sequence; Performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence; The optimized video frame sequence is rendered in blocks and in parallel according to the target platform parameters to generate MV files and metadata.

[0006] By adopting the above technical solutions, audio and visual information are deeply integrated, dynamic scenes that match emotional colors and visual styles are accurately generated according to user instructions, and lens switching is optimized through reinforcement learning to ensure high synchronization and interaction between audio and visual content. At the same time, through adaptive rendering strategies, it adapts to the needs of different platforms to ensure that high-quality MVs can be efficiently generated, thereby improving video creation efficiency and content quality.

[0007] Optionally, the step of extracting rhythm features and emotional features from the standardized audio signal to generate a music feature vector includes: Performing frame division and windowing processing on the standardized audio signal to generate a short-time signal frame sequence and performing time-frequency transformation to obtain a complex time-frequency matrix; Calculating a spectral flux sequence based on the complex time-frequency matrix, and generating a beat timestamp sequence by a dynamic programming beat tracking algorithm; Based on the Mel-scale filter bank, the spectral envelope of the complex time-frequency matrix is ​​extracted to generate an MFCC feature matrix, and a low-frequency energy proportion sequence is calculated; The MFCC feature matrix is ​​concatenated with the low-frequency energy proportion sequence to construct an enhanced feature matrix; Input the enhanced feature matrix into the Transformer encoder to obtain the sentiment label and time series feature matrix; The beat timestamp sequence, the emotion label and the time series feature matrix are integrated to generate a music feature vector including rhythm features and emotion features.

[0008] By adopting the above technical solution, after accurate processing of standardized audio signals, features are extracted from time-frequency features, rhythm, emotion and other aspects, and finally a music feature vector containing rich information is generated by fusion. This technical solution can accurately capture the rhythm and emotional changes in audio signals, providing strong support for applications such as music analysis, emotion recognition, and music recommendation. By combining multi-dimensional information such as spectrum, beat, and emotion tags, the accuracy of audio features and the breadth of applications are improved.

[0009] Optionally, aligning the music feature vector with the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; Mapping the music feature vector and the semantic vector to the same feature space, injecting position coding into the mapped music feature vector, and generating an aligned music feature vector; Taking the aligned music feature vector as the query vector, taking the semantic vector as the key vector and the value vector, and calculating the association weight between the query vector and the key vector based on the cross-modal attention mechanism, outputting the context-aware feature; Dynamically assigning fusion weights of the context-aware features and the semantic vector through a gating function and performing weighted summation to generate a joint feature vector; Obtaining user style parameters according to the user instruction; concatenating the joint feature vector and the user style parameter into a conditional latent vector and inputting the vector into a diffusion model; iterating the conditioned latent vector through a reverse denoising process to generate a latent space image representation; The latent space image representation is decoded to output an initial key frame sequence.

[0010] By adopting the above technical solutions, through the process of cross-modal feature alignment, dynamic weighted fusion and conditional generation, the deep fusion and interaction of audio and visual information is achieved, ensuring the high fit between the rhythm and emotional changes of the audio and the visual content, and adjusting the image generation process through the style parameters of the user's instructions. Finally, the diffusion model is used to generate a preliminary key frame sequence that meets the user's needs, laying the foundation for subsequent dynamic video generation.

[0011] Optionally, the step of dynamically adjusting the particle special effect density and generating the shot switching points for the initial key frame sequence according to the spectrum energy features in the music feature vector, and outputting the enhanced key frame sequence comprises: Extracting spectrum energy features according to the music feature vector; Based on the spectrum energy characteristics, the initial key frame sequence is subjected to frequency band separation and time alignment to generate frequency band energy curves and aligned key frame timestamps; According to the frequency-band energy curve, the density of particle special effects is dynamically adjusted through a piecewise linear mapping function to generate a key frame sequence with dynamic particle special effects; Optimizing the initial shot switching candidate points based on a reinforcement learning strategy network to generate a shot switching point sequence aligned with the sub-band energy curve; The key frame sequence with dynamic particle special effects is integrated with the lens switching point sequence, a dynamic transition effect is inserted, special effects-lens linkage parameters are superimposed, and an enhanced key frame sequence is output.

[0012] By adopting the above technical solutions, special effects mapping based on spectral energy and reinforcement learning to optimize the lens switching strategy, the perfect combination of video fluency and emotional expression is ensured, and the high synchronization of video content with audio rhythm and emotion is achieved. This technical solution can be widely used in scenarios such as real-time concert visual effects generation, game cutscene animation production, and short video AI soundtrack editing.

[0013] Optionally, the step of optimizing the initial shot switching candidate points based on the reinforcement learning strategy network to generate a shot switching point sequence aligned with the sub-band energy curve includes: Determine an initial shot switching candidate point according to the beat timestamp sequence and the frequency band energy curve; According to the sub-frequency band energy curve and the initial lens switching candidate point, a state vector set including real-time sub-frequency band energy flow data, historical switching intervals and candidate point offsets is constructed; Designing a multi-objective reward function based on the state vector set, and calculating the instantaneous reward values ​​of rhythm alignment error, switching fluency penalty, and emotional consistency reward; Inputting the state vector set and the instant reward value into the policy network, generating a probability distribution of shot switching actions through PPO algorithm training, and screening the initial shot switching candidate points to generate an optimized shot switching point sequence; According to the optimized lens switching point sequence and the real-time frequency-band energy flow data, the dynamic action space is expanded and a transition frame or a hard-cut action is inserted, and the final lens switching point sequence and dynamic transition parameters are output.

[0014] By adopting the above technical solutions, the entire shot switching optimization process uses the reinforcement learning network to accurately optimize the shot switching point sequence, and enhances the expressiveness of audio and video synchronization by dynamically adjusting the transition effect, creating a dynamic visual effect that is more in line with the audio characteristics, and achieving a high balance in rhythm alignment, visual fluency and emotional consistency. The optimized shot switching not only conforms to the rhythm and emotional changes of the audio, but also provides natural and expressive visual effects, greatly improving the quality and immersion of the video content.

[0015] Optionally, the step of performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence includes: Shot grouping is performed on the enhanced key frame sequence according to the beat timestamp sequence to obtain a shot group sequence; Extracting multi-scale feature maps and particle special effect parameters of each shot group in the shot group sequence to obtain a mixed feature tensor of each shot group; Based on the mixed feature tensor of adjacent lens groups, interpolation is performed through the RAFT optical flow model, and the interpolation result is optimized using the residual correction network to obtain the optical flow interpolation result; Perform long-term time series prediction according to the interpolated shot group sequence and the particle special effect parameters, and output a long-term time series prediction result; the long-term time series prediction result includes a character position heat map and a scene logic label; According to the difference in motion vectors between the optical flow interpolation result and the long-term prediction result, weighted fusion is performed to generate an optimized video frame sequence, and the particle special effect parameters are adjusted according to the scene logic label.

[0016] By adopting the above technical solution, after weighted fusion and dynamic particle effects adjustment, the final optimized video frame sequence has a visual effect of smooth transition, accurate motion prediction and consistent emotions. The density adjustment of particle effects further enhances the expressiveness of the video, making the entire video more natural and smooth, and highly consistent with the audio rhythm and emotion.

[0017] Optionally, after the step of weighted fusion to generate an optimized video frame sequence according to the motion vector difference between the optical flow interpolation result and the long time series prediction result, and adjusting the particle special effect parameters according to the scene logic label, the method further includes: Extracting a shot switching point sequence according to the optimized video frame sequence; Aligning the shot switching point sequence with the beat timestamp sequence according to the frame rate to obtain a set of audio and video timestamp pairs; Obtaining an optical flow motion vector field according to the optical flow interpolation result; Calculating the shot switching deviation and the rhythm consistency index according to the audio and video timestamp pair set and the optical flow motion vector field, and generating a synchronization rate score; Obtaining user style parameters according to the user instruction; Based on the comparison of the adjusted particle special effect parameters with the user style parameters, the style similarity and the particle parameter compliance index are calculated to generate a style consistency score; The synchronization rate score and the style consistency score are combined to generate a quality assessment result.

[0018] By adopting the above technical solution, automated evaluation of video quality is achieved, which can provide accurate feedback for video production, thereby helping the production team to quickly identify and improve problems in various aspects of video creation, thereby improving the visual effects of the video content, audio and video synchronization, and satisfaction of users' personalized needs.

[0019] In the second aspect, the present application provides an AIGC-based intelligent MV generation system, which adopts the following technical solutions: An AIGC-based intelligent MV generation system, the generation system comprising: A receiving module, used to receive the original music file and user instructions input by the user; wherein the user instructions include text keywords, reference images and style parameters; A processing module, configured to perform audio preprocessing on the original music file to generate a standardized audio signal, and map the user instruction into a semantic vector through a cross-modal coding model; A feature extraction module, used to extract rhythm features and emotional features from the standardized audio signal to generate a music feature vector; A frame sequence generation module, used for aligning the music feature vector with the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; A frame sequence enhancement module, used to dynamically adjust the particle special effect density and generate lens switching points for the initial key frame sequence according to the spectrum energy characteristics in the music feature vector, and output an enhanced key frame sequence; A frame sequence optimization module, used for performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence; The rendering module is used to perform block-by-block parallel rendering on the optimized video frame sequence according to the target platform parameters to generate MV files and metadata.

[0020] In a third aspect, the present application provides a computer device, which adopts the following technical solution: A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods in the first aspect.

[0022] In summary, the present application includes at least one of the following beneficial technical effects: through precise alignment of audio and video timing and multi-dimensional dynamic fusion optimization, high-quality evaluation and adaptive correction of generated videos are achieved; based on the frame rate alignment mechanism of music beat timestamps and lens switching points, combined with the coupling analysis of optical flow motion vector fields and rhythm frequencies, the synchronization rate score of generated videos is quantified, and at the same time, through multi-scale style similarity calculation and particle compliance verification driven by user style parameters, it is ensured that special effects density, visual style and preset rules are strictly matched; finally, through weighted fusion of motion differences and scene logic feedback adjustment, a smooth and coherent optimized video frame sequence is output, which significantly improves the automation level of film and television content generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a first flow chart of an AIGC-based intelligent MV generation method according to one of the embodiments of the present application.

[0024] Figure 2 This is a second flow chart of an AIGC-based intelligent MV generation method according to one of the embodiments of the present application.

[0025] Figure 3 This is a third flow chart of an AIGC-based intelligent MV generation method in one of the embodiments of the present application.

[0026] Figure 4 This is a fourth flow chart of an AIGC-based intelligent MV generation method according to one of the embodiments of the present application.

[0027] Figure 5 This is a fifth flow chart of an AIGC-based intelligent MV generation method according to one of the embodiments of the present application.

[0028] Figure 6 This is the sixth flow chart of an AIGC-based intelligent MV generation method in one of the embodiments of the present application.

[0029] Figure 7 This is the seventh flow chart of an AIGC-based intelligent MV generation method in one of the embodiments of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-7 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] The embodiment of the present application discloses an intelligent MV generation method based on AIGC.

[0032] Reference Figure 1 , an intelligent MV generation method based on AIGC, the generation method comprising: Step S101, receiving an original music file and a user instruction input by a user; wherein the user instruction includes text keywords, reference images and style parameters; Specifically, the original music file (audio stream) and user instructions (text keywords / reference images / style parameters) can be received through the API interface or UI interaction layer, a unified input data format can be constructed, and the user instructions can be structured and classified to provide an input basis for subsequent cross-modal alignment.

[0033] For example, a user uploads an MP3 file and enters keywords such as "science fiction, futuristic", etc. The system converts the music file into PCM format, and the text instructions are stored in the cache queue after word segmentation through natural language processing (NLP).

[0034] Step S102, performing audio preprocessing on the original music file to generate a standardized audio signal, and mapping the user instruction into a semantic vector through a cross-modal coding model; Among them, the original music files include multiple formats (MP3, WAV, MIDI, etc.), and the system first performs audio preprocessing. The goal of audio preprocessing is to standardize the audio data so that it meets the requirements of subsequent processing.

[0035] In some embodiments, audio preprocessing mainly includes three key operations: unified sampling rate (unifying the audio sampling rate to 44.1kHz, which is crucial for subsequent feature extraction), noise reduction (using spectral subtraction or other noise reduction algorithms to process the audio signal to reduce the interference of background noise) and frame processing (dividing the audio signal into small segments, usually about 20ms, for subsequent audio analysis and feature extraction).

[0036] In addition, the user instruction part includes text keywords, reference images and style parameters. The system can map text keywords to a 512-dimensional vector space by using the CLIP model. The image information is extracted through ResNet-50 and then weightedly fused with the text vector. By converting text and image information into semantic vectors, these semantic vectors provide guidance for the visual style, emotional color, etc. in the subsequent generation process.

[0037] Step S103, extracting rhythm features and emotional features from the standardized audio signal to generate a music feature vector; In one of the embodiments of the present application, rhythm feature extraction can use dynamic programming and beat tracking algorithms to detect the rhythm change points of the audio (i.e., the position of each beat). The system generates a timestamp sequence by analyzing the rhythm features of the audio signal. These timestamps are the key reference for subsequent camera switching. Emotion feature extraction can extract the spectrum envelope through MFCC, combine the Transformer model (12-layer encoder) to analyze the pitch and chord progression, output the emotion label (such as "excited" corresponds to a high-frequency energy ratio of >60%), and form a music feature vector together with the rhythm timestamp.

[0038] It can be understood that by extracting rhythmic and emotional features, the system can accurately capture the time and emotional dimensions in the audio, ensuring that the generated MV can accurately match the rhythmic changes and emotional expressions of the audio.

[0039] Step S104, aligning the music feature vector with the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; Among them, the audio feature vector and the semantic vector of the user command can be fused through the cross-modal attention mechanism. This mechanism ensures that the audio information and visual information can work together in the same feature space to generate a joint feature vector, so that the audio and user commands can guide each other in the generation process and jointly influence the visual content.

[0040] Furthermore, a preliminary key frame sequence is generated using diffusion models. Diffusion models are a technique for generating high-quality images based on a random process. Images that match the expected style can be generated based on the user's style parameters (such as reference images and text instructions). These key frames will become the basis for subsequent dynamic video generation.

[0041] It can be understood that by aligning the visual information in the audio and the user's instructions, the system is able to generate preliminary visual scenes that meet the user's needs and ensure that these scenes echo the music content and emotional characteristics.

[0042] Step S105, dynamically adjusting the particle special effect density and generating the shot switching points for the initial key frame sequence according to the spectrum energy characteristics in the music feature vector, and outputting an enhanced key frame sequence; The system dynamically couples the audio features with the generated initial key frame sequence to generate an enhanced key frame sequence with special effects. Special effects generation and camera switching can accurately adjust the screen special effects and visual rhythm according to the emotional changes and rhythm changes of the audio, making the interaction between audio and vision closer and smoother, greatly enhancing the audience's sense of immersion.

[0043] Specifically, the density of particle effects can be dynamically calculated based on the spectral energy characteristics (RMS) in the audio. The size of the spectral energy directly determines the intensity of the particle effects, so that the strength of the audio corresponds to the changes in the visual effects. For example, high-frequency audio signals may trigger light spots or other visual effects. In addition, during the generation of lens switching points, the system can optimize the lens switching strategy through reinforcement learning (PPO algorithm) to ensure that the lens switching frequency in the video is aligned with the timestamp of the audio rhythm. The accuracy of lens switching can ensure that the video remains smooth and natural when watching.

[0044] Step S106, performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence; In some embodiments, the RAFT optical flow model can be used to interpolate between enhanced key frames to solve the problem of picture jumps and ensure smooth video transitions. At the same time, the Transformer model is used for long-term time series prediction to predict subsequent frames to maintain the consistency of the position of the video characters and the scene logic.

[0045] Step S107, performing block-by-block parallel rendering on the optimized video frame sequence according to the target platform parameters to generate MV files and metadata.

[0046] The video is divided into multiple segments and accelerated rendering is performed using a GPU cluster. At the same time, the video resolution is dynamically adapted for different platforms (such as short video platforms, professional video platforms, etc.). For example, for short video platforms, the ESRGAN model is used to perform super-resolution enhancement on 1080P videos; while for professional platforms, the original 8K resolution is maintained and encoding parameters are optimized to ensure video output quality.

[0047] It can be understood that through block parallel rendering and resolution adaptation, the system can efficiently generate MVs suitable for different target platforms, ensuring that the final video has good visual effects on all platforms.

[0048] In the above implementation, audio and visual information are deeply integrated to accurately generate dynamic scenes that match emotional colors and visual styles according to user instructions, and lens switching is optimized through reinforcement learning to ensure high synchronization and interaction between audio and visual content. At the same time, adaptive rendering strategies are used to adapt to the needs of different platforms to ensure that high-quality MVs can be efficiently generated, thereby improving video creation efficiency and content quality.

[0049] Reference Figure 2 As an implementation of step S103, the steps of extracting rhythm features and emotional features from the standardized audio signal and generating a music feature vector include: Step S201, performing frame division and windowing processing on the standardized audio signal to generate a short-time signal frame sequence and perform time-frequency transformation to obtain a complex time-frequency matrix; Specifically, the continuous audio signal is cut into short-time frames (such as 25ms / frame) using a predefined frame length and frame shift, and windowing is performed using the Hamming window function to reduce spectrum leakage caused by signal truncation and ensure short-time stability. Next, the short-time Fourier transform (STFT) is used to convert each frame of the signal from the time domain to the frequency domain, generating a time-frequency matrix in complex form to characterize the energy distribution of the signal at different frequencies.

[0050] It can be understood that by converting the continuous audio signal into a processable time-frequency matrix, the subsequent analysis of the frequency and time characteristics of the audio is facilitated.

[0051] Step S202, calculating the spectrum flux sequence based on the complex time-frequency matrix, and generating a beat timestamp sequence by a dynamic programming beat tracking algorithm; Among them, spectral flux describes the rhythmic characteristics of audio signals by calculating the amplitude of spectrum changes between adjacent frames. It is usually used to identify sudden changes in audio signals (such as rhythm changes, the appearance of drum beats, etc.). The dynamic programming beat tracking algorithm is used to extract rhythm information from spectral flux, track and generate beat timestamp sequences in audio. The algorithm finds the optimal beat positioning through optimization problem solving, making rhythm tracking more accurate.

[0052] In one of the embodiments of the present application, the signal mutation points (such as drum beats and note starts) are detected by using the spectral energy difference between adjacent frames to generate a spectral flux sequence reflecting the rhythm intensity, and then a candidate beat grid is generated within a preset rhythm range (such as 60-180 BPM) based on the spectral flux sequence. Based on the matching degree between the flux sequence and the candidate beats, a dynamic programming algorithm selects the global optimal beat sequence, minimizes the timing error function, and outputs the beat timestamp.

[0053] It can be understood that by extracting rhythm change information through spectral flux, the beat timestamp sequence can accurately reflect the rhythm structure of the audio signal, which is helpful for the subsequent recognition and analysis of audio features.

[0054] Step S203, extracting the spectrum envelope of the complex time-frequency matrix based on the Mel-scale filter bank, generating an MFCC feature matrix, and calculating a low-frequency energy proportion sequence; Among them, the Mel scale filter bank can simulate the perceptual characteristics of human hearing. By extracting the spectrum envelope through the filter bank, the spectrum can be compressed to the Mel scale. By extracting the spectrum envelope of the time-frequency matrix, the MFCC (Mel frequency cepstral coefficient) feature matrix can be generated. It is a low-dimensional representation of the audio signal and is widely used in speech processing and music analysis. The low-frequency energy ratio sequence reflects the energy changes in the low-frequency part of the audio signal and is a way to analyze emotional characteristics.

[0055] In one of the embodiments of the present application, 40 Mel-scale triangular filters can be constructed to cover the frequency range of 20Hz-8kHz, and the linear frequency can be mapped to the Mel scale to highlight the key frequency bands of speech and music. In addition, the spectrum envelope features are extracted through logarithmic energy and discrete cosine transform (DCT) to characterize the timbre and instrument type, and a 13-dimensional MFCC matrix is ​​generated. Then, by extracting the spectral energy within the preset low-frequency range in the complex time-frequency matrix, the ratio of the low-frequency energy to the total energy of each frame signal is calculated to generate a low-frequency energy proportion sequence.

[0056] Exemplarily, the energy proportion of the 0-200 Hz frequency band is calculated to reflect the low-frequency strength of the music (such as bass and drums). For example, the low-frequency proportion of sad style music is >40%, and that of exciting style music is <20%.

[0057] Step S204, concatenating the MFCC feature matrix with the low-frequency energy proportion sequence to construct an enhanced feature matrix; Among them, the MFCC matrix M and the low-frequency energy ratio sequence E low Splice and get the enhanced feature matrix M′∈R n ×14 .

[0058] Step S205, inputting the enhanced feature matrix into the Transformer encoder to obtain the emotion label and the time series feature matrix; Among them, the Transformer encoder is a neural network architecture based on the self-attention mechanism. It is good at processing sequence data and can capture the temporal dependencies in audio signals through weighted combination of input features.

[0059] Specifically, a 12-layer Transformer encoder (number of heads = 8, hidden layers = 512) can be used to take the MFCC feature matrix and the low-frequency energy ratio sequence as input. After being processed by the Transformer encoder, the emotional labels and time series feature matrices are obtained. These emotional labels can be used to represent the emotional categories of the audio (such as excitement, sadness, etc.), and the time series feature matrix is ​​a feature representation of the audio signal changing over time. For example, when the high-frequency energy increases suddenly and continuously, the model outputs the "excited" label (probability>0.8).

[0060] Step S206, fusing the beat timestamp sequence, the emotion tag and the time series feature matrix to generate a music feature vector containing rhythm features and emotion features.

[0061] Among them, the beat timestamp sequence, emotion label and time series feature matrix are fused into an overall feature vector, which contains the rhythmic characteristics and emotional characteristics of the audio signal, and comprehensively considers the changes of the audio signal in time, frequency and emotion.

[0062] Specifically, the beat timestamp sequence is one-hot encoded to generate a beat indication vector. Through the gated attention mechanism, the similarity weights between the time series feature matrix and the beat indication vector are calculated, and the feature fusion ratio is dynamically allocated according to the weights. The weighted fusion features are generated and the fusion results are compressed and encoded, and a music feature vector of fixed dimension is output, finally obtaining an audio feature representation containing rich rhythm and emotional information.

[0063] In the above implementation, after accurate processing of the standardized audio signal, features are extracted from multiple aspects such as time-frequency features, rhythm, and emotion, and finally a music feature vector containing rich information is generated by fusion. This technical solution can accurately capture the rhythm and emotional changes in audio signals, and provides strong support for applications such as music analysis, emotion recognition, and music recommendation. By combining multi-dimensional information such as spectrum, beat, and emotion tags, the accuracy of audio features and the breadth of their application are improved.

[0064] Reference Figure 3 As an implementation method of step S104, the music feature vector and the semantic vector are feature aligned to generate a joint feature vector, and an initial key frame sequence is generated based on a diffusion model; Step S301, mapping the music feature vector and the semantic vector to the same feature space, injecting position coding into the mapped music feature vector, and generating an aligned music feature vector; In cross-modal learning, audio and visual information are usually embedded in different feature spaces. In order to enable effective interaction and alignment between these modalities, the audio feature vector (music feature vector) and the semantic vector need to be mapped to the same feature space. The music feature vector (128 dimensions) can be linearly projected to the same dimensional space as the semantic vector (512 dimensions) through a fully connected layer to eliminate the dimensional differences between the modalities and solve the problem of cross-modal data heterogeneity.

[0065] Furthermore, sinusoidal position coding is added to the projected music feature vector to preserve the temporal characteristics of the music signal (such as the order of beat points) and avoid the loss of temporal information. For example, for a piece of music with a crescendo rhythm, its position coding will mark the energy change trend at different time points.

[0066] It can be understood that position encoding can provide position information in the audio sequence, ensuring that the temporal order of the audio is effectively preserved, thereby providing the necessary context for the timing information of the audio, which is crucial for subsequent processing.

[0067] Step S302, using the aligned music feature vector as the query vector, the semantic vector as the key vector and the value vector, and calculating the association weight between the query vector and the key vector based on the cross-modal attention mechanism, and outputting the context-aware feature; Among them, the cross-modal attention mechanism is an important tool for processing audio and visual data. This mechanism can map information of different modalities to the same feature space and dynamically calculate weights based on the relationship between them. This process utilizes the "query-key-value" mechanism.

[0068] In this stage, the music feature vector after position encoding is used as the query vector, while the semantic vector is used as the key vector and value vector. By calculating the correlation between the query vector and the key vector (usually using dot product or cosine similarity), an association weight matrix can be obtained. Then, based on this weight matrix, the value vector can be weighted averaged to obtain context-aware features, that is, a representation that combines audio information and semantic information.

[0069] In one embodiment of the present application, the aligned music features are used as query vectors (Q), the semantic vectors are keys (K) and values ​​(V), and the cross-modal association weights are calculated through multi-head attention (8 heads). The specific calculation formula is: ; Among them, d k is the feature dimension of the key vector in a single attention head, such as d k =64;QK T It is the query-key dot product, which is used to calculate the similarity matrix between music features and semantic features, and quantify the association strength between music clips and semantic instructions (such as the high similarity between drum segments and the "intense" instruction). Softmax is a normalization function, which is used to probabilize the similarity matrix along the key vector dimension (i.e., the semantic feature dimension), convert the similarity into a probability distribution, and determine the contribution weight of semantic features to music features.

[0070] For example, when the music features contain the emotion of "sadness" (high low-frequency energy) and the semantic instruction is "rainy day scene", the formula automatically assigns high weights to semantic dimensions such as "gloomy" and "low saturation", driving the generation of a gray-toned picture.

[0071] Step S303, dynamically allocating the fusion weights of the context-aware features and the semantic vectors through a gating function and performing weighted summation to generate a joint feature vector; In cross-modal learning, the fusion weight between audio and visual information is usually not fixed, but needs to be dynamically adjusted according to the specific context. The gating mechanism can dynamically adjust the fusion weight according to the different feature values ​​and context of the input features, thereby controlling the degree of fusion of audio and visual features.

[0072] Specifically, the gating function determines the contribution of context-aware features and semantic vectors in the final joint features by calculating the relative importance of the input features. These weighted features are then summed to obtain the final joint feature vector. The joint feature vector contains the rhythm, emotional features, and semantic information of the audio signal, thus providing rich multimodal information for the subsequent image generation process.

[0073] In one of the embodiments of the present application, the gating function may use a Sigmoid function to generate a dynamic weight G∈[0,1] to control the fusion ratio of context-aware features and semantic vectors, and perform weighted summation of the context-aware features and semantic vectors according to the calculated weights to obtain a joint feature vector.

[0074] Specifically, the calculation formula of the gating function is: G = σ (W g ·[H attn ;V text ]); In the above formula, σ is the Sigmoid function, W g is the weight matrix, H attn is the context-aware feature, V text is the semantic vector, [H attn ;V text ] indicates a concatenation operation.

[0075] For example, if the user instruction emphasizes "rhythm priority", G approaches 1 to strengthen the guiding role of the music characteristics.

[0076] Step S304, obtaining user style parameters according to user instructions; The user style parameters include, but are not limited to: a visual style vector of a reference image and a semantic vector of text instructions (text keywords and style parameters).

[0077] In one of the embodiments of the present application, the CLIP-ViT model can be used to extract the visual style vector, encoded texture, color and other features of the reference image. The semantic vector can be directly reused or the style keywords (such as "ink style") in the text instructions can be extracted through an additional encoding layer.

[0078] It is understandable that the user's instructions contain the requirements for visual content and style descriptions, which are expressed through semantic vectors and style vectors of reference images. By parsing the user's instructions, these visual style parameters can be extracted, thereby providing the necessary conditions for subsequent image generation.

[0079] Step S305, concatenating the joint feature vector and the user style parameter into a conditional latent vector and inputting it into the diffusion model; Among them, when generating images, conditional generation models (such as diffusion models) usually need to input conditional vectors to control the style, content and other aspects of the generation process. Concatenating the joint feature vector and the user style parameters into a conditional latent vector enables the diffusion model to refer to the emotional changes of the audio and the user's visual needs during the image generation process and generate preliminary images that meet these conditions.

[0080] Step S306, iterating the conditioned latent vector through a reverse denoising process to generate a latent space image representation; The diffusion model generates high-quality images from the latent space through a reverse denoising process. This process gradually denoises and generates a qualified image representation, and converts the latent space representation into a visual image through a decoder. Through multiple iterations, the diffusion model can generate gradually clearer images and finally obtain a preliminary key frame sequence.

[0081] Specifically, the diffusion model predicts the noise of the current latent vector based on the conditioned latent vector and progressively denoises it according to the noise scheduling parameters to generate a high-quality latent space image representation.

[0082] The specific calculation formula for reverse denoising is: ; In the above formula, X t represents the potential vector at step t, α t represents the noise scheduling parameter, β t represents the noise variance parameter, represents the cumulative noise factor, represents the noise prediction network, σ t represents the noise injection coefficient; z represents independent Gaussian noise, and X t Same dimension.

[0083] For example, if the conditioned latent vector includes the feature of "electronic music", the network will predict high-frequency noise (corresponding to the light effect particles in the picture), and the denoising process tends to generate high-light-effect, technological-feeling pictures.

[0084] Step S307, decoding the latent space image representation and outputting an initial key frame sequence.

[0085] Specifically, the latent space image representation is mapped to the pixel space through the pre-trained VAE decoder, which is then decoded into a real image and outputs a key frame sequence. The decoder usually uses deconvolution or upsampling layers to gradually restore the resolution.

[0086] In the above implementation, through the process of cross-modal feature alignment, dynamic weighted fusion and conditional generation, the deep fusion and interaction of audio and visual information is achieved, ensuring the high degree of fit between the rhythm and emotional changes of the audio and the visual content, and adjusting the image generation process through the style parameters of the user's instructions. Finally, the diffusion model is used to generate a preliminary key frame sequence that meets the user's needs, laying the foundation for subsequent dynamic video generation.

[0087] Reference Figure 4As an implementation method of step S105, according to the spectrum energy characteristics in the music feature vector, the initial key frame sequence is dynamically adjusted in particle special effect density and the lens switching point is generated, and the step of outputting the enhanced key frame sequence includes: Step S401, extracting spectrum energy features according to music feature vectors; The spectral energy value of the music signal reflects the changes in audio intensity within different frequency ranges, and the spectral energy features can be extracted through frequency domain analysis (such as Fourier transform). The spectral energy features include the rhythm, emotional changes and frequency characteristics of the audio, which can provide a basis for subsequent visual effects and shot switching.

[0088] In some embodiments, the audio signal can be decomposed into multi-band energy distribution through short-time Fourier transform (STFT) or Mel spectrum analysis to extract the time-frequency domain feature matrix E(t,f), where t is the time frame and f is the frequency band index. For example, if the music contains high-frequency drum beats (such as above 8kHz), the system can identify the energy surge area of ​​the frequency band and mark it as a potential special effect trigger point.

[0089] Step S402, performing frequency band separation and time alignment on the initial key frame sequence based on the spectrum energy characteristics, generating frequency band energy curves and aligned key frame timestamps; The audio spectrum energy characteristics are dynamically changing, so aligning the spectrum energy of the audio signal with the time of the initial key frame sequence can ensure that each key frame is synchronized with the changes in audio rhythm and emotion. Frequency band separation makes the different special effects control of low-frequency and high-frequency bands clearly targeted, avoiding the asynchrony or disharmony between vision and audio.

[0090] Specifically, the short-time Fourier transform (STFT) can be used to calculate the spectrum energy in each time period, which is divided into low-frequency, medium-frequency and high-frequency bands (such as low-frequency 20-250Hz, medium-frequency 250-4kHz, and high-frequency 4-20kHz). The energy integral of each frequency band is calculated and the corresponding energy curve is obtained. Then, the mapping relationship between the audio sampling rate (such as 44.1kHz) and the video frame rate (such as 24fps) is used to align the audio timeline and the initial key frame timestamp through the interpolation algorithm to ensure that the energy information at each time point is synchronized with the corresponding visual effects. For example, if the audio duration is 60 seconds, corresponding to 1440 frames (24fps), each frame accurately matches about 30.7ms of audio energy data.

[0091] Step S403, dynamically adjusting the particle special effect density through a piecewise linear mapping function according to the frequency segment energy curve, and generating a key frame sequence with dynamic particle special effects; The density of particle effects (such as the number of particles, movement speed, etc.) should be closely related to the spectral energy of the audio. The energy change of the spectrum affects the intensity and type of special effects. Low frequencies usually correspond to lower density and slower moving special effects, while high frequencies are associated with more and faster particle effects. By converting energy values ​​into special effect density through a linear mapping function, the consistency of audio intensity and visual performance can be achieved.

[0092] Specifically, a piecewise linear function can be defined to map the energy values ​​(such as RMS values) of different frequency bands to density parameters in the interval [0,1]. These density parameters control the density of the particle effects, and then dynamically adjust the number, speed, size, direction of movement and other characteristics of the particles in the particle system according to the normalized density parameters. For example, when the energy value of the high frequency band of the audio is high, the particle effects will become more dense and faster. When the high frequency energy reaches its peak, it will trigger high-density light spot particles; otherwise, the particle density and speed will be reduced.

[0093] Step S404, optimizing the initial shot switching candidate points based on the reinforcement learning strategy network to generate a shot switching point sequence aligned with the sub-band energy curve; Among them, the use of reinforcement learning algorithms (such as PPO) to optimize the shot switching strategy can ensure that the switching points are highly consistent with the rhythm and emotional changes of the audio. The strategy network optimizes the accuracy of the switching points through the reward function, so that the switching timing is consistent with the rhythm and can enhance the emotional expression of the video, reducing the reliance on artificial rules.

[0094] Step S405, merging the key frame sequence with dynamic particle special effects and the lens switching point sequence, inserting dynamic transition effects and superimposing special effects-lens linkage parameters, and outputting an enhanced key frame sequence.

[0095] Among them, by fusing the keyframe sequence with particle effects and the sequence of lens switching points, the visual effects are ensured to have coherence and rhythm throughout the video. Dynamic transition effects (such as fade in and fade out, motion blur) further enhance the naturalness of the transition, while the special effects-lens linkage parameters ensure the coordination of visual effects and lens switching.

[0096] Specifically, dynamic transition effects are inserted before and after the shot switching point, the duration of the transition is calculated based on the energy gradient, and the natural transition is ensured through smoothing (such as motion blur, fade-in and fade-out, time and space distortion, etc.). In high-energy areas (such as the climax of the audio), strong visual effects such as lens vibration or particle bursts are superimposed to enhance emotional expression, and finally an enhanced key frame sequence with dynamic transition effects is generated and output through the rendering engine.

[0097] It should be noted that the enhanced keyframe sequence ensures the perfect fit between audio and visual effects. The visual effects not only respond to changes in audio, but also enhance the audience's immersion through shot switching and transition effects, thereby improving the overall video quality.

[0098] In the above implementation, the special effects mapping based on spectral energy and the reinforcement learning optimized shot switching strategy ensure the perfect combination of video fluency and emotional expression, and achieve a high degree of synchronization between video content and audio rhythm and emotion. This technical solution can be widely used in scenarios such as real-time concert visual effects generation, game cutscene animation production, and short video AI soundtrack editing.

[0099] Reference Figure 5 As an implementation method of step S404, the step of optimizing the initial shot switching candidate points based on the reinforcement learning strategy network and generating a shot switching point sequence aligned with the sub-band energy curve includes: Step S501, determining an initial shot switching candidate point according to a beat timestamp sequence and a frequency band energy curve; Among them, the beat timestamp sequence reflects the rhythm change of the music, that is, the time point when the rhythm occurs, which is usually extracted by the BPM (beats per minute) of the audio and the volume peak change of the audio signal. The initial shot switching candidate point should be selected according to the change of rhythm so that the shot switching conforms to the rhythm and change of the audio.

[0100] In some embodiments, audio analysis tools (such as fast Fourier transform, rhythm detection algorithm, etc.) can be used to extract beat timestamps from audio signals. These beat timestamps correspond to the rhythm changes of the audio, and their positions are used as initial shot switching candidate points based on the changes in the beats. The frequency band energy curve can be combined to filter out candidate points in low energy intervals (such as areas with energy below the average value), and retain high confidence (aligned with high frequency energy peaks) candidate points.

[0101] Step S502, constructing a state vector set including real-time sub-band energy flow data, historical switching intervals and candidate point offsets according to the sub-band energy curves and the initial shot switching candidate points; Among them, shot switching needs to consider not only rhythm changes, but also the dynamic changes of frequency band energy, the timing of historical switching, and the offset between the current switching point and the candidate point. The state vector set describes the environmental state required for each switching decision by integrating this information.

[0102] Specifically, the energy data of low frequency, medium frequency and high frequency are extracted from the frequency band energy curve, and the energy changes at each time point are updated in real time. These real-time frequency band energy flow data can reflect the energy intensity of the audio in different time periods; the historical switching interval is to calculate the time interval between the previous shot switch and the current shot candidate point, which is crucial to avoid too frequent or too long no switching phenomenon; the candidate point offset is to calculate the absolute deviation between the current candidate point and the most recent energy peak.

[0103] Step S503, designing a multi-objective reward function based on the state vector set, and calculating the instantaneous reward values ​​of the rhythm alignment error, the switching fluency penalty, and the emotion consistency reward; Specifically, the goal of reinforcement learning is to guide the network to learn the optimal strategy through rewards. It is crucial to design a reasonable reward function, which can guide the shot switching strategy through a multi-objective reward function to ensure that the shot switching is consistent with the rhythm, enhances emotional consistency and ensures fluency, and avoids overfitting of a single indicator.

[0104] Among them, the rhythm alignment error reward: the degree of time alignment between the shot switching point and the rhythm change point is scored. A higher alignment degree is rewarded positively, while a lower alignment degree is penalized. This reward is defined by calculating the rhythm switching error (for example, the offset between the switching point and the rhythm). Switching smoothness penalty: If the shot switches too frequently or at inappropriate times, the visual experience will be unsmooth, so a penalty term is added to encourage appropriate switching intervals. Emotional consistency reward: In order to ensure that the shot switching not only matches the rhythm, but also matches the emotional changes of the audio, emotional consistency is rewarded. The emotional strength of the audio is judged through emotional analysis tools, and the shot switching is encouraged to be consistent with the emotional changes.

[0105] For example, in the climax of rock music (high frequency accounts for 70%), the reward function encourages 2-3 switches per second to match the intense rhythm.

[0106] Understandably, the multi-objective reward function ensures that shot switching strikes a balance in multiple aspects, such as rhythm, fluidity, and emotional consistency, thereby generating shot switching points that are more in line with artistic requirements.

[0107] Step S504, inputting the state vector set and the instant reward value into the policy network, generating the probability distribution of the shot switching action through PPO algorithm training, screening the initial shot switching candidate points to generate the optimized shot switching point sequence; Among them, the PPO algorithm is based on the reinforcement learning framework of proximal policy optimization. It ensures training stability by limiting the policy update amplitude and is suitable for high-dimensional continuous state space problems. The policy network is trained through the PPO algorithm to generate the probability distribution of the lens switching action. At each moment, the policy network selects the most appropriate lens switching action based on the state vector set.

[0108] Specifically, the PPO algorithm is used to train the policy network, with the goal of maximizing the cumulative reward. Each time the network selects a shot switching action (switch or keep) based on the state vector, and updates the policy parameters based on the immediate reward. Through PPO training, the initial shot switching candidate points with a shot switching action probability greater than a preset threshold are screened to obtain the final optimized shot switching point sequence.

[0109] It is understandable that the trained strategy network can intelligently select the most appropriate switching point from multiple candidate points, reducing the complexity of manual settings and ensuring that the shot switching is highly consistent with the music rhythm and emotional changes.

[0110] Step S505, according to the optimized shot switching point sequence and the real-time frequency-band energy flow data, the dynamic action space is expanded and a transition frame or a hard-cut action is inserted, and the final shot switching point sequence and dynamic transition parameters are output.

[0111] Among them, the optimized shot switching point sequence needs to further enhance the transition effect to achieve a smooth transition between shots and improve the video viewing experience, ensuring that the switching points are coordinated with visual fluency and emotional changes.

[0112] Specifically, the action space is dynamically adjusted according to the real-time frequency-band energy flow data, and more possible shot switching types are added (for example, transition frames, hard switching, fade-in and fade-out, etc.). At the same time, for switching points with higher energy, transition frames (such as dynamic blur, fade-in and fade-out, etc.) are inserted to ensure visual fluency; for places with lower energy and more prominent rhythm, hard switching can be performed directly to quickly transition to the next shot.

[0113] In the above implementation, the entire shot switching optimization process uses a reinforcement learning network to accurately optimize the shot switching point sequence, and enhances the expressiveness of audio-visual synchronization by dynamically adjusting the transition effect, creating a dynamic visual effect that is more in line with the audio characteristics, and achieving a high balance in rhythm alignment, visual fluency, and emotional consistency. The optimized shot switching not only conforms to the rhythm and emotional changes of the audio, but also provides a natural and expressive visual effect, greatly improving the quality and immersion of the video content.

[0114] Reference Figure 6As an implementation of step S106, the step of performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence includes: Step S601, grouping the enhanced key frame sequence into shots according to the beat timestamp sequence to obtain a shot group sequence; The beat timestamp sequence is generated by the dynamic programming beat tracking algorithm of the audio signal and is used to mark the strong and weak beat positions in the music. The core of shot grouping is to divide the enhanced key frame sequence into shot groups aligned with the music beat, ensuring that each shot group corresponds to a continuous rhythm interval (such as the frame sequence between two adjacent strong beats).

[0115] For example, if the beat timestamp is T beat ={t1,t2,...,tK}, then the k-th lens group G k Contains timestamp in [t k ,t k +1). By strictly aligning the shot group boundaries with the beats, it is possible to avoid the camera switching across rhythm points causing the sound and picture to be out of sync (for example, the camera switching at the weak beat may cause a sense of visual disconnection).

[0116] Step S602, extracting a multi-scale feature map and particle special effect parameters of each shot group in the shot group sequence to obtain a mixed feature tensor of each shot group; Each shot group contains multiple visual features, such as object shape, motion, lighting changes, particle effects, etc. By extracting multi-scale feature maps and particle effect parameters, the visual content of the shot group can be fully described. By fusing these features into a hybrid feature tensor, sufficient contextual information is provided for subsequent interpolation and prediction models.

[0117] Step S603, interpolating frames using the RAFT optical flow model based on the mixed feature tensors of the adjacent shot groups, and optimizing the interpolation results using the residual correction network to obtain an optical flow interpolation result; Among them, the optical flow interpolation technology infers the content of the intermediate frames based on the pixel motion between video frames, ensuring that the visual effect transitions smoothly between two key frames. The RAFT optical flow model is an efficient optical flow estimation method that can capture fine-grained pixel-level motion and generate high-quality interpolated frames.

[0118] Specifically, the RAFT optical flow model is used to generate an optical flow field based on adjacent shot groups. The optical flow field represents the motion information of each pixel from one frame to another. Based on the estimated optical flow field, the model can insert intermediate frames between two shot groups. The interpolation process can produce a smooth transition effect and avoid sudden picture jumps.

[0119] Furthermore, the intermediate frames generated by interpolation may cause local blur or artifacts due to optical flow estimation errors. Therefore, the residual correction network can be used to optimize the optical flow interpolation results to reduce the errors in optical flow estimation. By comparing the optical flow interpolation results with the original key frames, the interpolation process can be adjusted to reduce visual distortion.

[0120] Step S604, performing long-term time series prediction according to the shot group sequence after interpolation and the particle special effect parameters, and outputting the long-term time series prediction result; Among them, the Transformer model performs well in processing long time series data and can learn long-term dependencies in time series. Combined with the dynamic adjustment of particle effects, the Transformer model can be used to predict the future state of video frames and generate output that is more in line with emotional changes and visual needs.

[0121] Specifically, the interpolated shot sequence is input into the Transformer model, and the particle special effect parameters are combined for timing prediction. These particle special effect parameters will be spliced ​​with the frame features as additional input channels, directly affecting the model prediction results. For example, high-density particles may suppress the confidence of the character motion prediction (adjusting the output heat map through the gating mechanism).

[0122] In an embodiment of the present application, the Transformer model processes the timing data of the shot group through the self-attention mechanism, learns the dependencies between each shot, generates predictions for future video frames, and outputs long-term time series prediction results, including character position heat maps (i.e., the movement trajectory of objects or characters in the video) and scene logic labels (such as scene changes, emotional turning points, etc.).

[0123] Step S605 , weighted fusion is performed to generate an optimized video frame sequence according to the motion vector difference between the optical flow interpolation result and the long-term prediction result, and particle special effect parameters are adjusted according to the scene logic label.

[0124] Among them, the motion vector difference provides the matching degree between the optical flow interpolation result and the long-term prediction result. Weighted fusion is performed based on this difference to ensure that the final generated video frame not only conforms to the motion trajectory but also reflects the change of emotion.

[0125] Specifically, the difference threshold can be set based on experience. When the motion vector difference exceeds the threshold, weighted fusion is triggered. Otherwise, the optical flow interpolation result is directly used. In the weighted fusion process, the weight coefficient formula of the optical flow interpolation result is: λ=exp(−δ avg / γ), where λ is the weighting coefficient; γ is the attenuation coefficient, which is used to control the fusion sensitivity; δ avg is the motion vector difference. For example, if γ is preset to 3, the calculated δ avgis 4.7, then λ is about 0.21, the weight of the optical flow interpolation result is 21%, and the weight of the long time series prediction result is 79%.

[0126] In addition, based on the scene logic labels in the long-term prediction results, the density of particle effects is dynamically adjusted to ensure that the visual effects are consistent with the emotional changes and movement patterns of the video. For example, if the scene label is "heavy rain", the greater the difference in motion vectors, the higher the density of raindrops.

[0127] In the above implementation, after weighted fusion and dynamic particle effects adjustment, the optimized video frame sequence finally generated has a visual effect of smooth transition, accurate motion prediction and consistent emotion. The density adjustment of the particle effects further enhances the expressiveness of the video, making the entire video more natural and smooth, and highly consistent with the audio rhythm and emotion.

[0128] Reference Figure 7 As a further implementation method of the intelligent MV generation method, after the steps of weighted fusion to generate an optimized video frame sequence according to the motion vector difference between the optical flow interpolation result and the long time series prediction result, and adjusting the particle special effect parameters according to the scene logic label, it also includes: Step S701, extracting a shot switching point sequence according to an optimized video frame sequence; Among them, shot switching points are key positions in the video, usually rhythm changes or emotional turning points. These points are usually located where the rhythm of the audio changes significantly and can be identified through automated visual analysis (such as feature extraction based on convolutional neural networks).

[0129] Step S702, aligning the shot switching point sequence with the beat timestamp sequence according to the frame rate to obtain a set of audio and video timestamp pairs; The shot switching point sequence and the beat timestamp sequence are aligned by nonlinear interpolation according to the frame rate to ensure that the switching timing of each shot is synchronized with the rhythm change point of the audio. For example, if the video frame rate is 30fps and the music sampling rate is 44.1kHz, the beat timestamp sequence is mapped to the video timeline through cubic spline interpolation.

[0130] Step S703, obtaining an optical flow motion vector field according to the optical flow interpolation result; Among them, the optical flow motion vector field represents the movement direction and speed of each pixel. The optical flow motion vector field can accurately describe the motion information between video frames, which plays an important role in subsequent synchronization analysis, motion compensation and quality assessment.

[0131] Step S704, calculating the shot switching deviation and rhythm consistency index according to the audio and video timestamp pair set and the optical flow motion vector field, and generating a synchronization rate score; Among them, the shot switching deviation and rhythm consistency indicators are used to evaluate the visual smoothness and rhythm matching of the video. By calculating the time deviation and rhythm alignment of the shot switching, a synchronization rate score can be generated to evaluate the synchronization quality of the video and audio.

[0132] Specifically, the time difference between the lens switching point and the rhythm change point is calculated to obtain the lens switching deviation BCD. A smaller deviation means that the lens switching and the rhythm are highly matched. Based on the optical flow motion vector field, the degree of fit between the motion after the lens switching and the audio rhythm is calculated to measure whether the visual effect after the switching is consistent with the rhythm and emotional changes. Based on the lens switching deviation and rhythm consistency indicators, a comprehensive synchronization rate score MRC is generated. The higher the synchronization rate score, the better the synchronization between the video and the audio.

[0133] In some embodiments, the joint entropy can be calculated by performing a short-time Fourier transform (STFT) correlation analysis on the optical flow motion vector field and the audio and video timestamp pair set. The lower the entropy value, the stronger the synchronization. In addition, the synchronization rate score is calculated as follows: sync =1−BCD+MRC. This score can be used to objectively evaluate the synchronization of audio and video in a video and help optimize the video production process.

[0134] Step S705, obtaining user style parameters according to user instructions; The user style parameters include, but are not limited to: a visual style vector of a reference image and a semantic vector of text instructions (text keywords and style parameters).

[0135] Step S706, comparing the adjusted particle special effect parameters with the user style parameters, calculating the style similarity and the particle parameter compliance index, and generating a style consistency score; Among them, the style consistency score is used to evaluate the consistency between the visual style of the video and the user's expected style. By comparing the adjusted particle special effects parameters with the user's style parameters, the style similarity and particle compliance are calculated, thereby quantifying the style matching of the video.

[0136] Specifically, the actual parameters of the particle effects (such as the number of particles, movement mode, etc.) are compared with the style parameters in the user's instructions, and the similarity between the two is calculated. Similarity measurement methods (such as cosine similarity or Euclidean distance) can be used to quantify the difference between the two. At the same time, the particle effects are compared with the constraints pre-set by the user (such as particle density, speed, etc.) to obtain the particle parameter compliance index.

[0137] In some embodiments, the weight of style similarity can be configured to be 0.6, and the weight of particle parameter compliance can be configured to be 0.4. A weighted fusion is performed based on the style similarity and compliance indicators to generate a style consistency score to indicate the degree of match between the visual style of the video and user requirements.

[0138] Step S707: Comprehensively combine the synchronization rate score and the style consistency score to generate a quality assessment result.

[0139] Among them, the quality assessment results can provide a comprehensive video quality evaluation, covering rhythm synchronization, visual effects and the matching degree of user needs, which helps to optimize the video production process and improve the overall video quality and user experience.

[0140] In the above implementation, automated evaluation of video quality is achieved, which can provide accurate feedback for video production, thereby helping the production team to quickly identify and improve problems in various aspects of video creation, thereby improving the visual effects of the video content, audio and video synchronization, and satisfaction of users' personalized needs.

[0141] The embodiment of the present application also discloses an intelligent MV generation system based on AIGC.

[0142] An intelligent MV generation system based on AIGC, the generation system comprising: A receiving module, used to receive the original music file and user instructions input by the user; wherein the user instructions include text keywords, reference images and style parameters; A processing module, used for performing audio preprocessing on the original music file to generate a standardized audio signal, and mapping the user instruction into a semantic vector through a cross-modal encoding model; A feature extraction module is used to extract rhythm features and emotional features from the standardized audio signal and generate a music feature vector; A frame sequence generation module is used to align the music feature vector with the semantic vector, generate a joint feature vector, and generate an initial key frame sequence based on a diffusion model; A frame sequence enhancement module is used to dynamically adjust the particle effect density and generate lens switching points for the initial key frame sequence according to the spectrum energy characteristics in the music feature vector, and output an enhanced key frame sequence; The frame sequence optimization module is used to perform optical flow interpolation and long-term temporal prediction on the enhanced key frame sequence to generate an optimized video frame sequence; The rendering module is used to perform block-by-block parallel rendering of the optimized video frame sequence according to the target platform parameters to generate MV files and metadata.

[0143] An AIGC-based intelligent MV generation system in an embodiment of the present application can implement any of the above-mentioned intelligent MV generation methods, and the specific working process of each module in the intelligent MV generation system can refer to the corresponding process in the above-mentioned method embodiment.

[0144] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0145] The embodiment of the present application also discloses a computer device.

[0146] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, an AIGC-based intelligent MV generation method as described above is implemented.

[0147] The embodiment of the present application also discloses a computer-readable storage medium.

[0148] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned AIGC-based intelligent MV generation methods.

[0149] Among them, computer-readable storage media can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0150] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for the parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. An intelligent MV generation method based on AIGC, characterized in that: The generation method comprises: Receiving an original music file and a user instruction input by a user; wherein the user instruction includes text keywords, reference images and style parameters; Performing audio preprocessing on the original music file to generate a standardized audio signal, and mapping the user instruction into a semantic vector through a cross-modal coding model; Extracting rhythm features and emotional features from the standardized audio signal to generate a music feature vector; Performing feature alignment on the music feature vector and the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; According to the spectrum energy characteristics in the music feature vector, dynamically adjust the particle special effect density and generate the shot switching points for the initial key frame sequence, and output an enhanced key frame sequence; Performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence; The optimized video frame sequence is rendered in blocks and in parallel according to the target platform parameters to generate MV files and metadata.

2. The AIGC-based intelligent MV generation method according to claim 1, characterized in that: The steps of extracting rhythm features and emotional features from the standardized audio signal and generating a music feature vector include: Performing frame division and windowing processing on the standardized audio signal to generate a short-time signal frame sequence and performing time-frequency transformation to obtain a complex time-frequency matrix; Calculating a spectral flux sequence based on the complex time-frequency matrix, and generating a beat timestamp sequence by a dynamic programming beat tracking algorithm; Based on the Mel-scale filter bank, the spectral envelope of the complex time-frequency matrix is ​​extracted to generate an MFCC feature matrix, and a low-frequency energy proportion sequence is calculated; The MFCC feature matrix is ​​concatenated with the low-frequency energy proportion sequence to construct an enhanced feature matrix; Input the enhanced feature matrix into the Transformer encoder to obtain the sentiment label and time series feature matrix; The beat timestamp sequence, the emotion label and the time series feature matrix are integrated to generate a music feature vector including rhythm features and emotion features.

3. The AIGC-based intelligent MV generation method according to claim 1, characterized in that: Aligning the music feature vector with the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; Mapping the music feature vector and the semantic vector to the same feature space, injecting position coding into the mapped music feature vector, and generating an aligned music feature vector; Taking the aligned music feature vector as the query vector, taking the semantic vector as the key vector and the value vector, and calculating the association weight between the query vector and the key vector based on the cross-modal attention mechanism, outputting the context-aware feature; Dynamically assigning fusion weights of the context-aware features and the semantic vector through a gating function and performing weighted summation to generate a joint feature vector; Obtaining user style parameters according to the user instruction; concatenating the joint feature vector and the user style parameter into a conditional latent vector and inputting the vector into a diffusion model; iterating the conditioned latent vector through a reverse denoising process to generate a latent space image representation; The latent space image representation is decoded to output an initial key frame sequence.

4. The AIGC-based intelligent MV generation method according to claim 2, characterized in that: According to the spectrum energy features in the music feature vector, the steps of dynamically adjusting the particle special effect density and generating the shot switching points for the initial key frame sequence and outputting the enhanced key frame sequence include: Extracting spectrum energy features according to the music feature vector; Based on the spectrum energy characteristics, the initial key frame sequence is subjected to frequency band separation and time alignment to generate frequency band energy curves and aligned key frame timestamps; According to the frequency-band energy curve, the density of particle special effects is dynamically adjusted through a piecewise linear mapping function to generate a key frame sequence with dynamic particle special effects; Optimizing the initial shot switching candidate points based on a reinforcement learning strategy network to generate a shot switching point sequence aligned with the sub-band energy curve; The key frame sequence with dynamic particle special effects is integrated with the lens switching point sequence, a dynamic transition effect is inserted, special effects-lens linkage parameters are superimposed, and an enhanced key frame sequence is output.

5. The AIGC-based intelligent MV generation method according to claim 4, characterized in that: The step of optimizing the initial shot switching candidate points based on the reinforcement learning strategy network and generating a shot switching point sequence aligned with the sub-band energy curve comprises: Determine an initial shot switching candidate point according to the beat timestamp sequence and the frequency band energy curve; According to the sub-frequency band energy curve and the initial lens switching candidate point, a state vector set including real-time sub-frequency band energy flow data, historical switching intervals and candidate point offsets is constructed; Designing a multi-objective reward function based on the state vector set, and calculating the instantaneous reward values ​​of rhythm alignment error, switching fluency penalty, and emotional consistency reward; Inputting the state vector set and the instant reward value into the policy network, generating a probability distribution of shot switching actions through PPO algorithm training, and screening the initial shot switching candidate points to generate an optimized shot switching point sequence; According to the optimized lens switching point sequence and the real-time frequency-band energy flow data, the dynamic action space is expanded and a transition frame or a hard-cut action is inserted, and the final lens switching point sequence and dynamic transition parameters are output.

6. The AIGC-based intelligent MV generation method according to claim 4, characterized in that: The steps of performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence include: Shot grouping is performed on the enhanced key frame sequence according to the beat timestamp sequence to obtain a shot group sequence; Extracting multi-scale feature maps and particle special effect parameters of each shot group in the shot group sequence to obtain a mixed feature tensor of each shot group; Based on the mixed feature tensor of adjacent lens groups, interpolation is performed through the RAFT optical flow model, and the interpolation result is optimized using the residual correction network to obtain the optical flow interpolation result; Perform long-term time series prediction according to the interpolated shot group sequence and the particle special effect parameters, and output a long-term time series prediction result; the long-term time series prediction result includes a character position heat map and a scene logic label; According to the difference in motion vectors between the optical flow interpolation result and the long-term prediction result, weighted fusion is performed to generate an optimized video frame sequence, and the particle special effect parameters are adjusted according to the scene logic label.

7. The AIGC-based intelligent MV generation method according to claim 6, characterized in that: After the steps of weighted fusion to generate an optimized video frame sequence according to the motion vector difference between the optical flow interpolation result and the long time series prediction result, and adjusting the particle special effect parameters according to the scene logic label, the method further includes: Extracting a shot switching point sequence according to the optimized video frame sequence; Aligning the shot switching point sequence with the beat timestamp sequence according to the frame rate to obtain a set of audio and video timestamp pairs; Obtaining an optical flow motion vector field according to the optical flow interpolation result; Calculating the shot switching deviation and the rhythm consistency index according to the audio and video timestamp pair set and the optical flow motion vector field, and generating a synchronization rate score; Obtaining user style parameters according to the user instruction; Based on the comparison of the adjusted particle special effect parameters with the user style parameters, the style similarity and the particle parameter compliance index are calculated to generate a style consistency score; The synchronization rate score and the style consistency score are combined to generate a quality assessment result.

8. An AIGC-based intelligent MV generation system, characterized in that: The generation system comprises: A receiving module, used to receive the original music file and user instructions input by the user; wherein the user instructions include text keywords, reference images and style parameters; A processing module, configured to perform audio preprocessing on the original music file to generate a standardized audio signal, and map the user instruction into a semantic vector through a cross-modal coding model; A feature extraction module, used to extract rhythm features and emotional features from the standardized audio signal to generate a music feature vector; A frame sequence generation module, used for aligning the music feature vector with the semantic vector to generate a joint feature vector, and generating an initial key frame sequence based on a diffusion model; A frame sequence enhancement module, used to dynamically adjust the particle special effect density and generate lens switching points for the initial key frame sequence according to the spectrum energy characteristics in the music feature vector, and output an enhanced key frame sequence; A frame sequence optimization module, used for performing optical flow interpolation and long time series prediction on the enhanced key frame sequence to generate an optimized video frame sequence; The rendering module is used to perform block-by-block parallel rendering on the optimized video frame sequence according to the target platform parameters to generate MV files and metadata.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images