Controllable video generation method and system based on multimodal fusion

Abstract

The present invention discloses a controllable video generation method and system based on multimodal fusion, the method comprising obtaining multimodal data and preprocessing it; extracting feature vectors and normalizing them, combining them to form a multimodal feature matrix; converting the multimodal feature matrix into a feature sequence; dynamically assigning weights to each modal feature in the feature sequence; performing multi-level deep fusion on the weighted feature sequence to obtain a fused feature representation; inputting the fused feature representation into a multi-scale spatiotemporal attention mechanism to obtain an enhanced feature representation; sequentially using spatial attention, temporal attention and cross-attention modules for processing to obtain a global enhanced feature representation and generate a multi-frame latent space representation; decoding to generate high-resolution video frames and performing post-processing to obtain a final high-quality video sequence. The present invention maintains the spatiotemporal coherence of the video sequence and generates video content with higher quality and greater controllability.

Description

Technical Field

[0001] The invention belongs to the field of video generation and discloses a controllable video generation method and system based on multi-modal fusion. Background Art

[0002] In the current field of video generation, most mainstream methods rely on single-modal input, such as plain text or images. Although these methods have achieved certain results to a certain extent, they still have many limitations in practical applications. Single-modal input cannot fully express complex multi-dimensional information, resulting in the generated video being limited in content expression and generation control.

[0003] Single-modal input usually includes the following typical methods: text-based generation, image-based generation, and image sequence-based generation. Text-based generation methods mainly rely on natural language processing technology to generate corresponding video content through descriptive text; image-based generation methods generate dynamic video sequences by providing static images; and image sequence-based generation methods generate videos through a series of continuous images. These methods perform well in specific scenarios, but when faced with complex applications that require the combination of multiple information sources, they often have shortcomings such as missing details and unstable quality. Therefore, a video generation method with multi-modal input is proposed. The quality of videos generated using existing multi-modal input video generation methods is low, and the spatiotemporal coherence of video sequences cannot be guaranteed, so further research and innovation are needed. Summary of the invention

[0004] The purpose of the invention is to provide a controllable video generation method and system based on multimodal fusion to solve the above-mentioned problems existing in the prior art.

[0005] The technical solution is a controllable video generation method based on multimodal fusion, comprising the following steps:

[0006] S1. Acquire multimodal data and preprocess it; extract feature vectors based on the preprocessed multimodal data using a pretrained encoder network; normalize the feature vectors and combine the normalized feature vectors to form a multimodal feature matrix; wherein the multimodal data includes text, RGB images, audio, infrared images, depth images, mask images, edge operator images, point cloud images, and posture image data;

[0007] S2. Use a predefined word segmenter to convert the multimodal feature matrix into a feature sequence, and add a modal identifier and position encoding information; apply a context-aware weight regulator to dynamically assign weights to each modal feature in the feature sequence to obtain a weighted feature sequence; input the weighted feature sequence into a hierarchical self-organizing fusion network for multi-level deep fusion to obtain a fused feature representation;

[0008] S3. Input the fused feature representation into the multi-scale spatiotemporal attention mechanism to obtain an enhanced feature representation; based on the enhanced feature representation, use the spatial attention module, the temporal attention module and the cross attention module in sequence to obtain a global enhanced feature representation; based on the global enhanced feature representation, use a dynamic latent space reconstructor to generate a multi-frame latent space representation; decode the multi-frame latent space representation through an adaptive resolution generation network to generate high-resolution video frames; post-process the high-resolution video frames to obtain the final high-quality video sequence.

[0009] A controllable video generation system based on multimodal fusion, including:

[0010] at least one processor; and,

[0011] a memory communicatively connected to at least one of the processors; wherein,

[0012] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the controllable video generation method based on multimodal fusion.

[0013] Beneficial effects: The present invention not only improves the accuracy of feature fusion, but also enhances the adaptability and flexibility of the model, so that the system can accurately control the generation of video content at different time and space scales, while maintaining the spatiotemporal coherence of the video sequence, and generating higher quality and more controllable video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention.

[0015] Figure 2 This is a flow chart of step S2 of the present invention.

[0016] Figure 3 This is a flow chart of step S3 of the present invention.

[0017] Figure 4 It is an overall framework diagram of an embodiment of the present invention.

[0018] Figure 5 FIG. 4 is a structural diagram of SiDiT according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] like Figure 1 As shown, the present invention proposes a controllable video generation method based on multimodal fusion, comprising the following steps:

[0020] S1. Acquire multimodal data and preprocess it; extract feature vectors based on the preprocessed multimodal data using a pretrained encoder network; normalize the feature vectors and combine the normalized feature vectors to form a multimodal feature matrix; wherein the multimodal data includes text, RGB images, audio, infrared images, depth images, mask images, edge operator images, point cloud images, and posture image data;

[0021] S2. Use a predefined word segmenter to convert the multimodal feature matrix into a feature sequence, and add a modal identifier and position encoding information; apply a context-aware weight regulator to dynamically assign weights to each modal feature in the feature sequence to obtain a weighted feature sequence; input the weighted feature sequence into a hierarchical self-organizing fusion network for multi-level deep fusion to obtain a fused feature representation;

[0022] S3. Input the fused feature representation into the multi-scale spatiotemporal attention mechanism to obtain an enhanced feature representation; based on the enhanced feature representation, use the spatial attention module, the temporal attention module and the cross attention module in sequence to obtain a global enhanced feature representation; based on the global enhanced feature representation, use a dynamic latent space reconstructor to generate a multi-frame latent space representation; decode the multi-frame latent space representation through an adaptive resolution generation network to generate high-resolution video frames; post-process the high-resolution video frames to obtain the final high-quality video sequence.

[0023] According to one aspect of the present application, step S1 further comprises:

[0024] S11. Receive user input or pre-stored multimodal data through the data interface module, perform format verification on the multimodal data to ensure that the data format meets the predefined standard; store the verified multimodal data in a temporary cache, and generate a data index to record the type, size and storage location information of each modal data;

[0025] S12, reading the multimodal data and preprocessing it to obtain the preprocessed multimodal data, specifically:

[0026] For text data, natural language processing tools are used to segment words and remove stop words to obtain words; pre-trained word embedding models are used to convert words into vector representations as pre-processed text data;

[0027] The RGB image data is adjusted to a uniform size and converted into a color space to obtain a converted image, and the pixel values ​​of the converted image are normalized to obtain preprocessed RGB image data;

[0028] The audio data is resampled to generate a spectrogram and extract spectral coefficients as the preprocessed audio data; specifically, the spectrogram is generated by short-time Fourier transform (STFT), and finally the Mel-frequency cepstral coefficient (MFCC) features are extracted.

[0029] Conduct heat distribution analysis on infrared image data, including hot spot detection and temperature gradient calculation, to obtain pre-processed infrared image data;

[0030] For the depth image data, the depth value is converted into a standardized depth code, and a depth histogram is generated as the preprocessed depth image data;

[0031] Apply the threshold algorithm to binarize the mask image data to obtain a clear foreground-background segmentation result as the preprocessed mask image data;

[0032] For edge operator graph data, Gaussian smoothing is used to remove noise, and edge detection operators are applied to enhance edge features to obtain preprocessed edge operator graph data;

[0033] The point cloud data is converted into a spatial coordinate system, unified into a predefined three-dimensional coordinate system, and the point cloud is downsampled to reduce the amount of data to obtain preprocessed point cloud data;

[0034] For the posture graph data, the skeleton key point detection algorithm is used to extract the coordinates of the key points of the human body, and the relative position and angle information between the key points are calculated as the preprocessed posture graph data;

[0035] S13, input the preprocessed multimodal data into the corresponding feature extraction modules respectively to extract feature vectors, specifically:

[0036] For the preprocessed text data, use the pre-trained BERT model as the encoder to extract the context-related feature vector of the text;

[0037] For the preprocessed RGB image data, the pre-trained ResNet model is used as the encoder to extract the high-level visual feature vector of the RGB image;

[0038] For the preprocessed audio data, use the pre-trained VGGish model as the encoder to extract the acoustic feature vector of the audio;

[0039] For the preprocessed infrared image data, a pre-trained convolutional neural network is used to extract the heat distribution feature vector;

[0040] For the preprocessed depth image data, the pre-trained PointNet model is used to extract the depth feature vector;

[0041] For the preprocessed mask image data, a lightweight U-Net model is used to extract the segmentation feature vector;

[0042] For the preprocessed edge operator graph data, the pre-trained EdgeNet model is used to extract edge feature vectors;

[0043] For the preprocessed point cloud data, use the pre-trained PointNet++ model to extract the 3D spatial feature vector;

[0044] For the preprocessed posture graph data, the pre-trained PoseNet model is used to extract the posture feature vector;

[0045] S14. All extracted feature vectors are normalized using the L2 norm normalization method to obtain normalized feature vectors; the normalized feature vectors are concatenated in a predefined order to form a unified multimodal feature matrix. Each row of the matrix represents a feature of a mode, and the number of columns is the sum of the dimensions of all feature vectors.

[0046] This embodiment lays a solid foundation for controllable video generation based on multimodal fusion through multimodal input processing and feature extraction. By receiving and preprocessing input data of multiple modalities (including text, RGB images, audio, infrared images, depth images, mask images, edge operator images, point cloud images, and posture images), the quality and consistency of input data are improved. Preprocessing technology ensures the effectiveness and accuracy of subsequent feature extraction. In the feature extraction stage, a specially designed encoder network (such as BERT for text, ResNet for images, VGGish for audio, etc.) is used to extract high-level features of each modal data, which can not only make full use of the transfer learning advantages of the pre-trained model, but also capture the deep semantic information of each modal data. By normalizing the extracted feature vectors, the problem of inconsistent scales of different modal features is solved, creating conditions for subsequent feature fusion. The normalized feature vectors are combined into a unified multimodal feature matrix, providing a rich and unified feature representation for the subsequent video generation process. This multimodal feature extraction and fusion method improves the content richness and semantic consistency of the generated video, so that the generated video can better reflect the complex information in the multimodal input, thereby achieving higher quality and diversity in the controllable video generation task. In addition, this embodiment also enhances the system's adaptability to different types of inputs, so that even when some modal data is missing or of low quality, high-quality video content can still be generated, thereby improving the robustness and practicality of the system.

[0047] like Figure 2 As shown, according to one aspect of the present application, step S2 is further:

[0048] S21, using a predefined word segmenter to convert each row in the multimodal feature matrix into discrete word units, adding a modality identifier to the word units of each modality to distinguish the features of different modalities; splicing the word units of all modalities into a feature sequence in a predefined order; obtaining position encoding information, and combining it with the feature sequence to obtain a combined feature sequence;

[0049] S22. Based on the combined feature sequence, a correlation graph between multimodal features is constructed using a modal correlation analysis algorithm; the importance score of each modal feature is calculated by analyzing the topological structure of the correlation graph; an initial weight is assigned to each modal feature based on the importance score; the specific requirements of the current task and the characteristics of the multimodal data are obtained, and the initial weight is dynamically adjusted to obtain an adjusted weight; the adjusted weight is applied to the combined feature sequence to obtain a weighted feature sequence;

[0050] S23. Based on the weighted feature sequence, a dynamic topology reconstruction algorithm is used to automatically adjust the hierarchical structure and connection mode of the hierarchical self-organizing fusion network to obtain an adjusted hierarchical self-organizing fusion network; based on the adjusted hierarchical self-organizing fusion network, the weighted feature sequence is fused layer by layer from the bottom layer, and each layer uses a self-attention mechanism to capture the relationship between features, and finally obtains a globally fused feature representation at the top layer; the globally fused feature representation is post-processed, including dimensionality adjustment and normalization, to obtain the final fused feature representation. In the fusion process of each layer, residual connection and layer normalization techniques are applied to improve the fusion effect and training stability; in the fusion process, the representation of each modal feature is continuously updated and optimized.

[0051] This embodiment improves the effect of multimodal fusion through efficient feature fusion and dynamic weight allocation, and provides a more accurate and rich feature representation for controllable video generation. Using predefined tokenizers to convert feature matrices into feature sequences, and adding modality identifiers and position encoding information, not only retains the original information of different modal features, but also introduces the relative position relationship between modalities, providing important context information for subsequent attention mechanisms. The context-aware weight regulator (CAWR) is applied to construct a feature association graph using the modality correlation analysis (MCA) algorithm, and dynamically adjusts the weights of each modal feature through the topological structure analysis of the graph. It can adaptively adjust the importance of different modal features according to the characteristics of the current task and input data, so as to better utilize the most relevant information in the video generation process. For example, when generating a video dominated by human actions, the system may automatically increase the weights of posture graph and depth graph features, while when generating a video emphasizing the environmental atmosphere, the weights of audio and infrared image features may be increased. This dynamic weight allocation mechanism improves the semantic consistency and visual quality of the generated video. The weighted feature sequence is input into the Hierarchical Self-Organizing Fusion Network (HSOFN) for multi-level deep fusion. The HSOFN uses the dynamic topology reconstruction (DTR) algorithm to automatically adjust the network structure, so that the hierarchical structure and connection mode of the network can be dynamically adjusted according to the complexity of the input data and the task requirements, thereby achieving more flexible and efficient feature fusion. Through multi-layer self-attention mechanism, residual connection and layer normalization technology, the HSOFN can capture the complex interactions between different modal features and generate highly abstract and information-rich fusion feature representations. This deep fusion method not only improves the expressive power of features, but also enhances the model's ability to understand complex scenes, so that the generated video can better reflect the subtle details and high-level semantic information in multimodal inputs. Through dimensionality adjustment and normalization processing, the final fusion feature representation is obtained, which concentrates the essence of multiple modal information and provides high-quality input for the subsequent video generation process. This embodiment improves the utilization efficiency of multimodal information through feature fusion and dynamic weight allocation technology, so that the generated video is improved in terms of content richness, semantic consistency and visual authenticity, laying a solid foundation for achieving high-quality controllable video generation.

[0052] like Figure 3 As shown, according to one aspect of the present application, step S3 is further:

[0053] S31. Decompose the fused feature representation into predetermined time and space scales using a hierarchical spatiotemporal pyramid structure; apply an adaptive scale selection algorithm to the features of each time and space scale to dynamically select the processing scale; calculate the correlation between features based on the selected processing scale using a self-attention mechanism; based on the correlation, use a cross-scale attention mechanism to integrate information at different processing scales to obtain multi-scale processing results; fuse the multi-scale processing results to obtain an enhanced feature representation;

[0054] S32, reshape the enhanced feature representation into a three-dimensional tensor; based on the three-dimensional tensor, generate a spatial attention map through a convolution operation; apply the spatial attention map to the three-dimensional tensor to obtain a weighted feature of the space; use a residual connection to add the original feature of the three-dimensional tensor to the weighted feature to obtain a first fused feature; based on the first fused feature, apply a layer normalization operation to obtain a spatial enhanced feature representation; wherein two dimensions in the three-dimensional tensor represent spatial information and one dimension represents channel information; the spatial attention map reflects the importance of each spatial position in the feature map;

[0055] S33, reorganizing the spatial enhanced feature representation into a time series; based on the time series, using the self-attention mechanism to calculate the attention weights between different time steps; based on the attention weights, weighted summing the features of each time step to obtain the time attention feature; using the gating mechanism to fuse the spatial enhanced feature representation and the time attention feature to obtain a second fused feature; based on the second fused feature, applying a layer normalization operation to obtain the spatiotemporal enhanced feature representation;

[0056] S34, extracting text features from the multimodal feature matrix, and calculating cross-attention weights based on the spatiotemporal enhanced feature representation and the text features; performing weighted summation on the spatiotemporal enhanced feature representation based on the cross-attention weights to obtain semantic alignment features; using a gating mechanism to fuse the spatiotemporal enhanced feature representation and the semantic alignment feature to obtain a third fused feature; based on the third fused feature, applying a layer normalization operation to obtain a semantically enhanced video feature representation;

[0057] S35. Based on the semantically enhanced video feature representation, a mapping relationship from the feature space to the latent space is constructed using a latent space topology preservation algorithm; based on the mapping relationship, the semantically enhanced video feature representation is projected into the latent space; a latent state sequence of a predetermined number of time steps is generated in the latent space using a recursive neural network; a post-processing operation is applied to the latent state sequence, including noise injection and regularization, to obtain a multi-frame latent space representation;

[0058] S36. Based on multi-frame latent space representation, a progressive detail enhancement algorithm is used to generate a low-resolution initial video frame; the specific scale information in the multi-frame latent space representation is extracted and gradually improved through upsampling and detail enhancement modules to obtain an initial high-resolution video frame; an adversarial training strategy is used to introduce a discriminator network to improve the authenticity of the initial high-resolution video frame to obtain a high-resolution video frame; according to preset quality indicators and available computing resources, the final resolution of the high-resolution video frame is dynamically adjusted to obtain a final high-resolution video frame; the final high-resolution video frame is post-processed, including color balancing, denoising and sharpening, to obtain a final high-quality video sequence.

[0059] This embodiment realizes a high-quality and controllable video generation process, and improves the quality, coherence and semantic consistency of the generated video. The multi-scale spatiotemporal attention mechanism (MSAM) uses a hierarchical spatiotemporal pyramid (HSP) structure and an adaptive scale selection (ASS) algorithm to process multi-scale features, and can simultaneously process features at different time and space scales, thereby improving the model's ability to understand and generate video content. For example, when generating a video containing fast movements and slow background changes, MSAM can simultaneously capture the details of fast movements and the gradient effect of the background, thereby generating more realistic and coherent video content. Through the sequential processing of the spatial attention module, the temporal attention module, and the cross-attention module, the system can effectively process spatial information, capture temporal relationships, and align text semantics and video features. This multi-level attention mechanism improves the generated video in terms of spatial details, temporal coherence, and semantic consistency. For example, when generating a video based on a text description, the cross-attention module can ensure that the video content is highly consistent with the text description, achieving accurate text-to-video conversion. The dynamic latent space reconstructor (DLSR) uses the latent space topology preservation (LSTP) algorithm to construct the mapping relationship from feature space to latent space. It maintains the topological structure of the original data when generating multi-frame latent space representations, effectively retains the spatiotemporal relationship in the video sequence, and improves the coherence and realism of the generated video. For example, when generating a video containing a complex action sequence, DLSR can ensure the smooth transition and natural continuation of the action, avoiding the common frame skipping or incoherence problems. The adaptive resolution generation network (ARGN) uses the progressive detail enhancement (PDE) algorithm to gradually improve the resolution and detail level of the video frame. It can not only generate high-resolution video content, but also flexibly adjust the resolution of the output video according to different application requirements and computing resources, improving the quality and flexibility of video generation. For example, when generating product display videos that require high details, ARGN can gradually enhance the texture and details of the product, while when generating fast preview videos, it can choose a lower resolution to increase the generation speed. Combined with the adversarial training strategy, ARGN is able to generate more realistic and high-quality video sequences. Post-processing steps, including color balancing, denoising, and sharpening, further improve the visual quality of the video. This embodiment achieves a highly controllable and high-quality video generation process through multi-scale spatiotemporal processing, dynamic latent space reconstruction, and adaptive resolution generation. It can not only generate video content that is highly consistent with multimodal input, but also ensure that the video reaches a high level in terms of spatial details, temporal coherence, and overall quality, providing strong technical support for controllable video generation tasks in various application scenarios.

[0060] According to one aspect of the present application, in step S12, the audio data is resampled to generate a spectrogram and extract spectral coefficients. The preprocessed audio data may also be:

[0061] S121, performing denoising processing on the audio data to obtain a denoised signal; performing volume normalization on the denoised signal to obtain a normalized signal; performing wavelet packet decomposition on the normalized signal to obtain sub-signals of a predetermined frequency band;

[0062] S122, performing time-frequency analysis on each sub-signal to obtain the center frequency of the sub-signal; dynamically calculating the optimal time window length based on the center frequency; performing frame division and windowing operations on the sub-signal based on the optimal time window length, and calculating the Fourier transform of each frame to obtain a time-frequency representation; merging the time-frequency representations of all sub-signals to form a complete time-frequency diagram;

[0063] S123, obtaining the frequency distribution characteristics of the audio data, dynamically generating a nonlinear frequency compression function, and mapping the time-frequency graph to the Mel frequency scale based on the nonlinear frequency compression function to obtain a mapped Mel frequency spectrum graph; applying adaptive equalization processing to the mapped Mel frequency spectrum graph to obtain an equalized Mel frequency spectrum graph;

[0064] S124. Calculate the logarithm of the equalized Mel-frequency spectrum, perform discrete cosine transform based on the logarithm, extract Mel-frequency cepstral coefficients as static features; calculate the increment and acceleration coefficients based on the Mel-frequency cepstral coefficients as dynamic features; combine the static features and the dynamic features to form preprocessed audio data.

[0065] In one embodiment of the present application, audio signal data is received; the audio signal data is denoised, noise segments are identified by calculating short-time energy and zero-crossing rate, and an adaptive filter is applied to reduce noise; the volume of the denoised signal is normalized, the root mean square energy of the entire audio is calculated, and the signal amplitude is adjusted to a preset standard level; wavelet packet decomposition is performed on the normalized signal, and an appropriate wavelet basis function is selected to decompose the signal into sub-signals of multiple frequency bands; time-frequency analysis is performed on each sub-signal, and the optimal time window length is dynamically calculated according to the center frequency of the sub-signal, using a longer window for lower frequencies and a shorter window for higher frequencies; a counter is used to calculate the optimal time window length. The optimal window length is calculated, and the sub-signals are framed and windowed, and the Fourier transform of each frame is calculated to obtain the time-frequency representation; the time-frequency representations of all sub-signals are merged to form a complete time-frequency diagram; according to the frequency distribution characteristics of the audio content, a nonlinear frequency compression function is dynamically generated to map the time-frequency diagram to the Mel-frequency scale; adaptive equalization is applied to the mapped Mel-frequency spectrum to enhance important time-frequency patterns; the logarithm of the Mel-frequency spectrum is calculated, and then a discrete cosine transform is performed to extract the Mel-frequency cepstrum coefficients; the increment and acceleration coefficients are additionally calculated as dynamic features; the static and dynamic features are combined to form the final audio feature vector.

[0066] In another embodiment of the present application, the input audio signal is preprocessed, including denoising and volume normalization. Wavelet Packet Transform (WPT) is applied to perform multi-scale decomposition of the signal to obtain sub-signals of different frequency bands. Adaptive-Window STFT (AW-STFT) is applied to each sub-signal. The window size is dynamically adjusted according to the frequency characteristics of the sub-signal, with a longer window used for low frequencies and a shorter window used for high frequencies.

[0067] The result of AW-STFT is converted into a Mel-spectrogram. The Adaptive Nonlinear Frequency Compression (ANFC) algorithm is applied, which dynamically adjusts the nonlinear mapping function of the Mel scale according to the audio content. The Dynamic Feature Enhancement (DFE) algorithm is applied to the compressed spectrogram, which uses the attention mechanism to highlight important time-frequency patterns. Improved cepstrum analysis techniques, including fractional cepstrum and complex cepstrum analysis, are used to extract richer acoustic features. This embodiment can capture more detailed time-frequency features, adapt to different types of audio content, and improve the expressiveness of features.

[0068] This embodiment improves the quality and adaptability of audio feature extraction and provides high-quality audio input for controllable video generation of multimodal fusion. The adaptive multi-resolution time-frequency analysis (AMRTFA) system realizes the refined processing of audio signals. The AMRTFA system uses wavelet packet transform to decompose the signal at multiple scales, and then applies adaptive window short-time Fourier transform (AW-STFT) to each sub-signal. It can dynamically adjust the analysis window according to the characteristics of different frequency bands, so as to obtain the optimal resolution in both time and frequency domains. For example, for audio containing rapidly changing speech and stable background music, AMRTFA can capture the instantaneous features of speech and the continuous features of music at the same time, improving the accuracy and comprehensiveness of feature extraction. The application of adaptive nonlinear frequency compression (ANFC) algorithm further optimizes the spectral representation. It can dynamically adjust the nonlinear mapping function of Mel scale according to the audio content, so that the system can better adapt to different types of audio input, such as human voice, music, ambient sound, etc. This adaptive compression not only improves the expressiveness of features, but also enhances the adaptability of the system to different audio scenes. Through the dynamic feature enhancement (DFE) algorithm and improved cepstrum analysis techniques, including fractional cepstrum and complex cepstrum analysis, the system is able to capture richer acoustic features, such as timbre, pitch change, rhythm and other complex features. These advanced features are essential for generating videos with precise audio-visual synchronization. For example, when generating music videos or dialogue scenes, the system can accurately align visual content with audio rhythm or speech prosody. This embodiment not only improves the quality and robustness of audio feature extraction, but also enhances the adaptability of the system when processing various complex audio scenes. In the context of controllable video generation with multimodal fusion, this high-quality audio feature extraction provides strong support for subsequent modal fusion and video generation, so that the generated video can achieve a high degree of consistency and realism in both visual content and audio content, thereby improving the overall quality of the generated video and user experience. Applying the adaptive multi-resolution time-frequency analysis (AMRTFA) system to audio data can not only capture more detailed time-frequency features, but also adaptively process different types of audio content, thereby improving the expressiveness and robustness of audio features.

[0069] According to one aspect of the present application, in step S12, the infrared image data is subjected to heat distribution analysis, including hot spot detection and temperature gradient calculation, and the pre-processed infrared image data is further:

[0070] S12a, based on the infrared image data, applying adaptive histogram equalization to obtain a contrast-enhanced infrared image; based on the contrast-enhanced infrared image, using an edge-preserving filter to process to obtain a denoised infrared image; based on the gradient information and heat intensity distribution of the denoised infrared image, calculating a segmentation threshold; based on the segmentation threshold, dividing the denoised infrared image into predetermined heat regions;

[0071] S12b, calculate the temperature characteristics of the pixels in each heat area, including the average temperature, the highest temperature, the lowest temperature and the temperature variance; analyze the changes in the heat area between adjacent frames, calculate the rate of change of the heat area area and the rate of change of the average temperature, and obtain the rate of change characteristics of the heat area; apply the improved optical flow algorithm to each heat area to estimate the direction and speed characteristics of heat flow; based on the heat area, calculate the heat gradient at a predetermined spatial scale; based on the heat gradient, use convolution kernels of different sizes to extract local and global heat change characteristics; extract heat texture features for each heat area, and based on the heat texture features, calculate the local temperature difference pattern and the statistical characteristics of the heat distribution; use the sliding time window method to calculate the time evolution characteristics of the heat distribution in the heat area;

[0072] S12c, normalizing the temperature features, rate of change features, heat flow direction and speed features, local and global heat change features, heat texture features, statistical characteristics and time evolution features to obtain normalized features; based on the normalized features, using the principal component analysis method to reduce the feature dimension to obtain preprocessed infrared image data.

[0073] In one embodiment of the present application, infrared image sequence data is received; adaptive histogram equalization is applied to each frame of infrared image to enhance image contrast; an edge-preserving filter is used to reduce image noise while retaining thermal boundary information; a segmentation threshold is automatically calculated based on the gradient information and thermal intensity distribution of the image to divide the image into multiple thermal regions; for each thermal region, the average temperature, maximum temperature, minimum temperature and temperature variance of the pixels in the region are calculated; the thermal region changes between adjacent frames are analyzed to calculate the rate of change of the regional area and the rate of change of the average temperature; an improved optical flow algorithm is applied to each thermal region to estimate the direction and speed of heat flow; thermal gradients are calculated at multiple spatial scales, and convolution kernels of different sizes are used to extract local and global thermal change information; thermal texture features are extracted for each thermal region, and local temperature difference patterns and statistical characteristics of thermal distribution are calculated; a sliding time window method is used to capture the time evolution characteristics of thermal distribution and calculate the change trend of each statistical quantity over time; all extracted features are normalized to ensure that different types of features have similar numerical ranges; a principal component analysis method is used to reduce feature dimensions, remove redundant information, and obtain a final thermal distribution feature representation.

[0074] Receive input multimodal feature data and description information of the current task; calculate the information entropy of the input data and evaluate the complexity of the data; analyze the relevance of each feature to the current task and calculate the feature importance score; determine the number of new nodes to be generated based on the data complexity and feature importance; assign initial weights and bias values ​​to each new node, and the weights are initialized based on the importance of the input features; use heuristic search methods to explore the optimal connection method between new nodes and existing nodes; score the explored connection schemes, considering the balance between network performance improvement and computational complexity; select the optimal connection scheme based on the scoring results and implement the connection; evaluate the contribution of each layer of the network to the overall performance and calculate the hierarchical importance index; based on the hierarchical importance, automatically decide whether to merge adjacent redundant layers or split complex single layers; The most suitable activation function is selected for each node in the network, and the network performance is evaluated by iteratively testing different combinations of activation functions; the learning rate of each layer is dynamically adjusted according to the changes in the network structure to ensure the stability of the training process; the gradient flow in the network is monitored in real time, and residual connections or batch normalization parameters are introduced when necessary to improve gradient propagation; the performance of the reconstructed network is regularly evaluated, including accuracy, computing time and memory usage; if the performance improvement is not obvious or reaches the preset threshold, the reconstruction process is stopped; otherwise, return to the starting position of the step and continue to optimize the network structure; after the reconstruction process, the importance of each connection and node in the network is analyzed, and the elements that contribute less to the performance are removed; the knowledge of the optimized large dynamic network is transferred to a smaller and more efficient fixed structure network to obtain the final network structure.

[0075] According to one aspect of the present application, a multi-scale thermal dynamic feature extraction (MTDFE) system may be constructed when processing infrared image data in step S12. The specific steps are as follows:

[0076] S12a, receiving input infrared image sequence data; performing preliminary noise reduction processing on each frame of infrared image, using a median filter to remove salt and pepper noise; applying adaptive Gaussian filtering, dynamically adjusting the filter parameters according to the heat intensity of the local area, retaining the heat boundary information while removing noise; calculating the global heat histogram of the image, and analyzing the heat distribution characteristics; based on the heat histogram, adaptively adjusting the image contrast and enhancing the heat details; converting the enhanced image into a floating point representation to ensure the accuracy of subsequent processing.

[0077] S12b, perform adaptive regional segmentation on the preprocessed infrared image; calculate the thermal gradient map of the image to identify areas with drastic thermal changes; use a multi-scale edge detection algorithm to detect thermal boundaries at different scales; combine thermal gradient and edge information to use a regional growing algorithm to preliminarily divide thermal regions; apply morphological operations to the results of the preliminary division to remove small isolated regions and smooth regional boundaries; calculate the average thermal intensity and variance of each region as regional features; use a clustering algorithm to merge similar regions to obtain the final thermal region division result.

[0078] S12c. Perform heat flow analysis on each divided heat area; calculate the heat changes in the corresponding areas between adjacent frames to obtain the rate of change of heat over time; use the improved optical flow algorithm to estimate the movement direction and speed of heat in the area; calculate the heat gradient at the boundary of the area, and analyze the diffusion and contraction trends of heat; fit the heat conduction model to each area, estimate the heat source position and thermal conductivity coefficient; based on the heat conduction model, predict the change of heat distribution in the short term; compare the predicted results with the actual observations, adjust the heat conduction model parameters, and improve the model accuracy.

[0079] S12d. Calculate thermal features at multiple spatial scales; use Gaussian pyramid to perform multi-scale decomposition of thermal images; calculate local thermal statistical features at each scale, including mean, variance, skewness and kurtosis; apply local binary pattern (LBP) algorithm to extract thermal texture features; calculate thermal co-occurrence matrix to extract spatial correlation characteristics of thermal distribution; use wavelet transform to analyze the distribution characteristics of heat at different frequencies; combine multi-scale features to construct thermal feature descriptors.

[0080] S12e, extract heat time series features; for each heat area, construct a time series to record the changes in the average heat, maximum heat and area of ​​the area; use a sliding time window to calculate the short-term heat change trend; apply an autoregressive model to predict the periodic changes in heat; calculate the spectral characteristics of heat changes and analyze the frequency components of heat changes; extract the statistical characteristics of heat changes, such as the mean and variance of the change rate; detect heat mutation events and record the time point and amplitude of the mutation.

[0081] S12f, integrate and optimize the extracted features; combine spatial features and temporal features to form an initial feature set; calculate the correlation matrix between features and identify redundant features; use the principal component analysis (PCA) method to reduce the feature dimension; apply feature selection algorithms, such as recursive feature elimination, to select the most relevant feature subset; normalize the selected features to make different types of features comparable; construct a feature importance ranking to provide a reference for subsequent analysis; finally, combine the optimized features into the final heat distribution feature vector.

[0082] In one embodiment of the present application, the input infrared image sequence is preprocessed, including denoising and contrast enhancement. An adaptive region segmentation algorithm (ARS) is applied to automatically divide the image region according to the complexity of the heat distribution. An improved thermal flow analysis (ETFA) algorithm is applied to each region, which combines the optical flow method and the heat conduction model to capture the dynamic changes of heat. The multi-scale thermal gradient calculation (MTGC) method is used to calculate the thermal gradient at different spatial scales. The thermal pattern recognition (TPR) algorithm is applied, which uses an improved local binary pattern (LBP) and thermal texture descriptor (TTD) to capture the local heat distribution characteristics. A thermal temporal feature extraction (TTFE) module is developed, and a long short-term memory network (LSTM) is used to capture the time evolution characteristics of the heat distribution. All extracted features are integrated through a feature fusion network (FFN) to obtain the final heat distribution feature representation. This embodiment can fully capture the static and dynamic heat distribution features in infrared images and improve the analysis capability of complex thermal scenes.

[0083] This embodiment improves the system's ability to process heat information through infrared heat distribution analysis technology, and provides rich and accurate heat feature input for controllable video generation with multimodal fusion. The multi-scale heat dynamic feature extraction (MTDFE) system can fully capture static and dynamic heat distribution features. Through the adaptive region segmentation algorithm (ARS), the system can automatically divide the image area according to the complexity of the heat distribution, which is particularly suitable for processing complex thermal scenes, such as accurately capturing the thermal characteristics of each area when generating videos containing multiple heat sources or uneven heat distribution. The enhanced heat flow analysis (ETFA) algorithm combines the optical flow method and the heat conduction model, enabling the system to accurately track and predict the dynamic changes of heat. This is crucial for generating videos involving heat changes. For example, when simulating fire scenes or showing heat changes in industrial processes, ETFA can provide realistic heat flow effects. The multi-scale heat gradient calculation (MTGC) method calculates heat gradients at different spatial scales. This multi-scale analysis enables the system to simultaneously capture large-scale heat distribution trends and small-scale heat detail changes. The introduction of the thermal pattern recognition (TPR) algorithm further enhances the system's ability to identify local heat distribution features, which is particularly important for generating videos containing complex thermal textures (such as infrared night vision scenes). The application of the thermal time series feature extraction (TTFE) module enables the system to effectively capture the temporal evolution characteristics of heat distribution, which provides key support for generating long-term series of heat change videos (such as weather forecast scenes). All extracted features are integrated through the feature fusion network (FFN) to obtain a comprehensive and accurate heat distribution feature representation. This high-quality thermal feature representation enhances the system's expressiveness when generating videos involving temperature and heat changes. For example, when generating videos that simulate natural phenomena (such as volcanic eruptions, day and night temperature differences in deserts) or industrial processes (such as metal smelting, chemical reactions), the system is able to provide extremely realistic and detailed heat change effects. In addition, this accurate thermal feature extraction also provides valuable input for multimodal fusion, so that the generated videos can achieve a high degree of consistency and realism in visual, audio and thermal information. This embodiment improves the system's ability to process and generate videos involving complex heat distribution, provides strong support for multimodal fusion controllable video generation tasks, and improves the quality and practicality of generated videos in specific application fields (such as thermal imaging, environmental monitoring, industrial process visualization, etc.).

[0084] According to one aspect of the present application, in step S12, the edge operator graph data is subjected to Gaussian smoothing to remove noise, and the edge detection operator is applied to enhance the edge features, so that the preprocessed edge operator graph data is further:

[0085] S12i, performing preliminary analysis on the edge operator graph data, calculating the cumulative distribution function of the image; performing histogram equalization based on the cumulative distribution function, and calculating the global noise level; adaptively determining the parameters of the Gaussian filter based on the global noise level, including the kernel size and the standard deviation; constructing a two-dimensional Gaussian kernel based on the determined parameters, performing a Gaussian filtering operation on the edge operator graph data, and obtaining a preliminary denoised image;

[0086] S12ii, based on the image with preliminary denoising, using the finite difference method, calculating the gradient distribution; based on the gradient distribution, adaptively selecting the threshold parameter of edge detection; based on the threshold parameter, selecting the Sobel operator or the Canny operator to perform edge detection to obtain an edge map; post-processing the edge map, including analyzing connectivity, removing noise, connecting edges, thinning edges and analyzing directions to obtain a processed edge map;

[0087] S12iii. Based on the processed edge map, calculate global statistical features, including edge density, average length and direction distribution; use Hough transform to detect straight and circular edges in the processed edge map, and record their parameters; calculate curvature statistics of the processed edge map, including average curvature and curvature variance; based on the processed edge map, repeat edge detection at different scales to construct multi-scale edge features; analyze the texture features of the processed edge map, calculate the edge co-occurrence matrix and extract related statistics;

[0088] S12iv. Based on global statistical features, parameters of straight and circular edges, curvature statistical information, multi-scale edge features and edge co-occurrence matrix, generate initial feature vectors; calculate the correlation between the initial feature vectors, identify and remove highly correlated redundant features, and obtain filtered features; based on the filtered features, apply principal component analysis to reduce feature dimensions and obtain reduced-dimensional features; normalize the reduced-dimensional features to obtain normalized features;

[0089] S12v. Calculate the importance score of each normalized feature, select the Top-K features from the normalized features according to the importance score, and form the final edge feature descriptor as the preprocessed edge operator graph data.

[0090] In one embodiment of the present application, input edge operator graph data is received; a preliminary analysis is performed on the image to calculate the grayscale histogram and cumulative distribution function of the image; based on the cumulative distribution function, histogram equalization is performed to improve the contrast of the image; the global noise level of the image is calculated to estimate the signal-to-noise ratio of the image; based on the estimated noise level, the parameters of the Gaussian filter are adaptively determined, including the kernel size and the standard deviation; a two-dimensional Gaussian kernel is constructed using the determined parameters; a Gaussian filtering operation is performed on the image, and the image boundary is processed using an edge filling method to obtain a preliminarily denoised image.

[0091] Prepare edge detection for the preliminary denoised image. Calculate the gradient of the image in the x and y directions, and use the finite difference method to approximate the gradient calculation; estimate the overall gradient distribution of the image based on the calculated gradient; analyze the gradient distribution and adaptively select the threshold parameters for edge detection; if the Sobel operator is selected, construct a 3x3 Sobel operator matrix for the x and y directions respectively; if the Canny operator is selected, additionally calculate the gradient direction and determine the high and low thresholds, usually the high threshold is 2 to 3 times the low threshold.

[0092] Apply edge detection operators to the prepared image data. For example, if the Sobel operator is used, apply the Sobel kernel to perform convolution operations in the x and y directions respectively to obtain Gx and Gy; calculate the gradient magnitude G = sqrt(Gx 2 +Gy 2 ) to obtain the edge intensity map; if the Canny operator is used, first apply Gaussian filtering, then calculate the gradient amplitude and direction; perform non-maximum suppression to retain the local maximum gradient value; apply the double threshold method to determine the strong edge and the weak edge; use the hysteresis threshold method to connect the edges to obtain the final edge map.

[0093] Post-process the obtained edge map. Analyze the connectivity of the edge and mark the independent edge segments; calculate the length and curvature of each edge segment; remove the edge segments that are too short or have abnormal curvature to reduce the impact of noise; connect the discontinuous edges and use local direction information to assist in connection judgment; refine the edges to ensure that the edge width is a single pixel; calculate the direction histogram of the edge and analyze the main direction distribution of the edge.

[0094] Calculate the global statistical features of the processed edge map, including edge density, average length and direction distribution; use Hough transform to detect straight and circular edges in the image and record their parameters; calculate the curvature statistics of the edge, including the average curvature and curvature variance; repeat the edge detection process at different scales of the image to construct multi-scale edge features; analyze the texture features of the edge, calculate the edge co-occurrence matrix and extract related statistics.

[0095] Integrate and optimize edge features. Combine the extracted edge features into an initial feature vector; calculate the correlation between features, identify and remove highly correlated redundant features; apply principal component analysis (PCA) to reduce feature dimensions and retain the principal component with the highest proportion of explained variance; normalize the reduced-dimensional features to ensure that different types of features are comparable; calculate the importance score of each feature, and use a tree-based feature importance evaluation method; select the Top-K most important features based on the importance score; reorganize the selected features to form the final edge feature descriptor.

[0096] Save the final edge feature descriptor in a structured data format, such as JSON or XML; generate visualization results of edge detection, including the original image, enhanced edge map, and multi-scale edge representation; calculate and record performance indicators of the entire processing process, such as processing time and memory usage; generate a processing report that records the parameter settings and intermediate results of each step in detail; package all output results and prepare them for transmission to subsequent processing modules or storage systems.

[0097] This embodiment improves the system's processing ability for image structure and contour information through edge detection and feature extraction technology, and provides high-quality edge feature input for controllable video generation of multimodal fusion. The application of adaptive Gaussian filtering improves the quality of the initial image. By dynamically adjusting the filter parameters, the system can retain important edge information while denoising. It is particularly suitable for processing input images of different qualities and types, such as when generating videos containing complex textures and details, it can effectively suppress noise without losing key structural information. The adaptive threshold selection technology introduced in the edge detection preparation stage enables the system to automatically determine the optimal detection parameters based on the overall gradient distribution of the image. This adaptive method improves the robustness of edge detection and enables the system to adapt to various images of different contrast and complexity. When applying the edge detection operator, the advantages of Sobel and Canny operators are combined, and more accurate and coherent edge detection is achieved by comprehensively utilizing gradient amplitude and direction information. This is crucial for generating video content with clear contours and structures, such as when generating videos of buildings, mechanical parts or human contours, it can provide extremely high edge accuracy and continuity. The edge connection and refinement techniques introduced in the post-processing stage further improve the edge quality. By intelligently connecting disconnected edge segments and removing noisy edges, the system is able to generate clearer and more coherent edge maps. Multi-scale edge feature extraction enables the system to capture edge information at different scales, thereby providing a richer and more comprehensive structural description, which is particularly suitable for generating videos containing multiple levels of details, such as scene transitions from distant views to close-ups. By integrating and optimizing edge features, including dimensionality reduction using principal component analysis and feature importance evaluation, the system obtains a highly refined and informative edge feature descriptor. This high-quality edge feature provides strong structural and shape guidance for the subsequent video generation process. Under the controllable video generation content of multimodal fusion, these high-quality edge features enable the system to generate video content with precise contours and rich details. For example, when generating sketch-based animation videos, the system can accurately transform simple line sketches into animated scenes with rich details and dynamic changes. At the same time, when generating video sequences that need to maintain the consistency of object shapes, such as animations of human movements or object deformations, these precise edge features can ensure the continuity and authenticity of object shapes. This embodiment improves the system's ability to process and generate videos with complex structures and fine contours, provides key visual structure information for the task of controlling video generation with multimodal fusion, and improves the visual quality, realism, and detail richness of the generated video.

[0098] According to one aspect of the present application, the dynamic topology reconstruction (DTR) algorithm in step S23 is further:

[0099] S231. Initialize the network structure. Create an initial network with a minimum number of nodes and basic connections; assign random initial weights and biases to each node; define a set of available activation functions, including ReLU, Sigmoid, Tanh, etc.; set network evaluation indicators, including accuracy, computational complexity, and memory usage; initialize a performance recorder to track changes in network performance.

[0100] S232. Analyze input data and task requirements. Calculate the statistical properties of the input data, such as mean, variance, and distribution type; estimate the intrinsic dimensions of the data, using principal component analysis or manifold learning methods; evaluate the complexity of the task, considering factors such as the number of classifications and the range of regression targets; analyze the correlation between each input feature and the task target, using indicators such as mutual information or correlation coefficient; based on the data analysis results, preliminarily estimate the required network capacity and complexity.

[0101] S233, Generate and optimize network nodes. Calculate the number of new nodes that need to be added based on data complexity and task requirements; assign initial weights to each new node, and initialize the weight values ​​based on the importance of the input features; use heuristic search algorithms to explore the optimal connection between new nodes and existing nodes; evaluate the performance of different connection schemes, considering the balance between accuracy improvement and computational overhead; select the optimal connection scheme and implement new connections in the network; perform local training on newly added nodes and connections to adapt to the network structure.

[0102] S234. Adjust the hierarchical structure of the network. Evaluate the contribution of each layer of the network to the overall performance and calculate the hierarchical importance index; identify redundant layers and information bottleneck layers in the network; compress or merge redundant layers to reduce unnecessary calculations; expand or decompose information bottleneck layers to improve information transmission efficiency; rebalance the number of nodes in each layer to ensure the balance of the network structure; adjust the connections between layers and add jump connections when necessary to improve gradient flow.

[0103] S235. Optimize the activation function of nodes. Select the most suitable activation function for each node in the network; evaluate the network performance by iteratively testing different combinations of activation functions; consider using mixed activation functions to combine different types of activation functions in the same layer; for key nodes, consider using activation functions with learnable parameters, such as PReLU; adjust the activation function selection strategy according to the position and function of the node in the network; evaluate the impact of different activation functions on the interpretability of the network, and seek a balance between performance and interpretability.

[0104] S236. Adaptively adjust the learning process. Dynamically adjust the learning rate of each layer according to changes in the network structure; monitor the gradient flow in the network in real time to detect gradient vanishing or gradient exploding problems; introduce gradient clipping or gradient normalization technology when necessary to stabilize the training process; adjust the batch size and number of training iterations to adapt to the new network structure; implement dynamic regularization intensity adjustment and adjust the L1 / L2 regularization parameters according to changes in network complexity; introduce a learning rate scheduler to automatically adjust the learning rate according to the progress of training.

[0105] S237, evaluate and iteratively optimize network performance. Regularly evaluate the performance of the reconstructed network, including accuracy, computing time, and memory usage; compare the current performance with the best historical performance to determine whether the stopping condition for performance improvement has been reached; if the performance continues to improve, return to step S232 to continue the optimization process; if the performance improvement is not obvious or reaches the preset threshold, enter the network solidification stage; during the iteration process, maintain an elite network structure pool and save several network versions with the best performance; if necessary, you can fall back to the previous excellent structure to avoid falling into the local optimum.

[0106] S238, network structure solidification and knowledge extraction. Analyze the importance of each connection and node in the network, use pruning technology to remove elements with less contribution; retrain the simplified network to recover the performance loss caused by pruning; use knowledge distillation technology to transfer the knowledge of a large dynamic network to a smaller and more efficient fixed structure network; optimize the numerical accuracy of the final network structure, consider quantization technology to reduce the model size; generate a visual representation of the network structure for easy analysis and interpretation; output the final network structure description, including information such as the number of layers, the number of nodes per layer, the connection method, and the activation function used.

[0107] In one embodiment of the present application, a dynamic topology reconstruction (DTR) algorithm is used to automatically adjust the hierarchical structure and connection mode of the network according to the complexity of the input data and the task requirements. Create a basic network structure containing a predefined minimum number of nodes and connections. Perform input analysis: use information entropy to calculate the complexity of the input data; apply Task Relevance Analysis (TRA) to evaluate the importance of each feature to the current task. Dynamically generate new nodes based on data complexity and feature importance; use an adaptive node generation threshold (ANGT), which is dynamically adjusted according to network performance and resource constraints. Apply a heuristic connection search (HCS) algorithm that uses a hybrid method of genetic algorithm and simulated annealing to search for the optimal connection method. Introduce a connection sparsity constraint (CSC) to balance network complexity and performance.

[0108] Adjust the hierarchical structure: Use the Layer Importance Evaluation (LIE) algorithm to evaluate the contribution of each layer to network performance; apply the Adaptive Layer Compression (ALC) technology to merge redundant layers or split complex layers. Use the Activation Function Search (AFS) algorithm to dynamically select the most suitable activation function for each node. The activation function search algorithm combines reinforcement learning and Bayesian optimization to select the optimal function from a predefined activation function library. Construct a dynamic learning rate adjustment (DLRA) mechanism to automatically adjust the learning rate of each layer according to changes in the network structure; introduce a gradient flow regulator (GFR) to ensure that the gradient can be effectively propagated in a dynamically changing network. Regularly evaluate network performance, including accuracy, computational complexity, and memory usage. If the performance improvement is not significant or reaches a preset threshold, stop the reconstruction process; otherwise, re-analyze the input and continue iterating. After the reconstruction process, apply the Structure Pruning (SP) algorithm to remove unimportant connections and nodes. Using the knowledge distillation (KD) technique, the knowledge of a large dynamic network is transferred to a smaller and more efficient fixed structure network. The dynamic topology reconstruction algorithm can better adapt to different input data and task requirements by dynamically adjusting the network structure, thereby improving the flexibility and performance of the network.

[0109] This embodiment improves the adaptability and performance of the network structure and provides a highly flexible and efficient deep learning framework for controllable video generation of multimodal fusion. The random weight allocation and diversified activation function set in the initialization phase provide a good starting point for the network and increase the possibility of the network exploring the optimal structure. This initial diversity is particularly important for processing complex multimodal inputs, enabling the network to capture the relationship between different modalities from multiple perspectives. In the in-depth analysis of input data and task requirements, by estimating the intrinsic dimensions of the data and evaluating the relevance of features to tasks, the system can more accurately determine the required network capacity and complexity. This data-driven approach enables the network structure to be dynamically adjusted according to actual input and tasks, avoiding the limitations of traditional fixed-structure networks. In the node generation and optimization stage, the application of the heuristic connection search (HCS) algorithm improves the efficiency of network structure search. By combining the advantages of genetic algorithms and simulated annealing, HCS can quickly find the approximate optimal connection mode in a large structure space. This is particularly important for processing high-dimensional, multimodal input data, enabling the network to effectively model complex interactions between different modalities. The hierarchical importance evaluation (LIE) algorithm and adaptive layer compression (ALC) technology introduced in the hierarchical structure adjustment enable the network to dynamically adjust its depth and width. This adaptive ability is crucial for handling video generation tasks of different complexity. For example, a shallower network can be used when generating simple action sequences, while it can automatically expand to a deeper structure when generating complex scene transitions. Through the activation function search (AFS) algorithm, the system is able to select the most suitable activation function for each node. This refined adjustment improves the network's expressiveness and training efficiency, especially when processing multimodal data, different activation functions can better capture various nonlinear relationships. Backpropagation adaptation techniques, such as dynamic learning rate adjustment (DLRA) and gradient flow regulator (GFR), further improve the network's training stability and convergence speed. This is particularly important for training large-scale, multimodal video generation models, which can effectively prevent the problem of gradient disappearance or explosion. The pruning and knowledge distillation techniques in the structure solidification stage enable complex dynamic networks to be transformed into more efficient fixed-structure networks, improving the model's reasoning speed and deployment efficiency. In the context of controllable video generation with multimodal fusion, this highly adaptive and efficient network structure enables the system to better process and fuse complex features from different modalities, thereby generating higher quality and more diverse video content. For example, when generating video sequences that require fine control, such as generating animations with specific styles and content based on text descriptions, the system can automatically adjust the network structure to best map text semantics to visual features. At the same time, for application scenarios that require real-time generation, such as interactive virtual reality content, the optimized fixed-structure network can provide fast reasoning capabilities.This embodiment improves the system's ability and efficiency in processing complex multimodal inputs and generating high-quality controllable videos, and provides a powerful and flexible deep learning framework for multimodal fusion controllable video generation tasks.

[0110] According to one aspect of the present application, in step S35, based on semantically enhanced video feature representation, a latent space topology preservation algorithm is used to construct a mapping relationship from feature space to latent space, which is further:

[0111] S35a, based on semantic enhanced video feature representation, calculate the Euclidean distance matrix between data samples; based on the Euclidean distance matrix, construct a K nearest neighbor graph; calculate the Laplacian matrix of the K nearest neighbor graph; perform eigendecomposition on the Laplacian matrix to obtain eigenvalues ​​and eigenvectors; based on the attenuation of the eigenvalues, estimate the intrinsic dimension of the data;

[0112] S35b, based on the intrinsic dimension, initialize the dimension and structure of the latent space to obtain the initial latent space coordinates; based on the initial latent space coordinates and the semantically enhanced video feature representation, construct a cost function, including a local distance preservation term and a global structure preservation term; use the gradient descent method to iteratively optimize the cost function to obtain the optimized latent space coordinates;

[0113] S35c. Based on the optimized latent space coordinates, a mapping function from the high-dimensional feature space to the latent space is constructed, and a radial basis function network is used as the basic form of the mapping function; the center point and width parameters of the radial basis function network are adjusted to obtain the adjusted mapping function; in the adjusted mapping function, a regularization term is introduced to obtain a simplified mapping function; an alternating optimization strategy is used to optimize the simplified mapping function to obtain an optimized mapping relationship.

[0114] In one embodiment of the present application, an input semantically enhanced video feature representation is received; a Euclidean distance matrix between data samples is calculated; based on the distance matrix, a K nearest neighbor graph is constructed, wherein the K value is determined by cross-validation; the Laplacian matrix of the K nearest neighbor graph is calculated; the Laplacian matrix is ​​eigen-decomposed to obtain its eigenvalues ​​and eigenvectors; and based on the attenuation of the eigenvalues, the intrinsic dimension of the data is estimated.

[0115] Based on the estimated intrinsic dimension, the dimension and structure of the latent space are initialized. A set of anchor points are randomly selected in the high-dimensional feature space, and the number is related to the estimated intrinsic dimension; the distance from each data point to these anchor points is calculated; the anchor points are projected into the low-dimensional latent space using the multidimensional scaling (MDS) algorithm; based on the projection results of the anchor points, the coordinates of other data points in the latent space are initialized.

[0116] Optimize the data representation in the latent space. Construct a cost function that includes a local distance-preserving term and a global structure-preserving term; the local distance-preserving term ensures that neighboring points remain close in the latent space; the global structure-preserving term maintains the overall structure by maintaining the shortest path distance; iteratively optimize the cost function using gradient descent; in each iteration, update the coordinates of the data points in the latent space; calculate the reconstruction error and topology-preserving metric of the current latent space representation.

[0117] Adaptively adjust the mapping function. Based on the current latent space representation, construct a mapping function from the high-dimensional feature space to the latent space; use the radial basis function (RBF) network as the basic form of the mapping function; adjust the center point and width parameters of the RBF network to minimize the mapping error; introduce regularization terms to prevent the mapping function from being overly complicated; use an alternating optimization strategy to optimize the latent space representation while fixing the mapping function, and then fix the latent space representation to optimize the mapping function.

[0118] Detect local nonlinear regions in the feature space; for highly nonlinear local regions, increase the complexity of the local mapping function; introduce the idea of ​​local linear embedding (LLE) and use linear mapping to approximate nonlinear relationships in local regions; smoothly transition the mapping results of different local regions to ensure global consistency.

[0119] Construct an inverse mapping function. Based on the optimized latent space representation, construct an inverse mapping function from the latent space back to the high-dimensional feature space; use a multi-layer perceptron (MLP) as the basic form of the inverse mapping function; train the inverse mapping function by minimizing the reconstruction error; introduce the idea of ​​adversarial training and use a discriminator network to evaluate the authenticity of the reconstructed data; iteratively optimize the inverse mapping function until the reconstruction quality reaches the preset threshold.

[0120] Evaluate and fine-tune the mapping quality. Use the holdout method to divide the dataset into a training set and a test set; evaluate the generalization ability of the mapping function on the test set; calculate the performance of the latent space representation on different tasks (such as classification, clustering); analyze abnormal points or outliers in the latent space and handle them if necessary; fine-tune the parameters and structure of the mapping function based on the evaluation results.

[0121] Generate the final latent space representation. Convert the entire data set to the latent space through the optimized mapping function; calculate the statistical properties of the data points in the latent space, such as the mean, variance, and covariance matrix; perform normalization to ensure that the dimensions of the latent space representation have similar scales; generate visualization results of the latent space representation for intuitive understanding of the data structure; package the final latent space representation, mapping function parameters, and evaluation indicators as output results.

[0122] The latent space representation is divided into multiple time steps, each time step corresponds to a video frame; the difference in latent space representation between adjacent time steps is calculated to analyze the temporal continuity; based on the temporal continuity analysis, the latent space representation is smoothed to ensure the smoothness of the generated video; timestamp information is added to the latent space representation of each time step; the processed multi-frame latent space representation is serialized and prepared for transmission to the subsequent video generation module.

[0123] This embodiment improves the quality and efficiency of the mapping from feature space to latent space through the latent space topology preservation (LSTP) algorithm, and provides a highly structured and information-rich latent space representation for controllable video generation of multimodal fusion. By constructing a K-nearest neighbor graph and calculating the Laplace matrix, the system can effectively capture local structural information in high-dimensional feature space, which is particularly suitable for processing complex multimodal data, such as when fusing text, image and audio features, it can retain the intrinsic relationship between each modality. The method of estimating the intrinsic dimension of data based on eigenvalue decay provides important guidance for the subsequent latent space construction, ensuring that the generated latent space can effectively represent the essential characteristics of the original data. When initializing the latent space structure, the method of projecting anchor points using the multidimensional scaling (MDS) algorithm provides a good starting point for the subsequent optimization process, which is particularly conducive to processing highly nonlinear data distributions, such as when generating videos containing complex action sequences or scene transitions, the topological structure of the original data can be better preserved. In the process of optimizing the latent space representation, the cost function of the local distance preservation and global structure preservation terms is introduced. This dual constraint ensures that both the neighbor relationship and the overall data structure are preserved during the dimensionality reduction process. For video generation tasks, this means being able to capture long-term temporal dependencies while maintaining inter-frame continuity. The step of adaptively adjusting the mapping function, especially using the radial basis function (RBF) network as the mapping function, improves the accuracy of the mapping from the high-dimensional feature space to the low-dimensional latent space. This flexible mapping method enables the system to handle a variety of complex nonlinear relationships, which is particularly important for generating complex video content containing multi-modal information (such as vision, audio, and text). Dealing with the local characteristics of the manifold, by increasing the mapping complexity of the local nonlinear region and introducing the idea of ​​local linear embedding (LLE), the system is able to more accurately capture the local structure of the data distribution. This is particularly effective for generating video sequences containing subtle changes or fast transitions, such as subtle changes in facial expressions or fast movements of objects. The step of constructing the inverse mapping function, especially the introduction of the idea of ​​adversarial training, improves the quality of the reconstruction from the latent space representation to the original feature space. This ensures that the generated video content can accurately reflect the characteristics of the original multimodal input, improving the authenticity and detail richness of the generated video.

[0124] The process of evaluating and fine-tuning the mapping quality, including evaluating the performance of the latent space representation on different tasks, enhances the versatility and robustness of the system. This makes the generated videos excellent not only in visual quality, but also in semantic consistency and task relevance. The step of dividing the latent space representation into multiple time steps and performing temporal continuity analysis provides key support for generating coherent and smooth video sequences. By smoothing the latent space representation, the system is able to generate more natural and coherent video content, avoiding common frame skipping or discontinuity problems. This high-quality latent space representation method brings advantages in the context of controllable video generation content with multimodal fusion. It enables the system to more effectively fuse information from different modalities and generate highly consistent and expressive video content. For example, when generating animated videos based on text descriptions and audio inputs, the system can accurately align text semantics, audio features, and visual elements in the latent space to create highly coordinated and creative video content. At the same time, it also enhances the control over the generation process, allowing users to precisely adjust various aspects of the video, such as style, content, motion, etc., by manipulating the latent space representation. This embodiment improves the system's ability to process complex multimodal inputs and generate high-quality, highly controllable videos, provides a powerful and flexible technical framework for multimodal fusion controllable video generation tasks, and expands the application scope and effects of video generation technology.

[0125] According to one aspect of the present application, a controllable video generation system based on multimodal fusion includes:

[0126] at least one processor; and,

[0127] a memory communicatively connected to at least one of the processors; wherein,

[0128] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the controllable video generation method based on multimodal fusion described in any one of the above embodiments.

[0129] In another embodiment of the present application, the video generation model based on multimodal fusion can accept inputs of multiple modes, including text, RGB images, audio, infrared images, depth images, mask images, edge operator images, point cloud images, and posture images. Information of different modes is processed by a specially designed encoder to extract useful features. These features can fully reflect the multidimensional information of the input modality and provide rich basic data for subsequent video generation.

[0130] For example, Pengi et al. used the CLAP model as their audio encoder. The CLAP model can efficiently extract feature information from audio and provide important sound information reference for multimodal video generation. In this way, audio features can be fused with other modal features to generate more realistic and expressive video content.

[0131] In addition, the ImageBind-LLM model uses the ImageBind encoder, which has multi-modal encoding capabilities and supports encoding of RGB images, text, audio, depth maps, thermal imaging, and IMU data. With the powerful performance of the ImageBind encoder, ImageBind-LLM can process and respond to inputs of multiple modalities, thereby achieving highly flexible and accurate video generation.

[0132] The multimodal features processed by their respective encoders are processed by a tokenizer to concatenate the features of different modalities into a unified feature sequence. This step ensures that information from different modalities can interact and merge in the same sequence.

[0133] In the process of feature fusion, a dynamic weight allocation mechanism is used to automatically adjust the weight of each modal feature according to the importance of different modal features in a specific task. The self-attention mechanism is used to weight the features of each modality in the feature sequence. Through the learned weight parameters, it is ensured that important modal features occupy a larger proportion in the fusion process. Dynamic weight allocation not only improves the accuracy of feature fusion, but also enhances the adaptability and flexibility of the model to different modal inputs.

[0134] After the initial fusion of the feature sequence, a multi-level deep fusion process is performed. Each level of fusion is achieved by stacking multiple layers of Transformer modules. In each layer of Transformer modules, the multi-head self-attention mechanism is used to deeply interact and fuse the features of each modality in the feature sequence. The multi-head self-attention mechanism can capture the complex correlation between different modal features and improve the fusion effect. Multi-level fusion can not only fully explore the potential information of different modal features, but also optimize the feature sequence layer by layer to generate more refined and coherent video content.

[0135] When only a single modality is provided as input, such as a text description or a single image, the model can generate a corresponding video based on the information in that modality. By deeply mining and understanding the features of a single modality, the generated video has achieved a high level of content consistency and visual effects.

[0136] like Figure 4As shown in the figure, X represents the multi-modality of the input. The feature vectors are obtained by encoding the corresponding ImageBind encoders (these feature vectors can capture the key information of the corresponding modality and mathematically express the semantic content of the original data). Then, the features of different modalities are spliced ​​into a unified feature sequence through the tokenizers. This step ensures that the information of different modalities can interact and merge in the same sequence. This embodiment uses a variant of DiT (Diffusion Transformer), SiDiT (Serial Diffusion Transformer). The structure of each layer is as follows: Figure 5 As shown. It superimposes a one-dimensional temporal attention module on a two-dimensional spatial attention module in a serial manner to model temporal relationships. After the temporal attention module, the cross-attention module is used to align the semantics of the text. Compared with the full attention mechanism, such a structure reduces the training and inference overhead. Specifically: Spatial attention module: responsible for processing the spatial features of video frames and extracting the key visual information of each frame. Temporal attention module: based on the spatial features, further processes the temporal relationship between video frames to capture actions and dynamic changes. Cross-attention module: used to align the semantic information extracted by the text encoder with the video features to achieve text-guided video generation. After passing through the SiDiT module, the latent space representation Z' of multiple frames can be obtained, and finally the video is restored using a unified VAE decoder Decode.

[0137] This embodiment has a wide range of beneficial effects in practical applications, can improve the quality and controllability of video generation, and meet diverse application needs. By introducing multimodal input processing, it is possible to receive and process data in multiple modes such as text, RGB images, audio, infrared images, depth images, mask images, edge operator images, point cloud images, and posture images. The specially designed encoder can efficiently extract the key features of each modality and splice these features into a unified feature sequence through the tokenizer. The dynamic weight allocation mechanism and multi-level deep fusion technology further optimize the interaction and fusion of features to ensure that the generated video reaches a higher level in content consistency, detail richness, and overall coherence.

[0138] The video generation model of this embodiment is highly adaptable and flexible; through multimodal input processing, users can choose a single modality or multiple modalities for input according to different needs. The model can generate video content that meets the needs based on the characteristics of the input modality. For example, when only text descriptions are provided, the model can generate a video that matches the text content; when images and audio are provided, the model can generate a video that combines visual and auditory information. This flexible input method enables the model to adapt to a variety of scenarios and needs in practical applications.

[0139] This embodiment has made an important breakthrough in the field of video generation by introducing multimodal fusion technology. Methods such as dynamic weight allocation and multi-level fusion not only improve the accuracy of feature fusion, but also enhance the adaptability and flexibility of the model. Compared with the existing technology, it can generate higher quality and more controllable video content, with technical advantages and market competitiveness.

[0140] This embodiment has broad application prospects in multiple fields. For example, in the production of movies, animations, and games, high-quality visual content and audio effects are generated using multimodal input, which can reduce production costs and time. At the same time, through dynamic weight distribution and multi-level fusion technology, the generated content is more realistic and expressive. In the field of education and training, rich multimedia teaching content is generated through the combination of text, images, and audio to improve teaching effects and learning experience. Especially in distance education, multimodal video generation technology can provide more intuitive and vivid teaching resources. In virtual reality (VR) and augmented reality (AR) applications, high-quality virtual environments and interactive content are generated using multimodal input to enhance user immersion and interactive experience. By combining data such as depth images and point cloud maps, the generated virtual scenes are more realistic and three-dimensional. In medical imaging and health monitoring, high-quality diagnostic videos and visualization reports are generated through multimodal data fusion to help doctors diagnose and treat more accurately. At the same time, using data such as infrared images and depth images, the generated health monitoring videos are more comprehensive and accurate.

[0141] The present invention improves the quality, diversity and controllability of generated videos through multimodal fusion and controllable video generation technology. In the multimodal input processing stage, technologies such as adaptive multi-resolution time-frequency analysis (AMRTFA) and multi-scale thermal dynamic feature extraction (MTDFE) are used to improve the feature extraction quality of each modal data. Not only does it enhance the system's ability to process complex inputs, but it also provides a richer and more accurate feature representation for the subsequent fusion process. In the feature fusion stage, the context-aware weight regulator (CAWR) and hierarchical self-organizing fusion network (HSOFN) are introduced to achieve dynamic weight allocation and multi-level deep fusion. It can adaptively adjust the importance of different modal features according to the characteristics of the current task and input data, and improve the expressiveness and semantic consistency of the fused features. In the video generation stage, the application of multi-scale spatiotemporal attention mechanism (MSAM) and dynamic latent space reconstructor (DLSR) enables the system to accurately control the generation of video content at different time and space scales, while maintaining the spatiotemporal coherence of the video sequence. The introduction of the Adaptive Resolution Generation Network (ARGN) further improves the quality and detail level of the generated video, enabling the system to flexibly adjust the resolution of the output video according to demand. This embodiment has advantages in processing complex multimodal inputs and generating high-quality controllable videos. It can not only generate video content that is highly consistent with the multimodal input, but also ensure that the video reaches a high level in spatial detail, temporal coherence and overall quality. It provides technical support for controllable video generation tasks in various application scenarios.

[0142] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A controllable video generation method based on multimodal fusion, characterized in that: The steps include: S1. Obtain multimodal data and preprocess it; based on the preprocessed multimodal data, extract feature vectors using a pre-trained encoder network; Normalize the feature vectors and combine the normalized feature vectors to form a multimodal feature matrix; the multimodal data includes text, RGB images, audio, infrared images, depth images, mask images, edge operator images, point cloud images, and posture image data; S2. Use a predefined word segmenter to convert the multimodal feature matrix into a feature sequence, and add a modal identifier and position encoding information; apply a context-aware weight regulator to dynamically assign weights to each modal feature in the feature sequence to obtain a weighted feature sequence; input the weighted feature sequence into a hierarchical self-organizing fusion network for multi-level deep fusion to obtain a fused feature representation; S3, input the fused feature representation into the multi-scale spatiotemporal attention mechanism to obtain enhanced feature representation; based on Enhanced feature representation, processed sequentially using the spatial attention module, the temporal attention module, and the cross attention module to obtain a global enhanced feature representation; Based on the global enhanced feature representation, a dynamic latent space reconstructor is used to generate multi-frame latent space representations. The multi-frame latent space representations are decoded by an adaptive resolution generation network to generate high-resolution video frames. The high-resolution video frames are post-processed to obtain the final high-quality video sequence.

2. The controllable video generation method based on multimodal fusion according to claim 1 is characterized in that: Step S1 is further as follows: S11. Receive user input or pre-stored multimodal data through the data interface module, perform format verification on the multimodal data to ensure that the data format meets the predefined standard; store the verified multimodal data in a temporary cache, and generate a data index to record the type, size and storage location information of each modal data; S12, reading the multimodal data and preprocessing it to obtain the preprocessed multimodal data, specifically: For text data, natural language processing tools are used to segment words and remove stop words to obtain words; pre-trained word embedding models are used to convert words into vector representations as pre-processed text data; The RGB image data is adjusted to a uniform size and converted into a color space to obtain a converted image, and the pixel values ​​of the converted image are normalized to obtain preprocessed RGB image data; Resample the audio data, generate a spectrum diagram and extract spectrum coefficients as preprocessed audio data; Conduct heat distribution analysis on infrared image data, including hot spot detection and temperature gradient calculation, to obtain pre-processed infrared image data; For the depth image data, the depth value is converted into a standardized depth code, and a depth histogram is generated as the preprocessed depth image data; Apply the threshold algorithm to binarize the mask image data to obtain a clear foreground-background segmentation result as the preprocessed mask image data; For edge operator graph data, Gaussian smoothing is used to remove noise, and edge detection operators are applied to enhance edge features to obtain preprocessed edge operator graph data; The point cloud data is converted into a spatial coordinate system, unified into a predefined three-dimensional coordinate system, and the point cloud is downsampled to reduce the amount of data to obtain preprocessed point cloud data; For the posture graph data, the skeleton key point detection algorithm is used to extract the coordinates of the key points of the human body, and the relative position and angle information between the key points are calculated as the preprocessed posture graph data; S13, input the preprocessed multimodal data into the corresponding feature extraction modules respectively to extract feature vectors, specifically: For the preprocessed text data, use the pre-trained BERT model as the encoder to extract the context-related feature vector of the text; For the preprocessed RGB image data, the pre-trained ResNet model is used as the encoder to extract the high-level visual feature vector of the RGB image; For the preprocessed audio data, use the pre-trained VGGish model as the encoder to extract the acoustic feature vector of the audio; For the preprocessed infrared image data, a pre-trained convolutional neural network is used to extract the heat distribution feature vector; For the preprocessed depth image data, the pre-trained PointNet model is used to extract the depth feature vector; For the preprocessed mask image data, a lightweight U-Net model is used to extract the segmentation feature vector; For the preprocessed edge operator graph data, the pre-trained EdgeNet model is used to extract edge feature vectors; For the preprocessed point cloud data, use the pre-trained PointNet++ model to extract the 3D spatial feature vector; For the preprocessed posture graph data, the pre-trained PoseNet model is used to extract the posture feature vector; S14. Use the L2 norm normalization method to normalize all extracted feature vectors to obtain normalized feature vectors; concatenate the normalized feature vectors in a predefined order to form a unified multimodal feature matrix.

3. The controllable video generation method based on multimodal fusion according to claim 2 is characterized in that: Step S2 is further as follows: S21, using a predefined word segmenter to convert each row in the multimodal feature matrix into discrete word units, adding a modality identifier to the word units of each modality to distinguish the features of different modalities; splicing the word units of all modalities into a feature sequence in a predefined order; obtaining position encoding information, and combining it with the feature sequence to obtain a combined feature sequence; S22. Based on the combined feature sequence, a modal correlation analysis algorithm is used to construct a correlation graph between multimodal features; by analyzing the topological structure of the correlation graph, the importance score of each modal feature is calculated; Based on the importance score, an initial weight is assigned to each modality feature; Obtain the specific requirements of the current task and the characteristics of the multimodal data, dynamically adjust the initial weights to obtain the adjusted weights; apply the adjusted weights to the combined feature sequence to obtain the weighted feature sequence; S23. Based on the weighted feature sequence, a dynamic topology reconstruction algorithm is used to automatically adjust the hierarchical structure and connection method of the hierarchical self-organizing fusion network to obtain an adjusted hierarchical self-organizing fusion network; based on the adjusted hierarchical self-organizing fusion network, the weighted feature sequence is fused layer by layer starting from the bottom layer, and each layer uses a self-attention mechanism to capture the relationship between features, and finally a globally fused feature representation is obtained at the top layer; the globally fused feature representation is post-processed, including dimensionality adjustment and normalization, to obtain the final fused feature representation.

4. The controllable video generation method based on multimodal fusion according to claim 3 is characterized in that: Step S3 is further as follows: S31. Decompose the fused feature representation into predetermined time and space scales using a hierarchical spatiotemporal pyramid structure; apply an adaptive scale selection algorithm to the features of each time and space scale to dynamically select the processing scale; and calculate the correlation between features using a self-attention mechanism based on the selected processing scale; Based on the correlation, the cross-scale attention mechanism is used to integrate the information of different processing scales to obtain multi-scale processing results; the multi-scale processing results are fused to obtain enhanced feature representation; S32, reshape the enhanced feature representation into a three-dimensional tensor; based on the three-dimensional tensor, generate a spatial attention map through a convolution operation; apply the spatial attention map to the three-dimensional tensor to obtain a spatial weighted feature; Using residual connection to add the original features of the three-dimensional tensor to the weighted features to obtain a first fused feature; based on the first fused feature, applying a layer normalization operation to obtain a spatial enhanced feature representation; S33, reorganize the spatial enhanced feature representation into a time series; based on the time series, use the self-attention mechanism to calculate the attention weights between different time steps; Based on the attention weight, the features of each time step are weighted summed to obtain the temporal attention features; The spatial enhancement feature representation and the temporal attention feature are fused using a gating mechanism to obtain a second fused feature; based on the second fused feature, a layer normalization operation is applied to obtain a spatiotemporal enhancement feature representation; S34, extracting text features from the multimodal feature matrix, and calculating cross attention weights based on spatiotemporal enhanced feature representation and text features; Based on the cross attention weights, the spatiotemporal enhanced feature representations are weighted summed to obtain semantically aligned features; Using a gating mechanism to fuse the spatiotemporal enhancement feature representation and the semantic alignment feature, a third fused feature is obtained; based on the third fused feature, a layer normalization operation is applied to obtain a semantically enhanced video feature representation; S35. Based on semantically enhanced video feature representation, a latent space topology preservation algorithm is used to construct a mapping relationship from feature space to latent space. Based on the mapping relationship, the semantically enhanced video feature representation is projected into the latent space; Generate a hidden state sequence of a predetermined number of time steps in a latent space using a recurrent neural network; apply post-processing operations, including noise injection and regularization, to the hidden state sequence to obtain a multi-frame latent space representation; S36. Based on multi-frame latent space representation, a progressive detail enhancement algorithm is used to generate a low-resolution initial video frame; the specific scale information in the multi-frame latent space representation is extracted and gradually improved through upsampling and detail enhancement modules to obtain an initial high-resolution video frame; an adversarial training strategy is used to introduce a discriminator network to improve the authenticity of the initial high-resolution video frame to obtain a high-resolution video frame; according to preset quality indicators and available computing resources, the final resolution of the high-resolution video frame is dynamically adjusted to obtain a final high-resolution video frame; the final high-resolution video frame is post-processed, including color balancing, denoising and sharpening, to obtain a final high-quality video sequence.

5. The controllable video generation method based on multimodal fusion according to claim 4 is characterized in that: In step S12, the audio data is resampled to generate a spectrogram and extract spectrum coefficients, and the preprocessed audio data is further: S121, performing denoising processing on the audio data to obtain a denoised signal; performing volume normalization on the denoised signal to obtain a normalized signal; performing wavelet packet decomposition on the normalized signal to obtain sub-signals of a predetermined frequency band; S122, performing time-frequency analysis on each sub-signal to obtain the center frequency of the sub-signal; based on the center frequency, dynamically calculating the optimal time window length; Based on the optimal time window length, the sub-signals are framed and windowed, and the Fourier transform of each frame is calculated to obtain the time-frequency representation; the time-frequency representations of all sub-signals are merged to form a complete time-frequency diagram; S123, obtaining the frequency distribution characteristics of the audio data, dynamically generating a nonlinear frequency compression function, and mapping the time-frequency graph to a Mel frequency scale based on the nonlinear frequency compression function to obtain a mapped Mel frequency spectrum graph; Applying adaptive equalization processing to the mapped Mel-spectrogram to obtain an equalized Mel-spectrogram; S124, calculating the logarithm of the equalized Mel frequency spectrum, performing discrete cosine transform based on the logarithm, extracting Mel frequency cepstrum coefficients as static features; and calculating increment and acceleration coefficients based on the Mel frequency cepstrum coefficients as dynamic features; The static features and dynamic features are combined to form preprocessed audio data.

6. The controllable video generation method based on multimodal fusion according to claim 4 is characterized in that: In step S12, the infrared image data is subjected to heat distribution analysis, including hot spot detection and temperature gradient calculation, and the pre-processed infrared image data is further: S12a, based on the infrared image data, applying adaptive histogram equalization to obtain a contrast-enhanced infrared image; based on the contrast-enhanced infrared image, using an edge-preserving filter to process to obtain a denoised infrared image; based on the gradient information and heat intensity distribution of the denoised infrared image, calculating a segmentation threshold; Based on the segmentation threshold, the denoised infrared image is divided into predetermined heat regions; S12b, calculating the temperature characteristics of the pixels in each heat region, including the average temperature, the highest temperature, the lowest temperature and the temperature variance; analyzing the change of the heat region between adjacent frames, calculating the change rate of the heat region area and the change rate of the average temperature, and obtaining the change rate characteristics of the heat region; An improved optical flow algorithm is applied to each heat region to estimate the heat flow direction and speed characteristics; based on the heat region, the heat gradient is calculated at a predetermined spatial scale; based on the heat gradient, convolution kernels of different sizes are used to extract local and global heat change characteristics; Extract thermal texture features from each thermal region, and calculate the local temperature difference pattern and the statistical characteristics of thermal distribution based on the thermal texture features; use the sliding time window method to calculate the time evolution characteristics of the thermal distribution in the thermal region; S12c, normalizing the temperature features, rate of change features, heat flow direction and speed features, local and global heat change features, heat texture features, statistical characteristics and time evolution features to obtain normalized features; based on the normalized features, using the principal component analysis method to reduce the feature dimension to obtain preprocessed infrared image data.

7. The controllable video generation method based on multimodal fusion according to claim 4 is characterized in that: In step S12, the edge operator graph data is subjected to Gaussian smoothing to remove noise, and the edge detection operator is applied to enhance edge features, so that the preprocessed edge operator graph data is further obtained as follows: S12i, performing preliminary analysis on the edge operator graph data, calculating the cumulative distribution function of the image; performing histogram equalization based on the cumulative distribution function, and calculating the global noise level; Based on the global noise level, the parameters of the Gaussian filter, including the kernel size and the standard deviation, are adaptively determined; based on the determined parameters, a two-dimensional Gaussian kernel is constructed, and a Gaussian filtering operation is performed on the edge operator graph data to obtain a preliminary denoised image; S12ii, based on the preliminary denoised image, using the finite difference method, calculating the gradient distribution; based on the gradient distribution, adaptively selecting the threshold parameter of edge detection; Based on the threshold parameter, the Sobel operator or the Canny operator is selected to perform edge detection to obtain an edge map; the edge map is post-processed, including analyzing connectivity, removing noise, connecting edges, thinning edges, and analyzing directions to obtain a processed edge map; S12iii. Based on the processed edge map, calculate global statistical features, including edge density, average length and direction distribution; use Hough transform to detect straight and circular edges in the processed edge map, and record their parameters; calculate curvature statistics of the processed edge map, including average curvature and curvature variance; based on the processed edge map, repeat edge detection at different scales to construct multi-scale edge features; Analyze the texture features of the processed edge map, calculate the edge co-occurrence matrix and extract relevant statistics; S12iv, generating initial feature vectors based on global statistical features, parameters of straight and circular edges, curvature statistical information, multi-scale edge features and edge co-occurrence matrix; calculating the correlation between the initial feature vectors, identifying and removing highly correlated redundant features, and obtaining the screened features; Based on the screened features, the principal component analysis method is used to reduce the feature dimension and obtain the reduced-dimensional features; Normalize the features after dimensionality reduction to obtain normalized features; S12v. Calculate the importance score of each normalized feature, select the Top-K features from the normalized features according to the importance score, and form the final edge feature descriptor as the preprocessed edge operator graph data.

8. The controllable video generation method based on multimodal fusion according to claim 4 is characterized in that: In step S35, based on the semantically enhanced video feature representation, a latent space topology preservation algorithm is used to construct a mapping relationship from the feature space to the latent space: S35a, based on the semantic enhanced video feature representation, calculating the Euclidean distance matrix between data samples; based on the Euclidean distance matrix, constructing a K nearest neighbor graph; Calculate the Laplacian matrix of the K nearest neighbor graph; perform eigendecomposition on the Laplacian matrix to obtain eigenvalues ​​and eigenvectors; estimate the intrinsic dimension of the data based on the decay of the eigenvalues; S35b, based on the intrinsic dimension, initialize the dimension and structure of the latent space to obtain the initial latent space coordinates; based on the initial latent space coordinates and the semantically enhanced video feature representation, construct a cost function including a local distance preservation item and a global structure preservation item; Use the gradient descent method to iteratively optimize the cost function to obtain the optimized latent space coordinates; S35c. Based on the optimized latent space coordinates, a mapping function from the high-dimensional feature space to the latent space is constructed, and a radial basis function network is used as the basic form of the mapping function; the center point and width parameters of the radial basis function network are adjusted to obtain the adjusted mapping function; in the adjusted mapping function, a regularization term is introduced to obtain a simplified mapping function; an alternating optimization strategy is used to optimize the simplified mapping function to obtain an optimized mapping relationship.

9. A controllable video generation system based on multimodal fusion, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the controllable video generation method based on multimodal fusion as described in any one of claims 1 to 8.

Images