AI video processing method and system based on cross-modal dynamic weight allocation

Abstract

The invention relates to the cross technical field of artificial intelligence and multimedia processing, in particular to an AI video processing method and system based on cross-modal dynamic weight allocation, and the method comprises the following steps: S1, receiving multi-modal input data; s2, respectively extracting semantic features of each modal data through a modal special feature extractor; s3, performing semantic alignment on the extracted features through a cross-modal attention mechanism, wherein the cross-modal attention mechanism dynamically allocates modal weights; s4, the features after semantic alignment are input into a multi-modal Transform for feature fusion, and a unified multi-modal semantic representation is generated; s5, generating a video frame sequence based on the multi-modal semantic representation and a generative neural network; S6, post-processing the video frame sequence; and S7, outputting the video file. Through cross-modal dynamic weight distribution, the key modal feature weakening problem is solved, and a cross-modal attention unit accurately responds to a semantic scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and multimedia processing, and particularly relates to an AI video processing method and system based on cross-modal dynamic weight distribution. BACKGROUND

[0002] The existing video generation technology has the following limitations:

[0003] Insufficient multi-modal fusion

[0004] Traditional solutions (such as Adobe Premiere plug-ins based on pre-defined templates) only support single-modal input dominance (such as text-to-video), and cannot coordinate the semantic association between text descriptions, image materials, and background music (for example, product appearance images and function description audio cannot be synchronized in product demonstration videos).

[0005] Although mainstream multi-modal models (such as Flamingo) can process text and image data, they lack optimization mechanisms for cross-modal temporal alignment such as audio-visual synchronization and lip-sync matching.

[0006] Defects in generated quality

[0007] End-to-end video generation models (such as Phenaki) exhibit picture tearing or object distortion when generating videos longer than 5 seconds due to the lack of spatiotemporal continuity constraints (inter-frame PSNR fluctuation > 10 dB).

[0008] The existing digital human driving solution (such as MetaVoice-OFB) has a speech-lip synchronization error rate of > 15%, making it difficult to meet the precise expression needs of educational videos.

[0009] Resource adaptation is rigid

[0010] Commercial video generation tools (such as Synthesia) rely on fixed material libraries and cannot dynamically call and adapt resources based on text semantics (such as incorrectly calling snow material when generating a "beach sunset" scene).

[0011] Low computational efficiency

[0012] Video generation systems based on diffusion models (such as Stable Video Diffusion) have a single video generation time of > 10 minutes (RTX 4090 graphics card) at 1080P resolution, which cannot meet real-time requirements. SUMMARY

[0013] To solve the above technical problems, the present application provides an AI video processing method and system based on cross-modal dynamic weight distribution.

[0014] In a first aspect, an AI video processing method based on cross-modal dynamic weight distribution includes the following steps:

[0015] S1, receiving multi-modal input data, the multi-modal input data including text data, image data and audio data;

[0016] S2, extracting semantic features of each modal data by a modal-specific feature extractor: using a pre-trained language model to extract text semantic features and identify entity relationships and sentiment orientation, using a convolutional neural network to extract image visual features and identify objects, scene classification and color distribution, and using an acoustic model to extract audio features and identify tone, rhythm and emotional tone;

[0017] S3, performing semantic alignment on the extracted features by a cross-modal attention mechanism, which dynamically allocates modal weights:

[0018] (a) calculating feature similarity using dot product attention:

[0019]

[0020] where Q, K, V come from text, image and audio feature vectors respectively; (b) updating the weight coefficient based on the scene classification result:

[0021] (b) dynamic allocation rule:

[0022] When the text contains product appearance description, the image weight ω image = min(1.0+0.5×S similarity ,1.8);

[0023] When the audio is background music, the audio-picture correlation weight ω audio-visual = 0.9×sigmoid(E emotion );

[0024] (c) semantic alignment process: through 3 rounds of iteration optimization until the feature difference loss <0.05;

[0025] S4, inputting the semantically aligned features into a multi-modal Transformer for feature fusion to generate a unified multi-modal semantic representation;

[0026] S5, based on the multi-modal semantic representation, generating a generative neural network to generate a video frame sequence, and generating digital human expression parameters, motion parameters and lip synchronization parameters according to text semantic features and audio features, and using a spatio-temporal convolution network to ensure the temporal coherence between video frames;

[0027] S6, post-processing the video frame sequence: synchronously mixing the video frame sequence with the input audio, generating subtitles from the text data and superimposing them onto the video frames, adjusting the video color tone, picture element layout or rhythm based on the user preference data;

[0028] S7, outputting the video file.

[0029] Preferably, the weight distribution rule of step S3 further comprises:

[0030] When the input data is identified as an educational video, the weight coefficient of the text feature is increased to 1.3-1.5 times of the baseline weight;

[0031] When the input data is identified as a music MV, the weight coefficient of the audio feature is increased to 1.5-1.8 times of the baseline weight.

[0032] Preferably, the user preference data in step S6 is obtained in the following way:

[0033] S31, collecting user historical operation behavior data;

[0034] S32, generating a user preference profile by analyzing the behavior data through an LSTM neural network model;

[0035] S33, dynamically adjusting the video color tone parameters, background music style or digital human motion amplitude based on the profile.

[0036] In a second aspect, a system for AI video based on cross-modal dynamic weight distribution according to the first aspect, comprising:

[0037] A multi-modal input interface module for receiving text, image and audio data;

[0038] A feature extraction engine module, including: a text feature extractor configured with a pre-trained language model, an image feature extractor configured with a convolutional neural network, and an audio feature extractor configured with an acoustic model;

[0039] A multi-modal fusion module containing a cross-modal attention unit and a multi-modal Transformer, the cross-modal attention unit being configured with a weight distribution strategy library;

[0040] A video generation module containing a GAN-based frame sequence generator and a digital human driving engine;

[0041] An output processing module configured with a video encoder, an audio synchronizer and a subtitle superimposer;

[0042] Preferably, it further comprises a resource matching engine module, which comprises:

[0043] a semantic analysis unit configured to label input data with keywords, entities, and sentiment tags;

[0044] a multi-modal resource index bank storing images and audio materials tagged with semantics;

[0045] a resource combination optimizer configured with a semantic similarity calculation algorithm and connected to the multi-modal resource index bank.

[0046] Preferably, the resource combination optimizer employs a reinforcement learning mechanism, and the resource combination optimizer comprises:

[0047] a user feedback collection interface configured to receive rating data of generated videos from users;

[0048] a policy update unit configured to update parameters of the semantic similarity calculation model based on the rating data;

[0049] a weight self-adaptive unit configured to automatically increase a priority of text resource matching when an education video request is identified.

[0050] Preferably, the system further comprises a user preference learning module, and the user preference learning module comprises:

[0051] an operation log collector configured to record user adjustment behaviors of video color tone and rhythm parameters;

[0052] a portrait generator configured with a three-layer fully connected neural network model;

[0053] a parameter mapper configured to convert a user preference portrait into tuning parameters of a video generation module.

[0054] Preferably, the digital human driving engine comprises:

[0055] a lip sync analyzer configured with a phoneme-mouth shape mapping table;

[0056] a motion parameter generator configured to output joint motion parameters based on text sentiment features;

[0057] a timing verification unit configured to detect motion continuity of adjacent video frames through a spatio-temporal convolution network.

[0058] Preferably, the system is based on a micro-service module, and the micro-service module comprises:

[0059] at least 5 independently packaged Docker containers respectively deployed with a feature extraction engine, a multi-modal fusion module, and a video generation module;

[0060] a Kubernetes cluster management platform configured with an automatic scaling strategy;

[0061] The model compression component adopts channel pruning and quantization techniques to compress the generative neural network to 40%-60% of the original model.

[0062] Preferably, the generative model compression component adopts the following steps:

[0063] (1) Structured pruning is adopted for the convolutional neural network to remove channels with weights below a threshold value;

[0064] (2) 8-bit quantization is adopted for the Transformer layer to map floating-point parameters to INT8 format;

[0065] (3) The output distribution of the original generative neural network is migrated to the compressed model through knowledge distillation.

[0066] Compared with the prior art, the present application has the following advantages:

[0067] (1) The present application solves the problem of weakening of key modal features through cross-modal dynamic weight distribution, and accurately responds to semantic scenes through cross-modal attention units: product appearance description enhances image feature weight, and function description strengthens text features, with an accuracy improvement of 32% for educational video knowledge points and a reduction of MV audio-visual synchronization error to ±80ms

[0068] (2) The present application updates the semantic similarity model based on reinforcement learning, iterates the material matching strategy driven by user feedback, and retrieves the multi-modal resource index library through entity-emotion double labels, with an accuracy improvement of 92% for material matching, and the resource combination optimizer supports dynamic priority switching: automatically increases the weight of text material matching for educational requests

[0069] (3) The present application constructs a user preference portrait through LSTM, realizes parameter mapping through a three-layer fully connected neural network, automatically adjusts video tone / layout / rhythm parameters, reduces 87% of manual adjustment operations, and generates expressions through a phoneme-mouth mapping table and joint motion parameters, with an expression naturalness score improvement of 40%

[0070] (4) The present application adopts a high-performance lightweight deployment microservice architecture: 5 containers are independently deployed for feature extraction / fusion / generation modules, Kubernetes achieves 200% resource utilization, three-layer model compression technology compresses the overall model size to 45%, and the edge device inference speed reaches 24fps. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is an AI video processing method flowchart;

[0072] Figure 2 is an AI video processing method system diagram;

[0073] Figure 3 Workflow diagram for resource matching engine; DETAILED DESCRIPTION

[0074] The technical solutions of the present application will be described clearly and completely below in conjunction with specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application.

[0075] Embodiment 1, as shown in Figure 1

[0076] An AI video processing method based on cross-modal dynamic weight distribution, the specific steps are as follows:

[0077] Multimodal data receiving: receiving user uploaded text (JSON format), image (JPEG / PNG) and audio (MP3 / WAV) through RESTful API interface. For example, in the e-commerce scenario, the text is "waterproof Bluetooth speaker, beach party essential", the image is a three-dimensional rendering diagram of the speaker, and the audio is a mixed file of sea wave sound and electronic music.

[0078] Modal feature extraction:

[0079] Text features: BERT model is used to extract semantic features to identify "waterproof" (functional description), "beach party" (scene) and "excited" emotional tendency;

[0080] Image features: ResNet-50 convolutional network is used to extract visual features to identify the shell material of the speaker and the color distribution of the blue LED light effect;

[0081] Audio features: WaveNet model is used to extract acoustic features to identify the rhythm value 120BPM and "cheerful" emotional tone.

[0082] Cross-modal semantic alignment:

[0083] Detecting the "waterproof" functional description in the text, the text feature weight is increased to 1.4 times (baseline weight 1.0);

[0084] Identifying the audio as background music, setting the audio-picture correlation weight coefficient to 0.9 (range 0-1);

[0085] Through the attention matrix, the similarity score of the image features and the "beach" scene in the text is calculated as 0.87;

[0086] The calculation process includes:

[0087] The specific process includes: ​

[0088] Step 1: Construct the text-image feature matrix Where W represents the trainable weights;

[0089] Step 2: Calculate the similarity score S = cos(E) text E image );

[0090] Step 3: If S > 0.8 and the text contains a functional description, then update the text weight ω. test =1.4×ω base ;

[0091] Step 4: Iterate twice until S converges.

[0092] Feature fusion and generation:

[0093] Multimodal Transformer fuses features to output a 128-dimensional semantic vector;

[0094] A GAN generator generates a sequence of video frames (25 FPS) of a beach scene, and a spatiotemporal convolutional network verifies the continuity of wave fluctuations.

[0095] Digital Human Drive Engine: Generates waving motion parameters (joint angle ±30°) based on "excitement" emotion, and synchronizes lip movements with audio phonemes.

[0096] Post-processing optimization:

[0097] The audio synchronizer aligns the music rhythm with the digital human's movements;

[0098] Add the subtitle "IPX7 waterproof rating" using the subtitle overlay tool;

[0099] The video color temperature was adjusted to 6500K based on user preferences (historical adjustment of cool tones).

[0100] Output: Generates a 1080P MP4 video file with a duration of 15 seconds.

[0101] Implementation method 2, such as Figure 2 As shown

[0102] Resource matching engine workflow:

[0103] Semantic analysis: The input text "Children's Educational Math Class" was annotated with the keywords "children" (subject), "mathematics" (topic), and the sentiment tag "relaxed";

[0104] Resource retrieval: Matching cartoon digital materials (similarity 0.92) and piano background music (emotional match 0.85) from the index database;

[0105] Reinforcement learning optimization:

[0106] User rating of generated video 4.2 / 5.0 → policy update unit adjusts mathematical class material priority weight +15%

[0107] Identify educational class videos, text resource matching priority automatically promoted to the highest level.

[0108] Embodiment 3, as Figure 3 shown

[0109] Model compression and deployment:

[0110] Structured pruning: remove channels with absolute weight values <0.01 for ResNet's convolutional layers, parameter quantity reduced by 45%;

[0111] 8-bit quantization: map Transformer floating-point parameters to INT8 format, model volume compressed to 52%;

[0112] Knowledge distillation:

[0113] Teacher model: original GAN generator (parameter quantity 120M)

[0114] Student model: 3-layer lightweight convolutional network (parameter quantity 48M)

[0115] Loss function: minimize the SSIM difference between the frames generated by the teacher and student (target value >0.95)

[0116] The specific embodiments further describe the present application, but it should be understood that the specific description herein should not be construed in a limiting sense. Various modifications within the scope of the present application made by those skilled in the art after reading the description are within the scope of the present application.

Claims

1. An AI video processing method based on cross-modal dynamic weight allocation, characterized in that, Includes the following steps: S1. Receive multimodal input data, wherein the multimodal input data includes text data, image data, and audio data; S2. Extract semantic features of each modality data using a modality-specific feature extractor: use a pre-trained language model to extract text semantic features and identify entity relationships and sentiment tendencies; use a convolutional neural network to extract image visual features and identify object, scene classification, and color distribution; use an acoustic model to extract audio features and identify pitch, rhythm, and emotional tone. S3. Semantic alignment of extracted features is performed through a cross-modal attention mechanism. The cross-modal attention mechanism dynamically allocates modal weights: when the text data contains product appearance descriptions, the weight of image features is increased; when the text data contains functional descriptions, the weight of text features is increased; when the audio data is background music, the weight of the correlation between audio emotional features and video images is increased. The specific process for dynamically allocating modal weights includes: (a) Calculate the cross-modal feature similarity matrix: using the dot product attention formula: Where Q, K, and V are the query, key, and value vectors, respectively; (b) Update weight coefficients based on scene classification results: If identified as an educational video, the text feature weights are: oh test =1.4×ω base (c) Optimize semantic alignment through three rounds of iteration until the feature difference threshold is <0.05; S4. Input the semantically aligned features into the multimodal Transformer for feature fusion to generate a unified multimodal semantic representation; S5. Based on the multimodal semantic representation, a generative neural network generates a video frame sequence, and generates digital human facial expression parameters, action parameters and lip-sync parameters according to text semantic features and audio features. A spatiotemporal convolutional network is used to ensure the temporal continuity between video frames. S6. Post-process the video frame sequence: Synchronously mix the video frame sequence with the input audio, generate subtitles based on the text data and overlay them onto the video frames, and adjust the video tone, layout of screen elements or rhythm based on user preference data. S7. Output video file.

2. The method according to claim 1, characterized in that: The weight allocation rules for step S3 also include: When the input data is identified as educational video, the weight coefficient of the text features is increased to 1.3-1.5 times the baseline weight; When the input data is identified as a music video, the weight coefficient of the audio features is increased to 1.5-1.8 times the baseline weight.

3. The method according to claim 1, characterized in that, The method for obtaining user preference data in step S6 is as follows: S31. Collect user's historical operation behavior data; S32. Generate user preference profiles by analyzing behavioral data using an LSTM neural network model; S33. Dynamically adjust the video tone parameters, background music style, or digital human motion range based on the image.

4. A system for AI video based on cross-modal dynamic weight allocation as described in claims 1-3, characterized in that, include: A multimodal input interface module for receiving text, image, and audio data; The feature extraction engine module includes: a text feature extractor, configured with a pre-trained language model; an image feature extractor, configured with a convolutional neural network; and an audio feature extractor, configured with an acoustic model. The multimodal fusion module includes a cross-modal attention unit and a multimodal Transformer. The cross-modal attention unit is configured with a weight allocation strategy library. The weight allocation strategy library includes a real-time update unit and a gradient optimizer. The real-time update unit dynamically generates a weight coefficient table based on the scene classification results, and the gradient optimizer minimizes the modal feature difference loss through backpropagation. The video generation module includes a GAN-based frame sequence generator and a digital human driving engine; The output processing module is configured with a video encoder, audio synchronizer, and subtitle overlay.

5. The system according to claim 4, characterized in that, It also includes a resource matching engine module, which includes: The semantic analysis unit is used to label input data with keywords, entities, and sentiment tags; A multimodal resource index library that stores semantically tagged image and audio materials; The resource combination optimizer configures the semantic similarity calculation algorithm and connects to the multimodal resource index.

6. The system according to claim 5, characterized in that: The resource combination optimizer employs a reinforcement learning mechanism and includes: The user feedback collection interface is used to receive user rating data for the generated videos; The strategy update unit updates the parameters of the semantic similarity calculation model based on the scoring data; The weighted adaptive unit automatically increases the priority of text resource matching when it recognizes educational video requests.

7. The system according to claim 4, characterized in that, It also includes a user preference learning module, which includes: Operation log collector, used to record user adjustments to video tone and rhythm parameters; Image generator, configured with a three-layer fully connected neural network model; The parameter mapper converts user preference profiles into tuning parameters for the video generation module.

8. The system according to claim 4, characterized in that, The digital human drive engine includes: Lip-sync analyzer, configured with phoneme-lip mapping table; Motion parameter generator, which outputs joint motion parameters based on text sentiment features; The timing verification unit uses a spatiotemporal convolutional network to detect the continuity of action between adjacent video frames.

9. The system according to claim 4, characterized in that, The system is based on a microservice module, which includes: At least five independently packaged Docker containers are used to deploy the feature extraction engine, multimodal fusion module, and video generation module, respectively. Configure automatic scaling policies for Kubernetes cluster management platforms; The model compression component uses channel pruning and quantization techniques to compress generative neural networks to 40%-60% of the original model.

10. The system according to claim 9, characterized in that, The generative model compression component employs the following steps: (1) Apply structured pruning to the convolutional neural network and remove channels whose absolute weight values ​​are below the threshold; (2) Use 8-bit quantization for the Transformer layer to map floating-point parameters to INT8 format; (3) The output distribution of the original generative neural network is transferred to the compressed model through knowledge distillation.

Images