AIGC content generation method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of AIGC content generation, discloses an AIGC content generation method and system based on multi-modal fusion, and aims to solve the problems of decentralization, low efficiency and insufficient originality of a traditional content generation tool. Multi-modal data such as texts, images, videos and audios are integrated, user intentions are analyzed in combination with intelligent retrieval and a domain knowledge base, automatic generation from multi-modal input to high-quality creative content is achieved, a cross-modal collaborative generation technology is adopted, semantic features are dynamically aligned, and logically coherent content is generated. The content emotional value is enhanced through an emotional analysis and dynamic optimization strategy, the homogenization bottleneck is broken through, meanwhile, an automatic quality evaluation and format adaptation mechanism is integrated, deep application of scenes such as text travel, advertisement, e-commerce and interactive network television service is supported, marketing copywriting, short videos and cross-platform distribution schemes can be efficiently generated, and the market competitiveness is improved. And the content production efficiency and the creativity transmission are obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AIGC content generation, and in particular to a method and system for generating AIGC content based on multimodal fusion. Background Art

[0002] AIGC, leveraging deep learning algorithms and large-scale pre-trained models, has achieved a remarkable evolution from single-modality to multimodal content generation. Early AIGC technologies primarily focused on single-modality generation, such as text generation (e.g., the GPT series) and image generation (e.g., GANs and diffusion models). However, these technologies were limited by their inability to support cross-modal collaborative creation. Currently, AIGC technology has transitioned from experimental applications to industrialized deployment. Its core drivers include innovative generation algorithms (e.g., diffusion models replacing GANs), improved generalization capabilities of pre-trained models (e.g., GPT-4 and DeepSeek-V3), and the deep integration of multimodal technologies. For example, the text-to-video generation model Sora uses spatiotemporal diffusion technology to achieve dynamic content construction for complex scenes, while large multimodal models support cross-modal semantic understanding and content creation. This technological evolution has enabled AIGC to break through the limitations of single-modality models and accomplish complex tasks such as video semantic description generation and cross-language audio and video synthesis.

[0003] For example, the patent with announcement number CN119299806A discloses an AIGC multimodal audio-visual content creation method and system. The input module supports the access of text descriptions and multimodal materials; the AIGC generation module realizes cross-modal content conversion based on deep learning, and produces video sequences and soundtracks; the multimodal fusion module completes the synchronous integration of audio and video through the spatiotemporal alignment algorithm; the intelligent editing module uses dynamic editing technology to optimize the connection between clips to ensure visual continuity; the output module generates standard media formats such as MP4 / AVI on demand.

[0004] The inventors believe that these and other existing technologies often suffer from the following drawbacks: creative resources are dispersed across different departments, manually driven linear processes struggle to meet real-time dissemination needs, and algorithmic template-driven content production easily falls into a rut. This model creates a significant time lag between content production cycles and market demand, and struggles to overcome the industry bottleneck of excessive creative reuse. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing technology has the shortcomings of traditional content generation tools being decentralized, inefficient, and lacking in creativity. To this end, we propose an AIGC content generation method and system based on multimodal fusion.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions: a multimodal fusion-based The AIGC content generation method includes the following specific implementation steps: S1. Multimodal Input Parsing: Receive input data from at least two modalities, including text, images, audio, and video. Standardize the input data through a preprocessing module. Combined with RAG search enhancement technology, analyze user intent based on industry-specific knowledge graphs to generate demand tags that include emotional tone, target audience, and scenario type. S2. Multimodal Collaborative Generation: Based on the content analysis model, we extract the narrative structure, visual rhythm, and emotional triggers of popular content, generate reusable creative templates, and perform the following operations through the cross-modal generation engine: Text-to-image / video generation: Use the diffusion model to generate candidate images and combine it with video interpolation technology to generate dynamic clips. Image-to-text / audio generation: Use the CLIP model to match images and text, and use TTS to generate narration or sound effects. Multimodal feature alignment: Using the Transformer architecture to align semantic features of different modalities, generating coherent content with logical causal chains; S3, Creativity Enhancement and Emotion Optimization: Based on sentiment analysis models, we identify the emotional tendencies of generated content, dynamically adjust the generation strategy through strong learning, introduce diffusion model variants to generate multiple candidate solutions, and select the optimal solution based on user feedback to achieve precise control of the error rate of emotional intensity. S4. Quality control and output adaptation: CLIPScore is used to evaluate the matching degree of images and texts, ASR is used to verify the synchronization of text and audio, the format and size are automatically adjusted according to the distribution channel, and the loading speed is optimized. The quality assessment module collects data to optimize the allocation of computing power to drive the resource scheduling process.

[0007] Preferably, the reinforcement learning of the sentiment analysis model in S3 adopts the PPO algorithm, which is based on click rate, dwell time, The length is used as the reward function to dynamically adjust the parameter weights of the generation strategy.

[0008] Preferably, the specific steps for creativity enhancement and emotion optimization in S3 are as follows: S31. Emotional tendency recognition: Use a pre-trained sentiment analysis model to perform fine-grained sentiment classification on generated content, extract multi-emotional feature vectors of text, voice intonation, and visual elements, and construct an emotion intensity distribution map. S32, Reinforcement Learning Strategy Adjustment: Construct a reward function based on the emotion intensity distribution map, use a proximal policy optimization algorithm to dynamically adjust the parameter weights of the generative model, and generate multiple sets of adversarial generation schemes through Monte Carlo tree search; S33, candidate solution generation: Call the diffusion model variant to perform multi-scale generation, and generate a set of candidate content with gradient differences through latent space interpolation technology. Each candidate solution contains a sentiment intensity tag and a semantic consistency check code; S34. User Feedback Integration: Establish a real-time interactive evaluation system to collect users' sentiment ratings and correction instructions for candidate solutions. Use differential privacy technology to desensitize the feedback data and then update the reinforcement learning strategy. S35. Error rate control: Establish an emotion intensity error prediction model through a bidirectional long short-term memory network, and dynamically adjust the generation threshold in combination with adversarial training so that the standard deviation of the final output emotion intensity is less than the preset tolerance value.

[0009] Preferably, the industry vertical knowledge graph in S1 contains causal triplets, and the logical contradictions of the generated content are dynamically corrected through the graph neural network.

[0010] Preferably, the dynamic correction process of the industry vertical knowledge graph includes a contradictory relationship resolution algorithm, which maps triple conflict information to the latent semantic space through knowledge distillation technology, uses a contrastive learning framework to decouple the features of contradictory entities, and realizes the targeted elimination of logical contradictions based on the attention weight allocation mechanism.

[0011] Preferably, in S4, the specific steps of quality control and output adaptation are as follows: S41. Multimodal quality assessment: The CLIP model is used to extract the image and text feature vectors of the generated content, calculate the cross-modal cosine similarity score, and use the preset threshold to screen the content combinations that meet the semantic consistency. S42, Audio and Video Synchronization Verification: An end-to-end ASR model is used to perform voice activity detection and beamforming noise reduction on the audio stream. A dynamic time warping algorithm is used to compare text timestamps with audio spectrum features to ensure that the audio and video synchronization error is less than 200ms. S43, Channel Adaptation Optimization: Analyze the technical parameters of the target distribution channel, automatically adjust content resolution, bitrate, and packaging format based on a responsive design framework, and apply HEVC encoding compression to achieve a bandwidth saving rate of ≥ 40%; S44. Loading performance optimization: Build a content block preloading strategy, use WebP / AVIF format instead of traditional JPEG / PNG, cache hot resources through CDN edge nodes, and control the first screen loading time within 1.5 seconds; S45. Closed-loop quality control: Deploy a quality assessment module to collect core indicators such as CLIPScore, ASR word error rate, and first-frame rendering time in real time. Use the reinforcement learning framework to dynamically adjust the generation strategy to drive the load balancing and elastic scaling of the computing resource pool.

[0012] Preferably, the HEVC encoding compression process in S43 introduces a configurable rate control strategy, by analyzing The heat map distribution of content-sensitive areas, adaptive quantization parameter adjustment for key frames, and inter-frame prediction optimization for non-key frames are used to control the subjective visual quality loss rate within 3% and meet the preset bandwidth saving rate requirements.

[0013] Preferably, in S4, the quality assessment module is based on the multimodal task generation characteristics and uses nonlinear The resource allocation formula drives the resource scheduling process to promote AIGC content generation. The formula is as follows:

[0014] Where: represents the real-time quality score of module i; Indicates the task queue length of module i; Indicates the current resource saturation of module j, i.e. GPU utilization / memory occupancy; represents the mass attenuation coefficient; Indicates the system load uniformity coefficient; Represents the long task compensation factor.

[0015] The present invention also provides another technical solution: an AIGC content generation system based on multimodal fusion, using the above-mentioned AIGC content generation method based on multimodal fusion, comprising: Multimodal input processing module, which receives and standardizes multiple input data, supports dynamic priority sorting, and implements cross-modal data cleaning, conversion, and semantic enhancement; The multimodal fusion hub module is used to integrate the semantic associations of multimodal data, perform logical verification and sentiment mapping, and generate executable cross-modal content strategies; The content generation engine module is used to generate text, images, videos, and audio content through multimodal collaborative creation, combining diffusion models and feature alignment technology for precise output; The creative optimizer module uses reinforcement learning and emotional symbol constraints to improve the creativity and emotional value of generated content, supporting personalized adaptation and diversity enhancement; The output adaptation module is used to convert the generated content into a format suitable for multiple platforms, and optimize loading performance and transmission efficiency through dynamic compression and intelligent distribution.

[0016] Preferably, the system further includes The industry knowledge base module is used to provide professional data support and correct logical contradictions by integrating vertical field knowledge graphs and real-time retrieval capabilities; The user rights center module is used to manage user collaboration processes and rights allocation, and implement security operation audits through a visual interface and low-code tools; The security and privacy module combines federated learning with encryption technology to protect data privacy, conduct copyright traceability, and conduct content compliance audits. The quality assessment module is used to continuously improve the quality and effectiveness of generated content through multimodal indicators and automated test-driven model optimization.

[0017] The technical effects and advantages of the present invention are as follows: In the present invention, by integrating multi-source data such as text, images, audio, and video with RAG search enhancement technology, combined with the logical verification capability of industry vertical knowledge graphs, a full-link closed loop from demand analysis to creative output is constructed; in the interactive network TV service scenario, the system relies on the multimodal collaborative generation mechanism to break through the single modality limitation, and the creative enhancement module extracts the narrative structure and emotional trigger points of popular content through the BERT+ResNet model, and combines the reinforcement learning dynamic optimization generation strategy to significantly improve the click-through rate of advertising marketing content; the quality control layer adopts CLIPScore and ASR synchronous verification technology to ensure the subjective visual quality of cross-platform distributed content at 4K resolution, and at the same time achieves low-latency transmission through intelligent routing algorithms; finally, in scenarios such as cultural and tourism promotion and e-commerce sales, the efficiency of multimodal content production is significantly improved and the user stay time is effectively extended, building a complete technical ecosystem for interactive network TV services from personalized content generation to cross-terminal intelligent distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The disclosure of the present invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components: Figure 1 Schematic diagram of the overall process of the method and system for generating AIGC content based on multimodal fusion according to the present invention; Figure 2 A logical diagram of an AIGC content generation method based on multimodal fusion according to the present invention; Figure 3 Schematic diagram of RAG retrieval enhancement generation architecture of an AIGC content generation method based on multimodal fusion according to the present invention; Figure 4 Schematic diagram of the system architecture of an AIGC content generation method based on multimodal fusion of the present invention. DETAILED DESCRIPTION

[0019] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention and should not be regarded as the entire invention or as a limitation or restriction of the technical solution of the present invention.

[0020] Reference Figure 1-Figure 3 As shown, the present invention provides a technical solution: a multimodal fusion-based The AIGC content generation method includes the following specific implementation steps: Step 1: Multimodal input parsing and demand analysis: Receive input data in at least two modalities: text, image, audio, and video. Standardize the input data through a preprocessing module. Combined with RAG search enhancement technology, analyze user intent based on the industry vertical knowledge graph and generate demand labels that include emotional tone, target audience, and scenario type. In step 1, the industry vertical knowledge graph contains causal triplets, and the logical contradictions of the generated content are dynamically corrected through the graph neural network; The dynamic correction process of the industry vertical knowledge graph includes a contradictory relationship resolution algorithm, which maps triple conflict information to the latent semantic space through knowledge distillation technology, uses a contrastive learning framework to decouple the features of contradictory entities, and realizes the targeted elimination of logical contradictions based on the attention weight allocation mechanism.

[0021] Step 2: Multimodal collaborative generation: Based on the content analysis model, we extract the narrative structure, visual rhythm, and emotional triggers of popular content, generate reusable creative templates, and perform the following operations through the cross-modal generation engine: (1) Text → Image / Video Generation: Call the diffusion model to generate candidate images, and combine video interpolation technology to generate dynamic clips; (2) Image → text / audio generation: Use the CLIP model to achieve image-text matching and use TTS to generate narration or sound effects; (3) Multimodal feature alignment: Utilize the Transformer architecture to align semantic features of different modalities and generate coherent content containing logical causal chains.

[0022] Step 3: Creative Enhancement and Emotion Optimization: This involves identifying the emotional tendencies of generated content using a sentiment analysis model. Dynamically adjusting the generation strategy through strong learning, introducing a diffusion model variant to generate multiple candidate solutions, and filtering out the optimal solution based on user feedback, achieving precise control of the emotional intensity error rate to ≤8%. The reinforcement learning of the sentiment analysis model in S3 adopts the PPO algorithm, takes click rate and dwell time as reward functions, and dynamically adjusts the parameter weights of the generation strategy.

[0023] The specific steps for creativity enhancement and emotion optimization in S3 are as follows: S31. Emotional tendency recognition: Use a pre-trained sentiment analysis model to perform fine-grained sentiment classification on generated content, extract multi-emotional feature vectors of text, voice intonation, and visual elements, and construct an emotion intensity distribution map. S32, Reinforcement Learning Strategy Adjustment: Construct a reward function based on the emotion intensity distribution map, use the Proximal Policy Optimization (PPO) algorithm to dynamically adjust the parameter weights of the generative model, and generate multiple sets of adversarial generation schemes through Monte Carlo tree search; S33, candidate solution generation: Call the diffusion model variant to perform multi-scale generation, and generate a set of candidate content with gradient differences through latent space interpolation technology. Each candidate solution contains a sentiment intensity tag and a semantic consistency check code; S34. User Feedback Integration: Establish a real-time interactive evaluation system to collect users' sentiment ratings and correction instructions for candidate solutions. Use differential privacy technology to desensitize the feedback data and then update the reinforcement learning strategy. S35, Error rate control: A bidirectional long short-term memory network (Bi-LSTM) is used to establish an emotion intensity error prediction model. Combined with adversarial training, the generation threshold is dynamically adjusted to ensure that the standard deviation of the final output emotion intensity is less than the preset tolerance value. The error rate control in S35 adopts a real-time dynamic adjustment strategy, which realizes parameter adaptive adjustment by building a multi-dimensional feedback loop of generated content. Specifically, it includes: (1) Establish a feedback channel based on a generative adversarial network to map user correction instructions into gradient update signals in adversarial training; (2) Design a dual-path feature extractor to process the original generated content and the corrected content respectively, and calculate the feature space offset as the basis for adjustment; (3) A time decay factor is introduced to distribute weights on historical feedback data, and the exploration-exploitation mechanism of reinforcement learning is combined to dynamically balance generation stability and diversity.

[0024] Step 4: Quality Control and Output Adaptation: CLIPScore is used to evaluate the matching between text and images, ASR is used to verify the synchronization between text and audio, the format and size are automatically adjusted according to the distribution channel, and the loading speed is optimized to ≤ 2 seconds / MB. The quality assessment module collects data to optimize computing power allocation to drive the resource scheduling process.

[0025] In step 4, the specific steps for quality control and output adaptation are as follows: S41. Multimodal quality assessment: The CLIP model is used to extract the image and text feature vectors of the generated content, calculate the cross-modal cosine similarity score, and use the preset threshold to screen the content combinations that meet the semantic consistency. S42, Audio and Video Synchronization Verification: An end-to-end ASR model is used to perform voice activity detection (VAD) and beamforming noise reduction on the audio stream. Dynamic Time Warping (DTW) is used to compare text timestamps with audio spectrum features to ensure that the audio and video synchronization error is less than 200ms. S43, Channel Adaptation Optimization: Analyze the technical parameters of the target distribution channel and automatically adjust content resolution (supporting dynamic switching from 720P to 4K), bitrate (500kbps-10Mbps adaptive), and packaging format (MP4 / WebM / HLS) based on a responsive design framework. Apply HEVC encoding compression to achieve a bandwidth saving rate of ≥ 40%. S44. Loading performance optimization: Build a content block preloading strategy, use WebP / AVIF format instead of traditional JPEG / PNG, cache hot resources through CDN edge nodes, and control the first screen loading time within 1.5 seconds; S45. Closed-loop quality control: Deploy a quality assessment module to collect core indicators such as CLIPScore (benchmark value ≥ 0.75), ASR word error rate (≤ 3%), and first frame rendering time (≤ 800ms) in real time. Use the reinforcement learning framework to dynamically adjust the generation strategy to drive the load balancing and elastic scaling of the computing resource pool.

[0026] The HEVC encoding compression process in S43 introduces a configurable bit rate control strategy. By analyzing the heat map distribution of content-sensitive areas, adaptive quantization parameter adjustment is adopted for key frames, and inter-frame prediction optimization is implemented for non-key frames, so that the subjective visual quality loss rate is controlled within 3% and meets the preset bandwidth saving rate requirements.

[0027] In step 4, the quality assessment module uses a nonlinear resource allocation formula to drive the resource scheduling process based on the multimodal task generation characteristics to promote AIGC content generation. The formula is as follows: Where: represents the real-time quality score of module i (0-1, CLIPScore + manual scoring); Indicates the task queue length (number of tasks to be processed) of module i; Indicates the current resource saturation (GPU / memory usage) of module j; represents the mass attenuation coefficient (0.3-0.7, dynamically adjusted by reinforcement learning); Indicates the system load uniformity coefficient (0<λ<1, determined by cluster resources); Indicates the long task compensation factor (0.1-0.5, preset for task type).

[0028] Scenario: Simultaneously process three tasks: image restoration (low priority, module A), short video generation (high priority, module B), and 3D modeling (long task, module C). GPU resources need to be dynamically allocated based on the current status.

[0029] Step 1: Set initial parameters First, we set some basic parameters: Mass attenuation coefficient β = 0.5 (medium attenuation) System load balancing coefficient λ=0.8 (higher balancing requirements) Long mission compensation factor γ = 0.3 (medium compensation) Step 2: Determine the status parameters of each module Assume that the current state of the system is as follows:

[0030] Step 3: Calculate the molecular part For each module, calculate : Module A:

[0031] Module B:

[0032] Module C:

[0033] Step 4: Calculate the denominator calculate : Module A:

[0034] Module B:

[0035] Module C:

[0036] Denominator sum = 0.116 + 0.0081 + 0.252 = 0.3761 Step 5: Calculate the final resource quota Calculate the resource quota Qi for each module: Module A: (0.131 / 0.3761)×0.3≈0.348×0.3≈0.104 Module B: (0.0113 / 0.3761)×0.3≈0.0301×0.3≈0.009 Module C: (0.250 / 0.3761)×0.3≈0.665×0.3≈0.199 Step 6: Result analysis and adjustment Based on the calculation results, resource quotas are allocated as follows: Module A: 10.4% Module B: 0.9% Module C: 19.9%.

[0037] Specifically, by building an intelligent closed-loop system for collaborative processing of multi-source data, the system achieves automated generation of the entire chain, from demand analysis to creative output. The system integrates multimodal input data such as text, images, audio, and video, dynamically analyzing causal triplets in the industry knowledge graph using RAG search enhancement technology. Combined with graph neural networks, it eliminates logical contradictions in real time, forming a precise demand labeling system. The cross-modal generation engine uses diffusion models and CLIP feature alignment technology to achieve collaborative creation of text, visual, and auditory elements. The BERT+ResNet-based popular content analysis model dynamically extracts narrative structure and emotional triggers, supporting intelligent association of cross-modal elements such as drone aerial footage and ocean wave sound effects. In the creative optimization phase, PPO reinforcement learning drives diffusion model variants to generate multiple candidate solutions. A Bi-LSTM error prediction model and adversarial training are used to precisely control emotional intensity. A real-time interactive evaluation system based on user feedback dynamically adjusts the generation strategy. The quality control module ensures content consistency through simultaneous CLIPScore and ASR verification. HEVC adaptive coding technology dynamically optimizes bitrate allocation, achieving bandwidth savings and improved loading performance while ensuring subjective visual quality.

[0038] This method breaks through the limitations of single-modal generation through knowledge distillation and multimodal alignment technology, and uses a dynamic elastic quota algorithm to balance the allocation of computing resources, ultimately achieving diverse generation of creative content, precise delivery of emotional resonance, and efficient cross-platform distribution, effectively solving the core pain points of traditional tools such as fragmentation, homogenization, and inefficiency.

[0039] Reference Figure 1 and Figure 4 As shown, in this embodiment: the present invention also provides an AIGC content generation system based on multimodal fusion, using the above-mentioned AIGC content generation method based on multimodal fusion, including: 1. Multi-mode input processing module Function: Receives and standardizes text, image, audio, and video input, supporting dynamic prioritization. Sub-functions: Data cleaning (denoising / word segmentation), format conversion (Base64 encoding), and RAG enhancement (knowledge graph retrieval + requirement tag generation). Technical support: CLIP model (cross-modal alignment), Elasticsearch (semantic retrieval). 2. Multi-mode fusion hub module Function: Implement cross-modal semantic association and logic verification, and generate executable generation strategies.

[0040] Sub-functions: spatiotemporal alignment (video frame and audio synchronization), causal reasoning (avoiding logical contradictions), Emotion mapping (emotional symbols → multimodal expression). Technical support: 3D Transformer (spatiotemporal modeling), GNN (causal graph reasoning). 3. Content Generation Engine Module Function: Performs multimodal content generation, supporting collaborative creation of text, images, videos, and audio. Sub-functions: Text generation (GPT-4o copywriting), image generation (StableDiffusion), video generation (Sora dynamic clips), and audio generation (Whisper narration). Technical support: Diffusion model (high-resolution generation), Transformer (feature alignment). 4. Creative Optimizer Module Function: Enhance content creativity and emotional value, and achieve personalized adaptation. Sub-functions: Hot template call (industry template library), emotion intensity control (RoBERTa+PPO reinforcement learning), diversity enhancement (diffusion model variant). Technical support: Federated Learning (personalization strategy), NS-CLIP (emotional symbol constraints). 5. Output adapter module Function: Generate content formats that are suitable for multiple platforms and optimize loading performance. Sub-functions: cross-platform conversion (portrait / landscape), dynamic compression (H.265 encoding), CDN distribution (Edge node acceleration). Technical support: FFmpeg (format conversion), intelligent routing algorithm (low-latency distribution). Industry knowledge base module Function: Provide vertical field knowledge support and enhance the professionalism of generated content. Sub-functions: knowledge graph construction (culture and tourism / e-commerce fields), RAG enhancement (industry data retrieval), and logical verification (causal contradiction correction). Technical support: Neo4j (graph database), vector search (Milvus). User Rights Center Module Function: Manage user permissions and collaboration processes, and provide a visual interactive interface. Sub-functions: low-code editor (drag-and-drop storyboard script), RBAC permission allocation, and operation log audit. Technical support: React+TypeScript (front-end), OAuth2.0 (identity authentication). Security and Privacy Module Function: Ensure data security and compliance and prevent abuse. Sub-functions: federated learning (privacy-preserving training), digital watermarking (copyright tracing), and content review (BERT violation filtering). Technical support: homomorphic encryption (HE), zero-knowledge proof (ZKP). Quality Assessment Module Function: Evaluate the quality of generated content and drive continuous optimization of the model. Sub-functions: multimodal indicator calculation (CLIPScore / NISQA), A / B testing framework, and automatic parameter adjustment (Bayesian optimization). Technical support: Prometheus (monitoring), Optuna (hyperparameter optimization). 10. Resource Scheduling Center Module Function: Dynamically allocate computing resources to improve system throughput. Sub-functions: GPU elastic scheduling (NVIDIAA100 / A10G), distributed training (Horovod), cache acceleration (Redis). Technical support: Kubernetes (container orchestration), Ray (distributed computing).

[0041] Specifically: The system is based on a multimodal input processing module, uses RAG enhancement technology to dynamically analyze semantic associations in industry knowledge graphs, and combines CLIP cross-modal alignment capabilities to generate precise demand labels; the multimodal fusion hub module uses the spatiotemporal Transformer and causal reasoning engine to achieve synchronous verification of video frames and audio waveforms, and uses GNN to dynamically eliminate logical contradictions to form an executable generation strategy; the content generation engine integrates the diffusion model and Transformer architecture, links GPT-4o copy generation and StableDiffusion visual creation, and uses federated learning to call the industry template library in real time to achieve intelligent reuse of creative elements; the creative optimizer uses RoBERTa sentiment analysis combined with reinforcement learning to dynamically adjust generation parameters, uses diffusion model variants to generate multiple candidate solutions, and achieves precise control of emotional intensity through adversarial training. The output adaptation module dynamically converts the vertical / horizontal screen formats through FFmpeg, combines the intelligent routing algorithm to compress the bit rate and accelerate CDN edge distribution, so as to achieve a balanced optimization of subjective visual quality and bandwidth consumption; the industry knowledge base module relies on the graph database to build a vertical field causal relationship network, and dynamically corrects the logical paradox of generated content through vector retrieval; the security and privacy center uses homomorphic encryption and digital watermarking technology to ensure the security of data flow, and the content review module embeds the BERT model to filter illegal information; the quality assessment module integrates multimodal indicators to build a closed-loop feedback system, and the resource scheduling center dynamically allocates GPU computing power based on container orchestration, and finally forms an intelligent collaborative network from data input to creative output. On the premise of ensuring privacy, security and compliance, it significantly improves the efficiency of cross-domain content production and the intensity of emotional resonance, and promotes the paradigm upgrade of content production from tool decentralization to intelligent ecology.

[0042] The technical scope of the present invention is not limited to the contents of the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A method for generating AIGC content based on multimodal fusion, characterized in that: The specific implementation steps include the following: S1. Multimodal Input Parsing: Receive input data from at least two modalities, including text, images, audio, and video. Standardize the input data through a preprocessing module. Combined with RAG search enhancement technology, analyze user intent based on industry-specific knowledge graphs to generate demand tags that include emotional tone, target audience, and scenario type. S2. Multimodal Collaborative Generation: Based on the content analysis model, we extract the narrative structure, visual rhythm, and emotional triggers of popular content, generate reusable creative templates, and perform the following operations through the cross-modal generation engine: Text-to-image / video generation: Use the diffusion model to generate candidate images and combine it with video interpolation technology to generate dynamic clips. Image-to-text / audio generation: Use the CLIP model to match images and text, and use TTS to generate narration or sound effects. Multimodal feature alignment: Using the Transformer architecture to align semantic features of different modalities, generating coherent content with logical causal chains; S3, Creativity Enhancement and Emotion Optimization: Based on sentiment analysis models, we identify the emotional tendencies of generated content, dynamically adjust the generation strategy through strong learning, introduce diffusion model variants to generate multiple candidate solutions, and select the optimal solution based on user feedback to achieve precise control of the error rate of emotional intensity. S4. Quality control and output adaptation: CLIPScore is used to evaluate the matching degree of images and texts, ASR is used to verify the synchronization of text and audio, the format and size are automatically adjusted according to the distribution channel, and the loading speed is optimized. The quality assessment module collects data to optimize the allocation of computing power to drive the resource scheduling process.

2. The AIGC content generation method based on multimodal fusion according to claim 1, characterized in that: In S3, the reinforcement learning of the sentiment analysis model adopts the PPO algorithm, takes the click rate and the length of stay as the reward function, and dynamically adjusts the parameter weights of the generation strategy.

3. The AIGC content generation method based on multimodal fusion according to claim 2, characterized in that: In S3, the specific steps for creativity enhancement and emotion optimization are as follows: S31. Emotional tendency recognition: Use a pre-trained sentiment analysis model to perform fine-grained sentiment classification on generated content, extract multi-emotional feature vectors of text, voice intonation, and visual elements, and construct an emotion intensity distribution map. S32, Reinforcement Learning Strategy Adjustment: Construct a reward function based on the emotion intensity distribution map, use a proximal policy optimization algorithm to dynamically adjust the parameter weights of the generative model, and generate multiple sets of adversarial generation schemes through Monte Carlo tree search; S33, candidate solution generation: Call the diffusion model variant to perform multi-scale generation, and generate a set of candidate content with gradient differences through latent space interpolation technology. Each candidate solution contains a sentiment intensity tag and a semantic consistency check code; S34. User Feedback Integration: Establish a real-time interactive evaluation system to collect users' sentiment ratings and correction instructions for candidate solutions. Use differential privacy technology to desensitize the feedback data and then update the reinforcement learning strategy. S35. Error rate control: Establish an emotion intensity error prediction model through a bidirectional long short-term memory network, and dynamically adjust the generation threshold in combination with adversarial training so that the standard deviation of the final output emotion intensity is less than the preset tolerance value.

4. The AIGC content generation method based on multimodal fusion according to claim 1, characterized in that: In S1, the industry vertical knowledge graph contains causal triples, and the logical contradictions of the generated content are dynamically corrected through the graph neural network.

5. The AIGC content generation method based on multimodal fusion according to claim 4 is characterized in that: The dynamic correction process of the industry vertical knowledge graph includes a contradictory relationship resolution algorithm, which maps triple conflict information to the latent semantic space through knowledge distillation technology, uses a contrastive learning framework to decouple the features of contradictory entities, and realizes the targeted elimination of logical contradictions based on the attention weight allocation mechanism.

6. The AIGC content generation method based on multimodal fusion according to claim 1, characterized in that: In S4, the specific steps of quality control and output adaptation are as follows: S41. Multimodal quality assessment: The CLIP model is used to extract the image and text feature vectors of the generated content, calculate the cross-modal cosine similarity score, and use the preset threshold to screen the content combinations that meet the semantic consistency. S42, Audio and Video Synchronization Verification: An end-to-end ASR model is used to perform voice activity detection and beamforming noise reduction on the audio stream. A dynamic time warping algorithm is used to compare text timestamps with audio spectrum features to ensure that the audio and video synchronization error is less than 200ms. S43, Channel Adaptation Optimization: Analyze the technical parameters of the target distribution channel, automatically adjust content resolution, bitrate, and packaging format based on a responsive design framework, and apply HEVC encoding compression to achieve a bandwidth saving rate of ≥ 40%; S44. Loading performance optimization: Build a content block preloading strategy, use WebP / AVIF format instead of traditional JPEG / PNG, cache hot resources through CDN edge nodes, and control the first screen loading time within 1.5 seconds; S45. Closed-loop quality control: Deploy a quality assessment module to collect core indicators such as CLIPScore, ASR word error rate, and first-frame rendering time in real time. Use the reinforcement learning framework to dynamically adjust the generation strategy to drive the load balancing and elastic scaling of the computing resource pool.

7. The AIGC content generation method based on multimodal fusion according to claim 6, characterized in that: In S43, the HEVC encoding compression process introduces a configurable bit rate control strategy. By analyzing the heat map distribution of content-sensitive areas, adaptive quantization parameter adjustment is adopted for key frames, and inter-frame prediction optimization is implemented for non-key frames, so that the subjective visual quality loss rate is controlled within 3% and meets the preset bandwidth saving rate requirements.

8. The AIGC content generation method based on multimodal fusion according to claim 6, characterized in that: In S4, the quality assessment module drives the resource scheduling process based on the multimodal task generation characteristics using a nonlinear resource allocation formula to promote AIGC content generation. The formula is as follows: Where: represents the real-time quality score of module i; Indicates the task queue length of module i; Indicates the current resource saturation of module j, i.e. GPU utilization / memory occupancy; represents the mass attenuation coefficient; Indicates the system load uniformity coefficient; Represents the long task compensation factor.

9. An AIGC content generation system based on multimodal fusion, characterized by: The AIGC content generation method based on multimodal fusion according to any one of claims 1 to 8 comprises: Multimodal input processing module, which receives and standardizes multiple input data, supports dynamic priority sorting, and implements cross-modal data cleaning, conversion, and semantic enhancement; The multimodal fusion hub module is used to integrate the semantic associations of multimodal data, perform logical verification and sentiment mapping, and generate executable cross-modal content strategies; The content generation engine module is used to generate text, images, videos, and audio content through multimodal collaborative creation, combining diffusion models and feature alignment technology for precise output; The creative optimizer module uses reinforcement learning and emotional symbol constraints to improve the creativity and emotional value of generated content, supporting personalized adaptation and diversity enhancement; The output adaptation module is used to convert the generated content into a format suitable for multiple platforms, and optimize loading performance and transmission efficiency through dynamic compression and intelligent distribution.

10. The AIGC content generation system based on multimodal fusion according to claim 9, characterized in that: The system also includes The industry knowledge base module is used to provide professional data support and correct logical contradictions by integrating vertical field knowledge graphs and real-time retrieval capabilities; The user rights center module is used to manage user collaboration processes and rights allocation, and implement security operation audits through a visual interface and low-code tools; The security and privacy module combines federated learning with encryption technology to protect data privacy, conduct copyright traceability, and conduct content compliance audits. The quality assessment module is used to continuously improve the quality and effectiveness of generated content through multimodal indicators and automated test-driven model optimization.

Images