AIGC intelligent agent based on fusion of multiple models

Abstract

The invention provides an AIGC intelligent agent based on multi-model fusion, which comprises a task analysis module, a multi-model dynamic selection and scheduling module, a cross-modal feature fusion module, a multi-modal content collaborative generation module and a result output module. According to the method, task demand parameters are matched with a constructed model capability library, a modal candidate model is obtained in combination with resource scheduling, capability limitation of a single model can be avoided through a dynamic selection mode, and then intermediate features output by the modal candidate model are fused to obtain a modal candidate model; the obtained global fusion feature vector can be processed by the modal candidate model to obtain the preliminary multi-modal content, the improved CLIP model is introduced to carry out consistency verification on the preliminary multi-modal content, semantic consistency errors can be reduced, finally, the generation result passing the consistency verification is transmitted to the user, and the satisfaction degree of the user to the generation result can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AIGC intelligent agent based on the fusion of multiple models. Background Technology

[0002] Existing single-modal AI-generated content (AIGC) models, such as the text generation model GPT, the image generation model Stable Diffusion, the speech generation model Tacotron, and simple multi-model concatenation systems, operate on the core logic of calling the corresponding single-modal model to output results for single-modal generation needs, or by concatenating multiple models through a fixed process to achieve simple cross-modal generation such as text→image or speech→text. These technologies are widely used in content creation, speech synthesis, and virtual content generation, but they suffer from the following drawbacks and limitations: Limitations of multimodal generation capabilities: Single-modal models can only handle a single type of input / output. For example, GPT cannot directly generate images, and Stable Diffusion cannot understand complex text semantics. Simple multi-model concatenation lacks deep intermodal interaction, and the generated results are prone to semantic fragmentation. For example, the text description "red roses under a blue sky" may result in a white rose in the generated image, and the cross-modal consistency error rate exceeds 30%. Poor model fusion flexibility: Existing multi-model systems mostly use fixed model combinations and fixed processes, such as only supporting two combinations: text→image and text→speech. They cannot dynamically select and adapt models according to the type of input task, such as "generating animations with voice narration" or "generating story text + illustrations from images". Adapting to new tasks requires system reconstruction, which takes more than 2 weeks.

[0003] The contradiction between generation quality and efficiency: To improve generation quality, existing technologies often use a single model with a large number of parameters, but the reasoning speed is slow; if a small model is used to improve efficiency, the logical coherence and richness of detail of the generated content will decrease.

[0004] Lack of dynamic feedback optimization: Existing systems mostly generate data in a one-time manner, and cannot dynamically adjust model parameters or fusion strategies based on user feedback on the initial generated results. Users need to re-initiate the generation request, resulting in low interaction efficiency and insufficient user satisfaction. Summary of the Invention

[0005] In view of this, the present invention proposes an AIGC agent based on the fusion of multiple models. Through deep fusion of cross-modal features and consistency verification, the semantic consistency of multimodal generation results can be improved, thereby enhancing user satisfaction with the generated content.

[0006] The technical solution of this invention is implemented as follows: An AIGC agent based on multi-model fusion includes: The task parsing module is used to receive the generation requirements input by the user, identify the task type through the task classifier, and output structured task requirement parameters. The multi-model dynamic selection and scheduling module is used to match task requirement parameters with model capabilities in a pre-built model capability library through a matching algorithm, filter out a set of suitable candidate models, and schedule the set of candidate models according to the current system resources to obtain modal candidate models. The cross-modal feature fusion module is used to assign different correlation weights to the intermediate features generated by the modal candidate model and then generate a global fusion feature vector. The multimodal content collaborative generation module is used to perform consistency verification on the preliminary multimodal content generated based on the global fusion feature vector using an improved CLIP model; The result output module is used to encapsulate the preliminary multimodal content that has passed the consistency check into a unified format of generated results and output the generated results to the user terminal. The task parsing module, multi-model dynamic selection and scheduling module, cross-modal feature fusion module, multi-modal content collaborative generation module, and result output module are sequentially connected in data.

[0007] Preferably, the task classifier is constructed using an improved lightweight BERT model. The basic architecture of the improved lightweight BERT model includes a 6-layer Transformer encoder, a hidden layer dimension of 512, and 8 attention heads. A multi-label classification head and an entity attribute extraction head are added before the output layer to be responsible for task type classification, input modality, output modality combination, and entity attribute extraction, respectively.

[0008] Preferably, the execution steps of the task classifier include: Receive user input of generation requirements, and convert the generation requirements into a text description + modal tag format, wherein the modal tags include input modality, output modality, and style requirements; Remove meaningless characters and redundant expressions from the text description, and perform entity recognition, annotation, and requirement completion. The text description is converted into word vectors through a word embedding layer, and after the context features are extracted by the Transformer encoder, the task type probability distribution and input / output modality probability distribution are output through the multi-label classification head, and the structured attributes are output through the entity attribute extraction head. Based on the probability distribution of task type and the probability distribution of input / output mode, obtain the input task type, input mode and output mode combination, and convert the task type, input mode, output mode combination and structured input and output into structured task requirement parameters.

[0009] Preferably, the model capability library sets several categories of labels for each model and quantifies the scores for different labels. The labels include task type adaptability, modal input / output support, quality score, resource consumption ratio, inference latency, style adaptability range, accuracy level, fault tolerance, update iteration cycle, and cross-modal collaboration capability.

[0010] Preferably, the execution steps of the multi-model dynamic selection and scheduling module are as follows: Preliminary screening: Based on the task requirement parameters, query preliminary candidate models whose modal input / output support score and task type suitability score are greater than the preliminary screening threshold; Secondary screening: Models with resource consumption ratio scores and inference latency scores greater than the secondary screening threshold in the initial candidate models are removed to obtain secondary candidate models; Third ranking: Calculate the comprehensive matching degree of the secondary candidate models. The comprehensive matching degree is calculated as follows: M=Σ(ω_s×ω_t×L_i), where M is the comprehensive matching degree, ω_s is the task scenario weight coefficient, which is dynamically adjusted according to the task type in the task requirement parameters, ω_t is the basic weight of the label, and Li is the quantitative score of the label. The top 3 models with the comprehensive matching degree are output as modal candidate models. Final decision: Query the system resource status, which includes the current video memory utilization rate and the current CPU load rate. Calculate the resource availability based on the system resource status. The formula for calculating resource availability is R = 0.7 × (1 - current video memory utilization rate) + 0.3 × (1 - current CPU load rate). Select the candidate model with the highest overall matching degree × resource availability as the final execution model.

[0011] Preferably, the execution steps of the cross-modal feature fusion module are as follows: Perform dimensional unification on the intermediate features generated by the modality candidate model; The association weights of different intermediate features are calculated using a cross-modal attention mechanism; Assign the associated weights to the corresponding intermediate features, and then concatenate the features to obtain the global fusion feature vector.

[0012] Preferably, the basic architecture of the improved CLIP model is as follows: Text encoder: The original CLIP Text Transformer is replaced with a lightweight BERT; Visual encoder: Retains CLIP's ViT-B / 16 architecture and adds an adaptive pooling layer to the output layer; A new speech encoder has been added: 80-dimensional Mel-frequency features are extracted based on Wav2Vec2.0 and encoded into a 512-dimensional speech feature vector through a 2-layer CNN+Transformer.

[0013] Preferably, the execution steps of the multimodal content collaborative generation module are as follows: Modal features are extracted from the initial multimodal content, and semantic feature vectors, visual feature vectors, and speech feature vectors are obtained respectively. Cosine similarity is used to calculate the similarity between any two modal features, and text-image similarity, text-speech similarity, and image-speech similarity are obtained. The consistency score is calculated using a weighted summation formula, which is: Consistency Score = 0.4 × Text-Image Similarity + 0.3 × Text-Speech Similarity + 0.3 × Image-Speech Similarity; If the consistency score is less than the preset verification threshold, the feedback is sent to the cross-modal feature fusion module to adjust the association weights. If the consistency score is greater than the preset verification threshold, the preliminary multimodal content is sent to the result output module.

[0014] Preferably, it also includes a resource adaptive scheduling module, which is data-connected to the multi-model dynamic selection and scheduling module, and is used to perform lightweight adaptation processing on modal candidate models. Its execution steps are as follows: Set lightweight trigger conditions, including GPU memory utilization ≥ 80%. When the CPU load is ≥70%, the task queue length is ≥50, and the real-time task response latency is ≥500ms, lightweight adaptation is started when the modal candidate model meets any lightweight triggering condition and continues for 3 sampling periods. Based on the type of the modal candidate model, a lightweight strategy is selected from a preset lightweight strategy library, and the modal candidate model is subjected to lightweight adaptation processing. The lightweight strategy includes layer pruning, quantization, and knowledge distillation.

[0015] Preferably, it also includes a multi-round feedback iteration module, which receives user feedback instructions on the generated results, parses the requirements in the feedback instructions and converts them into model adjustment parameters, and adjusts the modal candidate models according to the model adjustment parameters. The multi-round feedback iteration module is data-connected to the multi-model dynamic selection and scheduling module.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention discloses an AIGC agent based on multi-model fusion. After parsing the user's generation requirements, it can generate task requirement parameters. Then, through a preset matching algorithm, it can match the task requirement parameters with the model capabilities in a pre-built model capability library and filter to obtain a set of suitable candidate models. Then, through scheduling the current system resources, it can obtain multiple modal candidate models for generating the corresponding internal motion. By constructing the model capability library and introducing matching algorithms and resource scheduling strategies, it can achieve efficient adaptation between tasks and models without reconstructing the system. It can dynamically select the optimal model and avoid the limitations of a single model capability. Then, the intermediate features generated by multiple modal candidate models are fused to obtain a global fused feature vector. An improved CLIP model is then used for consistency verification. The preliminary multimodal content that passes the consistency verification is encapsulated into a unified format and output. This can reduce the semantic consistency error rate and improve user satisfaction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an AIGC agent based on the fusion of multiple models according to the present invention.

[0019] In the diagram, 1. Task parsing module; 2. Multi-model dynamic selection and scheduling module; 3. Cross-modal feature fusion module; 4. Multi-modal content collaborative generation module; 5. Result output module; 6. Resource adaptive scheduling module; 7. Multi-round feedback iteration module; 8. Dynamic weight adjustment and quality optimization module. Detailed Implementation

[0020] To better understand the technical content of this invention, a specific embodiment is provided below, and the invention will be further described in conjunction with the accompanying drawings.

[0021] See Figure 1 The present invention provides an AIGC agent based on the fusion of multiple models, comprising: Task parsing module 1 is used to receive the generation requirements input by the user, identify the task type through the task classifier, and output structured task requirement parameters. The multi-model dynamic selection and scheduling module 2 is used to match the task requirement parameters with the model capabilities in the pre-built model capability library through a matching algorithm, filter out a set of suitable candidate models, and schedule the set of candidate models according to the current system resources to obtain modal candidate models. The cross-modal feature fusion module 3 is used to assign different correlation weights to the intermediate features generated by the modal candidate model and then generate a global fusion feature vector. The multimodal content collaborative generation module 4 is used to perform consistency verification on the preliminary multimodal content generated based on the global fusion feature vector using an improved CLIP model; Result output module 5 is used to encapsulate the preliminary multimodal content that has passed the consistency check into a unified format of generated results and output the generated results to the user terminal. The task parsing module 1, the multi-model dynamic selection and scheduling module 2, the cross-modal feature fusion module 3, the multi-modal content collaborative generation module 4, and the result output module 5 are connected in sequence.

[0022] This invention discloses an AIGC agent based on multi-model fusion. It dynamically selects and combines multiple models to generate content that meets user needs. After receiving the user's input generation requirements through the task parsing module 1, the module classifies the requirements using a pre-set task classifier to determine the corresponding task type. It then outputs structured task requirement parameters, including task type, input / output modality, and corresponding attributes. After obtaining the task requirement parameters, the module determines the models for processing based on these parameters. The multi-model dynamic selection and scheduling module 2 internally constructs a model capability library containing capability scores for various models, which are then matched... The algorithm can match task requirement parameters with model capabilities and find multiple suitable candidate model sets. Then, it can schedule the candidate model sets according to the current system resources to achieve further filtering. The final modal candidate models can then dynamically generate content according to the user's generation requirements. Through the pre-built model capability library, matching algorithm and resource scheduling, the appropriate model can be dynamically selected according to different generation requirements. Compared with the traditional static fixed model combination, it can adapt to more complex tasks and supports arbitrary combination generation of 8 modalities such as text, image, voice and animation. New task adaptation only requires updating the model capability library without reconstructing the system, which is a significant improvement over existing technologies. It can dynamically select the optimal model and avoid the limitations of a single model capability.

[0023] After determining the modal candidate model, the modal candidate model can be processed based on the user's generation requirements. Different modal candidate models can generate intermediate features. Then, the cross-modal feature fusion module 3 can perform weighted fusion on the intermediate features. The resulting global fusion feature vector can be sent to the multimodal content collaborative generation module 4. The multimodal content collaborative generation module 4 uses an improved CLIP model to perform consistency verification on the global fusion feature vector. Only the preliminary multimodal content that passes the consistency verification will be sent to the result output module 5. If the consistency verification fails, the association weights of the cross-modal feature fusion module 3 need to be adjusted to generate a new global fusion feature vector, and the processing process of the multimodal content collaborative generation module 4 is repeated until the consistency verification passes. Through the consistency verification, the semantic consistency error rate of the final generated result can be significantly reduced, the content matching degree of text with images and animations can be improved, and the user's satisfaction with the generated content can be improved.

[0024] Preferably, the task classifier is constructed using an improved lightweight BERT model. The basic architecture of the improved lightweight BERT model includes a 6-layer Transformer encoder, a hidden layer dimension of 512, and 8 attention heads. A multi-label classification head and an entity attribute extraction head are added before the output layer to be responsible for task type classification, input modality, output modality combination, and entity attribute extraction, respectively.

[0025] The task classifier is built using an improved lightweight BERT model. The encoder was originally BERT-base with 12 layers. By using a 6-layer Transformer encoder, the computational cost can be reduced. The hidden layers were originally 768-dimensional, but were reduced to 512 dimensions. This allows for the preservation of feature representation through linear compression. The specific training process is as follows: Training dataset construction: Dataset size: 100,000 labeled samples, covering 8 input modalities, 12 output modalities, and 20 task types (cross-modal generation / single-modal optimization / style transfer / content modification, etc.). Label fields: Task type (1 main label), input modality (1-N labels), output modality combination (1-N labels), core requirement attributes (generation duration / style / precision requirements, etc., in key-value format).

[0026] Training parameters and loss function: Optimizer: AdamW; Training batch size: 32, number of iterations: 100 epochs, early stopping condition (validation set F1 score has not improved for 5 consecutive epochs). Loss function: Multi-label cross-entropy loss (task type + modal label) + MSE loss (numerical attribute, such as generation time).

[0027] Preferably, the execution steps of the task classifier include: Receive user input of generation requirements, and convert the generation requirements into a text description + modal tag format, wherein the modal tags include input modality, output modality, and style requirements; Remove meaningless characters and redundant expressions from the text description, and perform entity recognition, annotation, and requirement completion. The text description is converted into word vectors through a word embedding layer, and after the context features are extracted by the Transformer encoder, the task type probability distribution and input / output modality probability distribution are output through the multi-label classification head, and the structured attributes are output through the entity attribute extraction head. Based on the probability distribution of task type and the probability distribution of input / output mode, obtain the input task type, input mode and output mode combination, and convert the task type, input mode, output mode combination and structured input and output into structured task requirement parameters.

[0028] User requests require preprocessing, including data structuring and cleaning. Data structuring involves converting the requests into a format of text description + modal tags. For example, if the original input is "Upload a picture of a seaside landscape and generate a 30-second animation + gentle female voice narration," after standardization, the text description becomes "Generate a 30-second animation + gentle female voice narration." Modal tags are: Input modality: image, Output modality: animation + voice, Style requirement: gentle. Then, the text description is cleaned to remove meaningless characters such as punctuation and special symbols, as well as redundant expressions. Additionally, core entities such as "30 seconds," "seaside landscape," and "gentle female voice" are extracted using the spaCy tool. Finally, for ambiguous expressions, requirements are completed using predefined rules.

[0029] After preprocessing, the recognition and classification process begins. The standardized input text is converted into word vectors, and after the encoder extracts contextual features, the multi-label classification head and entity attribute extraction head process and output the results respectively. The output content is combined as structured task requirement parameters.

[0030] Preferably, the model capability library sets several categories of labels for each model and quantifies the scores for different labels. The labels include task type adaptability, modal input / output support, quality score, resource consumption ratio, inference latency, style adaptability range, accuracy level, fault tolerance, update iteration cycle, and cross-modal collaboration capability.

[0031] The model capability database contains 10 categories of labels, all of which are quantized into scores ranging from 0 to 1 to facilitate the calculation of the matching algorithm. The specific labels and scores are shown in Table 1. Table 1. Tags and Quantitative Scores

[0032] The model capability library sets up a label data update mechanism, in which the resource consumption ratio and inference latency are updated every 50ms based on sampled data, while the quality score, fault tolerance rate, and cross-modal collaboration capability are updated daily. The model update iteration cycle is automatically updated when the version is released, and the task type adaptability and style adaptability range are updated synchronously.

[0033] Preferably, the execution steps of the multi-model dynamic selection and scheduling module 2 are as follows: Preliminary screening: Based on the task requirement parameters, preliminary candidate models with modal input / output support scores ≥ 0.6 and task type fit scores ≥ 0.7 are selected; Secondary screening: Models with a resource consumption ratio score ≥ 0.5 and an inference latency score ≥ 0.6 in the initial candidate models are removed to obtain secondary candidate models; Third ranking: Calculate the comprehensive matching degree of the secondary candidate models. The comprehensive matching degree is calculated by the formula: M=Σ(ω_s×ω_t×L_i), where M is the comprehensive matching degree, ω_s is the task scenario weight coefficient, which is dynamically adjusted according to the task type in the task requirement parameters, such as the ω_s improvement of the style adaptation range in the stylization generation task, ω_t is the basic weight of the label, which is determined by the analytic hierarchy process and reflects the importance of the label, and L_i is the quantitative score of the label, i=1~10. The top 3 models with the best comprehensive matching degree are output as modal candidate models. Final decision: Query the system resource status, which includes the current video memory utilization rate and the current CPU load rate. Calculate the resource availability based on the system resource status. The formula for calculating resource availability is R = 0.7 × (1 - current video memory utilization rate) + 0.3 × (1 - current CPU load rate). Select the candidate model with the highest overall matching degree × resource availability as the final execution model.

[0034] When selecting suitable modal candidate models based on model capability library, matching algorithm, and resource scheduling, a four-step screening method is adopted. In the initial screening, the modal input / output support score and task type fit score of all models are selected according to task type to obtain preliminary candidate models. Then, a second screening is carried out based on the resource consumption ratio score and inference latency score calculated in real time by the model to obtain secondary candidate models that meet the resource and latency requirements. Then, the comprehensive matching degree of the secondary candidate models is calculated according to the preset comprehensive matching degree calculation formula. The top 3 models are output as modal candidate models based on the comprehensive matching degree. These 3 modal candidate models are the models used to handle user-generated requirements. The setting of the label basic weight and task scenario weight coefficient in the matching algorithm is shown in Tables 2 and 3.

[0035] Table 2. Basic Tag Weights (Default Values)

[0036] Table 3 Task Scenario Weighting Coefficients (Example)

[0037] When making the final decision, it is necessary to query the system resource status in real time and calculate the resource availability of the modal candidate models. Finally, the comprehensive matching degree and resource availability are multiplied together, and the final execution model is determined based on the multiplication result. If the resource availability of the first-ranked model is <0.5, the second-ranked model is automatically selected, and the lightweight strategy is triggered to release resources for the first model.

[0038] The following example demonstrates the execution process of the multi-model dynamic selection and scheduling module 2: Task requirements: Based on the text "Summer Iced Drinks", generate a Monet-style illustration and a 30-second promotional voiceover; Preliminary screening (modal support: text → image, text → speech; task adaptation: ad generation) → candidate model pool: GPT-4 (text), SD (image), Tacotron (speech), Midjourney (image), ElevenLabs (speech); Secondary filtering (resource consumption ≤ 50%, inference latency ≤ 2 seconds) → After filtering: SD (image), ElevenLabs (speech), GPT-4 (text); Overall matching degree calculation (stylization task, style adaptation range ω_s=1.5; real-time task, inference latency ω_s=1.4): SD: Matching degree = 0.25×0.9 + 0.2×0.8 + 0.18×0.92 + 0.12×0.6 + 0.1×1.4×0.8 + 0.08×1.5×0.95 + ... = 0.87; ElevenLabs: Match score = 0.83; GPT-4: Match score = 0.85; Resource availability calculation (current GPU memory usage 40%, CPU load 30%) → R = 0.7 × 0.6 + 0.3 × 0.7 = 0.63; Final decision: SD (0.87×0.63=0.5481) → Perform image generation; ElevenLabs (0.83×0.63=0.5229) → Perform speech generation; GPT-4 → Perform text polishing (assist in generating speech scripts).

[0039] Preferably, the execution steps of the cross-modal feature fusion module 3 are as follows: Perform dimensional unification on the intermediate features generated by the modality candidate model; The association weights of different intermediate features are calculated using a cross-modal attention mechanism; Assign the associated weights to the corresponding intermediate features, and then concatenate the features to obtain the global fusion feature vector.

[0040] The 768-dimensional semantic vector output by the text model is mapped to 512 dimensions through a linear layer; the 128×128×256 visual feature map output by the image model is compressed into a 512-dimensional vector through global average pooling; and the 80×100 Mel spectrum output by the speech model is reduced to 512 dimensions through 1D convolution, thus achieving the unification of multimodal feature dimensions.

[0041] When performing cross-modal attention calculation, a modality-feature attention matrix is ​​constructed, and the dot product similarity of different intermediate features is calculated to obtain the association weight. For example, the weight value of "blue" in the text corresponds to "blue pixel area" in the image with a weight value of 0.8, and the weight value of "red pixel area" with a weight value of 0.1. Feature association is strengthened through weighted summation. At the same time, a modality mask is introduced. If a certain modality is missing (such as no voice input), the attention calculation of that modality is automatically blocked to avoid interference.

[0042] Preferably, the basic architecture of the improved CLIP model is as follows: Text encoder: The original CLIP Text Transformer is replaced with a lightweight BERT; Visual encoder: Retains CLIP's ViT-B / 16 architecture and adds an adaptive pooling layer to the output layer; A new speech encoder has been added: 80-dimensional Mel-frequency features are extracted based on Wav2Vec2.0 and encoded into a 512-dimensional speech feature vector through a 2-layer CNN+Transformer.

[0043] After determining the modality candidate model, the user's generation requirements can be processed through the modality candidate model to generate preliminary multimodal content, such as text stories, image sequences, and audio. Then, the improved CLIP model (cross-modal contrastive learning model) is used to solve the semantic alignment problem of multimodal content and check the semantic consistency of the generated content. The improved CLIP model has been modified in terms of basic architecture. The text encoder replaces the original CLIP's TextTransformer with a lightweight BERT, which can reduce the amount of computation. The visual encoder adds an adaptive pooling layer and uniformly outputs 512-dimensional features. In addition, the newly added speech encoder can process the speech modality.

[0044] The pre-training task design for the improved CLIP model is as follows: Main task: Cross-modal contrastive learning (bringing matching samples of text-image, text-speech, and image-speech closer together, and pushing unmatched samples further apart); Auxiliary tasks: cross-modal matching classification (determining whether the input multimodal features are semantically consistent, a binary classification task), attribute alignment regression (such as regression of the correlation between "red" in text and RGB values ​​in an image).

[0045] Pre-trained dataset: Scale: 5 million multimodal alignment samples (2 million text-to-image, 1.5 million text-to-speech, 1.5 million image-to-speech); Data sources: Public datasets (COCO, Flickr30k) + self-made labeled datasets (manually labeled semantically consistent multimodal content).

[0046] Preferably, the execution steps of the multimodal content collaborative generation module 4 are as follows: Modal features are extracted from the initial multimodal content, and semantic feature vectors, visual feature vectors, and speech feature vectors are obtained respectively. Cosine similarity is used to calculate the similarity between any two modal features, and text-image similarity, text-speech similarity, and image-speech similarity are obtained. The consistency score is calculated using a weighted summation formula, which is: Consistency Score = 0.4 × Text-Image Similarity + 0.3 × Text-Speech Similarity + 0.3 × Image-Speech Similarity; If the consistency score is less than the preset verification threshold, it is fed back to the cross-modal feature fusion module 3 to adjust the association weights. If the consistency score is greater than the preset verification threshold, the preliminary multimodal content is sent to the result output module 5.

[0047] After extracting modal features from the initial multimodal content, feature vectors for different modalities can be obtained. Then, the similarity between any two modal features is calculated using the cosine similarity method. A weighted summation method is then used to calculate the consistency score of the three sets of similarity values. The consistency score is compared with a verification threshold of 0.8. Based on statistics from 100,000 validation samples, the false positive rate is low at this threshold. When the consistency score is greater than 0.8, the consistency is considered satisfactory. When the consistency score is less than 0.8, the consistency is considered unsatisfactory. The inconsistent feature localization result is output and fed back to the cross-modal feature fusion module 3 to adjust the association weights of the corresponding modalities until preliminary multimodal content that can pass the consistency verification is generated.

[0048] Preferably, it also includes a resource adaptive scheduling module 6, which is data-connected to the multi-model dynamic selection and scheduling module 2, and is used to perform lightweight adaptation processing on modal candidate models. Its execution steps are as follows: Set lightweight trigger conditions, including GPU memory utilization ≥ 80%. When the CPU load is ≥70%, the task queue length is ≥50, and the real-time task response latency is ≥500ms, lightweight adaptation is started when the modal candidate model meets any lightweight triggering condition and continues for 3 sampling periods. Based on the type of the modal candidate model, a lightweight strategy is selected from a preset lightweight strategy library, and the modal candidate model is subjected to lightweight adaptation processing. The lightweight strategy includes layer pruning, quantization, and knowledge distillation.

[0049] The resource adaptive scheduling module 6 can monitor GPU / CPU resource load in real time. When a certain modality candidate model meets the lightweight triggering condition, it needs to be lightweight adapted. The selected lightweight strategies include layer pruning, quantization, and knowledge distillation. Under the premise of ensuring that the generation quality loss is less than 5%, 20%-30% of the video memory is released. For low-priority tasks, a time-sharing inference strategy is adopted to stagger resource usage and improve the overall throughput of the system.

[0050] Depending on the type of modal candidate model, at least one lightweight strategy can be selected from the preset lightweight strategy library, and the lightweight strategy library and selection logic are shown in Table 4: Table 4 Lightweight Strategy Library and Selection Logic

[0051] Preferably, it also includes a multi-round feedback iteration module 7, which receives user feedback instructions on the generated results, parses the requirements in the feedback instructions and converts them into model adjustment parameters, and adjusts the modal candidate models according to the model adjustment parameters. The multi-round feedback iteration module 7 is data-connected to the multi-model dynamic selection and scheduling module 2.

[0052] The multi-round feedback iteration module 7 can parse user feedback commands and adjust the modal candidate model accordingly, shortening the feedback response time. Through multi-round feedback iteration, there is no need to re-initiate the task, improving user interaction efficiency, supporting personalized style customization, and meeting the generation needs of different scenarios.

[0053] Additionally, there is a dynamic weight adjustment and quality optimization module 8, which monitors the quality indicators of the generated content in real time, including text fluency, image clarity, and speech naturalness. If the quality of a certain modality does not meet the standard, the fusion weight of the corresponding model is automatically adjusted, such as increasing the feature weight of the text model from 0.3 to 0.5, and regenerating the content.

[0054] The technical solution of this invention requires the use of a dedicated AIGC hardware environment for implementation: High-performance computing server: Equipped with multiple GPUs, a single server supports 4-8 GPUs, providing computing power support for model training and inference; the CPU adopts a multi-core high-performance processor, responsible for logical calculations such as task parsing and resource scheduling.

[0055] Distributed storage devices: Employ an all-flash storage cluster to store model capability libraries, pre-trained model parameters, user-generated content, and task logs, supporting petabyte-level storage and high IOPS to ensure efficient model loading and data read / write operations.

[0056] User interaction terminals include PC, mobile, and tablet terminals, supporting text input, image / voice upload, feedback command sending, and previewing and downloading of multimodal generation results; the terminals communicate with the AIGC intelligent agent core service through API interfaces, with latency controlled within 100ms.

[0057] Network equipment: Employs 10 Gigabit Ethernet switches and routers to support high-speed data transmission between computing servers, storage devices, and user terminals, avoiding network bandwidth becoming a bottleneck for multimodal content output.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An AIGC intelligent agent based on the fusion of multiple models, characterized in that, include: The task parsing module is used to receive the generation requirements input by the user, identify the task type through the task classifier, and output structured task requirement parameters. The multi-model dynamic selection and scheduling module is used to match task requirement parameters with model capabilities in a pre-built model capability library through a matching algorithm, filter out a set of suitable candidate models, and schedule the set of candidate models according to the current system resources to obtain modal candidate models. The cross-modal feature fusion module is used to assign different correlation weights to the intermediate features generated by the modal candidate model and then generate a global fusion feature vector. The multimodal content collaborative generation module is used to perform consistency verification on the preliminary multimodal content generated based on the global fusion feature vector using an improved CLIP model; The result output module is used to encapsulate the preliminary multimodal content that has passed the consistency check into a unified format of generated results and output the generated results to the user terminal. The task parsing module, multi-model dynamic selection and scheduling module, cross-modal feature fusion module, multi-modal content collaborative generation module, and result output module are sequentially connected in data.

2. The AIGC agent based on multi-model fusion according to claim 1, characterized in that, The task classifier is built using an improved lightweight BERT model. The basic architecture of the improved lightweight BERT model includes a 6-layer Transformer encoder, a hidden layer dimension of 512, and 8 attention heads. A multi-label classification head and an entity attribute extraction head are added before the output layer to be responsible for task type classification, input modality, output modality combination, and entity attribute extraction, respectively.

3. The AIGC agent based on multi-model fusion according to claim 2, characterized in that, The execution steps of the task classifier include: Receive user input of generation requirements, and convert the generation requirements into a text description + modal tag format, wherein the modal tags include input modality, output modality, and style requirements; Remove meaningless characters and redundant expressions from the text description, and perform entity recognition, annotation, and requirement completion. The text description is converted into word vectors through a word embedding layer, and after the context features are extracted by the Transformer encoder, the task type probability distribution and input / output modality probability distribution are output through the multi-label classification head, and the structured attributes are output through the entity attribute extraction head. Based on the probability distribution of task type and the probability distribution of input / output mode, obtain the input task type, input mode and output mode combination, and convert the task type, input mode, output mode combination and structured input and output into structured task requirement parameters.

4. The AIGC agent based on multi-model fusion according to claim 1, characterized in that, The model capability library sets several categories of labels for each model and quantifies the scores for different labels. The labels include task type adaptability, modal input / output support, quality score, resource consumption ratio, inference latency, style adaptability range, accuracy level, fault tolerance, update iteration cycle, and cross-modal collaboration capability.

5. The AIGC agent based on multi-model fusion according to claim 4, characterized in that, The execution steps of the multi-model dynamic selection and scheduling module are as follows: Preliminary screening: Based on the task requirement parameters, query preliminary candidate models whose modal input / output support score and task type suitability score are greater than the preliminary screening threshold; Secondary screening: Models with resource consumption ratio scores and inference latency scores greater than the secondary screening threshold in the initial candidate models are removed to obtain secondary candidate models; Third ranking: Calculate the comprehensive matching degree of the secondary candidate models. The comprehensive matching degree is calculated as follows: M=Σ(ω_s×ω_t×L_i), where M is the comprehensive matching degree, ω_s is the task scenario weight coefficient, which is dynamically adjusted according to the task type in the task requirement parameters, ω_t is the basic weight of the label, and Li is the quantitative score of the label. The top 3 models with the comprehensive matching degree are output as modal candidate models. Final decision: Query the system resource status, which includes the current video memory utilization rate and the current CPU load rate. Calculate the resource availability based on the system resource status. The formula for calculating resource availability is R = 0.7 × (1 - current video memory utilization rate) + 0.3 × (1 - current CPU load rate). Select the candidate model with the highest overall matching degree × resource availability as the final execution model.

6. The AIGC agent based on multi-model fusion according to claim 1, characterized in that, The execution steps of the cross-modal feature fusion module are as follows: Perform dimensional unification on the intermediate features generated by the modality candidate model; The association weights of different intermediate features are calculated using a cross-modal attention mechanism; Assign the associated weights to the corresponding intermediate features, and then concatenate the features to obtain the global fusion feature vector.

7. The AIGC agent based on multi-model fusion according to claim 1, characterized in that, The basic architecture of the improved CLIP model is as follows: Text encoder: The original CLIP Text Transformer is replaced with a lightweight BERT; Visual encoder: Retains CLIP's ViT-B / 16 architecture and adds an adaptive pooling layer to the output layer; A new speech encoder has been added: 80-dimensional Mel-frequency features are extracted based on Wav2Vec2.0 and encoded into a 512-dimensional speech feature vector through a 2-layer CNN+Transformer.

8. The AIGC agent based on multi-model fusion according to claim 7, characterized in that, The execution steps of the multimodal content collaborative generation module are as follows: Modal features are extracted from the initial multimodal content, and semantic feature vectors, visual feature vectors, and speech feature vectors are obtained respectively. Cosine similarity is used to calculate the similarity between any two modal features, and text-image similarity, text-speech similarity, and image-speech similarity are obtained. The consistency score is calculated using a weighted summation formula, which is: Consistency Score = 0.4 × Text-Image Similarity + 0.3 × Text-Speech Similarity + 0.3 × Image-Speech Similarity; If the consistency score is less than the preset verification threshold, the feedback is sent to the cross-modal feature fusion module to adjust the association weights. If the consistency score is greater than the preset verification threshold, the preliminary multimodal content is sent to the result output module.

9. The AIGC agent based on multi-model fusion according to claim 1, characterized in that, It also includes a resource adaptive scheduling module, which is data-connected to the multi-model dynamic selection and scheduling module. This module is used to perform lightweight adaptation processing on modal candidate models, and its execution steps are as follows: Set lightweight trigger conditions, including GPU memory utilization ≥ 80%. When the CPU load is ≥70%, the task queue length is ≥50, and the real-time task response latency is ≥500ms, lightweight adaptation is started when the modal candidate model meets any lightweight triggering condition and continues for 3 sampling periods. Based on the type of the modal candidate model, a lightweight strategy is selected from a preset lightweight strategy library, and the modal candidate model is subjected to lightweight adaptation processing. The lightweight strategy includes layer pruning, quantization, and knowledge distillation.

10. The AIGC agent based on multi-model fusion according to claim 1, characterized in that, It also includes a multi-round feedback iteration module, which receives user feedback instructions on the generated results, parses the requirements in the feedback instructions and converts them into model adjustment parameters, and adjusts the modal candidate models according to the model adjustment parameters. The multi-round feedback iteration module is data-connected to the multi-model dynamic selection and scheduling module.

Images