A multi-modal content collaborative generation method
By constructing a unified semantic space for multimodal use and dynamically allocating computing power, the problems of semantic disconnect and logical coherence between modalities in multimodal generation technology are solved, achieving semantic consistency and efficient adjustment of multimodal content and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING DINGDIAN PERSPECTIVE PUBLIC RELATIONS CONSULTING CO LTD
- Filing Date
- 2025-10-22
- Publication Date
- 2026-05-26
AI Technical Summary
Existing multimodal generation technologies suffer from problems such as semantic disconnect between modalities, poor content consistency, weak logical coherence, and low adjustment efficiency, making it difficult to meet the needs of semantic consistency, content relevance, and ease of adjustment in practical applications.
By constructing a unified multimodal semantic space, establishing associations between modal semantic tags, monitoring the system's computing power status in real time, dynamically allocating computing power, providing a visual interactive interface that supports user modification, and verifying and optimizing through a multimodal quality closed-loop evaluation module, the semantic consistency and logical coherence of multimodal content are achieved.
It achieves strong semantic association of multimodal content, improves content consistency and logical coherence, enhances the interactive convenience and adjustment efficiency of content modification, and meets the needs of practical applications.
Smart Images
Figure CN121328772B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence multimodal generation technology, specifically a method for collaborative generation of multimodal content. Background Technology
[0002] With the rapid development of AIGC (Generative Artificial Intelligence) technology, the demand for multimodal content (integrating various types of content such as text, images, audio, and video) is exploding. For example, e-commerce platforms need to generate combined content of "product text description + promotional images + explanatory audio"; educational institutions need to develop courseware of "knowledge point text + animated demonstrations + voice interpretation"; and self-media creators need to produce short video materials of "script text + scene images + background audio".
[0003] Current multimodal generation technologies mostly adopt a "single-modal independent generation + post-assembly" approach. This involves separately calling text generation models (such as the GPT series), image generation models (such as StableDiffusion), and audio generation models (such as AudioLDM) to generate single-modal content, which is then manually or using simple tools to stitch it into multimodal content. While this approach can achieve the stacking of multimodal content, it struggles to meet the demands of practical applications for semantic consistency, content relevance, and ease of adjustment, gradually revealing several technical pain points, specifically the following issues:
[0004] I. Semantic disconnect between modalities and poor content consistency
[0005] Existing methods lack a unified semantic coordination mechanism, leading to semantic conflicts between different modalities. For example, when an e-commerce platform uses existing tools to generate promotional content for "waterproof sports shoes," the text description is "suitable for outdoor walking in the rain, the upper is waterproof and leak-proof," but the image generation model outputs "sports shoes for a sunny gym setting," and the audio explanation mentions "suitable for indoor sports." These three modalities are semantically fragmented, preventing users from accurately understanding the core selling points of the product through multimodal content, resulting in a decrease in product conversion rates.
[0006] II. Modal content misalignment and weak logical coherence
[0007] In educational courseware generation scenarios, existing tools are prone to "step misalignment" problems. For example, the text section of a math courseware clearly states that "the steps to solve a quadratic equation are: ① remove parentheses → ② transpose terms → ③ combine like terms → ④ solve," but the animation generation module outputs a demonstration process of "① remove parentheses → ② combine like terms → ③ transpose terms → ④ solve," while the audio narration follows the animation steps. This leads to a confusion in the logical order of the text, animation, and audio, which can easily cause cognitive biases for students and affect the teaching effect.
[0008] Third, the adjustment process is inefficient and lacks interactivity.
[0009] When a user needs to modify a modal content, existing methods require that modal content be regenerated independently, and cannot automatically synchronize and adjust other related modalities. For example, after a self-media creator generates short video material using existing tools, they may find that "viewing flowers in the park at 9 am" in the text script needs to be changed to "viewing flowers in the park at 3 pm." This requires regenerating the text script, manually adjusting the image (replacing it with the park scene at 3 pm), and re-recording the audio (modifying the time description). The entire process takes 1-2 hours and is prone to secondary problems where "the new text and new image times match, but the new audio and new image times are misaligned again," severely impacting creative efficiency.
[0010] Based on the above, a multimodal content collaborative generation method is invented. Summary of the Invention
[0011] To solve the above-mentioned technical problems, the present invention provides the following technical solution:
[0012] A multimodal content collaborative generation method includes the following specific steps:
[0013] S1: Receive the user's input multimodal generation requirements and collect the initial features of each modality;
[0014] S2: Based on the scenario to which the requirement belongs, automatically integrate industry compliance rules, user historical preferences, and cross-scenario format requirements to transform non-standard user requirements into structured and executable constraint adaptation parameters, providing clear boundary conditions for subsequent semantic tag generation.
[0015] S3: Through three-dimensional classification and storage, intelligent semantic similarity retrieval, and adaptive optimization of historical resources, high-quality semantic tags, association rules, and modal fragments generated in the past are transformed into reusable resources;
[0016] S4: First, based on the pre-trained cross-modal semantic model, and combined with the constraint report and reuse resource package, construct a multimodal unified semantic space. Then, map the initial features of each modality output by the modal feature acquisition module to this space, generate unified semantic labels, and establish the association relationship between the semantic labels of each modality.
[0017] S5: Real-time monitoring of system computing power status, combined with the differences in resource requirements of each modality generator and the urgency of user needs, through dynamic computing power allocation algorithm and priority ranking of generated tasks, realizes the scheduling logic of tilting computing power towards high-demand modalities and prioritizing the execution of urgent tasks;
[0018] S6: Based on semantic tags, relationships, constraint adaptation reports and scheduling strategies, schedule each modality generator and verify the matching degree between the generated content and the unified semantic tags and constraints in real time; if the matching degree is lower than the preset threshold, it is fed back to the corresponding modality generator for adjustment until all modality content meets the semantic consistency requirements.
[0019] S7: Multimodal quality closed-loop assessment module, used to perform dual verification of collaboratively generated multimodal content from both professional and user dimensions through quantitative assessment based on professional standards, collection of user preview feedback, and generation of problem iteration optimization suggestions; triggers secondary generation for substandard content.
[0020] S8: Provides a visual interactive interface that allows users to modify any modal content; when a user triggers a modification command, it will automatically regenerate constraint parameters and update semantic tags, and at the same time trigger the multimodal resource management and reuse module to re-retrieve matching resources, and then the collaborative generation control module will schedule other related modal generators to adjust synchronously.
[0021] S9: Integrate and output the collaboratively generated multimodal content according to the user's required format, generate a semantic consistency report, and store the high-quality generated content in the multimodal resource database and update reusable resources.
[0022] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the specific steps of step S2 are as follows:
[0023] S21: Based on the compliance rule library of various industries, automatically identify the scenario to which the requirement belongs and extract the corresponding compliance rules;
[0024] S22: Connect to the user's historical generation record database, extract the user's past style preferences and format preferences, and generate personalized constraint parameters;
[0025] S23: Based on the target scenario in the requirements, match the preset format standard and generate format constraint parameters;
[0026] S24: Integrate the three types of constraint parameters to generate a constraint adaptation report.
[0027] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the specific steps of S3 are as follows:
[0028] S31: Connects to a multimodal resource database, storing historically generated semantic tags, association rules, and high-quality modal fragments according to three dimensions: scenario, theme, and constraint;
[0029] S32: Based on the demand topic and constraint parameters, retrieve historical resources in the database with a matching degree of ≥80% using a semantic similarity algorithm;
[0030] S33: Adaptively adjust the retrieved historical resources to generate a reusable resource package.
[0031] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the specific steps of step S5 are as follows:
[0032] S51: Real-time monitoring of system computing power, memory usage, and resource requirements of each modality generator;
[0033] S52: Based on monitoring data, computing power is allocated using a demand-resource matching algorithm;
[0034] S53: Based on the urgency of user needs and the time required to generate modalities, generate tasks in order.
[0035] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the specific steps of S7 are as follows:
[0036] S71: Quantitatively evaluate the professional quality of generated content based on professional standard libraries for each scenario;
[0037] S72: Supports users to mark quality issues during the preview stage, and automatically associates them with the corresponding modal generator;
[0038] S73: When the professional quality score is less than 80 points or there are user feedback issues, generate optimization suggestions and feed them back to the collaborative generation control module to trigger secondary generation.
[0039] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, a multimodal content collaborative generation system includes:
[0040] The modal feature acquisition module is used to receive the user's input multimodal generation requirements and acquire the initial features of each modality;
[0041] The multi-dimensional constraint adaptation module is used to automatically integrate industry compliance rules, user historical preferences, and cross-scenario format requirements based on the scenario to which the requirement belongs. It transforms non-standardized user requirements into structured and executable constraint adaptation parameters, providing clear boundary conditions for subsequent semantic tag generation.
[0042] The multimodal resource management and reuse module is used to transform previously generated high-quality semantic tags, association rules, and modal fragments into reusable resources through three-dimensional classification and storage, intelligent semantic similarity retrieval, and adaptive optimization of historical resources.
[0043] The semantic unified mapping module is used to first construct a multimodal unified semantic space based on a pre-trained cross-modal semantic model, combined with constraint reports and reused resource packages. Then, the initial features of each modality output by the modal feature acquisition module are mapped to this space to generate unified semantic labels and establish the association relationship between the semantic labels of each modality.
[0044] The cross-modal resource scheduling optimization module is used to monitor the system's computing power status in real time. Combining the differences in resource requirements of each modal generator with the urgency of user needs, it implements a scheduling logic that tilts computing power towards high-demand modalities and prioritizes urgent tasks through dynamic computing power allocation algorithms and task priority ranking.
[0045] The collaborative generation control module is used to schedule each modality generator according to semantic tags, association relationships, constraint adaptation reports and scheduling strategies, and to verify the matching degree between the generated content and the unified semantic tags and constraints in real time. If the matching degree is lower than the preset threshold, it will be fed back to the corresponding modality generator for adjustment until all modality content meets the semantic consistency requirements.
[0046] The multimodal quality closed-loop assessment module is used to perform dual verification of collaboratively generated multimodal content from both professional and user dimensions through quantitative assessment based on professional standards, collection of user preview feedback, and generation of problem iteration optimization suggestions. Substandard content is triggered for secondary generation.
[0047] The dynamic interaction module provides a visual interactive interface that allows users to modify any modal content. When a user triggers a modification command, the constraint parameters are automatically regenerated and the semantic tags are updated. At the same time, the multimodal resource management and reuse module is triggered to re-retrieve matching resources, and then the collaborative generation control module schedules other related modal generators to adjust synchronously.
[0048] The content output module is used to integrate and output the collaboratively generated multimodal content according to the user's required format, generate a semantic consistency report, store high-quality generated content to the multimodal resource database, and update reusable resources.
[0049] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the multi-dimensional constraint adaptation module includes:
[0050] The industry compliance constraint module is used to automatically identify the scenario to which the requirement belongs and extract the corresponding compliance rules based on the compliance rule library of each industry.
[0051] The user preference constraint module is used to connect to the user's historical generation record database, extract the user's past style preferences and format preferences, and generate personalized constraint parameters.
[0052] Cross-scenario format constraint module: used to match preset format standards and generate format constraint parameters according to the target scenario in the requirements;
[0053] The integration module is used to integrate the three types of constraint parameters and generate a constraint adaptation report.
[0054] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the multimodal resource management and reuse module includes:
[0055] The resource classification and storage module is used to connect to a multimodal resource database and store historically generated semantic tags, association rules, and high-quality modal fragments according to three dimensions: scenario, theme, and constraint.
[0056] The intelligent search and matching module is used to retrieve historical resources in the database with a matching degree of ≥80% based on the demand topic and constraint parameters using a semantic similarity algorithm;
[0057] The reuse optimization module is used to adaptively adjust the retrieved historical resources and generate reuse resource packages.
[0058] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the cross-modal resource scheduling optimization module includes:
[0059] The resource monitoring module is used to monitor the system's computing power, memory usage, and resource requirements of each modal generator in real time.
[0060] The dynamic allocation module is used to allocate computing power based on monitoring data and a demand-resource matching algorithm.
[0061] The priority sorting module is used to sort and generate tasks according to the urgency of user needs and the time required to generate modalities.
[0062] As a preferred embodiment of the multimodal content collaborative generation method described in this invention, the multimodal quality closed-loop evaluation module includes:
[0063] The professional domain quality module is used to quantitatively evaluate the professional quality of generated content based on professional standard libraries for various scenarios.
[0064] The user experience feedback module is used to support users in marking quality issues during the preview stage, and automatically associates them with the corresponding modal generator;
[0065] The iterative optimization module generates optimization suggestions when the professional quality score is less than 80 points or when there are user feedback issues. These suggestions are then fed back to the collaborative generation control module to trigger secondary generation.
[0066] Compared with existing technologies:
[0067] 1. By constructing a unified multimodal semantic space, mapping the initial features of each modality to this space and establishing the association relationship of modal semantic labels, it has the advantages of achieving strong semantic association of multimodal content, avoiding semantic conflicts between different modalities, and improving content consistency.
[0068] 2. By using a collaborative generation control module to schedule each modal generator based on a unified semantic tag and to verify the matching degree between the generated content and the semantic tag in real time, it has the advantages of achieving logical coherence in multimodal content generation, eliminating misalignment of content steps between modalities, and ensuring the logical integrity of the content.
[0069] 3. A visual modification interface is provided through a dynamic interaction module. When the user triggers a modification command, the semantic tags are automatically updated synchronously, and the associated modality generator is scheduled to adjust synchronously. This has the advantages of achieving efficient synchronous adjustment of multimodal content, reducing manual operation, and improving the convenience of content modification interaction. Attached Figure Description
[0070] Figure 1 This is a schematic diagram of the overall framework of the present invention;
[0071] Figure 2 This is a schematic diagram of the multi-dimensional constraint adaptation module framework of the present invention;
[0072] Figure 3 This is a schematic diagram of the multimodal resource management and reuse module framework of the present invention;
[0073] Figure 4 This is a schematic diagram of the cross-modal resource scheduling optimization module framework of the present invention;
[0074] Figure 5 This is a schematic diagram of the framework of the multimodal quality closed-loop evaluation module of the present invention. Detailed Implementation
[0075] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0076] This invention provides a method for collaborative generation of multimodal content, comprising the following specific steps:
[0077] S1: Receive the user's input multimodal generation requirements and collect the initial features of each modality;
[0078] S2: Based on the scenario to which the requirement belongs, automatically integrate industry compliance rules, user historical preferences, and cross-scenario format requirements to transform non-standard user requirements into structured and executable constraint adaptation parameters, providing clear boundary conditions for subsequent semantic tag generation.
[0079] The specific steps of S2 are as follows:
[0080] S21: Based on the compliance rule library of various industries, automatically identify the scenario to which the requirement belongs and extract the corresponding compliance rules;
[0081] S22: Connect to the user's historical generation record database, extract the user's past style preferences and format preferences, and generate personalized constraint parameters;
[0082] S23: Based on the target scenario in the requirements, match the preset format standard and generate format constraint parameters;
[0083] S24: Integrate the three types of constraint parameters to generate a constraint adaptation report;
[0084] S3: Through three-dimensional classification and storage, intelligent semantic similarity retrieval, and adaptive optimization of historical resources, high-quality semantic tags, association rules, and modal fragments generated in the past are transformed into reusable resources;
[0085] The specific steps of S3 are as follows:
[0086] S31: Connects to a multimodal resource database, storing historically generated semantic tags, association rules, and high-quality modal fragments according to three dimensions: scenario, theme, and constraint;
[0087] S32: Based on the demand topic and constraint parameters, retrieve historical resources in the database with a matching degree of ≥80% using a semantic similarity algorithm;
[0088] S33: Adaptively adjust the retrieved historical resources to generate a reused resource package;
[0089] S4: First, based on the pre-trained cross-modal semantic model, and combined with the constraint report and reuse resource package, construct a multimodal unified semantic space. Then, map the initial features of each modality output by the modal feature acquisition module to this space, generate unified semantic labels, and establish the association relationship between the semantic labels of each modality.
[0090] S5: Real-time monitoring of system computing power status, combined with the differences in resource requirements of each modality generator and the urgency of user needs, through dynamic computing power allocation algorithm and priority ranking of generated tasks, realizes the scheduling logic of tilting computing power towards high-demand modalities and prioritizing the execution of urgent tasks;
[0091] The specific steps of S5 are as follows:
[0092] S51: Real-time monitoring of system computing power, memory usage, and resource requirements of each modality generator;
[0093] S52: Based on monitoring data, computing power is allocated using a demand-resource matching algorithm;
[0094] S53: Generate tasks in order of urgency of user needs and modality generation time;
[0095] S6: Based on semantic tags, relationships, constraint adaptation reports and scheduling strategies, schedule each modality generator and verify the matching degree between the generated content and the unified semantic tags and constraints in real time; if the matching degree is lower than the preset threshold, it is fed back to the corresponding modality generator for adjustment until all modality content meets the semantic consistency requirements.
[0096] S7: Multimodal quality closed-loop assessment module, used to perform dual verification of collaboratively generated multimodal content from both professional and user dimensions through quantitative assessment based on professional standards, collection of user preview feedback, and generation of problem iteration optimization suggestions; triggers secondary generation for substandard content.
[0097] The specific steps of S7 are as follows:
[0098] S71: Quantitatively evaluate the professional quality of generated content based on professional standard libraries for each scenario;
[0099] S72: Supports users to mark quality issues during the preview stage, and automatically associates them with the corresponding modal generator;
[0100] S73: When the professional quality score is less than 80 points or there are user feedback issues, generate optimization suggestions and feed them back to the collaborative generation control module to trigger secondary generation;
[0101] S8: Provides a visual interactive interface that allows users to modify any modal content; when a user triggers a modification command, it will automatically regenerate constraint parameters and update semantic tags, and at the same time trigger the multimodal resource management and reuse module to re-retrieve matching resources, and then the collaborative generation control module will schedule other related modal generators to adjust synchronously.
[0102] S9: Integrate and output the collaboratively generated multimodal content according to the user's required format, generate a semantic consistency report, and store the high-quality generated content in the multimodal resource database and update reusable resources.
[0103] Please see Figures 1-5 A multimodal content collaborative generation system, comprising:
[0104] The modal feature acquisition module is used to receive user input of multimodal generation requirements (including target modal type, content theme, and style requirements) and acquire initial features of each modality (such as semantic vectors of text, visual features of images (color, scene, objects), and acoustic features of audio (pitch, rhythm, scene sound effects)).
[0105] The multi-dimensional constraint adaptation module is used to automatically integrate industry compliance rules, user historical preferences, and cross-scenario format requirements based on the scenario to which the requirement belongs. It transforms non-standardized user requirements into structured and executable constraint adaptation parameters, providing clear boundary conditions for subsequent semantic tag generation.
[0106] The multimodal resource management and reuse module is used to transform previously generated high-quality semantic tags, association rules, and modal fragments into reusable resources through three-dimensional classification and storage, intelligent semantic similarity retrieval, and adaptive optimization of historical resources. This provides a reference template for the semantic mapping process and reduces the cost of repeated calculations and building from scratch.
[0107] The semantic unified mapping module is used to first construct a multimodal unified semantic space based on a pre-trained cross-modal semantic model (such as CLIP-4 extended version) and combined with constraint reports and reused resource packages. Then, the initial features of each modality output by the modal feature acquisition module are mapped to this space to generate unified semantic labels (such as the semantic labels corresponding to the topic "flower viewing in the park at 3 pm": [time: 15:00, scene: park, action: flower viewing, style: fresh]). The module also establishes the association relationship between the semantic labels of each modality (such as "time: 15:00" associating the image with "warm-toned lighting" and the audio with "evening birdsong").
[0108] The cross-modal resource scheduling optimization module is used to monitor the system's computing power status in real time. Combining the differences in resource requirements of each modal generator with the urgency of user needs, it implements a scheduling logic that tilts computing power towards high-demand modalities and prioritizes urgent tasks through dynamic computing power allocation algorithms and task priority ranking.
[0109] The collaborative generation control module is used to schedule each modal generator (text generator, image generator, audio generator, and video generator) based on semantic tags, associations, constraint adaptation reports, and scheduling strategies, and to verify the matching degree between the generated content and the unified semantic tags and constraints in real time (such as verifying whether the image contains "warm-toned lighting" or "park scene", and whether the audio contains "evening birdsong"). If the matching degree is lower than a preset threshold (such as 85%), it will be fed back to the corresponding modal generator for adjustment until all modal content meets the semantic consistency requirements.
[0110] The multimodal quality closed-loop assessment module is used to perform dual verification of collaboratively generated multimodal content from both professional and user dimensions through quantitative assessment based on professional standards, collection of user preview feedback, and generation of problem iteration optimization suggestions. Substandard content is triggered for secondary generation.
[0111] The dynamic interaction module provides a visual interactive interface, allowing users to modify content in any modality. When a user triggers a modification command, the constraint parameters are automatically regenerated and the semantic tags are updated. At the same time, the multimodal resource management and reuse module is triggered to re-retrieve matching resources. Then, the collaborative generation control module schedules other related modal generators to adjust synchronously (e.g., modifying the text "3 PM" to "9 AM" automatically updates the image to "cool-toned morning light" and the audio to "early morning insect chirping").
[0112] The content output module is used to integrate and output collaboratively generated multimodal content (text, images, audio, and video) according to user-required formats (such as e-commerce promotional packages, courseware formats, and short video formats), and generate a semantic consistency report (including matching data for each modality). At the same time, it stores high-quality generated content (score ≥ 90 points) to the multimodal resource database and updates reusable resources.
[0113] The multi-dimensional constraint adaptation module includes:
[0114] The industry compliance constraint module is used to automatically identify the scenario to which the requirement belongs (such as "e-commerce" or "education") based on the compliance rule library of various industries (such as the prohibited word library of the Advertising Law in the e-commerce industry, the knowledge point accuracy verification library in the education industry, and the content review standard library in the self-media industry), and extract the corresponding compliance rules (such as filtering prohibited words such as "best" and "first" in the e-commerce scenario, and verifying the accuracy of "steps for solving a quadratic equation in one variable" in the education scenario).
[0115] The user preference constraint module is used to connect to the user's historical generation record database, extract the user's past style preferences (such as an e-commerce user's habit of "concise and professional" text style, or a teacher user's common courseware image style of "combining animation and real scene") and format preferences (such as a self-media user's fixed output of "16:9 horizontal video + 500-word script"), and generate personalized constraint parameters.
[0116] Cross-scene format constraint module: Used to match preset format standards (such as "e-commerce detail page", "classroom courseware", "short video platform") to generate format constraint parameters based on the target scene in the requirements (such as "e-commerce detail page", "classroom courseware", "short video platform").
[0117] The integration module is used to integrate three types of constraint parameters and generate a constraint adaptation report (such as [compliance constraints: disable the word "most"; user preferences: concise and professional text; format constraints: white background and no watermark for images]).
[0118] The multimodal resource management and reuse module includes:
[0119] The resource classification and storage module is used to connect to a multimodal resource database. It stores historically generated semantic tags, association rules, and high-quality modal fragments (such as high-conversion e-commerce text templates and highly adaptable courseware animation clips) according to three dimensions: scenario (e-commerce / education / self-media), theme (waterproof sneakers / quadratic equations), and constraints (compliance / preference / format).
[0120] The intelligent search and matching module is used to retrieve historical resources with a matching degree of ≥80% in the database based on the demand theme (such as "waterproof sports shoe promotion") and constraint parameters (such as "e-commerce scenario + white background image") and semantic similarity algorithm (such as cosine similarity). These resources include semantic tags for "waterproof running shoes" and white background promotional image templates.
[0121] The reuse optimization module is used to adaptively adjust the retrieved historical resources (such as replacing the semantic tag of "waterproof running shoes" with "waterproof sports shoes" while retaining the core parameters of "white background + waterproof test scenario"), generate a reuse resource package (including reference semantic tags and association rule templates), and output it to the semantic unified mapping module to help it quickly build a unified semantic space.
[0122] The cross-modal resource scheduling optimization module includes:
[0123] The resource monitoring module is used to monitor the system's computing power (CPU / GPU utilization), memory usage, and resource requirements of each modal generator in real time (e.g., video generation requires high GPU resources, and text generation requires high CPU resources).
[0124] The dynamic allocation module is used to allocate computing power based on monitoring data using a demand-resource matching algorithm (e.g., automatically increasing GPU utilization to 70% when generating video, and prioritizing CPU resources when generating text).
[0125] The priority sorting module is used to sort generation tasks based on the urgency of user needs (such as "instant generation" and "timed output") and the time taken to generate modalities (such as video generation time > image generation). This prioritizes image generation for urgent needs over video generation for non-urgent needs, thus avoiding generation lag under high load scenarios.
[0126] The multimodal quality closed-loop evaluation module includes:
[0127] The professional domain quality module is used to quantitatively evaluate the professional quality of generated content based on professional standard libraries for various scenarios (such as "conversion rate-oriented indicators" (keyword density, selling point prominence) for e-commerce content and "knowledge point depth indicators" (logical completeness, difficulty suitability) for educational content).
[0128] The user experience feedback module supports users in marking quality issues (such as "image color deviation" and "audio noise") during the preview stage, and automatically associates them with the corresponding modality generator.
[0129] The iterative optimization module generates optimization suggestions (such as "increase the keyword density of 'waterproof' in the text to 5%" or "adjust the color saturation of the image to 60%) when the professional quality score is less than 80 or there are user feedback issues. These suggestions are then fed back to the collaborative generation control module to trigger secondary generation.
[0130] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. A method for collaborative generation of multimodal content, characterized in that, The specific steps are as follows: S1: Receive the user's input multimodal generation requirements and collect the initial features of each modality; S2: Based on the scenario to which the requirement belongs, automatically integrate industry compliance rules, user historical preferences, and cross-scenario format requirements to transform non-standard user requirements into structured and executable constraint adaptation parameters, providing clear boundary conditions for subsequent semantic tag generation. S3: Through three-dimensional classification and storage, intelligent semantic similarity retrieval, and adaptive optimization of historical resources, high-quality semantic tags, association rules, and modal fragments generated in the past are transformed into reusable resources; S4: First, based on the pre-trained cross-modal semantic model, and combined with the constraint report and reuse resource package, construct a multimodal unified semantic space. Then, map the initial features of each modality output by the modal feature acquisition module to this space, generate unified semantic labels, and establish the association relationship between the semantic labels of each modality. S5: Real-time monitoring of system computing power status, combined with the differences in resource requirements of each modality generator and the urgency of user needs, through dynamic computing power allocation algorithm and priority ranking of generated tasks, realizes the scheduling logic of tilting computing power towards high-demand modalities and prioritizing the execution of urgent tasks; S6: Based on semantic tags, relationships, constraint adaptation reports and scheduling strategies, schedule each modality generator and verify the matching degree between the generated content and the unified semantic tags and constraints in real time; if the matching degree is lower than the preset threshold, it is fed back to the corresponding modality generator for adjustment until all modality content meets the semantic consistency requirements. S7: Multimodal quality closed-loop assessment module, used to perform dual verification of collaboratively generated multimodal content from both professional and user dimensions through quantitative assessment based on professional standards, collection of user preview feedback, and generation of problem iteration optimization suggestions; triggers secondary generation for substandard content. S8: Provides a visual interactive interface that allows users to modify any modal content; when a user triggers a modification command, it will automatically regenerate constraint parameters and update semantic tags, and at the same time trigger the multimodal resource management and reuse module to re-retrieve matching resources, and then the collaborative generation control module will schedule other related modal generators to adjust synchronously. S9: Integrate and output the collaboratively generated multimodal content according to the user's required format, generate a semantic consistency report, and store the high-quality generated content in the multimodal resource database and update reusable resources.
2. The multimodal content collaborative generation method according to claim 1, characterized in that, The specific steps of S2 are as follows: S21: Based on the compliance rule library of various industries, automatically identify the scenario to which the requirement belongs and extract the corresponding compliance rules; S22: Connect to the user's historical generation record database, extract the user's past style preferences and format preferences, and generate personalized constraint parameters; S23: Based on the target scenario in the requirements, match the preset format standard and generate format constraint parameters; S24: Integrate the three types of constraint parameters to generate a constraint adaptation report.
3. The multimodal content collaborative generation method according to claim 1, characterized in that, The specific steps of S3 are as follows: S31: Connects to a multimodal resource database, storing historically generated semantic tags, association rules, and high-quality modal fragments according to three dimensions: scenario, theme, and constraint; S32: Based on the demand topic and constraint parameters, retrieve historical resources in the database with a matching degree of ≥80% using a semantic similarity algorithm; S33: Adaptively adjust the retrieved historical resources to generate a reusable resource package.
4. The multimodal content collaborative generation method according to claim 1, characterized in that, The specific steps of S5 are as follows: S51: Real-time monitoring of system computing power, memory usage, and resource requirements of each modality generator; S52: Based on monitoring data, computing power is allocated using a demand-resource matching algorithm; S53: Based on the urgency of user needs and the time required to generate modalities, generate tasks in order.
5. The multimodal content collaborative generation method according to claim 1, characterized in that, The specific steps of S7 are as follows: S71: Quantitatively evaluate the professional quality of generated content based on professional standard libraries for each scenario; S72: Supports users to mark quality issues during the preview stage, and automatically associates them with the corresponding modal generator; S73: When the professional quality score is less than 80 points or there are user feedback issues, generate optimization suggestions and feed them back to the collaborative generation control module to trigger secondary generation.
6. A multimodal content collaborative generation system, characterized in that, include: The modal feature acquisition module is used to receive the user's input multimodal generation requirements and acquire the initial features of each modality; The multi-dimensional constraint adaptation module is used to automatically integrate industry compliance rules, user historical preferences, and cross-scenario format requirements based on the scenario to which the requirement belongs. It transforms non-standardized user requirements into structured and executable constraint adaptation parameters, providing clear boundary conditions for subsequent semantic tag generation. The multimodal resource management and reuse module is used to transform previously generated high-quality semantic tags, association rules, and modal fragments into reusable resources through three-dimensional classification and storage, intelligent semantic similarity retrieval, and adaptive optimization of historical resources. The semantic unified mapping module is used to first construct a multimodal unified semantic space based on a pre-trained cross-modal semantic model, combined with constraint reports and reused resource packages. Then, the initial features of each modality output by the modal feature acquisition module are mapped to this space to generate unified semantic labels and establish the association relationship between the semantic labels of each modality. The cross-modal resource scheduling optimization module is used to monitor the system's computing power status in real time. Combining the differences in resource requirements of each modal generator with the urgency of user needs, it implements a scheduling logic that tilts computing power towards high-demand modalities and prioritizes urgent tasks through dynamic computing power allocation algorithms and task priority ranking. The collaborative generation control module is used to schedule each modality generator according to semantic tags, association relationships, constraint adaptation reports and scheduling strategies, and to verify the matching degree between the generated content and the unified semantic tags and constraints in real time. If the matching degree is lower than the preset threshold, it will be fed back to the corresponding modality generator for adjustment until all modality content meets the semantic consistency requirements. The multimodal quality closed-loop assessment module is used to perform dual verification of collaboratively generated multimodal content from both professional and user dimensions through quantitative assessment based on professional standards, collection of user preview feedback, and generation of problem iteration optimization suggestions. Substandard content is triggered for secondary generation. The dynamic interaction module provides a visual interactive interface that allows users to modify any modal content. When a user triggers a modification command, the constraint parameters are automatically regenerated and the semantic tags are updated. At the same time, the multimodal resource management and reuse module is triggered to re-retrieve matching resources, and then the collaborative generation control module schedules other related modal generators to adjust synchronously. The content output module is used to integrate and output the collaboratively generated multimodal content according to the user's required format, generate a semantic consistency report, store high-quality generated content to the multimodal resource database, and update reusable resources.
7. A multimodal content collaborative generation system according to claim 6, characterized in that, The multi-dimensional constraint adaptation module includes: The industry compliance constraint module is used to automatically identify the scenario to which the requirement belongs and extract the corresponding compliance rules based on the compliance rule library of each industry. The user preference constraint module is used to connect to the user's historical generation record database, extract the user's past style preferences and format preferences, and generate personalized constraint parameters. Cross-scenario format constraint module: used to match preset format standards and generate format constraint parameters according to the target scenario in the requirements; The integration module is used to integrate the three types of constraint parameters and generate a constraint adaptation report.
8. A multimodal content collaborative generation system according to claim 6, characterized in that, The multimodal resource management and reuse module includes: The resource classification and storage module is used to connect to a multimodal resource database and store historically generated semantic tags, association rules, and high-quality modal fragments according to three dimensions: scenario, theme, and constraint. The intelligent search and matching module is used to retrieve historical resources in the database with a matching degree of ≥80% based on the demand topic and constraint parameters using a semantic similarity algorithm; The reuse optimization module is used to adaptively adjust the retrieved historical resources and generate reuse resource packages.
9. A multimodal content collaborative generation system according to claim 6, characterized in that, The cross-modal resource scheduling optimization module includes: The resource monitoring module is used to monitor the system's computing power, memory usage, and resource requirements of each modal generator in real time. The dynamic allocation module is used to allocate computing power based on monitoring data and a demand-resource matching algorithm. The priority sorting module is used to sort and generate tasks according to the urgency of user needs and the time required to generate modalities.
10. A multimodal content collaborative generation system according to claim 6, characterized in that, The multimodal quality closed-loop evaluation module includes: The professional domain quality module is used to quantitatively evaluate the professional quality of generated content based on professional standard libraries for various scenarios. The user experience feedback module is used to support users in marking quality issues during the preview stage, and automatically associates them with the corresponding modal generator; The iterative optimization module generates optimization suggestions when the professional quality score is less than 80 points or when there are user feedback issues. These suggestions are then fed back to the collaborative generation control module to trigger secondary generation.