Content generation method, device and equipment based on multi-modal large model platform

Abstract

The invention relates to the technical field related to content generation, in particular to a content generation method, device and equipment based on a multi-modal large model platform. The method comprises the following steps: acquiring multi-modal data; the multi-modal data comprises text data, picture data and video data; uniformly encoding the text data, the picture data and the video data into potential vectors based on a preset encoder, and performing alignment and association based on Transform to obtain multi-modal features; based on a multi-head self-attention mechanism, fusing the multi-modal features to obtain target features; inputting the target feature into a preset multi-modal generator, and outputting multi-modal content; wherein the multi-mode content comprises text content, picture content and video content; wherein the equipment running the multi-mode generator function optimizes resource allocation of a GPU (Graphics Processing Unit) and a memory in real time by intelligently predicting load and task requirements, and ensures that a high-priority task is quickly completed; and carrying out content optimization and quality evaluation on the multi-modal content.

Description

Technical Field

[0001] The present application relates to the technical field related to content generation, and specifically to a content generation method, apparatus and device based on a multimodal large model platform. Background Art

[0002] With the continuous development of artificial intelligence technology, the demand for multimodal information processing is increasing. For example, in the fields of multimedia content creation, intelligent video editing, and virtual reality scene construction, a single modality processing method is difficult to meet the complex and changing creation needs. Traditional technologies have many limitations in multimodal fusion. It is difficult to achieve efficient collaborative processing of different modal data such as text, images, and videos, resulting in insufficient richness and immersion in the generated content, and unable to fully meet users' expectations for high-quality and diverse content creation. At the same time, in terms of resource utilization, existing technologies often have inefficiencies and cannot be optimized in real time according to load and task requirements, resulting in slow response speeds and high costs in high-concurrency scenarios, which limits their application in large-scale user scenarios. Summary of the Invention

[0003] In view of this, the embodiments of the present application are dedicated to providing a content generation method, apparatus and device based on a multimodal large model platform.

[0004] The first aspect of the present application provides a content generation method based on a multimodal large model platform, comprising:

[0005] Acquire multimodal data; the multimodal data includes: text data, image data, and video data;

[0006] Based on a preset encoder, the text data, image data, and video data are uniformly encoded into potential vectors, and aligned and associated based on the Transformer to obtain multimodal features;

[0007] Based on the multi-head self-attention mechanism, multimodal features are integrated to obtain target features;

[0008] Inputting the target feature into a preset multimodal generator to output multimodal content; wherein the multimodal content includes: text content, image content, and video content;

[0009] Among them, devices running the multimodal generator function can optimize GPU and memory resource allocation in real time by intelligently predicting load and task requirements, ensuring that high-priority tasks are completed quickly;

[0010] Perform content optimization and quality assessment on multimodal content.

[0011] In some embodiments, inputting the target feature into a preset multimodal generator to output multimodal content includes:

[0012] Inputting the target features into a preset multimodal generator;

[0013] The multimodal generator determines task complexity;

[0014] A multimodal generation model that matches the complexity of the task is selected to generate and output multimodal content.

[0015] In some embodiments, selecting a multimodal generative model that matches the complexity of the task includes:

[0016] When the task complexity is lower than a first preset value, selecting a lightweight modality generation model;

[0017] When the task complexity is not less than a first preset value and less than a second preset value, selecting a standard-level modality generation model;

[0018] When the task complexity is not less than a second preset value, a heavyweight modal generation model is selected.

[0019] In some embodiments, the preset encoder uniformly encodes the text data, image data, and video data into a potential vector, including:

[0020] For text data, use BERT-base to extract CLS token representation and reduce dimensionality to obtain latent vector;

[0021] For image data, use Vision Transformer (ViT-B / 16) to perform segmentation and dimensionality reduction to obtain latent vectors;

[0022] The video data is combined with 3D-CNN (I3D) and temporal Transformer to reduce the dimension and obtain the latent vector.

[0023] In some embodiments, including:

[0024] Obtain user evaluation of the multimodal generator;

[0025] The multimodal generator is adjusted based on the evaluation.

[0026] In some embodiments, the multimodal content is optimized and the quality is evaluated:

[0027] Optimize the clarity of video and image content;

[0028] Based on the gap between the input multimodal data and the output multimodal content, and the consistency of the content of different modes in the multimodal content, the quality of the multimodal content is evaluated.

[0029] In some embodiments, it further includes:

[0030] Get user feedback on content optimization;

[0031] Based on the feedback, adjust the content optimization strategy.

[0032] The present application provides a content generation device based on a multimodal large model platform, comprising:

[0033] An acquisition module, configured to acquire multimodal data, wherein the multimodal data includes text data, image data, and video data;

[0034] A feature processing module is used to uniformly encode the text data, image data, and video data into potential vectors based on a preset encoder, and align and associate them based on a Transformer to obtain multimodal features;

[0035] The fusion module is used to fuse multimodal features based on the multi-head self-attention mechanism to obtain the target features;

[0036] An output module, configured to input the target features into a preset multimodal generator and output multimodal content; wherein the multimodal content includes: text content, image content, and video content;

[0037] Among them, devices running the multimodal generator function can optimize GPU and memory resource allocation in real time by intelligently predicting load and task requirements, ensuring that high-priority tasks are completed quickly;

[0038] The optimization module is used to optimize and evaluate the quality of multimodal content.

[0039] The present application provides an electronic device, including:

[0040] A processor, and a memory for storing a program executable by the processor;

[0041] The processor is used to implement the above-mentioned content generation method based on the multimodal large model platform by running the program in the memory.

[0042] The present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor is enabled to execute the content generation method based on the multimodal large model platform as described above.

[0043] The present application provides a content generation method based on a multimodal large model platform, which first obtains multimodal data; the multimodal data includes: text data, image data, and video data; based on a preset encoder, the text data, image data, and video data are uniformly encoded into potential vectors, and aligned and associated based on Transformer to obtain multimodal features; based on a multi-head self-attention mechanism, the multimodal features are fused to obtain target features; the target features are input into a preset multimodal generator to output multimodal content; wherein, the multimodal content includes: text content, image content, and video content; wherein, the device running the multimodal generator function optimizes the resource allocation of GPU and memory in real time by intelligently predicting load and task requirements to ensure that high-priority tasks are completed quickly; content optimization and quality evaluation are performed on the multimodal content. In this way, multimodal data such as text, images, and videos are integrated, and after encoding, alignment, association, and fusion, an output containing multiple content forms is generated, which enriches the creative content, enhances the sense of immersion, and makes the generated results more comprehensive and vivid. The device intelligently predicts load and task requirements, optimizing GPU and memory resource allocation in real time to ensure the rapid completion of high-priority tasks. This improves resource utilization, reduces costs, and enhances system stability in high-concurrency scenarios. It optimizes and evaluates the quality of multimodal content to ensure high-quality output. Further adjustments can be made based on evaluation results and user feedback to meet personalized needs and improve user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0045] Figure 1 This is a flowchart of a content generation method based on a multimodal large model platform provided in one embodiment of the present application.

[0046] Figure 2 This is a partial flow chart of a method provided in one embodiment of the present application.

[0047] Figure 3 It is a structural diagram of a content generation device based on a multimodal large model platform provided by an embodiment of the present application.

[0048] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] Figure 1 This is a flow chart of a content generation method based on a multimodal large model platform provided by an embodiment of the present application. Figure 1 As shown, the method includes the following contents.

[0051] Step S110, acquiring multimodal data; the multimodal data includes: text data, image data, and video data;

[0052] The first step in starting the multimodal content generation process is to collect multimodal data. The data types involved are broad and specifically cover the following three categories:

[0053] Text data: This type of data is obtained through various channels, including but not limited to sentences and paragraphs provided by users through various text input interfaces, as well as text information extracted from various storage media such as internet webpages and electronic documents. This text content can take the form of structured tabular text or unstructured free-text paragraphs. Its core characteristic is that it carries and conveys information through linguistic symbols.

[0054] Image data: These primarily come from images captured by digital photography equipment, graphic data acquired by scanners, and copyright-compliant image materials from internet resource platforms. These image data convey intuitive visual perception information through their unique visual elements, such as color, line, shape, and texture.

[0055] Video data is recorded using digital cameras, webcams, and other devices, or collected from legitimate video resource libraries. Video data combines spatial and temporal dimensions, consisting of a series of continuous frames that can dynamically display the development of events or changes in scenes.

[0056] Step S120: uniformly encode the text data, image data, and video data into latent vectors based on a preset encoder, and align and associate them based on a Transformer to obtain multimodal features;

[0057] After completing the multimodal data collection, it is necessary to use a preset encoding system to uniformly encode the data into vectors and use the Transformer architecture to achieve cross-modal feature alignment and association:

[0058] Unified encoding process: For text, image, and video data, the corresponding preset encoders are used to perform encoding operations. The text encoder uses advanced natural language processing technology to convert text sequences into fixed-dimensional latent vectors, accurately capturing the text's semantic information. The image encoder uses computer vision algorithms to extract key visual features from images and map them to a latent vector space. The video encoder comprehensively considers spatiotemporal characteristics to convert video data into latent vectors that incorporate dynamic information.

[0059] The encoded latent vector is fed into the Transformer model, leveraging its self-attention mechanism to achieve deep interaction and fusion within the multimodal feature space. This process eliminates information discrepancies between modal data, constructs a unified and coherent feature representation framework, and ultimately generates multimodal fusion features, laying the foundation for subsequent content generation.

[0060] Step S130: Based on the multi-head self-attention mechanism, multimodal features are integrated to obtain target features;

[0061] Based on the multi-head self-attention mechanism, a deep fusion operation is performed on multimodal features. Leveraging its multi-channel parallel processing capabilities, the multi-head self-attention mechanism focuses on the associations between features from different dimensions and granularities, capturing latent semantic and visual association information. This mechanism integrates text features into the spatial characteristics of images and videos to enhance semantic expression; image and video features can also draw on text semantic constraints to optimize the presentation of visual information. After full interaction and integration, the target feature is generated. This feature vector comprehensively integrates the multimodal information of text, images, and videos to form a unified and expressive feature expression, accurately outlining the entire data landscape and providing a core basis for content generation.

[0062] Step S140: inputting the target feature into a preset multimodal generator to output multimodal content; wherein the multimodal content includes: text content, image content, and video content;

[0063] The fused target features are input into the preset multimodal generator to officially start the multimodal content generation process. Relying on an advanced generative model architecture, the generator synchronously outputs multimodal content including text, pictures and videos based on the comprehensive information contained in the target features. In this process, the generation of text content focuses on following semantic logic and language specifications to ensure that the output text is fluent and clear in meaning; while the picture and video content emphasizes the optimization of visual effects and the guarantee of dynamic coherence. At the same time, the hardware equipment responsible for running the multimodal generator is equipped with an intelligent resource management system. By accurately predicting load conditions and task requirements, it dynamically allocates GPU and memory resources, prioritizes the supply of resources for high-priority tasks, and ensures the efficient execution and rapid completion of key tasks in multi-tasking concurrent scenarios, thereby improving the overall system performance.

[0064] Among them, devices running the multimodal generator function can optimize GPU and memory resource allocation in real time by intelligently predicting load and task requirements, ensuring that high-priority tasks are completed quickly;

[0065] Step S150: Optimize and evaluate the quality of the multimodal content.

[0066] Optimization and quality assessment of the generated multimodal content are carried out to further improve the quality of the content and ensure that the output meets high-quality standards.

[0067] In some embodiments, inputting the target feature into a preset multimodal generator and outputting multimodal content includes: inputting the target feature into a preset multimodal generator; the multimodal generator determines the task complexity; selecting a multimodal generation model that matches the task complexity, and generating and outputting multimodal content.

[0068] Specifically, the selection of a multimodal generation model that matches the task complexity includes: when the task complexity is lower than a first preset value, selecting a lightweight modal generation model; when the task complexity is not lower than the first preset value and lower than a second preset value, selecting a standard-level modal generation model; when the task complexity is not lower than the second preset value, selecting a heavyweight modal generation model.

[0069] The process of inputting target features into a preset multimodal generator and outputting multimodal content can be broken down into the following steps:

[0070] Input target features: The target features obtained through fusion processing are input into the preset multimodal generator. This target feature integrates key information from multiple modalities such as text, images, and videos, providing a core basis for subsequent content generation.

[0071] Determine the complexity of the task: The task complexity assessment module within the multimodal generator is immediately activated to conduct a comprehensive assessment of the complexity of the current task. During the assessment, multiple factors are taken into consideration, including but not limited to the size of the input data (such as the length of the text, the resolution of the image, the length of the video, etc.), the number of modalities involved (such as only text and images, or text, images and videos at the same time), the type of content required to be generated (such as simple text descriptions or complex video generation, etc.), and the high-level semantic requirements for the output content, such as the level of sophistication, style, and emotion. Through quantitative analysis of these factors, a specific quantitative value of task complexity is obtained.

[0072] Select a matching multimodal generative model: Based on the determined task complexity quantification value, select a matching multimodal generative model from the model library. Specifically:

[0073] When the task complexity is lower than the first preset value: If the task is relatively simple, such as generating only a short text description or basic image content, a lightweight modal generation model is selected. Lightweight models have fewer parameters, a relatively simple structure, fast operation, and low resource consumption. They can quickly complete the task while ensuring basic quality. They are suitable for scenarios where the requirements for generated content quality are relatively low, but the generation speed is high.

[0074] When the task complexity is at least the first preset value and less than the second preset value: When the task complexity is medium, such as generating logically structured text, relatively clear images, or simple video content, select the Standard-level modality generation model. The Standard-level model strikes a balance between generation capability and resource consumption, meeting the content generation needs of most common scenarios and providing reliable and timely output.

[0075] When the task complexity is at least the second preset value: For complex tasks, such as generating high-quality video content, image and text combinations with complex interactive logic, or content that requires the precise capture and expression of multiple high-level semantic information (such as specific style, emotion, artistic effect, etc.), choose a heavyweight modal generation model. Heavyweight models have rich parameters and powerful generation capabilities. They can deeply understand and process complex multimodal information, generating detailed, high-quality, and semantically coherent multimodal content to meet the stringent content quality requirements in complex scenarios.

[0076] Generate and output multimodal content: After selecting a multimodal generation model that matches the complexity of the task, the model begins generating content based on the multimodal information contained in the target features. The text generation component focuses on ensuring accurate semantics, logical coherence, and natural language flow. Image and video generation focuses on image clarity, color coordination, dynamic coherence, and a close match with the text. Ultimately, multimodal content, including text, images, and videos, is generated and output that meets the task requirements and quality standards to meet diverse user needs.

[0077] In some embodiments, the preset encoder uniformly encodes the text data, image data, and video data into a potential vector, including:

[0078] For text data, we use BERT-base to extract CLS token representations and reduce dimensionality to obtain latent vectors. BERT-base is a pre-trained language model based on Transformer that excels at capturing semantic information in text. The specific steps are as follows:

[0079] Text data is fed into the BERT-base model, which outputs a vector representation for each token (word or subword unit). The vector representation of the CLS token (located at the first position in the text sequence) is extracted. This vector summarizes the semantic information of the entire text sequence. The extracted CLS token vector is subjected to dimensionality reduction, converting it into a latent vector of a fixed dimension (e.g., 512 dimensions) for subsequent processing and fusion.

[0080] The Vision Transformer (ViT-B / 16) is used to slice and reduce the image data to obtain a latent vector. ViT-B / 16 can effectively process image data and capture the visual features of the image.

[0081] The specific steps are as follows:

[0082] Image data is fed into the ViT-B / 16 model, which segments the image into multiple 16x16 patches. Each patch is independently encoded, and the features of these patches are fused and processed by the model's internal Transformer encoder. A vector containing the overall semantic information of the image is extracted and dimensionality reduced to obtain a latent vector of fixed dimension (e.g., 512).

[0083] For video data, we combine 3D-CNN (I3D) and Temporal Transformer to reduce dimensionality and obtain latent vectors. We also combine 3D-CNN (I3D) and Temporal Transformer for encoding. I3D can capture both spatial and temporal features in the video, while Temporal Transformer further models temporal features. The specific steps are as follows:

[0084] Video data is fed into the I3D model, which extracts spatial features from each frame and uses a temporal convolutional layer to capture dynamic information between frames. The features extracted by I3D are fed into a temporal transformer for further processing to capture long-term dependencies in the video. The processed features are then subjected to dimensionality reduction to obtain a latent vector of fixed dimension (e.g., 512).

[0085] In some embodiments, the method further includes: obtaining a user's evaluation of the multimodal generator; and adjusting the multimodal generator based on the evaluation.

[0086] The multimodal generator's user interface provides users with a comprehensive and convenient evaluation interface. For example, a dedicated evaluation portal can be placed near the generated content display area, allowing users to access the evaluation page by clicking a button or link. This page offers a variety of evaluation methods, including but not limited to a rating system (e.g., 1 to 5 stars), categorized rating options (e.g., individual ratings for content accuracy, relevance, and visual quality), and a text comment box (for users to enter specific comments and suggestions), allowing users to fully express their opinions and feelings about the generated content. Furthermore, a questionnaire can be set up to be completed by users after they complete a certain number of generation tasks. The questionnaire can focus on the various functions of the multimodal generator, the quality of the generated content, and the user experience, thereby collecting more systematic and comprehensive user feedback. In addition to passively waiting for users to provide feedback through the evaluation interface, users can also actively collect behavioral data and interaction information during use as implicit evaluation. For example, this can record user behavior such as the length of time they viewed the generated content, the number of times they edited it, and whether they saved or shared the generated content. If a user edits a piece of generated content multiple times and only views it for a short time, it may indicate that the content does not meet user needs well. Conversely, if a user views it for a long time and directly saves or shares it, it indicates that the content is of high quality and meets user expectations. At the same time, we monitor the user's operation path and the pages they stay on during use, analyze the frequency and order of users switching between different functional modules, and understand users' attention to and usage habits of various functions, thereby mining potential user evaluation information on various aspects of the multimodal generator.

[0087] Collected user evaluation data is systematically analyzed to uncover valuable insights. For quantitative evaluation data (e.g., ratings and questionnaire options), statistical analysis methods are used to calculate the mean, median, and standard deviation of each evaluation metric to understand the distribution of user satisfaction with the multimodal generator as a whole and across various aspects. Furthermore, data visualization techniques (e.g., bar charts, line graphs, and heat maps) are used to visually present the distribution characteristics and changing trends of evaluation results, facilitating the rapid identification of challenges and strengths. For qualitative evaluation data (e.g., text reviews), natural language processing techniques are employed for text analysis, including sentiment analysis (determining whether user reviews are positive, negative, or neutral), topic mining (identifying themes and key topics discussed in user reviews, such as content quality, generation speed, and ease of use), and keyword extraction (identifying frequently occurring and representative keywords in reviews). This provides a deeper understanding of users' specific needs, pain points, and expectations. By combining quantitative and qualitative evaluation data, a comprehensive evaluation model is developed to comprehensively and in-depth evaluate the performance of the multimodal generator and pinpoint areas requiring improvement and optimization.

[0088] Based on the evaluation and analysis results, targeted adjustments can be made to the parameters and models of the multimodal generator. If user reviews indicate issues with semantic coherence in the generated content, the parameters of the text generation model can be optimized, such as adjusting the temperature of the language model and the weight distribution of the attention mechanism, to improve the logic and fluency of text generation. For low-quality image and video generation, fine-tuning the corresponding visual generation model can be performed, such as modifying the loss function parameters of the generator and discriminator in the generative adversarial network (GAN) and optimizing the configuration of the convolutional and pooling layers in the network architecture, to enhance the clarity, realism, and visual aesthetics of the generated images and videos. If user feedback indicates that the multimodal generator needs improvement in terms of content consistency across different modalities, the training of the multimodal feature fusion module can be strengthened, and the parameters of the cross-modal attention mechanism can be adjusted to achieve better consistency in the semantics and visual style of the text, images, and videos. Furthermore, based on user requirements for generation speed, the generator's computational process and resource allocation strategy can be optimized to improve overall operational efficiency and shorten generation time. During the adjustment process, experimental verification and test evaluation are continuously carried out to ensure that the adjustments made can effectively improve the performance of the multimodal generator and user satisfaction, and will not introduce new problems or have negative impacts on other aspects.

[0089] In some embodiments, the multimodal content is optimized and the quality is evaluated:

[0090] Optimize the clarity of video and image content;

[0091] Based on the gap between the input multimodal data and the output multimodal content, and the consistency of the content of different modes in the multimodal content, the quality of the multimodal content is evaluated.

[0092] Specifically, various technologies are employed to enhance video clarity. For example, super-resolution reconstruction uses deep learning algorithms to upscale low-resolution video frames to high resolution, enhancing details and making blurry images legible. Furthermore, video denoising algorithms are employed to reduce noise interference and improve image purity. Furthermore, video color correction and enhancement are performed, optimizing parameters such as color saturation and contrast to achieve more vivid and natural colors and enhance the overall visual quality. Image processing techniques, such as image sharpening, are used to highlight edge details in images, making them clearer and sharper. Image enhancement algorithms are also employed to adjust image brightness, contrast, and color for greater visual appeal. For noisy images, denoising algorithms are employed to reduce the impact of noise on image quality. During the optimization process, the content and stylistic characteristics of the images are fully considered to ensure that the optimized images maintain their original artistic style and semantic information while meeting high standards of clarity and visual quality.

[0093] For video and image content, objective image / video quality assessment metrics, such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), are used to compare videos and images before and after optimization to quantify the improvement in clarity. PSNR compares the mean squared error (MSE) of the original and processed images to the square of the maximum pixel value, yielding a numerical metric. Higher values ​​indicate better image quality. SSIM comprehensively assesses similarity based on image brightness, contrast, and structure, better aligned with human visual perception and effectively measuring the improvement in clarity of optimized videos and images. Based on the gap between the input multimodal data and the output multimodal content, the consistency of content across different modalities within the multimodal content is measured. On the one hand, the consistency of information conveyed by text, images, and videos is analyzed at a semantic level. For example, whether the scene described in the text matches the content presented in the image or the video, and whether there are any semantic deviations or inconsistencies. On the other hand, the consistency of content in different modalities is evaluated in terms of style and visual features. For example, whether the color style and composition characteristics of pictures and videos match the atmosphere and emotions described in the text, ensuring that the multimodal content is highly coordinated and consistent as a whole, providing inspiration and help to users.

[0094] In some embodiments, the method further includes: obtaining user feedback on content optimization; and adjusting the content optimization strategy based on the feedback.

[0095] In the interface of multimodal content display, add a user feedback portal, such as setting a "Feedback" button below the video or image. Users can click the button to enter the feedback page and submit their specific opinions and suggestions for content optimization.

[0096] Create a user feedback form with various feedback types, such as "Clarity meets requirements," "Clarity needs improvement," "Content does not meet expectations," and "Unreasonable color matching," to facilitate users to quickly select and fill out feedback. Establish online customer service or a feedback email address to promptly record and organize user issues and optimization needs during use.

[0097] Monitor user behavior on optimized multimodal content, such as whether users repeatedly optimize the same content. Repeated optimizations may indicate that the optimization results are still not meeting their expectations and require further improvement. Analyze how long users spend viewing optimized content. Shorter dwell times may indicate that the content is not engaging and that there may be issues with the optimization approach. Monitor how often users share and save optimized content. Higher sharing and saving frequencies generally indicate user satisfaction with the optimization results and can serve as a reference for effective optimization strategies.

[0098] Collected user feedback is categorized and statistically analyzed to identify common user issues and key needs. For example, if the majority of users report insufficient video clarity, improving video clarity will be a key optimization priority.

[0099] Combine user feedback and quality assessment metrics to identify weak links in the content optimization process. For example, if user feedback indicates an image's color combination is inappropriate and the quality assessment also reveals low scores for color-related metrics, you need to adjust the image color optimization algorithm accordingly.

[0100] Based on user feedback, we adjust the parameters of the content optimization algorithm. For example, if a user wants to further improve video clarity, they can increase the magnification of the super-resolution reconstruction algorithm or optimize the strength of the denoising algorithm to reduce the impact of noise on clarity while preserving details.

[0101] To address the style inconsistency issues reported by users, we optimized the multimodal feature fusion module and adjusted the parameters of the cross-modal attention mechanism to achieve greater consistency in style across text, images, and videos. Furthermore, we strengthened the extraction and application of style features during content generation to ensure that the generated multimodal content meets the user's desired style.

[0102] Regularly update and improve content optimization models based on user feedback. If new requirements or issues arise from user feedback, retrain or fine-tune the optimization model to better adapt to changing user needs and continuously improve the effectiveness and quality of multimodal content optimization.

[0103] The following combination Figure 2 The following describes the solution provided in this application:

[0104] The platform provided by this application innovatively realizes the deep integration of large models such as text, picture, and video generation, realizes the collaborative processing and consistent output of cross-modal input, significantly improves the richness and immersion of content creation, and breaks through the limitations of single-modal processing of traditional platforms. The core components include: 1. Input encoder: uniformly processes text, images, and videos to generate shared latent space representations. 2. Cross-modal attention mechanism: based on Transformer alignment and correlation of multimodal features. 3. Multimodal generator: outputs generated videos, text, pictures, audio, etc. according to user needs.

[0105] The specific implementation is as follows:

[0106] 1. Input encoder:

[0107] The input encoder uses a pre-trained model to uniformly encode text, images, and videos into a 512-dimensional latent vector:

[0108] z t =fi (x i θ i )

[0109] The input x for each mode i Through a dedicated encoder f i (·) is mapped into a shared latent space vector z i =f i (x i ), thereby achieving cross-modal feature alignment.

[0110] Text: Use BERT-base, fine-tune on the semantic understanding task, process a maximum of 512 token sequences, extract CLS tag representations and perform dimensionality reduction.

[0111] Images: Use the Vision Transformer (ViT-B / 16), pre-trained on the ImageNet dataset, to process 224x224 images, split them into 16x16 patches and then perform dimensionality reduction.

[0112] Video: Combines 3D-CNN (I3D) and temporal Transformer, pre-trained on the Kinetics-400 dataset, processes 10-second videos (16fps), and outputs reduced dimensionality.

[0113] 2. Cross-modal attention mechanism

[0114] The cross-modal attention mechanism is based on Transformer and fuses modal features through multi-head self-attention:

[0115] Perform weighted summation of the query, key, and value vectors of text, image, and video to obtain the fused representation z fused This mechanism enables semantic information of different modalities to refer to and complement each other, achieving semantic consistency between modalities.

[0116] 3. Multimodal Generator

[0117] The multimodal generator generates coherent multimodal content based on the conditional diffusion model. Through a diffusion process with gradual denoising:

[0118] Using the U-Net structure to fused The system generates continuous video frames with a resolution of 256×256 (16fps) under the following conditions; GPT-2 decoder is used to generate the corresponding text, WaveNet is used to generate background music, and the image generation is directly derived from the fusion representation z through the improved model based on Stable Diffusion. fusedBy sampling the video, we generate high-definition images that match the theme and style of the video. These generated contents are consistent in style and semantics, greatly enhancing the immersiveness of content creation.

[0119] The solution provided by this application has the following effects:

[0120] Cross-modal semantic collaboration: The attention mechanism achieves deep alignment of text, image, and video features, generating consistent output, which is superior to the isolated processing of traditional single-modality models.

[0121] Diversified generation capabilities: The diffusion model supports the simultaneous generation of video, text, and audio, meeting complex creative needs and transcending the limitations of a single output platform.

[0122] Efficient real-time processing: DDIM sampling and TensorRT optimization shorten inference time to seconds, making it suitable for high-concurrency scenarios and surpassing the complex calculations of traditional diffusion models.

[0123] Flexible scalability: The platform supports flexible selection of output combinations based on user needs (such as generating images alone or combining video, images, and audio), improving the freedom and adaptability of content creation and outperforming traditional content generation platforms with fixed-format output.

[0124] Scalable architecture: The image generation module is integrated in a modular manner, and other modalities such as voice and 3D modeling can be further expanded in the future to maintain the platform's forward-looking nature and evolutionary space.

[0125] Furthermore, the dynamic resource scheduling of this application is as follows:

[0126] Dynamic resource scheduling optimizes the allocation of resources such as GPU and memory in real time by intelligently predicting load and task requirements, ensuring the rapid completion of high-priority tasks while reducing costs and significantly improving the efficiency and responsiveness of the platform in high-concurrency scenarios.

[0127] The core components include:

[0128] Load Predictor: Predicts system load and task demand.

[0129] Resource Allocator: Dynamically adjust GPU, memory, and network resources.

[0130] Task Scheduler: Optimize task priority and execution order.

[0131] The specific implementation is as follows:

[0132] 1. Load predictor

[0133] The load predictor uses an LSTM model to predict future system load based on historical load, task characteristics, and user priority information. Its prediction formula can be expressed as:

[0134] L(t+1)=f LSTM (L(t),X task ,X user ;θ)

[0135] Where L(t) is the current resource utilization rate, X task Including task characteristics such as input size and number of modalities, X user Represents information such as user priority. The predictor improves prediction accuracy and real-time performance through strategies such as sequence length adjustment and early stopping.

[0136] 2. Resource Allocator

[0137] The resource allocator dynamically allocates computing resources such as GPU and memory based on the Deep Q Network (DQN) algorithm. The reward function is designed as:

[0138] R=w1·Quality-w2·Latency-w3·Cost

[0139] The DQN optimizes the trade-off between generation quality, response latency, and resource cost. Based on the current state, it outputs a resource allocation plan (such as the number of GPUs, memory allocation, and batch size). Through techniques such as state normalization and mixed-precision inference, it controls inference latency to milliseconds to support high-concurrency processing.

[0140] 3. Task Scheduler

[0141] The task scheduler optimizes task execution based on the Earliest Deadline First (EDF) algorithm:

[0142]

[0143] Ensure that high-priority tasks are completed first. Combine mechanisms such as queue slicing, batching, and task retry, and use Pod Autoscaling (HPA) to maintain system stability under high load.

[0144] With this configuration, the solution provided by this application has the following effects:

[0145] Accurate load prediction: The LSTM model accurately predicts resource requirements through multi-dimensional feature analysis, reducing allocation waste and outperforming the inefficiency of traditional static scheduling.

[0146] Intelligent resource optimization: DQN dynamically adjusts GPU and memory allocation to ensure high-priority tasks run with low latency, staying ahead of resource bottlenecks in fixed allocation strategies.

[0147] High concurrency and robustness: EDF scheduling combined with a preemptive mechanism maintains fast response under high loads, making it suitable for large-scale user scenarios and exceeding the performance limitations of conventional queue management.

[0148] Cost-effectiveness: Reinforcement learning optimizes resource usage, significantly reducing cloud deployment costs and making it more economical than traditional scheduling.

[0149] The solution provided in this application also involves adaptive model optimization

[0150] Adaptive model optimization dynamically selects models suitable for tasks and fine-tunes them in real time based on user feedback, balancing computing efficiency and generation quality to provide personalized, high-performance AI services that adapt to diverse application scenarios.

[0151] The core components of adaptive model optimization include: a model selector that dynamically selects a model suitable for a task; a resource allocator that optimizes the allocation of computing resources; and a feedback loop that fine-tunes the model based on user feedback.

[0152] The specific implementation is as follows:

[0153] Model Selector

[0154] The model selector selects the best model based on the task complexity score. Define the complexity score:

[0155] Model={Lightweight,ifS<θ1Standard,ifθ1≤S<θ2Heavy,ifS≥θ2][S

[0156] =w1·InputSize+w2·ModalityCount+w3·TaskType

[0157] Based on thresholds 01 and 02, models are divided into three tiers: when S < θ1, lightweight models (such as DistilBERT and MobileViT) are used; when θ1 ≤ S < θ2, standard models (such as BERT-base and ViT-B / 16) are used; and when S ≥ θ2, heavyweight models (such as RoBERTa-large and ViT-L / 16) are used. Through dynamic threshold adjustment and model preloading mechanisms, model switching delays are controlled to milliseconds, ensuring precise matching of tasks and models.

[0158] 2. Resource Allocator

[0159] The resource allocator dynamically adjusts the inference resource configuration based on the Proximal Policy Optimization (PPO) algorithm. Set the reward function:

[0160] R = w1·Quality - w2·Latency

[0161] Minimize latency while ensuring build quality. System state includes task complexity and current load, while actions can include GPU and memory allocation. Improve resource utilization efficiency and responsiveness through state normalization and batch inference optimization.

[0162] 3. Feedback loop

[0163] The feedback loop fine-tunes the model through user feedback, combining the base loss with the feedback loss to construct a new loss:

[0164]

[0165] in, Defined by user ratings or edit operations. The system uses incremental updates and low-quality feedback filtering mechanisms, and regularly uses evaluation metrics (such as BLEU and FID) to fine-tune the model to ensure continuous improvement in generation quality.

[0166] The solution of this application has the following technical advantages:

[0167] Accurate model matching: The complexity scoring mechanism ensures that the model is perfectly adapted to the task requirements, balancing efficiency and quality, and is superior to single models that waste resources or have insufficient performance.

[0168] Dynamic resource management: PPO optimizes resource allocation in real time, reducing latency under high loads and surpassing the low flexibility of static allocation.

[0169] Personalized generation: Feedback-driven fine-tuning adapts to user preferences and generates customized content, going beyond the single output limitations of general models.

[0170] Scalable architecture: Model pool and resource allocation support dynamic expansion to adapt to diverse tasks and user scale, surpassing the limitations of fixed model systems

[0171] The intelligent content optimization and quality assessment mechanism of this application is described as follows:

[0172] The intelligent content optimization and quality assessment mechanism automatically optimizes content generation and provides objective quality assessment to ensure the high clarity, relevance and consistency of the output content. At the same time, it achieves personalized optimization based on user feedback to enhance the user experience.

[0173] The core components of the intelligent content optimization and quality assessment mechanism include: Content Optimizer: Automatically adjusts generated content; Quality Evaluator: Quantifies content quality; User Preference Learner: Optimizes content based on feedback.

[0174] Specifically, the implementation methods include:

[0175] 1. Content Optimizer

[0176] The content optimizer uses a generative adversarial network (GAN) to post-process the generated content to improve visual quality. The optimization loss function is:

[0177]

[0178] Combining adversarial loss and perceptual loss (based on the VGG model) enables the U-Net generator to enhance the clarity of images and videos, while the PatchGAN discriminator ensures the authenticity of the content. Through progressive training (gradually increasing from low resolution to 256×256) and FP16 acceleration, training stability and efficiency are balanced.

[0179] 2. Quality Evaluator

[0180] The quality evaluator calculates a composite quality score for generated content. Define quality metrics:

[0181] Q=w1·Q ckarity +w2·Q relevance +w3·Q consistency

[0182] The clarity Q clarity Through the entropy evaluation of the Inception V3 model, the correlation Q relevance BERT is used to calculate the similarity between the input text and the output text, and the consistency Q consistency CLIP is used to calculate the cross-modal embedding similarity between images and text. Through feature normalization and batch inference optimization, evaluation latency is controlled to milliseconds, enabling real-time monitoring and optimization of generated results.

[0183] 3. User Preference Learner

[0184] The user preference learner fine-tunes the optimizer based on feedback:

[0185]

[0186] Convert user ratings or edits on generated results into feedback signals and construct feedback loss The system updates model parameters through gradient descent, thereby learning and adapting to the user's specific preferences. It also combines low-quality feedback filtering and regular evaluation mechanisms to ensure that feedback-driven optimization is stable and effective, continuously improving the personalized generation effect.

[0187] This has the following technical advantages:

[0188] High-quality content generation: GAN optimization significantly improves the clarity and realism of images and videos, surpassing the blur or distortion problems of traditional generative models.

[0189] Multidimensional quality assessment: A comprehensive scoring mechanism provides an objective measure of clarity, relevance, and consistency, surpassing the uncertainty of subjective assessments.

[0190] Personalized user experience: Feedback-driven fine-tuning enables customized output to meet user-specific needs, surpassing the inflexibility of general-purpose models.

[0191] Real-time optimization capabilities: TensorRT and quantization technologies support fast inference, meeting the needs of real-time applications and outperforming the complex calculations of traditional post-processing.

[0192] The device embodiments of this application can be used to execute the method embodiments of this application. For details not disclosed in the device embodiments of this application, please refer to the method embodiments of this application.

[0193] Figure 3 FIG. 1 is a block diagram of a content generation device based on a multimodal large model platform provided by an embodiment of the present application. Figure 3 As shown, the device includes:

[0194] An acquisition module, configured to acquire multimodal data, wherein the multimodal data includes text data, image data, and video data;

[0195] A feature processing module is used to uniformly encode the text data, image data, and video data into potential vectors based on a preset encoder, and align and associate them based on a Transformer to obtain multimodal features;

[0196] The fusion module is used to fuse multimodal features based on the multi-head self-attention mechanism to obtain the target features;

[0197] An output module, configured to input the target features into a preset multimodal generator and output multimodal content; wherein the multimodal content includes: text content, image content, and video content;

[0198] Among them, devices running the multimodal generator function can optimize GPU and memory resource allocation in real time by intelligently predicting load and task requirements, ensuring that high-priority tasks are completed quickly;

[0199] The optimization module is used to optimize and evaluate the quality of multimodal content.

[0200] Below, reference Figure 4 To describe the electronic device according to the embodiment of the present application. Figure 4 The figure shows a block diagram of an electronic device according to an embodiment of the present application.

[0201] like Figure 4 As shown, electronic device 400 includes one or more processors 410 and memory 420 .

[0202] The processor 410 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0203] The memory 420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 410 may run the program instructions to implement the content generation method based on the multimodal large model platform of the various embodiments of the present application described above and / or other desired functions. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0204] In one example, the electronic device 400 may further include an input device 430 and an output device 440 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0205] In addition, the input device 430 may also include, for example, a keyboard, a mouse, an interface, etc. The output device 440 may output various information to the outside, including analysis results, etc. The output device 440 may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0206] Of course, to simplify, Figure 4 Only some of the components in the electronic device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0207] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the content generation method based on a multimodal large model platform according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0208] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0209] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the content generation method based on a multimodal large model platform according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0210] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0211] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A content generation method based on a multimodal large model platform, characterized in that: include: Acquire multimodal data; The multimodal data includes: text data, image data, and video data; Based on a preset encoder, the text data, image data, and video data are uniformly encoded into potential vectors, and aligned and associated based on the Transformer to obtain multimodal features; Based on the multi-head self-attention mechanism, multimodal features are integrated to obtain target features; Inputting the target feature into a preset multimodal generator to output multimodal content; wherein the multimodal content includes: text content, image content, and video content; Among them, devices running the multimodal generator function can optimize GPU and memory resource allocation in real time by intelligently predicting load and task requirements, ensuring that high-priority tasks are completed quickly; Perform content optimization and quality assessment on multimodal content.

2. The content generation method based on the multimodal large model platform according to claim 1 is characterized in that: Inputting the target feature into a preset multimodal generator to output multimodal content includes: Inputting the target features into a preset multimodal generator; The multimodal generator determines task complexity; A multimodal generation model that matches the complexity of the task is selected to generate and output multimodal content.

3. The content generation method based on the multimodal large model platform according to claim 2 is characterized in that: The selecting of a multimodal generative model that matches the complexity of the task includes: When the task complexity is lower than a first preset value, selecting a lightweight modality generation model; When the task complexity is not less than a first preset value and less than a second preset value, selecting a standard-level modality generation model; When the task complexity is not less than a second preset value, a heavyweight modal generation model is selected.

4. The content generation method based on the multimodal large model platform according to claim 1 is characterized in that: The preset encoder uniformly encodes the text data, image data, and video data into a potential vector, including: For text data, use BERT-base to extract CLS token representation and reduce dimensionality to obtain latent vector; For image data, use Vision Transformer (ViT-B / 16) to perform segmentation and dimensionality reduction to obtain latent vectors; The video data is combined with 3D-CNN (I3D) and temporal Transformer to reduce the dimension and obtain the latent vector.

5. The content generation method based on the multimodal large model platform according to claim 1 is characterized in that: include: Obtain user evaluation of the multimodal generator; The multimodal generator is adjusted based on the evaluation.

6. The content generation method based on the multimodal large model platform according to claim 1 is characterized in that: The above mentioned method is to optimize the content and evaluate the quality of multimodal content: including Optimize the clarity of video and image content; Based on the gap between the input multimodal data and the output multimodal content, and the consistency of the content of different modes in the multimodal content, the quality of the multimodal content is evaluated.

7. The content generation method based on the multimodal large model platform according to claim 1 is characterized in that: Also includes: Get user feedback on content optimization; Based on the feedback, adjust the content optimization strategy.

8. A content generation device based on a multimodal large model platform, characterized in that: include: Acquisition module, used to acquire multimodal data; The multimodal data includes: text data, image data, and video data; A feature processing module is used to uniformly encode the text data, image data, and video data into potential vectors based on a preset encoder, and align and associate them based on a Transformer to obtain multimodal features; The fusion module is used to fuse multimodal features based on the multi-head self-attention mechanism to obtain the target features; An output module, configured to input the target features into a preset multimodal generator and output multimodal content; wherein the multimodal content includes: text content, image content, and video content; Among them, devices running the multimodal generator function can optimize GPU and memory resource allocation in real time by intelligently predicting load and task requirements, ensuring that high-priority tasks are completed quickly; The optimization module is used to optimize and evaluate the quality of multimodal content.

9. An electronic device, characterized in that: include: A processor, and a memory for storing a program executable by the processor; The processor is configured to implement the content generation method based on a multimodal large model platform as described in any one of claims 1 to 7 by running the program in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the content generation method based on a multimodal large model platform according to any one of claims 1 to 7.

Images