Multimodal content generation method, system and storage medium

Abstract

The present invention provides a multimodal content generation method, system, and storage medium, relating to the field of multimodal interaction technology. The method includes: obtaining a user's text description, determining the user's need type, target need, and a unimodal generator corresponding to the need type; when there are multiple need types, generating unimodal content corresponding to each need type based on the target need and the user's historical data by the unimodal generator corresponding to each need type; comparing each unimodal content to determine the thematic similarity between each unimodal content and other unimodal content; updating the unimodal content based on the thematic similarity, and fusing all updated unimodal content to obtain multimodal content corresponding to the target need. By updating the unimodal content, the present invention eliminates the thematic inconsistencies between different modalities and improves the matching degree between multimodal content and user needs.

Description

Technical Field

[0001] The present invention relates to the field of multimodal interaction technology, and in particular to a multimodal content generation method, system, and storage medium. Background Art

[0002] Existing multimodal content generation systems primarily rely on artificial intelligence (AI) to generate content in multiple modalities, including text, images, audio, and animation. Typically, these systems generate monomodal content consisting of text, sound, and images based on user input or requirements, and then integrate these to create multimodal content that meets user requirements.

[0003] In related technologies, since the generation process of content in different modalities may be driven by different algorithms and models, it is difficult to ensure synchronization and consistency between different modalities. For example, when making a PPT on a science and technology theme, there may be style or logical mismatches between the text content, dubbing and animation effects, resulting in the final generated content being unable to meet user needs, thereby affecting the user experience. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the matching degree between multimodal content and user needs.

[0005] To solve the above problems, the present invention provides a multimodal content generation method, system and storage medium.

[0006] In a first aspect, a multimodal content generation method of the present invention is applied to an intelligent agent system, wherein the intelligent agent system includes a multimodal creation module, wherein the multimodal creation module includes multiple single-modal generators, and the method includes:

[0007] Get the user's text description;

[0008] Determining, according to the text description, the user's demand type, target demand, and the single-modal generator corresponding to the demand type;

[0009] When there are multiple demand types, generating a single modal content corresponding to each demand type according to the target demand and the user's historical data by the single modal generator corresponding to each demand type;

[0010] Comparing the unimodal content corresponding to each demand type to determine the subject similarity between each unimodal content and other unimodal content;

[0011] The unimodal content is updated according to the subject similarity between each unimodal content and other unimodal content, and all the updated unimodal content are merged to obtain multimodal content corresponding to the target demand.

[0012] Optionally, determining the user's requirement type, target requirement, and the single-modal generator corresponding to the requirement type according to the text description includes:

[0013] Performing feature extraction on the text description to obtain multiple keywords of the text description;

[0014] Performing intent recognition and context analysis on the keywords through a deep learning model to obtain at least one intended target and at least one required keyword of the user;

[0015] Determining the user's need type according to each of the intended goals, and then determining the single-modal generator corresponding to the need type according to the need type;

[0016] When there is a demand keyword, determining the target demand according to the demand keyword;

[0017] When there are multiple demand keywords, a weight value is assigned to each demand keyword by natural language processing technology, and each demand keyword is clustered by a clustering algorithm to obtain a clustering result for each demand keyword;

[0018] The target demand is obtained according to the weight value corresponding to the demand keyword and the clustering result.

[0019] Optionally, the unimodal generator includes a text generator, and generating unimodal content corresponding to each requirement type by the unimodal generator corresponding to each requirement type according to the target requirement and the user's historical data includes:

[0020] When the single-modal generator is the text generator, the demand keywords are integrated according to the clustering results of all the demand keywords by using natural language generation technology to obtain a keyword list;

[0021] Perform content planning based on the keyword list to obtain initial text content;

[0022] The text content corresponding to the demand keyword in the initial text content is enhanced according to the historical data and the weight value of the demand keyword to obtain final text content, and the text content is used as the single modal content of the text generator.

[0023] Optionally, the unimodal generator includes an image generator, and generating unimodal content corresponding to each demand type by the unimodal generator corresponding to each demand type according to the target demand and the user's historical data includes:

[0024] When the single-modality generator is the image generator, determining the theme and style of the image corresponding to the demand keyword according to the clustering results of all the demand keywords;

[0025] Generate image content corresponding to the required keywords based on the theme and style using computer vision technology and image processing algorithms;

[0026] The image content corresponding to the demand keyword is optimized and integrated with the historical data of the theme and style to obtain final image content, and the final image content is used as the single-modal content of the image generator.

[0027] Optionally, the unimodal generator includes an audio generator, and generating unimodal content corresponding to each demand type according to the target demand and the user's historical data by the unimodal generator corresponding to each demand type includes:

[0028] When the unimodal generator is the audio generator, determining the theme and style of the audio corresponding to the demand keyword according to the clustering results of all the demand keywords;

[0029] According to the theme and style of the audio, audio elements are selected and speech synthesis is performed on the audio elements to obtain audio content corresponding to the required keywords, and the audio content is used as the unimodal content of the audio generator.

[0030] Optionally, comparing the unimodal content corresponding to each demand type to determine the subject similarity between each unimodal content and other unimodal content includes:

[0031] Performing feature extraction on the unimodal content of each demand type to obtain a feature vector of each unimodal content;

[0032] Normalizing the feature vector of each unimodal content to obtain feature vectors of the same scale;

[0033] Cosine similarity calculation is performed based on the feature vectors of the same scale to obtain similarity scores between each of the unimodal content and other unimodal content, and the similarity scores are used as the topic similarity.

[0034] Optionally, updating the unimodal content according to the subject similarity between each unimodal content and other unimodal content includes:

[0035] generating a similarity matrix according to the subject similarities between each unimodal content and other unimodal content;

[0036] Analyzing the similarity matrix to determine the unimodal content that needs to be adjusted;

[0037] Obtaining a parameter adjustment strategy for the unimodal generator corresponding to the unimodal content that needs to be adjusted based on the topic similarity between the unimodal content that needs to be adjusted and other unimodal contents;

[0038] updating, according to the parameter adjustment strategy, the model parameters of the unimodal generator corresponding to the unimodal content that needs to be adjusted;

[0039] The updated unimodal generator is used to regenerate unimodal content corresponding to each of the demand types according to the target demand and the user's historical data, and the regenerated unimodal content is used as the updated unimodal content.

[0040] Optionally, the intelligent agent system further includes a cross-modal alignment submodule, which fuses all the updated single-modal contents to obtain multimodal content corresponding to the target demand, including:

[0041] Inputting all the updated unimodal contents into the cross-modal alignment submodule;

[0042] Performing temporal and structural alignment on all the updated unimodal contents and other unimodal contents through a cross-modal alignment submodule to obtain all cross-modally aligned unimodal contents;

[0043] Performing feature extraction on all the cross-modal aligned unimodal contents to obtain multiple key interaction points;

[0044] The monomodal content after all cross-modal alignment is matched according to the key interaction points to obtain the multimodal content corresponding to the target demand.

[0045] In a second aspect, the present invention provides a multimodal content generation system, which is applied to an intelligent agent system. The intelligent agent system includes a multimodal creation module, which includes multiple single-modal generators. The system includes:

[0046] A receiving unit, configured to obtain a user's text description;

[0047] An analyzing unit, configured to determine, based on the text description, the user's requirement type, target requirement, and the single-modal generator corresponding to the requirement type;

[0048] a single-modal processing unit, configured to generate, when there are multiple demand types, single-modal content corresponding to each demand type according to the target demand and the user's historical data by the single-modal generator corresponding to each demand type;

[0049] a comparing unit, configured to compare the unimodal content corresponding to each of the demand types to determine a theme similarity between each of the unimodal content and the other unimodal content;

[0050] An updating unit is configured to update the unimodal content according to the subject similarity between each unimodal content and other unimodal content, and fuse all the updated unimodal content to obtain multimodal content corresponding to the target demand.

[0051] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multimodal content generation method described above is implemented.

[0052] The multimodal content generation method, system, and storage medium of the present invention first receive a user's text description and identify their need type and target need. It then matches a unimodal generator based on the need type. Using the matched unimodal generator, the system combines the user's historical data and the target need to generate preliminary unimodal content. When multiple need types exist, this indicates that the user desires to generate multimodal content encompassing multiple forms of expression. The unimodal generator corresponding to each need type is then called to generate corresponding unimodal content based on the target need and the user's historical data. This fully utilizes the user's historical data to enhance the personalization and relevance of the generated content. This satisfies the user's diverse needs for multimodal content and, by integrating the user's historical data, makes the generated unimodal content more aligned with the user's personalized preferences, improving content relevance and user satisfaction. Each unimodal content is then compared with other unimodal content to calculate thematic similarity between the content, thereby quantitatively assessing the degree of correlation between content in different modalities. This allows for timely identification of thematic differences or mismatches between content in different modalities, providing a clear basis and direction for subsequent adjustments. According to the determined subject similarity, the unimodal content with lower similarity is updated and optimized to improve its subject consistency with other content. After all unimodal content has undergone necessary adjustments and optimizations, it is merged to form the final target multimodal content. By updating the unimodal content, the present invention eliminates the thematic incoordination problem between different modalities, improves the overall quality and content accuracy of the multimodal content, and thus improves the degree of matching between the multimodal content and user needs. On the other hand, the successful fusion of multiple modal contents provides users with a complete, rich and coordinated multimodal experience, which significantly enhances user satisfaction and the attractiveness of the content. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 1 is a flow chart of a method for generating multimodal content according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic structural diagram of an intelligent agent system in another embodiment of the present invention;

[0055] Figure 3 FIG. 4 is a structural diagram of a multimodal content generation system in yet another embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0057] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0058] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0059] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0060] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0061] In response to the problems existing in the above-mentioned related technologies, this embodiment provides a multimodal content generation method, system and storage medium.

[0062] Combine Figure 1 As shown, an embodiment of the present invention provides a multimodal content generation method, which is applied to an intelligent agent system. The intelligent agent system includes a multimodal creation module, and the multimodal creation module includes multiple single-modal generators.

[0063] Specifically, combined Figure 2As shown in the figure, the intelligent agent system, as a highly integrated intelligent architecture, can efficiently process and generate multimodal content. Its core component is the multimodal creation module, which integrates multiple unimodal generators specifically for generating different types of content output. These unimodal generators include text generators, image generators, and audio generators. The text generators, image generators, and audio generators work together to meet the complex and diverse needs of users. Among them, the text generator uses natural language processing technology to create text content, the image generator uses computer vision technology to generate visual materials, and the audio generator focuses on creating voice or music content.

[0064] The method comprises:

[0065] Gets a text description of the user.

[0066] Specifically, users express their needs and expectations for target content by entering a text description. This text description contains a wealth of information, such as topic, style, and specific requirements. To accurately generate content that meets user expectations, it is necessary to capture this text description completely and accurately and perform preliminary preprocessing on it, such as removing irrelevant characters and standardizing the text format. This ensures clarity and usability, laying a solid foundation for subsequent needs analysis and content generation.

[0067] According to the text description, the user's demand type, target demand, and the single-modal generator corresponding to the demand type are determined.

[0068] Specifically, after obtaining the user's text description, it is necessary to conduct an in-depth analysis to identify the key information therein. In a preferred embodiment of the present invention, natural language processing technology can be used to extract keywords, phrases and semantic structures in the text, and then determine the user's specific demand type (such as text content creation, image generation, audio production, etc.) and target needs, wherein the target needs include elements such as the theme and style of the content that the user expects to generate (such as generating a PPT with a science and technology theme, making a promotional video, etc.). At the same time, a single-modal generator that can meet these needs must be matched. This step requires the system to have strong semantic understanding and pattern recognition capabilities to ensure accurate identification of user intentions and selection of appropriate tools to provide precise direction guidance for subsequent content generation.

[0069] When there are multiple demand types, the unimodal content corresponding to each demand type is generated by the unimodal generator corresponding to each demand type according to the target demand and the historical data of the user.

[0070] Specifically, when the user's needs involve multiple modalities, it indicates that the final multimodal content needs to integrate different types of expressions. At this time, the system will call the unimodal generator corresponding to each demand type. These generators create content based on the description of the target demand and the user's historical data. In a preferred embodiment of the present invention, if the user wants to generate a promotional material containing text, images and audio, the system will use the corresponding text generator to create relevant copy, the image generator to create illustrations, and the audio generator to produce background music or dubbing. In this process, the user's historical data plays a key role, which can help the generator better understand the user's preferences, common styles, etc., so that the generated unimodal content is more in line with the user's personalized needs and improve the relevance and satisfaction of the content.

[0071] Comparison is performed based on the unimodal content corresponding to each of the demand types to determine the subject similarity between each of the unimodal content and the other unimodal content.

[0072] Specifically, since the generated unimodal content needs to maintain coherence and consistency in the subject, in order to ensure that the multimodal content after the final fusion is holistic and coordinated. Therefore, the system will compare and analyze each unimodal content with other content. By calculating the subject similarity between the generated content, the degree of subject relevance of different modal contents can be quantitatively evaluated. In a preferred embodiment of the present invention, feature extraction and similarity calculation algorithms can be used to implement it. For example, text feature extraction technology is used to convert text content into vector form, and then the similarity with other modal content is calculated by methods such as cosine similarity. In this way, it is possible to accurately identify which unimodal content has deviations or mismatches in the subject, provide a clear basis for subsequent optimization and adjustment, and help improve the overall quality of multimodal content.

[0073] The unimodal content is updated according to the subject similarity between each unimodal content and other unimodal content, and all the updated unimodal content are merged to obtain multimodal content corresponding to the target demand.

[0074] Specifically, targeted updates and optimizations are performed on unimodal content with low similarity, thereby adjusting the style of the content, supplementing or modifying theme-related elements, and other operations to enhance its consistency with other content. After all unimodal content has undergone necessary adjustments and optimizations, the system will merge all content. The fusion process needs to comprehensively consider factors such as the characteristics of different modalities, the logical relationship of the content, and the presentation method to achieve seamless integration. The resulting multimodal content not only meets the user's target needs, but also maintains coordination and unity in theme, style, and expression, providing users with a complete, rich, and high-quality multimodal experience, effectively improving user satisfaction and content appeal.

[0075] The multimodal content generation method of the present invention first receives a user's text description and identifies their need type and target need. It then matches a unimodal generator based on the need type. Using the matched unimodal generator, the method combines the user's historical data and the target need to generate preliminary unimodal content. When there are multiple need types, this indicates that the user desires to generate multimodal content that includes multiple forms of expression. The unimodal generator corresponding to each need type is then called to generate corresponding unimodal content based on the target need and the user's historical data. This fully utilizes the user's historical data to enhance the personalization and relevance of the generated content. This method meets the user's diverse needs for multimodal content and, by combining user historical data, makes the generated unimodal content more aligned with the user's personalized preferences, improving content relevance and user satisfaction. Each unimodal content is then compared with other unimodal content to calculate thematic similarity between the content, thereby quantitatively assessing the degree of correlation between content in different modalities. This allows for timely identification of thematic differences or mismatches between content in different modalities, providing a clear basis and direction for subsequent adjustments. According to the determined subject similarity, the unimodal content with lower similarity is updated and optimized to improve its subject consistency with other content. After all unimodal content has undergone necessary adjustments and optimizations, it is merged to form the final target multimodal content. By updating the unimodal content, the present invention eliminates the thematic incoordination problem between different modalities, improves the overall quality and content accuracy of the multimodal content, and thus improves the degree of matching between the multimodal content and user needs. On the other hand, the successful fusion of multiple modal contents provides users with a complete, rich and coordinated multimodal experience, which significantly enhances user satisfaction and the attractiveness of the content.

[0076] Optionally, determining the user's requirement type, target requirement, and the single-modal generator corresponding to the requirement type according to the text description includes:

[0077] Performing feature extraction on the text description to obtain multiple keywords of the text description;

[0078] Performing intent recognition and context analysis on the keywords through a deep learning model to obtain at least one intended target and at least one required keyword of the user;

[0079] Determining the user's need type according to each of the intended goals, and then determining the single-modal generator corresponding to the need type according to the need type;

[0080] When there is a demand keyword, determining the target demand according to the demand keyword;

[0081] When there are multiple demand keywords, a weight value is assigned to each demand keyword by natural language processing technology, and each demand keyword is clustered by a clustering algorithm to obtain a clustering result for each demand keyword;

[0082] The target demand is obtained according to the weight value corresponding to the demand keyword and the clustering result.

[0083] Specifically, the text description is first feature extracted, and multiple keywords are identified using natural language processing techniques. These keywords are key information units for subsequent demand analysis. Next, a deep learning model is used to identify the keywords' intent and analyze their context. In a preferred embodiment of the present invention, the deep learning model can be a pre-trained language model such as BERT or GPT based on the Transformer architecture, which is used to capture semantic information and contextual dependencies in the text. For example, in a practical application, if a user enters "create a technological PPT introducing artificial intelligence," the system encodes the text using the BERT model and then performs contextual analysis to identify demand keywords such as "create," "introduce artificial intelligence," "technological," and "PPT." The model then further analyzes the semantic roles and relationships between these demand keywords, determining that the user's intended goal is to create a presentation on a specific theme. Based on each intended goal, the system determines the user's need type, such as text content creation or image design, and matches it with a corresponding unimodal generator. When there are multiple demand keywords, natural language processing techniques are used to assign a weight to each keyword, which reflects the keyword's importance in the text and its impact on the user's need. In a preferred embodiment of the present invention, the natural language processing technology may be TF-IDF or a word embedding method. Subsequently, a clustering algorithm is used to cluster the demand keywords to obtain clustering results for each keyword, so as to better understand the structure and focus of user needs. In a preferred embodiment of the present invention, the clustering algorithm may be K-means or hierarchical clustering. Finally, based on the weight values ​​corresponding to the demand keywords and the clustering results, the average value of the weight values ​​of all keywords in the cluster indicated by each clustering result is used as the total weight of the corresponding cluster, wherein the cluster with a higher total weight has a greater impact on the user's target needs. The clustering results are integrated according to the total weight of the cluster, for example, the clustering results and their corresponding total weights are associated and stored, thereby accurately providing guidance for subsequent content generation.

[0084] In an embodiment of the present invention, by introducing deep learning models and natural language processing technology, accurate analysis of user needs is achieved. By leveraging the semantic understanding and contextual analysis capabilities of deep learning models, the system can accurately identify the user's intended goals and demand keywords. At the same time, weight allocation and clustering processing further refine the understanding of multi-keyword needs, enhancing the accuracy and comprehensiveness of demand analysis. This not only improves the quality of the system's response to user input, but also provides more targeted guidance for subsequent single-modal content generation, ensuring that the generated content is more in line with user expectations.

[0085] Optionally, the unimodal generator includes a text generator, and generating unimodal content corresponding to each requirement type by the unimodal generator corresponding to each requirement type according to the target requirement and the user's historical data includes:

[0086] When the single-modal generator is the text generator, the demand keywords are integrated according to the clustering results of all the demand keywords by using natural language generation technology to obtain a keyword list;

[0087] Perform content planning based on the keyword list to obtain initial text content;

[0088] The text content corresponding to the demand keyword in the initial text content is enhanced according to the historical data and the weight value of the demand keyword to obtain final text content, and the text content is used as the single modal content of the text generator.

[0089] Specifically, in a preferred embodiment of the present invention, the text generator comprises a natural language generation unit, a content planning unit, and a text enhancement unit. In this preferred embodiment, the natural language generation unit incorporates a pre-trained language model based on the Transformer architecture, leveraging the language model's predictive capabilities to generate sentences that conform to grammatical rules and are semantically coherent. The natural language generation unit first analyzes input keywords, phrases, or data to determine the core content and theme to be conveyed. Next, it constructs the overall structure of the text, including determining the text type (e.g., narrative, expository, argumentative, etc.), paragraph division, and the logical order of each section. Based on this content determination and structure, the natural language generation unit generates specific sentences, selecting appropriate vocabulary and grammatical structures to ensure grammatical correctness and semantic coherence. Finally, the generated sentences are adjusted to make the text read more naturally and fluently, including by using connectives, adjusting sentence length and structure, and optimizing transitions. The content planning unit first conducts a thorough analysis of user needs and intent, clarifying the target audience, purpose, and core message of the text. Based on this needs analysis, the text's theme and main discussion direction are determined. Then, the structure of the text is planned, including the arrangement of the beginning, the middle body and the end, to ensure that the content is logically clear and well-structured. In a preferred embodiment of the present invention, the content planning unit includes a recurrent neural network and a keyword extraction model based on deep learning. The text sequence input by the user is processed by the LSTM model to capture the semantic information in the text to identify the user's intention, and then a convolutional neural network is used to perform a convolution operation on the text to extract the key features in the text, thereby identifying the keywords related to the topic. The text enhancement unit strengthens the relevant keywords and their surrounding content in the text according to the weight value of the required keywords, increases the frequency of occurrence of the keywords, highlights the keyword-related parts, and performs synonym expansion on the keywords, etc., to enhance the topic relevance and attractiveness of the text. In a preferred embodiment of the present invention, the text enhancement unit includes a Word2Vec word embedding model and a style transfer model. The Word2Vec word embedding model maps keywords to a continuous vector space so that semantically similar words are closer in the vector space. When performing synonym expansion, words that are closer to the target word vector can be replaced or expanded. Then, using a GAN-based style transfer model, the generator and discriminator are trained to achieve text style transformation, ultimately generating the unimodal content of the text generator. Therefore, after receiving user requirements, the text generator first integrates the clustering results of all required keywords using natural language generation technology. The key to this step lies in the rational selection and organization of keywords to form a structured keyword list.For example, in a real-world application, suppose a user's demand keywords are clustered into several key categories, such as "artificial intelligence," "technological trends," and "application scenarios." The text generator then organizes these keywords into a hierarchical list based on logical order and semantic associations. Next, content planning is performed based on this keyword list. This step involves designing the text structure, such as determining the introduction, middle, and conclusion of the text, as well as the detailed arrangement of each section. During the content planning stage, the focus of the content is adjusted based on pre-set templates or user historical data. For example, if a user's historical data indicates a high interest in "artificial intelligence applications in the medical field," the system will appropriately increase the weight of related content when planning the text, with this weight being reflected in the weight. Finally, the text generator enhances the initial text based on the user's historical data and the weight of the demand keywords. This step enhances the relevance and depth of the text by highlighting keyword-related content, elaborating on it, or providing additional examples. For example, in the generated text, the system may insert relevant recent research findings or industry data for the highly weighted keyword "artificial intelligence" to enhance the content's authority and appeal.

[0090] In the embodiments of the present invention, the accuracy and personalization of text generation are significantly improved through refined keyword processing and text enhancement mechanisms. The combination of natural language generation technology and cluster analysis enables the text generation subunit to efficiently integrate scattered keyword information and construct logically clear and semantically coherent text content. The content planning and text enhancement process fully considers the user's historical preferences and keyword weights, ensuring that the generated text not only meets the user's specific needs but also highlights the key content that the user is concerned about.

[0091] Optionally, the unimodal generator includes an image generator, and generating unimodal content corresponding to each demand type by the unimodal generator corresponding to each demand type according to the target demand and the user's historical data includes:

[0092] When the single-modality generator is the image generator, determining the theme and style of the image corresponding to the demand keyword according to the clustering results of all the demand keywords;

[0093] Generate image content corresponding to the required keywords based on the theme and style using computer vision technology and image processing algorithms;

[0094] The image content corresponding to the demand keyword is optimized and integrated with the historical data of the theme and style to obtain final image content, and the final image content is used as the single-modal content of the image generator.

[0095] Specifically, the image generator's structure includes a theme identification unit, a style determination unit, an image generation unit, and an image optimization unit. The theme identification unit is responsible for extracting key visual elements from the target keywords, such as building outlines, aircraft, and neon lights in the context of "future city," and converting them into specific image content. The style determination unit focuses on identifying and defining the aesthetic characteristics of the image from the target keywords. For example, "technological" may involve the use of cool tones, metallic textures, and luminous effects, ensuring that the generated image meets the user's style expectations. In a preferred embodiment of the present invention, the theme identification unit can be a Transformer-based model (such as BERT) combined with a visual vocabulary to extract key visual elements from the target keywords and convert them into image content. The style determination unit can be a convolutional neural network (CNN) model, such as VGG16 or ResNet, which can analyze high-level image features and identify aesthetic characteristics. The image generation unit can be a generative adversarial network, generating high-quality images through adversarial training. The image optimization unit combines image super-resolution models such as SRCNN and EDSR with image style transfer models such as CycleGAN and StarGAN to enhance image resolution and detail, and optimize the image style to suit a specific theme or style. The core component is the image generation unit, which includes deep learning models such as generative adversarial networks (GANs) or variational autoencoders. These models, trained on large amounts of image data, are capable of generating high-quality images that conform to specific styles and themes. For example, in practical applications, the system might invoke a GAN model trained on a dataset of futuristic imagery to generate images featuring futuristic cityscapes, high-tech buildings, and neon lights. The generated images might include elements such as building outlines, street layouts, and aircraft in the sky, all presented in a manner consistent with the futuristic aesthetic, such as using cool tones, metallic textures, and luminous effects. In a preferred embodiment of the present invention, computer vision techniques and image processing algorithms are combined with multiple deep learning models to achieve efficient and high-quality image generation. Generative adversarial networks (GANs), particularly variants such as deep convolutional GANs (DCGANs), utilize adversarial training of generators and discriminators to generate realistic images. Variational autoencoders (VAEs), such as β-VAEs, generate diverse image samples through mapping and reconstruction of latent spaces. Autoregressive models such as PixelCNN++ capture complex image dependencies through pixel-by-pixel predictions, while self-attention models, such as the Transformer-based Vision Transformer, leverage self-attention mechanisms to generate globally consistent images. Conditional generative models such as the CGAN and CVAE models can generate images with specific themes or styles based on additional conditional information, while image style transfer techniques such as CycleGAN allow the style of one image to be applied to another to adjust the style of the generated image.In addition, image super-resolution techniques such as SRCNN and EDSR are used to enhance the resolution and details of generated images. These models and networks together form the technical foundation of the image generator, enabling it to determine the image theme and style based on the clustering results of the required keywords and generate high-quality image content that meets user expectations.

[0096] In this embodiment of the present invention, the image generator achieves efficient response to user needs and generates high-quality image content. By accurately understanding and converting required keywords into specific image themes and styles, the generated image content is ensured to be highly consistent with user expectations, thereby improving user satisfaction. Secondly, by leveraging computer vision technology and deep learning models, the generated images achieve a high level of quality and detail, capable of meeting the needs of professional scenarios.

[0097] Optionally, the unimodal generator includes an audio generator, and generating unimodal content corresponding to each demand type according to the target demand and the user's historical data by the unimodal generator corresponding to each demand type includes:

[0098] When the unimodal generator is the audio generator, determining the theme and style of the audio corresponding to the demand keyword according to the clustering results of all the demand keywords;

[0099] According to the theme and style of the audio, audio elements are selected and speech synthesis is performed on the audio elements to obtain audio content corresponding to the required keywords, and the audio content is used as the unimodal content of the audio generator.

[0100] Specifically, the audio generator first determines the theme and style of the audio corresponding to all required keywords based on the clustering results of these keywords. This process involves semantic analysis of the keywords and audio feature mapping. For example, if the required keyword clustering results indicate that the user requires audio with a "technological feel" and a "future city" theme, the system will identify the theme as "future city" and the style as "technological feel." Next, based on the determined theme and style, the audio generator selects appropriate audio elements from the audio library, such as background music, sound effects, and voice clips. These audio elements may include ambient sound effects of a futuristic city, such as the roar of aircraft or the hum of electronic devices, and musical rhythms that match the technological style. Then, using speech synthesis technology, these audio elements are combined into a coherent audio content. In a preferred embodiment of the present invention, the audio generator can be a speech synthesis model such as DeepVoice, which uses an attention mechanism to quickly generate high-quality speech. In a preferred embodiment of the present invention, speech synthesis technology typically utilizes deep learning models, such as WaveNet or Tacotron, to generate natural and fluent speech and sound effects. For example, in actual applications, the system may call a WaveNet model trained on a technological audio dataset to generate audio content containing futuristic city background sound effects and technological music. The generated audio content may include environmental sound effects, background music, and voice commentary, all of which are presented in a manner that is consistent with the technological style, such as using electronic music elements, high-frequency sound effects, and synthesized speech. Finally, the audio generator optimizes and integrates the generated audio content with relevant thematic and style information in the user's historical data. This step may involve audio volume adjustment, equalization optimization, and reverberation processing to ensure that the final audio content not only meets current needs, but also matches the user's past preferences and usage habits. For example, if the user's historical data shows that they prefer clearer voices and lower background music volume, the system will make corresponding adjustments to the generated audio to make the audio more audible and professional while maintaining a technological feel.

[0101] In an embodiment of the present invention, using speech synthesis technology and deep learning models, the generated audio reaches a high level in quality and naturalness, can meet the usage requirements in professional scenarios, ensure that the generated audio content is highly consistent with the user's expectations, and improve user satisfaction.

[0102] Optionally, comparing the unimodal content corresponding to each demand type to determine the subject similarity between each unimodal content and other unimodal content includes:

[0103] Performing feature extraction on the unimodal content of each demand type to obtain a feature vector of each unimodal content;

[0104] Normalizing the feature vector of each unimodal content to obtain feature vectors of the same scale;

[0105] Cosine similarity calculation is performed based on the feature vectors of the same scale to obtain similarity scores between each of the unimodal content and other unimodal content, and the similarity scores are used as the topic similarity.

[0106] Specifically, we first need to extract features from the unimodal content of each demand type to obtain a feature vector for each unimodal content. Feature extraction is the foundation of this process, transforming complex text, image, or audio content into quantifiable numerical vectors that capture the core characteristics of the content, such as keyword frequency in text, visual elements in images, or spectral features in audio. In a preferred embodiment of the present invention, when processing text content, word embedding technology such as Word2Vec or GloVe can be used to map the words in the text into a high-dimensional vector space, thereby forming a feature vector of the text; when processing image content, a convolutional neural network can be used to extract the visual features of the image and generate the feature vector of the image; when processing audio content, the spectral features of the audio can be extracted using technologies such as Mel-frequency cepstral coefficients to form the feature vector of the audio. Next, the feature vector of each unimodal content is normalized to obtain feature vectors of the same scale. Since the feature vectors of different modalities may have different numerical ranges and scales, all feature vectors can be scaled to a uniform scale range through normalization, for example, the numerical range of all feature vectors can be adjusted to between 0 and 1. In a preferred embodiment of the present invention, commonly used normalization methods include Z-score normalization and Min-Max normalization. Z-score normalization subtracts the mean and divides by the standard deviation to make the mean of the feature vector 0 and the standard deviation 1; Min-Max normalization scales the numerical range of the feature vector to a specified interval through linear transformation, such as [0, 1]. After normalization, the feature vectors of content from different modalities can be compared and analyzed on the same scale. Finally, cosine similarity is calculated based on the feature vectors at the same scale to obtain similarity scores between each unimodal content and other unimodal content. These similarity scores are used as thematic similarity. Cosine similarity is a commonly used similarity metric that measures similarity by calculating the cosine of the angle between two vectors. Cosine similarity values ​​range from -1 to 1, where 1 indicates that the two vectors are identical, 0 indicates that the two vectors are orthogonal (no similarity), and -1 indicates that the two vectors are completely opposite. In practical applications, cosine similarity can effectively capture thematic and stylistic consistency of content from different modalities. For example, when creating a science-themed PPT, the system can extract feature vectors for the text, image, and audio content. Using cosine similarity, similarity scores are calculated between text and image, text and audio, and image and audio. These scores quantify thematic similarity between content from different modalities and help the system identify content with thematic deviations or mismatches.

[0107] In the embodiment of the present invention, feature extraction and standardization processing ensure that different modal contents can be compared on the same scale, eliminating comparison bias caused by scale differences and improving the accuracy of similarity calculation. Secondly, the use of cosine similarity calculation provides an effective method for quantitatively evaluating the similarity of topics in different modal contents, which can help the system to promptly discover inconsistencies between contents and provide clear guidance for subsequent adjustments and optimizations. Finally, this process helps to improve the overall quality and user experience of multimodal content, ensure that the generated content is highly consistent in theme and style, and provide users with a higher quality and more attractive multimodal content experience.

[0108] Optionally, updating the unimodal content according to the subject similarity between each unimodal content and other unimodal content includes:

[0109] generating a similarity matrix according to the subject similarities between each unimodal content and other unimodal content;

[0110] Analyzing the similarity matrix to determine the unimodal content that needs to be adjusted;

[0111] Obtaining a parameter adjustment strategy for the unimodal generator corresponding to the unimodal content that needs to be adjusted based on the topic similarity between the unimodal content that needs to be adjusted and other unimodal contents;

[0112] updating, according to the parameter adjustment strategy, the model parameters of the unimodal generator corresponding to the unimodal content that needs to be adjusted;

[0113] The updated unimodal generator is used to regenerate unimodal content corresponding to each of the demand types according to the target demand and the user's historical data, and the regenerated unimodal content is used as the updated unimodal content.

[0114] Specifically, the process of updating unimodal content based on the thematic similarity between each unimodal content and other unimodal content involves first generating a similarity matrix based on the thematic similarity between each unimodal content and other unimodal content. A similarity matrix is ​​a two-dimensional matrix in which each element represents the similarity score between two unimodal contents. For example, in a multimodal generation task involving text, images, and audio content, the similarity matrix would quantify the similarity between text and image, text and audio, and image and audio. These similarity scores, derived through cosine similarity calculations or other similarity metrics, can reflect the thematic and stylistic consistency of content across modalities. Next, the similarity matrix is ​​analyzed to identify unimodal content that requires adjustment. This step typically involves setting a similarity threshold, below which content pairs are considered inconsistent and require adjustment. For example, if the thematic similarity between a particular unimodal content A and all other unimodal content B falls below the similarity threshold, then unimodal content A is directly deemed to require adjustment. If the topic similarity between a certain unimodal content A and only one other unimodal content B is below the similarity threshold, the topic similarity between this other unimodal content B and all other unimodal content is further examined. If the topic similarity between this other unimodal content B and other unimodal content also falls below the similarity threshold, this other unimodal content B is prioritized as the content requiring adjustment. If the topic similarity between unimodal content A and some unimodal content B is below the similarity threshold, the unimodal content requiring adjustment is determined by comparing the number of content pairs with unimodal content A that fall below the similarity threshold and the number of content pairs with unimodal content B that fall below the similarity threshold. The unimodal content with the largest number of content pairs below the similarity threshold is selected as the unimodal content requiring adjustment. In this way, the system can identify which unimodal content deviates from the topic of other content. Based on the topic similarity between the unimodal content requiring adjustment and other unimodal content, the parameter adjustment strategy for the unimodal generator corresponding to the unimodal content requiring adjustment is derived. This step requires the system to analyze the specific characteristics of inconsistent content and determine how to adjust the parameters of the generator to improve the consistency of the generated content. For example, if the text content and the image content are inconsistent in topic, the system may adjust the keyword weights of the text generator, increase the weights of keywords related to the image theme, or adjust the style parameters of the image generator so that the generated image is more consistent with the theme of the text. According to the parameter adjustment strategy, the model parameters of the unimodal generator corresponding to the unimodal content that needs to be adjusted are updated. This step involves actually modifying the model parameters of the generator, such as adjusting the weights of the neural network, changing the parameters of the generation algorithm, etc. The updated model parameters will guide the generator to produce content that is more in line with the theme requirements in the subsequent generation process.Finally, the updated unimodal generator regenerates the unimodal content corresponding to each requirement type based on the target requirements and the user's historical data, and uses this regenerated unimodal content as the updated unimodal content. For example, in generating a science-themed PPT, if the similarity between text and image falls below a threshold, the system will regenerate the text and image content using the updated text and image generators to ensure consistency in theme and style. The regenerated content will better align with the overall multimodal content's thematic requirements, improving content coordination and quality.

[0115] In an embodiment of the present invention, by generating a similarity matrix and an analysis matrix to identify unimodal content that needs to be adjusted, and adjusting the parameters of the unimodal generator according to the subject similarity, accurate coordination and consistency optimization of multimodal content are achieved. First, the similarity matrix provides an effective tool for quantitatively evaluating the consistency between different modal contents, helping the system to promptly discover inconsistencies between contents. Secondly, the parameter adjustment strategy ensures that the generator can perform targeted optimization based on the similarity requirements of the content, thereby improving the accuracy and relevance of the generated content. Finally, by regenerating the content, the system can ensure that all unimodal content remains highly consistent in theme and style, thereby improving the overall quality of multimodal content and user experience.

[0116] Optionally, the intelligent agent system further includes a cross-modal alignment submodule, which fuses all the updated single-modal contents to obtain multimodal content corresponding to the target demand, including:

[0117] Inputting all the updated unimodal contents into the cross-modal alignment submodule;

[0118] Performing temporal and structural alignment on all the updated unimodal contents and other unimodal contents through a cross-modal alignment submodule to obtain all cross-modally aligned unimodal contents;

[0119] Performing feature extraction on all the cross-modal aligned unimodal contents to obtain multiple key interaction points;

[0120] The monomodal content after all cross-modal alignment is matched according to the key interaction points to obtain the multimodal content corresponding to the target demand.

[0121] Specifically, all updated unimodal content is first input into the cross-modal alignment submodule, which comprises a sequence model, a Transformer model, and a graph neural network. The sequence model is used to capture the temporal dependencies between modalities, such as text and audio. The Transformer model then analyzes the structural features of modalities like text and images. Template matching or format conversion algorithms are then used to unify the layout structures of each modality. Finally, a graph neural network is used to establish intermodal associations and perform content fusion. The cross-modal alignment submodule ensures that content from different modalities is consistent in terms of time and structure, thus providing a coherent foundation for the final multimodal content. Regarding temporal alignment, the cross-modal alignment submodule analyzes the temporal characteristics of each unimodal content. For example, for an educational video project containing text, images, and audio, the reading time of the text, the display time of the images, and the playback time of the audio all need to be synchronized. The system may analyze the text's speaking rate, the switching frequency of the images, and the rhythm of the audio to adjust their respective temporal parameters so that they are presented coherently on the timeline. Structural alignment focuses on the logical and formatting consistency of the content. For example, when generating a presentation on a science and technology theme, the cross-modal alignment submodule ensures that the text paragraphs, image positions, and audio narrations follow a unified layout structure on each slide, in line with the overall design specifications of the presentation. After completing the time alignment and structural alignment, the system performs feature extraction on all cross-modal aligned unimodal content. This step aims to identify key interaction points in each modal content, that is, those elements with strong correlation and interaction potential between different modalities. For example, in a multimodal content explaining the application of artificial intelligence, the keyword "machine learning algorithm" mentioned in the text, the algorithm flowchart shown in the image, and the detailed explanation of the algorithm principle in the audio together constitute a key interaction point. Specifically, feature extraction techniques are used, such as using BERT to extract text features, ResNet to extract image features, and VGGish to extract audio features, to obtain feature vectors from each unimodal content. Next, the system uses the cosine similarity algorithm and semantic association analysis, combined with historical user data, to identify feature combinations with high similarity and significant semantic value. The cosine similarity algorithm is used to calculate similarity, while semantic association analysis is used to identify semantically significant feature combinations (for example, using a large transformer-based language model to identify semantically significant feature combinations). These combinations constitute key interaction points. Based on these key interaction points, the system matches all cross-modally aligned unimodal content. This matching process uses an attention mechanism algorithm to calculate attention weights for feature vectors from different modalities, highlighting important features at key interaction points and focusing the model on relevant content, ensuring that content from different modalities seamlessly integrates and complements each other at these key interaction points.For example, at the key interaction point "machine learning algorithm", the system will adjust the level of detail of the text content, the visual emphasis of the image, and the depth of the audio commentary, so that they together construct a complete and coherent explanation segment.

[0122] In an embodiment of the present invention, by introducing a cross-modal alignment submodule and a sophisticated fusion step, the coordination of multimodal content and the user experience are significantly improved. Time alignment and structural alignment ensure the synchronization and logical consistency of content in different modalities during presentation, eliminating the inconsistency in the content display process. Feature extraction and key interaction point matching further enhance the inherent coherence of multimodal content, allowing different modalities such as text, images, and audio to complement each other and jointly construct a complete and attractive multimodal narrative.

[0123] Combine Figure 3 As shown, a multimodal content generation system provided by an embodiment of the present invention is applied to an intelligent agent system. The intelligent agent system includes a multimodal creation module, which includes multiple single-modal generators. The system includes:

[0124] A receiving unit, configured to obtain a user's text description;

[0125] An analyzing unit, configured to determine, based on the text description, the user's requirement type, target requirement, and the single-modal generator corresponding to the requirement type;

[0126] a single-modal processing unit, configured to generate, when there are multiple demand types, single-modal content corresponding to each demand type according to the target demand and the user's historical data by the single-modal generator corresponding to each demand type;

[0127] a comparing unit, configured to compare the unimodal content corresponding to each of the demand types to determine a theme similarity between each of the unimodal content and the other unimodal content;

[0128] An updating unit is configured to update the unimodal content according to the subject similarity between each unimodal content and other unimodal content, and fuse all the updated unimodal content to obtain multimodal content corresponding to the target demand.

[0129] The advantages of the multimodal content generation system of the present invention over the prior art are the same as the advantages of the multimodal content generation method described above over the prior art, and will not be repeated here.

[0130] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the multimodal content generation method described above is implemented.

[0131] In other words, a non-volatile computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following operations:

[0132] Get the user's text description;

[0133] Determining, according to the text description, the user's demand type, target demand, and the single-modal generator corresponding to the demand type;

[0134] When there are multiple demand types, generating a single modal content corresponding to each demand type according to the target demand and the user's historical data by the single modal generator corresponding to each demand type;

[0135] Comparing the unimodal content corresponding to each demand type to determine the subject similarity between each unimodal content and other unimodal content;

[0136] The unimodal content is updated according to the subject similarity between each unimodal content and other unimodal content, and all the updated unimodal content are merged to obtain multimodal content corresponding to the target demand.

[0137] The advantages of the computer-readable storage medium of the present invention over the prior art are the same as the advantages of the above-mentioned multimodal content generation method over the prior art, and will not be repeated here.

[0138] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multimodal content generation method, characterized in that: The method is applied to an intelligent agent system, the intelligent agent system including a multimodal creation module and a cross-modal alignment submodule, the multimodal creation module including a plurality of single-modal generators, and the method comprising: Get the user's text description; Determining, according to the text description, the user's demand type, target demand, and the single-modal generator corresponding to the demand type; When there are multiple demand types, generating a single modal content corresponding to each demand type according to the target demand and the user's historical data by the single modal generator corresponding to each demand type; Comparing the unimodal content corresponding to each demand type to determine the subject similarity between each unimodal content and other unimodal content; updating the unimodal content according to the subject similarity between each unimodal content and other unimodal content, and fusing all the updated unimodal content to obtain multimodal content corresponding to the target demand; Wherein, the updating of the unimodal content according to the subject similarity between each unimodal content and other unimodal content specifically includes: generating a similarity matrix according to the subject similarity between each unimodal content and other unimodal content; analyzing the similarity matrix to determine the unimodal content that needs to be adjusted; obtaining a parameter adjustment strategy for the unimodal generator corresponding to the unimodal content that needs to be adjusted according to the subject similarity between the unimodal content that needs to be adjusted and other unimodal content; updating the model parameters of the unimodal generator corresponding to the unimodal content that needs to be adjusted according to the parameter adjustment strategy; regenerating the unimodal content corresponding to each of the demand types according to the target demand and the user's historical data through the updated unimodal generator, and using the regenerated unimodal content as the updated unimodal content; The fusing of all the updated unimodal contents to obtain the multimodal content corresponding to the target demand specifically includes: inputting all the updated unimodal contents into the cross-modal alignment submodule; performing time alignment and structural alignment on all the updated unimodal contents with other unimodal contents through the cross-modal alignment submodule to obtain all the cross-modally aligned unimodal contents; performing feature extraction on all the cross-modally aligned unimodal contents to obtain multiple key interaction points; and matching all the cross-modally aligned unimodal contents according to the key interaction points to obtain the multimodal content corresponding to the target demand.

2. The multimodal content generation method according to claim 1, wherein: The determining, based on the text description, the user's requirement type, target requirement, and the single-modal generator corresponding to the requirement type includes: Performing feature extraction on the text description to obtain multiple keywords of the text description; Performing intent recognition and context analysis on the keywords through a deep learning model to obtain at least one intended target and at least one required keyword of the user; Determining the user's need type according to each of the intended goals, and then determining the single-modal generator corresponding to the need type according to the need type; When there is a demand keyword, determining the target demand according to the demand keyword; When there are multiple demand keywords, a weight value is assigned to each demand keyword by natural language processing technology, and each demand keyword is clustered by a clustering algorithm to obtain a clustering result for each demand keyword; The target demand is obtained according to the weight value corresponding to the demand keyword and the clustering result.

3. The multimodal content generation method according to claim 2, wherein: The single-modality generator includes a text generator, and the single-modality generator corresponding to each requirement type generates the single-modality content corresponding to each requirement type according to the target requirement and the user's historical data, including: When the single-modal generator is the text generator, the demand keywords are integrated according to the clustering results of all the demand keywords by using natural language generation technology to obtain a keyword list; Perform content planning based on the keyword list to obtain initial text content; The text content corresponding to the demand keyword in the initial text content is enhanced according to the historical data and the weight value of the demand keyword to obtain final text content, and the text content is used as the single modal content of the text generator.

4. The multimodal content generation method according to claim 2, wherein: The single-modality generator includes an image generator, and the single-modality generator corresponding to each demand type generates the single-modality content corresponding to each demand type according to the target demand and the user's historical data, including: When the single-modality generator is the image generator, determining the theme and style of the image corresponding to the demand keyword according to the clustering results of all the demand keywords; Generate image content corresponding to the required keywords based on the theme and style using computer vision technology and image processing algorithms; The image content corresponding to the demand keyword is optimized and integrated with the historical data of the theme and style to obtain final image content, and the final image content is used as the single-modal content of the image generator.

5. The multimodal content generation method according to claim 2, wherein: The unimodal generator includes an audio generator, and generating unimodal content corresponding to each demand type according to the target demand and the user's historical data by the unimodal generator corresponding to each demand type includes: When the unimodal generator is the audio generator, determining the theme and style of the audio corresponding to the demand keyword according to the clustering results of all the demand keywords; According to the theme and style of the audio, audio elements are selected and speech synthesis is performed on the audio elements to obtain audio content corresponding to the required keywords, and the audio content is used as the unimodal content of the audio generator.

6. The multimodal content generation method according to claim 1, wherein: The comparing the unimodal content corresponding to each demand type to determine the subject similarity between each unimodal content and other unimodal content includes: Performing feature extraction on the unimodal content of each demand type to obtain a feature vector of each unimodal content; Normalizing the feature vector of each unimodal content to obtain feature vectors of the same scale; Cosine similarity calculation is performed based on the feature vectors of the same scale to obtain similarity scores between each of the unimodal content and other unimodal content, and the similarity scores are used as the topic similarity.

7. A multimodal content generation system, characterized in that: The system is applied to an intelligent agent system, the intelligent agent system includes a multimodal creation module, the multimodal creation module includes multiple single-modal generators, and the system includes: A receiving unit, configured to obtain a user's text description; An analyzing unit, configured to determine, based on the text description, the user's requirement type, target requirement, and the single-modal generator corresponding to the requirement type; a single-modal processing unit, configured to generate, when there are multiple demand types, single-modal content corresponding to each demand type according to the target demand and the user's historical data by the single-modal generator corresponding to each demand type; a comparing unit, configured to compare the unimodal content corresponding to each of the demand types to determine a theme similarity between each of the unimodal content and the other unimodal content; an updating unit, configured to update the unimodal content according to the subject similarity between each unimodal content and other unimodal content, and fuse all the updated unimodal content to obtain multimodal content corresponding to the target demand; Wherein, the updating of the unimodal content according to the subject similarity between each unimodal content and other unimodal content specifically includes: generating a similarity matrix according to the subject similarity between each unimodal content and other unimodal content; analyzing the similarity matrix to determine the unimodal content that needs to be adjusted; obtaining a parameter adjustment strategy for the unimodal generator corresponding to the unimodal content that needs to be adjusted according to the subject similarity between the unimodal content that needs to be adjusted and other unimodal content; updating the model parameters of the unimodal generator corresponding to the unimodal content that needs to be adjusted according to the parameter adjustment strategy; regenerating the unimodal content corresponding to each of the demand types according to the target demand and the user's historical data through the updated unimodal generator, and using the regenerated unimodal content as the updated unimodal content; The fusing of all the updated unimodal contents to obtain the multimodal content corresponding to the target demand specifically includes: inputting all the updated unimodal contents into the cross-modal alignment submodule; performing time alignment and structural alignment on all the updated unimodal contents with other unimodal contents through the cross-modal alignment submodule to obtain all the cross-modally aligned unimodal contents; performing feature extraction on all the cross-modally aligned unimodal contents to obtain multiple key interaction points; and matching all the cross-modally aligned unimodal contents according to the key interaction points to obtain the multimodal content corresponding to the target demand.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal content generation method according to any one of claims 1 to 6 is implemented.

Images