Multi-modal content generation method and related device

Abstract

The invention provides a multi-modal content generation method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of artificial intelligence content generation. According to the specific implementation scheme, at least one material is displayed to a user, wherein the material comprises at least one mode; obtaining a user instruction of the user on the material; and generating multi-modal content based on the material and the user instruction, wherein the multi-modal content comprises at least one modal. The detail control granularity of the user on the generated content is enhanced, the flexibility of content generation is improved, and the user experience of personalized generation of the multi-modal content is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of artificial intelligence content generation technology, and more particularly to a multimodal content generation method and device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of AI content generation, existing technologies mainly focus on generating content using single-modal materials based on user instructions. For example, in the field of text generation, large language models use text materials and user instructions to generate new text; in the field of image generation, large models use image materials and user instructions to generate new images.

[0003] However, in existing AI content generation methods, users provide user instructions by describing all materials as a whole. This user instruction method limits the granularity of user control over material utilization and generated content. The lack of flexibility in content generation often leads to a gap between generated content and user expectations. Users find that although they provide an overall description, the generated content often fails to fully meet their personalized expectations in terms of details.

[0004] In addition, the existing single-modal content generation method limits the diversity and richness of the content. Multimodal content, such as content that contains a combination of images, text, audio, video, etc., is increasingly widely used in various fields, and users have a stronger demand for personalized content. However, existing technologies still face challenges in achieving personalized generation of multimodal content. Summary of the invention

[0005] The present disclosure provides a multimodal content generation method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to a first aspect, a multimodal content generation method is provided, the method comprising: displaying at least one material to a user, the material comprising at least one modality; obtaining a user instruction of the user for the material; and generating multimodal content based on the material and the user instruction, the multimedia content comprising at least one modality.

[0007] According to a second aspect, a multimodal content generation device is provided, which includes: a display unit, which displays at least one material to a user, and the material includes at least one modality; an instruction unit, which obtains the user's instructions for the material; and a generation unit, which generates multimodal content based on the material and the user instructions, and the multimedia content includes at least one modality.

[0008] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect.

[0009] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0010] According to a fifth aspect, a computer program product is provided, comprising a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0011] The multimodal content generation method and device provided by the present disclosure, first of all, enhance the user's control over the details of material utilization and content generation. Traditional methods can only use overall prompt words as user instructions, which are severely limited in terms of material utilization and content generation details. The present disclosure innovatively associates user instructions with single materials, specific subsets of materials, and specific parts of materials, significantly enhancing the user's control over the details of material utilization and content generation. This enhancement is specifically reflected in the interaction between materials and user instructions. Users can operate materials at a micro level based on their own needs, with the help of custom instructions or system predefined instructions.

[0012] Secondly, the flexibility of content generation is improved. The disclosed method displays at least one material containing at least one modality to the user, and the user can issue user instructions for a collection of materials, a single material, or part of the material. As a result, the user can control the content generation process more finely, break through the limitations of the overall description, better meet personalized expectations, and effectively improve the flexibility of content generation. Whether it is the combination of materials or the associated use of user instructions, diversified adjustments can be achieved according to user needs. In addition, compared with the limitations of existing single-modal content generation methods on content diversity and richness, the new method disclosed in the present invention generates multimodal content, including a combination of multiple forms such as images, text, audio, and video, which further enhances the flexibility of the content in terms of form and details.

[0013] Finally, the user experience of personalized generation of multimodal content is improved. By enhancing the granularity of detail control and improving flexibility, the present disclosure can fully meet the needs of users to personalize content. Each user has his or her unique creativity and expression demands, and the technical solution of the present disclosure provides strong support for the realization of these needs. The personalized experience also runs through the entire process of interaction between the user and the system, from material selection, instruction issuance to content generation, and each link is centered on the user. This personalized capability gives users a better user experience during the creation process.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0016] Figure 1 is a schematic diagram of a content generation scenario provided by an embodiment of the present disclosure;

[0017] Figure 2 is a flowchart of a multimodal content generation method provided by an embodiment of the present disclosure;

[0018] Figure 3 is a schematic diagram of a multimodal content generation method provided by an embodiment of the present disclosure;

[0019] Figure 4 is a schematic diagram of a multimodal content generation method provided by an embodiment of the present disclosure;

[0020] Figure 5 is a flowchart of a multimodal content generation method provided by an embodiment of the present disclosure;

[0021] Figure 6 is a flowchart of a multimodal content generation method provided by an embodiment of the present disclosure;

[0022] Figure 7 is a structural diagram of a multimodal content generation device provided by an embodiment of the present disclosure;

[0023] Figure 8 It is a block diagram of an electronic device used to implement the multimodal content generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In the field of artificial intelligence content generation, current technologies have significant technical problems. Most existing technologies generate content based on user instructions using single-modal materials, such as large language models in the field of text generation using text materials and instructions to generate new text, and large models in the field of image generation using image materials and instructions to generate new images. However, its user instruction method is to describe all materials as a whole, which severely limits the user's control over the details of material utilization and content generation, resulting in poor flexibility in content generation. The generated content often differs from user expectations. Even if the user gives an overall description, it is difficult to meet personalized expectations in details. Moreover, the single-modal content generation method limits the diversity and richness of the content.

[0026] In response to the above technical problems, the present disclosure proposes a multimodal content generation method.

[0027] It is understandable that the method can be integrated into various computing devices, including but not limited to terminal devices with text processing capabilities (such as smart phones, tablets, personal computers, etc.) and servers (such as application servers, Web servers, whether single servers or cluster-deployed server systems). The method does not rely on a specific hardware platform or software architecture. During the execution of the method, both independently running terminal devices and terminal-server architectures that work collaboratively through a network can effectively utilize the method disclosed in the present invention. When the terminal device is executed independently, the method can be independent of the external network. In scenarios where higher processing performance or wider data resource support is required, the method of the present invention also supports communication between the terminal device and the server, utilizing the powerful computing power and rich data resources of the server to jointly complete the method. The method can adapt to different operating systems and platform environments, including mobile operating systems such as iOS, Android, and Hongmeng, desktop operating systems such as Windows and macOS, and server operating systems such as Linux and Unix.

[0028] It can be understood that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] For ease of understanding, Figure 1A content generation scenario according to an embodiment of the present disclosure is shown.

[0030] In a typical video recommendation application scenario, a user accesses a server with content generation capabilities through a client application (e.g., a browser or APP) on his smart device (e.g., a mobile phone, tablet, or computer). The user first starts the client and connects to the server.

[0031] like Figure 1 As shown in the figure, after the client is started, there is a special material operation area on the interface. Users can select files from local devices through the upload function. These file types are diverse, covering various modalities such as text, images, audio, and video. At the same time, users can also use the search function to find materials in the platform's material library. The search keywords can be subject type, style characteristics, etc. The materials in the material library are also rich and diverse modal forms. After the server receives the user's upload or search request, it quickly displays the acquired materials in the display area of ​​the client. For example, the current material operation area may include 5 materials, material 1 is a document about an ancient myth, material 2 is an audio clip telling the background of the myth, and materials 3-5 are illustrations of several mythological characters. After that, the user issues various instructions for the displayed materials. For example, the user hopes to use the style of a certain illustration as a blueprint, combine a certain plot fragment in the myth document, and add a specific audio atmosphere to generate a multimodal introduction document. Based on these materials and user instructions, the server integrates, processes and innovates the materials through complex algorithms and models, and ultimately generates multimodal content that meets user needs. These contents can be viewed and edited directly on the client and can also be shared on other platforms.

[0032] The multiple embodiments of the present disclosure are not isolated individuals from each other, but rather present mutually complementary characteristics. The schemes involved in each embodiment are intended to exemplify the overall scheme from a specific aspect or dimension. The contents of each embodiment in the present disclosure can be integrated with other embodiments to constitute one of the multiple feasible schemes of the present disclosure. Such a design enables the present disclosure to form an organic whole, which together provides comprehensive and systematic technical support for achieving the purpose of the invention, so as to meet the needs of different application scenarios, and ensure the integrity and effectiveness of the technical scheme of the present disclosure in multiple dimensions and levels.

[0033] Figure 2 The flowchart 100 of the multimodal content generation method according to an embodiment of the present disclosure is shown. The multimodal content generation method comprises the following steps:

[0034] Step 101: display at least one material to a user, wherein the material includes at least one modality.

[0035] Step 101 breaks the limitation of the prior art that only operates around single-modal materials by displaying materials of at least one modality to the user, and solves the problem that the display of single-modal materials cannot meet the initial requirements of multimodal content generation. Step 101 presents materials containing at least one modality to the user, so that the user can be exposed to a rich variety of material forms, such as text, images, audio, video and other modalities. After seeing materials of multiple modalities, the user can more intuitively understand the role of different modalities in content expression. For example, through the display of image and audio materials, the user can better imagine how to combine them with text to generate more vivid and expressive multimodal content.

[0036] Unimodal material refers to material that contains only one way of expressing information. This type of material is relatively pure, and is mainly centered around this single mode when it is processed and understood. For example, text material only conveys information through words, such as academic papers, novels, news reports, etc., all of which rely on the organization and expression of words. Image material conveys information through visual elements such as color, shape, composition, etc., such as a painting, a satellite remote sensing image, etc. Audio material relies on factors such as the frequency, timbre, and volume of sound, such as music works, audiobooks, etc. If video material is considered from a single perspective, such as only focusing on its picture part or audio part, it can also be regarded as unimodal material, such as a surveillance video without sound (unimodal with picture as the main part) or a radio drama with only audio (unimodal with audio as the main part).

[0037] Multimodal materials are materials that contain two or more different ways of expressing information. These different modes complement each other and can provide richer and more three-dimensional information. Take graphic documents as an example. They combine text and images. The text can provide detailed descriptions, explanations or supplement relevant background knowledge of the image, while the image can make the text content more intuitive and vivid. For example, in a children's picture book, the text describes a simple fairy tale, while the exquisite illustrations bring the scenes and characters in the story to life, making it easier for children to understand and immerse themselves in the story.

[0038] Graphic documents are a typical multimodal material that organically combines text and images. In a graphic document, there are not only views, events, knowledge, etc. explained in text, but also visual elements such as scenes, objects, and characters that are intuitively displayed through images. For example, a travel brochure contains detailed text introductions of various tourist attractions, including the history, characteristics, and travel guides of the attractions, as well as corresponding photos of the attractions, so that readers can understand the style of the attractions more intuitively. In the content generation scenario, when such graphic documents are used as materials, more diverse creations can be made according to user instructions. For example, if the user's instructions are to generate more vivid introduction content based on an introduction to the tourist attraction, then a multimedia display content that includes an in-depth interpretation of the characteristics of the attraction and is accompanied by exquisite pictures can be generated by combining the detailed description of the text in the graphic document and the intuitive display of the image, or a voice explanation content that combines the visual sense of the attraction photos and the emotional expression of the text can be generated.

[0039] Multimodal materials can continue to add new modalities. For example, a slideshow document (PPT) is a multimodal document mainly composed of text and images (including charts, graphs, etc.). When audio or video content is added, it becomes a more complex multimodal document. For another example, an animated video itself is a multimodal material that contains images (animated pictures) and audio (character dialogue, background music, etc.). When subtitles are added, a richer multimodal document is formed.

[0040] Multimodal materials can exist in the form of files or can be presented as multimedia content that can be opened online. For example, augmented reality (AR) or virtual reality (VR) documents. Taking AR tourist guide materials as an example, when tourists use mobile devices to scan scenic spots, virtual information will be displayed on the device screen superimposed on the real scene. This virtual information may include text introductions (such as historical allusions of scenic spots), images (such as restored images of ancient buildings), audio (such as tour guides’ explanations), and even simple animations (such as showing the construction process of buildings). This kind of multimodal material allows tourists to immerse themselves in a rich information environment and experience and understand tourist attractions in a whole new way.

[0041] When displaying materials to users, two effective approaches, direct display and preview mode, can be used according to the material type. For relatively simple modal content such as text and images that have low system resource and loading requirements, direct display can be used. When multimodal content involves relatively complex and data-intensive modalities such as audio, video, and animation, preview mode can provide users with a better experience. For example, in the preview mode of audio materials, the browser presents users with a simple audio playback interface, including basic control elements such as play / pause buttons, volume adjustment sliders, and progress bars. For another example, in the preview mode of animation materials, the animation will be played at a lower complexity, and some details may be simplified or the frame rate may be reduced so that users can quickly understand the theme, characters, and main actions of the animation.

[0042] The user can obtain the material from the local device or the network device. The user can select files from the local device through the upload function. These files are of various types, covering various modalities such as text, image, audio, video, etc. At the same time, the user can also use the search function to find materials in the material library of the network device.

[0043] For obtaining materials on the local device, the present invention supports searching for materials that meet specific conditions in the user's local hard disk or other storage devices. For example, if the user needs image materials, the present invention can automatically search and filter out relevant image documents based on the keywords or specific themes, color features, etc. entered by the user.

[0044] For the acquisition of materials on network devices, the present disclosure supports the use of network connections to access online network disks, image libraries, text resource libraries, audio libraries or video libraries, etc., and select appropriate materials from them. For example, if a user needs article materials, the present disclosure can search for articles on specific topics in the text resource library based on the keywords entered by the user. During the material acquisition process, the present disclosure allows users to preview and select search results to ensure that the materials ultimately used fully meet the user's needs and expectations. Users can sort, filter and compare search results to find the most suitable materials.

[0045] The material library can store a large amount of materials, which are of various types and cover various modes. It can include materials in text form, such as academic papers, literary works, news reports, technical documents, etc.; it can include image materials, including photos, paintings, design drawings, icons, etc.; it can include audio materials, such as music clips, voice recordings, sound effects, etc.; it can include video materials, such as movie clips, advertising videos, teaching videos, etc. There can also be various multi-modal materials, such as documents and slides with pictures and texts.

[0046] The material library provides both regular search and smart search functions. The smart search function is based on advanced technology and provides users with an efficient and accurate search experience. When users have specific material needs, they can enter keywords through the smart search interface. The smart search function is more than just simple text matching. It uses natural language processing technology to understand the semantics of the keywords entered by users. The smart search function can also make personalized recommendations based on the user's search history and usage habits. If the user frequently searches for art-related materials, then in subsequent searches, the system will prioritize materials that are more relevant to the art field at the top of the search results, and may recommend some new materials that the user may be interested in but has not yet searched, further improving the efficiency and satisfaction of users in finding materials.

[0047] The uploading and display of materials are carried out in the material operation area. In different scenarios, the material operation area can be called a canvas or a whiteboard.

[0048] The canvas can be configured with a commonly used basic toolbar, which includes all necessary operation buttons and functional modules, such as text editing, shape drawing, line adding, free brush and file uploading, etc., with clear classification.

[0049] The canvas also has a core visual space for users to create, where users can freely drag, drop, upload and edit various elements. Users can easily drag and drop materials, whether they are locally stored images, audio, video files, or text content obtained from other sources, they can be placed in the center of the canvas through simple drag and drop operations. At the same time, users can also upload materials here. The upload function supports multiple file formats and modal types to ensure that materials from different sources can be imported smoothly. In terms of editing, users have rich editing permissions and can make comprehensive modifications and adjustments to uploaded materials.

[0050] The canvas provides users with functions such as point selection, multiple selection, and box selection to further help users enrich and improve their designs. The point selection function allows users to select a single element by clicking. The multiple selection function allows users to select multiple elements by clicking and using shortcut keys. The box selection function provides users with more powerful batch processing capabilities, allowing users to select multiple materials by box selection.

[0051] In order to achieve more accurate user command input and operation, the canvas has a user command interaction function. When the user right-clicks the material with the mouse, the user command menu can be called up, which contains a series of preset command options for the material. At the same time, the user command input component is closely integrated with the selection control. After the user selects a specific material or material combination, the user can attach personalized user commands through the input component to ensure that the system can accurately understand the user's specific requirements for material processing and creation, thereby achieving high-quality, personalized multimodal content creation.

[0052] Step 102: obtaining the user's instruction on the material.

[0053] Step 102 innovatively associates user instructions with at least one material (in the following embodiments, it can also be a specific subset of the material and a specific part of the material), which significantly enhances the user's detailed control granularity over the use of the material and the generation of content. This method of associating user instructions with specific materials brings about many significant technical effects. First, the user can guide the content generation process more accurately. For example, when the material is a set of historical documents and related historical pictures, the user can instruct the generation of interpretation content closely related to a specific paragraph in a document, and specify the use of a certain picture to assist in the explanation, rather than treating the entire material collection in a general way as in the traditional method, which makes the process control granularity of content generation finer and the user instructions more targeted.

[0054] The specific presentation of user instructions is highly flexible and diverse. On the one hand, users can use natural language to input instructions in the text box. This method is extremely convenient. Users only need to type their ideas in the text box like daily communication. For example, when facing a text material introducing a tourist attraction, the user can enter "Generate content using the style of this classic beach introduction as a template" in the text box related to the text material of the attraction introduction.

[0055] At the same time, voice input is also an important way to present user commands. This brings great convenience to users, especially when their hands are busy or they need to input commands quickly. Users can use the device's microphone to clearly speak their commands. Speech recognition technology can accurately convert user voice commands into information that the system can recognize, thereby achieving efficient content generation.

[0056] In addition, user instructions can also be pre-set in the system. In subsequent embodiments, this approach will show unique advantages. For example, for some specific types of materials, system developers can set up a series of instruction templates in the system in advance based on common needs and patterns. When users browse related materials, these preset instructions can be directly called or used as a reference. This diversified user instruction presentation method can meet the needs of different users in different scenarios, further improving the practicality and ease of use of the multimodal content generation system.

[0057] Step 103: Generate multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode.

[0058] Step 103 enhances the user's detailed control granularity over material utilization and content generation. Associate user instructions with a single material, a specific subset of the material, or a specific part. In the interaction between the material and the user instruction, the user can operate the material at a micro level through customized or system predefined instructions according to their own needs, overcoming the shortcomings of traditional methods in detail control. Secondly, the flexibility of content generation is improved. Whether it is material combination or user instruction association utilization, it can be adjusted in a diversified manner as needed. Unlike the existing single-modal content generation method, the present disclosure can generate multimodal content, covering multiple forms such as images, text, audio, and video. This further enhances the flexibility of the content in terms of form and details. Finally, by enhancing the granularity of detail control and improving flexibility, the user's content personalization needs are fully met.

[0059] When dealing with content generation with multiple modalities and each material has user instructions attached, the following generative large model technologies and related details can be considered. First, the technology of the multimodal Transformer architecture can be used. This architecture can process information of different modalities at the same time. For each modal information of the material, whether it is text, image, audio or video, it can be converted into a vector representation that the model can understand through a specific encoding method. For example, for text, word embedding technology can be used, images can be feature vectors extracted by convolutional neural networks, audio can be extracted through audio feature extraction methods, and videos can be combined with image and audio processing methods. These vector representations can be fused and interacted in the Transformer architecture, and the model learns the association and combination between different materials based on user instructions. At the same time, the model can learn the generation mode of different modal materials under different instructions from a large amount of pre-training data, such as how to combine images and text to generate more style-appropriate content under specific artistic style instructions. In addition, the attention mechanism is crucial in this process. It can help the model focus on the part of the material that the user instruction focuses on. When user instructions point to a specific subset or part of a material, the attention mechanism can highlight these key parts among many modal information, focus computing resources and generation on these places, and more accurately generate multimodal content that meets user instructions. Moreover, between different modalities, the attention mechanism can also coordinate their weights and associations, making the generated content more natural and reasonable in multimodal fusion. During the training process, it is necessary to build a large-scale multimodal dataset, which contains materials of different modalities and the corresponding user instructions and expected multimodal content. By continuously adjusting the model parameters, the difference between the generated content and the expected content is reduced, and the content generation ability of the model in such complex scenarios is improved. At the same time, in order to deal with possible overfitting problems, regularization techniques such as dropout can be used to ensure the generalization ability of the model, so that it can accurately generate high-quality multimodal content when facing new materials and instructions.

[0060] When dealing with multiple content generation with multiple modalities and each material has user instructions attached, other technologies besides the following generative large models can be considered. First, rule-based expert system technology can process materials and user instructions to generate content based on pre-set rules and knowledge. Domain experts need to build a huge rule base that covers the processing methods of different modal materials and the response rules for various user instructions. For example, for image materials, if the user instruction is to adjust the color style, the rule base should have specific color adjustment algorithms and steps corresponding to it; for text materials, if the instruction is to polish, there are corresponding grammar, vocabulary replacement and other rules. Secondly, template-based generation technology can also process materials and user instructions to generate content. This technology first creates a large number of different types of multimodal content templates. These templates cover various common material combinations and corresponding styles, uses, etc. For example, there are templates for promotion, including attractive images, concise and powerful texts, dynamic audio, etc.; there are also templates for teaching explanations, with clear images, detailed texts, soothing audio, etc. When receiving material with user instructions, the system will analyze the material and instructions to find the one that best matches the existing template. Once a matching template is found, the material provided by the user will be replaced and adjusted according to the structure and requirements of the template. If the image style in the template is different from that provided by the user, the image will be processed accordingly according to the user instructions to make it conform to the template style; the same is true for text, the format and content of the text will be optimized according to the template to generate multimodal content that conforms to the user instructions. Finally, data mining and knowledge graph technology can also process materials and user instructions to generate content. Through data mining technology, the association rules between different modal materials and the corresponding relationship with user instructions can be mined from a large amount of multimodal material data. At the same time, a knowledge graph is constructed to represent these relationships. The knowledge graph uses nodes to represent elements such as materials, user instructions, and generated content, and edges to represent the relationship between them. When receiving new material with user instructions, the system will find relevant material processing methods and content generation paths based on the relationships in the knowledge graph.

[0061] The following embodiment is about content generation of graphic and text materials.

[0062] Step 1: Display the material. In the material operation interface, at least one picture and at least one text description are displayed to the user. For example, the picture may be a landscape photo, and the text description is an introduction to the landscape.

[0063] Step 2: Get user instructions. The system requests the user to input instructions to guide how to combine pictures and text to generate new multimodal content. Users can provide instructions for different materials through an interactive interface. These instructions can be specific modification requirements, such as "replace the blue sky in the picture with the orange-red sky at sunset", or "use this text description as the beginning of the generated content", or "add a text describing the history of the building in the picture".

[0064] Step 3: Generate content based on the material and user instructions. Based on the user instructions and the original material, the system generates new multimodal content. In this example, the system will generate a new landscape image in which the sky is replaced with orange-red, place a text description from the material at the beginning of the generated content, and generate a new text description to introduce the history of the building. The system integrates these contents into a new graphic document.

[0065] Step 4: User feedback. The system displays the generated graphic document to the user and requests user feedback. If the user is satisfied, the system saves and outputs the final multimodal content. This can be displayed directly on the screen, saved as a file for the user to download or print, or allowed to be forwarded and shared by the user. If the user is not satisfied, the system uses the generated graphic document as a new material, allowing the user to provide further instructions, and then repeats the above until the generated content meets the user's needs.

[0066] Through this embodiment, users can adjust and control the graphic materials in a personalized manner to generate multimodal content that meets their needs. This method improves the granularity of the user's control over the details of the generated content, improves the flexibility of content generation and user satisfaction.

[0067] The following example is content generation for generating an audio podcast.

[0068] Step 1: Display the material. In the audio material operation interface, display at least one background music and at least one audio copy to the user. For example, the background music may be a soothing piano piece, while the audio copy is an emotional story content that tells the protagonist's growth process.

[0069] Step 2: Get user instructions. The system prompts the user to enter instructions to guide how to combine background music and audio copy to generate new multimodal content. The user can make user instructions for background music, such as "enhance the piano sound in the background music and add some violin concerto in the middle part." The user can also make user instructions for audio copy, such as "move the first paragraph of the audio copy to the end" or "add some description of environmental sound effects at a certain plot in the audio copy."

[0070] Step 3: Generate content based on the material and user instructions. Based on user instructions and the original material, the system generates new multimodal content. In this example, the system adjusts the background music, enhances the piano sound and incorporates a violin concerto in the middle, adjusts the position of the audio copy, moves the first paragraph to the end, and generates new environmental sound effect descriptions such as wind and rain at the corresponding plot according to user instructions. The adjusted background music and the modified audio copy are then integrated into a new audio file to form new audio podcast content.

[0071] Step 4: User feedback. The system plays the generated audio file to the user and asks for user feedback. If the user is satisfied, the system saves and outputs the final multimodal content. This can be to provide a download option directly on the playback interface, or to allow the user to share the audio link to other social platforms. If the user is not satisfied, the system uses this audio file as a new material, allowing the user to provide further instructions, and then repeats the above steps until the generated content meets the user's expectations.

[0072] Through this embodiment, users can adjust and control audio materials in a personalized way to generate audio podcast multimodal content that meets their needs. This method enhances the user's control over the details of generated content, improves the flexibility of content generation, and thus improves user satisfaction.

[0073] The following embodiment generates a video animation of a single modality based on video content of a single modality.

[0074] Step 1: Display the material. In the video material operation interface, at least one original video is displayed to the user. The original video can be an animation short with a simple plot, such as an animation of a little boy adventuring in the forest, with a relatively simple color and simple movements.

[0075] Step 2: Get user instructions. The system prompts the user to enter instructions, which will guide how to modify and improve the original video to generate new video animation content. The user provides instructions to the video material through the interactive interface, such as "make the color of the little boy's clothes brighter, change the top to red, and the pants to blue", or "add details to the boy's actions during the adventure, such as adding an action of slipping and standing up quickly when crossing the stream", or "adjust the color of the background of the picture to make the forest look more mysterious, with a dark purple color."

[0076] Step 3: Generate content based on the material and user instructions. Based on the user instructions and the original video material, the system generates new video animation content. In this process, the system will modify the color of the boy's clothes according to the instructions to make it more colorful and eye-catching. At the same time, it will add more action details to the boy to make his adventure more vivid and interesting. In addition, it will adjust the color of the forest in the background of the picture to create a mysterious atmosphere and make the entire video animation more attractive.

[0077] Step 4: User feedback. The system displays the generated new video animation to the user and requests user feedback. If the user is satisfied, the system saves and outputs the final video animation content. This can be done by providing playback and sharing functions directly on the playback platform, or by allowing the user to download the video file to a local device. If the user is not satisfied, the system uses the newly generated video animation as a new material, allowing the user to provide further instructions, and then repeats the previous steps until the generated content meets the user's needs.

[0078] Through this embodiment, the user can make personalized fine adjustments and controls to the original video material to generate video animation content that meets his or her expectations. This method greatly improves the granularity of the user's control over the details of the generated content, enhances the flexibility of content generation, and effectively improves user satisfaction.

[0079] Figure 3 A schematic diagram of a multimodal content generation method according to an embodiment of the present disclosure is shown.

[0080] In one implementation, obtaining the user instruction of the user for the material includes:

[0081] Retrieving a specific subset of the material from the material;

[0082] A user instruction of the user for the specific subset is received.

[0083] In one implementation, obtaining a specific subset of the material from the material includes:

[0084] Obtaining a selection result of the user on a specific subset of the material;

[0085] Based on the selection result, a specific subset of the material is obtained from the material.

[0086] The following embodiments are based on the user attaching user instructions to a specific subset of the material.

[0087] Step 1: The system displays a series of materials to the user. These materials can include multiple modalities such as images, text, audio, and video. The user can browse these materials and select one or more specific subsets from them. For example, if there are 5 materials in the material library, namely multiple pictures and several paragraphs of text, the user can select materials 1, 2, and 4 as the subsets they want to further operate.

[0088] During the selection process, the system can provide various tools and interface elements to help users make selections. For example, users can select images or texts by clicking on the checkboxes next to them, or use the drag-and-drop function to move materials to a specific work area to indicate that these materials are a subset that they want to process further.

[0089] Step 2: Once the user selects a specific subset, the system will receive the user's instructions for these subsets. These instructions can be any natural language requirements, such as "refer to the composition of the above 3 images". The user can provide these instructions through an interactive interface, for example, by filling in an input box, using a right-click menu, or directly marking and annotating on the material.

[0090] Step 3: The system will generate content based on the user's selection results and user instructions. For example, if the user selects 3 pictures and adds the user instruction "refer to the composition of the above 3 pictures" to the 3 pictures, the system will generate content based on the user instruction and the 3 selected pictures.

[0091] In this way, users can not only select specific subsets from multiple materials to operate, but also provide detailed instructions for these subsets, thereby achieving more personalized and sophisticated content generation.

[0092] In one implementation, obtaining the selection result of the user on the specific subset of the material includes:

[0093] Obtaining recognition results for a specific subset of the material;

[0094] Based on the recognition result, recommending the specific subset to the user;

[0095] The selection result of the user on the specific subset is obtained.

[0096] The following embodiment describes an implementation method in which the system automatically recommends some materials from multiple materials to a user and generates multimodal content according to the user's selection.

[0097] The system automatically analyzes the user's historical instructions and preferences, as well as the type of current task, to identify specific subsets of materials that may meet the user's needs. This process can be achieved through machine learning algorithms, in which the system learns the user's past behavior patterns, including the types of materials they choose, editing preferences, and commonly used instructions. For example, if the user often chooses landscape pictures and inspirational quotes when editing graphic documents, the system uses this information as a basis for identifying and recommending subsets. The system also takes into account the type of task to determine which subsets of materials are most suitable for the current task. For example, if the task is to make a business presentation, the system may recommend a subset of charts, graphics, and professional copywriting.

[0098] In step 1, the system identifies and recommends specific subsets of materials, and the user will receive a list or preview of these recommendations. The user can browse these recommendations and select the subset they think is most suitable for further operation. The user's selection can be made by simple clicks, drag and drop or other interactive methods, and the system will record each user's selection to further optimize future recommendations.

[0099] Step 2, after the user selects specific subsets, the system will receive user instructions for these subsets.

[0100] Step 3: Generate content based on the user's selection result of the material subset and the user's instruction.

[0101] By combining automatic recommendation with user selection, the system can not only improve the efficiency of material selection, but also ensure that the generated content is more in line with the user's personalized needs and preferences. This method further improves the flexibility of multimodal content generation and user satisfaction.

[0102] Figure 4 A schematic diagram of a multimodal content generation method according to an embodiment of the present disclosure is shown.

[0103] In one implementation, obtaining the user instruction of the user for the material includes:

[0104] Retrieving a specific portion of the material from the material;

[0105] A user instruction of the specific part is received from the user.

[0106] In one implementation, obtaining a specific portion of the material from the material includes:

[0107] Obtaining a selection instruction of the user for the specific part;

[0108] Based on the selection instruction, a specific portion of the material is obtained from the material.

[0109] The following embodiment describes an implementation method for generating multimodal content based on a specific portion of a material.

[0110] Step 1: The system first obtains specific parts of the materials from the material library. These materials can include multiple modalities such as images, texts, audios, and videos. Instead of selecting the entire material, the user selects specific parts of the material and provides user instructions for these specific parts. For example, select a specific part of the material from 5 materials, select the entire presentation document of material 2, several paragraphs of text 3A of the document of material 3, or a part of the data 5A of the table of material 5 as the specific part for further operation. During the selection process, the system provides various tools and interface elements to help users make selections, such as clicking a checkbox or using the drag and drop function.

[0111] Step 2: The user attaches specific user instructions to specific parts of the material. The user can provide these instructions through an interactive interface, such as filling in an input box, clicking with the mouse, using the right-click menu to select system predefined instructions, or marking directly on the material. Multiple personalized instructions can be attached to a material. For example, a user instruction "Use this part of the content as the beginning" is attached to the first two paragraphs of the document, and another user instruction "Use the style of this part of the content to explain my conclusion" is attached to the middle three paragraphs of the same document.

[0112] Step 3: Based on the user's selection results and user instructions, the system obtains the corresponding specific parts from the material and generates content in combination with the corresponding user instructions.

[0113] In this way, users can not only select specific parts from multiple materials to operate, but also provide detailed instructions for these specific parts, thereby achieving more personalized and sophisticated content generation.

[0114] In one implementation, obtaining the user's selection instruction for the specific part includes:

[0115] Obtaining a recognition result of a specific portion of the material;

[0116] Based on the recognition result, recommending the specific part to the user;

[0117] A selection instruction of the user on the specific part is obtained.

[0118] The following embodiment describes an implementation method in which the system automatically recommends a specific part of the material to the user and generates multimodal content according to the user's selection.

[0119] The system automatically analyzes the user's historical instructions and preferences, as well as the type of current task, to identify specific parts of the material that may meet the user's needs. This process can be achieved through machine learning algorithms, in which the system learns the user's past behavior patterns, including the types of materials they choose, editing preferences, and commonly used instructions. For example, if the user often chooses the sky part in a landscape picture as the background when setting the background, or the user often indicates to refer to the color matching of PPT, the system uses this information as the basis for identifying and recommending parts. The system also takes into account the task type to determine which parts of the material are most suitable for the current task.

[0120] In step 1, the system identifies and recommends specific parts of the material, and the user will receive a list or preview of these recommendations. The user can browse these recommendations and select the parts they think are most appropriate for further operation. For example, for a landscape picture, the system automatically recommends "select the sky of the picture". For another example, for a PPT, the system automatically recommends "select the color scheme of the presentation document". The user's selection can be made by simple clicks, drag and drop, or other interactive methods. The system will record every choice of the user in order to further optimize future recommendations.

[0121] Step 2: After the user selects a specific part, the system will receive the user's instructions for these parts. For example, for the sky in a landscape picture, the user instruction is "use the sky in this picture as the background of the new presentation document generated." For another example, for the color scheme of a PPT, the user instruction is "use the color scheme of this presentation document as the color scheme of the new presentation document generated."

[0122] Step 3: Generate content based on the user's selection result of the material part and the user's instructions.

[0123] By combining automatic recommendation with user selection, the system can not only improve the granularity and refinement of material selection, but also ensure that the generated content is more in line with the user's personalized needs and preferences. This method further improves the flexibility of multimodal content generation and user satisfaction.

[0124] In one embodiment, the user instructions include user-defined instructions. In the interactive scenario of content generation, the user-defined instructions used by the user provide the greatest degree of freedom for the entire content creation process. When the user is in the content generation interface, he or she will use natural language, voice or other interactive methods to input various detailed and clear requirements and expectations based on his or her unique needs and rich imagination.

[0125] For example, when a user is faced with a text material full of fantasy, he or she may enter the following command: "Generate a related video based on this text. The video length must be controlled within 5 minutes. The video style should be science fiction, and the picture colors should be mainly cold tones, such as dark blue and blue-gray, to create a mysterious and profound atmosphere. The characters in the video should have a futuristic feel, wearing clothes with luminous lines and technological accessories. In terms of scene design, a grand cosmic starry sky background should be presented, with huge interstellar battleships shuttling through it. The surface of the battleships should have complex mechanical structures and flashing lights. At the same time, add sound effects that are in line with the science fiction style, such as engine roars, energy weapon firing sounds, etc., to make the video more immersive." User-defined commands do not limit the length or the terms used by users, thus giving users maximum freedom.

[0126] In another embodiment, the user instruction includes a system predefined instruction.

[0127] System predefined instructions play an important role in content generation systems. Especially when users lack relevant operating experience, system predefined instructions play a key role in providing high-quality demonstrations. System predefined instructions follow a scientific model that has been practiced and verified for a long time in the field of content generation, allowing users to intuitively understand how content generation tasks should start from original materials and gradually transform into target content based on what steps and rules. In terms of the specific content of the instructions, it covers a series of complete and detailed information from material type identification, topic analysis to target content key feature determination, presenting users with a clear and actionable example, so that users can quickly master basic operating ideas and methods when facing unfamiliar content generation scenarios. In addition, the functions covered by system predefined instructions are often the more commonly used and core parts of content generation systems. These functions have been screened through extensive market research and practical applications, and are a set of functions that meet the needs of most users in daily content generation.

[0128] In this way, the system predefined instructions can reasonably plan the general direction and main features of the generated content according to the inherent characteristics of the material and / or the user's usage habits, laying the foundation for the subsequent content generation process.

[0129] In one embodiment, the multimodal content is generated based on the material and the user instruction, and the multimedia content includes at least one modality, including:

[0130] A large model is used to generate multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one modality.

[0131] In one embodiment, the use of the macro model generates multimodal content based on the material and the user instruction, and the multimedia content includes at least one modality, including:

[0132] Based on a user instruction, obtaining unimodal content from the material;

[0133] Based on a unimodal macromodel corresponding to the unimodal content, processing the unimodal content to obtain a unimodal processing result;

[0134] Based on the unimodal processing result, multimodal content is obtained.

[0135] The following example illustrates the generation of multimodal content based on a unimodal macro model.

[0136] Step 1: Display the material. For example, a detailed text material on the knowledge point of "electromagnetic induction" in the physics course, which includes concept explanations, formula derivations, experimental examples, etc., as well as some simple hand-drawn schematic diagrams related to this. These texts and images together constitute the material.

[0137] Step 2: Obtain user instructions. User instructions for text materials: "Generate a teaching outline for the high school physics electromagnetic induction course based on this material." User instructions for hand-drawn drawings: "Generate illustrations for the teaching outline based on this image material."

[0138] Step 3: Generate content based on materials and user instructions. Based on user instructions and original materials, the system generates new multimodal content. Here, the generated syllabus is in text mode, and the generated illustrations are in image mode. The two are two different types of unimodal content, and the unimodal content can be processed based on the unimodal large model.

[0139] First, for the text unimodal content, the system calls a large model dedicated to natural language processing (NLP). This large model has been pre-trained on a large amount of physics education text data. The large model processes the extracted electromagnetic induction text content and obtains the text unimodal processing result, that is, an optimized, easier to understand and logically clear electromagnetic induction teaching outline text content, in which key knowledge points are highlighted and some necessary explanations and related information are added.

[0140] Secondly, for image unimodal content, the system uses a large model specifically for image understanding and processing. This large model has been trained on image data in the field of education and can identify physical elements and relationships in images. For our hand-drawn schematics, the image large model processes them, including image clarity enhancement, annotation of key elements (such as marking the direction and strength of magnetic field lines next to them), and conversion of hand-drawn images into more standardized vector images for subsequent operations. Through these processes, the image unimodal processing results are obtained, that is, a set of high-quality, clear and fully annotated electromagnetic induction teaching outline illustrations.

[0141] Finally, multimodal content is obtained based on the unimodal processing results. The system integrates the processed text unimodal processing results and image unimodal processing results. For example, according to the associated information in the knowledge graph, the corresponding image is inserted into the appropriate position in the text. Finally, a multimodal learning material containing text and images is generated, which meets the user's requirements for high school physics electromagnetic induction course materials.

[0142] In one embodiment, the use of the macro model generates multimodal content based on the material and the user instruction, and the multimedia content includes at least one modality, including:

[0143] Based on the user instruction, obtaining multimodal content from the material;

[0144] Based on a multimodal macro model corresponding to the multimodal content, processing the multimodal content to obtain a multimodal processing result;

[0145] Based on the multimodal processing result, multimodal content is obtained.

[0146] The following embodiments illustrate generating multimodal content based on a multimodal macro model.

[0147] Step 1: Display the material. For example, a detailed text material on the knowledge point of "electromagnetic induction" in the physics course, which includes concept explanations, formula derivations, experimental examples, etc., as well as some simple hand-drawn schematic diagrams related to this. These texts and images together constitute the material.

[0148] Step 2: Obtain user instructions. User instructions for text materials: "Generate a teaching outline for the high school physics electromagnetic induction course based on this material." User instructions for a collection of text materials and hand-drawn pictures: "Generate illustrations for the teaching outline based on this text material and image material."

[0149] Step 3: Generate content based on materials and user instructions. Based on user instructions and original materials, the system generates new multimodal content. Here, the generated syllabus is in text mode, and the generated illustrations are in image mode. The two are two different unimodal contents, which can be processed based on the multimodal large model.

[0150] The multimodal model has the ability to process multiple modal information at the same time. It has been pre-trained on a large amount of multimodal data containing physics education text and images. For materials composed of text and images, the multimodal model processes them according to user instructions.

[0151] The multimodal large model is a highly complex and collaborative process when processing text and images at the same time. The model combines image information when processing text. According to the description of the experimental steps or physical phenomena in the text, the model adjusts the image. Guided by the steps of the electromagnetic induction experiment in the text, the model converts the hand-drawn image into a more standardized vector image. According to the associated information in the knowledge graph, the corresponding image is inserted into the appropriate position in the text. Finally, a multimodal learning material containing text and images is generated, which meets the user's requirements for high school physics electromagnetic induction course materials.

[0152] In one embodiment, the generated content can be used as new material, that is, the generated content of the previous cycle can be used for content generation in the next cycle. When the generated multimodal content is displayed to the user again as material, this opens a new content creation cycle.

[0153] The generated content of the previous cycle participates in the subsequent content generation as new materials, and the user instructions can be further modified and optimized based on the previous user instructions. When users face these new materials, the formation of their instructions has a new basis and direction. Users can further refine the instructions for new materials based on the user instructions in the previous creation process. For example, in the animation clips of the multimodal content generated previously, the user may feel that the actions of a certain character can be more exaggerated, or the tone of the narration needs to be more passionate. The continuation of this instruction allows content creation to be continuously optimized on the original basis. At the same time, users can also generate new instructions based on the combination of new materials and other original materials. For example, an element in the new material is associated with a specific element in the original material, requiring the generation of a new content that combines the styles of both.

[0154] In one embodiment, the material of the current cycle is generated based on the first material of the previous cycle and the first user instruction;

[0155] The generating multimodal content based on the material and the user instruction includes:

[0156] Multimodal content is generated based on the material, the user instruction, the first material, and the first user instruction.

[0157] In one implementation, the generating multimodal content based on the material, the user instruction, the first material, and the first user instruction includes:

[0158] analyzing the material relevance between the material and the first material;

[0159] Combining the material with the first material based on the material association to obtain a material combination;

[0160] Multimodal content is generated based on the user instruction, the first user instruction, and the combination of the materials.

[0161] When using the big model to generate subsequent multimedia content based on these new materials and user instructions, the big model needs to have strong adaptability. For unimodal content in the new material, the big model must be able to recognize traces of previous processing and make corresponding adjustments based on the new instructions. For example, for text content in the new material, the big model may need to adjust the rhythm or rhetoric of the language based on user instructions while maintaining the original style. For multimodal content, the big model must be able to analyze the new relationship between the various modes. For example, the rhythm coordination between the video and audio in the new material may need to be readjusted under new instructions to adapt to the new creative goals.

[0162] Since the new material itself may be generated based on the previous multi-layer material and user instructions, it is necessary to consider the relevance of more layers of materials when subsequently generating multimedia content. Not only should the relevance of the new material to the current original material be analyzed, but also the relationship between the different layers of materials within the new material should be considered. For example, an element in the new material may be a fusion of two different materials before. In subsequent creation, it is necessary to ensure that this element does not destroy its original semantic and emotional information when it is fused with other elements. This relevance and fusion of multi-layer materials brings more complexity and richness to the generation of multimedia content, making the final generated content more in-depth and meaningful.

[0163] This continuous cycle improves the user experience. Users can continuously improve their ideas in a continuous creative environment, and see their creativity gradually develop from a preliminary conception into a complex and wonderful multimedia work. Every addition of new materials and the generation of new content is a sublimation of the user's creativity, which makes the user feel a continuous sense of achievement and satisfaction, and also inspires more creative inspiration for the user, forming a virtuous circle.

[0164] The following example is about the association of materials in multiple loops.

[0165] Step 1: Display the material. In the material operation interface, at least one picture and at least one text description are displayed to the user. For example, the picture may be a "landscape photo" and the text description may be "an introduction to the scenery" (the first material).

[0166] Step 2: Get user instructions. The system requests the user to input instructions to guide how to combine pictures and text to generate new multimodal content. Users can provide instructions for different materials through an interactive interface. These instructions can be specific modification requirements, such as "replace the blue sky in the picture with the orange-red sky at sunset" or "use this text description as the beginning of the generated content" (first user instruction).

[0167] Step 3: Generate content based on the material and user instructions. Based on the user instructions and the original material, the system generates new multimodal content. In this example, the system will generate a new landscape image in which the sky is replaced with orange-red and a text description from the material is placed at the beginning of the generated content. The system integrates these contents into a new "landscape graphic document".

[0168] Step 4: User feedback. The system displays the generated "landscape graphic document" (current material) to the user and asks for user feedback. If the user is not satisfied, the user continues to operate.

[0169] Step 5, the user adds a user instruction to the graphic document of the building: "Add a section of the building's history" (current user instruction).

[0170] At this time, when the system uses the current material and the current user's instructions to generate content, it is not necessary to unavoidably refer to the historical material. The first material ("introduction to the scenery") referenced by the current material ("scenery graphic document" generated in the previous cycle) is the historical material that can be referenced.

[0171] The system starts to analyze the relevance between the current material (the "landscape graphic document" generated in the previous cycle) and the first material ("introduction to the scenery"). The scenery in the current material and the first material are closely related to each other. The building elements to be added this time are also derived from this scenery.

[0172] In the field of knowledge representation, triples are a commonly used structure, usually composed of (subject, predicate, object). It can effectively represent the relationship between entities. In the representation of material relevance, we can regard the current material and the first material as entities, and the relationship between them as the predicate. Therefore, the relevance between the current material and the first material is ("landscape graphic document", "originating from", "introduction to the scenery"), which constitutes a material combination.

[0173] It is worth noting that when triples are used to express the relevance of materials, the predicate can be any of the following: originated from, adapted from, referenced from, similar to, thematically related to, additional explanation, etc.

[0174] Step 7, based on the user instruction of the current loop ("add a history of the building"), the first user instruction of the previous loop ("use this text description as the beginning of the generated content") and the material combination ("landscape graphic document", "originating from", "introduction to the scenery"), generate the multimodal content "new landscape graphic document".

[0175] Step 8: User feedback. The system displays the generated "new landscape graphic document" to the user and asks for user feedback. If the user is satisfied, content generation is complete.

[0176] In one implementation, the generating multimodal content based on the material, the user instruction, the first material, and the first user instruction includes:

[0177] analyzing a user instruction correlation between the user instruction and the first user instruction;

[0178] Combining the user instruction and the first user instruction based on the user instruction association to obtain a user instruction combination;

[0179] Multimodal content is generated based on the combination of the user instructions and the material, and the first material.

[0180] The following embodiments are about the correlation of user instructions in multiple cycles.

[0181] Step 1: Display the material. In the material operation interface, at least one picture and at least one text description are displayed to the user. For example, the picture may be a "landscape photo" (the first material), and the text description may be "an introduction to the scenery".

[0182] Step 2: Get user instructions. The system requests the user to input instructions to guide how to combine pictures and text to generate new multimodal content. Users can provide instructions for different materials through an interactive interface. These instructions can be specific modification requirements, such as "replace the blue sky in the picture with the orange-red sky at sunset" (the first user instruction), or "use this text description as the beginning of the generated content."

[0183] Step 3: Generate content based on the material and user instructions. Based on the user instructions and the original material, the system generates new multimodal content. In this example, the system will generate a new landscape image in which the sky is replaced with orange-red and a text description from the material is placed at the beginning of the generated content. The system integrates these contents into a new "landscape graphic document".

[0184] Step 4: User feedback. The system displays the generated "landscape graphic document" (current material) to the user and asks for user feedback. If the user is not satisfied, the user continues to operate.

[0185] Step 5, the user adds a user instruction to the graphic document of the building: "Make the range of replacement smaller" (current user instruction).

[0186] At this time, when the system uses the current material and the current user instructions to generate content, it is inevitable to reference historical materials and historical instructions. The first material ("introduction to the scenery") referenced by the current material ("scenery graphic document" generated in the previous cycle) is the historical material that can be referenced. The first user instruction ("replace the blue sky in the picture with the orange-red sky at sunset") referenced by the current user instruction ("reduce the range of replacement") is the historical instruction that can be referenced.

[0187] The system begins to analyze the correlation between the user instruction ("make the replacement area smaller") and the first user instruction ("replace the blue sky in the picture with the orange-red sky at sunset"). The current user instruction is not valid when understood alone, and must be understood together with the first user instruction.

[0188] In the field of knowledge representation, triples are a commonly used structure, usually composed of (subject, predicate, object). It can effectively represent the relationship between entities. In the representation of the relevance of user instructions, we can regard the current user instruction and the first user instruction as entities, and the relationship between them as the predicate. Therefore, the relevance between the current user instruction and the first user instruction is ("make the range of replacement smaller", "modify", "replace the blue sky in the picture with the orange-red sky at sunset"), which constitutes a user instruction combination.

[0189] At this time, the correlation between the current material and the first material is ("landscape graphic document", "originating from", "landscape photo"), which also constitutes a material combination.

[0190] It is worth noting that when triples are used to represent the relevance of user instructions, the predicate can be any of the following: refinement, generalization, modification, continuation, supplementary explanation, etc.

[0191] Step 7, based on the user instruction combination ("make the replacement range smaller", "modify", "replace the blue sky in the picture with the orange-red sky at sunset") and the material combination ("landscape graphic document", "originating from", "landscape photo"), generate the multimodal content "new landscape graphic document".

[0192] Step 8: User feedback. The system displays the generated "new landscape graphic document" to the user and asks for user feedback. If the user is satisfied, content generation is complete.

[0193] Figure 5 A multimodal content generation method 200 according to an embodiment of the present disclosure is shown, which may include the following steps:

[0194] Step 201, displaying at least one material to a user, wherein the material includes at least one modality;

[0195] Step 202, obtaining a user instruction of the user for the material;

[0196] Step 203: Generate multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode.

[0197] Step 204, obtaining the modification of the user instruction by the user as a second user instruction;

[0198] Step 205: Generate multimedia content based on the second user instruction and second material related to the second user instruction.

[0199] During the process of generating multimedia contents in multiple cycles, the system allows the previous user instructions to be modified. To distinguish them, the modified user instructions may be named as second user instructions.

[0200] Modification of a user instruction means abandonment of the user instruction, that is, replacing the user instruction with a second user instruction.

[0201] The second user instruction has a close and complex relationship with the user instructions issued by the previous user and the corresponding materials. The previous user instructions have guided the selection, processing and content generation of the initial materials, and the second user instruction is a further expansion on this basis. First, it can be a continuation of the unfinished part of the previous user instruction. For example, in the previously generated content, the user instructions mainly focus on the picture style and text content, while the second user instruction may make more detailed provisions for the audio part to improve the entire multimedia experience. Second, the second user instruction is also closely related to the new user material. The previously used materials are no longer sufficient to complete the user's intentions, and the second user instruction can determine the new creative direction based on the information contained in the new materials.

[0202] Step 205 also reflects another role of user feedback in multiple cycles. The system displays the generated multimodal content to the user and requests user feedback. If the user is satisfied, the system will allow the user to provide further instructions, and also allow the user to directly modify the previous user instructions.

[0203] Figure 6 A multimodal content generation method 300 according to an embodiment of the present disclosure is shown, which may include the following steps:

[0204] Step 301, displaying at least one material to a user, wherein the material includes at least one modality;

[0205] Step 302, obtaining a user instruction of the user for the material;

[0206] Step 303: Generate multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode.

[0207] Step 304, obtaining the modification of the material by the user as the second material;

[0208] Step 305: Generate multimedia content based on the second material and a second user instruction related to the material.

[0209] In the process of generating multiple cycles of multimedia content, the system allows the previous material to be modified. To distinguish it, the modified material can be named as the second material.

[0210] Modification of a material means abandonment of the material, that is, using the second material to replace the previous one.

[0211] The second material has a close and complex relationship with the previous material and the corresponding user instructions. The previous user instructions have guided the selection, processing and content generation of the previous material. After the previous material is modified into the second material, the relevant user instructions are needed to further expand on this basis. The previously used material is no longer sufficient to complete the user's intention, and the second material can cooperate with the relevant user instructions to determine the new creative direction.

[0212] Step 305 also reflects another role of user feedback in multiple cycles. The system displays the generated multimodal content to the user and requests user feedback. If the user is satisfied, the system will allow the user to further add content, or directly modify the previous content.

[0213] In one implementation, the multimedia content is shared externally based on offline files or online content.

[0214] For offline file methods, this is a traditional and reliable way of sharing. Creators can save the generated multimedia content in a specific file format. For example, video content can be saved in common MP4, AVI and other formats, audio content can be saved in MP3, WAV and other formats, and image content can be saved in JPEG, PNG and other formats. The online content method is more in line with the current development trend of digitalization and networking. By uploading multimedia content to a network server, the advantages of the Internet can be used to achieve extensive sharing. You can also generate a shared link through cloud storage services and send the link to those who need to receive the content. This online method is not limited by geographical distance and can quickly spread multimedia content to audiences around the world, greatly improving the efficiency and influence of content dissemination.

[0215] like Figure 7 As shown, the multimodal content generating device 700 provided in this embodiment includes: a display unit 701, an instruction unit 702, and a generating unit 703.

[0216] The display unit 701 displays at least one material to the user, wherein the material includes at least one modality;

[0217] An instruction unit 702 is used to obtain a user instruction of the user for the material;

[0218] The generating unit 703 generates multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode.

[0219] In this embodiment, in the multimodal content generating device 700, the specific processing of each mode and the technical effects brought about by it can refer to the relevant descriptions of each step in the aforementioned embodiment, and will not be repeated here.

[0220] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0221] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0222] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0223] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0224] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the multimodal content generation method. For example, in some embodiments, the multimodal content generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the multimodal content generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the multimodal content generation method in any other appropriate manner (e.g., by means of firmware).

[0225] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0226] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable information processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0227] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0228] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0229] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0230] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0231] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0232] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal content generation method, characterized in that: The method comprises: Displaying at least one material to a user, the material comprising at least one modality; Obtaining a user instruction of the user for the material; Multimodal content is generated based on the material and the user instruction, and the multimedia content includes at least one modality.

2. The method according to claim 1, characterized in that The modality includes one of image, text, audio, and video.

3. The method according to claim 1, characterized in that The obtaining of the user instruction of the user for the material includes: Retrieving a specific subset of the material from the material; A user instruction of the user for the specific subset is received.

4. The method according to claim 3, characterized in that The obtaining a specific subset of the material from the material includes: Obtaining a selection result of the user on a specific subset of the material; Based on the selection result, a specific subset of the material is obtained from the material.

5. The method according to claim 4, characterized in that The obtaining of the selection result of the user on the specific subset of the material includes: Obtaining recognition results for a specific subset of the material; Based on the recognition result, recommending the specific subset to the user; The selection result of the user on the specific subset is obtained.

6. The method according to claim 1, characterized in that The obtaining of the user instruction of the user for the material includes: Retrieving a specific portion of the material from the material; A user instruction of the user for the specific part is received.

7. The method according to claim 6, characterized in that The obtaining a specific portion of the material from the material includes: Obtaining a selection instruction of the user for the specific part; Based on the selection instruction, a specific portion of the material is obtained from the material.

8. The method according to claim 7, characterized in that The obtaining the user's selection instruction for the specific part includes: Obtaining a recognition result of a specific portion of the material; Based on the recognition result, recommending the specific part to the user; A selection instruction of the user on the specific part is obtained.

9. The method according to claim 1, characterized in that: The user instructions include user-defined instructions.

10. The method according to claim 1, characterized in that The user instructions include system predefined instructions.

11. The method according to claim 1, characterized in that: The generating multimodal content based on the material and the user instruction includes: A large model is used to generate multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one modality.

12. The method according to claim 11, characterized in that The using of the macro model generates multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode, including: Based on a user instruction, obtaining unimodal content from the material; Based on a unimodal macromodel corresponding to the unimodal content, processing the unimodal content to obtain a unimodal processing result; Based on the unimodal processing result, multimodal content is obtained.

13. The method according to claim 11, characterized in that The using of the macro model generates multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode, including: Based on the user instruction, obtaining multimodal content from the material; Based on a multimodal macro model corresponding to the multimodal content, processing the multimodal content to obtain a multimodal processing result; Based on the multimodal processing result, multimodal content is obtained.

14. The method according to claim 1, characterized in that The material is generated based on the first material and the first user instruction; The generating multimodal content based on the material and the user instruction includes: Multimodal content is generated based on the material, the user instruction, the first material, and the first user instruction.

15. The method according to claim 14, characterized in that The generating multimodal content based on the material, the instruction, the first material, and the first user instruction includes: analyzing the material relevance between the material and the first material; Combining the material with the first material based on the material association to obtain a material combination; Multimodal content is generated based on the user instruction, the first user instruction, and the combination of the materials.

16. The method according to claim 14, characterized in that The generating multimodal content based on the material, the instruction, the first material, and the first user instruction includes: analyzing a user instruction correlation between the user instruction and the first user instruction; Combining the user instruction and the first user instruction based on the user instruction association to obtain a user instruction combination; Multimodal content is generated based on the combination of the user instructions and the material, and the first material.

17. The method according to claim 1, characterized in that Also includes: Acquire the modification of the user instruction by the user as a second user instruction; Multimedia content is generated based on the second user instruction and a second material related to the second user instruction.

18. The method according to claim 1, characterized in that: Acquire the modification of the material by the user as the second material; Multimedia content is generated based on the second material and a second user instruction related to the material.

19. The method according to claim 1, characterized in that: The multimedia content is shared externally based on offline files or online content.

20. A multimodal content generation device, the device comprising: A display unit, for displaying at least one material to a user, wherein the material includes at least one modality; An instruction unit, for obtaining a user instruction of the user for the material; A generating unit generates multimodal content based on the material and the user instruction, wherein the multimedia content includes at least one mode.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 19.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 19.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 19.

Images