A multi-modal generation method, apparatus, device, medium, and product

Abstract

This disclosure relates to a multimodal generation method, apparatus, device, medium, and product. The method includes: receiving a user query; wherein the user query contains at least one input data, each input data belonging to a modality; assigning a corresponding encoder to each input data based on the modality to which the input data belongs, and determining the feature vector corresponding to each input data through the encoder; inputting the feature vector into a vector database for retrieval to obtain retrieval results; wherein the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; and determining a multimodal output result based on the user query and the retrieval results. This disclosure can accurately process multimodal user queries, ensuring independent retrieval and semantic association of information from different modalities, thereby generating multimodal content that is highly accurate, well-coordinated, and better meets the personalized needs of users.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a multimodal generation method, apparatus, device, medium, and product. Background Technology

[0002] Retrieval Augmented Generation (RAG) technology has become an important method to improve the output quality of large models by retrieving relevant knowledge from vector databases and combining it with generative models. In multimodal application scenarios, existing technologies usually encode different modal data such as text and images into a unified vector space for retrieval to support subsequent multimodal content generation.

[0003] However, existing multimodal generation methods struggle to effectively balance the semantic differences and relevance between different modalities, and they also cannot flexibly adjust the utilization of information from different modalities according to user needs. These limitations of existing technologies affect the accuracy, consistency, and user satisfaction of multimodal content generation. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, medium, and product for generating multimodal data.

[0005] According to a first aspect of this disclosure, a multimodal generation method is provided, the method comprising: Receive user queries; wherein the user query contains at least one input data, and each input data belongs to a modality; Based on the modality to which the input data belongs, a corresponding encoder is assigned to each input data, and the feature vector corresponding to each input data is determined through the encoder; The feature vector is input into a vector database for retrieval to obtain retrieval results; wherein, the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; The multimodal output result is determined based on the user query and the search results.

[0006] Furthermore, the modality includes at least one of the following: text modality, image modality, and video modality; Assigning a corresponding encoder to each input data based on the modality to which the input data belongs includes: Assign a text encoder to the input data belonging to the text modality; Assign an image encoder to the input data belonging to the image modality; Assign a video encoder to the input data belonging to the video modality.

[0007] Further, determining the multimodal output result based on the user query and the search results includes: The search results are subjected to semantic consistency processing to obtain multimodal search information; wherein, the multimodal search information includes at least one of the following: text search results, image search results, and video search results; A first search result is determined based on the text search results, the image search results, and the video search results; wherein, the first search result is a search result obtained by integrating the text search results, the image search results, and the video search results based on modal weights; The user query and the first search result are input into the multimodal large model to obtain the multimodal output result.

[0008] Further, determining the first search result based on the text search result, the image search result, and the video search result includes: The modal weights corresponding to the text retrieval results, the image retrieval results, and the video retrieval results are calculated using a first formula; wherein, the first formula is: , Representing search results for different modalities, The modal weights, As the initial weights, Here, is the decay coefficient, and t is the interval between the weight update time and the corresponding search result generation time. Output a content acceptance rating for the user's search results for the different modalities; The modal weights are normalized to obtain the normalized weights corresponding to the text retrieval results, the image retrieval results, and the video retrieval results; Based on the normalized weights, a corresponding number of search segments are selected from the text search results, image search results, and video search results, respectively. The search fragments are used to construct the first search result.

[0009] Furthermore, before inputting the feature vector into the vector database for retrieval, the method further includes: Obtain an external data source, wherein the external data source includes at least one of the following: text data, image data, and video data; Construct a text vector database based on the text data; An image vector database is constructed based on the image data; A video vector database is constructed based on the video data.

[0010] Furthermore, the step of constructing a video vector database based on the video data includes: The video data is segmented to obtain multiple video segments; Extract the frame feature vector of each video frame in each video segment, and average the frame feature vectors to obtain the segment feature vector of each video segment. The feature vectors of the segments are concatenated to obtain the visual feature vectors of the video data; Extract the audio content from the video data, and determine the text feature vector based on the audio content; The visual feature vector and the text feature vector are fused to obtain the fused feature vector of the video data, and the video vector database is constructed based on the fused feature vector.

[0011] According to a second aspect of this disclosure, a multimodal generation apparatus is provided, the apparatus comprising: A receiving module is used to receive user queries; wherein the user query contains at least one input data, and each input data belongs to a modality; The first determining module is used to assign a corresponding encoder to each input data based on the mode to which the input data belongs, and to determine the feature vector corresponding to each input data through the encoder; The retrieval module is used to input the feature vector into a vector database for retrieval and obtain retrieval results; wherein, the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; The second determining module is used to determine the multimodal output result based on the user query and the search result.

[0012] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0013] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0014] According to a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, implements the methods described above in this disclosure.

[0015] This disclosure provides a multimodal generation method, apparatus, device, medium, and product. In this embodiment, firstly, a user query is received; wherein the user query includes at least one input data, each input data belonging to a modality; then, based on the modality to which the input data belongs, a corresponding encoder is assigned to each input data, and the encoder determines the feature vector corresponding to each input data; subsequently, the feature vector is input into a vector database for retrieval to obtain retrieval results; wherein the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; finally, a multimodal output result is determined based on the user query and the retrieval results.

[0016] As described above, this embodiment of the present disclosure assigns corresponding encoders to input data of different modalities, enabling input data of different modalities such as text, images, and videos to obtain more accurate feature representations that preserve modality-specific semantics through their dedicated encoders. This effectively alleviates the problem of semantic information loss and confusion caused by unified encoding between different modalities at the source. Furthermore, this embodiment of the present disclosure determines the feature vector corresponding to each input data through the encoder, and then inputs the feature vector into the vector database corresponding to the modality to which the input data to which the feature vector belongs for retrieval, and obtains the retrieval results. This ensures that the retrieval process can be carried out in the independent semantic spaces of text, images, and videos, avoiding semantic interference during cross-modal retrieval. Finally, this embodiment of the present disclosure determines the multimodal output results based on user queries and retrieval results, thereby effectively taking into account the semantic differences between different modalities, and can flexibly adjust the utilization degree of different modal information according to user queries, thereby improving the accuracy, coordination, and user satisfaction of multimodal content generation. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A flowchart of a multimodal generation method provided as an exemplary embodiment of this disclosure; Figure 2 One of the flowcharts for a multimodal generation method provided as another exemplary embodiment of this disclosure; Figure 3 A second flowchart of a multimodal generation method provided for another exemplary embodiment of this disclosure; Figure 4A schematic block diagram of the functional modules of a multimodal generation apparatus provided for an exemplary embodiment of the present disclosure; Figure 5 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure; Figure 6 A structural block diagram of a computer system provided as an exemplary embodiment of this disclosure; Figure 7 A structural block diagram of a computer program product provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0021] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more". The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0025] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure; other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] In one embodiment, such as Figure 1 As shown, a multimodal generation method is provided, including the following steps: Step 101: Receive user queries.

[0027] Here, the executing entity can receive user queries. The executing entity can be a server or a client device. The user query contains at least one input data. Each input data belongs to a modality, which defines the basic information type of the input data, such as text, image, or video. It should be noted that the user query can contain only one input data of a single modality, or it can contain multiple input data of multiple modalities, thus constituting a multimodal query.

[0028] In one possible embodiment, a user initiates a query to the server through an application on a smart terminal. For example, user A wants to know about the habits of giant pandas; they can enter the plain text "What do giant pandas usually eat?" in the application's input box. This constitutes a user query containing only text modality input data. User B sees an unfamiliar plant while traveling, takes a picture of it, and uploads it; this constitutes a user query containing only image modality input data. User C uploads a concept image of a new sneaker and enters the text "Please generate a 30-second promotional video script highlighting its lightweight and cushioning characteristics for this design" in the description box. This constitutes a multimodal user query containing both image and text modalities. The executing entity can receive and process these different types of user queries.

[0029] Step 102: Based on the modality to which the input data belongs, assign a corresponding encoder to each input data, and determine the feature vector corresponding to each input data through the encoder.

[0030] Here, after receiving a user query, the execution entity can assign a corresponding encoder to each input data based on the modality to which the input data belongs, and determine the feature vector corresponding to each input data through the encoder.

[0031] In one possible embodiment, the modality includes at least one of the following: text modality, image modality, and video modality. Based on the modality to which the input data belongs, a corresponding encoder is assigned to each input data, including the following steps: Assign a text encoder to input data that belongs to the text modality; Assign an image encoder to the input data belonging to the image modality; Assign a video encoder to the input data that belongs to the video modality.

[0032] Specifically, after receiving a user query, the executing entity can assign a text encoder to input data belonging to the text modality, an image encoder to input data belonging to the image modality, and an image encoder to input data belonging to the image modality.

[0033] In one possible embodiment, following the previous example, after receiving the multimodal query from user C as in step 101, the executing entity first parses and identifies the modality type of the received user query. Specifically, the executing entity detects that the query contains two independent input data units: the file format or data stream characteristics of one data unit conform to the characteristics of image data, and therefore it is identified as belonging to the image modality; the content of the other data unit is a character sequence, and therefore it is identified as belonging to the text modality.

[0034] After recognition, the executing agent assigns a pre-trained image encoder, specifically designed for extracting deep visual features, to the sneaker concept image belonging to the image modality. This image encoder can be a convolutional neural network model. It processes the image data, converting it into a high-dimensional image feature vector representing its visual semantics. Simultaneously, the executing agent assigns a pre-trained text encoder, specifically designed for understanding text semantics, to the descriptive text belonging to the text modality. This text encoder can be a Transformer-based model. The executing agent processes the text data using this text encoder, converting it into a high-dimensional text feature vector representing its linguistic semantics. Through this step, the original input data from different modalities are independently converted into feature vector representations in their respective modal semantic spaces, laying the foundation for subsequent independent retrieval.

[0035] In another possible embodiment, the modal types included in the user query are not limited to images and text, but can be extended to video or other types. For example, a user query may include a short video clip as input data for the video modality, along with text instructions. In this case, the executing entity will first identify the video and text modalities, then assign a dedicated video encoder to the video data to extract its spatiotemporal features, and simultaneously assign a text encoder to the text data. This process follows the same principle as the above embodiment, that is, independently assigning a corresponding encoder to each identified modality for processing. It should be noted that this embodiment illustrates that the present disclosure has the extended capability to process multimodal queries containing any number and combination, the core of which lies in performing precise, one-to-one encoding processing on each independent input data in the query according to its modality.

[0036] Step 103: Input the feature vector into the vector database for retrieval and obtain the retrieval results.

[0037] Here, after the execution entity determines the feature vectors corresponding to each input data through the encoder, it can input the feature vectors into the vector database for retrieval and obtain the retrieval results. The vector database corresponds to the mode to which the input data corresponding to the feature vectors belong.

[0038] In one possible implementation, continuing the previous example, after obtaining the image feature vector and text feature vector in step 102, the executing entity will initiate the retrieval process. Specifically, the executing entity maintains multiple independent vector databases that strictly correspond to different modalities, such as an image vector database specifically storing massive amounts of image data, and a text vector database specifically storing massive amounts of text data. Subsequently, the executing entity inputs the image feature vector into the image vector database and performs independent retrieval within that database. During the retrieval, the executing entity uses cosine similarity as a similarity metric, specifically calculating the similarity between the feature vector to be retrieved and the vector entries in the database using the following formula:

[0039] Here, A represents the feature vector to be retrieved, and B represents a vector entry in the vector database. Through this similarity measure, the executing entity finds several image segments that are most visually and semantically related to the query image from massive image data, such as the top k1 results with the highest similarity, as the retrieval results of the image modality.

[0040] Simultaneously, the executing entity inputs the text feature vector into a text vector database and independently performs a search within that database using cosine similarity to find the few text fragments most semantically relevant to the query text from the massive amount of text data. For example, the top k2 results with the highest similarity are used as the search results for the text modality. In this way, queries of different modalities are searched in their own independent, modality-matched dedicated databases, ensuring that the search process is not interfered with by data from other modalities. This results in accurate sets of image search results and text search results, which together constitute search results that are highly relevant to the user's multimodal query intent and clearly modal.

[0041] It should be noted that the above embodiments are illustrated using user queries containing both image and text modalities as examples. This is merely a demonstration of a specific scenario for ease of understanding and is not intended to limit the scope of application of this application. Regardless of whether the user query contains one, two, or more different modalities of input data, such as further including other modalities like video or audio, the same principle is strictly followed: the feature vector generated for each modal of input data is independently retrieved in a dedicated vector database that strictly corresponds to its modality. This independent retrieval mechanism ensures that the retrieval process is conducted only within the semantic space of a single modality, eliminating the possibility of mutual interference between different modal data during the retrieval stage from a technical perspective. This provides a reliable and pure foundation for generating high-quality, highly relevant multimodal content.

[0042] In a preferred embodiment, to further enhance cross-modal information, the executing entity first constructs a multimodal representation for the user query before retrieval. Specifically, for text-type input, the executing entity not only uses a text encoder to obtain text feature vectors but also uses a CLIP model to generate a reference image corresponding to the text and extract its image feature vectors. Simultaneously, the executing entity can also use a CogVideoX model to generate corresponding video data segments and extract video feature vectors. For image-type input, the executing entity not only uses an image encoder to obtain image feature vectors but also uses a BLIP model to generate corresponding text descriptions and extract text feature vectors. Simultaneously, the executing entity can also use a StableVideoX model to generate corresponding video data segments and extract video feature vectors. The Diffusion model generates video clips and extracts video feature vectors. For video input, the execution entity extracts the feature vectors of the video itself, as well as the text feature vectors converted from the audio and the image feature vectors of the keyframes. Subsequently, the execution entity inputs the multimodal feature vector set constructed above into the vector databases corresponding to each feature vector modality, namely the text, image, and video vector databases, for independent retrieval. For example, for a text query, the system will simultaneously search for relevant text in the text database, search for images related to the derived images in the image database, and search for video clips related to the derived videos in the video database. Finally, all these retrieval results constitute the output of step 103.

[0043] Step 104: Determine the multimodal output results based on the user query and retrieval results.

[0044] Here, the executing entity inputs the feature vector into the vector database for retrieval, and after obtaining the retrieval results, it can determine the multimodal output results based on the user query and the retrieval results.

[0045] In one possible embodiment, such as Figure 2 As shown, determining the multimodal output results based on user queries and retrieval results includes the following steps: Step 1041: Perform semantic consistency processing on the search results to obtain multimodal search information.

[0046] Here, after inputting the feature vector into the vector database for retrieval and obtaining the retrieval results, the executing entity can perform semantic consistency processing on the retrieval results to generate multimodal retrieval information that is semantically related and content-coordinated. The multimodal retrieval information includes at least one of the following: text retrieval results, image retrieval results, and video retrieval results.

[0047] In one possible embodiment, the executing entity has retrieved text modal information, image modal information, and video modal information related to the user query through step 103. For ease of description, the set of retrieved text information is denoted as S={ , ,..., The set of image information is denoted as I={ , ,..., The set of video information is denoted as V={ , ,..., }

[0048] In this embodiment, the first step in semantic consistency processing is to select the dominant modality based on weights. The weights can be determined based on the overall relevance of each modality's retrieval results to the query, historical data, or preset rules. For example, in this embodiment, the executing entity calculates that the text information set S has the highest weight, and therefore selects the text information as the dominant modality for this processing. Subsequently, the executing entity performs cross-modal semantic alignment and filtering. Using the dominant modality text information set S as a benchmark, the executing entity calculates text-image pairs, such as text fragments. With image fragments The semantic similarity between the two modalities is used to assess whether they are semantically related. The semantic similarity can be calculated using methods such as the vector space model, and there are no restrictions here. The executing entity will preset a similarity threshold. ,For example =0.5, if the calculated text-image pair { , If the semantic similarity of the text fragment to the image fragment is less than the threshold θ, and the index p of the text fragment is greater than or equal to the index q of the image fragment (i.e., p ≥ q), then the image fragment is considered valid. Information that is irrelevant to the dominant text semantics is filtered out from the image information set I. After this round of calculation and filtering, the content retained in the text information set S and the image information set I achieves preliminary semantic alignment.

[0049] Finally, the executing entity further aligns the video information with the aligned modalities. For example, the executing entity can calculate the semantic similarity between each video segment in the video information set V and the aligned and filtered text information set S and image information set I, respectively, and filter or associate them according to similar threshold rules, thereby ensuring the semantic association between video information and text and image information.

[0050] Through the cross-modal semantic computation, filtering, and alignment process based on the dominant modality in the above embodiments, the final set of text, image, and video information obtained by the executing entity is highly related and complementary at the semantic level. These retrieval results of different modalities after semantic consistency processing together constitute multimodal retrieval information. This multimodal retrieval information provides a reliable guarantee for the subsequent generation of multimodal outputs with coordinated content and semantic consistency.

[0051] Step 1042: Determine the first search result based on the text search results, image search results, and video search results.

[0052] Here, after performing semantic consistency processing on the search results and obtaining multimodal search information, the executing entity can determine the first search result based on the text search results, image search results, and video search results. The first search result is the search result obtained by integrating the text search results, image search results, and video search results based on modal weights.

[0053] In one possible embodiment, determining the first search result based on text search results, image search results, and video search results includes the following steps: The modal weights corresponding to text retrieval results, image retrieval results, and video retrieval results are calculated using the first formula. The modal weights are normalized to obtain the normalized weights corresponding to text retrieval results, image retrieval results, and video retrieval results; Based on normalized weights, a corresponding number of search segments are selected from the text search results, image search results, and video search results, respectively. The search fragments are used to form the first search result.

[0054] Specifically, after performing semantic consistency processing on the search results to obtain multimodal search information, the executing entity first calculates the modal weights corresponding to the text search results, image search results, and video search results using a first formula. Then, the executing entity normalizes the modal weights to obtain the normalized weights corresponding to the text search results, image search results, and video search results. After that, based on the normalized weights, the executing entity selects a corresponding number of search segments from the text search results, image search results, and video search results, respectively. Finally, the executing entity constructs the first search result from the search segments.

[0055] In one possible embodiment, firstly, the executing entity calculates initial modal weights for text retrieval results, image retrieval results, and video retrieval results. These initial weights are calculated based on the objective relevance of each modal retrieval result to the user's query. Specifically, during the retrieval process in step 103, for each modality, the initial weights are calculated using... This represents the frequency of occurrence of content in the vector database corresponding to the modality whose similarity to the vector to be retrieved is greater than a preset threshold θ. j represents different modalities. In this embodiment, θ = 0.5. At this time, the initial weight wj of each modality can be calculated by the following formula:

[0056] The initial weight wj reflects the proportion of objective relevance between each modality and the query in this retrieval. Subsequently, this initial weight... The system can be dynamically optimized and adjusted based on user interaction history and real-time feedback to adapt to users' personalized preferences. The weight adjustment follows the first formula, which is shown below:

[0057] in, This refers to the initial weights calculated in the previous step. Here, represents the modal weights, i.e., the updated weights; α is the decay coefficient, taking values ​​between 0 and 1; and t is the interval between the weight update time and the time the corresponding search result is generated. This function outputs a rating of the user's acceptance of search results across different modalities. A shorter time interval t indicates more targeted user feedback, resulting in a larger weight adjustment range; conversely, a longer time interval t leads to a decrease in the weighting of the evaluation terms. This will decrease, weakening the impact of historical feedback on the current query, as user acceptance of a certain modality increases. At lower levels, modal weights This will significantly increase, thereby enhancing the information of this modality in subsequent searches. This step enables the personalization and adaptive updating of weights.

[0058] Next, the executing entity updates the modal weights. Normalization is performed to obtain the final normalized weights for text search results, image search results, and video search results. Then, the executing entity selects a corresponding number of search segments from the text search results, image search results, and video search results according to the normalized weights. Specifically, the search results of each modality have been pre-sorted according to their similarity to the query. For example, the text search results are sorted in descending order of similarity to obtain a list. The normalized weights determine the proportion of the number of segments selected from each modality in the total number of segments k. For example, if the normalized weight of the text modality is 0.5 and the total number of search segments k is set to 10, then the 5 segments with the highest similarity are selected from the text search results. Finally, the search segments selected by the executing entity from each modality according to the weight proportions constitute a set containing k multimodal segments, which is the first search result.

[0059] Step 1043: Input the user query and the first search result into the multimodal large model to obtain the multimodal output result.

[0060] Here, after determining the first search result, the executing entity can input the user query and the first search result into the multimodal large model to obtain the multimodal output result.

[0061] In one possible embodiment, the executing agent combines the user query with the structured first search result to construct an enhanced multimodal input context (Prompt). This context clearly includes the user's original intent and relevant knowledge fragments retrieved from the knowledge base, after filtering and weighting. Subsequently, the executing agent inputs this constructed enhanced context into a pre-trained multimodal large model. Based on its deep understanding and generation capabilities of multimodal information, the large model performs reasoning and content synthesis guided by the context. Finally, the large model outputs a multimodal output result. For example, the output result can be a promotional video script draft that integrates visuals and text, which includes both textual descriptions of the lightweight and cushioning characteristics of the athletic shoe and several keyframe sketches generated based on the input concept map and the retrieved image.

[0062] In this embodiment, firstly, the executing entity performs semantic consistency processing on the search results to obtain multimodal search information; then, the executing entity determines the first search result based on the text search results, image search results, and video search results; wherein, the first search result is the search result obtained after integrating the text search results, image search results, and video search results based on modal weights; finally, the executing entity inputs the user query and the first search result into the multimodal large model to obtain the multimodal output result.

[0063] As described above, in this embodiment, the executing entity effectively ensures the semantic association and complementarity between different modal retrieval information by performing semantic consistency processing on the retrieval results. Furthermore, in this embodiment, the executing entity integrates text, image, and video retrieval results based on modal weights to determine the first retrieval result, thereby dynamically adjusting the utilization ratio of different modal information according to query relevance and user preferences. Finally, the executing entity inputs the user query and this refined integrated first retrieval result into the multimodal large model, thereby driving the model to generate semantically consistent, content-accurate, and more user-expected multimodal output results.

[0064] This disclosure provides a multimodal generation method, apparatus, device, medium, and product. In this embodiment, firstly, a user query is received; wherein the user query includes at least one input data, each input data belonging to a modality; then, based on the modality to which the input data belongs, a corresponding encoder is assigned to each input data, and the encoder determines the feature vector corresponding to each input data; subsequently, the feature vector is input into a vector database for retrieval to obtain retrieval results; wherein the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; finally, a multimodal output result is determined based on the user query and the retrieval results.

[0065] As described above, this embodiment of the present disclosure assigns corresponding encoders to input data of different modalities, enabling input data of different modalities such as text, images, and videos to obtain more accurate feature representations that preserve modality-specific semantics through their dedicated encoders. This effectively alleviates the problem of semantic information loss and confusion caused by unified encoding between different modalities at the source. Furthermore, this embodiment of the present disclosure determines the feature vector corresponding to each input data through the encoder, and then inputs the feature vector into the vector database corresponding to the modality to which the input data to which the feature vector belongs for retrieval, and obtains the retrieval results. This ensures that the retrieval process can be carried out in the independent semantic spaces of text, images, and videos, avoiding semantic interference during cross-modal retrieval. Finally, this embodiment of the present disclosure determines the multimodal output results based on user queries and retrieval results, thereby effectively taking into account the semantic differences between different modalities, and can flexibly adjust the utilization degree of different modal information according to user queries, thereby improving the accuracy, coordination, and user satisfaction of multimodal content generation.

[0066] In one embodiment, before inputting the feature vector into the vector database for retrieval, the following steps are also included: Obtain external data source; A text vector database is constructed based on text data; an image vector database is constructed based on image data; and a video vector database is constructed based on video data.

[0067] Specifically, before inputting the feature vectors into the vector database for retrieval, the executing entity can first obtain external data sources, which include at least one of the following: text data, image data, and video data. Then, the executing entity can construct a text vector database based on the text data, an image vector database based on the image data, and a video vector database based on the video data.

[0068] In one possible embodiment, for the construction of a text vector database, the executing entity first preprocesses the externally acquired document data. This preprocessing includes data cleaning and long text segmentation to obtain well-structured text fragments. Subsequently, the executing entity uses a text encoder to encode each text fragment, converting it into a high-dimensional text feature vector based on the semantic information of the text. Finally, the executing entity organizes and stores all the generated text feature vectors to construct a text vector database.

[0069] In one possible embodiment, for the construction of the image vector database, the executing entity uses an image encoder to process the raw image data acquired from the outside, extracts image features, generates an image feature vector corresponding to each image, and then constructs an image vector database.

[0070] In one possible embodiment, such as Figure 3 As shown, constructing a video vector database based on video data includes the following steps: Step 301: The video data is segmented to obtain multiple video segments.

[0071] In one possible embodiment, the executing entity acquires the raw video data and performs segmentation processing on each video, thereby dividing the continuous video stream into multiple independent parts that are consecutive in time, resulting in multiple video segments. The segmentation processing can be automatically completed based on fixed time intervals, keyframe detection, or scene changes.

[0072] Step 302: Extract the frame feature vector of each video frame in each video segment, and average the frame feature vectors to obtain the segment feature vector of each video segment.

[0073] In one possible embodiment, for each video segment obtained in step 301, the executing entity extracts the frame feature vectors of the multiple video frames contained therein. Specifically, the executing entity uses a pre-trained image encoder to extract features from each video frame to obtain the frame feature vectors of each frame. Subsequently, the executing entity performs average processing on all the frame feature vectors in the segment, such as an arithmetic average, to obtain the segment feature vector that represents the overall visual content of the segment.

[0074] Step 303: Concatenate the feature vectors of the segments to obtain the visual feature vectors of the video data.

[0075] In one possible embodiment, the executing entity sequentially concatenates the segment feature vectors of each video segment obtained in step 302 according to the time sequence, thereby obtaining a coherent overall visual feature vector that can represent the visual content of the entire video data.

[0076] Step 304: Extract the audio content from the video data and determine the text feature vector based on the audio content.

[0077] In one possible embodiment, the executing entity separates and extracts the audio track from the video data to obtain the audio content. Subsequently, the executing entity converts the audio content into corresponding text data using automatic speech recognition technology. Finally, the executing entity uses a text encoder to encode the text data, extract its semantic features, and generate the corresponding text feature vector.

[0078] Step 305: The visual feature vector and the text feature vector are fused to obtain the fused feature vector of the video data, and a video vector database is constructed based on the fused feature vector.

[0079] In one possible embodiment, the executing entity uses multimodal fusion technology, such as vector concatenation or a neural network-based fusion module, to fuse the overall visual feature vector obtained in step 303 with the text feature vector obtained in step 304 to generate a unified fusion feature vector that represents the audiovisual semantics of the video. Finally, the executing entity constructs a video vector database based on this fusion feature vector for subsequent retrieval.

[0080] In this embodiment, firstly, the execution entity segments the video data to obtain multiple video segments; then, the execution entity extracts the frame feature vectors of the video frames in each video segment and averages the frame feature vectors to obtain the segment feature vectors of each video segment; next, the execution entity concatenates the segment feature vectors to obtain the visual feature vectors of the video data; then, the execution entity extracts the audio content from the video data and determines the text feature vectors based on the audio content; finally, the execution entity fuses the visual feature vectors and the text feature vectors to obtain the fused feature vectors of the video data, and constructs a video vector database based on the fused feature vectors.

[0081] As described above, in this embodiment, the executing entity systematically segments the video data, extracts and averages frame features, and concatenates segment feature vectors to fully capture the visual semantic information of the video, forming an overall visual feature vector. Simultaneously, the executing entity extracts and identifies the audio content in the video to obtain corresponding text feature vectors, thereby fusing the audiovisual dual-modal information of the video. Finally, through multimodal fusion technology, the visual and text feature vectors are combined into a unified fused feature vector, which is then used to construct a video vector database. This embodiment not only achieves the full extraction and effective association of multimodal information in video data, but also the constructed video vector database supports subsequent accurate and efficient retrieval based on unified semantic representation, providing a high-quality, highly relevant video information foundation for the generation of multimodal content.

[0082] As described above, the technical solution disclosed herein effectively solves the semantic confusion and modal interference problems caused by cross-modal unified encoding and retrieval in the prior art by independently assigning corresponding encoders to input data of different modalities and retrieving them in their respective independent modal vector databases. Furthermore, the technical solution disclosed herein ensures the semantic relevance, complementarity, and flexibility of multimodal information in the generated content through semantic consistency processing and a modal weight integration mechanism that dynamically adjusts based on user preferences, significantly improving the accuracy, coordination, and user satisfaction of multimodal content generation.

[0083] By dividing each functional module according to its corresponding function, this disclosure provides a multimodal generation device, which can be a server or a chip applied to a server. Figure 4 A schematic block diagram of the functional modules of a multimodal generation apparatus provided for an exemplary embodiment of this disclosure. Figure 4 As shown, the multimodal generation device includes: The receiving module 401 is used to receive user queries; wherein the user query contains at least one input data, and each input data belongs to a modality; The first determining module 402 is used to assign a corresponding encoder to each input data based on the mode to which the input data belongs, and to determine the feature vector corresponding to each input data through the encoder; The retrieval module 403 is used to input the feature vector into a vector database for retrieval and obtain retrieval results; wherein, the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; The second determining module 404 is used to determine the multimodal output result based on the user query and the search result.

[0084] In one embodiment, the first determining module 402 includes: The first allocation unit is used to allocate a text encoder to the input data belonging to the text modality; The second allocation unit is used to allocate an image encoder to the input data belonging to the image modality; The third allocation unit is used to allocate video encoders to the input data belonging to the video modality.

[0085] In one embodiment, the second determining module 404 includes: A semantic processing unit is used to perform semantic consistency processing on the search results to obtain multimodal search information; wherein the multimodal search information includes at least one of the following: text search results, image search results, and video search results; The first determining unit is configured to determine a first search result based on the text search result, the image search result, and the video search result; wherein the first search result is a search result obtained by integrating the text search result, the image search result, and the video search result based on modal weights; The first determining unit is used to input the user query and the first retrieval result into the multimodal large model to obtain the multimodal output result.

[0086] In one embodiment, the second determining module 404 includes: The calculation unit is configured to calculate the modal weights corresponding to the text retrieval result, the image retrieval result, and the video retrieval result using a first formula; wherein, the first formula is... , Representing search results for different modalities, The modal weights, As the initial weights, Here, is the decay coefficient, and t is the interval between the weight update time and the corresponding search result generation time. Output a content acceptance rating for the user's search results for the different modalities; The normalization unit is used to normalize the modal weights to obtain the normalized weights corresponding to the text retrieval results, the image retrieval results, and the video retrieval results. The selection unit is used to select a corresponding number of search segments from the text search results, image search results, and video search results based on the normalized weights. A construction unit is used to construct the first search result from the search fragments.

[0087] In one embodiment, the apparatus further includes: An acquisition module is used to acquire external data sources, wherein the external data sources include at least one of the following: text data, image data, and video data; The first construction module is used to construct a text vector database based on the text data; The second construction module is used to construct an image vector database based on the image data; The third construction module is used to construct a video vector database based on the video data.

[0088] In one embodiment, the third building module includes: The segmentation unit is used to segment the video data to obtain multiple video segments; The first extraction unit is used to extract the frame feature vector of each video frame in each video segment, and to perform average processing on the frame feature vector to obtain the segment feature vector of each video segment. The splicing unit is used to splice the feature vectors of the segments to obtain the visual feature vectors of the video data; The second extraction unit is used to extract the audio content from the video data and determine the text feature vector based on the audio content; The fusion construction unit is used to fuse the visual feature vector and the text feature vector to obtain the fused feature vector of the video data, and to construct the video vector database based on the fused feature vector.

[0089] Figure 5 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 5 As shown, the electronic device 500 includes at least one processor 501 and a memory 502 coupled to the processor 501, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0090] The processor 501 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 501 or by software instructions. The processor 501 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 502, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 501 reads information from the memory 502 and, in conjunction with its hardware, completes the steps of the method described above.

[0091] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 6The computer system 600 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 6 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0092] Computer system 600 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0093] like Figure 6 As shown, the computer system 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the computer system 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] Multiple components in the computer system 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device capable of inputting information into the computer system 600. The input unit 606 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 608 may include, but is not limited to, a hard disk and an optical disk. The communication unit 609 allows the computer system 600 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ device, WiFi device, WiMax device, cellular communication device, and / or the like.

[0095] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 602 and / or communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0096] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0097] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0099] Figure 7 A computer program product 700 is provided as an exemplary embodiment of the present disclosure. The computer program product 700 includes a computer program 701, wherein the computer program 701, when executed by a processor, implements the methods disclosed in the embodiments of the present disclosure.

[0100] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0103] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0104] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0105] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A multimodal generation method, characterized in that, include: Receive user queries; wherein the user query contains at least one input data, and each input data belongs to a modality; Based on the modality to which the input data belongs, a corresponding encoder is assigned to each input data, and the feature vector corresponding to each input data is determined through the encoder; The feature vector is input into a vector database for retrieval to obtain retrieval results; wherein, the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; The multimodal output result is determined based on the user query and the search results.

2. The method according to claim 1, characterized in that, The modality includes at least one of the following: text modality, image modality, and video modality; Assigning a corresponding encoder to each input data based on the modality to which the input data belongs includes: Assign a text encoder to the input data belonging to the text modality; Assign an image encoder to the input data belonging to the image modality; Assign a video encoder to the input data belonging to the video modality.

3. The method according to claim 1, characterized in that, The process of determining the multimodal output result based on the user query and the search results includes: The search results are subjected to semantic consistency processing to obtain multimodal search information; wherein, the multimodal search information includes at least one of the following: text search results, image search results, and video search results; A first search result is determined based on the text search results, the image search results, and the video search results; wherein, the first search result is a search result obtained by integrating the text search results, the image search results, and the video search results based on modal weights; The user query and the first search result are input into the multimodal large model to obtain the multimodal output result.

4. The method according to claim 3, characterized in that, Determining the first search result based on the text search result, the image search result, and the video search result includes: The modal weights corresponding to the text retrieval results, the image retrieval results, and the video retrieval results are calculated using a first formula; wherein, the first formula is: , Representing search results for different modalities, The modal weights, As the initial weights, Here, is the decay coefficient, and t is the interval between the weight update time and the corresponding search result generation time. Output a content acceptance rating for the user's search results for the different modalities; The modal weights are normalized to obtain the normalized weights corresponding to the text retrieval results, the image retrieval results, and the video retrieval results; Based on the normalized weights, a corresponding number of search segments are selected from the text search results, image search results, and video search results, respectively. The search fragments are used to construct the first search result.

5. The method according to claim 1, characterized in that, Before inputting the feature vector into the vector database for retrieval, the method further includes: Obtain an external data source, wherein the external data source includes at least one of the following: text data, image data, and video data; Construct a text vector database based on the text data; An image vector database is constructed based on the image data; A video vector database is constructed based on the video data.

6. The method according to claim 5, characterized in that, The construction of a video vector database based on the video data includes: The video data is segmented to obtain multiple video segments; Extract the frame feature vector of each video frame in each video segment, and average the frame feature vectors to obtain the segment feature vector of each video segment. The feature vectors of the segments are concatenated to obtain the visual feature vectors of the video data; Extract the audio content from the video data, and determine the text feature vector based on the audio content; The visual feature vector and the text feature vector are fused to obtain the fused feature vector of the video data, and the video vector database is constructed based on the fused feature vector.

7. A multimodal generation device, characterized in that, include: A receiving module is used to receive user queries; wherein the user query contains at least one input data, and each input data belongs to a modality; The first determining module is used to assign a corresponding encoder to each input data based on the mode to which the input data belongs, and to determine the feature vector corresponding to each input data through the encoder; The retrieval module is used to input the feature vector into a vector database for retrieval and obtain retrieval results; wherein, the vector database corresponds to the modality to which the input data corresponding to the feature vector belongs; The second determining module is used to determine the multimodal output result based on the user query and the search result.

8. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images