Multi-modal information-based training, index construction and search methods and devices

Abstract

The embodiment of the invention provides a training method and device, an index construction method and device and a searching method and device based on multi-modal information. The model training method comprises the steps that a pre-trained generative model is acquired, and the generative model comprises an image-text processing unit and a language model unit; obtaining an instruction word generated based on the cue word and object association information of the target object, wherein the object association information comprises object description information and image information; constructing input data based on the instruction word, the object description information and the image information, inputting the input data into a generative model, and outputting a corresponding keyword; calculating loss information through the keyword; and adjusting an image-text processing unit and a language model unit of the generative model based on the loss information. The precision of the generated model can be improved, so that the accuracy of keywords is improved, the accuracy of search results is improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method based on multimodal information, an index construction method based on multimodal information, a search method based on multimodal information, an electronic device, a computer-readable storage medium and a computer program product. Background Art

[0002] On e-commerce platforms, information about commodity objects is usually presented in the form of multimodal data. For example, information about commodity objects includes text information such as title, category, and commodity description information, as well as information such as images and videos. When users search on e-commerce commodity platforms, they often search by simply entering search information with text descriptions.

[0003] The platform usually segments the user's search information into keywords and then searches based on the keywords. However, since the search keywords are text information, they are usually matched based on the subject, category, product description information, etc. of the product object. However, users usually use some colloquial expressions in actual searches. When searching for texts based on such colloquial expressions, the keyword matches are often inaccurate, resulting in inaccurate search results. Summary of the invention

[0004] The embodiment of the present application provides a model training method based on multimodal information to improve the accuracy of the model and thereby improve the accuracy of keywords.

[0005] Correspondingly, an embodiment of the present application also provides an index construction method based on multimodal information, a search method based on multimodal information, an electronic device, a computer-readable storage medium and a computer program product to ensure the implementation and application of the above-mentioned system.

[0006] In order to solve the above problems, the embodiment of the present application discloses a model training method based on multimodal information, wherein the method includes:

[0007] Obtaining a pre-trained generative model, wherein the generative model includes: a graphic and text processing unit and a language model unit;

[0008] Acquire the instruction word generated based on the prompt word and the object association information of the target object, wherein the object association information includes object description information and image information;

[0009] constructing input data based on the instruction words, object description information, and image information, inputting the input data into a generation model, and outputting corresponding keywords;

[0010] Calculate loss information using the keywords;

[0011] The graphic processing unit and the language model unit of the generation model are adjusted based on the loss information.

[0012] Optionally, parameters of the image coding unit and the fully connected layer are frozen during the fine-tuning process.

[0013] Optionally, the method further includes the step of generating instruction words based on the prompt words: constructing prompt words based on object description information of the target object, and generating instruction words based on the prompt words.

[0014] Optionally, constructing a prompt word based on the object description information of the target object, and generating an instruction word based on the prompt word includes:

[0015] Construct prompt words based on the object description information of the target object and the instruction template;

[0016] Inputting the instruction word into a language model, and outputting a predicted keyword corresponding to the instruction word;

[0017] Screening the predicted keywords to determine at least one screened keyword;

[0018] Based on the at least one filtered keyword and the instruction template, an instruction word corresponding to the prompt word is generated.

[0019] Optionally, the generation model further includes: an image encoding unit and a fully connected layer; constructing input data based on the instruction word, object description information, and image information, inputting the input data into the generation model, and outputting corresponding keywords, including:

[0020] Inputting the image information into an image encoding unit for encoding processing, and outputting corresponding image features;

[0021] Combining the instruction word and the object description information to construct a corresponding object instruction text, inputting the image feature and the object instruction text into a graphic processing unit, and outputting a corresponding multimodal feature;

[0022] Input the multimodal features into a fully connected layer for processing, and output corresponding intermediate features;

[0023] The intermediate features and instruction words are input into the language model unit for processing, and the corresponding keywords are output.

[0024] Optionally, the adjusting the image and text processing unit and the language model unit of the generation model based on the loss information includes:

[0025] Parameters of the image and text processing unit are adjusted based on the loss information, and parameters of the language model unit are adjusted based on the loss information.

[0026] The present application also discloses a method for constructing an index based on multimodal information, wherein the method includes:

[0027] Acquire object association information of a target object, wherein the object association information includes object description information and image information;

[0028] Constructing input data based on the object description information and image information, inputting the input data into a generation model, and outputting corresponding keywords, wherein the generation model is fine-tuned based on instruction words after pre-training is completed, and the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction words are generated based on prompt words;

[0029] The index information of the target object is constructed based on the keyword, and the index information is added to the target index to perform a search process based on the target index.

[0030] Optionally, constructing input data based on the object description information and image information, inputting the input data into a generation model, and outputting corresponding keywords include:

[0031] Inputting the object description information and image information into the generation model;

[0032] Determining multimodal features based on a graphic processing unit of the generative model;

[0033] The multimodal features are input into the language model unit, and corresponding keywords are output.

[0034] Optionally, the generation model further includes: an image encoding unit; the image and text processing unit based on the generation model determines the multimodal features, including:

[0035] Inputting the image information into an image encoding unit for encoding processing, and outputting corresponding image features;

[0036] The object description information and the image features are input into a graphics processing unit, and corresponding multimodal features are output.

[0037] Optionally, the generation model further includes: a fully connected layer, wherein the multimodal features are input into the language model unit and the corresponding keywords are output, including:

[0038] Input the multimodal features into a fully connected layer for processing, and output corresponding intermediate features;

[0039] The intermediate features are input into the language model unit for processing, and the corresponding keywords are output.

[0040] Optionally, the target index includes at least one of the following: an inverted index, a vectorized index, and a competitive index;

[0041] The step of constructing the index information of the target object based on the keyword and adding the index information to the target index includes at least one of the following steps:

[0042] Performing word segmentation processing on the keyword, and adding the word segmentation information of the keyword to the inverted index;

[0043] Inputting the keywords into a semantic model, and adding the output multi-dimensional vector representation to the vectorized index;

[0044] The keywords and corresponding target objects are combined, and the combined information is added to the competition index.

[0045] Optionally, also include the step of fine-tuning the pre-trained generative model:

[0046] Obtain a pre-trained generative model, wherein the generative model includes: an image encoding unit, a graphic processing unit, a fully connected layer, and a language model unit;

[0047] Obtaining object association information of the instruction word and the target object generated based on the prompt word;

[0048] constructing input data based on the instruction words, object description information, and image information, inputting the input data into a generation model, and outputting corresponding keywords;

[0049] Calculate loss information using the keywords;

[0050] The graphic and text processing unit and the language model unit of the generation model are adjusted based on the loss information.

[0051] The embodiment of the present application also discloses a search method based on multimodal information, including:

[0052] Receive keywords;

[0053] The keyword is matched with a target index to determine at least one target object matched by the keyword, wherein the target index is added with keywords generated based on object association information of the target object, and the generation model for generating the keyword is fine-tuned based on the instruction word after pre-training is completed, and the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction word is generated based on the prompt word;

[0054] generating search results based on the at least one target object;

[0055] Feedback on the search results.

[0056] The embodiment of the present application also discloses a search method based on multimodal information, including:

[0057] Receive keywords;

[0058] Using the keyword matching index information, at least one commodity object matched by the keyword is determined, the commodity object includes an advertisement object, wherein the target index is added with a keyword generated based on object association information of the advertisement object, the generation model for generating the keyword is generated by fine-tuning based on the instruction word after pre-training is completed, the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction word is generated based on the prompt word;

[0059] generating search results based on the at least one product object;

[0060] Feedback on the search results.

[0061] The embodiment of the present application also discloses an electronic device, which includes: a processor, and a memory communicatively connected to the processor;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the present application.

[0064] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the embodiment of the present application.

[0065] The embodiment of the present application further discloses a computer program product, including a computer program / computer executable instructions, wherein the computer program / computer executable instructions, when executed by a processor in an electronic device, implement the method described in the embodiment of the present application.

[0066] Compared with the prior art, the embodiments of the present application have the following advantages:

[0067] In an embodiment of the present application, fine-tuning is performed on a pre-trained generative model to obtain instruction words generated based on prompt words and object association information of the target object. The instruction words are used to guide the fine-tuning task of the generative model, and input data is constructed based on the instruction words, object description information, and image information. The input data is input into the generative model, and corresponding keywords are output. Loss information is then calculated using the keywords. The graphic processing unit and the language model unit of the generative model are adjusted in a targeted manner based on the loss information to complete the fine-tuning of the generative model, thereby improving the accuracy of the generative model, thereby improving the accuracy of the keywords, and then improving the accuracy of the search results, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flowchart of a step of an embodiment of a model training method based on multimodal information of the present application;

[0069] Figure 2 is a schematic diagram of an example of a generation model according to an embodiment of the present application;

[0070] Figure 3 It is a flowchart of the steps of an optional embodiment of a model training method based on multimodal information of the present application;

[0071] Figure 4 It is a flowchart of a method for constructing an index based on multimodal information according to an embodiment of the present application;

[0072] Figure 5 It is a flowchart of the steps of an optional embodiment of an index construction method based on multimodal information of the present application;

[0073] Figure 6 It is a flowchart of a step of an embodiment of a search method based on multimodal information of the present application;

[0074] Figure 7 is a flowchart of another embodiment of a model training method based on multimodal information of the present application;

[0075] Figure 8 is a flowchart of another embodiment of an index construction method based on multimodal information of the present application;

[0076] Fig. 9 is a flowchart of another embodiment of a search method based on multimodal information of the present application;

[0077] Fig.10 It is a schematic diagram of the structure of an exemplary device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0079] The embodiments of the present application can be applied to search scenarios, such as searching for commodity objects on an e-commerce platform, as well as advertising search, advertising bidding search and other scenarios. It is possible to analyze the multimodal information of the target object, such as images, texts, etc., to determine the search keywords of the target object, so that the corresponding target object can be searched based on the search keywords during the search, thereby improving the search efficiency. Among them, multimodality refers to multiple forms of expression of the data information of an object, such as text, images, videos, audio and other forms of data.

[0080] In order to make the search for the target object more accurate, a generative model can be built and trained. The generative model processes multimodal information and generates corresponding keywords to make the keywords describing the target object more accurate and meet user needs. The keywords are added to the index as index information of the target object.

[0081] In an embodiment of the present application, a generative model can be pre-constructed and pre-trained. After the pre-training of the generative model is completed, some parameters in the generative model can be frozen, so as to fine-tune some units in the training model in a targeted manner and improve the accuracy of the generative model.

[0082] Reference Figure 1 , showing a step flow chart of an embodiment of a model training method based on multimodal information of the present application.

[0083] Step 102, obtaining a pre-trained generative model.

[0084] Obtain a pre-trained generative model, which is a generative model that has been trained in advance and can be used. However, in order to further improve the supervision of the generative model, the generative model can be fine-tuned, where the fine-tuning is mainly on the image and text processing unit and the language model unit.

[0085] like Figure 1 As shown, the generation model includes: an image encoding unit 10, a graphic processing unit 20, a fully connected layer 30 and a language model unit 40, wherein the image encoding unit is used to encode the image and determine the corresponding image features. The graphic processing unit is used to analyze multimodal information such as images and texts to obtain corresponding multimodal features. The image features output by the image encoding unit will be input into the graphic processing unit together with other multimodal information such as text information for processing. The fully connected layer is used to perform dimensionality reduction processing on the features, and the multimodal features output by the graphic processing unit can be reduced in dimensionality to obtain corresponding intermediate features. The language model unit is used to process the intermediate features and generate corresponding search keywords.

[0086] By fine-tuning the image and text processing unit and the language model unit, we can make more effective use of image and text information, more accurately improve language processing capabilities, and improve the accuracy of the generated model.

[0087] The embodiments of the present application perform supervised fine-tuning on the generative model, such as fine-tuning through instructions. The instruction-tuning technology can decouple the fine-tuning of the model from various tasks, thereby performing targeted fine-tuning.

[0088] Step 104: Obtain object association information of the instruction word generated based on the prompt word and the target object.

[0089] Obtain the instruction words required for fine-tuning, which are generated based on the prompt words. The instruction words can be generated by combining the prompt words with the data of the target object. Among them, the prompt word can be understood as a piece of instruction or text, which is used to prompt the content of the task to be performed, such as constructing a search keyword for the target object in a certain aspect, which can assist the model in generating more accurate and more relevant content. The instruction words are used to perform the fine-tuning task corresponding to the prompt words. The object association information refers to various information used to describe the corresponding, such as object association information including object description information and image information.

[0090] In an optional embodiment, the step of generating instruction words based on prompt words includes: constructing prompt words based on the object description information of the target object, and generating instruction words based on the prompt words. Wherein, the object description information of the target object refers to information used to describe the target object, such as various information such as name, attributes, etc. Taking the commodity object as an example, its description information may include various text information such as name, category, information in the commodity object page such as title, commodity introduction information in the page, etc. Another example is the advertising object information of the advertising object, such as advertising words, product name, advertising information, etc. Prompt words suitable for the target object can be constructed based on the object description information, and then the prompt words are analyzed to determine the corresponding instruction words. Wherein, the object description information of each target object is associated with the search keyword of the target object, and the search keyword is the search keyword entered by the user when actually searching for the target object. For example, the search keyword entered by the user and the target object clicked are collected through the user's behavior log, and the search keyword and the target object clicked are associated, so as to associate the search keyword with the object description information of the target object, so as to filter the predicted keywords required to generate the instruction words in the subsequent instruction word generation process.

[0091] In an optional embodiment of the present application, instruction templates corresponding to various prompt tasks can be pre-generated, and instruction words can be constructed based on the instruction templates and prompt words to perform instruction fine-tuning tasks. In an optional embodiment, the prompt words are constructed based on the object description information of the target object, and the instruction words are generated based on the prompt words, including: constructing prompt words based on the object description information of the target object and the instruction template; inputting the instruction words into the language model, and outputting the predicted keywords corresponding to the instruction words; filtering the predicted keywords to determine at least one filtered keyword; and generating the instruction words corresponding to the prompt words based on the at least one filtered keyword and the instruction template.

[0092] In the embodiment of the present application, an instruction template can be constructed, and then the object description information of the target object is matched according to the instruction template to obtain the corresponding instruction word. The instruction word is input into a language model for processing, and the language model is used to predict keywords, and the predicted keyword corresponding to the instruction word can be output accordingly. The predicted keyword is the keyword corresponding to the prompt of the instruction word predicted by the model, that is, the keyword that the user may enter. Then the predicted keyword is screened, and the screening of the predicted keyword can be carried out in a variety of ways, such as calculating the similarity between the predicted keyword and the actual search keyword of the user, and screening based on the BLEU (Bilingual Evaluation Understudy, bilingual evaluation substitute) index, calculating the difference index between the predicted keyword and the actual search keyword of the user based on the BLEU index, and screening the most similar top N predicted keywords based on the difference index, where N is a positive integer greater than 1, such as N is 3, 4, 5...10, etc. The at least one screened keyword and the instruction template are respectively combined to generate the instruction word corresponding to the prompt word.

[0093] Step 106, construct input data based on the instruction words, object description information, and image information, input the input data into the generation model, and output corresponding keywords.

[0094] After the construction of the instruction word is completed, the generation model can be fine-tuned based on the instruction word. The training data for fine-tuning consists of instruction words and object association information. The object association information can be obtained based on log information. The user's behavior data and the click data and exposure data of the target object are obtained from the log data. Then the data is cleaned to determine the keywords entered by the user and the target object clicked. The corresponding relationship between the keywords and the object association information of the target object is established, which is convenient for subsequent training as annotation information for fine-tuning.

[0095] The command words, object description information, and image information are used to construct input data, and then the input data is input into the generation model for processing, and the corresponding search keywords are output. The image information is input into the image encoding unit for encoding processing, and the corresponding image features are output; the command words and object description information are combined to construct the corresponding object command text, and the image features and object command text are input into the image and text processing unit, and the corresponding multimodal features are output; the multimodal features are input into the fully connected layer for processing, and the corresponding intermediate features are output; the intermediate features and the command words are input into the language model unit for processing, and the corresponding keywords are output.

[0096] In an optional embodiment, the generative model includes: an image encoding (Image Encoder) unit, a graphic processing (Q-Former) unit, a fully connected (Projection) layer and a language model (Large Language Model, LLM) unit. In an embodiment of the present application, the generative model is pre-trained. After completing the pre-training of the generative model, the parameters of the image encoding unit and the fully connected layer are frozen, and the graphic processing unit and the language model unit are fine-tuned, wherein the graphic processing unit and the language model unit can also freeze some parameters, and only fine-tune other parameters. The specific frozen and fine-tuned parameters can be determined based on demand. The frozen parameters will not change during the fine-tuning training process, thereby greatly reducing the training data required for model training and the training machine overhead.

[0097] Taking the commodity object as an example, the textual object description information such as the commodity object's category, title, attribute, commodity introduction information, etc. can be obtained and combined with the instruction words as the object instruction text instruction in model training.

[0098] The image information of the commodity object is first input into the image encoding unit that has been pre-trained and has frozen parameters to obtain the corresponding commodity image features. Then the image features and the combined object instruction text are used as input and input into the image and text processing unit. The image and text processing unit can analyze the intention of the instruction, combine the text and image features, obtain multimodal features, and then perform dimensionality reduction processing through the fully connected layer. Then, it is input into the language model unit together with the instruction word to output the virtual keyword corresponding to the commodity object.

[0099] Step 108, calculating loss information using the keywords.

[0100] Calculate loss information based on the keyword and the user's actual search keyword, such as calculating loss value, loss function and other loss information. In an optional embodiment, the calculation of loss information through the keyword includes: calculating a loss function based on the keyword and the search keyword as loss information. In one example, the loss function is an entropy loss function, an entropy loss function, etc.

[0101] Step 110: Adjust the image and text processing unit and the language model unit of the generation model based on the loss information.

[0102] Based on the loss function, the parameters of the image and text processing unit and the language model unit are adjusted. If the parameters in the image and text processing unit and the language model unit are partially frozen, the unfrozen parameters are adjusted. The parameters are updated and iterated through fine-tuning, and the other pre-trained parameters do not change, and only serve as gradient feedback. Finally, a generative model with efficient parameter fine-tuning is obtained.

[0103] In summary, by fine-tuning the pre-trained generation model, the instruction words generated based on the prompt words and the object association information of the target object can be obtained, the fine-tuning task of the generation model is guided by the instruction words, input data is constructed based on the instruction words, object description information, and image information, the input data is input into the generation model, the corresponding keywords are output, and then the loss information is calculated through the keywords. The graphic processing unit and the language model unit of the generation model are adjusted in a targeted manner based on the loss information to complete the fine-tuning of the generation model, improve the accuracy of the generation model, thereby improving the accuracy of the keywords, and then improving the accuracy of the search results and improving the user experience.

[0104] Based on the above embodiments, the present application also provides a model training method based on multimodal information, such as Figure 3 As shown:

[0105] Step 302, obtaining a pre-trained generative model, the generative model comprising: an image coding unit, a graphic processing unit, a fully connected layer and a language model unit. The parameters of the image coding unit and the fully connected layer are frozen during the fine-tuning process. Furthermore, some parameters of the graphic processing unit and the language model unit are frozen, and other parameters are fine-tuned.

[0106] Step 304: construct prompt words based on the object description information of the target object, and generate instruction words based on the prompt words.

[0107] Among them, the constructing of prompt words based on the object description information of the target object and generating instruction words based on the prompt words include: constructing prompt words based on the object description information of the target object and an instruction template; inputting the instruction words into a language model and outputting predicted keywords corresponding to the instruction words; filtering the predicted keywords to determine at least one filtered keyword; and generating an instruction word corresponding to the prompt words based on the at least one filtered keyword and the instruction template.

[0108] Step 306: Obtain object association information of the instruction word generated based on the prompt word and the target object.

[0109] Step 308: input the image information into an image encoding unit for encoding processing, and output corresponding image features.

[0110] Step 310, combining the instruction words and the object description information to construct a corresponding object instruction text, inputting the image features and the object instruction text into a graphic processing unit, and outputting corresponding multimodal features.

[0111] Step 312: input the multimodal features into a fully connected layer for processing, and output corresponding intermediate features.

[0112] Step 314: input the intermediate features and instruction words into the language model unit for processing, and output corresponding keywords.

[0113] Step 316, calculating loss information using the keywords.

[0114] Step 318: Adjust the graphic processing unit and the language model unit of the generation model based on the loss information.

[0115] The embodiment of the present application utilizes the capability of a large model and fine-tunes the model by means of efficient parameter fine-tuning, which can fit the data very well and obtain a generative model. Through fine-tuning, the keywords generated by the generative model have similar expression capabilities to the search keywords of the user, thereby improving the accuracy of the generative model.

[0116] After completing the fine-tuning of the generative model, the keywords of the target object can be predicted based on the generative model, and an index can be built based on the keywords to improve the accuracy and hit rate of subsequent searches.

[0117] Reference Figure 4 , showing a step flow chart of an embodiment of an index construction method based on multimodal information of the present application.

[0118] Step 402: Obtain object association information of the target object, wherein the object association information includes object description information and image information.

[0119] After fine-tuning the pre-trained generative model based on the above steps, a trained generative model is obtained. Keywords of the target object are generated based on the generative model so that the target object can be retrieved more accurately later. Object association information of the target object can be obtained first, such as object association information of the advertising object, including object description information such as advertising words, product names, attributes, and image information such as advertising pictures and videos.

[0120] Step 404: construct input data based on the object description information and image information, input the input data into the generation model, and output corresponding keywords.

[0121] Input data is constructed based on the object description information and image information, and then the input data is input into a generation model. Through processing by the generation model, keywords of the target object are determined.

[0122] The processing is similar to that in the above-mentioned fine-tuning process, but after the fine-tuning of the generative model is completed, the instruction words are no longer needed, and the input data can be constructed based on the object association information of the target object. In the embodiment of the present application, the generative model combines multimodal features such as images and texts to generate keywords, so that the data of the target object can be more fully used to improve the accuracy of the keywords.

[0123] In an optional embodiment, the input data is constructed based on the object description information and image information, the input data is input into the generative model, and the corresponding keywords are output, including: inputting the object description information and image information into the generative model; determining multimodal features based on the image and text processing unit of the generative model; inputting the multimodal features into the language model unit, and outputting corresponding keywords. The object description information and image information are input as input data into the generative model, and each unit in the generative model can process the input data. Among them, the image and text processing unit processes based on the features of image, text and other data to obtain multimodal features, and thus can perform analysis and processing based on the object description information and image information. Based on the processing of the image and text processing unit, the language model unit further analyzes and processes in combination with the multimodal features output by the image and text processing unit to obtain corresponding keywords.

[0124] Wherein, the generation model further includes: an image encoding unit; the image and text processing unit based on the generation model determines the multimodal features, including: inputting the image information into the image encoding unit for encoding processing and outputting the corresponding image features; inputting the object description information and the image features into the image and text processing unit and outputting the corresponding multimodal features. The generation model further includes: a fully connected layer, the multimodal features are input into the language model unit and the corresponding keywords are output, including: inputting the multimodal features into the fully connected layer for processing and outputting the corresponding intermediate features; inputting the intermediate features into the language model unit for processing and outputting the corresponding keywords.

[0125] After the input data is input into the generative model, the image encoding unit first processes the image information and outputs the corresponding image features. The image features and object description information are then combined to form a combined input, which is input into the image and text processing unit. The image and text processing unit processes multimodal data such as text and images and outputs the corresponding multimodal features. The multimodal features are then reduced in dimension through the fully connected layer, and the reduced multimodal features are output as intermediate features. The intermediate features are then input into the language model unit for processing, and the keywords of the target object are output through semantic analysis and other processing.

[0126] Step 406: construct index information of the target object based on the keyword, and add the index information to the target index to perform search processing based on the target index.

[0127] In an embodiment of the present application, index information of the target object is generated based on the keywords generated by the generation model, the keywords are used as the keywords of the target object to update the index information, and the index information is added to the target index so that the target object can be matched when a subsequent search is performed based on the target index, thereby improving the query rate of the target object.

[0128] In an embodiment of the present application, the target index includes multiple indexes, such as one or more of an inverted index, a vectorized index, and a competitive index. Among them, the inverted index is also called a reverse index, which can query objects based on keywords, so based on the inverted index, the object information of the target object can be queried through the keywords of the target object. The vectorized index vectorizes the query terms and objects, queries the vector of the object through the vectorized query terms, and then determines the target object. The competitive index refers to an index that competitively ranks the target objects based on resources, such as based on click-through rate.

[0129] The step of constructing the index information of the target object based on the keyword and adding the index information to the target index includes at least one of the following steps: performing word segmentation on the keyword and adding the word segmentation information of the keyword to the inverted index; inputting the keyword into the semantic model and adding the output multidimensional vector representation to the vectorized index; combining the keyword and the corresponding target object and adding the combination information to the competitive index. Perform word segmentation on the keyword to obtain word segmentation information, and then add the word segmentation information and the target object to the inverted index. Input the keyword into the semantic model, convert the keyword into a multidimensional vector representation, determine the multidimensional vector representation of the target object, and add the multidimensional vector representation corresponding to the keyword and the multidimensional vector representation of the target object to the vectorized index. Combine the keyword and the target object to obtain competitive combination information of the target object and add the combination information to the competitive index.

[0130] Thus, the index information of various indexes can be updated based on keywords and target objects and added to the target index to facilitate subsequent retrieval.

[0131] Based on the above embodiments, the present application also provides an index construction method based on multimodal information, such as Figure 5 As shown:

[0132] Step 502: Obtain object association information of the target object, wherein the object association information includes object description information and image information.

[0133] Step 504: construct input data based on the object description information and image information, and input the input data into the generation model.

[0134] Step 506: input the image information into an image encoding unit for encoding processing, and output corresponding image features.

[0135] Step 508: input the object description information and the image features into a graphics processing unit, and output corresponding multimodal features.

[0136] Step 510: input the multimodal features into a fully connected layer for processing, and output corresponding intermediate features.

[0137] Step 512: input the intermediate features into the language model unit for processing, and output corresponding keywords.

[0138] Index information of the target object is constructed based on the keyword, and the index information is added to a target index to perform search processing based on the target index, wherein the target index includes at least one of the following: an inverted index, a vectorized index, and a competitive index.

[0139] Step 514: perform word segmentation processing on the keyword, and add the word segmentation information of the keyword to the inverted index.

[0140] Step 516: input the keyword into the semantic model, and add the output multi-dimensional vector representation to the vectorized index.

[0141] Step 518: Combine the keyword and the corresponding target object, and add the combined information to the competitive index.

[0142] Through fine-tuning, the generative model can generate keywords that are more in line with user expressions, thereby improving the quality of the keywords, and then update the target index based on the keywords, so that the keywords can be more easily matched based on the target index in the future, thereby improving the conversion rate of keywords and improving the retrieval effect.

[0143] On the basis of the above-mentioned embodiment, the embodiment of the present application further provides a search method based on multimodal information, which can retrieve the target object based on the above-mentioned updated target index, and can retrieve the target object more accurately.

[0144] Reference Figure 6 , showing a step flow chart of an embodiment of a search method based on multimodal information of the present application.

[0145] Step 602, receiving keywords.

[0146] In various search pages, keywords input by users can be received and searches can be performed based on the keywords. For example, keywords can be entered to search for commodity objects in an e-commerce platform, and keywords can be entered to search for ticket information in a ticketing website.

[0147] Step 604, using the keyword to match the target index, determine at least one target object matched by the keyword, wherein the target index is added with keywords generated based on object association information of the target object, and the generation model for generating the keyword is fine-tuned based on the instruction word after pre-training is completed, and the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction word is generated based on the prompt word.

[0148] Based on the target index updated in the above steps, the received keyword is matched with the target index to determine at least one target object matched by the keyword. For example, based on the keyword, the inverted index is queried to query at least one keyword that matches in the inverted index to determine at least one target object corresponding to the at least one keyword that matches. In another example, the keyword is input into a semantic model to obtain a multidimensional vector representation, and the multidimensional vector representation is used to query the vectorized index to determine the multidimensional vector representation of at least one keyword that matches, and then determine the corresponding at least one target object. In another example, based on the keyword, a competitive index is queried to determine at least one corresponding target object based on the competition information.

[0149] Step 606: Generate search results based on the at least one target object.

[0150] The at least one target object is screened, for example, weighted calculation is performed based on the matching degree corresponding to each index to obtain a corresponding search score, the target object is screened based on the search score, and search results are generated using the screened target object.

[0151] Step 608: Feedback the search results.

[0152] The search results are fed back to the user terminal so that the user can query the search results and click on the desired target object.

[0153] Based on the target index, it is easier to match the keyword, improve the conversion rate of the keyword, and improve the retrieval effect.

[0154] In the embodiment of the present application, the target object may be any retrieval object containing multimodal information, such as product objects, advertising objects, news information, audio and video, etc.

[0155] On the basis of the above embodiment, taking the advertisement object as the target object as an example, in various websites such as e-commerce websites and ticketing websites, advertisement objects that meet the user's needs can be fed back during the search to improve the click-through rate.

[0156] Reference Figure 7 , showing a step flow chart of another embodiment of a model training method based on multimodal information of the present application.

[0157] Step 702, obtaining a pre-trained generative model, the generative model comprising: an image encoding unit, a graphic processing unit, a fully connected layer and a language model unit; freezing the parameters of the image encoding unit and the fully connected layer during the fine-tuning process.

[0158] Step 704: Obtain object association information of the instruction word generated based on the prompt word and the advertisement object.

[0159] The step of generating instruction words based on the prompt words is also included: constructing prompt words based on object description information of the advertisement object, and generating instruction words based on the prompt words.

[0160] Among them, the constructing of prompt words based on the object description information of the advertising object and generating instruction words based on the prompt words include: constructing prompt words based on the object description information of the advertising object and an instruction template; inputting the instruction words into a language model and outputting predicted keywords corresponding to the instruction words; filtering the predicted keywords to determine at least one filtered keyword; and generating instruction words corresponding to the prompt words based on the at least one filtered keyword and the instruction template.

[0161] Step 706, construct input data based on the instruction words, object description information, and image information, input the input data into the generation model, and output corresponding keywords.

[0162] Among them, the image information is input into the image encoding unit for encoding processing, and the corresponding image features are output; the instruction words and object description information are combined to construct the corresponding object instruction text, and the image features and object instruction text are input into the image and text processing unit, and the corresponding multimodal features are output; the multimodal features are input into the fully connected layer for processing, and the corresponding intermediate features are output; the intermediate features and instruction words are input into the language model unit for processing, and the keywords of the advertising object are output.

[0163] Step 708, calculating loss information using the keywords.

[0164] Step 710: Adjust the image and text processing unit and the language model unit of the generation model based on the loss information.

[0165] After completing fine-tuning of the generative model, keywords for advertising objects can be generated based on the trained generative model to update the target index. Figure 8 As shown:

[0166] Step 802: Obtain object association information of the advertisement object, wherein the object association information includes object description information and image information.

[0167] Step 804, construct input data based on the object description information and image information, input the input data into the generation model, and output corresponding keywords, wherein the generation model is generated by fine-tuning based on the instruction words after pre-training is completed, and the fine-tuning units include a graphic processing unit and a language model unit, and the instruction words are generated based on the prompt words.

[0168] Among them, constructing input data based on the object description information and image information, inputting the input data into the generation model, and outputting corresponding keywords, includes: inputting the object description information and image information into the generation model; determining multimodal features based on the graphic processing unit of the generation model; inputting the multimodal features into the language model unit, and outputting corresponding keywords.

[0169] The generation model also includes: an image encoding unit; the image and text processing unit based on the generation model determines the multimodal features, including: inputting the image information into the image encoding unit for encoding processing and outputting the corresponding image features; inputting the object description information and the image features into the image and text processing unit and outputting the corresponding multimodal features. The generation model also includes: a fully connected layer, the multimodal features are input into the language model unit and the corresponding keywords are output, including: inputting the multimodal features into the fully connected layer for processing and outputting the corresponding intermediate features; inputting the intermediate features into the language model unit for processing and outputting the corresponding keywords.

[0170] Step 806: construct index information based on the keyword, and add the index information to the target index, so that the advertisement object can be retrieved when searching based on the target index.

[0171] The target index includes at least one of the following: an inverted index, a vectorized index, and a competitive index;

[0172] The step of constructing index information based on the keyword and adding the index information to the target index includes at least one of the following steps: performing word segmentation on the keyword and adding the word segmentation information of the keyword to the inverted index; inputting the keyword into the semantic model and adding the output multidimensional vector representation to the vectorized index; combining the keyword and the corresponding target object and adding the combined information to the competitive index.

[0173] After completing the update of the target index, you can feedback the advertising objects that meet the user's needs when searching on the corresponding website or application (Application, APP). Taking the search for product objects on an e-commerce website as an example, you can add advertising objects to the search results to improve the click-through rate of the advertisement while meeting the user's needs. Fig. 9 As shown:

[0174] Step 902, receiving keywords.

[0175] Step 904, using the keyword matching index information, determine at least one commodity object matched by the keyword, the commodity object includes an advertising object, wherein the target index is added with keywords generated based on object association information of the advertising object, and the generation model for generating the keywords is generated by fine-tuning based on instruction words after pre-training is completed, the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction words are generated based on prompt words.

[0176] Step 906: Generate search results based on the at least one commodity object.

[0177] Step 908: Feedback the search results.

[0178] Generate keywords that match user expressions based on the generative model, so that the retrieval of advertising objects is more in line with user needs, express advertising information in a way that matches user semantic expressions, and improve advertising retrieval efficiency and click-through rate. The generative model can process multimodal information, so that it can comprehensively utilize various information about advertisements and products to improve matching effects and user experience.

[0179] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0180] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0181] On the basis of the above embodiment, this embodiment further provides a model training device based on multimodal information, which is applied to electronic devices such as server devices, and the device includes:

[0182] The acquisition module is used to acquire a pre-trained generation model, wherein the generation model includes: a graphic processing unit and a language model unit; and acquire the instruction words generated based on the prompt words and the object association information of the target object, wherein the object association information includes object description information and image information.

[0183] The model processing module is used to construct input data based on the instruction words, object description information, and image information, input the input data into the generation model, and output corresponding keywords.

[0184] The loss calculation module is used to calculate the loss information through the keywords.

[0185] A model adjustment module is used to adjust the image and text processing unit and the language model unit of the generation model based on the loss information.

[0186] In summary, by fine-tuning the pre-trained generation model, the instruction words generated based on the prompt words and the object association information of the target object can be obtained, the fine-tuning task of the generation model is guided by the instruction words, input data is constructed based on the instruction words, object description information, and image information, the input data is input into the generation model, the corresponding keywords are output, and then the loss information is calculated through the keywords. The graphic processing unit and the language model unit of the generation model are adjusted in a targeted manner based on the loss information to complete the fine-tuning of the generation model, improve the accuracy of the generation model, thereby improving the accuracy of the keywords, and then improving the accuracy of the search results and improving the user experience.

[0187] Wherein, the parameters of the image coding unit and the fully connected layer are frozen during the fine-tuning process.

[0188] It also includes: an instruction word generation module, which is used to construct prompt words based on the object description information of the target object and generate instruction words based on the prompt words.

[0189] The instruction word generation module is used to construct a prompt word based on the object description information of the target object and the instruction template; input the instruction word into the language model and output the predicted keyword corresponding to the instruction word; filter the predicted keyword to determine at least one filtered keyword; and generate an instruction word corresponding to the prompt word based on the at least one filtered keyword and the instruction template.

[0190] The generation model also includes: an image encoding unit and a fully connected layer; the model processing module is used to input the image information into the image encoding unit for encoding processing and output corresponding image features; combine the instruction words and object description information to construct corresponding object instruction texts, input the image features and object instruction texts into the image and text processing unit, and output corresponding multimodal features; input the multimodal features into the fully connected layer for processing and output corresponding intermediate features; input the intermediate features and instruction words into the language model unit for processing and output corresponding keywords.

[0191] The model adjustment module is used to adjust the parameters of the image and text processing unit based on the loss information, and to adjust the parameters of the language model unit based on the loss information.

[0192] The embodiment of the present application utilizes the capability of a large model and fine-tunes the model by means of efficient parameter fine-tuning, which can fit the data very well and obtain a generative model. Through fine-tuning, the keywords generated by the generative model have similar expression capabilities to the search keywords of the user, thereby improving the accuracy of the generative model.

[0193] On the basis of the above embodiment, this embodiment further provides an index construction device based on multimodal information, which is applied to electronic devices such as server devices, and the device includes:

[0194] The information acquisition module is used to acquire object association information of the target object, wherein the object association information includes object description information and image information.

[0195] A keyword matching module is used to construct input data based on the object description information and image information, input the input data into the generation model, and output corresponding keywords, wherein the generation model is generated by fine-tuning based on instruction words after pre-training is completed, and the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction words are generated based on prompt words.

[0196] The index updating module is used to construct the index information of the target object based on the keyword, and add the index information to the target index to perform search processing based on the target index.

[0197] In summary, object description information and image information are obtained to construct input data, the input data is input into the generation model, and the corresponding keywords are output. On the basis of improving the accuracy of the generation model through fine-tuning, the keywords generated are more accurate, so that the index information of the target object is constructed based on the keyword, and the index information is added to the target index. It can match the corresponding target object based on the keyword, thereby improving the user experience.

[0198] Among them, the keyword matching module is used to input the object description information and image information into the generation model; determine the multimodal features based on the graphic processing unit of the generation model; input the multimodal features into the language model unit, and output the corresponding keywords.

[0199] The generation model also includes: an image encoding unit and a fully connected layer; the keyword matching module is used to input the image information into the image encoding unit for encoding processing and output corresponding image features; input the object description information and the image features into the image and text processing unit and output corresponding multimodal features; input the multimodal features into the fully connected layer for processing and output corresponding intermediate features; input the intermediate features into the language model unit for processing and output corresponding keywords.

[0200] Among them, the target index includes at least one of the following: an inverted index, a vectorized index, and a competitive index; the index update module is used to perform word segmentation on the keyword and add the word segmentation information of the keyword to the inverted index; input the keyword into the semantic model and add the output multidimensional vector representation to the vectorized index; combine the keyword and the corresponding target object and add the combination information to the competitive index.

[0201] It also includes: a model fine-tuning module, which is used to obtain a pre-trained generation model, wherein the generation model includes: an image encoding unit, a graphic processing unit, a fully connected layer and a language model unit; obtain an instruction word generated based on a prompt word and object association information of a target object; construct input data based on the instruction word, object description information and image information, input the input data into the generation model, and output corresponding keywords; calculate loss information through the keywords; and adjust the graphic processing unit and the language model unit of the generation model based on the loss information.

[0202] Through fine-tuning, the generative model can generate keywords that are more in line with user expressions, thereby improving the quality of the keywords, and then update the target index based on the keywords, so that the keywords can be more easily matched based on the target index in the future, thereby improving the conversion rate of keywords and improving the retrieval effect.

[0203] On the basis of the above embodiment, this embodiment further provides a search device based on multimodal information, which is applied to electronic devices such as server devices, and the device includes:

[0204] The receiving module is used to receive keywords.

[0205] An object matching module is used to use the keyword to match the target index and determine at least one target object matched by the keyword, wherein the target index is added with keywords generated based on object association information of the target object, and the generation model for generating the keywords is fine-tuned based on instruction words after pre-training is completed. The fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction words are generated based on prompt words.

[0206] A result generation module is used to generate search results based on the at least one target object.

[0207] The feedback module is used to provide feedback on the search results.

[0208] Based on the target index, it is easier to match the keyword, improve the conversion rate of the keyword, and improve the retrieval effect.

[0209] Generate keywords that match user expressions based on the generative model, so that the retrieval of advertising objects is more in line with user needs, express advertising information in a way that matches user semantic expressions, and improve advertising retrieval efficiency and click-through rate. The generative model can process multimodal information, so that it can comprehensively utilize various information about advertisements and products to improve matching effects and user experience.

[0210] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0211] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the embodiment of the present application.

[0212] The embodiment of the present application also provides an electronic device, comprising: a processor, and a memory connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the present application. In the embodiment of the present application, the electronic device includes a server, a terminal device, and other devices.

[0213] The embodiment of the present application further discloses a computer program product, including a computer program / computer executable instructions, wherein the computer program / computer executable instructions, when executed by a processor in an electronic device, implement the method described in the embodiment of the present application.

[0214] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as a server (cluster), a terminal, etc. Fig.10 An exemplary apparatus 1000 that can be used to implement various embodiments described in this application is schematically illustrated.

[0215] For one embodiment, Fig.10 An exemplary apparatus 1000 is shown having one or more processors 1002, a control module (chip set) 1004 coupled to at least one of the (one or more) processors 1002, a memory 1006 coupled to the control module 1004, a non-volatile memory (NVM) / storage device 1008 coupled to the control module 1004, one or more input / output devices 1010 coupled to the control module 1004, and a network interface 1012 coupled to the control module 1004.

[0216] The processor 1002 may include one or more single-core or multi-core processors, and the processor 1002 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1000 can be used as a server, terminal, or other device described in the embodiments of the present application.

[0217] In some embodiments, the apparatus 1000 may include one or more computer-readable media (e.g., memory 1006 or NVM / storage device 1008) having instructions 1014 and one or more processors 1002 combined with the one or more computer-readable media and configured to execute the instructions 1014 to implement a module to perform the actions described in the present disclosure.

[0218] For one embodiment, the control module 1004 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1002 and / or any suitable device or component in communication with the control module 1004 .

[0219] The control module 1004 may include a memory controller module to provide an interface to the memory 1006. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0220] The memory 1006 may be used, for example, to load and store data and / or instructions 1014 for the device 1000. For one embodiment, the memory 1006 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1006 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0221] For one embodiment, control module 1004 may include one or more input / output controllers to provide an interface to NVM / storage device 1008 and input / output device(s) 1010 .

[0222] For example, NVM / storage 1008 may be used to store data and / or instructions 1014. NVM / storage 1008 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0223] NVM / storage device 1008 may include storage resources that are part of the device on which apparatus 1000 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1008 may be accessed via (one or more) input / output devices 1010 over a network.

[0224] (One or more) input / output devices 1010 may provide an interface for the apparatus 1000 to communicate with any other appropriate device, and the input / output device 1010 may include a communication component, an audio component, a sensor component, etc. The network interface 1012 may provide an interface for the apparatus 1000 to communicate through one or more networks, and the apparatus 1000 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as Bluetooth, WiFi, 2G, 3G, 4G, 5G, 6G, etc., or a combination thereof for wireless communication.

[0225] For one embodiment, at least one of the processor(s) 1002 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1004. For one embodiment, at least one of the processor(s) 1002 may be packaged together with the logic of one or more controllers of the control module 1004 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1002 may be integrated on the same die with the logic of one or more controllers of the control module 1004. For one embodiment, at least one of the processor(s) 1002 may be integrated on the same die with the logic of one or more controllers of the control module 1004 to form a system-on-chip (SoC).

[0226] In various embodiments, the apparatus 1000 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 1000 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0227] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0228] The embodiment of the present application also provides an electronic device, including: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor executes one or more methods as described in the embodiment of the present application. In the embodiment of the present application, the memory can store various data, such as target files, file and application association data, and user behavior data, so as to provide a data basis for various processing.

[0229] The embodiments of the present application also provide one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0230] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0231] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0232] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0233] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0234] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0235] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0236] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0237] The above is a detailed introduction to a model training method based on multimodal information, an index construction method based on multimodal information, a search method based on multimodal information, an electronic device, a computer-readable storage medium and a computer program product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method based on multimodal information, wherein: The method comprises: Obtaining a pre-trained generative model, wherein the generative model includes: a graphic and text processing unit and a language model unit; Acquire the instruction word generated based on the prompt word and the object association information of the target object, wherein the object association information includes object description information and image information; constructing input data based on the instruction words, object description information, and image information, inputting the input data into a generation model, and outputting corresponding keywords; Calculate loss information using the keywords; The graphic processing unit and the language model unit of the generation model are adjusted based on the loss information.

2. The method according to claim 1, wherein: The parameters of the image coding unit and the fully connected layers are frozen during the fine-tuning process.

3. The method according to claim 2, wherein: It also includes the steps of generating instruction words based on the prompt words: A prompt word is constructed based on the object description information of the target object, and an instruction word is generated based on the prompt word.

4. The method according to claim 3, wherein: The step of constructing a prompt word based on the object description information of the target object and generating an instruction word based on the prompt word includes: Construct prompt words based on the object description information of the target object and the instruction template; Inputting the instruction word into a language model, and outputting a predicted keyword corresponding to the instruction word; Screening the predicted keywords to determine at least one screened keyword; Based on the at least one filtered keyword and the instruction template, an instruction word corresponding to the prompt word is generated.

5. The method according to claim 2, wherein: The generation model also includes: an image encoding unit and a fully connected layer; The step of constructing input data based on the instruction words, object description information, and image information, inputting the input data into a generation model, and outputting corresponding keywords includes: Inputting the image information into an image encoding unit for encoding processing, and outputting corresponding image features; Combining the instruction word and the object description information to construct a corresponding object instruction text, inputting the image feature and the object instruction text into a graphic processing unit, and outputting a corresponding multimodal feature; Input the multimodal features into a fully connected layer for processing, and output corresponding intermediate features; The intermediate features and instruction words are input into the language model unit for processing, and the corresponding keywords are output.

6. The method according to claim 1, wherein: The image and text processing unit and the language model unit for adjusting the generation model based on the loss information include: Parameters of the image and text processing unit are adjusted based on the loss information, and parameters of the language model unit are adjusted based on the loss information.

7. A method for constructing an index based on multimodal information, wherein: The method comprises: Acquire object association information of a target object, wherein the object association information includes object description information and image information; Constructing input data based on the object description information and image information, inputting the input data into a generation model, and outputting corresponding keywords, wherein the generation model is fine-tuned based on instruction words after pre-training is completed, and the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction words are generated based on prompt words; The index information of the target object is constructed based on the keyword, and the index information is added to the target index to perform a search process based on the target index.

8. The method according to claim 7, wherein: The step of constructing input data based on the object description information and the image information, inputting the input data into the generation model, and outputting corresponding keywords includes: Inputting the object description information and image information into the generation model; Determining multimodal features based on a graphic processing unit of the generative model; The multimodal features are input into the language model unit, and corresponding keywords are output.

9. The method according to claim 8, wherein: The generation model further includes: an image encoding unit; the image and text processing unit based on the generation model determines the multimodal features, including: Inputting the image information into an image encoding unit for encoding processing, and outputting corresponding image features; The object description information and the image features are input into a graphics processing unit, and corresponding multimodal features are output.

10. The method according to claim 8, wherein: The generation model further includes: a fully connected layer, wherein the multimodal features are input into the language model unit and the corresponding keywords are output, including: Input the multimodal features into a fully connected layer for processing, and output corresponding intermediate features; The intermediate features are input into the language model unit for processing, and the corresponding keywords are output.

11. The method according to claim 7, wherein: The target index includes at least one of the following: an inverted index, a vectorized index, and a competitive index; The step of constructing the index information of the target object based on the keyword and adding the index information to the target index includes at least one of the following steps: Performing word segmentation processing on the keyword, and adding the word segmentation information of the keyword to the inverted index; Inputting the keywords into a semantic model, and adding the output multi-dimensional vector representation to the vectorized index; The keywords and corresponding target objects are combined, and the combined information is added to the competition index.

12. The method according to claim 7, wherein: It also includes steps for fine-tuning the pre-trained generative model: Obtain a pre-trained generative model, wherein the generative model includes: an image encoding unit, a graphic processing unit, a fully connected layer, and a language model unit; Obtaining object association information of the instruction word and the target object generated based on the prompt word; constructing input data based on the instruction words, object description information, and image information, inputting the input data into a generation model, and outputting corresponding keywords; Calculate loss information using the keywords; The graphic processing unit and the language model unit of the generation model are adjusted based on the loss information.

13. A search method based on multimodal information, comprising: Receive keywords; The keyword is matched with a target index to determine at least one target object matched by the keyword, wherein the target index is added with keywords generated based on object association information of the target object, and the generation model for generating the keyword is fine-tuned based on the instruction word after pre-training is completed, and the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction word is generated based on the prompt word; generating search results based on the at least one target object; Feedback on the search results.

14. A search method based on multimodal information, comprising: Receive keywords; Using the keyword matching index information, at least one commodity object matched by the keyword is determined, the commodity object includes an advertisement object, wherein the target index is added with a keyword generated based on object association information of the advertisement object, the generation model for generating the keyword is generated by fine-tuning based on the instruction word after pre-training is completed, the fine-tuning unit includes a graphic processing unit and a language model unit, and the instruction word is generated based on the prompt word; generating search results based on the at least one product object; Feedback on the search results.

15. An electronic device, wherein: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 14 when executed by a processor.

17. A computer program product comprising a computer program / computer executable instructions, wherein: When the computer program / computer executable instructions are executed by a processor in an electronic device, the method according to any one of claims 1 to 14 is implemented.

Images