Article retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides an article retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of data search. The method comprises the following steps: processing based on acquired retrieval information to obtain retrieval features; determining a target article in the plurality of articles based on the similarity between the retrieval feature and the text feature, the multimedia feature and the fusion feature corresponding to each article; the fusion feature is obtained by fusing the text feature and the multimedia feature; wherein the retrieval features, the text features, the multimedia features and the fusion features are obtained based on preset model processing; the preset model is obtained by training based on the matching degree between the sample retrieval information and the text information of the sample article, the matching degree between the text information of the sample article and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information. According to the method, the matching degree of the target article and the retrieval information is improved, so that the target article is better matched with the real demand of the user.

Description

Technical Field

[0001] The present application relates to the field of data search technology, and in particular to an object retrieval method, device, electronic device and storage medium. Background Art

[0002] In the related art, the traditional method of item retrieval on e-commerce platforms is usually based on text search. Users can enter keywords or phrases in the search box, and the system will automatically match relevant items and return search results. The server's search algorithm will segment and understand the search keywords, match the text information such as item titles, descriptions, and categories, and return items with a high degree of match with the search keywords to the user end. However, traditional text search methods have some limitations. There will be errors in the processing of synonyms and ambiguous words, resulting in poor matching between the returned items determined by the server and the search keywords, and the returned items do not match the actual needs of users. Summary of the invention

[0003] An item retrieval method, device, electronic device and storage medium provided in the embodiments of the present application can improve the matching degree between the returned items determined by the server and the search keywords, and improve the matching degree between the returned items and the actual needs of the user.

[0004] The technical solution of this application is implemented as follows:

[0005] The present application embodiment provides an item retrieval method, including:

[0006] Obtaining retrieval features based on the acquired retrieval information;

[0007] Based on the similarity between the search feature and the text feature, multimedia feature and fusion feature corresponding to each item, a target item is determined from the multiple items; the fusion feature is obtained by fusion of the text feature and the multimedia feature;

[0008] Among them, the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained based on the preset model; the preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information.

[0009] In the above solution, the step of determining a target item from a plurality of items based on the similarity between the search feature and the text feature, multimedia feature and fusion feature corresponding to each item includes:

[0010] Based on the similarity between the search feature and the text feature, the multimedia feature and the fusion feature corresponding to each of the items, sorting the multiple items to obtain an item sequence;

[0011] The target item is determined in the item sequence.

[0012] In the above solution, the step of sorting the plurality of items to obtain an item sequence based on the similarity between the search feature and the text feature, the multimedia feature and the fusion feature corresponding to each item includes:

[0013] Based on a first similarity between the search feature and the text feature of each of the items, sorting the plurality of items to obtain a first item sequence; each of the items in the first item sequence corresponds to the first similarity;

[0014] Based on the second similarity between the search feature and the multimedia feature of each of the items, sorting the plurality of items to obtain a second item sequence; each of the items in the first item sequence corresponds to the second similarity;

[0015] Based on the third similarity between the search feature and the fusion feature of each of the items, the plurality of items are sorted to obtain a third item sequence; each item in the third item sequence corresponds to the third similarity.

[0016] In the above solution, determining the target item in the item sequence includes:

[0017] Based on the first similarity of each of the items in the first item sequence, the second similarity of each of the items in the second item sequence, and the third similarity of each of the items in the third item sequence, deduplication and fusion are performed on the items in the first item sequence, the second item sequence, and the third item sequence to obtain a fusion sequence; and the multiple items in the fusion sequence are arranged in descending order of similarity;

[0018] The first N items in the fusion sequence are determined to be the target items; N is an integer.

[0019] In the above solution, before obtaining the search features based on the acquired search information, the method further includes:

[0020] The text information and multimedia information of each of the items are processed based on the preset model to obtain the text features, multimedia features and fusion features corresponding to each of the items.

[0021] In the above scheme, the preset model includes: at least one of a first text model, a second text model, a multimedia feature model and a fusion model; the first text model is used to process the search information to obtain the search feature;

[0022] The processing of the text information and multimedia information of each of the items based on the preset model to obtain the text features, multimedia features and fusion features corresponding to each of the items includes:

[0023] Processing the text information of each of the items based on the second text model to obtain the text feature;

[0024] Processing the multimedia information of each of the items based on the multimedia feature model to obtain the multimedia feature;

[0025] The text feature and the multimedia feature are fused based on the fusion feature to obtain the fusion feature.

[0026] The present application also provides a model training method, including:

[0027] Obtain sample retrieval information, as well as text information and multimedia information of sample items;

[0028] The initial preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information, and the training is stopped until the initial preset model meets the training conditions to obtain the preset model.

[0029] In the above scheme, the initial preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information until the initial preset model meets the training conditions, and the preset model is obtained, including:

[0030] Processing the sample retrieval information based on the initial preset model to obtain a sample retrieval feature, processing the text information based on the initial preset model to obtain a sample text feature, processing the multimedia information based on the initial preset model to obtain a sample multimedia feature, and fusing the sample text feature and the sample multimedia feature based on the initial preset model to obtain a sample fusion feature;

[0031] The initial preset model is trained based on a first contrast loss between the sample retrieval feature and the sample text feature, a second contrast loss between the sample retrieval feature and the sample fusion feature, and a third contrast loss between the sample text feature and the sample multimedia feature, and the training is stopped until the initial preset model meets the training conditions to obtain the preset model.

[0032] In the above solution, the initial preset model includes: at least one of a first initial text model, a second initial text model, an initial multimedia feature model and an initial fusion model;

[0033] The processing of the sample retrieval information based on the initial preset model to obtain a sample retrieval feature, the processing of the text information based on the initial preset model to obtain a sample text feature, the processing of the multimedia information based on the initial preset model to obtain a sample multimedia feature, and the fusion of the sample text feature and the sample multimedia feature based on the initial preset model to obtain a sample fusion feature include:

[0034] Processing the sample retrieval information based on the first initial text model to obtain the sample retrieval feature;

[0035] Processing the text information based on the second initial text model to obtain the sample text features;

[0036] Processing the multimedia information based on the initial multimedia feature model to obtain the sample multimedia features;

[0037] The sample text features and the sample multimedia features are processed based on the initial fusion model to obtain the sample fusion features.

[0038] In the above scheme, at least one of the first text model, the second text model, the multimedia feature model and the fusion model;

[0039] The initial preset model is trained based on a first contrast loss between the sample retrieval feature and the sample text feature, a second contrast loss between the sample retrieval feature and the sample fusion feature, and a third contrast loss between the sample text feature and the sample multimedia feature until the initial preset model meets the training condition, thereby obtaining the preset model, including:

[0040] The first initial text model is adjusted based on the first contrast loss and the second contrast loss, the second initial text model is adjusted based on the first contrast loss and the third contrast loss, the initial fusion model is adjusted based on the second contrast loss, and the initial multimedia feature model is adjusted based on the third contrast loss until the first initial text model, the second initial text model, the initial multimedia feature model, and the initial fusion model all reach a predetermined number of training times, thereby obtaining the first text model, the second text model, the multimedia feature model, and the fusion model.

[0041] The present application also provides an object retrieval device, including:

[0042] An information processing unit, used for processing the acquired search information to obtain search features;

[0043] A determination unit, configured to determine a target item from a plurality of items based on similarities between the search feature and text features, multimedia features, and fusion features corresponding to each item; the fusion features are obtained by fusion of the text features and the multimedia features;

[0044] Among them, the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained based on the preset model; the preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information.

[0045] The present application also provides a model training device, including:

[0046] An information acquisition unit, used to acquire sample retrieval information, as well as text information and multimedia information of sample items;

[0047] A training unit is used to train an initial preset model based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information, and stop when the initial preset model meets the training conditions to obtain a preset model.

[0048] An embodiment of the present application also provides an electronic device, including a first memory and a first processor, wherein the first memory stores a computer program that can be run on the first processor, and the sample retrieval information processor implements the steps in the above-mentioned item retrieval method when executing the computer program.

[0049] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a first processor, the steps in the above-mentioned item retrieval method are implemented.

[0050] An embodiment of the present application also provides an electronic device, including a second memory and a second processor, wherein the second memory stores a computer program that can be executed on the second processor, and the sample retrieval information processor implements the steps in the above-mentioned model training method when executing the computer program.

[0051] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a second processor, the steps in the above-mentioned model training method are implemented.

[0052] In the embodiment of the present application, the retrieval feature is obtained based on the acquired retrieval information processing; the target item is determined from multiple items based on the similarity between the retrieval feature and the text feature, multimedia feature and fusion feature corresponding to each item; the fusion feature is obtained by fusion of the text feature and the multimedia feature; wherein the retrieval feature, text feature, multimedia feature and fusion feature are obtained based on the preset model processing; the preset model is obtained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information. Since the present application not only considers the similarity between the retrieval feature of the retrieval information and the text feature of the item in the process of determining the target item, but also considers the multimedia feature and fusion feature of the item, starting from the multi-dimensional feature, the matching degree between the target item and the retrieval information is improved, and since the preset model training process is also based on the multi-dimensional feature of the sample item, the preset model can more fully learn the matching degree between the sample retrieval information and the text information and multimedia information of the sample item, thereby also improving the matching degree between the target item and the retrieval information, so that the target item sent to the client is more in line with the real needs of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0054] Figure 2 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0055] Figure 3 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0056] Figure 4 An optional effect diagram of the object retrieval method provided in an embodiment of the present application;

[0057] Figure 5 An optional flowchart of the model training method provided in the embodiment of the present application;

[0058] Figure 6 An optional flowchart of the model training method provided in the embodiment of the present application;

[0059] Figure 7 An optional flowchart of the model training method provided in the embodiment of the present application;

[0060] Figure 8 An optional effect diagram of the model training method provided in the embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the structure of an object retrieval device provided in an embodiment of the present application;

[0062] Fig.10 A hardware entity diagram of an electronic device provided in an embodiment of the present application Figure 1 ;

[0063] Fig.11 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0064] Fig.12 A hardware entity diagram of an electronic device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0066] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0067] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] The traditional method of searching for items based on search information on e-commerce platforms has the following disadvantages:

[0070] 1. Unable to search non-text content: Traditional text search methods can only search and match text information such as item titles, descriptions, and categories, but cannot search non-text content such as pictures and videos, resulting in a mismatch between the search results and the search information.

[0071] 2. Inaccurate handling of synonyms and ambiguous words: Due to the existence of synonyms and ambiguous words, the search information entered by the user may have multiple interpretations and intentions, resulting in a mismatch between the search results and the search information.

[0072] 3. Inaccurate understanding of user intent: Traditional methods of retrieving information for item retrieval can only match keywords and text information, and cannot deeply understand the user's search intent, resulting in the returned search results not fully matching the user's actual needs.

[0073] This application embodiment provides an item retrieval method, see Figure 1 , is an optional flow chart of the item retrieval method provided in the embodiment of the present application, which will be combined with Figure 1 The steps shown are explained.

[0074] S101, obtaining search features based on the acquired search information.

[0075] In the embodiment of the present application, the server may receive the search information sent by the client, analyze the text information of the search information, and obtain the search feature.

[0076] In the embodiment of the present application, the user can enter the keywords (search information) of the desired items in the search interface of the client, the client sends the keywords to the server, and the server performs semantic analysis on the keywords to obtain search features.

[0077] In an embodiment of the present application, the server may also use a text model to analyze the search information to obtain corresponding search features.

[0078] S102. Determine a target item from a plurality of items based on the similarity between the retrieval feature and the text feature, multimedia feature and fusion feature corresponding to each item; the fusion feature is obtained by fusion of the text feature and the multimedia feature; wherein the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained by processing based on a preset model; the preset model is trained based on the matching degree between sample retrieval information and the text information of a sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information.

[0079] In an embodiment of the present application, the server stores text information and multimedia information of each of the multiple items. The server can process the text information of each item based on a preset model to obtain text features, process the multimedia information of each item based on a preset model to obtain multimedia features, and process the text features and multimedia features based on a preset model to obtain fusion features. The server can determine the similarity between the retrieval features and the text features, multimedia features, and fusion features of each item. Based on the similarity between the retrieval features and each item, the N items with the greatest similarity among the multiple items are determined as target items. N is an integer. Among them, the multimedia information includes: at least one of the image information, video information, and audio information of the corresponding item. The text information can be text information such as the corresponding item title, description, and classification.

[0080] Among them, the preset model is trained based on the contrast loss between sample retrieval information and text information of sample items, the contrast loss between sample retrieval information and fusion information of text information and multimedia information of sample items, and the contrast loss between text information and multimedia information of sample items.

[0081] Among them, contrast loss is a binary classification loss function, which is usually used to measure the matching degree between two samples. Its basic idea is to take the feature vectors of two samples as input and determine whether they belong to the same category by calculating their similarity. If the two samples belong to the same category, the contrast loss is 0. If they do not belong to the same category, the contrast loss is a positive number. The priority of contrast loss is that it is simple to calculate and is applicable to large-scale data sets.

[0082] In some other embodiments, the server may sort the multiple items based on the similarity between the search feature and the text feature, multimedia feature and fusion feature corresponding to each item. The items ranked higher have a higher matching degree with the search information. The server may determine the first N items as the target items among the sorted multiple items.

[0083] In the embodiment of the present application, after determining the target item, the server can send at least one of the text information, image information, video information and audio information corresponding to the target item to the client. The client receives the text information, image information, video information and audio information of the target item and renders and displays them on the current page using a rendering component.

[0084] In an embodiment of the present application, a retrieval feature is obtained based on the acquired retrieval information; a target item is determined from multiple items based on the similarity between the retrieval feature and the text feature, multimedia feature and fusion feature corresponding to each item; the fusion feature is obtained by fusion of the text feature and the multimedia feature; wherein the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained by processing based on a preset model; the preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information. Since, in the process of determining the target item, the present application not only considers the similarity between the retrieval features of the retrieval information and the text features of the item, but also considers the multimedia features and fusion features of the item, starting from the multi-dimensional features, the matching degree between the target item and the retrieval information is improved. Moreover, since the preset model training process is also based on the multi-dimensional features of the sample item, the preset model can more fully learn the matching degree between the sample retrieval information and the text information and multimedia information of the sample item, thereby improving the matching degree between the target item and the retrieval information, so that the target item sent to the client is more in line with the user's real needs.

[0085] In some embodiments, see Figure 2 , Figure 2 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 1 The illustrated S101 to S102 may also include S103 to S106 , which will be described in conjunction with each step.

[0086] S103: Process the text information and multimedia information of each of the items based on the preset model to obtain the text features, multimedia features and fusion features corresponding to each of the items.

[0087] In the embodiment of the present application, after completing the training of the preset model, the server obtains: a first text model, a second text model, a multimedia feature model and a fusion model. The text information of each of the items can be processed based on the second text model to obtain the text feature; the multimedia information of each of the items can be processed based on the multimedia feature model to obtain the multimedia feature; and the text feature and the multimedia feature can be fused based on the fusion feature to obtain the fusion feature.

[0088] In the embodiment of the present application, the server can process the text information of each of the items based on any one of a word vector model, a deep neural network (DNN) model, and a convolutional neural network (CNN) model to obtain corresponding text features. The multimedia information of each item can be processed based on at least one of a CNN model, an LSTM model, a transforer model, and a VGG model to obtain corresponding multimedia features. The multimedia features and sample features of each item can be fused based on several sublayers in the transforer model to obtain fused features.

[0089] S104: Process the search information based on the first text model to obtain the search feature.

[0090] In an embodiment of the present application, the client sends corresponding search information to the server, and the server processes the search information based on the first text model to obtain search features.

[0091] In an embodiment of the present application, the server can process the retrieval information based on any one of a word vector model, a deep neural network (DNN) and a convolutional neural network (CNN) to obtain corresponding retrieval features.

[0092] S105 , sorting the plurality of items to obtain an item sequence based on the similarity between the search feature and the text feature, multimedia feature and fusion feature corresponding to each item.

[0093] In the embodiment of the present application, the server can calculate the similarity between the search feature and the text feature, multimedia feature and fusion feature of each item to obtain the similarity between the search feature and each item. The server sorts multiple items based on the similarity of each item to obtain an item sequence.

[0094] S106: Determine the target item in the item sequence.

[0095] In the embodiment of the present application, since the similarities corresponding to multiple items in the item sequence can increase or decrease, the server can determine the N items with the greatest similarity in the item sequence as the target items.

[0096] In the embodiment of the present application, in the process of determining the target object, not only the similarity between the retrieval features of the retrieval information and the text features of the object is considered, but also the similarity between the multimedia features and fusion features of the object and the retrieval features is considered. Starting from the multi-dimensional features, the analysis of the item features is more comprehensive, thereby improving the matching degree between the target object and the retrieval information.

[0097] In some embodiments, see Figure 3 , Figure 3 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 2 The illustrated S105 to S106 may also be implemented by including S107 to S111, which will be described in conjunction with each step.

[0098] S107: Sort the plurality of items based on a first similarity between the search feature and the text feature of each item to obtain a first item sequence.

[0099] In an embodiment of the present application, the server may determine a first similarity between the search feature and the text feature of each item. Each item corresponds to a first similarity, and the server may sort each item in descending order of the first similarity based on the first similarity corresponding to each item to obtain a first item sequence. Each item in the first item sequence corresponds to the first similarity.

[0100] The server can calculate the first similarity cosθ between the search feature and the text feature of each of the items through formula (1).

[0101]

[0102] The retrieval feature vector is a, and the text feature vector is b. The server can multiply a and b, and then multiply the modulus of a and the modulus of b. The first similarity cosθ is obtained.

[0103] S108: Sort the plurality of items based on a second similarity between the search feature and the multimedia feature of each item to obtain a second item sequence.

[0104] In the embodiment of the present application, the server may determine the second similarity between the search feature and the multimedia feature of each item. Each item corresponds to the second similarity, and the server may sort each item in descending order of the second similarity based on the second similarity corresponding to each item to obtain a second item sequence. Each item in the second item sequence corresponds to the second similarity.

[0105] The server may determine the second similarity between the search feature and the text feature of each item based on a cosine distance algorithm.

[0106] In an embodiment of the present application, the multimedia features may include: image features, video features, and audio features corresponding to each item. The server may determine the second similarity between the retrieval feature and any one of the image features, video features, and audio features of each item. The server may sort each item in descending order of the second similarity of the image features based on the second similarity of the image features corresponding to each item, and obtain a second item sequence 1. The server may sort each item in descending order of the second similarity of the video features based on the second similarity of the video features corresponding to each item, and obtain a second item sequence 2. The server may sort each item in descending order of the second similarity of the audio features based on the second similarity of the audio features corresponding to each item, and obtain a second item sequence 3.

[0107] S109: Sort the plurality of items based on a third similarity between the search feature and the fusion feature of each item to obtain a third item sequence.

[0108] In the embodiment of the present application, the server may determine the third similarity between the search feature and the text feature time of each item. Each item corresponds to the third similarity, and the server may sort each item in descending order of the third similarity based on the third similarity corresponding to each item to obtain a third item sequence. Each item in the third item sequence corresponds to the third similarity. Each item in the third item sequence corresponds to the third similarity;

[0109] The server may determine the third similarity between the search feature and the fusion feature of each item based on a cosine distance algorithm.

[0110] S110: Based on the first similarity of each item in the first item sequence, the second similarity of each item in the second item sequence, and the third similarity of each item in the third item sequence, perform deduplication fusion on items in the first item sequence, the second item sequence, and the third item sequence to obtain a fused sequence.

[0111] In the embodiment of the present application, the server may reorder the items in the first item sequence, the second item sequence, and the third item sequence based on the first similarity, the second similarity, and the third similarity corresponding to the items in the first item sequence, the second item sequence, and the third item sequence. During the reordering process, when the fused sequence includes at least two identical items, the identical items in the latter order may be deleted, thereby obtaining a fused sequence without duplicate items.

[0112] In the embodiment of the present application, the server may reorder the items in the first item sequence, the second item sequence 1, the second item sequence 2, the second item sequence 3, and the third item sequence based on the first similarity, the second similarity, and the third similarity corresponding to the items in the first item sequence, the second item sequence 1, the second item sequence 2, the second item sequence 3, and the third item sequence. During the reordering process, when the fused sequence includes at least two identical items, the identical items in the latter order may be deleted, thereby obtaining a fused sequence without duplicate items.

[0113] S111. Determine the first N items in the fusion sequence as the target items.

[0114] In the embodiment of the present application, the multiple items in the fusion sequence are arranged in descending order of similarity; the server can determine that the first N items in the fusion sequence are the target items.

[0115] Exemplary, combined Figure 4 ,The multimodal adaptive item retrieval process mainly consists of the following key steps:

[0116] (1) The server obtains the search information, which can be encoded through the first text model to obtain its search features.

[0117] (2) For each item, the second text model, multimedia feature model and fusion model are used to extract features to obtain corresponding text features, multimedia features and fusion features; wherein the text features and multimedia features can be used as unimodal features of the item. The text features, multimedia features and fusion features constitute an item feature library.

[0118] (3) For each of the text features, multimedia features, and fusion features of each item, determine the similarity between the feature and the search feature, and sort all items from high to low according to the similarity between the text features, multimedia features, and fusion features and the search feature, respectively, to obtain a first item sequence corresponding to the text features, a second item sequence corresponding to the multimedia features, and a third item sequence corresponding to the fusion features. The first item sequence, the second item sequence, and the third item sequence constitute the preliminary search results.

[0119] (4) The items in the first item sequence, the second item sequence, and the third item sequence are merged and sorted. Specifically, items are selected from the three sequences in descending order of similarity to form a new fused sequence. If an item already exists in the fused sequence, the item will no longer be added.

[0120] (5) Determine the target item with the greatest similarity from the fusion sequence obtained in (4), and return the target item to the user as the final retrieval result.

[0121] In the embodiment of the present application, the purpose of item retrieval based on multimodal features is to improve the accuracy of item search and user experience. The item retrieval method based on multimodal technology can simultaneously utilize multiple different types of data, such as images, text, voice, etc., to improve the accuracy of search results and user experience. In addition, multimodal technology can also use natural language processing technology to more accurately understand the user's search intent to obtain more accurate search results. Therefore, item retrieval based on multimodal technology aims to improve the accuracy of item search and user experience.

[0122] This application embodiment also provides a model training method, please refer to Figure 5 , each step will be explained.

[0123] S201, obtaining sample search information, as well as text information and multimedia information of sample items.

[0124] In the embodiment of the present application, the server can determine the sample retrieval information of each preset model training in the historical retrieval information library. The server can determine any one of the stored multiple items as a sample item. And extract the text information and multimedia information of the sample item.

[0125] The historical search library can store the search information sent by various clients within a certain period of time.

[0126] S202. Train the initial preset model based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information, until the initial preset model meets the training conditions, thereby obtaining the preset model.

[0127] In an embodiment of the present application, the initial preset model can be obtained by combining a text model, an image model, a video model, an audio model and a fusion model. The server can train the text model in the initial preset model based on the contrast loss between the sample retrieval information and the text information of the sample item. The text model, image model, video model and audio model in the initial preset model can be trained based on the contrast loss between the text information of the sample item and the multimedia information. The fusion model in the initial preset model is trained based on the contrast loss between the sample retrieval information and the fusion information of the text information and the multimedia information.

[0128] In the embodiment of the present application, since the contrast loss between the sample retrieval information and the text information of the sample item, the contrast loss between the text information of the sample item and the multimedia information, and the contrast loss between the sample retrieval information and the fusion information of the text information and multimedia information are taken into account in the process of training the preset model, the preset model can more fully learn the degree of matching between the sample retrieval information and the text information and multimedia information of the sample item, thereby improving the matching degree between the target item and the retrieval information, so that the target item sent to the client is more in line with the user's actual needs.

[0129] In some embodiments, see Figure 6 , Figure 6 An optional flowchart of the model training method provided in the embodiment of the present application is shown in FIG. Figure 5 The illustrated S202 can be implemented through S203 to S204 , which will be described in conjunction with each step.

[0130] S203, based on the initial preset model, the sample retrieval information is processed to obtain sample retrieval features, based on the initial preset model, the text information is processed to obtain sample text features, based on the initial preset model, the multimedia information is processed to obtain sample multimedia features, and based on the initial preset model, the sample text features and the sample multimedia features are fused to obtain sample fusion features.

[0131] In the embodiment of the present application, the initial preset model can be obtained by combining a text model, an image model, a video model, an audio model and a fusion model. The server can process the sample retrieval information based on the text model in the initial preset model to obtain a sample retrieval feature. The text information of the sample item is processed based on the text model in the initial preset model to obtain a sample text feature, and the multimedia information of the sample item is processed based on at least one of the image model, video model and audio model in the initial preset model to obtain a sample multimedia feature. The sample text feature and the sample multimedia feature are fused based on the fusion model in the initial preset model to obtain a sample fusion feature.

[0132] The fusion model fusing the sample text feature and the sample multimedia feature to obtain the sample fusion feature may include: the fusion model weighting the sample text feature and the sample multimedia feature to obtain the sample fusion feature.

[0133] S204. Training the initial preset model based on a first contrast loss between the sample retrieval feature and the sample text feature, a second contrast loss between the sample retrieval feature and the sample fusion feature, and a third contrast loss between the sample text feature and the sample multimedia feature, until the initial preset model meets the training condition, thereby obtaining the preset model.

[0134] In the embodiment of the present application, the server can adjust the model parameters of the initial preset model based on the first contrast loss, the second contrast loss, and the third contrast loss during the first training of the initial preset model, and complete the first training of the initial preset model. Then, the second training of the initial preset model is performed until the initial preset model completes the predetermined number of trainings, and the preset model is obtained.

[0135] In an embodiment of the present application, since the first contrast loss between the sample retrieval feature and the sample text feature, the second contrast loss between the sample retrieval feature and the sample fusion feature, and the third contrast loss between the sample text feature and the sample multimedia feature are taken into account during the training of the preset model, the preset model can more fully learn the degree of matching between the sample retrieval information and the text information and multimedia information of the sample item, thereby improving the matching degree between the target item and the retrieval information, so that the target item sent to the client is more in line with the user's actual needs.

[0136] In some embodiments, see Figure 7 , Figure 7 An optional flowchart of the model training method provided in the embodiment of the present application is shown in FIG. Figure 6 S203 to S204 shown may also be implemented through S205 to S209, which will be described in conjunction with each step.

[0137] S205: Process the sample retrieval information based on the first initial text model to obtain the sample retrieval feature.

[0138] In an embodiment of the present application, the initial preset model includes: at least one of a first initial text model, a second initial text model, an initial multimedia feature model and an initial fusion model; the server can process the sample retrieval information based on the first initial text model to obtain a corresponding sample retrieval feature vector.

[0139] Among them, the first initial text model may include: any one of a word vector model, a deep neural network (Deep Neural Network, DNN) and a convolutional neural network (Convolutional Neural Network, CNN).

[0140] S206: Process the text information based on the second initial text model to obtain the sample text features.

[0141] In an embodiment of the present application, the server may process the text information of the sample item based on the second initial text model to obtain a sample text feature vector corresponding to the sample item.

[0142] The second initial text model may also include: any one of a word vector model, a deep neural network (DNN) model and a convolutional neural network (CNN) model.

[0143] S207: Process the multimedia information based on the initial multimedia feature model to obtain the sample multimedia features.

[0144] In an embodiment of the present application, the server can process the image information, video information and audio information corresponding to the sample item based on the initial multimedia feature model to obtain a sample multimedia feature vector corresponding to the sample item.

[0145] Among them, the initial multimedia feature vector may include: a convolutional neural network (CNN) model, a long short-term memory network (LSTM) model, a transforer model, and a deep learning (Oxford Visual Geometry Group, VGG) model.

[0146] In the embodiment of the present application, when the multimedia information of the sample item is image information, the corresponding multimedia feature vector can be obtained based on the initial CNN model. When the multimedia information of the sample item is video information, the corresponding multimedia feature vector can be obtained based on the initial LSTM model or the initial Ransforer model. When the multimedia information of the sample item is audio information, the corresponding multimedia feature vector can be obtained based on the initial VGG model.

[0147] S208: Process the sample text features and the sample multimedia features based on the initial fusion model to obtain the sample fusion features.

[0148] In an embodiment of the present application, the server may fuse the sample text features and the sample multimedia features based on the initial fusion model of the sample item to obtain the sample fusion features corresponding to the sample item.

[0149] Among them, the initial sample fusion features may include: several sub-layers in the transformer model.

[0150] S209, adjusting the first initial text model based on the first contrast loss and the second contrast loss, adjusting the second initial text model based on the first contrast loss and the third contrast loss, adjusting the initial fusion model based on the second contrast loss, and adjusting the initial multimedia feature model based on the third contrast loss, until the first initial text model, the second initial text model, the initial multimedia feature model, and the initial fusion model all reach a predetermined number of training times, thereby obtaining the first text model, the second text model, the multimedia feature model, and the fusion model.

[0151] In an embodiment of the present application, the preset model includes: at least one of a first text model, a second text model, a multimedia feature model and a fusion model; when the server performs one training of the initial preset model, the server can adjust the model parameters of the first initial text model based on the first contrast loss and the second contrast loss, adjust the model parameters of the second initial text model based on the first contrast loss and the third contrast loss, adjust the model parameters of the initial fusion model based on the second contrast loss, and adjust the model parameters of the initial multimedia feature model based on the third contrast loss, until the training times of the first initial text model, the second initial text model, the initial multimedia feature model and the initial fusion model all reach the predetermined training times, and the first text model, the second text model, the multimedia feature model and the fusion model are obtained.

[0152] Among them, the purpose of the first contrast loss is to match the sample retrieval information with the text information of the sample item, the purpose of the second contrast loss is to match the sample retrieval information with the fusion information of the text information and multimedia information of the sample item, and the purpose of the third contrast loss is to align the text information of the sample item with each multimedia information so that the fusion model can perform effective feature fusion.

[0153] Exemplary, combined Figure 8 The preset model consists of a first initial text model, a second initial text model on the item side, an initial multimedia feature model, and an initial fusion model. The first initial text model is responsible for encoding the sample query words entered by the user to form sample retrieval features. The second initial text model is responsible for encoding the text information contained in the item, such as item title, specifications, brand, category, etc., to form the sample text features of the item. The initial multimedia feature model is responsible for encoding the image information, video information, and audio information contained in the object to form the multimedia features of the object sample. The initial fusion model on the item side is responsible for fusing the sample text features and sample multimedia features of the item and outputting the fusion features. The training of the initial preset model is mainly completed by optimizing the following three sets of contrast losses, namely, sample retrieval features Sample text features on the item side Sample retrieval features Fusion features with item-side samples And sample retrieval features With sample multimedia features in and The purpose of the contrast loss is to match the sample retrieval information with the text information of the sample item. and The purpose of the contrast loss is to match the sample retrieval information with the fusion information of the sample items. and The purpose of the contrast loss is to align the information of different modalities on the sample item side so that the fusion model can perform effective feature fusion. Through such model training, four models can be finally obtained: the first text model, the second text model, the multimedia feature model, and the fusion model.

[0154] In an embodiment of the present application, the first initial text model is adjusted based on the first contrast loss and the second contrast loss, the second initial text model is adjusted based on the first contrast loss and the third contrast loss, the initial fusion model is adjusted based on the second contrast loss, and the initial multimedia feature model is adjusted based on the third contrast loss. After multiple training of the first initial text model, the second initial text model, the initial fusion model and the initial multimedia feature model, a preset model is obtained by combining them. During the training process of the preset model, the various modal information of the sample retrieval information and the sample item, as well as the contrast loss between the fused modal information are fully considered, so that the preset model can more fully learn the degree of matching between the sample retrieval information and the text information and multimedia information of the sample item, thereby improving the matching degree between the target item and the retrieval information, so that the target item sent to the client is more matched with the user's actual request.

[0155] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of an item retrieval device provided in an embodiment of the present application.

[0156] The embodiment of the present application further provides an item retrieval device 600 , comprising: an information processing unit 601 and a determination unit 602 .

[0157] An information processing unit 601 is used to obtain a search feature based on the acquired search information;

[0158] A determination unit 602, configured to determine a target item from a plurality of items based on similarities between the search feature and text features, multimedia features, and fusion features corresponding to each item; the fusion features are obtained by fusion of the text features and the multimedia features;

[0159] Among them, the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained based on the preset model; the preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and multimedia information of the sample item.

[0160] In the embodiment of the present application, the determining unit 602 in the item retrieval device 600 is used to sort the multiple items to obtain an item sequence based on the similarity between the retrieval feature and the text feature, the multimedia feature and the fusion feature corresponding to each item;

[0161] The target item is determined in the item sequence.

[0162] In the embodiment of the present application, the determining unit 602 in the item retrieval device 600 is used to sort the multiple items to obtain a first item sequence based on the first similarity between the retrieval feature and the text feature of each item; each item in the first item sequence corresponds to the first similarity;

[0163] Based on the second similarity between the search feature and the multimedia feature of each of the items, sorting the plurality of items to obtain a second item sequence; each of the items in the first item sequence corresponds to the second similarity;

[0164] Based on the third similarity between the search feature and the fusion feature of each of the items, the plurality of items are sorted to obtain a third item sequence; each item in the third item sequence corresponds to the third similarity.

[0165] In the embodiment of the present application, the determining unit 602 in the item retrieval device 600 is used to remove duplicates and fuse the items in the first item sequence, the second item sequence, and the third item sequence based on the first similarity of each item in the first item sequence, the second similarity of each item in the second item sequence, and the third similarity of each item in the third item sequence to obtain a fused sequence; the multiple items in the fused sequence are arranged in descending order of similarity;

[0166] The first N items in the fusion sequence are determined to be the target items; N is an integer.

[0167] In the embodiment of the present application, the information processing unit 602 in the item retrieval device 600 is used to process the text information and multimedia information of each item based on the preset model to obtain the text features, multimedia features and fusion features corresponding to each item.

[0168] In an embodiment of the present application, the preset model includes: at least one of a first text model, a second text model, a multimedia feature model and a fusion model; the first text model is used to process the retrieval information to obtain the retrieval feature; the information processing unit 602 in the item retrieval device 600 is used to process the text information of each item based on the second text model to obtain the text feature; based on the multimedia feature model, the multimedia information of each item is processed to obtain the multimedia feature; based on the fusion feature, the text feature and the multimedia feature are fused to obtain the fusion feature.

[0169] It should be noted that in the embodiments of the present application, if the above-mentioned item retrieval method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for enabling an item retrieval device (which can be a personal computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0170] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method for item retrieval are implemented.

[0171] Correspondingly, an embodiment of the present application provides an electronic device 700, including a first memory 702 and a first processor 701, wherein the first memory 702 stores a computer program that can be executed on the first processor 701, and the sample retrieval information first processor 701 implements the steps in the above method when executing the program.

[0172] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0173] It should be noted that Fig.10 A hardware entity diagram of an electronic device provided in an embodiment of the present application Figure 1 ,like Fig.10 As shown, the hardware entity of the electronic device 700 includes: a first processor 701 and a first memory 702, wherein;

[0174] The first processor 701 generally controls the overall operations of the electronic device 700 .

[0175] The first memory 702 is configured to store instructions and applications executable by the first processor 701, and can also cache data to be processed or processed by the first processor 701 and various modules in the electronic device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0176] See also Fig.11 , which is a structural diagram of the model training device provided in an embodiment of the present application.

[0177] The embodiment of the present application also provides a model training device 800, including: an information acquisition unit 801 and a training unit 802.

[0178] The information acquisition unit 801 is used to acquire sample search information, as well as text information and multimedia information of sample items;

[0179] The training unit 802 is used to train the initial preset model based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information, and stops when the initial preset model meets the training conditions to obtain the preset model.

[0180] In the embodiment of the present application, the training unit 802 in the model training device 800 is used to process the sample retrieval information based on the initial preset model to obtain a sample retrieval feature, process the text information based on the initial preset model to obtain a sample text feature, process the multimedia information based on the initial preset model to obtain a sample multimedia feature, and fuse the sample text feature and the sample multimedia feature based on the initial preset model to obtain a sample fusion feature;

[0181] The initial preset model is trained based on a first contrast loss between the sample retrieval feature and the sample text feature, a second contrast loss between the sample retrieval feature and the sample fusion feature, and a third contrast loss between the sample text feature and the sample multimedia feature, and the training is stopped until the initial preset model meets the training conditions to obtain the preset model.

[0182] In the embodiment of the present application, the initial preset model includes: at least one of a first initial text model, a second initial text model, an initial multimedia feature model, and an initial fusion model; the training unit 802 in the model training device 800 is used to process the sample retrieval information based on the first initial text model to obtain the sample retrieval feature;

[0183] Processing the text information based on the second initial text model to obtain the sample text features;

[0184] Processing the multimedia information based on the initial multimedia feature model to obtain the sample multimedia features;

[0185] The sample text features and the sample multimedia features are processed based on the initial fusion model to obtain the sample fusion features.

[0186] In an embodiment of the present application, the preset model includes: at least one of a first text model, a second text model, a multimedia feature model and a fusion model; in an embodiment of the present application, the training unit 802 in the model training device 800 is used to adjust the first initial text model based on the first contrast loss and the second contrast loss, adjust the second initial text model based on the first contrast loss and the third contrast loss, adjust the initial fusion model based on the second contrast loss, and adjust the initial multimedia feature model based on the third contrast loss, until the first initial text model, the second initial text model, the initial multimedia feature model and the initial fusion model all reach a predetermined number of training times and stop, thereby obtaining the second text model, the second text model, the multimedia feature model and the fusion model.

[0187] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method on the model training side are implemented.

[0188] Correspondingly, an embodiment of the present application provides an electronic device 900, including a second memory 902 and a second processor 901, wherein the second memory 902 stores a computer program that can be executed on the second processor 901, and the sample retrieval information second processor 901 implements the steps in the above method when executing the program.

[0189] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0190] It should be noted that Fig.12 A hardware entity diagram of an electronic device provided in an embodiment of the present application Figure 2 ,like Fig.12 As shown, the hardware entity of the electronic device 900 includes: a second processor 901 and a second memory 902, wherein;

[0191] The second processor 901 generally controls the overall operation of the electronic device 900 .

[0192] The second memory 902 is configured to store instructions and applications executable by the second processor 901, and can also cache data to be processed or processed by the second processor 901 and various modules in the electronic device 900 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0193] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0194] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0195] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0196] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0197] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0198] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes.

[0199] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a mobile storage device, a ROM, a magnetic disk, or an optical disk.

[0200] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An item retrieval method, characterized in that: include: Obtaining retrieval features based on the acquired retrieval information; Based on the similarity between the search feature and the text feature, multimedia feature and fusion feature corresponding to each item, a target item is determined from the multiple items; the fusion feature is obtained by fusion of the text feature and the multimedia feature; Among them, the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained based on the preset model; the preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information.

2. The article retrieval method according to claim 1, characterized in that: The determining of a target item from a plurality of items based on the similarity between the search feature and the text feature, multimedia feature and fusion feature corresponding to each item comprises: Based on the similarity between the search feature and the text feature, the multimedia feature and the fusion feature corresponding to each of the items, sorting the multiple items to obtain an item sequence; The target item is determined in the item sequence.

3. The article retrieval method according to claim 2, characterized in that: The step of sorting the plurality of items to obtain an item sequence based on the similarity between the search feature and the text feature, the multimedia feature, and the fusion feature corresponding to each item includes: Based on a first similarity between the search feature and the text feature of each of the items, sorting the plurality of items to obtain a first item sequence; each of the items in the first item sequence corresponds to the first similarity; Based on the second similarity between the search feature and the multimedia feature of each of the items, sorting the plurality of items to obtain a second item sequence; each of the items in the first item sequence corresponds to the second similarity; Based on the third similarity between the search feature and the fusion feature of each of the items, the plurality of items are sorted to obtain a third item sequence; each item in the third item sequence corresponds to the third similarity.

4. The article retrieval method according to claim 3, characterized in that: The step of determining the target item in the item sequence includes: Based on the first similarity of each of the items in the first item sequence, the second similarity of each of the items in the second item sequence, and the third similarity of each of the items in the third item sequence, deduplication and fusion are performed on the items in the first item sequence, the second item sequence, and the third item sequence to obtain a fusion sequence; and the multiple items in the fusion sequence are arranged in descending order of similarity; The first N items in the fusion sequence are determined to be the target items; N is an integer.

5. The object retrieval method according to any one of claims 1 to 4, characterized in that: Before obtaining the search features based on the acquired search information, the method further includes: The text information and multimedia information of each of the items are processed based on the preset model to obtain the text features, multimedia features and fusion features corresponding to each of the items.

6. The object retrieval method according to claim 5, characterized in that: The preset model includes: at least one of a first text model, a second text model, a multimedia feature model and a fusion model; the first text model is used to process the search information to obtain the search feature; The processing of the text information and multimedia information of each of the items based on the preset model to obtain the text features, multimedia features and fusion features corresponding to each of the items includes: Processing the text information of each of the items based on the second text model to obtain the text feature; Processing the multimedia information of each of the items based on the multimedia feature model to obtain the multimedia feature; The text feature and the multimedia feature are fused based on the fusion feature to obtain the fusion feature.

7. A model training method, characterized in that: include: Obtain sample retrieval information, as well as text information and multimedia information of sample items; The initial preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information, and the training is stopped until the initial preset model meets the training conditions to obtain the preset model.

8. The model training method according to claim 7, characterized in that: The initial preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information until the initial preset model meets the training condition, thereby obtaining the preset model, including: Processing the sample retrieval information based on the initial preset model to obtain a sample retrieval feature, processing the text information based on the initial preset model to obtain a sample text feature, processing the multimedia information based on the initial preset model to obtain a sample multimedia feature, and fusing the sample text feature and the sample multimedia feature based on the initial preset model to obtain a sample fusion feature; The initial preset model is trained based on a first contrast loss between the sample retrieval feature and the sample text feature, a second contrast loss between the sample retrieval feature and the sample fusion feature, and a third contrast loss between the sample text feature and the sample multimedia feature, and the training is stopped until the initial preset model meets the training conditions to obtain the preset model.

9. The model training method according to claim 8, characterized in that: The initial preset model includes: at least one of a first initial text model, a second initial text model, an initial multimedia feature model, and an initial fusion model; The processing of the sample retrieval information based on the initial preset model to obtain a sample retrieval feature, the processing of the text information based on the initial preset model to obtain a sample text feature, the processing of the multimedia information based on the initial preset model to obtain a sample multimedia feature, and the fusion of the sample text feature and the sample multimedia feature based on the initial preset model to obtain a sample fusion feature include: Processing the sample retrieval information based on the first initial text model to obtain the sample retrieval feature; Processing the text information based on the second initial text model to obtain the sample text features; Processing the multimedia information based on the initial multimedia feature model to obtain the sample multimedia features; The sample text features and the sample multimedia features are processed based on the initial fusion model to obtain the sample fusion features.

10. The model training method according to claim 8, characterized in that: The preset model includes: at least one of a first text model, a second text model, a multimedia feature model and a fusion model; The initial preset model is trained based on a first contrast loss between the sample retrieval feature and the sample text feature, a second contrast loss between the sample retrieval feature and the sample fusion feature, and a third contrast loss between the sample text feature and the sample multimedia feature until the initial preset model meets the training condition, thereby obtaining the preset model, including: The first initial text model is adjusted based on the first contrast loss and the second contrast loss, the second initial text model is adjusted based on the first contrast loss and the third contrast loss, the initial fusion model is adjusted based on the second contrast loss, and the initial multimedia feature model is adjusted based on the third contrast loss until the first initial text model, the second initial text model, the initial multimedia feature model, and the initial fusion model all reach a predetermined number of training times, thereby obtaining the first text model, the second text model, the multimedia feature model, and the fusion model.

11. An item retrieval device, characterized in that: include: An information processing unit, used for processing the acquired search information to obtain search features; A determination unit, configured to determine a target item from a plurality of items based on similarities between the search feature and text features, multimedia features, and fusion features corresponding to each item; the fusion features are obtained by fusion of the text features and the multimedia features; Among them, the retrieval feature, the text feature, the multimedia feature and the fusion feature are obtained based on the preset model; the preset model is trained based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information and the multimedia information of the sample item, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information.

12. A model training device, characterized in that: include: An information acquisition unit, used to acquire sample retrieval information, as well as text information and multimedia information of sample items; A training unit is used to train an initial preset model based on the matching degree between the sample retrieval information and the text information of the sample item, the matching degree between the text information of the sample item and the multimedia information, and the matching degree between the sample retrieval information and the fusion information of the text information and the multimedia information, and stop when the initial preset model meets the training conditions to obtain a preset model.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and the sample retrieval information processor implements the steps of the method described in any one of claims 1 to 6 or the steps of the method described in any one of claims 7 to 10 when executing the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6, or implements the steps of the method according to any one of claims 7 to 10.

Images