Article retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides an article retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of data search. The method comprises the steps of obtaining retrieval information sent by a client, and processing the retrieval information and corresponding user information based on a preset model to obtain retrieval features; based on the similarity between the retrieval feature and the image-text fusion feature corresponding to each article, determining a target article in the plurality of articles; the image-text fusion features are obtained by processing image-text information of each article based on a preset model; wherein the preset model is obtained by training based on fusion of graphic and text information corresponding to the article and clicking of the graphic and text information of the article; the click article is determined by the user for the fed back sample target article; and sending the related information of the target article to the client. According to the method and the device, the matching degree of the target article and the retrieval information is improved, and meanwhile, the target article sent to the client is more matched with the real demand of the user.

Description

Technical Field

[0001] The present application relates to the field of data search technology, and in particular to an object retrieval method, device, electronic device and storage medium. Background Art

[0002] In the related art, the traditional method of item retrieval on e-commerce platforms is usually based on text search. Users can enter keywords or phrases in the search box, and the system will automatically match relevant items and return search results. Existing search methods mainly rely on text features such as item titles, brands, and item categories. Single feature recall based on text lacks a complete description and understanding of the item, resulting in a poor match between the returned items determined by the server and the search keywords, and the returned items do not match the actual needs of users. Summary of the invention

[0003] An item retrieval method, device, electronic device and storage medium provided in the embodiments of the present application can improve the matching degree between the returned items determined by the server and the search keywords, and improve the matching degree between the returned items and the actual needs of the user.

[0004] The technical solution of this application is implemented as follows:

[0005] The present application embodiment provides an item retrieval method, including:

[0006] Acquire the search information sent by the client, and process the search information and corresponding user information based on a preset model to obtain search features;

[0007] Based on the similarity between the search feature and the image-text fusion feature corresponding to each item, a target item is determined from the multiple items; the image-text fusion feature is obtained by processing the image-text information of each item based on the preset model;

[0008] The preset model is obtained by training based on the fusion of the graphic information corresponding to the item and the graphic information of the clicked item; the clicked item is determined by the user based on the sample target item fed back;

[0009] Send the relevant information of the target item to the client.

[0010] In the above solution, before obtaining the search information sent by the client and processing the search information and the corresponding user information based on the preset model to obtain the search features, the method further includes:

[0011] Based on the first sample retrieval information intercepted from the text information corresponding to the item and the graphic information corresponding to the item, the initial model is trained until a first training condition is reached, thereby obtaining an intermediate model; wherein the text information belongs to the graphic information corresponding to the item;

[0012] Acquire a clicked item identifier based on the second sample retrieval information sent by the sample client; wherein the clicked item identifier is used to represent identification information of the clicked item;

[0013] Determine the clicked item and the non-clicked item from a plurality of sample target items based on the clicked item identifier, and form hard training labels with the clicked item and the non-clicked item through the second sample retrieval information; wherein the plurality of sample target items are items fed back to the sample client in response to the second retrieval information sent by the sample client;

[0014] Training the intermediate model based on the hard training labels and the soft training labels determined during the training of the hard training labels, and stopping until a second training condition is met to obtain the preset model;

[0015] Among them, the soft training label includes: the second sample retrieval information, predicted click items and predicted non-click items information; the predicted click items and the predicted non-click items are obtained by distributing the multiple sample target items based on the sample probability distribution; the sample probability distribution is determined based on the parameter update information of the hard training label during the training process of the intermediate model.

[0016] In the above scheme, the initial model is trained based on the first sample retrieval information intercepted from the text information corresponding to the item and the graphic information corresponding to the item until the first training condition is reached to obtain the intermediate model, including:

[0017] The first sample retrieval information is processed by the initial model to obtain a first sample retrieval feature, the text information is processed to obtain a first text feature, the image information is processed to obtain a first image feature, and the first text feature and the first image feature are fused to obtain a first sample image-text feature; wherein the image information belongs to the image-text information corresponding to the item;

[0018] The initial model is gradient updated based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image and text feature, until the initial model reaches a first predetermined number of training times, thereby obtaining the intermediate model.

[0019] In the above scheme, the initial model includes: an initial retrieval model and an initial image-text fusion model;

[0020] The step of performing gradient updating on the initial model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature, and the first sample image-text feature, until the initial model reaches a first predetermined number of training times, and obtaining the intermediate model, comprises:

[0021] Synchronously performing gradient update on the initial retrieval model and the initial image-text fusion model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature, and the first sample image-text feature, until the initial retrieval model and the initial image-text fusion model reach a first predetermined number of training times, thereby obtaining an intermediate retrieval model and an intermediate image-text fusion model;

[0022] The intermediate retrieval model and the intermediate image-text fusion model are combined to determine the intermediate model.

[0023] In the above solution, the step of obtaining the clicked item identifier based on the second sample search information sent by the sample client includes:

[0024] Processing the second sample retrieval information sent by the sample client based on the intermediate model to obtain a second sample retrieval feature;

[0025] Based on the similarity between the second sample retrieval feature and the intermediate image-text fusion feature corresponding to each item, a plurality of sample target items are determined from the plurality of items; the intermediate image-text fusion feature is obtained by processing the image-text information of each item based on the intermediate model;

[0026] Feedback the plurality of sample target items to the sample client, and obtain the clicked item identifier sent by the sample client; wherein the clicked item identifier is sent by the sample client in response to instructions for the plurality of sample target items.

[0027] In the above scheme, the intermediate model is trained based on the hard training label and the soft training label determined during the training process of the hard training label until the second training condition is reached to obtain the preset model, including:

[0028] Using the second sample retrieval information and the graphic information of the clicked item and the non-clicked item, the intermediate model is calculated to obtain a first contrast loss, and parameter update information of the intermediate model during the training process for the hard training label is extracted;

[0029] Obtaining the sample probability distribution determined by processing the parameter update information by the network momentum model, and dividing the plurality of sample target items based on the sample probability distribution to obtain the predicted click items and the predicted non-click items;

[0030] Traversing the clicked items, the non-clicked items, the predicted clicked items and the predicted non-clicked items to determine a pair of items with the same item identifier, performing weighted averaging based on the sample labels corresponding to the pair of items, and determining a new sample target item and a new sample label corresponding to the item identifier, until the traversal is completed to obtain a plurality of the new sample target items and the new sample label corresponding to each of the new sample target items;

[0031] Calculating the intermediate model based on the new sample label and the graphic information corresponding to each of the new sample target items to obtain a second contrast loss;

[0032] The intermediate model is gradient updated based on the first contrast loss and the second contrast loss until the intermediate model reaches a second predetermined number of training times, thereby obtaining the preset model.

[0033] In the above scheme, the intermediate model includes: an intermediate retrieval model and an intermediate image-text fusion model; the intermediate model is gradient updated based on the first contrast loss and the second contrast loss until the intermediate model reaches a second predetermined number of training times, and the preset model is obtained, including:

[0034] Synchronously performing gradient update on the intermediate retrieval model and the intermediate image-text fusion model based on the first contrast loss and the second contrast loss until the intermediate retrieval model and the intermediate image-text fusion model reach a second predetermined training number, thereby obtaining a preset retrieval model and a preset image-text fusion model;

[0035] The preset retrieval model and the preset image-text fusion model are combined to determine the preset model.

[0036] In the above solution, the step of determining a target item from a plurality of items based on the similarity between the search feature and the image-text fusion feature corresponding to each item includes:

[0037] Determine the similarity between the search feature and the image-text fusion feature corresponding to each of the items;

[0038] Based on the similarity between the search feature and the image-text fusion feature corresponding to each of the items, the multiple items are sorted to obtain an item sequence; wherein the multiple items in the item sequence are arranged in order according to the similarity;

[0039] The first N items in the item sequence are determined to be the target items; N is an integer.

[0040] In the above scheme, the preset model includes: a preset image-text fusion model;

[0041] Before determining the target item from the plurality of items based on the similarity between the search feature and the image-text fusion feature corresponding to each item, the method further includes:

[0042] The image-text fusion feature is obtained by processing the image-text information of each of the items based on the preset image-text fusion model.

[0043] In the above scheme, the preset model includes: a preset retrieval model;

[0044] The step of processing the search information and the corresponding user information based on a preset model to obtain search features includes:

[0045] The search information and the user information are processed based on the preset search model to obtain the search feature.

[0046] The present application also provides an object retrieval device, including:

[0047] An information processing unit, used to obtain the search information sent by the client, and process the search information and corresponding user information based on a preset model to obtain search features;

[0048] A determination unit, configured to determine a target item from a plurality of items based on the similarity between the search feature and a graphic-text fusion feature corresponding to each item; the graphic-text fusion feature is obtained by processing the graphic-text information of each item based on the preset model;

[0049] The preset model is obtained by training based on the fusion of the graphic information corresponding to the item and the graphic information of the clicked item; the clicked item is determined by the user based on the sample target item fed back;

[0050] A sending unit is used to send the relevant information of the target item to the client.

[0051] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the sample retrieval information processor implements the steps in the above method when executing the computer program.

[0052] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method are implemented.

[0053] In the embodiment of the present application, the retrieval information sent by the client is obtained, and the retrieval information and the corresponding user information are processed based on the preset model to obtain the retrieval feature; based on the similarity between the retrieval feature and the image-text fusion feature corresponding to each item, the target item is determined from multiple items; the image-text fusion feature is obtained by processing the image-text information of each item based on the preset model; wherein the preset model is obtained by training based on the fusion of the image-text information corresponding to the item and the image-text information of the clicked item; the clicked item is determined by the user for the sample target item fed back; the relevant information of the target item is sent to the client. Since the present application not only considers the similarity between the retrieval feature of the retrieval information and the text feature of the item in the process of determining the target item, but also considers the image feature of the item, the matching degree between the target item and the retrieval information is improved from the perspective of multi-dimensional features. And since the preset model is trained from multiple angles based on the image-text information of the item and the clicked item, the preset model can more fully learn the matching degree between the retrieval information and the image-text information, and also enables the preset model to more fully learn the matching degree between the image-text information of the item actually clicked by the user and the retrieval information. Furthermore, the similarity between the retrieval features obtained through the preset model processing and the image-text fusion features calculated is more accurate, making the target items sent to the client more closely match the user's actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0055] Figure 2 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0056] Figure 3 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0057] Figure 4 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0058] Figure 5 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0059] Figure 6 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0060] Figure 7 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0061] Figure 8An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0062] Fig. 9 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0063] Fig.10 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0064] Fig.11 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0065] Fig.12 An optional flowchart of the item retrieval method provided in the embodiment of the present application;

[0066] Fig.13 A schematic diagram of the structure of an object retrieval device provided in an embodiment of the present application;

[0067] Fig.14 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0069] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0070] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] Search in e-commerce scenarios mainly goes through four stages: recall, rough sorting, fine sorting, and re-ranking. The main task of the recall module is to quickly retrieve items related to the user's search intent from a massive item library. Traditional inverted index recall can only recall items through literal text information. The existing vector recall method solves the problem that the inverted index cannot recall items that are not similar to the search terms but have similar semantics, and is widely used in recall retrieval. The vector recall model calculates the relevance between the user's search terms and the items, and realizes item retrieval through similarity retrieval. The existing vector recall method mainly relies on text features such as item title, brand, and item category. Therefore, the single-modal vector recall based on text lacks a complete description and understanding of the item, and has certain limitations when users use search terms with visual elements.

[0073] This application embodiment provides an item retrieval method, see Figure 1 , is an optional flow chart of the item retrieval method provided in the embodiment of the present application, which will be combined with Figure 1 The steps shown are explained.

[0074] S101. Acquire search information sent by a client, and process the search information and corresponding user information based on a preset model to obtain search features.

[0075] In the embodiment of the present application, the server may receive the search information sent by the client, and process the search information and the user information of the client user based on a preset model to obtain the search feature.

[0076] In an embodiment of the present application, a user may enter keywords (search information) of a desired item in a search interface of a client, and the client sends the keywords to a server, which processes the keywords and user information based on a preset model to obtain search features.

[0077] The user information may include: the gender, age, estimated transaction ability value, whether the client is a member, and other information of the user.

[0078] S102. Determine a target item from among the multiple items based on the similarity between the search feature and the image-text fusion feature corresponding to each item; the image-text fusion feature is obtained by processing the image-text information of each item based on the preset model; wherein the preset model is trained based on the fusion of the image-text information corresponding to the items and the image-text information of the clicked item; the clicked item is determined by the user based on the sample target item fed back.

[0079] In an embodiment of the present application, the server stores text information and image information of each of the multiple items. The server can process the text information of each item based on a preset model to obtain text features, process the image information of each item to obtain image features, and fuse the text features and image features to obtain image-text fusion features. The server can determine the similarity between the search feature and the image-text fusion feature of each item. Based on the similarity corresponding to each item, the N items with the greatest similarity among the multiple items are determined as target items. N is an integer. Among them, the text information can be text information such as the corresponding item title, description, and classification.

[0080] The preset model is obtained by training based on the fusion of the graphic and text information corresponding to the object and the graphic and text information of the clicked object; the clicked object is determined by the user based on the sample target object fed back.

[0081] In an embodiment of the present application, the training process of the preset model includes: intercepting the first sample retrieval information based on the text information of the object, and training the initial model based on the first sample retrieval information and the corresponding graphic information to obtain an intermediate model. The intermediate model is trained based on the graphic information of the clicked object, and the parameter update information of the intermediate model during the training process is extracted. Based on the parameter update information, a new sample probability distribution is determined, the clicked objects are re-divided, and the predicted clicked objects are determined. The intermediate model is trained based on the graphic information of the clicked object and the predicted clicked object to obtain the preset model.

[0082] In some other embodiments, the server may sort the multiple items based on the similarity between the search feature and the image-text fusion feature corresponding to each item. The items ranked higher have a higher matching degree with the search information. The server may determine the first N items as the target items among the sorted multiple items.

[0083] S103: Send relevant information of the target item to the client.

[0084] In the embodiment of the present application, after determining the target item, the server can send at least one of the text information, image information, video information and audio information corresponding to the target item to the client. The client receives the text information, image information, video information and audio information of the target item and renders and displays them on the current page using a rendering component.

[0085] In the embodiment of the present application, the retrieval information sent by the client is obtained, and the retrieval information and the corresponding user information are processed based on the preset model to obtain the retrieval feature; based on the similarity between the retrieval feature and the image-text fusion feature corresponding to each item, the target item is determined from multiple items; the image-text fusion feature is obtained by processing the image-text information of each item based on the preset model; wherein the preset model is obtained by training based on the fusion of the image-text information corresponding to the item and the image-text information of the clicked item; the clicked item is determined by the user for the sample target item fed back; the relevant information of the target item is sent to the client. Since the present application not only considers the similarity between the retrieval feature of the retrieval information and the text feature of the item in the process of determining the target item, but also considers the image feature of the item, the matching degree between the target item and the retrieval information is improved from the perspective of multi-dimensional features. And since the preset model is trained from multiple angles based on the image-text information of the item and the clicked item, the preset model can more fully learn the matching degree between the retrieval information and the image-text information, and also enables the preset model to more fully learn the matching degree between the image-text information of the item actually clicked by the user and the retrieval information. Furthermore, the similarity between the retrieval features obtained through the preset model processing and the image-text fusion features calculated is more accurate, making the target items sent to the client more closely match the user's actual needs.

[0086] In some embodiments, see Figure 2 , Figure 2 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 1 The illustrated S101 may also include S104 to S107, which will be described in conjunction with each step.

[0087] S104. Training the initial model based on the first sample retrieval information intercepted from the text information corresponding to the item and the graphic information corresponding to the item, and stopping until a first training condition is reached to obtain an intermediate model; wherein the text information belongs to the graphic information corresponding to the item.

[0088] In the embodiment of the present application, the graphic information of each item includes: image information and text information. The server can randomly intercept a text in the text information of each item as the first sample retrieval information. The first sample retrieval information, text information and image information of an item are input into the initial model for one training, and then the first sample retrieval information, text information and image information of the next item are input into the initial model for the next training, until the training of the initial model reaches the first training condition, and then the intermediate model is obtained.

[0089] S105. Obtain a clicked item identifier based on the second sample retrieval information sent by the sample client; wherein the clicked item identifier is used to represent identification information of the clicked item.

[0090] In the embodiment of the present application, the server uses the intermediate model for online application. The image and text information of each item is processed based on the intermediate model to obtain the corresponding intermediate image and text fusion features. The second search information sent by a sample client and the user information corresponding to the sample client are processed based on the intermediate model. Multiple sample target items are determined from multiple items and sent to the sample client. The sample client displays the multiple sample target items, the user clicks on the item on the sample client, and the sample client feeds back the identification information of the item clicked by the user.

[0091] Among them, the sample client is the client that interacts with the server to retrieve items after the intermediate model is applied online.

[0092] S106. Determine the clicked item and the non-clicked item among multiple sample target items based on the clicked item identifier, and form hard training labels with the clicked item and the non-clicked item through the second sample retrieval information; wherein, the multiple sample target items are items fed back to the sample client in response to the second retrieval information sent by the sample client.

[0093] In an embodiment of the present application, each item in the server corresponds to an item identifier. Multiple sample target items belong to multiple items. The server can determine the clicked item among the multiple sample target items based on the acquired clicked item identifier. And determine that the items other than the clicked item among the multiple sample target items are non-clicked items. Based on the second sample retrieval information, a hard training label is formed for each clicked item and each non-clicked item. Among them, the multiple sample target items are items that are fed back to the sample client in response to the second retrieval information sent by the sample client. Both the clicked item and the non-clicked item include graphic information.

[0094] For example, when the server feeds back relevant information of 10 sample target items to the sample client, the user clicks on the first 3 items among the 10 items displayed by the sample client, and the sample client feeds back the identifiers of the first 3 items to the server. The server determines the clicked item and the non-clicked item based on the identifiers of the 3 clicked items.

[0095] S107. Train the intermediate model based on the hard training labels and the soft training labels determined during the training process of the hard training labels, and stop when the second training condition is reached to obtain the preset model; wherein the soft training labels include: the second sample retrieval information, predicted click items, and predicted non-click items.

[0096] In an embodiment of the present application, the server trains the intermediate model based on the hard training labels to determine the first contrast loss, and uses the parameter update information when the intermediate model is trained corresponding to the hard training labels. Based on the parameter update information, the sample probability distribution for multiple sample target items is determined. Based on the sample probability distribution, multiple sample target items are divided to determine predicted click items and predicted non-click items. Based on the soft training labels (the second retrieval information, predicted click items, and predicted non-click items), the intermediate model is trained to determine the second contrast loss, and the intermediate model is gradient updated using the first contrast loss and the second contrast loss to complete one training. Stop until the intermediate model reaches the second training condition to obtain a preset model.

[0097] Among them, contrast loss is a binary classification loss function, which is usually used to measure the matching degree between two samples. Its basic idea is to take the feature vectors of two samples as input and determine whether they belong to the same category by calculating their similarity. If the two samples belong to the same category, the contrast loss is 0. If they do not belong to the same category, the contrast loss is a positive number. The priority of contrast loss is that it is simple to calculate and is applicable to large-scale data sets.

[0098] The embodiment of the present application uses multimodal retrieval to realize item recall. The multimodal retrieval model (preset model) introduces the image information of the item on the basis of single text domain feature retrieval. The model learns the semantic relationship between the search term and the item's graphic information from a large number of users clicking on items, and uses the intuitive visual features contained in the image to supplement the text features. Through the fusion of graphic and text information, the model's ability to represent the semantics of the item is improved, and the user's search intent is matched more accurately.

[0099] In some embodiments, see Figure 3 , Figure 3 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 2 The illustrated S104 can also be implemented through S108 to S109, which will be described in conjunction with each step.

[0100] S108. Using the initial model, the first sample retrieval information is processed to obtain a first sample retrieval feature, the text information is processed to obtain a first text feature, the image information is processed to obtain a first image feature, and the first text feature and the first image feature are merged to obtain a first sample image-text feature.

[0101] In an embodiment of the present application, the server uses the initial model to process the first sample retrieval information to obtain the first sample retrieval feature, processes the text information corresponding to the item to obtain the first text feature, processes the image information to obtain the first image feature, and fuses the first text feature and the first image feature to obtain the first sample graphic feature.

[0102] In the embodiment of the present application, the initial model includes a Transformer encoder and a Cross Attention model. The Transformer encoder can be used to process the first sample retrieval information to obtain a first sample retrieval feature, the Transformer encoder can be used to process the first text information to obtain a first text feature, and the Transformer encoder can be used to process the first image information to obtain a first image feature. The Cross Attention model is used to fuse the first text feature and the first image feature to obtain a first sample image-text feature.

[0103] S109, performing gradient update on the initial model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image-text feature, until the initial model reaches a first predetermined number of training times, thereby obtaining the intermediate model.

[0104] In the embodiment of the present application, the server performs a gradient update on the initial model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image and text feature, and completes one training. Then, the next item is used to perform the next training on the initial model until the initial model reaches the first predetermined number of training times, and then the intermediate model is obtained.

[0105] In the embodiment of the present application, there is no specific limitation on the specific number of the first predetermined training times.

[0106] In the embodiment of the present application, the initial model is gradient updated based on the contrast loss between the first sample retrieval feature intercepted from the text information of the item and the first text feature, the first image feature and the first sample image and text feature, until the initial model reaches the first predetermined number of training times, and then the intermediate model is obtained. This process uses the image and text information of the item to construct a training data set, which can improve the accuracy of the preset model in fusing the image information and text information of the item.

[0107] In some embodiments, see Figure 4 , Figure 4 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 3 The illustrated S109 can also be implemented through S110 to S111 , which will be described in conjunction with each step.

[0108] S110, synchronously performing gradient update on the initial retrieval model and the initial image-text fusion model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image-text feature, until the initial retrieval model and the initial image-text fusion model reach a first predetermined number of training times, thereby obtaining an intermediate retrieval model and an intermediate image-text fusion model.

[0109] In an embodiment of the present application, the initial model includes: an initial retrieval model and an initial image-text fusion model. Among them, the initial retrieval model may include: a Feature Embedding model and a Transformer model. The initial image-text fusion model includes: a text Feature Embedding model, an image Feature Embedding model, a text Transformer model, an image Transformer model, and a Cross Attention model for fusing image and text features. The server synchronously performs gradient updates on the initial retrieval model and the initial image-text fusion model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature, and the first sample image-text feature, until the initial retrieval model and the initial image-text fusion model reach the first predetermined number of training times and stop, and obtain an intermediate retrieval model and an intermediate image-text fusion model.

[0110] S111, combining the intermediate retrieval model and the intermediate image-text fusion model to determine the intermediate model.

[0111] In the embodiment of the present application, the intermediate model is the same as the preset model, which is a double-tower model. The intermediate model can be obtained based on the combination of the intermediate retrieval model and the intermediate image-text fusion model.

[0112] Exemplary, combined Figure 5, the initial model includes: an initial retrieval model and an initial fusion model. At the same time, the search terms determined in the item text information are input into the initial retrieval model, and the item title, category, store name, and item image of the item are input into the initial fusion model. The initial retrieval model processes the search terms to obtain feature vector representation, and the feature vector representation is processed by the deep learning model encoder to obtain the search term vector representation. The initial fusion model processes the item title, category, and store name to obtain the text feature vector representation, and the text feature vector representation is processed by the deep learning model encoder to obtain the item text vector representation. The initial fusion model processes the item image to obtain the image feature vector representation, and the image feature vector representation is processed by the deep learning model encoder to obtain the item image vector representation. The item image vector representation and the item text vector representation are fused by the attention mechanism model to obtain the item vector representation. The initial retrieval model and the initial fusion model are synchronously gradient updated based on the contrast loss between the search term vector representation and the item image vector representation, the item text vector representation, and the item vector representation to obtain the intermediate model.

[0113] In the embodiment of the present application, in the model Pretrain stage (the training stage of the intermediate model), query-sku, query-sku text and query-sku image matching tasks are designed, that is, the search terms (first sample retrieval information) are paired with the graphic features of the item, the pure text features of the item and the image features of the item, respectively, so that the text and image can play a role in the recall task in different degrees and in different fusion modes. In the Pretrain stage, Fake search terms are constructed by randomly intercepting the text information of the item. This method of constructing a pre-training data set does not require the use of the user's real query information, and only uses the graphic information of the item to generate a training data set, which can improve the accuracy of the preset model in fusing the image information and text information of the item.

[0114] In some embodiments, see Figure 6 , Figure 6 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 2 The illustrated S105 can also be implemented through S112 to S114, which will be described in conjunction with each step.

[0115] S112: Process the second sample retrieval information sent by the sample client based on the intermediate model to obtain a second sample retrieval feature.

[0116] In an embodiment of the present application, the server applies the intermediate model online, receives second sample retrieval information sent by the sample client, and processes the second sample retrieval information and corresponding sample user information based on the intermediate model to obtain corresponding second sample retrieval features.

[0117] S113. Determine a plurality of sample target items from a plurality of items based on the similarity between the second sample retrieval feature and the intermediate image-text fusion feature corresponding to each item; the intermediate image-text fusion feature is obtained by processing the image-text information of each item based on the intermediate model.

[0118] In the embodiment of the present application, after the intermediate model is online, the intermediate model can be used to process the image and text information corresponding to each item to obtain the intermediate image and text fusion features corresponding to each item. The server determines the similarity between the second sample retrieval feature and the intermediate image and text fusion features of each item, and determines the top N items with the greatest similarity as multiple sample target items. The relevant information of the multiple sample target items is sent to the corresponding sample client.

[0119] S114. Feedback the multiple sample target items to the sample client, and obtain the clicked item identifier sent by the sample client; wherein the clicked item identifier is sent by the sample client in response to instructions for the multiple sample target items.

[0120] In an embodiment of the present application, the server feeds back a plurality of the sample target items to the sample client, the sample client displays the plurality of sample target items, the user clicks on an item on the sample client, and the sample client feeds back identification information of the item clicked by the user.

[0121] In an embodiment of the present application, multiple sample target items are determined by utilizing the second sample retrieval information of a real sample client, the multiple sample target items are sent to the client, and the item identifiers actually clicked by the user sent by the client are received. This process uses the user's actual item click data for training, so that the preset model can be more familiar with the real intentions of various types of users, thereby improving the matching degree between the feedback target items and the user's real intentions.

[0122] In some embodiments, see Figure 7 , Figure 7 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 2 The illustrated S107 can also be implemented through S115 to S119, which will be described in conjunction with each step.

[0123] S115. Utilize the second sample retrieval information and the graphic information of the clicked item and the non-clicked item to calculate the first contrast loss for the intermediate model, and extract parameter update information of the intermediate model during the training process for the hard training label.

[0124] In an embodiment of the present application, the server processes the second sample retrieval information and the corresponding user information based on the intermediate model to obtain the second sample retrieval feature, processes the image and text information of the clicked items and the non-clicked items based on the intermediate model to obtain the intermediate sample image and text features, calculates the first contrast loss based on the second sample retrieval feature and the intermediate sample image and text features, uses the second contrast loss to perform gradient update on the intermediate model, and obtains parameter update information during the gradient update process of the intermediate model.

[0125] S116. Obtain the sample probability distribution determined by processing the parameter update information with the network momentum model, and divide the plurality of sample target items based on the sample probability distribution to obtain the predicted click items and the predicted non-click items.

[0126] In the embodiment of the present application, the server uses the network momentum model to process the parameter update information to determine the sample probability distribution. Based on the sample probability distribution, multiple sample target items are divided into predicted click items and predicted non-click items.

[0127] Among them, the network momentum model may include: Momentum model.

[0128] S117, traverse the clicked items, the non-clicked items, the predicted clicked items and the predicted non-clicked items to determine a pair of items with the same item identification, and perform weighted averaging based on the sample labels corresponding to the pair of items to determine a new sample target item and a new sample label corresponding to the item identification, until the traversal is completed to obtain multiple new sample target items and a new sample label corresponding to each new sample target item.

[0129] In an embodiment of the present application, the server traverses clicked items, non-clicked items, predicted clicked items, and predicted non-clicked items to determine a pair of items with the same item identification, and performs weighted averaging based on the sample labels corresponding to the pair of items to determine a new sample target item and a new sample label corresponding to the item identification, until the traversal is completed to obtain multiple new sample target items and a new sample label corresponding to each new sample target item.

[0130] S118. Calculate the intermediate model based on the new sample label and the graphic information corresponding to each of the new sample target items to obtain a second contrast loss.

[0131] In an embodiment of the present application, the server processes the image and text information of the predicted clicked items and the predicted non-clicked items based on the intermediate model to obtain the predicted intermediate sample image and text features, and calculates the second contrast loss based on the second sample retrieval features and the predicted intermediate sample image and text features.

[0132] S119. Performing gradient update on the intermediate model based on the first contrast loss and the second contrast loss until the intermediate model reaches a second predetermined number of training times, thereby obtaining the preset model.

[0133] In the embodiment of the present application, during the training process of the preset model, the preset model is trained from multiple angles based on the graphic information of the items and the items clicked, so that the preset model can more fully learn the matching degree between the search information and the graphic information, and also the matching degree between the graphic information of the items actually clicked by the user and the search information. In addition, the similarity between the search features obtained by the preset model and the graphic fusion features calculated is more accurate, so that the target items sent to the client are more in line with the real needs of the user.

[0134] In the embodiment of the present application, the server performs a gradient update on the intermediate model based on the first contrast loss and the second contrast loss until the intermediate model reaches a second predetermined number of training times, thereby obtaining a preset model.

[0135] In some embodiments, see Figure 8 , Figure 8 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 7 The illustrated S119 can also be implemented through S120 to S121, which will be described in conjunction with each step.

[0136] S120. Based on the first contrast loss and the second contrast loss, the intermediate retrieval model and the intermediate image-text fusion model are synchronously gradient updated until the intermediate retrieval model and the intermediate image-text fusion model reach a second predetermined number of training times, thereby obtaining a preset retrieval model and a preset image-text fusion model.

[0137] In the embodiment of the present application, the intermediate model includes: an intermediate retrieval model and an intermediate image-text fusion model, and the intermediate model and the preset model are both dual-tower model structures. The server performs synchronous gradient updates on the intermediate retrieval model and the intermediate image-text fusion model based on a weighted combination of the first contrast loss and the second contrast loss as the overall loss, until the intermediate retrieval model and the intermediate image-text fusion model reach the second predetermined training times, and then obtains the preset retrieval model and the preset image-text fusion model.

[0138] S121. Combining the preset retrieval model with the preset image-text fusion model to determine the preset model.

[0139] In the embodiment of the present application, the server obtains a preset model based on a combination of a preset retrieval model and a preset image-text fusion model.

[0140] Exemplary, combined Fig. 9, the initial model includes: an initial retrieval model and an initial fusion model. At the same time, the search terms and user information of the sample client are input into the intermediate retrieval model, and the item titles, categories, store names, and item images of the corresponding clicked items and non-clicked items are input into the intermediate image-text fusion model. The intermediate retrieval model processes the search terms and user information to obtain feature vector representations, and the deep learning model encoder processes the feature vector representations to obtain search terms & user vector representations. The intermediate image-text fusion model processes the item titles, categories, and store names to obtain text feature vector representations, and the deep learning model encoder processes the text feature vector representations to obtain item text vector representations. The intermediate image-text fusion model processes the item images to obtain image feature vector representations, and the deep learning model encoder processes the image feature vector representations to obtain item image vector representations. The item image vector representations and the item text vector representations are fused through the attention mechanism model to obtain item vector representations. The first contrast loss is calculated based on the search term vector representation and the item image vector representation, and the parameter update information of the intermediate retrieval model and the intermediate image-text fusion model is determined by the first contrast loss, and the information is passed to the network momentum model and processed to obtain the distillation information, which is passed to the intermediate retrieval model and the intermediate image-text fusion model. The sample labels of the sample target items are updated, and the second contrast loss is calculated. Based on the weighted combination of the first contrast loss and the second contrast loss as the overall loss, the intermediate retrieval model and the intermediate image-text fusion model are synchronously gradient updated until the intermediate retrieval model and the intermediate image-text fusion model reach the second predetermined training times, and the preset retrieval model and the preset image-text fusion model are obtained.

[0141] In an embodiment of the present application, in the Finetune stage of the model (the stage for training the intermediate model), the contrast loss function is calculated for the search term & user embedding and the item embedding based on the correspondence between the user's search terms and the clicked items as hard training labels. Since there is a correlation between the items themselves, taking the correlation information of the items into consideration can enrich and optimize the correspondence between the query terms and the items, so that the model can map the query terms and the items to a better embedding space. In this method, a Momentum model is designed to perform knowledge distillation to achieve this optimization. The Momentum model has the same structure as the multimodal retrieval model, but the parameters of the Momentum model do not participate in the gradient update. Instead, the parameters are updated by sliding the average of the parameters of the historical versions of the multimodal retrieval model. During the training process, in order to fuse the distillation information of the Momentum model into the multimodal retrieval model, a fusion method at the loss function level and a fusion method at the label level are adopted. The loss function consists of two parts. One part uses the real item click information as the label to calculate the loss. The other part uses the real item click information as the hard label and the Momentum model prediction result as the soft label. The loss is calculated by weighted averaging the hard label and the soft label. Finally, the two parts of the loss are weighted and combined in proportion as the overall loss function to improve the generalization performance of the model.

[0142] In an embodiment of the present application, a multimodal retrieval model (preset model) optimizes the fusion interaction of text and image multimodal information based on Transformer to achieve item vector recall. The multimodal retrieval model uses the Pretrain+Finetune training method. The Pretrain stage uses item data to construct a data set to achieve the fusion of image and text information. The Finetune stage is trained on real user click data. The retrieval model side of the preset model adds user information on the basis of user search terms to enhance the personalized recall capability of the model; the image-text fusion model side of the preset model first extracts the unimodal features of the item text domain and image domain respectively, and then fuses the modal information, so that the multimodal feature information can be fully learned. In addition, the Momentum model is introduced in the Finetune stage to supplement the correspondence between query terms and items, guiding the multimodal retrieval model to learn better embeddings.

[0143] In some embodiments, see Fig.10 , Fig.10 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 1 The illustrated S101 may also be implemented through S122 to S123, which will be described in conjunction with each step.

[0144] S122: Process the image and text information of each of the items based on the preset image and text fusion model to obtain the image and text fusion feature.

[0145] In the embodiment of the present application, the server processes the image and text information of each object based on the preset image and text fusion model in the double tower model of the preset model to obtain the image and text fusion features.

[0146] Exemplary, combined Fig.11 The preset image-text fusion model processes the item title, category, and store name to obtain a text feature vector representation, and the deep learning model encoder processes the text feature vector representation to obtain the item text vector representation. The preset image-text fusion model processes the item image to obtain an image feature vector representation, and the deep learning model encoder processes the image feature vector representation to obtain the item image vector representation. The attention mechanism model fuses the item image vector representation and the item text vector representation to obtain the item vector representation (image-text fusion feature).

[0147] In the embodiment of the present application, the text information such as the title, category, store name, etc. of the item is sent to the Transformer encoder on the preset text-image fusion model side through the text Feature Embedding, and the picture of the item is sent to the Transformer through the image Feature Em bedding, thereby obtaining the single domain features of the item in the text domain and the image domain respectively. Then the two single domain features are sent to the Cross Attention module for cross-modal feature fusion, and the item embedding (text-image fusion feature) is calculated.

[0148] S123: Process the search information and the user information based on the preset search model to obtain the search feature.

[0149] In the embodiment of the present application, the server processes the graphic and text information of each object based on the preset retrieval model in the double tower model of the preset model to obtain the retrieval features.

[0150] Exemplary, combined Fig.11 The preset retrieval model processes the search terms and user information to obtain a feature vector representation, and the deep learning model encoder processes the feature vector representation to obtain a search term & user vector representation.

[0151] In the embodiment of the present application, the user inputs the search keyword, and the search word and user information (gender, age, estimated transaction capacity value, whether or not a member) are spliced ​​and sent to the preset retrieval model side of the multimodal retrieval model, and the data is converted into a vector of a specific dimension through the FeatureEmbedding module. The search word & user embedding is then calculated through the Transformer encoder.

[0152] In the embodiment of the present application, the retrieval information and the user information are processed by the preset retrieval model of the preset model to obtain the retrieval features, and the image and text information of each item is processed by the preset image and text fusion model of the preset model to obtain the image and text fusion features. Then, the target item is determined by the similarity between the image and text fusion features and the retrieval features. In the process of determining the target item, not only the similarity between the retrieval features and the text features of the item is considered, but also the image features of the item are considered, and the matching degree between the target item and the retrieval information is improved from the perspective of multi-dimensional features.

[0153] In some embodiments, see Fig.10 , Fig.10 An optional flowchart of the item retrieval method provided in the embodiment of the present application is shown in FIG. Figure 1 The illustrated S103 may also be implemented through S124 to S126, which will be described in conjunction with each step.

[0154] S124: Determine the similarity between the search feature and the image-text fusion feature corresponding to each of the items.

[0155] In the embodiment of the present application, the server calculates the similarity between the search feature and the image-text fusion feature corresponding to each item.

[0156] In the embodiment of the present application, the server can calculate the similarity cosθ between the search feature and the image-text fusion feature of each of the items through formula (1).

[0157]

[0158] Among them, the retrieval feature vector is a, and the image-text fusion feature vector is b. The server can multiply a and b, and then multiply the modulus length of a and the modulus length of b. The similarity cosθ is obtained.

[0159] S125. Sort the multiple items to obtain an item sequence based on the similarity between the search feature and the image-text fusion feature corresponding to each of the items; wherein the multiple items in the item sequence are arranged in order according to the similarity.

[0160] In an embodiment of the present application, the server sorts multiple items in descending order of similarity based on the similarity between the search feature and the image-text fusion feature corresponding to each item to obtain an item sequence; wherein the multiple items in the item sequence are arranged in order according to the size of the similarity.

[0161] S126. Determine the first N items in the item sequence as the target items; N is an integer.

[0162] In the embodiment of the present application, the server determines that the first N items in the item sequence are the target items; N is an integer.

[0163] Exemplary, combined Fig.11 , calculate the similarity between the search term & user vector representation (embedding) and the image-text fusion features (item embedding), and recall the top N products with the highest similarity.

[0164] In the embodiment of the present application, by calculating the similarity between the search feature and the image-text fusion feature, multiple items are sorted to determine the target item that is ranked first. This process uses the multi-dimensional perspective of the image feature dimension and the text feature dimension of the item. Compared with the prior art method that only starts from the text feature, the granularity is finer, and the matching degree between the target item and the search information is improved from the multi-dimensional feature.

[0165] In some embodiments, see Fig.12 , Fig.12 An optional flowchart of the item retrieval method provided for an embodiment of the present application will be described in conjunction with each step.

[0166] S201: The user inputs a search term.

[0167] In the embodiment of the present application, the user enters a search term in the client, and the client sends the search term to the server.

[0168] S202: Obtain user characteristic information.

[0169] In the embodiment of the present application, the server receives the search term sent by the client and extracts the corresponding user feature information according to the client identifier.

[0170] S203: Calculate search term & user vector representation.

[0171] In an embodiment of the present application, the server processes the search term and user information based on a preset model to obtain a search term & user vector representation.

[0172] S204: Retrieve the vector representation of the offline item library.

[0173] In the embodiment of the present application, the server searches the offline item library to determine the similarity between the item vector representation of each item and the search term & user vector representation.

[0174] S205. Obtain the top N items ranked by similarity.

[0175] In the embodiment of the present application, the server obtains relevant information of the top N items in similarity ranking and sends it to the client.

[0176] S206: sort items.

[0177] In the embodiment of the present application, the client receives relevant information of the Top N items ranked by similarity, and sorts each item according to the similarity.

[0178] S207: Present search results.

[0179] In the embodiment of the present application, the client displays the relevant information of the Top N items on the display interface.

[0180] See also Fig.13 , Fig.13 A schematic diagram of the structure of an item retrieval device provided in an embodiment of the present application.

[0181] This community young embodiment also provides an item retrieval device 800, including: an information processing unit 801, a determination unit 802 and a sending unit 803.

[0182] The information processing unit 801 is used to obtain the search information sent by the client, and process the search information and the corresponding user information based on a preset model to obtain a search feature;

[0183] A determination unit 802 is used to determine a target item from a plurality of items based on the similarity between the search feature and a graphic-text fusion feature corresponding to each item; the graphic-text fusion feature is obtained by processing the graphic-text information of each item based on the preset model;

[0184] The preset model is obtained by training based on the fusion of the graphic information corresponding to the item and the graphic information of the clicked item; the clicked item is determined by the user based on the sample target item fed back;

[0185] The sending unit 803 is used to send the relevant information of the target item to the client.

[0186] In the embodiment of the present application, the item retrieval device 800 is used to obtain the clicked item identifier based on the second sample retrieval information sent by the sample client; wherein the clicked item identifier is used to represent the identification information of the clicked item;

[0187] Determine the clicked item and the non-clicked item from a plurality of sample target items based on the clicked item identifier, and form hard training labels with the clicked item and the non-clicked item through the second sample retrieval information; wherein the plurality of sample target items are items fed back to the sample client in response to the second retrieval information sent by the sample client;

[0188] Training the intermediate model based on the hard training labels and the soft training labels determined during the training of the hard training labels, and stopping until a second training condition is met to obtain the preset model;

[0189] Among them, the soft training label includes: the second sample retrieval information, predicted click items and predicted non-click items; the predicted click items and the predicted non-click items are obtained by distributing the multiple sample target items based on the sample probability distribution; the sample probability distribution is determined based on the parameter update information of the hard training label during the training of the intermediate model.

[0190] In the embodiment of the present application, the object retrieval device 800 is used to use the initial model to process the first sample retrieval information to obtain a first sample retrieval feature, process the text information to obtain a first text feature, process the image information to obtain a first image feature, and fuse the first text feature and the first image feature to obtain a first sample image-text feature; wherein the image information belongs to the image-text information corresponding to the object;

[0191] The initial model is gradient updated based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image and text feature, until the initial model reaches a first predetermined number of training times, thereby obtaining the intermediate model.

[0192] In the embodiment of the present application, the initial model includes: an initial retrieval model and an initial image-text fusion model; the item retrieval device 800 is used to synchronously perform gradient updates on the initial retrieval model and the initial image-text fusion model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image-text feature, respectively, until the initial retrieval model and the initial image-text fusion model reach a first predetermined number of training times, and then stop to obtain an intermediate retrieval model and an intermediate image-text fusion model;

[0193] The intermediate retrieval model and the intermediate image-text fusion model are combined to determine the intermediate model.

[0194] In the embodiment of the present application, the item retrieval device 800 is used to process the second sample retrieval information sent by the sample client based on the intermediate model to obtain a second sample retrieval feature;

[0195] Based on the similarity between the second sample retrieval feature and the intermediate image-text fusion feature corresponding to each item, a plurality of sample target items are determined from the plurality of items; the intermediate image-text fusion feature is obtained by processing the image-text information of each item based on the intermediate model;

[0196] Feedback the plurality of sample target items to the sample client, and obtain the clicked item identifier sent by the sample client; wherein the clicked item identifier is sent by the sample client in response to instructions for the plurality of sample target items.

[0197] In the embodiment of the present application, the item retrieval device 800 is used to calculate the first contrast loss of the intermediate model using the second sample retrieval information and the graphic information of the clicked item and the non-clicked item, and extract parameter update information of the intermediate model in the training process for the hard training label;

[0198] Obtaining the sample probability distribution determined by processing the parameter update information by the network momentum model, and dividing the plurality of sample target items based on the sample probability distribution to obtain the predicted click items and the predicted non-click items;

[0199] Traversing the clicked items, the non-clicked items, the predicted clicked items and the predicted non-clicked items to determine a pair of items with the same item identifier, performing weighted averaging based on the sample labels corresponding to the pair of items, and determining a new sample target item and a new sample label corresponding to the item identifier, until the traversal is completed to obtain a plurality of the new sample target items and the new sample label corresponding to each of the new sample target items;

[0200] Calculating the intermediate model based on the new sample label and the graphic information corresponding to each of the new sample target items to obtain a second contrast loss;

[0201] The intermediate model is gradient updated based on the first contrast loss and the second contrast loss until the intermediate model reaches a second predetermined number of training times, thereby obtaining the preset model.

[0202] In the embodiment of the present application, the intermediate model includes: an intermediate retrieval model and an intermediate image-text fusion model; the item retrieval device 800 is used to synchronously perform gradient updates on the intermediate retrieval model and the intermediate image-text fusion model based on the first contrast loss and the second contrast loss until the intermediate retrieval model and the intermediate image-text fusion model reach a second predetermined training number, thereby obtaining a preset retrieval model and a preset image-text fusion model;

[0203] The preset retrieval model and the preset image-text fusion model are combined to determine the preset model.

[0204] In the embodiment of the present application, the determination unit 802 in the item search device 800 is used to determine the similarity between the search feature and the image-text fusion feature corresponding to each item;

[0205] Based on the similarity between the search feature and the image-text fusion feature corresponding to each of the items, the multiple items are sorted to obtain an item sequence; wherein the multiple items in the item sequence are arranged in order according to the similarity;

[0206] The first N items in the item sequence are determined to be the target items; N is an integer.

[0207] In the embodiment of the present application, the preset model includes: a preset image-text fusion model; the item retrieval device 800 is used to process the image-text information of each item based on the preset image-text fusion model to obtain the image-text fusion feature.

[0208] In the embodiment of the present application, the preset model includes: a preset retrieval model; the item retrieval device 800 is used to process the retrieval information and the user information based on the preset retrieval model to obtain the retrieval feature.

[0209] It should be noted that in the embodiments of the present application, if the above-mentioned item retrieval method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for enabling an item retrieval device (which can be a personal computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0210] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the server-side method are implemented.

[0211] Correspondingly, an embodiment of the present application provides an electronic device 900, including a memory 902 and a processor 901, wherein the memory 902 stores a computer program that can be executed on the processor 901, and the sample retrieval information processor 901 implements the steps in the above method when executing the program.

[0212] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0213] It should be noted that Fig.14 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.14 As shown, the hardware entity of the electronic device 900 includes: a processor 901 and a memory 902, wherein;

[0214] The processor 901 generally controls the overall operation of the electronic device 900 .

[0215] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or processed by the processor 901 and various modules in the electronic device 900 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0216] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0217] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0218] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0219] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0220] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0221] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes.

[0222] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a mobile storage device, a ROM, a magnetic disk, or an optical disk.

[0223] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An item retrieval method, characterized in that: include: Acquire the search information sent by the client, and process the search information and corresponding user information based on a preset model to obtain search features; Based on the similarity between the search feature and the image-text fusion feature corresponding to each item, a target item is determined from the multiple items; the image-text fusion feature is obtained by processing the image-text information of each item based on the preset model; The preset model is obtained by training based on the fusion of the graphic information corresponding to the item and the graphic information of the clicked item; the clicked item is determined by the user based on the sample target item fed back; Send the relevant information of the target item to the client.

2. The article retrieval method according to claim 1, characterized in that: Before obtaining the search information sent by the client and processing the search information and the corresponding user information based on a preset model to obtain the search features, the method further includes: Based on the first sample retrieval information intercepted from the text information corresponding to the item and the graphic information corresponding to the item, the initial model is trained until a first training condition is reached, thereby obtaining an intermediate model; wherein the text information belongs to the graphic information corresponding to the item; Acquire a clicked item identifier based on the second sample retrieval information sent by the sample client; wherein the clicked item identifier is used to represent identification information of the clicked item; Determine the clicked item and the non-clicked item from a plurality of sample target items based on the clicked item identifier, and form hard training labels with the clicked item and the non-clicked item through the second sample retrieval information; wherein the plurality of sample target items are items fed back to the sample client in response to the second retrieval information sent by the sample client; Training the intermediate model based on the hard training labels and the soft training labels determined during the training of the hard training labels, and stopping until a second training condition is met to obtain the preset model; Among them, the soft training label includes: the second sample retrieval information, predicted click items and predicted non-click items; the predicted click items and the predicted non-click items are obtained by distributing the multiple sample target items based on the sample probability distribution; the sample probability distribution is determined based on the parameter update information of the hard training label during the training of the intermediate model.

3. The article retrieval method according to claim 2, characterized in that: The initial model is trained based on the first sample retrieval information intercepted from the text information corresponding to the item and the graphic information corresponding to the item until the first training condition is reached to obtain an intermediate model, including: The first sample retrieval information is processed by the initial model to obtain a first sample retrieval feature, the text information is processed to obtain a first text feature, the image information is processed to obtain a first image feature, and the first text feature and the first image feature are fused to obtain a first sample image-text feature; wherein the image information belongs to the image-text information corresponding to the item; The initial model is gradient updated based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature and the first sample image and text feature, until the initial model reaches a first predetermined number of training times, thereby obtaining the intermediate model.

4. The article retrieval method according to claim 3, characterized in that: The initial model includes: an initial retrieval model and an initial image-text fusion model; The step of performing gradient updating on the initial model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature, and the first sample image-text feature, until the initial model reaches a first predetermined number of training times, and obtaining the intermediate model, comprises: Synchronously performing gradient update on the initial retrieval model and the initial image-text fusion model based on the contrast loss between the first sample retrieval feature and the first text feature, the first image feature, and the first sample image-text feature, until the initial retrieval model and the initial image-text fusion model reach a first predetermined number of training times, thereby obtaining an intermediate retrieval model and an intermediate image-text fusion model; The intermediate retrieval model and the intermediate image-text fusion model are combined to determine the intermediate model.

5. The article retrieval method according to claim 2, characterized in that: The step of obtaining the clicked item identifier based on the second sample retrieval information sent by the sample client includes: Processing the second sample retrieval information sent by the sample client based on the intermediate model to obtain a second sample retrieval feature; Based on the similarity between the second sample retrieval feature and the intermediate image-text fusion feature corresponding to each item, a plurality of sample target items are determined from the plurality of items; the intermediate image-text fusion feature is obtained by processing the image-text information of each item based on the intermediate model; Feedback the plurality of sample target items to the sample client, and obtain the clicked item identifier sent by the sample client; wherein the clicked item identifier is sent by the sample client in response to instructions for the plurality of sample target items.

6. The article retrieval method according to claim 2, characterized in that: The step of training the intermediate model based on the hard training labels and the soft training labels determined during the training of the hard training labels until a second training condition is met to obtain the preset model includes: Using the second sample retrieval information and the graphic information of the clicked item and the non-clicked item, the intermediate model is calculated to obtain a first contrast loss, and parameter update information of the intermediate model during the training process for the hard training label is extracted; Obtaining the sample probability distribution determined by processing the parameter update information by the network momentum model, and dividing the plurality of sample target items based on the sample probability distribution to obtain the predicted click items and the predicted non-click items; Traversing the clicked items, the non-clicked items, the predicted clicked items and the predicted non-clicked items to determine a pair of items with the same item identifier, performing weighted averaging based on the sample labels corresponding to the pair of items, and determining a new sample target item and a new sample label corresponding to the item identifier, until the traversal is completed to obtain a plurality of the new sample target items and the new sample label corresponding to each of the new sample target items; Calculating the intermediate model based on the new sample label and the graphic information corresponding to each of the new sample target items to obtain a second contrast loss; The intermediate model is gradient updated based on the first contrast loss and the second contrast loss until the intermediate model reaches a second predetermined number of training times, thereby obtaining the preset model.

7. The article retrieval method according to claim 6, characterized in that: The intermediate model includes: an intermediate retrieval model and an intermediate image-text fusion model; the gradient updating of the intermediate model based on the first contrast loss and the second contrast loss is stopped until the intermediate model reaches a second predetermined training number, and the preset model is obtained, including: Synchronously performing gradient update on the intermediate retrieval model and the intermediate image-text fusion model based on the first contrast loss and the second contrast loss until the intermediate retrieval model and the intermediate image-text fusion model reach a second predetermined training number, thereby obtaining a preset retrieval model and a preset image-text fusion model; The preset retrieval model and the preset image-text fusion model are combined to determine the preset model.

8. The article retrieval method according to any one of claims 1 to 7, characterized in that: The step of determining a target item from a plurality of items based on the similarity between the search feature and the image-text fusion feature corresponding to each item includes: Determine the similarity between the search feature and the image-text fusion feature corresponding to each of the items; Based on the similarity between the search feature and the image-text fusion feature corresponding to each of the items, the multiple items are sorted to obtain an item sequence; wherein the multiple items in the item sequence are arranged in order according to the similarity; The first N items in the item sequence are determined to be the target items; N is an integer.

9. The object retrieval method according to any one of claims 1 to 7, characterized in that: The preset model includes: a preset image-text fusion model; Before determining the target item from the plurality of items based on the similarity between the search feature and the image-text fusion feature corresponding to each item, the method further includes: The image-text fusion feature is obtained by processing the image-text information of each of the items based on the preset image-text fusion model.

10. The article retrieval method according to any one of claims 1 to 7, characterized in that: The preset model includes: a preset retrieval model; The step of processing the search information and the corresponding user information based on a preset model to obtain search features includes: The search information and the user information are processed based on the preset search model to obtain the search feature.

11. An item retrieval device, characterized in that: include: An information processing unit, used to obtain the search information sent by the client, and process the search information and corresponding user information based on a preset model to obtain search features; A determination unit, configured to determine a target item from a plurality of items based on the similarity between the search feature and a graphic-text fusion feature corresponding to each item; the graphic-text fusion feature is obtained by processing the graphic-text information of each item based on the preset model; The preset model is obtained by training based on the fusion of the graphic information corresponding to the item and the graphic information of the clicked item; the clicked item is determined by the user based on the sample target item fed back; A sending unit is used to send the relevant information of the target item to the client.

12. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and the sample retrieval information processor implements the steps of the method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 10 are implemented.

Images