Image-text retrieval method and device

Abstract

The invention provides an image-text retrieval method and device, and relates to the technical field of information retrieval, and the method comprises the steps: obtaining a retrieval request of a target retrieval platform, and determining a target account, a retrieval text corresponding to the target account and a user feature corresponding to the target account according to the retrieval request; the target account, the retrieval text and the user features are input into a recall model, a recall data set corresponding to the target retrieval platform and output by the recall model is obtained, and the recall model is used for screening out data with multiple targets matched with a first preset condition set from the candidate pool; the retrieval text, the user features and the recalled data set are input into a fine arrangement model, a target data set is obtained, the fine arrangement model is used for screening out data with multiple targets matched with a second preset condition set from the recalled data set, and the screened data are sorted; and sending the target data set to a target account corresponding to the target retrieval platform. And the balance of the image-text retrieval method in multiple targets is improved.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a method and device for image and text retrieval. Background Art

[0002] Image and text retrieval refers to the technology of obtaining information through mutual retrieval between images and texts. Specifically, it includes searching for texts by images and searching for images by texts. In a personalized search system, relevance is the basis for a good user experience, and conversion rate (including click-through rate and download volume) is the key to system revenue. Therefore, after a user searches, content with high relevance and conversion rate needs to be displayed to the user first in order to improve the user experience or increase revenue.

[0003] The image and text retrieval methods in the related technology usually only consider relevance, or only consider click-through rate, or only consider download volume, that is, only consider one of them, but this image and text retrieval method has no way to achieve a balance between multiple goals (i.e. relevance, click-through rate, download volume, etc.), that is, only consider one of the multiple goals. For example, the image and text retrieval method that only considers relevance, although it improves the user experience, will cause a decrease in revenue. If only the conversion rate is considered, the relevance will decrease, thereby bringing a bad experience to the user.

[0004] With regard to the problem that the existing image and text retrieval methods cannot achieve a balance among multiple objectives, no effective technical solution has been proposed so far. Summary of the invention

[0005] The embodiments of the present application provide a method and device for image-text retrieval, so as to at least solve the problem that the image-text retrieval method in the prior art cannot achieve a balance among multiple objectives.

[0006] According to one aspect of an embodiment of the present application, a graphic and text retrieval method is provided, including: obtaining a retrieval request of a target retrieval platform, and determining a target account, a retrieval text corresponding to the target account, and a user feature corresponding to the target account according to the retrieval request; inputting the target account, the retrieval text, and the user feature into a recall model to obtain a recall data set output by the recall model and corresponding to the target retrieval platform, wherein the recall model is a model obtained by training an initial recall model using a first sample set, and the recall model is used to filter out data whose multiple targets all match a first preset condition set from a candidate pool; inputting the retrieval text, the user feature, and the recall data set into a refined ranking model to obtain a target data set, wherein the refined ranking model is a model obtained by training an initial refined ranking model using a second sample set, and the refined ranking model is used to filter out data whose multiple targets all match a second preset condition set from the recall data set, and sort the filtered data; and sending the target data set to the target account corresponding to the target retrieval platform.

[0007] According to another aspect of an embodiment of the present application, a graphic and text retrieval device is also provided, including: an acquisition unit, used to acquire a retrieval request of a target retrieval platform, and determine a target account, a retrieval text corresponding to the target account, and a user feature corresponding to the target account according to the retrieval request; a first input unit, used to input the target account, the retrieval text, and the user feature into a recall model, and obtain a recall data set corresponding to the target retrieval platform output by the recall model, wherein the recall model is a model obtained by training an initial recall model using a first sample set, and the recall model is used to filter out data whose multiple targets all match a first preset condition set from a candidate pool; a second input unit, used to input the retrieval text, the user feature, and the recall data set into a fine ranking model, and obtain a target data set, wherein the fine ranking model is a model obtained by training an initial fine ranking model using a second sample set, and the fine ranking model is used to filter out data whose multiple targets all match a second preset condition set from the recall data set, and sort the filtered data; a sending unit, used to send the target data set to the target account corresponding to the target retrieval platform.

[0008] Optionally, the above-mentioned image and text retrieval device also includes a first training unit, which is used to train the initial recall model using a first sample set; the first training unit includes: obtaining multiple first search texts of multiple first search accounts within a first preset time period, and a first click sequence and a first download sequence corresponding to each first search text, wherein the click information included in the first click sequence is used to indicate the information of the clicked image, and the download information included in the first download sequence is used to indicate the information of the downloaded image corresponding to the first search text; a fine-tuning unit, which fine-tunes the initial multimodal correlation model according to the multiple first search texts, the multiple first click sequences and the multiple first download sequences to obtain a target multimodal correlation model; a first training subunit, which is used to construct a first sample set, and train the initial recall model according to the first sample set and the target multimodal correlation model.

[0009] Optionally, the above-mentioned first training subunit includes: a first acquisition module, which is used to acquire multiple second search texts within a second preset time period, and determine multiple second search accounts based on the second search texts, wherein the first sample set includes multiple second search texts; a first determination module, which is used to determine the second click sequence and the second download sequence corresponding to each second search account based on the multiple second search accounts; a first input module, which is used to input the second click sequence into the target multimodal correlation model, output the click text feature set and the click visual feature set corresponding to the second click sequence, and input the second download sequence into the target multimodal correlation model, and output the download text feature set and the download visual feature set corresponding to the second download sequence; an averaging module, which is used to average the click text feature set, the click visual feature set, the download text feature set and the download visual feature set, respectively, to obtain the click text average feature set, the click visual average feature set, the download text average feature set and the download visual average feature set included in the first sample set; a second acquisition module, which is used to acquire the display information set corresponding to the target retrieval platform, and input the display information set into the target multimodal correlation model, and output the display text feature set and the display visual feature set included in the first sample set.

[0010] Optionally, the above-mentioned first training sub-unit includes a third acquisition module, which is used to obtain an initial recall model constructed using an end-to-end dual-tower multi-objective modeling method; a second input module, which is used to input multiple second retrieval texts, click text average feature sets, click visual average feature sets, download text average feature sets and download visual average feature sets into the user sub-network of the initial recall model to obtain user feature vector representations corresponding to each of the multiple second retrieval accounts; a third input module, which is used to input display text feature sets and display visual feature sets into the item sub-network of the initial recall model to obtain item feature vector representations corresponding to each of the multiple displayed items in the target retrieval platform; and an iterative training module, which is used to iteratively train the initial recall model according to the user feature vector representation and the item feature vector representation.

[0011] Optionally, the second input module includes: a feature extraction module, which is used to extract text features from multiple second retrieval texts using a target multimodal correlation model in the user subnetwork to obtain text features; a first splicing module, which is used to splice the text features with the click text average feature set, the click visual average feature set, the downloaded text average feature set and the downloaded visual average feature set to obtain spliced ​​features; a normalization module, which is used to normalize the spliced ​​features and process the normalized features through a multi-layer perceptron to obtain a user feature vector representation.

[0012] Optionally, the third input module includes: a keyword extraction module, which is used for the item sub-network to perform keyword extraction on the display text features in the display text feature set to obtain a core text feature set; an extraction module, which is used for the target multimodal correlation model in the item sub-network to perform semantic extraction on the core text feature set to obtain a semantic feature set; a second splicing module, which is used to splice the semantic feature set and the display visual feature set to obtain a display splicing feature; then a normalization module is used to normalize the display splicing features, and the normalized display splicing features are processed by a multi-layer perceptron to obtain an item feature vector representation.

[0013] Optionally, the above-mentioned first iterative training module includes: an acquisition submodule, used to obtain the cosine similarity between the user feature vector representation and the item feature vector representation; a multiplication calculation submodule, used to multiply the cosine similarity with the target hyperparameter in the initial recall model to obtain the multiplication result; a loss value calculation submodule, used to input the multiplication result into the target loss function for loss value calculation to obtain the model loss value; an iterative training submodule, used to iteratively train the initial recall model according to the model loss value.

[0014] Optionally, the above-mentioned image and text retrieval device also includes: a sample construction unit for constructing a second sample set; the sample construction unit includes: a first acquisition subunit for acquiring multiple third search texts of multiple third search accounts within a third preset time period, and determining a search text feature set based on the multiple third search texts, wherein the second sample set includes a search text feature set; a second acquisition subunit for acquiring user features corresponding to each of the multiple third search accounts, wherein the second sample set includes multiple user features, each user feature includes a third click identification sequence, a third download identification sequence, and permission information of multiple third search accounts on multiple search platforms within the third preset time period of the corresponding third search account; a third acquisition subunit for acquiring statistical feature sets within multiple sub-preset time periods within the third preset time period, and scene features corresponding to each of the multiple search platforms, wherein the second sample set includes a statistical feature set and multiple scene features, and each statistical feature in the statistical feature set is used to indicate the weight of the displayed image in multiple search platforms within the third preset time period.

[0015] Optionally, the above-mentioned image and text retrieval device further includes a second training unit, which is used to train the initial refined ranking model using a second sample set.

[0016] Optionally, the second training unit includes: a deep cross calculation subunit, which is used for the initial fine ranking model to perform deep cross calculation on multiple sample feature subsets in the second sample set to obtain multiple attention weight results, and the multiple attention weight results are processed by multiple different multi-layer perceptrons to obtain attention perception results, wherein the deep difference calculation is used to calculate the attention weights of different sample feature subsets; a fourth acquisition subunit, which is used to obtain the cold start feature sets corresponding to each of the multiple retrieval platforms, and the multiple cold start feature sets are processed by multiple different multi-layer perceptrons to obtain cold start perception results, wherein the cold start features in the cold start feature sets are used to indicate that they have not been displayed within a third preset time period. an image given to the user; a fifth acquisition subunit, used to obtain scene features corresponding to each of the multiple retrieval platforms, and process the multiple scene features through a first multi-layer perceptron to obtain a scene perception result, wherein the first multi-layer perceptron is different from multiple different multi-layer perceptrons; a splicing subunit, used to splice the attention perception result, the cold start feature set, and the scene perception result to obtain a target splicing result, and process the target splicing result through a second multi-layer perceptron to obtain an output result of the initial refined ranking model, wherein the second multi-layer perceptron is different from multiple different multi-layer perceptrons and the first multi-layer perceptron; a second iterative training module, used to iteratively train the initial refined ranking model according to the output result.

[0017] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above image and text retrieval method.

[0018] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the above-mentioned image and text retrieval method.

[0019] The above-mentioned image-text retrieval method solves the problem that the image-text retrieval method in the prior art cannot achieve balance among multiple objectives, thereby improving the balance of the image-text retrieval method among multiple objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a schematic diagram of a hardware environment of a graphic and text retrieval method according to an embodiment of the present invention;

[0022] Figure 2 The figure is a flow chart of an optional image-text retrieval method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the technical personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiment of the present application will be clearly and completely described in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only an embodiment of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work should belong to the scope of protection of the present application. The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. The terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described below with reference to the drawings and in conjunction with the embodiments.

[0024] Relevance and conversion rate are both goals in the multi-goals mentioned in this application, and in addition to relevance and conversion rate, there are many other goals in the multi-goals, which will not be discussed one by one here. Multiple goals are very critical. If only relevance is considered, although the user experience is good, the goals related to revenue may not be paid attention to. If only conversion rate is considered, the relevance will decrease, which will give users a bad experience, and long-term use will also have a bad impact on the system that implements the image and text retrieval method. If you want to ensure that content with better multiple goals is displayed to users first throughout the entire link, it often involves a balance between multiple goals, and the image and text retrieval method in the prior art has no way to achieve a balance among multiple goals.

[0025] In order to solve the above problems, the embodiment of the present application provides a set of full-link consistency modeling algorithms (graphic and text retrieval methods), which follow consistent modeling goals from recall to refined ranking links to ensure efficient conversion of the funnel at each stage. Through online AB experimental comparison, the conversion of the entire link is significantly improved, and the low feedback rate of multiple targets is effectively reduced. As an optional implementation method, the graphic and text retrieval method can be applied to, but is not limited to, Figure 1In the picture and text retrieval system composed of the terminal device 102 and the server 104 shown in the figure, the terminal device 102 is connected to the server 104 via the network 110. The terminal device 102 is also provided with a display 106, a processor 108 and a memory 112. The display 106 can be used to display the target data set, etc., the processor 108 can be used to process the sample data (the first sample set, the second sample set, etc.), etc., and the memory 112 can be used for the sample data, the target data set, etc. After the terminal device 102 executes S102, that is, sends the search request to the server 104 via the network 110, the server 104 can execute the following steps: Figure 1 S104 to S112 shown are performed to complete the image and text retrieval.

[0026] The server 104 may be a single server, a server cluster consisting of multiple servers, or a cloud server. The server 104 includes a database 114 and a processing engine 116. The database 114 may be used to store the first sample set, the second sample set, the target data set, etc., and the processing engine 116 may be used to process the first sample set, the second sample set, etc.

[0027] In the above embodiment of the present invention, the above-mentioned image-text retrieval method is adopted to improve the balance of the image-text retrieval method in multiple objectives, and achieve retrieval results with high multiple objectives for user recommendation.

[0028] The above is only an example and is not limited in this embodiment.

[0029] As an optional implementation, please refer to Figure 2 , which shows a flow chart of a method for image and text retrieval provided by an embodiment of the present application. The method may include at least one of the following steps:

[0030] S202, obtaining a search request from a target search platform, and determining a target account, a search text corresponding to the target account, and a user feature corresponding to the target account according to the search request; S204, inputting the target account, the search text, and the user feature into a recall model, and obtaining a recall data set output by the recall model and corresponding to the target search platform, wherein the recall model is a model obtained by training an initial recall model with a first sample set, and the recall model is used to filter out data whose multiple targets all match a first preset condition set from a candidate pool; S206, inputting the search text, the user feature, and the recall data set into a refined ranking model, and obtaining a target data set, wherein the refined ranking model is a model obtained by training an initial refined ranking model with a second sample set, and the refined ranking model is used to filter out data whose multiple targets all match a second preset condition set from the recall data set, and sort the filtered data; S208, sending the target data set to the target account corresponding to the target search platform.

[0031] It should be noted that the target search platform in the above S202 is used to indicate the platform where the user can search. The target search platform includes a variety of applications with search functions (various APPs including search functions, such as shopping platforms, ordering platforms, etc., and the corresponding scenarios of different platforms may be different, such as the shopping platform corresponding to the scene is the commodity screening scene, and the ordering platform corresponding to the scene is the dining scene, etc.), or various search web pages, etc. The target account is the account of the user logging into the target search platform, the search text is the search text entered by the user (also known as the search term), and the user feature is used to indicate the user's preferences / preferences. It can be understood that the user feature is the user's feature on the target search platform (the user feature is the feature corresponding to the target search platform), and the user feature can be determined based on the user's historical search information on the target search platform, and the historical search information includes historical search text, historical click text, historical download text, etc.

[0032] The role of the recall model in S204 is to screen out rough screening results (i.e., recall data sets) of hundreds to tens of millions with high relevance and high conversion rate from a candidate pool of hundreds of millions. The candidate pool in S204 is all the data to be screened in the target retrieval platform. The candidate pool can be understood as including all the data in the target retrieval platform. When the recall model performs rough screening, it screens out some data that meet the conditions from the candidate pool, thereby obtaining a recall data set. The first preset condition set includes multiple first preset conditions, each target corresponds to a first preset condition, and multiple targets are matched with the first preset condition set, which can be understood, but not limited to, that each target satisfies its corresponding first preset condition; for example, there are three targets: relevance, click-through rate, and download rate. The relevance satisfies the first preset condition corresponding to the relevance, the click-through rate satisfies the first preset condition corresponding to the click-through rate, and the download rate satisfies the first preset condition corresponding to the download rate.

[0033] The refined ranking model in S206 is used to filter out data from the recall data set where multiple targets all match the second preset condition set. The second preset condition set includes multiple second preset conditions, each target corresponds to a second preset condition, and multiple targets all match the second preset condition set, which can be understood as, but not limited to, each target satisfies its corresponding second preset condition; if there are three targets, namely, relevance, click-through rate, and download rate, the relevance satisfies the second preset condition corresponding to the relevance, the click-through rate satisfies the second preset condition corresponding to the click-through rate, and the download rate satisfies the second preset condition corresponding to the download rate. Each first preset condition corresponds to a second preset condition, and each second preset condition is higher than its corresponding first preset condition, such as the first preset condition corresponding to the relevance is 60%, and the second preset condition corresponding to the relevance is 80%, that is, the second preset condition is more stringent than the first preset condition in screening data. The refined ranking stage corresponding to the refined ranking model obtains multiple data (such as obtaining a candidate pool of top ten to one hundred levels) that meet higher screening conditions for the candidate set (recall data set) of hundreds to tens of millions obtained in the recall stage corresponding to the recall model through a more complex sorting model (refined ranking model). The target data set is the data output after screening and sorting by the refined ranking model.

[0034] Through the above-mentioned implementation of the present application, the balance of the image and text retrieval method in multiple objectives is improved, and the retrieval results with high multiple objectives are achieved for user recommendations.

[0035] As an optional implementation, the initial recall model is trained using a first sample set, including: S1, obtaining multiple first search texts of multiple first search accounts within a first preset time period, and a first click sequence and a first download sequence corresponding to each first search text, wherein the click information included in the first click sequence is used to indicate information of a clicked image, and the download information included in the first download sequence is used to indicate information of a downloaded image corresponding to the first search text; S2, fine-tuning the initial multimodal correlation model according to the multiple first search texts, the multiple first click sequences, and the multiple first download sequences to obtain a target multimodal correlation model; S3, constructing a first sample set, and training the initial recall model according to the first sample set and the target multimodal correlation model.

[0036] When the scheme is implemented, the recall model is a model that is trained to perform accurate and rough screening. However, before use, the initial recall model needs to be trained to obtain the recall model, and S1 to S2 is the process of training the initial recall model. The first preset time period in S1 is, for example, within 15 days or 7 days. The multiple first search texts are search texts of multiple first search accounts on the target search platform. Each first search account corresponds to a first search text. When the user performs a search, at least one search result will be displayed to the user on the target search platform. The above-mentioned first click sequence can be but is not limited to being understood as the user's click record information on at least one search result corresponding to the first search text. Each search result in the at least one search result will have a click status identifier corresponding to the click, and the click status identifier is used to indicate whether the corresponding search result is clicked by the user. If product A is clicked by the user, then the click status identifier corresponding to product A is 1. If product B is not clicked by the user, then the click status identifier corresponding to product B is 0. Each search result in the at least one search result also includes the number of clicks of the search result in the first preset time period. For example, in the first preset time period, product A is clicked 3 times by user a and 1 time by user b, then the number of clicks corresponding to product A is 4.

[0037] The first download sequence can be understood as, but not limited to, the download record information of the user for at least one search record corresponding to the first search text. Each search result in the at least one search result will have a download status mark corresponding to download, and the download status mark is used to indicate whether the corresponding search result is downloaded by the user. If product A is downloaded by the user, the download status mark corresponding to product A is 1. If product B is not downloaded by the user, the download status mark corresponding to product B is 0. If the user clicks and downloads product A when searching using the q search text, the comprehensive mark corresponding to product A in the first click sequence and the first download sequence corresponding to the q search text is 11. If product A is clicked but not downloaded, the comprehensive mark corresponding to product A in the first click sequence and the first download sequence corresponding to the q search text is 10. Each search result in the at least one search result also includes the number of downloads of the search result in the first preset time period. For example, in the first preset time period, product A is downloaded 2 times by user a and 1 time by user b, and the number of downloads corresponding to product A is 3.

[0038] The information of the clicked image in S1 includes the clicked image and text (the image information of the clicked image corresponding to the first search text and the text information corresponding to the image), and the information of the downloaded image includes the downloaded image and text (the image information of the downloaded image corresponding to the first search text and the text information corresponding to the image).

[0039] The initial multimodal correlation model in S2 (the open source text-image search model) uses the open source pre-trained chinese-clip model parameters (the initial parameters of the model before training the initial multimodal correlation model) as the basic model parameters, and then uses the target retrieval platform to fine-tune the search terms (first search text) and / or search images in the user's click and download logs within a first preset time period (such as one month) (when training / fine-tuning the initial multimodal correlation model, the search image can also be input for training / fine-tuning) to obtain the target multimodal correlation model. The operations in S1 to S2 can be understood as first keeping the parameters of the open source model unchanged, and then fine-tuning the model using samples within the first preset time period. Fine-tuning is to adjust the parameters in the model to obtain the target multimodal correlation model.

[0040] Through the above-mentioned implementation of the present application, a target multimodal correlation model can be trained, so that the target multimodal correlation model can be conveniently used in the subsequent training of the recall model.

[0041] As an optional implementation, constructing a first sample set includes: S1, obtaining multiple second search texts within a second preset time period, and determining multiple second search accounts based on the second search texts, wherein the first sample set includes multiple second search texts; S2, determining the second click sequence and the second download sequence corresponding to each second search account based on the multiple second search accounts; S3, inputting the second click sequence into the target multimodal correlation model, outputting the click text feature set and the click visual feature set corresponding to the second click sequence, and inputting the second download sequence into the target multimodal correlation model, and outputting the download text feature set and the download visual feature set corresponding to the second download sequence; S4, averaging the click text feature set, the click visual feature set, the download text feature set and the download visual feature set, respectively, to obtain the click text average feature set, the click visual average feature set, the download text average feature set and the download visual average feature set included in the first sample set; S5, obtaining the display information set corresponding to the target retrieval platform, and inputting the display information set into the target multimodal correlation model, and outputting the display text feature set and the display visual feature set included in the first sample set.

[0042] The first sample set is a training sample for training the initial recall model. The second preset time period in S1 can be the same as or different from the first preset time period. The multiple second search texts are search texts of the target search platform within the second preset time period. The search accounts corresponding to each second search text may be the same or different. Therefore, when determining multiple second search accounts, the second search account corresponding to each second search text can be determined first, and then only one of the same second search accounts in the multiple second search accounts is retained, thereby obtaining multiple second search accounts in S1. The second click sequence and the second download sequence can be understood as, but not limited to, the click record information and download record information corresponding to each second search account. The click record information also includes the clicked picture information, the text information corresponding to the picture, the click status identifier, the number of clicks, etc.; the download record information also includes the downloaded picture information, the text information corresponding to the picture, the download status identifier, the number of downloads, etc. The click text feature set in S3 includes the text features corresponding to the click record, and the click visual feature includes the visual features corresponding to the click record. The text feature can be understood as, but not limited to, the summary text of the click record (for example, the description text of a certain picture), and the visual feature can be understood as, but not limited to, the visually observable feature. The data in the four sets of click text average feature set, click visual average feature set, download text average feature set and download visual average feature set in S4 are the features of the user corresponding to the second search account. The display information set in S5 is the data information that can be displayed to the user in the candidate pool corresponding to the target search platform. The display information set includes, for example, multiple pictures to be retrieved included in the candidate pool corresponding to the target search platform. The display text feature set includes text features corresponding to each type of data information, and the display visual feature set includes visual features corresponding to each type of data information. The above processes in S1 to S4 are the construction process of the first sample set. The following is a detailed example of S1 to S4:

[0043] The construction of the first sample set can be understood as selecting the user's click log within a period of time (i.e., the second preset time period) in the target retrieval platform as a click positive sample, selecting the download log within this period of time as a download positive sample, and the negative sample is randomly sampled from the candidate pool corresponding to the target retrieval platform as a global negative sample (i.e., screening some data from all the data in the candidate pool as a negative sample). The positive and negative samples can follow a certain ratio according to the task requirements, for example, the positive and negative sample ratio is 1:2, that is, the ratio of positive samples (click positive samples + download positive samples) to global negative samples. The role of randomly selecting negative samples from the candidate pool corresponding to the target retrieval platform is to ensure the consistency of data distribution during model training and reasoning. In addition, in order to avoid the influence of user's misclick and misdownload behavior on the relevance of the model, the present invention filters the positive samples whose multimodal similarity threshold between the search term and the image in the positive sample is lower than 0.1 through the target multimodal relevance model. (That is, in the process of model training, filtering while training). After obtaining the positive samples and global negative samples, they will be randomly scattered, and then the scattered positive samples and negative samples will be input into the recall model in batches, that is, a certain amount of samples are selected each time to be input into the recall model for training.

[0044] During the recall model training process, each positive sample will be subjected to random negative sampling and difficult sample sampling within the batch (that is, after batching, before each batch of samples is input into the initial recall model for training, random negative sampling and difficult sample sampling will be performed on the samples within the batch). Random negative sampling within the batch is to randomly select a part (such as 20) from the current batch as negative examples of the current positive sample. Difficult sample sampling is to calculate the similarity with the positive sample in the current batch, and take the most similar samples (such as the top 20 samples of similarity) except the current positive sample as difficult samples. The purpose of adding difficult samples is to input search terms so that the model can not only distinguish the differences between large categories, but also distinguish the differences between subtle images, thereby improving the relevance of the model. For example, if the search text is "white background", the random samples obtained by random negative sampling are to allow the recall model to distinguish multiple samples and determine whether each sample is background. Difficult samples are to allow the recall model to distinguish multiple samples that are backgrounds and determine whether each background sample is white background, because it may also be background samples of other colors such as green and yellow.

[0045] The construction process of the first sample set can be understood as a feature construction process. In the feature construction, it is divided into search term features (i.e., the second retrieval text), user features (user preference features, i.e., features of user preferences reflected by the click and download results of the user over a period of time) and item-side features (data features of all data to be retrieved in the candidate pool corresponding to the target retrieval platform (also called image features of the image to be retrieved), such as the features corresponding to the dishes for the ordering platform and the product features for the shopping platform). The recall model is input with the above three features + the click status identifier of whether each data to be retrieved in the candidate pool has been clicked, and / or the download status identifier of whether it has been downloaded (referred to as label in the subsequent description).

[0046] During the feature construction process, in order to ensure better generalization of the recall model, no ID features were used in the recall model training, and only text and image features with better generalization were used. The search term feature is the current search term (the second search text), and the user feature includes the text information and image information corresponding to the user's recent click item sequence (multiple click records) and download item sequence (multiple download records). The text information and image information use the text side and visual side of the target multimodal relevance model to extract the core features of each item (click record or download record) text (a sentence or several keywords representing the item) embedding (converting this sentence or several keywords into a vector representation) and visual (visual information that the user can see) embedding (the process of converting visual information into a vector), thereby obtaining multiple text embeddings (i.e., multiple click text features included in the click text feature set and multiple download text features included in the download text set) and multiple visual embeddings (i.e., multiple click visual features included in the click visual feature set and multiple download visual features included in the download visual feature set). Assuming that the user has clicked on three types of content in the past, namely kittens, puppies, and hello kitty, then the user will have two historical features, namely the average text embedding and the average visual embedding of the text descriptions of these three images. These two average embeddings represent the user's preferences (i.e., the above-mentioned user features. The four sets obtained in the above S4 are the user features mentioned in this application).

[0047] Then, the above S4 is executed, that is, the multiple text embeddings and multiple visual embeddings clicked by the user in the second preset time period are averaged respectively through the average pool to obtain the average feature set of clicked text, the average feature set of clicked visual, the average feature set of downloaded text, and the average feature set of downloaded visual. The item side features include text features and visual features, the text features are the core terms of the item (keywords in the text features), and the visual features are the visual features extracted from the visual side of the target multimodal correlation model; that is, the item side features can be understood, but not limited to, as the keywords (display text feature set) of the text information corresponding to each data to be retrieved in the candidate pool corresponding to the target retrieval platform and the visual features (display visual feature set) of the visual information corresponding to the data to be retrieved. In addition, in order to enable the recall model to learn more conversion-related features, the number of clicks, downloads, exposure click rate, exposure download rate, click-down rate (both clicks and downloads) and other statistical features corresponding to each data to be retrieved in different historical periods (for example, three days, five days, seven days, fifteen days, etc.) are added to the item side features.

[0048] Through the above implementation of the present application, an accurate and representative first sample set can be constructed for subsequent training of the recall model.

[0049] As an optional implementation, the initial recall model is trained according to the first sample set and the target multimodal correlation model, including: S1, obtaining the initial recall model constructed using an end-to-end dual-tower multi-target modeling approach; S2, inputting multiple second retrieval texts, click text average feature sets, click visual average feature sets, download text average feature sets, and download visual average feature sets into the user subnetwork of the initial recall model to obtain user feature vector representations corresponding to each of the multiple second retrieval accounts; S3, inputting the display text feature set and the display visual feature set into the item subnetwork of the initial recall model to obtain item feature vector representations corresponding to each of the multiple displayed items in the target retrieval platform; S4, iteratively training the initial recall model according to the user feature vector representation and the item feature vector representation.

[0050] The initial recall model in S1 is a model built using an end-to-end dual-tower multi-objective modeling approach, which is divided into a user tower (user subnetwork) and an item tower (item subnetwork). After multiple second retrieval texts, click text average feature sets, and click visual average feature sets are input into the user subnetwork of the initial recall model in S2, the user subnetwork will perform relevant data processing on the input data to obtain a user feature vector representation; after the display text feature set and the display visual feature set are input into the item subnetwork of the initial recall model in S3, the item subnetwork will also perform relevant data processing on the input data to obtain an item feature vector representation. The process of data processing of the input data by the user sub-network includes: S2-1, the target multimodal correlation model in the user sub-network extracts text features from multiple second search texts to obtain text features; S2-2, the text features are spliced ​​with the click text average feature set, the click visual average feature set, the download text average feature set and the download visual average feature set to obtain spliced ​​features; S2-3, the spliced ​​features are normalized, and the normalized features are processed by a multi-layer perceptron to obtain a user feature vector representation. The process of data processing of the input data by the item sub-network includes: S3-1, the item sub-network extracts keywords from the display text features in the display text feature set to obtain a core text feature set; S3-2, the target multimodal correlation model in the item sub-network extracts semantics from the core text feature set to obtain a semantic feature set; S3-3, the semantic feature set is spliced ​​with the display visual feature set to obtain a display splicing feature; S3-4, the display splicing features are normalized, and the normalized display splicing features are processed by a multi-layer perceptron to obtain an item feature vector representation.

[0051] The detailed process of S3 to S5 above is as follows: the user tower inputs the search word features (multiple second search texts) and user features (click text average feature set, click visual average feature set, download text average feature set and download visual average feature set) into the text feature extraction side of the target multimodal correlation model to extract the text embedding (i.e., text features) of the search word; then the text embedding is concatenated with the user features (for example, multiple 728-dimensional vectors are concatenated into a vector of more than three thousand dimensions) to obtain the concatenated features, and then the concatenated features are batch normalized (normalized) to obtain the above-mentioned user feature vector representation after passing through multiple MLP layers. The item tower first performs keyword extraction on multiple display text features in the display text feature set to obtain a core text feature set, and then performs semantic extraction on the core text feature set through a multimodal correlation model to obtain multiple semantic embeddings (i.e., semantic feature sets), and then concatenates the semantic feature set with the display visual feature set (and in this application, statistical features are also input into the itm tower, so during splicing, the statistical features will also be spliced) to obtain display splicing features, and then batch normalizes the display splicing features, and the batch normalized results are processed through multiple MLP (multi-layer perceptron) layers to obtain the above-mentioned item feature vector representation.

[0052] During the training process of the recall model, random negative sampling and difficult sample sampling are performed within the batch for each click positive sample and download positive sample. The sampled negative samples (including random negative samples and difficult samples) are spliced ​​(the samples obtained by random negative sampling and difficult sample sampling are all called negative samples, so the splicing here is to splice the positive samples, the negative samples of random negative sampling and the difficult samples), and then input into the initial recall model to obtain the user feature vector representation and the item feature vector representation, and then execute S4, S4 includes: S4-1, obtaining the cosine similarity between the user feature vector representation and the item feature vector representation; S4-2, multiplying the cosine similarity with the target hyperparameter in the initial recall model to obtain the multiplication result; S4-3, inputting the multiplication result into the target loss function to calculate the loss value to obtain the model loss value; S4-4, iteratively training the initial recall model according to the model loss value.

[0053] The operation in S4 can be understood as, but not limited to: calculating the cosine similarity between the user feature vector representation and the item feature vector representation, and then multiplying the calculated cosine similarity with the preset parameter (temperature) in the recall model to obtain the hyperparameter multiplication result, and then inputting the multiplication result into the target loss function (such as softmax_cross_entropy_logits_loss) to calculate the loss value, and obtaining the model loss value (indicating the process of model gradient descent). Each time the model is trained, a model loss value will be obtained. According to the model loss value obtained each time, the parameters in the initial recall model can be updated, and the training process of the above recall model can be repeated to obtain the recall model. The above preset parameter is a hyperparameter, the purpose of which is to speed up the convergence of the model. In this application, the preset parameter in the initial recall model is initially set to 0.01.

[0054] During use, the above recall model is used. For example, there are hundreds of millions of images in the candidate pool corresponding to the target retrieval platform. If the user's search text is "kitten", then the recall model will search for images including kittens in the hundreds of millions of images. Among the hundreds of millions of images, each image corresponds to an image text feature and an image visual feature (that is, a display text feature and a display visual feature). According to the search account corresponding to the search text, the user characteristics of the user corresponding to the search account on the target retrieval platform can be determined. Similarity retrieval is performed based on the user characteristics, image text features and image visual features. For example, the 1,000 images with the highest similarity that the user prefers are found, and these 1,000 images are the results output by the recall model.

[0055] In the process of training the initial recall model in this application, the model evaluation of the recall model is also included. How to effectively evaluate the effect of each iterative version of the model is the key to ensure the success of the experiment. The model mainly includes three evaluation indicators. The first is the indicator for correlation evaluation. The NDCG (Normalized Discounted Cumulative Gain, an indicator for evaluating sorting quality) is calculated in the manually labeled correlation samples to evaluate the correlation. The second is the evaluation of click and download conversion tasks. The click and download logs of the new day are selected as the actual results (groudtruth) and the hit rate and other indicators are calculated. The third is the consistency evaluation with the refined ranking model to ensure that the candidate set with excellent recall can also output excellent and accurate results.

[0056] Through the above-mentioned implementation methods of the present application, the recall model can be effectively trained and an accurate recall model can be obtained, laying a strong data foundation for the implementation process of the subsequent refined ranking model.

[0057] As an optional implementation, it also includes constructing a second sample set, and constructing the second sample set includes: S1, obtaining multiple third search texts of multiple third search accounts within a third preset time period, and determining a search text feature set based on the multiple third search texts, wherein the second sample set includes a search text feature set; S2, obtaining user features corresponding to each of the multiple third search accounts, wherein the second sample set includes multiple user features, each user feature includes a third click identification sequence, a third download identification sequence, and permission information of multiple third search accounts on multiple search platforms of the corresponding third search account within the third preset time period; S3, obtaining statistical feature sets within multiple sub-preset time periods within the third preset time period, and scene features corresponding to each of the multiple search platforms, wherein the second sample set includes a statistical feature set and multiple scene features, and each statistical feature in the statistical feature set is used to indicate the weight of the displayed image in multiple search platforms within the third preset time period.

[0058] Before using the refined ranking model in the refined ranking stage, the initial refined ranking model also needs to be trained. The main function of the refined ranking stage is to obtain, for example, the top ten to hundred candidate sets (i.e., target datasets) through a more complex sorting model for the candidate sets of, for example, hundreds to thousands in the recall stage (i.e., recall datasets). The training of the refined ranking model also includes relevance, click-through rate, and download rate.

[0059] It should be noted that the above-mentioned search text feature set (also referred to as search term feature) includes query_id (the id of each third search text), 1_gran (a set of segmented words corresponding to each third search text obtained by segmenting each third search text, and the position information of each word in the third search text is not changed after segmentation), 2_gram (a combination result set obtained by performing adjacent pairwise combinations of multiple segmented words in the segmented word set corresponding to each third text, such as the combination of the first word and the second word, the combination of the second word and the third word, but not including the combination of the first word and the third word), for example, the third search text For "red background elements", 1_gram is {red, background, elements} (that is, to ensure that the front and back position relationship between multiple words remains unchanged), 2_gram is {red background, background elements} (that is, multiple words in the word set are combined in order), but 2_gram does not include words with jumping position relationships such as "red elements". Jieba word segmentation can be used for word segmentation. The word segmentation principle of Jieba word segmentation is to use a Chinese word library, compare the content to be segmented with the word library, and find the phrase with the highest probability through graph structure and dynamic programming methods. In addition, the Jieba library also provides the function of adding custom Chinese words. Jieba word segmentation supports three word segmentation modes: 1. Precise mode, which tries to separate sentences most accurately, suitable for text analysis; 2. Full mode, which scans all words that can form words in the sentence, which is fast, but cannot resolve ambiguity; 3. Search engine mode, on the basis of precise mode, long words are segmented again to improve the recall rate, which is suitable for search engine word segmentation.

[0060] The user features in S2 above are features used to indicate user preferences, and the user features include the user's recent (third preset time period) click and download item_id sequence (i.e., the third click identification sequence and the third download identification sequence). It also includes clustering features of display visual features corresponding to clicks or downloads of items (i.e., each click record and download record). Clustering features, for example: each image (item) has its corresponding visual embedding (i.e., display visual features), and then a cluster is made based on multiple visual embeddings corresponding to multiple images, such as kittens are one category, puppies are one category, or plants are one category, animals are one category, and the clustered features are the above clustering features. Permission information can be, but is not limited to, understood as the permissions that a user has on the target retrieval platform, for example, ordinary users and VIP users have different permissions, that is, permission information can also be understood as the user's membership information (such as whether the user is a member, what type of member (e.g., VIP, SVIP)), etc.

[0061] The above statistical feature set includes multiple data to be searched included in the candidate pool corresponding to the target search platform, and the number of exposures, clicks, downloads, exposure click-through rate, click-to-download rate and exposure download rate, etc., corresponding to each data to be searched in different historical periods (for example, the past three days, the past five days, the past seven days, etc.). The scene features include a pit position identifier (indicating whether the user is searching on the homepage of the target search platform or searching on the special page) and a scene identifier (a dimension identifier corresponding to the third search text, such as whether the search is for an element or a template). The pit position identifier is, for example, a user's direct search on the homepage, or a user's search on the special page. In other words, the pit position identifier indicates the user's search interface on the target search platform, that is, whether the user enters the third search text as soon as he enters the target search platform, or switches to a fixed category. For example, like searching for TV series, in a certain APP, is the search directly on the homepage, or is the search conducted on special pages such as TV series / variety shows?

[0062] The positive samples of the click task included in the second sample set constructed above are the click logs of users in the target retrieval system (the click set corresponding to the third click identification sequence), and the negative samples are the logs that are exposed but not clicked (i.e., the results are displayed after each search by the user, but not clicked). The positive samples of the download task are the download logs in the target retrieval system (i.e., the download set corresponding to the third download identification sequence), and the negative samples are the logs that are exposed but not downloaded. Random negative sampling is performed on the click negative samples to ensure that the ratio of positive and negative samples of the click task is 1:2, thereby obtaining the click set (the set corresponding to the third click identification sequence) and the download set (the set corresponding to the third download identification sequence).

[0063] Through the above implementation of the present application, the second sample set required for initial refined sorting model training can be accurately constructed.

[0064] As an optional implementation, the second sample set is used to train the initial fine ranking model, including: S1, the initial fine ranking model performs deep cross calculation on multiple sample feature subsets in the second sample set to obtain multiple attention weight results, and the multiple attention weight results are processed by multiple different multi-layer perceptrons to obtain attention perception results, wherein the deep difference calculation is used to calculate the attention weights of different sample feature subsets; S2, obtain the cold start feature sets corresponding to each of the multiple retrieval platforms, and the multiple cold start feature sets are processed by multiple different multi-layer perceptrons to obtain cold start perception results, wherein the cold start features in the cold start feature sets are used to indicate Display images that have not been displayed to the user within a third preset time period; S3, obtain scene features corresponding to each of the multiple retrieval platforms, and process the multiple scene features through a first multi-layer perceptron to obtain a scene perception result, wherein the first multi-layer perceptron is different from multiple different multi-layer perceptrons; S4, after splicing the attention perception result, the cold start feature set, and the scene perception result, obtain a target splicing result, and process the target splicing result through a second multi-layer perceptron to obtain an output result of the initial fine ranking model, wherein the second multi-layer perceptron is different from multiple different multi-layer perceptrons and the first multi-layer perceptron; S5, iteratively train the initial fine ranking model according to the output result.

[0065] The multiple different multi-layer perceptrons in S1 include MLP-512, MLP-256, and MLP-128, and the cold start feature set in S2 is the images that have never been shown to the user in the third preset time period (i.e., the data to be retrieved that has never been shown to the user in the candidate pool corresponding to the target retrieval platform). S1 to S5 can be specifically understood as, in the refined ranking model, all id-like features in the second sample set are converted to embeddings through an embedding table (i.e., all discrete id-like features are converted to continuous vector representations that can be processed by the machine learning model). This process not only improves the expressive power of the model, but also effectively reduces the dimensionality problem. The statistical features are batch normalized and then spliced ​​with all other features (i.e., the features in the second sample set except the statistical features) (it should be noted that the spliced ​​id-like features are the conversion results obtained by the above-mentioned embedding table conversion). Among them, the attention perception result is calculated through the attention mechanism through the third search text and the user behavior sequence (that is, the third click sequence and the third download sequence mentioned above) (which can be understood as calculating the attention weight according to the third search text and the user behavior sequence). The purpose is to extract the user's historical behavior related to each third search text, to avoid the user having no historical behavior (click or download) related to the third search text, and to splice the zero vector for the user's features. If the user has no behavior related to the third search text, a higher weight than the zero vector is learned. In addition, the third search text will calculate the attention weight with the historical search word sequence of each data to be retrieved in the target search platform.

[0066] In the candidate pool corresponding to the target retrieval platform, the attention weight is calculated for each id corresponding to the data to be retrieved and the user's historical click and download sequence (third click identification sequence, third download identification sequence). In this way, the depth of each feature is poorly differentiated and the model effect is improved. In addition, because there are a large number of cold start images in the target retrieval platform (multiple cold start features in the cold start feature set, that is, newly launched images that have never been displayed to users in the third preset time period), for the cold start image, in addition to using the visual and text features of the cold start image itself, the number of exposures of the cold start feature is input into an activation network, and the output of the activation network is multiplied by each layer output of the MLP.

[0067] Similarly, the present application also includes an evaluation of the refined ranking model. In the process of evaluating the refined ranking model, the exposure click logs of the users of the new day are selected to construct positive and negative correlation samples of click and download tasks, and auc and gauc are calculated on the samples. Among them, gauc is the granularity of each user's request as the key, including request id, user id, scene id, search terms and other distinguishing information. Through the above implementation methods of the present application, the recommendation effect of cold start images can be effectively improved. That is, the training objectives of the refined ranking model in the present application are also relevance, conversion rate and download rate. The last stage is the re-ranking stage. The click rate and download rate scores obtained in the refined ranking stage are fused and displayed to the user from high to low according to the fusion score.

[0068] The image and text retrieval method in this application is a full-link service stage: the service of the entire search link includes recall, fine sorting and re-ranking logic. In the recall stage, a candidate pool of hundreds to thousands is screened out from the candidate pool of the full amount of billions. This stage includes real-time user embedding reasoning service and similarity retrieval service, in which item embedding is imported into the similarity retrieval library after offline reasoning. In the fine sorting stage, the click rate and download rate are inferred in real time by inputting user features, context features and features of the recalled item to obtain the predicted scores of click rate and download rate. In the re-ranking stage, the click and download scores are fused, and the final sorting is sorted from high to low according to the fusion score. The fusion formula of the present invention is ctr_score*1 / (alpha+math.exp(-beta*cvr_score)), where alpha=0.01, beta=5. This application realizes personalized retrieval of billions of images through the full-link consistency modeling idea, divided into two stages of recall and fine sorting. Click rate, download rate and relevance are modeled in the recall and fine sorting stages respectively. Through the above implementation method, consistent modeling of the entire link is achieved, ensuring the consistency of the model objectives from recall to fine ranking, ensuring the balance between multiple objectives in each link, and improving the efficiency of the entire link funnel conversion. At the same time, open source multimodal technology is applied, combined with its own business, and different feature construction and modeling methods are adopted in the recall and fine ranking stages, effectively improving the effect of each stage under controllable costs.

[0069] According to another aspect of an embodiment of the present invention, a graphic and text retrieval device for implementing the above-mentioned graphic and text retrieval method is also provided, and the device includes: an acquisition unit, which is used to obtain a retrieval request of a target retrieval platform, and determine a target account, a retrieval text corresponding to the target account, and a user feature corresponding to the target account according to the retrieval request; a first input unit, which is used to input the target account, the retrieval text, and the user feature into a recall model, and obtain a recall data set corresponding to the target retrieval platform output by the recall model, wherein the recall model is a model obtained by training an initial recall model with a first sample set, and the recall model is used to filter out data whose multiple targets all match a first preset condition set from a candidate pool; a second input unit, which is used to input the retrieval text, the user feature, and the recall data set into a refined ranking model, and obtain a target data set, wherein the refined ranking model is a model obtained by training an initial refined ranking model with a second sample set, and the refined ranking model is used to filter out data whose multiple targets all match a second preset condition set from the recall data set, and sort the filtered data; a sending unit, which is used to send the target data set to the target account corresponding to the target retrieval platform. The specific manner in which each unit in the device embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on again here.

[0070] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0071] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0072] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for image and text retrieval, characterized in that: include: Obtaining a search request from a target search platform, and determining a target account, a search text corresponding to the target account, and a user feature corresponding to the target account according to the search request; Inputting the target account, the search text, and the user features into a recall model to obtain a recall data set corresponding to the target search platform output by the recall model, wherein the recall model is a model obtained by training an initial recall model using a first sample set, and the recall model is used to screen out data from a candidate pool in which multiple targets all match a first preset condition set; Inputting the search text, the user features and the recall data set into a refined ranking model to obtain a target data set, wherein the refined ranking model is a model obtained by training an initial refined ranking model using a second sample set, and the refined ranking model is used to filter out data whose multiple targets all match a second preset condition set from the recall data set, and sort the filtered data; The target data set is sent to the target account corresponding to the target retrieval platform.

2. The method according to claim 1, characterized in that The initial recall model is trained using the first sample set, including: Acquire multiple first search texts of multiple first search accounts within a first preset time period, and a first click sequence and a first download sequence corresponding to each first search text, wherein the click information included in the first click sequence is used to indicate information of a clicked image, and the download information included in the first download sequence is used to indicate information of a downloaded image corresponding to the first search text; Fine-tuning the initial multimodal correlation model according to the first plurality of search texts, the first plurality of click sequences, and the first plurality of download sequences to obtain a target multimodal correlation model; The first sample set is constructed, and the initial recall model is trained according to the first sample set and the target multimodal correlation model.

3. The method according to claim 2, characterized in that Constructing the first sample set includes: Acquire a plurality of second search texts within a second preset time period, and determine a plurality of second search accounts according to the second search texts, wherein the first sample set includes the plurality of second search texts; Determine, according to the plurality of the second search accounts, a second click sequence and a second download sequence corresponding to each of the second search accounts; Inputting the second click sequence into the target multimodal correlation model, outputting a click text feature set and a click visual feature set corresponding to the second click sequence, and inputting the second download sequence into the target multimodal correlation model, outputting a download text feature set and a download visual feature set corresponding to the second download sequence; Respectively averaging the click text feature set, the click visual feature set, the download text feature set, and the download visual feature set to obtain a click text average feature set, a click visual average feature set, a download text average feature set, and a download visual average feature set included in the first sample set; A display information set corresponding to the target retrieval platform is obtained, and the display information set is input into the target multimodal correlation model, and a display text feature set and a display visual feature set included in the first sample set are output.

4. The method according to claim 3, characterized in that Training the initial recall model according to the first sample set and the target multimodal correlation model includes: Obtain an initial recall model built using an end-to-end dual-tower multi-objective modeling approach; Inputting the plurality of second search texts, click text average feature sets, click visual average feature sets, download text average feature sets, and download visual average feature sets into the user subnetwork of the initial recall model to obtain user feature vector representations corresponding to the plurality of second search accounts respectively; Inputting the displayed text feature set and the displayed visual feature set into the item sub-network of the initial recall model to obtain item feature vector representations corresponding to each of a plurality of displayed items in the target retrieval platform; The initial recall model is iteratively trained according to the user feature vector representation and the item feature vector representation.

5. The method according to claim 4, characterized in that Inputting a plurality of the second search texts, click text average feature sets, click visual average feature sets, download text average feature sets, and download visual average feature sets into the user subnetwork of the initial recall model to obtain user feature vector representations corresponding to each of the plurality of the second search accounts, including: The target multimodal correlation model in the user sub-network performs text feature extraction on the plurality of second search texts to obtain text features; Splicing the text feature with the click text average feature set, the click visual average feature set, the download text average feature set, and the download visual average feature set to obtain a spliced ​​feature; The concatenated features are normalized, and the normalized features are processed by a multi-layer perceptron to obtain the user feature vector representation.

6. The method according to claim 4, characterized in that Inputting the displayed text feature set and the displayed visual feature set into the item sub-network of the initial recall model to obtain item feature vector representations corresponding to each of the multiple displayed items in the target retrieval platform, including: The item sub-network performs keyword extraction on the display text features in the display text feature set to obtain a core text feature set; The target multimodal correlation model in the item sub-network performs semantic extraction on the core text feature set to obtain a semantic feature set; splicing the semantic feature set and the display visual feature set to obtain a display splicing feature; The display splicing features are normalized, and the normalized display splicing features are processed by a multi-layer perceptron to obtain the feature vector representation of the object.

7. The method according to claim 4, characterized in that Iteratively training the initial recall model according to the user feature vector representation and the item feature vector representation includes: Obtaining the cosine similarity between the user feature vector representation and the item feature vector representation; Multiplying the cosine similarity by a target hyperparameter in the initial recall model to obtain a multiplication result; Input the multiplication result into the target loss function to calculate the loss value to obtain the model loss value; The initial recall model is iteratively trained according to the model loss value.

8. The method according to claim 1, characterized in that The method further includes constructing the second sample set, wherein the constructing the second sample set includes: Acquire multiple third search texts of multiple third search accounts within a third preset time period, and determine a search text feature set according to the multiple third search texts, wherein the second sample set includes the search text feature set; Obtaining user features corresponding to each of the plurality of third search accounts, wherein the second sample set includes a plurality of user features, each of the user features including a third click identification sequence, a third download identification sequence, and permission information of the plurality of third search accounts on a plurality of search platforms of the corresponding third search account within the third preset time period; Obtain a statistical feature set within multiple sub-preset time periods within the third preset time period, and scene features corresponding to each of the multiple retrieval platforms, wherein the second sample set includes the statistical feature set and the multiple scene features, and each statistical feature in the statistical feature set is used to indicate the weight of the images displayed in the multiple retrieval platforms within the third preset time period.

9. The method according to claim 8, characterized in that The second sample set is used to train the initial refined ranking model, including: The initial refined ranking model performs deep cross calculation on multiple sample feature subsets in the second sample set to obtain multiple attention weight results, and processes the multiple attention weight results through multiple different multi-layer perceptrons to obtain attention perception results, wherein the deep cross calculation is used to calculate the attention weights of different sample feature subsets; Acquire cold start feature sets corresponding to each of the plurality of retrieval platforms, and process the plurality of cold start feature sets through a plurality of different multi-layer perceptrons to obtain cold start perception results, wherein the cold start features in the cold start feature sets are used to indicate images that have not been displayed to the user within the third preset time period; Acquire scene features corresponding to each of the plurality of retrieval platforms, and process the plurality of scene features through a first multi-layer perceptron to obtain a scene perception result, wherein the first multi-layer perceptron is different from the plurality of different multi-layer perceptrons; After splicing the attention perception result, the cold start feature set, and the scene perception result, a target splicing result is obtained, and the target splicing result is processed by a second multi-layer perceptron to obtain an output result of the initial refined sorting model, wherein the second multi-layer perceptron is different from the multiple different multi-layer perceptrons and the first multi-layer perceptron; The initial refined sorting model is iteratively trained according to the output result.

10. A graphic and text retrieval device, characterized in that: include: An acquisition unit, configured to acquire a search request from a target search platform, and determine a target account, a search text corresponding to the target account, and a user feature corresponding to the target account according to the search request; A first input unit is used to input the target account, the search text and the user features into a recall model to obtain a recall data set corresponding to the target search platform output by the recall model, wherein the recall model is a model obtained by training an initial recall model using a first sample set, and the recall model is used to screen out data from a candidate pool in which multiple targets all match a first preset condition set; A second input unit is used to input the search text, the user characteristics and the recall data set into a refined ranking model to obtain a target data set, wherein the refined ranking model is a model obtained by training an initial refined ranking model with a second sample set, and the refined ranking model is used to filter out data whose multiple targets all match a second preset condition set from the recall data set, and sort the filtered data; A sending unit is used to send the target data set to the target account corresponding to the target retrieval platform.

Images