Correlation prediction model training method, device and correlation prediction method

Abstract

The embodiment of the present disclosure provides a method for training a correlation prediction model, including: obtaining a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result; the first sample pair in the first training sample set is input into the correlation prediction model training to obtain a correlation prediction model after the first round of training; the second sample pair in the second training sample set is input into the correlation prediction model after the first round of training for retraining to obtain a correlation prediction model after the second round of training. In the above process, the prediction effect of the model can be improved, which greatly alleviates the poor prediction effect caused by the deviation in sample construction, and the problem that the prediction effect and timeliness performance cannot be taken into account at the same time.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to a correlation prediction model training method, device and correlation prediction method. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Search relevance refers to the relevance between the search terms entered by the user and the search results provided by the search engine. For example, when a user uses a search engine to search, the search engine will present the most relevant search results to the user based on the search terms entered by the user, ensuring that the search results match and are relevant to the search terms entered by the user.

[0004] At present, there is no universal definition of search relevance that is suitable for all fields. Relevance definitions in a field are often constructed for specific businesses based on different search businesses. Usually, the basis for judging relevance includes text relevance or semantic relevance. Summary of the invention

[0005] In this context, embodiments of the present disclosure are intended to provide a correlation prediction model training method, device, and correlation prediction method.

[0006] In a first aspect of the embodiments of the present disclosure, a correlation prediction model training method is provided, comprising:

[0007] Acquire a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result;

[0008] Inputting the first sample pair in the first training sample set into the correlation prediction model for training to obtain a correlation prediction model after a first round of training;

[0009] The second sample pairs in the second training sample set are input into the correlation prediction model after the first round of training to be trained again, so as to obtain the correlation prediction model after the second round of training.

[0010] In one embodiment of the present disclosure, inputting the first sample pair in the first training sample set into the correlation prediction model for training to obtain the correlation prediction model after the first round of training includes:

[0011] Inputting the first search item and the first search result in the first sample pair into the relevance prediction model respectively, wherein the relevance prediction model comprises: a feature encoding layer, a concatenation layer and a fully connected layer;

[0012] The feature coding layer performs feature coding on the first search item and the first search result respectively, and generates corresponding first feature coding vectors and second feature coding vectors respectively;

[0013] The first feature coding vector and the second feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0014] After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output;

[0015] According to the first true label of the first sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained until the loss function converges, thereby obtaining the correlation prediction model after the first round of training.

[0016] In one embodiment of the present disclosure, the relevance prediction model further includes: an attention layer;

[0017] The first feature coding vector and the second feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector, including:

[0018] Based on the self-attention mechanism, the attention layer extracts features from the first feature coding vector and the second feature coding vector respectively to obtain corresponding first feature vectors and second feature vectors respectively;

[0019] The first eigenvector and the second eigenvector are concatenated to obtain a concatenated vector.

[0020] In one embodiment of the present disclosure, the feature coding layer performs feature coding on the first search item and the first search result respectively, and generates corresponding first feature coding vectors and second feature coding vectors respectively, including:

[0021] The feature encoding layer performs feature encoding on the text of the first search item to obtain a first search item encoding vector;

[0022] Performing entity recognition on the text of the first search item, encoding phrase information after entity recognition, and obtaining a first entity encoding vector;

[0023] The feature encoding layer performs feature encoding on the text of the first search result to obtain a first search result encoding vector;

[0024] Performing entity recognition on the text of the first search result, encoding phrase information after entity recognition, and obtaining a second entity encoding vector;

[0025] The feature coding layer performs feature coding on the audio information of the first search result to obtain an audio coding vector;

[0026] The feature coding layer performs feature coding on the picture information of the first search result to obtain a picture coding vector;

[0027] The first feature encoding vector includes: the first search term encoding vector and the first entity encoding vector;

[0028] The second feature coding vector includes: the first search result coding vector, a second entity coding vector, an audio coding vector and a picture coding vector.

[0029] In one embodiment of the present disclosure, based on the self-attention mechanism, the attention layer extracts features from the first feature coding vector and the second feature coding vector respectively to obtain the corresponding first feature vector and second feature vector respectively, including:

[0030] Based on the self-attention mechanism, the attention layer inputs the first search term encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first search term encoding vector to obtain an attention score of the first search term;

[0031] Based on the self-attention mechanism, the attention layer inputs the first entity encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first entity encoding vector to obtain an attention score of the first entity;

[0032] Concatenate the attention score of the first search term and the attention score of the first entity to obtain the first feature vector;

[0033] Based on the self-attention mechanism, the attention layer inputs the first search result encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first search result encoding vector to obtain an attention score of the first search result;

[0034] Based on the self-attention mechanism, the attention layer inputs the second entity encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the second entity encoding vector to obtain an attention score of the second entity;

[0035] The attention score of the first search result and the attention score of the second entity are concatenated, and then outer-product fused with the audio coding vector and the picture coding vector to obtain the second feature vector.

[0036] In one embodiment of the present disclosure, the first feature vector and the second feature vector are concatenated to obtain a concatenated vector, including:

[0037] The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

[0038] In one embodiment of the present disclosure, inputting the second sample pair in the second training sample set into the correlation prediction model after the first round of training to train again, to obtain the correlation prediction model after the second round of training, includes:

[0039] Inputting the second search item and the second search result in the second sample pair into the correlation prediction model after the first round of training respectively, wherein the correlation prediction model includes: a feature encoding layer, a concatenation layer and a fully connected layer;

[0040] The feature coding layer performs feature coding on the second search item and the second search result respectively, and generates a corresponding third feature coding vector and a fourth feature coding vector respectively;

[0041] The third feature coding vector and the fourth feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0042] After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output;

[0043] According to the second true label of the second sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained again until the loss function converges, thereby obtaining a correlation prediction model after the second round of training.

[0044] In one embodiment of the present disclosure, the method further includes:

[0045] Pre-dividing click rate intervals and setting a first real label corresponding to any click rate interval;

[0046] Pre-dividing the correlation level and setting the second real label corresponding to any level;

[0047] Determine a corresponding click rate interval and a corresponding first true label according to the click data of any training sample;

[0048] The corresponding gear position and the corresponding second true label are determined according to the relevance degree of any training sample.

[0049] In a second aspect of the embodiments of the present disclosure, a correlation prediction method is provided, comprising:

[0050] Obtaining a search term input by a user and search results corresponding to the search term;

[0051] Inputting the search term and the search result into the trained correlation prediction model according to any one of the claims in the first aspect of the embodiments of the present disclosure;

[0052] Based on the trained relevance prediction model, the prediction scores of the search terms and the search results are output.

[0053] In a third aspect of the embodiments of the present disclosure, a correlation prediction model training device is provided, comprising:

[0054] A residual calculation module, which calculates the residual of each coding unit to be encoded in a video frame to be encoded in the video;

[0055] An acquisition module, for acquiring a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result;

[0056] A first training module, inputting the first sample pair in the first training sample set into the correlation prediction model for training, to obtain a correlation prediction model after a first round of training;

[0057] The second training module inputs the second sample pair in the second training sample set into the correlation prediction model after the first round of training to perform retraining to obtain the correlation prediction model after the second round of training.

[0058] In a fourth aspect of the embodiments of the present disclosure, a correlation prediction device is provided, comprising:

[0059] A first acquisition module, acquiring a search term input by a user and search results corresponding to the search term;

[0060] An input module, inputting the search terms and the search results into the trained correlation prediction model according to any one of the claims in the first aspect of the embodiments of the present disclosure;

[0061] The output module outputs the predicted scores of the search terms and the search results based on the trained correlation prediction model.

[0062] In a fifth aspect of the embodiments of the present disclosure, a storage medium is provided; a computer instruction is stored on the storage medium, and when the instruction is executed by a processor, the steps of the method described below are implemented:

[0063] Acquire a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result;

[0064] Inputting the first sample pair in the first training sample set into the correlation prediction model for training to obtain a correlation prediction model after a first round of training;

[0065] The second sample pairs in the second training sample set are input into the correlation prediction model after the first round of training to be trained again, so as to obtain the correlation prediction model after the second round of training.

[0066] In a sixth aspect of the embodiments of the present disclosure, there is provided a computing device, including:

[0067] A processor; and a memory for storing instructions executable by the processor;

[0068] The processor executes the executable instructions to implement the following steps:

[0069] Acquire a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result;

[0070] Inputting the first sample pair in the first training sample set into the correlation prediction model for training to obtain a correlation prediction model after a first round of training;

[0071] The second sample pairs in the second training sample set are input into the correlation prediction model after the first round of training to be trained again, so as to obtain the correlation prediction model after the second round of training.

[0072] In a seventh aspect of the embodiments of the present disclosure, a storage medium is provided; a computer instruction is stored on the storage medium, and when the instruction is executed by a processor, the steps of the method described below are implemented:

[0073] Obtaining a search term input by a user and search results corresponding to the search term;

[0074] Inputting the search term and the search result into the trained correlation prediction model according to any one of the claims in the first aspect of the embodiments of the present disclosure;

[0075] Based on the trained relevance prediction model, the prediction scores of the search terms and the search results are output.

[0076] In an eighth aspect of the embodiments of the present disclosure, there is provided a computing device, comprising:

[0077] A processor; and a memory for storing instructions executable by the processor;

[0078] The processor executes the executable instructions to implement the following steps:

[0079] Obtaining a search term input by a user and search results corresponding to the search term;

[0080] Inputting the search term and the search result into the trained correlation prediction model according to any one of the claims in the first aspect of the embodiments of the present disclosure;

[0081] Based on the trained relevance prediction model, the prediction scores of the search terms and the search results are output.

[0082] The above embodiments of the present disclosure have at least the following beneficial effects:

[0083] Through the above technical solution, a training sample set that is more in line with the correlation rules can be obtained. By adopting a large-scale sample constructed based on the click data of the search results and training in a weakly supervised manner, the prediction effect of the model is improved compared to directly using whether or not a click is used as a sample for training; and continuing to train the trained model with precise samples of manually labeled correlation results can further improve the prediction effect of the model, which greatly alleviates the poor prediction effect caused by the deviation during sample construction and the problem that the prediction effect and timeliness performance cannot be taken into account at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0085] Figure 1 is a schematic diagram of a correlation prediction process provided by an exemplary embodiment;

[0086] Figure 2 is a schematic diagram of a video search result in a music scene provided by an exemplary embodiment;

[0087] Figure 3is a flowchart of a correlation prediction model training method provided by an exemplary embodiment;

[0088] Figure 4 is a schematic diagram of a first round of training of a correlation prediction model provided by an exemplary embodiment;

[0089] Figure 5 It is a schematic diagram of named entity recognition in a relevance prediction model structure provided by an exemplary embodiment;

[0090] Figure 6 is a schematic diagram of an attention layer in a correlation prediction model structure provided by an exemplary embodiment;

[0091] Figure 7 is a schematic diagram of another correlation prediction model structure provided by an exemplary embodiment;

[0092] Figure 8 is a flow chart of a correlation prediction method provided by an exemplary embodiment;

[0093] Fig. 9 is a block diagram of a correlation prediction model training device provided by an exemplary embodiment;

[0094] Fig.10 is a block diagram of a correlation prediction device provided by an exemplary embodiment;

[0095] Fig.11 is a schematic diagram of a medium corresponding to the above method provided by an exemplary embodiment;

[0096] Fig.12 It is a schematic diagram of an electronic device capable of implementing the above method provided by an exemplary embodiment.

[0097] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0098] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0099] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, device, equipment, method or computer-readable storage medium. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0100] According to an embodiment of the present disclosure, a correlation prediction model training method, device and correlation prediction method are proposed.

[0101] In this document, it should be understood that any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning. In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties.

[0102] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0104] The inventors have found that when judging the relevance between search terms and search results, it is better to judge the relevance from a semantic level by building a machine learning model than to judge whether there is an intersection between the two texts. However, when training a machine learning model, on the one hand, the accuracy of the training sample selection must be considered, and the quality of the sample is crucial to the effect of the model. On the other hand, when building the model structure, it is necessary to balance the computational efficiency of the model with the prediction effect of the model.

[0105] In view of this, the present specification provides a technical solution for training a relevance prediction model by first using a large-scale sample constructed based on click data on search results, training the model through weak supervision, and then continuing to train the trained model with precise samples of manually labeled relevance results, so as to obtain better relevance prediction results.

[0106] Through the above technical solution, a training sample set that is more in line with the correlation rules can be obtained. By adopting a large-scale sample constructed based on the click data of the search results and training in a weakly supervised manner, the prediction effect of the model is improved compared to directly using whether or not a click is used as a sample for training; and continuing to train the trained model with precise samples of manually labeled correlation results can further improve the prediction effect of the model, which greatly alleviates the poor prediction effect caused by the deviation during sample construction and the problem that the prediction effect and timeliness performance cannot be taken into account at the same time.

[0107] Furthermore, based on the above-mentioned trained relevance prediction model, this specification also provides a technical solution for predicting the relevance of search terms and search results based on the trained relevance prediction model.

[0108] Through the above technical solution, the correlation prediction effect during online prediction can be improved.

[0109] See also Figure 1 , Figure 1 It is a schematic diagram of a correlation prediction process provided by an exemplary embodiment.

[0110] like Figure 1 As shown, the correlation prediction model can be trained in an offline process, and the correlation prediction model trained in the offline process can be used to perform correlation prediction in the online process.

[0111] Specifically, the offline process includes the following steps:

[0112] First, according to the definition of relevance, a first training sample set and a second training sample set are constructed, wherein the first training sample set is constructed based on the collected user click data on the search results, and the second training sample set is constructed by manually annotating the relevance results.

[0113] Secondly, the first sample pair in the first training sample set is input into a pre-built correlation prediction model for training to obtain a correlation prediction model after the first round of training.

[0114] Then, the second sample pair in the second training sample set is input into the correlation prediction model after the first round of training to be trained again, so as to obtain the correlation prediction model after the second round of training.

[0115] Finally, the correlation prediction model after the second round of training is used as the trained correlation prediction model and provided to the online process for use.

[0116] Specifically, the online process includes the following steps:

[0117] First, the search engine determines the search results corresponding to the search terms according to the search terms input by the user.

[0118] Secondly, the search terms and the search results corresponding to the search terms are input into the relevance prediction model trained during the offline process.

[0119] Finally, the relevance prediction model outputs the relevance prediction score between the search term and the search results.

[0120] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0121] Application Scenario Overview

[0122] When judging the relevance between the search terms entered by the user and the search results given by the search engine, it can generally be divided into relevance judgment at the text level and relevance judgment at the semantic level.

[0123] When making a relevance judgment based on the text level, it can be judged according to the text of the search terms, and the search results will contain the words or characters that appear in the text of the search terms. In implementation, generally, means such as text similarity features, text edit distance, or the longest common subsequence can be used to judge the relevance between the search terms and the search results, without the need to build a machine learning model.

[0124] However, the relevance judgment at the text level has limitations and is prone to missing relevant resources. Relevance can only be judged when there is an intersection between the search terms and the search results in terms of text.

[0125] For example, please refer to Figure 2 , Figure 2 which is a schematic diagram of video search results in a music scenario provided by an exemplary embodiment.

[0126] As Figure 2 shown, when making a relevance judgment based on the text level, Figure 2 the interface on the left in Figure 2 is the search result corresponding to the search term "will it be", and the interface on the right in Figure 2 is the search result corresponding to the search term "lost". As

[0127] shown, the search results all contain the text that appears in the search terms. However, the user's real intention is to search for video resources of the songs "Will It Be" and "Lost", but the search engine does not present the video resources related to the songs in the resource library to the user, that is, the search engine misses relevant resources.

[0127] In addition, in actual situations, there will be a large number of cases where there is no text intersection but semantic relevance. For example, "Zhou XX" and "Sunny Day" have no intersection at the text level, and there is no intersection between them in terms of words or characters, but they have strong semantic relevance. When the user searches for "Sunny Day", it is very likely that they want to search for "Sunny Day" sung by Zhou XX.

[0128] In response to the above situation, relevance judgment can be made from the semantic level. In implementation, a relevance sample composed of the search terms and the search results can be constructed to train a relevance prediction model, so as to use the trained relevance prediction model to predict the relevance between the search terms entered by the user and the search results given by the search engine.

[0129] When constructing the training sample, among the multiple search results corresponding to the search terms, the search results clicked by the user and the search terms can be used as relevant positive sample pairs, while the search results not clicked by the user and the search terms can be used as irrelevant negative sample pairs.

[0130] However, when users click on search results, they may click on search results that are of interest to the users but are not relevant to the search terms. Figure 2 As shown in the figure, when a user searches for "Will it happen", he may click on a video resource that is not relevant to the song "Will it happen" but is very popular or novel. Therefore, it is not accurate to construct training samples directly based on whether the user clicks on the search results.

[0131] Furthermore, when building a relevance prediction model, a semantic matching model can usually be used to judge the relevance. The semantic matching model can generally be divided into two types: a representational model and an interactive model.

[0132] For example, representational models, such as the Deep Structured Semantic Models (DSSM) model based on a deep network, adopt a dual-tower model structure. By inputting the original text of the search terms and search results into the encoding model tower respectively, the encoding vectors corresponding to the search terms and search results are obtained, and then the relevance score is obtained by calculating the cosine distance of the vectors.

[0133] To continue with the example, an interactive model, such as the BERT (Bidirectional Encoder Representations from Transformers) model, inputs the original text of the search terms and search results into the same model. During vector encoding, the search terms and search results can refer to each other. This mutual reference can capture more semantic information between the two, and then a relevance score is obtained through a fully connected neural network.

[0134] It is worth noting that the representational model encodes the search terms and search results separately, so the computational efficiency is relatively high, but the semantic capture effect is not good enough. The interactive model encodes the search terms and search results together, so the computational efficiency is low, but it can achieve better semantic capture effect through interaction.

[0135] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0136] Exemplary Methods

[0137] The technical concept of this specification will be described in detail below through specific embodiments.

[0138] The present disclosure aims to provide a technical solution for training a correlation prediction model by first using a large-scale sample constructed based on click data on search results, training the model through a weak supervision method, and then continuing to train the trained model with precise samples of manually labeled correlation results, so as to obtain better correlation prediction results.

[0139] During implementation, a training sample set may be obtained.

[0140] The training sample set includes a first training sample set constructed based on collected user click data on search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result.

[0141] Then, the first sample pair in the first training sample set may be input into the correlation prediction model for training to obtain a correlation prediction model after a first round of training.

[0142] Finally, the second sample pair in the second training sample set may be input into the correlation prediction model after the first round of training to be trained again, so as to obtain the correlation prediction model after the second round of training.

[0143] Through the above technical solution, a training sample set that is more in line with the correlation rules can be obtained. By adopting a large-scale sample constructed based on the click data of the search results and training in a weakly supervised manner, the prediction effect of the model is improved compared to directly using whether or not a click is used as a sample for training; and continuing to train the trained model with precise samples of manually labeled correlation results can further improve the prediction effect of the model, which greatly alleviates the poor prediction effect caused by the deviation during sample construction and the problem that the prediction effect and timeliness performance cannot be taken into account at the same time.

[0144] See also Figure 3 , Figure 3 1 is a flowchart of a correlation prediction model training method provided by an exemplary embodiment, the method comprising the following steps:

[0145] Step 301: Obtain a training sample set.

[0146] In this embodiment, the training sample set includes a first training sample set constructed based on collected user click data on search results, and a second training sample set consisting of manually annotated relevance results; any training sample includes a sample pair consisting of a search term and a search result.

[0147] As described above, when constructing the training sample set, among the multiple search results corresponding to the search term, the search results clicked by the user and the search term can be used as relevant positive sample pairs, while the search results not clicked by the user and the search term can be used as irrelevant negative sample pairs, that is, the first training sample set is constructed based on the collected click data of the user on the search results.

[0148] However, it is not accurate enough to directly construct the training sample based on whether the user clicks on the search result.

[0149] Therefore, preferably, in an illustrated embodiment, the click-through rate interval can be pre-divided and the first true label corresponding to any click-through rate interval can be set; further, according to the click data of any training sample, the corresponding click-through rate interval is determined and the corresponding first true label is determined.

[0150] Specifically, according to the user's historical click log, the click-through rate between each search term and the search results corresponding to the search term can be calculated, divided into different intervals according to the numerical value of the click-through rate, and the corresponding relevance label is set for each interval.

[0151] For example, the following interval division as shown in Table 1 can be pre-constructed:

[0152] Table 1

[0153]

[0154] It should be noted that the division of the click-through rate interval and the setting of the corresponding label in Table 1 can be determined by those skilled in the art according to the actual situation, and the present application does not limit this.

[0155] Among them, for the construction of the negative sample pair, it can also be obtained based on the simple random sampling method.

[0156] For example, the search term "will it" and the search results corresponding to the search term "confused" can be constructed as a negative sample pair.

[0157] In addition, complex negative sample pairs can be constructed by replacing and rewriting some words in the positive sample pairs, so as to enhance the discrimination ability of the model.

[0158] For example, "sunny" in the search term "Zhou XX sunny" can be replaced with "cloudy".

[0159] Further, for any training sample in the first training sample set, according to the above-mentioned relevance definition based on the click-through rate, the numerical value of the click-through rate can be determined first, and then the corresponding relevance label can be determined according to the interval to which the click-through rate belongs.

[0160] Continuing with the example, suppose that within the preset time range, the historical click log records 100 searches for the search term "will it happen", and among the 100 search results presented, the number of clicks on search result A is 70, then the click rate of search result A is 0.7. According to Table 1, it belongs to the interval [0.5, 0.8), and the corresponding first true label is sub-correlated.

[0161] It is worth noting that when determining the click rate, those skilled in the art can select the time range according to actual conditions, and this application does not limit this.

[0162] Generally speaking, for the same search item, the relevance of search results with high click-through rates is higher than that of search results with low click-through rates. Through the above process, a stratified sample constructed based on click-through rates can be obtained. Since the stratified sample can be derived from the user's historical click logs and has a large natural magnitude, a large number of first training samples can be constructed to train the relevance prediction model, so that the parameters of the relevance prediction model can be fully learned.

[0163] It should be noted that a high click-through rate does not necessarily mean that the relevance is good, and there may still be sample bias. For example, a cover version of a song is very popular recently and has a high click-through rate, but when users search for the song, they may want to find the original version.

[0164] Furthermore, in order to improve the effect of correlation prediction, a second training sample set consisting of manually annotated correlation results may be constructed, and the correlation prediction model after the first round of training may be retrained based on the second training sample set.

[0165] Preferably, in one embodiment shown, the correlation levels can be pre-divided and the second real label corresponding to any level can be set; further, the corresponding level and the corresponding second real label can be determined according to the correlation degree of any training sample.

[0166] Specifically, relevance levels and relevance labels corresponding to each relevance level may be pre-divided based on multiple dimensions of relevance.

[0167] For example, taking the music scenario as an example, the relevance can be divided into the following three dimensions according to the needs of the business field:

[0168] (1) Text relevance refers to the fact that the search terms are included in the search results, that is, the search result resource contains the terms (words or characters) in the search term query.

[0169] For example, the search term is "Zhou XX Sunny Day", and the search result is "Zhou XX live concert singing "Sunny Day"".

[0170] (2) Semantic relevance refers to the semantic relevance between search results and search terms, which can usually be based on semantic association knowledge in the field of music and can include the correlation between singer names and single titles, album names, style types, languages, etc.

[0171] For example, "Zhou XX" and "Sunny Day", "Liu XX" and "Four Heavenly Kings", "Liu XX" and "The Rap of China", etc.

[0172] (3) Intent relevance means that when the search term contains entity intents such as specific songs, artists, albums, or lyrics, the corresponding intents in the search results remain consistent.

[0173] For example, when the search term is "Sunny Day sung by Zhou Moumou" and the search result is "Sunny Day covered by Zhang San", it can be considered that the search intent is inconsistent, and the user wants to find Sunny Day sung by Zhou Moumou, not the cover version.

[0174] Continuing with the example, based on the above three dimensions, the relevance in the music field can be divided into four levels, and a relevance label corresponding to each level is set, that is, the second true label, as shown in Table 2 below:

[0175] Table 2

[0176]

[0177] As shown in Table 2, when performing correlation judgment based on the above three dimensions, four different correlation levels can be determined according to the combination of matching or mismatching, and corresponding correlation labels are set for the four levels. Among them, matching is represented by √ and mismatch is represented by ×.

[0178] It is worth mentioning that when the search term is "Hxx", the search result is "Tian Moumou". Although the two do not match in the text relevance dimension, they both point to the same person. Both semantics and intentions are relevant, so the relevance level is Good.

[0179] Furthermore, when manually annotating the correlation results of the second training sample set, the correlation level of the sample can be determined based on a predetermined correlation degree, and then the second true label corresponding to the correlation level can be determined.

[0180] For example, assuming that a sample pair in the second training sample set is "Will Chen XX do it?" and "Chen XX's concert, fans singing "Will it"?", then referring to the three dimensions of correlation judgment shown in Table 2, it can be concluded that the degree of correlation is that all three dimensions match, and the correlation level can be determined to be Good, and the corresponding second true label is the most relevant.

[0181] In one example, tens of thousands of samples can be screened out from the user's historical click logs, and the relevance levels of the samples can be manually determined according to pre-divided relevance level rules, thereby determining the second true label of the sample.

[0182] It should be noted that Table 2 provides a definition of relevance in the field of music. It is understandable that when facing other business fields, technical personnel in this field can also determine the basis for dividing the relevance levels based on the characteristics of the fields.

[0183] After obtaining a weakly supervised training sample set based on click rate and a training sample set with manually labeled relevance that are more consistent with relevance through the above step 301, the model can be trained based on the training sample set.

[0184] Step 302: Input the first sample pair in the first training sample set into the correlation prediction model for training, and obtain the correlation prediction model after the first round of training.

[0185] Specifically, the first sample pair in the first training sample set constructed based on the collected user click data on the search results can be input into a pre-constructed correlation prediction model for training, so as to obtain the correlation prediction model after the first round of training, and determine the parameters of the correlation prediction model after the first round of training.

[0186] As can be seen from the above, when constructing a relevance prediction model, the representational model and the interactive model have their own strengths and weaknesses in terms of computational efficiency and the effect of semantic relevance discrimination. In order to balance computational efficiency and model effect, a relevance prediction model can be constructed that includes at least a feature encoding layer, a concatenation layer, and a fully connected layer.

[0187] In one embodiment shown, the first search item and the first search result in the first sample pair may be respectively input into the relevance prediction model, wherein the relevance prediction model includes: a feature encoding layer, a concatenation layer, and a fully connected layer;

[0188] The feature coding layer performs feature coding on the first search item and the first search result respectively, and generates corresponding first feature coding vectors and second feature coding vectors respectively;

[0189] The first feature coding vector and the second feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0190] After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output;

[0191] According to the first true label of the first sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained until the loss function converges, thereby obtaining the correlation prediction model after the first round of training.

[0192] Specifically, after constructing the first training sample set based on the collected user click data on search results, the first search item and the first search result in the first sample pair can be respectively input into a pre-constructed relevance prediction model for a first round of training. The relevance prediction model includes at least a feature encoding layer, a splicing layer, and a fully connected layer.

[0193] See also Figure 4 , Figure 4 A schematic diagram of a first round of training of a correlation prediction model provided by an exemplary embodiment includes the following steps:

[0194] Step S401: The feature coding layer may perform feature coding on the input first search term and the first search result, respectively, and generate corresponding first feature coding vectors and second feature coding vectors, respectively.

[0195] When feature encoding is performed on the first search item and the first search result, in order to improve encoding efficiency, a basic dual-tower structure may be used to feature encode the first search item and the first search result, respectively.

[0196] In addition, if the search result is a multimodal resource, such as video or audio, the image information or voice information can also be input into the feature encoding layer for feature encoding.

[0197] This application does not limit the specific encoding method for search terms and search results. Those skilled in the art know that the search terms and search results can be encoded in the same way.

[0198] In one example, the specific process of step S401 may include:

[0199] The feature coding layer performs feature coding on the text of the first search item to obtain a first search item coding vector; the feature coding layer performs feature coding on the text of the first search result to obtain a first search result coding vector; the feature coding layer performs feature coding on the audio information of the first search result to obtain an audio coding vector; the feature coding layer performs feature coding on the image information of the first search result to obtain an image coding vector.

[0200] Continuing with the example, the first feature coding vector generated after feature coding includes: a first search item coding vector; and the second feature coding vector includes: a first search result coding vector, an audio coding vector and a picture coding vector.

[0201] Step S402: the first feature coding vector and the second feature coding vector may be concatenated through a concatenation layer to obtain a concatenated vector.

[0202] The first feature coding vector and the second feature coding vector are both one-dimensional vectors, and the concatenated vector obtained after vector concatenation is also a one-dimensional vector.

[0203] Step S403: the concatenated vector may be processed by a fully connected layer and sigmoid to output a prediction score.

[0204] Among them, since the output of sigmoid is between 0 and 1, the value of the final output prediction score is in the range of (0, 1).

[0205] Furthermore, the prediction score intervals may be pre-divided, and the correlation results corresponding to any prediction score intervals may be set.

[0206] For example, see the interval division shown in Table 3:

[0207] Table 3

[0208] Prediction score (0，0.3) [0.3，0.5) [0.5，0.8) [0.8，1) Correlation results Not relevant Neutral related Secondary correlation Most relevant

[0209] It is worth noting that the division of the prediction partitions given in Table 3 and the setting of the corresponding correlation results can be determined by those skilled in the art according to actual conditions, and this application does not limit this.

[0210] Step S404: the correlation prediction model may be iteratively trained according to the first true label of the first sample pair, the prediction score and a preset loss function until the loss function converges, thereby obtaining a correlation prediction model after the first round of training.

[0211] In an example, the following loss function can be defined:

[0212]

[0213] Among them, q represents the search term query, r represents the search result resource; target q，r For the artificially defined target prediction score, score q，r is the prediction score output by the model;

[0214] For the positive sample pairs in the first training sample set, the prediction score is required to be greater than the threshold score corresponding to the correlation result; for the negative sample pairs, the prediction score needs to be less than the corresponding threshold score.

[0215] Taking Table 1 as an example, we can get the artificially defined target prediction scores shown in Table 4:

[0216] Table 4

[0217]

[0218] Similarly, for the second training sample set, the target prediction score shown in Table 5 below can also be manually defined:

[0219] Table 5

[0220]

[0221] By iteratively training the correlation prediction model until the loss function converges, the correlation prediction model after the first round of training and the parameters of the correlation prediction model after the first round of training are obtained.

[0222] At this point, the correlation prediction model after the first round of training can be trained again based on the second training sample set with manually annotated correlation results.

[0223] However, in order to further improve the relevance prediction effect and allow the model to learn more semantic information, the present disclosure also provides a technical solution for improving the aforementioned relevance prediction model based on named entity recognition technology (NER, Named Entity Recognition).

[0224] Among them, named entity recognition technology refers to the identification of entities with specific meanings in the text, mainly including names of people, places, institutions and proper nouns.

[0225] For example, in a music scenario, the recognized entity types can be artist name, song name, album name, and genre, etc.

[0226] In one example, entity recognition is performed on the search term "Chen Moumou will it" and two entity types of phrase information are obtained: artist name-Chen Moumou, single name-will it it. Therefore, the search term includes the following three dimensions when inputting the feature encoding layer:

[0227] The text of the search term: "Can Chen XX?"

[0228] Artist name entity text: "Chen XX";

[0229] Single title entity text: "Will it be?"

[0230] In another example, the search result "Chen Moumou's Sun Tour Concert, fans sing "Will It Be"" is recognized and four entity types of phrase information are obtained: artist name - Chen Moumou, single name - Will It Be, album name - Sun and video type - Concert. Therefore, the search result includes the following five dimensions when inputting the feature encoding layer:

[0231] The text of the search result: "Chen Moumou's Sun Tour Concert, fans sang "Will It Be"";

[0232] Artist name entity text: "Chen XX";

[0233] Single title entity text: "Will it be";

[0234] Album Title Entity Text: "Sun";

[0235] Video type entity text: "Concert".

[0236] In an embodiment shown, after the named entity recognition technology is introduced into the relevance prediction model, the above step S401 may further include the following process:

[0237] The feature encoding layer performs feature encoding on the text of the first search item to obtain a first search item encoding vector; performs entity recognition on the text of the first search item, encodes the phrase information after entity recognition, and obtains a first entity encoding vector;

[0238] The feature coding layer performs feature coding on the text of the first search result to obtain a first search result coding vector; performs entity recognition on the text of the first search result, encodes the phrase information after entity recognition, and obtains a second entity coding vector; the feature coding layer performs feature coding on the audio information of the first search result to obtain an audio coding vector; the feature coding layer performs feature coding on the picture information of the first search result to obtain a picture coding vector;

[0239] The first feature encoding vector includes: the first search term encoding vector and the first entity encoding vector;

[0240] The second feature coding vector includes: the first search result coding vector, a second entity coding vector, an audio coding vector and a picture coding vector.

[0241] It can be seen that by introducing named entity recognition technology, entity information is added to the first feature encoding vector and the second feature encoding vector, which allows the model to learn more text semantic information, thereby improving the prediction effect of the correlation model.

[0242] See also Figure 5 , Figure 5 It is a schematic diagram of named entity recognition in a relevance prediction model structure provided by an exemplary embodiment.

[0243] like Figure 5As shown, after the first search term and the first search result are input into the relevance prediction model, entity recognition can be performed on the text of the first search term and the first search result respectively to obtain phrase information after entity recognition.

[0244] Secondly, the text of the first search item and the phrase information after entity recognition, as well as the text of the first search result and the phrase information after entity recognition can be feature encoded through the feature encoding layer to obtain the first search item encoding vector and the first entity encoding vector, as well as the first search result encoding vector and the second entity encoding vector.

[0245] In addition, when the first search result is a multimodal resource, the audio information and picture information of the first search result may also be feature encoded through a feature encoding layer to obtain an audio encoding vector and a picture encoding vector.

[0246] Finally, the generated first feature coding vector includes: the first search item coding vector and the first entity coding vector; the second feature coding vector includes: the first search result coding vector, the second entity coding vector, the audio coding vector and the picture coding vector.

[0247] In addition to the above-mentioned named entity recognition technology, the present disclosure also provides a technical solution for improving the aforementioned relevance prediction model based on the Self-Attention mechanism to better capture semantic information.

[0248] Among them, the Attention mechanism refers to focusing limited attention on key information, thereby saving resources and quickly obtaining the most effective information.

[0249] Self-Attention is a special form of Attention. As the name suggests, Self-Attention means paying attention to yourself internally and looking for internal connections.

[0250] For example, for a sentence, each word in the sentence can be subjected to attention calculation together with all the words in the sentence, so as to learn the dependency relationship between the words in the sentence and capture the internal structure of the sentence.

[0251] In one illustrated embodiment, the relevance prediction model further includes: an attention layer.

[0252] Furthermore, after the attention layer is introduced into the relevance prediction model, the above step S402 may further include the following process:

[0253] Based on the self-attention mechanism, the attention layer extracts features from the first feature coding vector and the second feature coding vector respectively to obtain corresponding first feature vectors and second feature vectors respectively; and concatenates the first feature vector and the second feature vector to obtain a concatenated vector.

[0254] Among them, in one embodiment shown, when performing vector splicing, a weak interaction method can be used to splice the first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector to obtain a spliced ​​vector.

[0255] In one embodiment shown, when the relevance prediction model structure simultaneously introduces named entity recognition technology and self-attention mechanism, based on the self-attention mechanism, the attention layer inputs the first search term encoding vector into an activation function, and then performs convolution calculation on the result of the activation function, and performs dot product operation on the result of the convolution calculation and the first search term encoding vector to obtain the attention score of the first search term; based on the self-attention mechanism, the attention layer inputs the first entity encoding vector into an activation function, and then performs convolution calculation on the result of the activation function, and performs dot product operation on the result of the convolution calculation and the first entity encoding vector to obtain the attention score of the first entity; the attention score of the first search term and the attention score of the first entity are concatenated to obtain the first feature vector.

[0256] Furthermore, based on the self-attention mechanism, the attention layer inputs the first search result encoding vector into the activation function, and then performs convolution calculation on the result of the activation function, and performs dot product operation on the result of the convolution calculation and the first search result encoding vector to obtain the attention score of the first search result; based on the self-attention mechanism, the attention layer inputs the second entity encoding vector into the activation function, and then performs convolution calculation on the result of the activation function, and performs dot product operation on the result of the convolution calculation and the second entity encoding vector to obtain the attention score of the second entity; the attention score of the first search result and the attention score of the second entity are concatenated, and then outer product fused with the audio encoding vector and the picture encoding vector to obtain the second feature vector.

[0257] See also Figure 6 , Figure 6 It is a schematic diagram of an attention layer in a correlation prediction model structure provided by an exemplary embodiment.

[0258] like Figure 6As shown, after the first feature encoding vector and the second feature encoding vector obtained by introducing the named entity recognition technology are input into the attention layer, the following steps may be specifically included:

[0259] Step S601: The attention layer inputs the first search term encoding vector and the first entity encoding vector into the activation function respectively.

[0260] Step S602: performing convolution calculations on the calculation results of the activation functions respectively;

[0261] Step S603: performing dot product operations on the result of the convolution calculation, the first search term encoding vector and the first entity encoding vector, respectively, to obtain attention scores of the first search term and the first entity, respectively;

[0262] Step S604: concatenate the attention score of the first search item and the attention score of the first entity to obtain a first feature vector.

[0263] Similarly, for the first search result encoding vector and the second entity encoding vector, the above steps S601-S603 are also performed to obtain the attention scores of the first search result and the second entity respectively.

[0264] However, the difference is that when the first search result is a multimodal resource, step S604 does not need to be performed for the first search result encoding vector and the second entity encoding vector, but step S605 needs to be performed.

[0265] Step S605: concatenate the attention score of the first search result and the attention score of the second entity, and then perform outer product fusion with the audio coding vector and the picture coding vector to obtain the second feature vector.

[0266] Among them, the specific algorithm of outer product fusion can be determined by those skilled in the art, and this application does not limit it.

[0267] In an example, the above feature extraction process can be performed based on the following formula:

[0268] In step S601,

[0269] Among them, e q Represents each encoding vector of the input, W q and W k Represents the parameter matrix, and relu represents the ReLU activation function (The Rectified Linear Unit).

[0270] In step S602,

[0271] Among them, conv1d represents a one-dimensional convolution calculation, h represents the number of encoding vectors, and α is an h-dimensional vector.

[0272] In step S603, q=(q1, q2, ..., q h )=α·e q ;

[0273] Among them, q represents the degree of focus on the original input under each attention.

[0274] In step S604, q total =(q ori ,q pv1 ,q pv2 , …).

[0275] Among them, q ori represents the attention score of the first search term, q pv1 ,q pv2 , ... represents the attention score of the first entity, q total Represents the first eigenvector obtained after concatenation.

[0276] Similarly, the process of obtaining the attention scores of the first search result and the second entity, and the process of obtaining the second feature vector are described above and will not be repeated here.

[0277] It should be noted that since the computational efficiency of the CNN convolutional neural network is higher than that of the RNN recurrent neural network and the Transformer autoregressive model, it is used as a preferred example. In practical applications, those skilled in the art may also replace it with other coding structures.

[0278] See also Figure 7 , Figure 7 FIG. 1 is a schematic diagram of another correlation prediction model structure provided by an exemplary embodiment. Figure 7 As shown in the figure, in addition to the feature encoding layer, concatenation layer and fully connected layer, the relevance prediction model structure also introduces the named entity recognition technology and self-attention mechanism described above.

[0279] It can be understood that according to the multiple embodiments of the correlation prediction model described above, the following can be obtained: Figure 7 The correlation prediction model shown. The process of the first round of training of the correlation prediction model will not be described in detail here, and those skilled in the art can know it by combining the above multiple embodiments.

[0280] After the first sample pair in the first training sample set is input into the correlation prediction model for training to obtain the correlation prediction model after the first round of training, the second round of training may be continued to execute step 303 .

[0281] Step 303: input the second sample pair in the second training sample set into the correlation prediction model after the first round of training to perform retraining, so as to obtain the correlation prediction model after the second round of training.

[0282] Specifically, the second sample pairs in the second training sample set with manually annotated correlation results can be input into the correlation prediction model after the first round of training for retraining, thereby obtaining the correlation prediction model after the second round of training.

[0283] In one embodiment shown, the second search item and the second search result in the second sample pair may be respectively input into the correlation prediction model after the first round of training, wherein the correlation prediction model includes: a feature encoding layer, a concatenation layer, and a fully connected layer;

[0284] The feature coding layer performs feature coding on the second search item and the second search result respectively, and generates a corresponding third feature coding vector and a fourth feature coding vector respectively;

[0285] The third feature coding vector and the fourth feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0286] After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output;

[0287] According to the second true label of the second sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained again until the loss function converges, thereby obtaining a correlation prediction model after the second round of training.

[0288] It is worth noting that during the second round of training, the model structure of the correlation prediction model remains unchanged, so the training process of the second training sample set will not be repeated here, and those skilled in the art may refer to the training process of the first training sample set mentioned above.

[0289] In the above embodiment, when training the correlation prediction model, a large-scale sample constructed based on the click data of the search results is first used for training in a weakly supervised manner, and then the trained model is further trained using precise samples of manually labeled correlation results, so as to obtain a better correlation prediction effect. Through the above technical scheme, a training sample set that is more in line with the correlation rules can be obtained. By using a large-scale sample constructed based on the click data of the search results and training in a weakly supervised manner, the prediction effect of the model is improved compared to directly using whether or not a click is used as a sample for training; and by continuing to train the trained model using precise samples of manually labeled correlation results, the prediction effect of the model can be further improved, which greatly alleviates the poor prediction effect caused by the deviation in sample construction, and the problem that the prediction effect and timeliness performance cannot be taken into account at the same time.

[0290] Corresponding to the above-mentioned correlation prediction model training method, the present disclosure also provides a technical solution for predicting the correlation between search terms and search results based on a trained correlation prediction model.

[0291] See also Figure 8 , Figure 8 1 is a flowchart of a correlation prediction method provided by an exemplary embodiment, the method comprising the following steps:

[0292] Step 801: Obtain the search term input by the user and the search results corresponding to the search term.

[0293] Step 802: Input the search term and the search result into the trained relevance prediction model.

[0294] Step 803: Based on the trained relevance prediction model, output the prediction scores of the search terms and the search results.

[0295] In one embodiment shown, the trained correlation prediction model includes: a feature encoding layer, a concatenation layer, and a fully connected layer; the step 803 further includes:

[0296] The feature coding layer performs feature coding on the search item and the search result respectively, and generates a corresponding first feature coding vector and a second feature coding vector respectively; the first feature coding vector and the second feature coding vector are vector-concatenated by the splicing layer to obtain a splicing vector; after the splicing vector is processed by a fully connected layer and sigmoid, the prediction score is output.

[0297] Preferably, in one embodiment shown, based on the named entity recognition technology, step 803 further:

[0298] The feature encoding layer performs feature encoding on the text of the search item to obtain a search item encoding vector; performs entity recognition on the text of the search item, encodes the phrase information after entity recognition, and obtains a first entity encoding vector;

[0299] The feature encoding layer performs feature encoding on the text of the search result to obtain a search result encoding vector; performs entity recognition on the text of the search result, encodes the phrase information after entity recognition, and obtains a second entity encoding vector;

[0300] The feature coding layer performs feature coding on the audio information of the search result to obtain an audio coding vector; the feature coding layer performs feature coding on the picture information of the search result to obtain a picture coding vector;

[0301] The first feature coding vector includes: the search term coding vector and the first entity coding vector; the second feature coding vector includes: the search result coding vector, the second entity coding vector, the audio coding vector and the picture coding vector.

[0302] Preferably, in one embodiment shown, the trained correlation prediction model further includes: an attention layer; and step 803 further includes:

[0303] Based on the self-attention mechanism, the attention layer extracts features from the first feature coding vector and the second feature coding vector respectively to obtain corresponding first feature vectors and second feature vectors respectively; and concatenates the first feature vector and the second feature vector to obtain a concatenated vector.

[0304] In one embodiment shown, the first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector may be vector-concatenated to obtain a concatenated vector.

[0305] Furthermore, in one embodiment shown, step 803 further includes:

[0306] Based on the self-attention mechanism, the attention layer inputs the search term encoding vector into the activation function, and then performs convolution calculation on the calculation result of the activation function, and performs dot product calculation on the result of the convolution calculation and the first search term encoding vector to obtain the attention score of the search term; Based on the self-attention mechanism, the attention layer inputs the first entity encoding vector into the activation function, and then performs convolution calculation on the calculation result of the activation function, and performs dot product calculation on the result of the convolution calculation and the first entity encoding vector to obtain the attention score of the first entity; The attention score of the search term and the attention score of the first entity are concatenated to obtain the first feature vector;

[0307] Based on the self-attention mechanism, the attention layer inputs the search result encoding vector into the activation function, and then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the search result encoding vector to obtain the attention score of the search result; Based on the self-attention mechanism, the attention layer inputs the second entity encoding vector into the activation function, and then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the second entity encoding vector to obtain the attention score of the second entity; The attention score of the search result and the attention score of the second entity are concatenated, and then outer product fused with the audio encoding vector and the picture encoding vector to obtain the second feature vector.

[0308] The specific details of the search terms input by the above-mentioned user, and the search results corresponding to the search terms after being input into the trained correlation prediction model, are described in detail in the previously described correlation prediction model training method process. Those skilled in the art can refer to the relevant description after the first search term and the first search result are input into the correlation prediction model, which will not be repeated here.

[0309] It should be noted that, in one embodiment shown, before the feature coding layer performs feature coding on the search item and the search result respectively, it is also possible to:

[0310] Determine whether the first feature coding vector corresponding to the search item and / or the second feature coding vector corresponding to the search result are pre-stored; if so, obtain the first feature coding vector and / or the second feature coding vector.

[0311] Specifically, in the second round of training, after the loss function converges, the feature coding vectors corresponding to the search terms and search results in the sample pairs can be stored. When actual correlation prediction is performed based on the trained correlation prediction model, for the user input search terms and the search results corresponding to the search terms provided by the search engine, it can be determined whether the first feature coding vector corresponding to the search terms is pre-stored, and whether the second feature coding vector corresponding to the search results is pre-stored.

[0312] Furthermore, if there is a first feature coding vector, the first feature coding vector can be directly called without the need to perform feature coding again. Similarly, if there is a second feature coding vector, it can also be directly called, thereby eliminating the calculation process, improving the calculation efficiency when using the online correlation prediction model for correlation prediction, and saving time.

[0313] In addition, after determining the predicted score of the relevance between the search term input by the user and the search results, the predicted score can also be used as one of the basis for sorting and input into the sorting model, and finally the sorted search results are presented to the user on the search interface.

[0314] Through the above technical solution, a training sample set that is more in line with the correlation rules can be obtained. By adopting a large-scale sample constructed based on the click data of the search results and training in a weakly supervised manner, the prediction effect of the model is improved compared to directly using whether or not a click is used as a sample for training; and continuing to train the trained model with precise samples of manually labeled correlation results can further improve the prediction effect of the model, which greatly alleviates the poor prediction effect caused by the deviation during sample construction and the problem that the prediction effect and timeliness performance cannot be taken into account at the same time.

[0315] Exemplary Devices

[0316] After introducing the method of the exemplary embodiment of the present disclosure, next, see Fig. 9 , Fig. 9 It is a block diagram of a correlation prediction model training device provided by an exemplary embodiment.

[0317] The implementation process of the functions and effects of each module in the following device is specifically described in the implementation process of the corresponding steps in the above-mentioned correlation prediction model training method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment.

[0318] like Fig. 9 As shown, the correlation prediction model training device 900 may include: an acquisition module 901, a first training module 902 and a second training module 903, wherein:

[0319] The acquisition module 901 is configured to acquire a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result;

[0320] A first training module 902 is configured to input the first sample pair in the first training sample set into the correlation prediction model for training, to obtain a correlation prediction model after a first round of training;

[0321] The second training module 903 is configured to input the second sample pair in the second training sample set into the correlation prediction model after the first round of training to train again, so as to obtain the correlation prediction model after the second round of training.

[0322] In one embodiment, the first training module 902 further:

[0323] Inputting the first search item and the first search result in the first sample pair into the relevance prediction model respectively, wherein the relevance prediction model comprises: a feature encoding layer, a concatenation layer and a fully connected layer;

[0324] The feature coding layer performs feature coding on the first search item and the first search result respectively, and generates corresponding first feature coding vectors and second feature coding vectors respectively;

[0325] The first feature coding vector and the second feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0326] After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output;

[0327] According to the first true label of the first sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained until the loss function converges, thereby obtaining the correlation prediction model after the first round of training.

[0328] In one embodiment, the relevance prediction model further includes: an attention layer;

[0329] The first training module 902 further:

[0330] Based on the self-attention mechanism, the attention layer extracts features from the first feature coding vector and the second feature coding vector respectively to obtain corresponding first feature vectors and second feature vectors respectively;

[0331] The first eigenvector and the second eigenvector are concatenated to obtain a concatenated vector.

[0332] In one embodiment, the first training module 902 further:

[0333] The feature encoding layer performs feature encoding on the text of the first search item to obtain a first search item encoding vector;

[0334] Performing entity recognition on the text of the first search item, encoding phrase information after entity recognition, and obtaining a first entity encoding vector;

[0335] The feature encoding layer performs feature encoding on the text of the first search result to obtain a first search result encoding vector;

[0336] Performing entity recognition on the text of the first search result, encoding phrase information after entity recognition, and obtaining a second entity encoding vector;

[0337] The feature coding layer performs feature coding on the audio information of the first search result to obtain an audio coding vector;

[0338] The feature coding layer performs feature coding on the picture information of the first search result to obtain a picture coding vector;

[0339] The first feature encoding vector includes: the first search term encoding vector and the first entity encoding vector;

[0340] The second feature coding vector includes: the first search result coding vector, a second entity coding vector, an audio coding vector and a picture coding vector.

[0341] In one embodiment, the first training module 902 further:

[0342] Based on the self-attention mechanism, the attention layer inputs the first search term encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first search term encoding vector to obtain an attention score of the first search term;

[0343] Based on the self-attention mechanism, the attention layer inputs the first entity encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first entity encoding vector to obtain an attention score of the first entity;

[0344] Concatenate the attention score of the first search term and the attention score of the first entity to obtain the first feature vector;

[0345] Based on the self-attention mechanism, the attention layer inputs the first search result encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first search result encoding vector to obtain an attention score of the first search result;

[0346] Based on the self-attention mechanism, the attention layer inputs the second entity encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the second entity encoding vector to obtain an attention score of the second entity;

[0347] The attention score of the first search result and the attention score of the second entity are concatenated, and then outer-product fused with the audio coding vector and the picture coding vector to obtain the second feature vector.

[0348] In one embodiment, the first training module 902 further:

[0349] The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

[0350] In one embodiment, the second training module 903 further:

[0351] Inputting the second search item and the second search result in the second sample pair into the correlation prediction model after the first round of training respectively, wherein the correlation prediction model includes: a feature encoding layer, a concatenation layer and a fully connected layer;

[0352] The feature coding layer performs feature coding on the second search item and the second search result respectively, and generates a corresponding third feature coding vector and a fourth feature coding vector respectively;

[0353] The third feature coding vector and the fourth feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0354] After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output;

[0355] According to the second true label of the second sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained again until the loss function converges, thereby obtaining a correlation prediction model after the second round of training.

[0356] In one embodiment, the correlation prediction model training device 900 further includes:

[0357] A first division module 904, pre-dividing click rate intervals and setting a first real label corresponding to any click rate interval;

[0358] The second division module 905 pre-divides the correlation level and sets a second real label corresponding to any level;

[0359] A first real label determination module 906, which determines a corresponding click rate interval and a corresponding first real label according to the click data of any training sample;

[0360] The second real label determination module 907 determines the corresponding gear position and the corresponding second real label according to the correlation degree of any training sample.

[0361] The specific details of each module of the above-mentioned correlation prediction model training device 900 have been described in detail in the previous description of the correlation prediction model training method process, and therefore, they will not be repeated here.

[0362] Corresponding to the above correlation prediction method, see Fig.10 , Fig.10 It is a block diagram of a correlation prediction device provided by an exemplary embodiment.

[0363] The implementation process of the functions and effects of each module in the following device is specifically detailed in the implementation process of the corresponding steps in the above correlation prediction method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment.

[0364] like Fig.10 As shown, the correlation prediction device 1000 may include: a first acquisition module 1001, an input module 1002 and an output module 1003, wherein:

[0365] A first acquisition module 1001 is configured to acquire a search term input by a user and search results corresponding to the search term;

[0366] An input module 1002 is configured to input the search terms and the search results into a trained relevance prediction model;

[0367] The output module 1003 is configured to output the predicted scores of the search terms and the search results based on the trained relevance prediction model.

[0368] In one embodiment, the trained correlation prediction model includes: a feature encoding layer, a concatenation layer, and a fully connected layer;

[0369] The output module 1003 further:

[0370] The feature coding layer performs feature coding on the search item and the search result respectively, and generates corresponding first feature coding vectors and second feature coding vectors respectively;

[0371] The first feature coding vector and the second feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector;

[0372] After the concatenated vector is processed by a fully connected layer and sigmoid, the prediction score is output.

[0373] In one embodiment, the trained correlation prediction model further includes: an attention layer;

[0374] The output module 1003 further:

[0375] Based on the self-attention mechanism, the attention layer extracts features from the first feature coding vector and the second feature coding vector respectively to obtain corresponding first feature vectors and second feature vectors respectively;

[0376] The first eigenvector and the second eigenvector are concatenated to obtain a concatenated vector.

[0377] In one embodiment, the correlation prediction device 1000 further includes:

[0378] A determination module 1004 is configured to determine whether the first feature coding vector corresponding to the search item and / or the second feature coding vector corresponding to the search result is pre-stored.

[0379] The second acquisition module 1005 acquires the first feature coding vector and / or the second feature coding vector, if any.

[0380] In one embodiment, the output module 1003 further:

[0381] The feature encoding layer performs feature encoding on the text of the search term to obtain a search term encoding vector;

[0382] Performing entity recognition on the text of the search item, encoding phrase information after entity recognition, and obtaining a first entity encoding vector;

[0383] The feature encoding layer performs feature encoding on the text of the search result to obtain a search result encoding vector;

[0384] Performing entity recognition on the text of the search result, encoding the phrase information after entity recognition, and obtaining a second entity encoding vector;

[0385] The feature coding layer performs feature coding on the audio information of the search result to obtain an audio coding vector;

[0386] The feature coding layer performs feature coding on the image information of the search result to obtain an image coding vector;

[0387] The first feature encoding vector includes: the search term encoding vector and the first entity encoding vector;

[0388] The second feature coding vector includes: the search result coding vector, a second entity coding vector, an audio coding vector and a picture coding vector.

[0389] In one embodiment, the output module 1003 further:

[0390] Based on the self-attention mechanism, the attention layer inputs the search term encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first search term encoding vector to obtain an attention score of the search term;

[0391] Based on the self-attention mechanism, the attention layer inputs the first entity encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the first entity encoding vector to obtain an attention score of the first entity;

[0392] Concatenate the attention score of the search term and the attention score of the first entity to obtain the first feature vector;

[0393] Based on the self-attention mechanism, the attention layer inputs the search result encoding vector into an activation function, then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the search result encoding vector to obtain the attention score of the search result;

[0394] Based on the self-attention mechanism, the attention layer inputs the second entity encoding vector into an activation function, performs convolution calculation on the calculation result of the activation function, and performs a dot product operation on the result of the convolution calculation and the second entity encoding vector to obtain an attention score of the second entity;

[0395] The attention score of the search result and the attention score of the second entity are concatenated, and then outer-product fused with the audio coding vector and the picture coding vector to obtain the second feature vector.

[0396] In one embodiment, the output module 1003 further:

[0397] The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

[0398] The specific details of each module of the above-mentioned correlation prediction device 1000 have been described in detail in the previous description of the correlation prediction model training method process, so they will not be repeated here.

[0399] It should be noted that although the correlation prediction model training device 900 and several modules or units of the correlation prediction device 1000 are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0400] Exemplary Media

[0401] After introducing the apparatus of the exemplary embodiment of the present disclosure, next, see Fig.11 , Fig.11 It is a schematic diagram of a medium corresponding to the above method provided by an exemplary embodiment.

[0402] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0403] refer to Fig.11 As shown, a readable storage medium 1100 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the readable storage medium of the present disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0404] The readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0405] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0406] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0407] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0408] Exemplary Computing Devices

[0409] After introducing the method, apparatus and medium of the exemplary embodiments of the present disclosure, see Fig.12 , Fig.12 It is a schematic diagram of an electronic device capable of implementing the above method provided by an exemplary embodiment.

[0410] Refer to the following Fig.12 The electronic device 1200 according to this embodiment of the present disclosure is described in detail. The electronic device 1200 shown in FIG12 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.

[0411] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include but are not limited to: at least one processing unit 1201, at least one storage unit 1202, and a bus 1203 connecting different system components (including the storage unit 1202 and the processing unit 1201).

[0412] The storage unit stores program codes, and the program codes can be executed by the processing unit 1201, so that the processing unit 1201 executes the steps of the above-mentioned various embodiments of this specification.

[0413] The storage unit 1202 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 12021 and / or a cache storage unit 12022 , and may further include a read-only storage unit (ROM) 12023 .

[0414] The storage unit 1202 may also include a program / utility 12024 having a set (at least one) of program modules 12025, such program modules 12025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the reality of a network environment.

[0415] The bus 1203 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0416] The electronic device 1200 may also communicate with one or more external devices 1204 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or may communicate with any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1205. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1206. As shown, the network adapter 1206 communicates with other modules of the electronic device 1200 via a bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0417] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0418] It should be noted that, although several units / modules or subunits / modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0419] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0420] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A correlation prediction model training method, comprising: Acquire a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result; Inputting the first search item and the first search result in the first sample pair into the relevance prediction model respectively, wherein the relevance prediction model comprises: a feature encoding layer, a concatenation layer, a fully connected layer and an attention layer; The feature encoding layer performs feature encoding on the text of the first search item to obtain a first search item encoding vector; performs entity recognition on the text of the first search item, and encodes the phrase information after entity recognition to obtain a first entity encoding vector; performs feature encoding on the text of the first search result to obtain a first search result encoding vector; performs entity recognition on the text of the first search result, and encodes the phrase information after entity recognition to obtain a second entity encoding vector; performs feature encoding on the audio information of the first search result to obtain an audio encoding vector; performs feature encoding on the picture information of the first search result to obtain a picture encoding vector; Based on the self-attention mechanism, the attention layer inputs the first search term encoding vector, the first entity encoding vector, the first search result encoding vector, and the second entity encoding vector as input vectors to the activation function, and then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the corresponding input vector to obtain the attention score of the first search term, the attention score of the first entity, the attention score of the first search result, and the attention score of the second entity respectively; Concatenate the attention score of the first search item and the attention score of the first entity to obtain a first feature vector, and concatenate the attention score of the first search result and the attention score of the second entity, and then perform outer product fusion with the audio coding vector and the picture coding vector to obtain a second feature vector; Performing vector concatenation on the first eigenvector and the second eigenvector to obtain a concatenated vector; After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output; Iteratively training the correlation prediction model according to the first true label of the first sample pair, the prediction score and a preset loss function until the loss function converges, thereby obtaining a correlation prediction model after a first round of training; The second sample pairs in the second training sample set are input into the correlation prediction model after the first round of training to be trained again, so as to obtain the correlation prediction model after the second round of training.

2. According to the method of claim 1, concatenating the first feature vector and the second feature vector to obtain a concatenated vector, comprising: The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

3. The method according to claim 1, inputting the second sample pair in the second training sample set into the correlation prediction model after the first round of training to train again, so as to obtain the correlation prediction model after the second round of training, comprises: Inputting the second search item and the second search result in the second sample pair into the correlation prediction model after the first round of training respectively; The feature coding layer performs feature coding on the second search item and the second search result respectively, and generates a corresponding third feature coding vector and a fourth feature coding vector respectively; The third feature coding vector and the fourth feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector; After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output; According to the second true label of the second sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained again until the loss function converges, thereby obtaining a correlation prediction model after the second round of training.

4. The method according to claim 1, further comprising: Pre-dividing click rate intervals and setting a first real label corresponding to any click rate interval; Pre-dividing the correlation level and setting the second real label corresponding to any level; Determine a corresponding click rate interval and a corresponding first true label according to the click data of any training sample; The corresponding gear position and the corresponding second true label are determined according to the relevance degree of any training sample.

5. A correlation prediction method, comprising: Obtaining a search term input by a user and search results corresponding to the search term; Inputting the search terms and the search results into a trained relevance prediction model; wherein the trained relevance prediction model comprises: a feature encoding layer, a concatenation layer, a fully connected layer, and an attention layer; The feature coding layer performs feature coding on the text of the search item to obtain a search item coding vector; performs entity recognition on the text of the search item, and encodes the phrase information after entity recognition to obtain a first entity coding vector; performs feature coding on the text of the search result to obtain a search result coding vector; performs entity recognition on the text of the search result, and encodes the phrase information after entity recognition to obtain a second entity coding vector; performs feature coding on the audio information of the search result to obtain an audio coding vector; performs feature coding on the picture information of the search result to obtain a picture coding vector; Based on the self-attention mechanism, the attention layer inputs the search term encoding vector, the first entity encoding vector, the first search result encoding vector, and the second entity encoding vector as input vectors to the activation function, then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the corresponding input vector to obtain the attention score of the search term, the attention score of the first entity, the attention score of the search result, and the attention score of the second entity respectively; The attention score of the search item and the attention score of the first entity are concatenated to obtain a first feature vector; and the attention score of the search result and the attention score of the second entity are concatenated, and then the outer product fusion is performed with the audio coding vector and the picture coding vector to obtain a second feature vector; Performing vector concatenation on the first eigenvector and the second eigenvector to obtain a concatenated vector; After the concatenated vector is processed by a fully connected layer and sigmoid, the predicted scores of the search term and the search result are output.

6. The method according to claim 5, before the feature encoding layer performs feature encoding on the text of the search term, the method further comprises: Determine whether a first feature coding vector corresponding to the search item and / or a second feature coding vector corresponding to the search result is pre-stored; wherein the first feature coding vector includes: the search item coding vector and the first entity coding vector; the second feature coding vector includes: the search result coding vector, the second entity coding vector, the audio coding vector and the picture coding vector; If yes, then the first feature coding vector and / or the second feature coding vector are obtained.

7. The method according to claim 5, concatenating the first feature vector and the second feature vector to obtain a concatenated vector, comprising: The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

8. A correlation prediction model training device, comprising: An acquisition module, for acquiring a training sample set; wherein the training sample set includes a first training sample set constructed based on the collected user click data on the search results, and a second training sample set consisting of manually annotated correlation results; any training sample includes a sample pair consisting of a search term and a search result; A first training module, inputting the first search item and the first search result in the first sample pair into the relevance prediction model respectively, wherein the relevance prediction model includes: a feature encoding layer, a concatenation layer, a fully connected layer and an attention layer; The feature encoding layer performs feature encoding on the text of the first search item to obtain a first search item encoding vector; performs entity recognition on the text of the first search item, and encodes the phrase information after entity recognition to obtain a first entity encoding vector; performs feature encoding on the text of the first search result to obtain a first search result encoding vector; performs entity recognition on the text of the first search result, and encodes the phrase information after entity recognition to obtain a second entity encoding vector; performs feature encoding on the audio information of the first search result to obtain an audio encoding vector; performs feature encoding on the picture information of the first search result to obtain a picture encoding vector; Based on the self-attention mechanism, the attention layer inputs the first search term encoding vector, the first entity encoding vector, the first search result encoding vector, and the second entity encoding vector as input vectors to the activation function, and then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the corresponding input vector to obtain the attention score of the first search term, the attention score of the first entity, the attention score of the first search result, and the attention score of the second entity respectively; Concatenate the attention score of the first search item and the attention score of the first entity to obtain a first feature vector, and concatenate the attention score of the first search result and the attention score of the second entity, and then perform outer product fusion with the audio coding vector and the picture coding vector to obtain a second feature vector; Performing vector concatenation on the first eigenvector and the second eigenvector to obtain a concatenated vector; After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output; Iteratively training the correlation prediction model according to the first true label of the first sample pair, the prediction score and a preset loss function until the loss function converges, thereby obtaining a correlation prediction model after a first round of training; The second training module inputs the second sample pair in the second training sample set into the correlation prediction model after the first round of training to perform retraining to obtain the correlation prediction model after the second round of training.

9. The apparatus according to claim 8, wherein the first training module further comprises: The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

10. The apparatus according to claim 8, wherein the second training module further comprises: Inputting the second search item and the second search result in the second sample pair into the correlation prediction model after the first round of training respectively; The feature coding layer performs feature coding on the second search item and the second search result respectively, and generates a corresponding third feature coding vector and a fourth feature coding vector respectively; The third feature coding vector and the fourth feature coding vector are concatenated through the concatenation layer to obtain a concatenated vector; After the concatenated vector is processed by the fully connected layer and sigmoid, the prediction score is output; According to the second true label of the second sample pair, the prediction score and a preset loss function, the correlation prediction model is iteratively trained again until the loss function converges, thereby obtaining a correlation prediction model after the second round of training.

11. The device according to claim 8, further comprising: A first division module, pre-dividing click rate intervals and setting a first real label corresponding to any click rate interval; A second division module is used to pre-divide the correlation level and set a second real label corresponding to any level; A first real label determination module, which determines a corresponding click rate interval and a corresponding first real label according to the click data of any training sample; The second real label determination module determines the corresponding gear position and the corresponding second real label according to the relevance degree of any training sample.

12. A correlation prediction device, comprising: A first acquisition module, acquiring a search term input by a user and search results corresponding to the search term; An input module, inputting the search terms and the search results into a trained relevance prediction model; wherein the trained relevance prediction model includes: a feature encoding layer, a concatenation layer, a fully connected layer, and an attention layer; Output module, the feature coding layer performs feature coding on the text of the search item to obtain a search item coding vector; performs entity recognition on the text of the search item, and encodes the phrase information after entity recognition to obtain a first entity coding vector; performs feature coding on the text of the search result to obtain a search result coding vector; performs entity recognition on the text of the search result, and encodes the phrase information after entity recognition to obtain a second entity coding vector; performs feature coding on the audio information of the search result to obtain an audio coding vector; performs feature coding on the picture information of the search result to obtain a picture coding vector; Based on the self-attention mechanism, the attention layer inputs the search term encoding vector, the first entity encoding vector, the first search result encoding vector, and the second entity encoding vector as input vectors to the activation function, then performs convolution calculation on the calculation result of the activation function, and performs dot product operation on the result of the convolution calculation and the corresponding input vector to obtain the attention score of the search term, the attention score of the first entity, the attention score of the search result, and the attention score of the second entity respectively; The attention score of the search item and the attention score of the first entity are concatenated to obtain a first feature vector; and the attention score of the search result and the attention score of the second entity are concatenated, and then the outer product fusion is performed with the audio coding vector and the picture coding vector to obtain a second feature vector; Performing vector concatenation on the first eigenvector and the second eigenvector to obtain a concatenated vector; After the concatenated vector is processed by a fully connected layer and sigmoid, the predicted scores of the search term and the search result are output.

13. The device according to claim 12, further comprising: A judgment module is used to judge whether a first feature coding vector corresponding to the search item and / or a second feature coding vector corresponding to the search result is pre-stored; wherein the first feature coding vector includes: the search item coding vector and the first entity coding vector; the second feature coding vector includes: the search result coding vector, the second entity coding vector, the audio coding vector and the picture coding vector; The second acquisition module, if any, acquires the first feature coding vector and / or the second feature coding vector.

14. The device according to claim 12, wherein the output module further comprises: The first eigenvector, the second eigenvector, the vector sum of the first eigenvector and the second eigenvector, and the vector difference between the first eigenvector and the second eigenvector are vector-concatenated to obtain a concatenated vector.

15. A storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

16. A computing device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

Images