Training method of search result sorting model, search result sorting method, training device of search result sorting model, search result sorting device, electronic device, computer readable storage medium, and computer program

Abstract

To provide a training method of a search result sorting model and a search result sorting method.SOLUTION: A method includes steps of: obtaining a plurality of first data pairs, single target features in which search results in the plurality of first data pairs correspond to a plurality of search targets, and an annotation score in which the plurality of first data pairs respectively correspond to the plurality of search targets; obtaining a plurality of single search target sorting models by training based on the obtained data; obtaining a plurality of second data pairs and multi-target features in which respective search results in the plurality of second data pairs correspond to all search targets; and scoring the respective search results in the second data pairs using the plurality of single search target sorting models corresponding to the plurality of search targets to determine training search targets of the second data pairs; and obtaining the search result sorting models by training based on queries in the plurality of second data pairs, the multi-target features for which the respective search results correspond to all search targets, and scores for which the respective search results correspond to the training search targets.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to computer technology, particularly to the field of artificial intelligence technology such as natural language processing and deep learning, and specifically to a method, apparatus, electronic device, and readable storage medium for training a search result sorting model and sorting search results. [Background technology]

[0002] With the rapid development of search engine technology, users are increasingly using search engines to obtain various search results. When dealing with a large number of search results, there is a problem of sorting the search results and presenting them to users. Summary of the Invention [Means for solving the problem]

[0003] According to a first aspect of the present disclosure, a plurality of first data pairs, single-target features of search results in the plurality of first data pairs corresponding to a plurality of search targets, and annotation scores of the plurality of first data pairs corresponding to the plurality of search targets are obtained, each of the first data pairs including a query and one corresponding search result. For each search target, a first pre-trained language model is trained based on the query in the plurality of first data pairs, the single-target features of search results in the plurality of first data pairs corresponding to the search target, and the annotation scores of the plurality of first data pairs corresponding to the search target to obtain a single-search target sorting model corresponding to the search target; a plurality of second data pairs, and each search result in the plurality of second data pairs corresponding to all the search targets. and corresponding multi-target features, each second data pair including a query and two corresponding search results; for each second data pair, scoring each search result in the second data pair using a plurality of single-search target sorting models corresponding to the plurality of search targets; determining a training search target for the second data pair based on scores for each search result corresponding to the plurality of search targets; and training a second pre-trained language model based on the queries in the plurality of second data pairs, the multi-target features for each search result in the plurality of second data pairs corresponding to all the search targets, and the scores for each search result in the plurality of second data pairs corresponding to the training search target to obtain a search result sorting model.

[0004] According to a second aspect of the present disclosure, there is provided a search result sorting method including: obtaining a query to be processed and a plurality of corresponding search results; for each search result, obtaining multi-target features for which the search result corresponds to all search targets; for each search result, inputting the query to be processed and the multi-target features for which the search result corresponds to all search targets into a search result sorting model to obtain a score for the search result output by the search result sorting model; and sorting the plurality of search results based on the scores of the plurality of search results.

[0005] According to a third aspect of the present disclosure, there is provided a first acquisition unit that acquires a plurality of first data pairs, single-target features of search results in the plurality of first data pairs corresponding to a plurality of search targets, and annotation scores of the plurality of first data pairs corresponding to the plurality of search targets, each of the first data pairs including a query and one corresponding search result; a first training unit that, for each search target, trains a first pre-trained language model based on the query in the plurality of first data pairs, the single-target features of search results in the plurality of first data pairs corresponding to the search target, and the annotation scores of the plurality of first data pairs corresponding to the search target to obtain a single-search target sorting model corresponding to the search target; and a plurality of second data pairs, and acquires a single-search target sorting model corresponding to the search target, each of the search results in the plurality of second data pairs corresponding to all of the search targets. a processing unit that, for each second data pair, scores each search result in the second data pair using a plurality of single-search-target sorting models corresponding to the plurality of search targets, and determines a training search target for the second data pair based on scores for each search result corresponding to the plurality of search targets; and a second training unit that trains a second pre-trained language model to obtain a search result sorting model based on the queries in the plurality of second data pairs, the multi-target features for each search result in the plurality of second data pairs corresponding to all the search targets, and scores for each search result in the plurality of second data pairs corresponding to the training search target.

[0006] According to a fourth aspect of the present disclosure, there is provided a search result sorting device including a third acquisition unit that acquires a query to be processed and a plurality of corresponding search results; a fourth acquisition unit that acquires, for each search result, multi-target features for which the search result corresponds to all search targets; a scoring unit that inputs, for each search result, the query to be processed and the multi-target features for which the search result corresponds to all search targets into a search result sorting model and acquires a score for the search result output by the search result sorting model; and a sorting unit that sorts the plurality of search results according to the scores of the plurality of search results.

[0007] According to a fifth aspect of the present disclosure, there is provided an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores commands executable by the at least one processor, and wherein the commands, when executed by the at least one processor, cause the at least one processor to perform the method described above.

[0008] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the method set forth above.

[0009] According to a seventh aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method set out above.

[0010] As can be seen from the above technical solution, according to the present disclosure, there is no need to annotate training data for multiple search targets, and only the training data for a single search target is required, thereby reducing the cost and difficulty of data annotation. Furthermore, the search result sorting model obtained through training can achieve the goal of integrated scoring of search results across multiple search targets, thereby improving the accuracy of the search result sorting model when sorting search results.

[0011] It should be understood that the contents described in this section are not intended to identify key or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. [Figure 2] FIG. 10 is a schematic diagram according to a second embodiment of the present disclosure. [Figure 3] FIG. 10 is a schematic diagram according to a third embodiment of the present disclosure. [Figure 4] FIG. 10 is a schematic diagram according to a fourth embodiment of the present disclosure. [Figure 5] FIG. 10 is a schematic diagram according to a fifth embodiment of the present disclosure. [Figure 6] FIG. 10 is a schematic diagram according to a sixth embodiment of the present disclosure. [Figure 7] FIG. 10 is a schematic diagram according to a seventh embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram of an electronic device for implementing a search result sorting model training or search result sorting method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, exemplary embodiments of the present application will be described based on the drawings. For ease of understanding, various details of the embodiments of the present application are included and should be considered as merely examples. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of brevity, the following description will omit descriptions of well-known functions and structures.

[0014] 1 is a schematic diagram according to the first embodiment of the present disclosure. As shown in FIG. 1, the method for training a search result sorting model of this embodiment specifically includes the following steps:

[0015] In S101, a plurality of first data pairs, single-target features in which search results in the plurality of first data pairs respectively correspond to a plurality of search targets, and annotation scores in which the plurality of first data pairs respectively correspond to the plurality of search targets are obtained, and each first data pair includes a query and one corresponding search result.

[0016] In S102, for each search target, a first pre-trained language model is trained based on the queries in the plurality of first data pairs, the single-target features of the search results in the plurality of first data pairs corresponding to the search target, and the annotation scores of the plurality of first data pairs corresponding to the search target, to obtain a single-search target sorting model corresponding to the search target.

[0017] In S103, a plurality of second data pairs and a multi-target feature are obtained, where each search result in the plurality of second data pairs corresponds to all search targets, and each second data pair includes a query and two corresponding search results.

[0018] At S104, for each second data pair, a plurality of single search target sorting models corresponding to the plurality of search targets are used to score each search result in the second data pair, and a training search target for the second data pair is determined based on the scores of each search result corresponding to the plurality of search targets.

[0019] In S105, a second pre-trained language model is trained based on the queries in the plurality of second data pairs, the multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and the scores in which each search result in the plurality of second data pairs corresponds to the training search target to obtain a search result sorting model.

[0020] According to the method for training a search result sorting model of this embodiment, single-search target sorting models corresponding to different search targets are first trained based on training data corresponding to a single search target, then multiple single-search target sorting models are used to construct training data corresponding to multiple search targets, and finally, search result sorting models corresponding to all search targets are trained based on the training data corresponding to the multiple search targets. In this embodiment, there is no need to annotate the training data for multiple search targets, and only the training data for a single search target is required, thereby reducing the cost and difficulty of data annotation, and the trained search result sorting model can achieve the goal of integrated scoring of search results across multiple search targets, thereby improving the accuracy of the search result sorting model when sorting search results.

[0021] In this embodiment, the first data pair obtained by performing S101 includes a query and a corresponding search result, where the search result in the first data pair may be an entire web page, an entire document, or an entire multimedia data, or may be a paragraph in the web page, document, or multimedia data.

[0022] In practical applications, after a search engine obtains multiple search results corresponding to a query, it needs to score each search result and sort the multiple search results according to the scoring results. Unless the search engine combines multiple search goals, such as relevance goals, authority goals, and timeliness goals, and scores each search result, it cannot obtain more accurate scoring results.

[0023] Therefore, in this embodiment, the multiple search goals that are preset are multiple goals such as the above-mentioned relevance goal (representing the correlation between the search results and the query), authority goal (representing the authority of the search results), and timeliness goal (representing the timeliness of the search results).

[0024] In this embodiment, after executing S101 to obtain the first data pair, multiple search targets may be determined first, and then the single-target features of the search results in the multiple first data pairs corresponding to the multiple search targets respectively and the annotation scores of the multiple first data pairs corresponding to the multiple search targets respectively may be obtained.

[0025] In this embodiment, when S101 is executed to obtain annotation scores corresponding to multiple first data pairs respectively corresponding to multiple search targets, the annotation scores corresponding to each search target of the multiple first data pairs can be obtained by a preset annotation method.

[0026] For example, in this embodiment, when the multiple first data pairs obtained by executing S101 include data pair 1 (query1, para1), and the multiple search targets are correlation targets, authoritative targets, and timeliness targets, when executing S101, this embodiment can obtain, through manual annotation, the annotation score for data pair 1 corresponding to the correlation target, the annotation score for data pair 1 corresponding to the authoritative target, and the annotation score for data pair 1 corresponding to the timeliness target, respectively.

[0027] In this embodiment, when S101 is executed to obtain single-target features of search results in multiple first data pairs corresponding to multiple search goals, attribute information of the search results in multiple first data pairs is obtained, which may include information such as the title, summary, publication time, publication site, publisher, and publication address of the search results, and for each search goal, the attribute information among the attribute information corresponding to the search goal is used as the single-target feature of search results in multiple first data pairs corresponding to the search goal.

[0028] In other words, the different single-target features acquired in this embodiment correspond to different search targets, and the attribute information included in the single-target features whose search results correspond to the same search target may be one or more.

[0029] For example, in this embodiment, the attribute information corresponding to the correlation target obtained by executing S101 may be the title and / or abstract of the search result, the attribute information corresponding to the timeliness target may be the publication time of the search result, and the attribute information corresponding to the authority target may be at least one of the publication site, publisher, publication address, etc. of the search result.

[0030] In this embodiment, in S101, when, for each search target, information among the attribute information corresponding to the search target is used as a single-target feature corresponding to the search target in the search results in multiple first data pairs, if it is determined that the attribute information among the attribute information corresponding to the search target is not text, the attribute information corresponding to the search target is converted into text, and the text conversion result is used as a single-target feature corresponding to the search target in the search results in multiple first data pairs.

[0031] That is, this embodiment can ensure that the acquired single-target feature is text and avoid the use of other complex forms of content, thereby reducing the difficulty of model training and improving model training efficiency.

[0032] In this embodiment, when S101 is executed, the attribute information corresponding to the search target can be obtained from the attribute information of the search results as a single-target feature corresponding to each search target based on the correspondence between the search target and attribute information that has been set in advance, thereby improving the accuracy of the obtained single-target feature.

[0033] For example, in this embodiment, when the multiple first data pairs obtained by executing S101 include data pair 1 (query1, para1), and the multiple search targets are respectively a correlation target, an authoritative target, and a time-to-date target, when executing S101 in this embodiment, the title and abstract of para1 can be obtained as a single-target feature corresponding to the correlation target, the publication time of para1 can be obtained as a single-target feature corresponding to the time-to-date target, and the publication site of para1 can be obtained as a single-target feature corresponding to the authoritative target.

[0034] In this embodiment, S101 is executed to obtain a plurality of first data pairs, single-target features of the search results in the plurality of first data pairs corresponding to each of the plurality of search targets, and annotation scores of the plurality of first data pairs corresponding to each of the plurality of search targets, and then S102 is executed to train a first pre-trained language model for each search target based on the query in the plurality of first data pairs, the single-target features of the search results in the plurality of first data pairs corresponding to the search target, and the annotation scores of the plurality of first data pairs corresponding to the search target, to obtain a single-search target sorting model corresponding to the search target.

[0035] It should be understood that in this embodiment, multiple single search target sorting models are obtained by performing S102, and different single search target sorting models correspond to different search targets.

[0036] Specifically, in this embodiment, in S102, for each search target, when a first pre-trained language model is trained based on queries in a plurality of first data pairs, single-target features whose search results in a plurality of first data pairs correspond to the search target, and annotation scores whose plurality of first data pairs correspond to the search target, to obtain a single-search target sorting model corresponding to the search target, the queries in the plurality of first data pairs and the single-target features whose search results in a plurality of first data pairs correspond to the search target are input into the first pre-trained language model for each search target to obtain a predicted score to be output by the first pre-trained language model for each first data pair, and a loss function value is calculated based on the predicted scores of the plurality of first data pairs and the annotation scores whose plurality of first data pairs correspond to the search target. In this embodiment, the loss function value can be calculated using a cross-entropy loss function calculation method, and the parameters of the first pre-trained language model are adjusted based on the calculated loss function value until the first pre-trained language model converges to obtain a single-search target sorting model corresponding to the search target.

[0037] For example, if the multiple search targets are correlation targets, authoritative targets, and time-sensitive targets, executing S102 in this embodiment will result in a single search target sorting model corresponding to the correlation targets, a single search target sorting model corresponding to the authoritative targets, and a single search target sorting model corresponding to the time-sensitive targets. The above three single search target sorting models are used to output correlation scoring results, authoritative scoring results, and time-sensitive scoring results of the search results, respectively.

[0038] That is, in this embodiment, different single-search target sorting models are obtained according to different search targets, and the single-search target sorting models corresponding to different search targets can output scoring results corresponding to different search targets based on the single-target features of the input query and its search results corresponding to different search targets.

[0039] It should be understood that the first pre-trained language model trained by performing S102 in this embodiment is a neural network model based on a transformer network, and may be an Ernie model.

[0040] In this embodiment, S102 is performed to obtain a single-search target sorting model corresponding to multiple search targets, and then S103 is performed to obtain multiple second data pairs and multi-target features in which each search result in the multiple second data pairs corresponds to all the search targets.

[0041] The second data pair obtained by performing S103 in this embodiment includes a query and two corresponding search results, where the two search results in the second data pair may be an entire web page, an entire document, or an entire multimedia data, or may be a paragraph in the web page, document, or multimedia data.

[0042] In this embodiment, when executing S103 to obtain multiple second data pairs, first, multiple queries are obtained, then multiple search results corresponding to each query are obtained, and finally, for the same query, any two of the multiple search results corresponding to the query are selected and combined to obtain multiple second data pairs including the query and the two corresponding search results.

[0043] In this embodiment, when S103 is executed to obtain multi-target features for each search result in multiple second data pairs corresponding to all search targets, an optional implementation method may be adopted in which attribute information for each search result in multiple second data pairs is obtained, and the attribute information of each search result is used as a multi-target feature for each search result in multiple second data pairs corresponding to all search targets.

[0044] That is, the multi-target features acquired in this embodiment are all attribute information of the search results, that is, the multi-target features include all single-target features corresponding to different search targets in the search results.

[0045] In this embodiment, when S103 is executed to treat the attribute information of each search result in multiple second data pairs as a multi-target feature corresponding to all search targets for each search result, if it is determined that the attribute information of each search result is not text, the attribute information of the search result may be converted into text, and the text conversion result may be treated as a multi-target feature corresponding to all search targets for each search result.

[0046] That is, this embodiment can ensure that the acquired multi-target features are text, avoiding the use of other complex forms of content, reducing the difficulty of model training, and improving model training efficiency.

[0047] In this embodiment, S103 is executed to obtain multiple second data pairs and multi-target features in which each search result in the multiple second data pairs corresponds to all search targets, and then S104 is executed to score each search result in the second data pair using multiple single-search target sorting models corresponding to multiple search targets for each second data pair, and determine training search targets for the second data pair based on the scores in which each search result corresponds to multiple search targets.

[0048] In this embodiment, in S104, for each second data pair, multiple single-search target sorting models corresponding to multiple search targets are used to score each search result in the second data pair, and when determining the training search target for the second data pair based on the scores of each search result corresponding to the multiple search targets, the scoring order of the multiple single-search target sorting models is determined according to the priority of the search targets, and for each second data pair, the single-search target sorting model is used to score each search result in the second data pair in sequence according to the determined scoring order, and if it is determined that the score difference between the two search results exceeds the search target difference threshold corresponding to the current single-search target sorting model, the search target corresponding to the current single-search target sorting model is used as the training search target for the second data pair.

[0049] That is, in this embodiment, different single-search target sorting models are used to sequentially score the two search results in the second data pair according to a preset priority, and the training search target for the second data pair is determined. This improves the accuracy of the obtained training search target, and the determined training search target is used to train the search result sorting model in the next step. Therefore, there is no need to annotate the scores corresponding to the different search targets for the two search results in the second data pair, thereby reducing the difficulty and training cost of the model and improving the training efficiency of the model.

[0050] In this embodiment, priorities of different search targets are preset when S104 is executed. For example, if the preset priorities are "correlation target > time-sensitive target > authoritative target," the scoring order determined by executing S104 in this embodiment is as follows: first, scoring is performed using the single search target sorting model corresponding to the correlation target, then scoring is performed using the single search target sorting model corresponding to the time-sensitive target, and finally scoring is performed using the single search target sorting model corresponding to the authoritative target.

[0051] In this embodiment, when S104 is executed to score each search result in the second data pair using the single-search target sorting model, for each search result in the second data pair, the query in the second data pair and the single-target feature for which the search result corresponds to the current search target can be input into the single-search target sorting model corresponding to the current search target, and the score output by the single-search target sorting model for the search result can be obtained.

[0052] In this embodiment, when performing S104, difference thresholds corresponding to different search targets are also set in advance, and the purpose of determining the training search target of the second data pair is achieved based on the comparison result between the score difference between the search results and the difference threshold.

[0053] For example, if the single search target sorting model corresponding to the correlation target is Model 1, the single search target sorting model corresponding to the time-sensitive target is Model 2, and the single search target sorting model corresponding to the authoritative target is Model 3, and the scoring order determined by executing S104 in this embodiment is Model 1 to Model 2 to Model 3, then Model 1 is used to score the two search results. If it is determined that the score difference between the two search results in the second data pair exceeds the difference threshold for the correlation target, the correlation target is used as the training search target for the second data pair. Otherwise, Model 2 is used to continue scoring the two search results. If it is determined that the score difference between the two search results in the second data pair exceeds the difference threshold for the time-sensitive target, the time-sensitive target is used as the training search target for the second data pair. Otherwise, Model 3 is used to continue scoring the two search results. If it is determined that the score difference between the two search results in the second data pair exceeds the difference threshold for the authoritative target, the authoritative target is used as the training search target for the second data pair.

[0054] In this embodiment, when determining the training search target for the second data pair based on the score difference and the difference threshold in S104, there may be cases where the score difference between the two search results does not exceed the difference threshold for the different search targets. In this case, in this embodiment, when performing S104, the search target with the highest priority can be set as the training search target for such second data pair.

[0055] In addition, in this embodiment, in S104, for each second data pair, multiple single-search target sorting models corresponding to multiple search targets are used to score each search result in the second data pair, and when determining the training search target for this second data pair based on the scores of each search result corresponding to the multiple search targets, an optional implementation method may be adopted in which, for each second data pair, multiple single-search target sorting models corresponding to the multiple search targets are used to score each search result in this second data pair, and the search target corresponding to the largest score difference between the two search results is set as the training search target for this second data pair.

[0056] That is, in this embodiment, all single search target sorting models further determine the training search target for each second data pair based on the score of the search results output for each second data pair, so that the two search results in the second data pair have the most significant difference in the determined training search target, thereby improving the training effect of the search result sorting model.

[0057] In this embodiment, after performing S104 to determine the training search targets of the plurality of second data pairs, performing S105 to train a second pre-trained language model to obtain a search result sorting model based on the queries in the plurality of second data pairs, the multi-target features for which each search result in the plurality of second data pairs corresponds to all the search targets, and the scores for which each search result in the plurality of second data pairs corresponds to the training search target.

[0058] The second pre-trained language model used when the present embodiment performs S105 may be a neural network model based on a transformer network, for example, an Ernie model.

[0059] In this embodiment, in S105, when a search result sorting model is obtained by training a second pre-trained language model based on queries in a plurality of second data pairs, multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and scores in which each search result in the plurality of second data pairs corresponds to the training search target, for each second data pair, the query in the second data pair and the search result with a high score corresponding to the training search target in the second data pair are configured as a positive example data pair, and the query in the second data pair and the search result with a low score corresponding to the training search target in the second data pair are configured as a negative example data pair, and the query in the positive example data pair is and multi-target features in which search results correspond to all search targets, and the query in negative example data pairs and multi-target features in which search results correspond to all search targets are input to a second pre-trained language model, respectively, to obtain positive example prediction scores output by the second pre-trained language model for positive example data pairs and negative example prediction scores output for negative example data pairs, and calculate a loss function value based on the positive example prediction scores and negative example prediction scores. In this embodiment, the loss function value can be calculated using a calculation method for a cross-entropy loss function and a cut-off loss function, and the parameters of the second pre-trained language model are adjusted based on the calculated loss function value until the second pre-trained language model converges, thereby obtaining a search result sorting model.

[0060] That is, in this embodiment, a positive example data pair and a negative example data pair corresponding to the second data pair are obtained based on the training search target of the second data pair, and then the obtained positive example data pair and negative example data pair are used to complete the training of the second pre-trained language model, thereby eliminating the need to annotate each search result for the second data pair, reducing the cost and difficulty of model training and improving model training efficiency.

[0061] For example, if the second data pair is data pair 2 (query1, para1, para2), the training search target of data pair 2 is a correlation target, the score corresponding to para1 with the correlation target is 0.7, and the score corresponding to para2 with the correlation target is 0.3, in this embodiment, when executing S105, (query1, para1) is set as a positive example data pair, and (query1, para2) is set as a negative example data pair.

[0062] The attention mechanism in the second pre-trained language model based on Transformer does not distinguish between features corresponding to different search goals when performing attention calculations, and the existing second pre-trained language model performs attention calculations on the features of search goal 1 and search goal 2, even if search goal 1 and search goal 2 are in completely unrelated dimensions.

[0063] However, different search goals actually correspond to different degrees of matching, e.g., the recency goal is used to describe the match between the time of the search result and the query, and the authoritative goal is used to describe the match between the publisher of the search result and the query, so there is no need to consider the authoritative goal when modeling recency.

[0064] In order to further improve the effectiveness of the pre-trained language model in learning multiple search targets, when performing S105, this embodiment can also further adjust the attention mechanism in the second pre-trained language model so that the second pre-trained language model only performs attention calculation between the query and different target features, attention calculation between queries, and attention calculation between the same target features.

[0065] That is, the second pre-trained language model of this embodiment can avoid attention calculation between target features corresponding to different search goals during training, thereby reducing mutual interference between target features of different search goals and improving the sorting accuracy of the search result sorting model obtained by training.

[0066] For example, if the input of the second pre-trained language model includes a query, a target feature 1, a target feature 2, and a target feature 3, the second pre-trained language model in this embodiment performs attention calculation only for the query and target feature 1, the query and target feature 2, the query and target feature 3, the query and the query, the target feature 1 and target feature 1, and the target feature 2 and target feature 2.

[0067] FIG. 2 is a schematic diagram of a second embodiment of the present disclosure. As shown in FIG. 2, this embodiment illustrates a structural diagram of a single-target sorting model obtained by training. That is, a query in a first data pair is spliced ​​with a single-target feature whose search result in the first data pair corresponds to the current search target. The splicing result is used as input for a pre-trained language model to obtain a predicted score for the first data pair. Here, [CLS] represents a start character and [SEP] represents a delimiter.

[0068] FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure. As shown in FIG. 3, this embodiment illustrates a flowchart for determining a training search target for a second data pair. In this embodiment, multiple search targets are divided into a main search target and multiple sub-search targets. Two search results in the second data pair are sequentially scored using single-search target sorting models corresponding to different search targets. If it is determined that the score difference between the two search results exceeds the difference threshold for the main search target, the main search target is selected as the training search target for the second data pair. Otherwise, the two search results in the second data pair are continuously scored using a single-search target sorting model corresponding to sub-search target 1. If it is determined that the score difference between the two search results exceeds the difference threshold for sub-search target 1, sub-search target 1 is selected as the training search target for the second data pair. In this manner, the training search target for the second data pair can be determined. In particular, if the score differences of the second data pair for all sub-search targets do not exceed the corresponding difference threshold, the main search target is selected as the training search target for the second data pair.

[0069] FIG. 4 is a schematic diagram of a fourth embodiment of the present disclosure. As shown in FIG. 4, this embodiment illustrates a structural diagram of a search result sorting model obtained by training. That is, a query in the second data pair is spliced ​​with multi-target features (main target feature of the main search target, sub-target 1 feature of sub-search target 1, and sub-target n feature of sub-search target n) corresponding to all search targets in the second data pair. The splicing result is used as input to a pre-trained language model to obtain a predicted score for the second data pair. Here, [CLS] represents a start character and [SEP] represents a delimiter.

[0070] 5 is a schematic diagram according to a fifth embodiment of the present disclosure. As shown in FIG. 5, the search result sorting method of this embodiment specifically includes the following steps:

[0071] In S501, a query to be processed and a plurality of corresponding search results are obtained.

[0072] In S502, for each search result, multi-goal features corresponding to all search goals of the search result are obtained.

[0073] In S503, for each search result, the query to be processed and the multi-goal features of the search result corresponding to all search goals are input into a search result sorting model to obtain a score output by the search result sorting model for the search result.

[0074] In S504, the search results are sorted based on the scores of the search results.

[0075] In this embodiment, the search result sorting method obtains multiple search results corresponding to a query, first obtains multi-target features corresponding to all search targets for each search result, then inputs the query and the multi-target features corresponding to all search targets for each search result into a search result sorting model to obtain scores output by the search result sorting model for each search result, and finally completes sorting of the search results based on the scores of the search results. In this embodiment, by obtaining multi-target features corresponding to all search targets for the search results, the scores used for sorting can be integrated across multiple search targets, simplifying the search result sorting steps and improving the accuracy of search result sorting.

[0076] In this embodiment, when executing S501 to obtain a query to be processed, the query to be processed may be a query input on the input side, or a query selected on the input side over a network. The search result corresponding to the query to be processed, obtained by executing S501 in this embodiment, may be an entire web page, an entire document, or an entire multimedia data, or may be one paragraph in the web page, document, or multimedia data.

[0077] In this embodiment, after executing S501 to obtain a query to be processed and corresponding search results, S502 is executed to obtain, for each search result, multi-target features corresponding to all search targets of the search result. Note that, in this embodiment, the multiple search targets may be preset relevance targets, timeliness targets, authoritative targets, etc.

[0078] In this embodiment, when S502 is executed to obtain, for each search result, a multi-target feature corresponding to all search targets of the search result, attribute information of the search result is obtained for each search result, and the attribute information of the search result is used as a multi-target feature corresponding to all search targets of the search result.

[0079] In this embodiment, when S502 is executed to treat the attribute information of the search result as a multi-target feature corresponding to all search targets of the search result, if it is determined that the attribute information of the search result is not text, the attribute information of the search result may be converted into text, and the text conversion result may be treated as a multi-target feature corresponding to all search targets of the search result.

[0080] In this embodiment, S502 is executed to obtain the multi-target features corresponding to all the search targets of each of the multiple search results, and then S503 is executed to input, for each search result, the query to be processed and the multi-target features corresponding to all the search targets of the search result into the search result sorting model, and obtain the score output by the search result sorting model for the search result.

[0081] In this embodiment, when S503 is executed to input the query to be processed and the multi-target features of the search results corresponding to all search targets into the search result sorting model, the splicing results between the two are input into the search result sorting model, so that the search result sorting model can score the search results and obtain a score to be output from the search result sorting model for the search result.

[0082] In this embodiment, after S503 is executed to obtain scores for a plurality of search results, S504 is executed to sort the plurality of search results according to their scores.

[0083] In this embodiment, when S504 is executed to sort the search results according to the scores of the search results, the search results can be sorted in descending order of scores.

[0084] 6 is a schematic diagram according to the sixth embodiment of the present disclosure. As shown in FIG. 6, the apparatus 600 for training a search result sorting model of this embodiment includes the following means:

[0085] The first acquisition unit 601 acquires a plurality of first data pairs, single-target features in which search results in the plurality of first data pairs respectively correspond to a plurality of search targets, and annotation scores in which the plurality of first data pairs respectively correspond to the plurality of search targets, and each first data pair includes a query and one search result corresponding thereto.

[0086] The first training unit 602 trains a first pre-trained language model for each search target based on the queries in the plurality of first data pairs, the single-target features of the search results in the plurality of first data pairs corresponding to the search target, and the annotation scores of the plurality of first data pairs corresponding to the search target, to obtain a single-search target sorting model corresponding to the search target.

[0087] The second acquisition unit 603 acquires a plurality of second data pairs and a multi-target feature in which each search result in the plurality of second data pairs corresponds to all search targets, and each second data pair includes a query and two corresponding search results.

[0088] For each second data pair, the processing unit 604 scores each search result in the second data pair using a plurality of single search target sorting models corresponding to the plurality of search targets, and determines a training search target for the second data pair based on the scores of each search result corresponding to the plurality of search targets.

[0089] The second training unit 605 trains a second pre-trained language model to obtain a search result sorting model based on queries in the plurality of second data pairs, multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and scores in which each search result in the plurality of second data pairs corresponds to the training search target.

[0090] The first data pair acquired by the first acquisition unit 601 includes a query and one corresponding search result, where the search result in the first data pair may be an entire web page, an entire document, or an entire multimedia data, or may be a paragraph in the web page, document, or multimedia data.

[0091] In this embodiment, the multiple search goals that are preset include multiple goals such as a relevance goal (representing the correlation between the search results and the query), an authority goal (representing the authority of the search results), and a timeliness goal (representing the timeliness of the search results).

[0092] After acquiring the first data pairs, the first acquisition unit 601 may first determine multiple search targets, and then acquire single-target features of the search results in the multiple first data pairs corresponding to the multiple search targets, respectively, and annotation scores of the multiple first data pairs corresponding to the multiple search targets, respectively.

[0093] When a plurality of first data pairs acquire annotation scores corresponding to a plurality of search targets, the first acquisition unit 601 can acquire the annotation scores corresponding to each search target of the plurality of first data pairs according to a preset annotation method.

[0094] When the search results in a plurality of first data pairs acquire single-target features corresponding to a plurality of search targets, the first acquisition unit 601 may adopt an optional realization method in which it acquires attribute information of the search results in the plurality of first data pairs, and for each search target, the attribute information corresponding to the search target among the attribute information is set as the single-target feature in which the search results in the plurality of first data pairs acquire single-target features corresponding to the search target.

[0095] In other words, different single-target features acquired by the first acquisition unit 601 correspond to different search targets, and the attribute information included in the single-target features whose search results correspond to the same search target may be one or more.

[0096] The first acquisition unit 601 may include converting the attribute information corresponding to the search goal from among the attribute information for each search goal into text and using the text conversion result as a single-target feature for which search results in the multiple first data pairs correspond to the search goal when it is determined that the attribute information corresponding to the search goal from among the attribute information for each search goal is not text.

[0097] That is, the first acquisition unit 601 can ensure that the acquired single-target feature is text and avoid the use of other complex forms of content, thereby reducing the difficulty of model training and improving model training efficiency.

[0098] The first acquisition unit 601 can acquire attribute information corresponding to the search target from the attribute information of the search results as single-target features corresponding to each search target based on a correspondence relationship between the search target and attribute information that has been set in advance, thereby improving the accuracy of the acquired single-target features.

[0099] In this embodiment, the first acquisition unit 601 acquires a plurality of first data pairs, single-target features of the search results in the plurality of first data pairs corresponding to each of the plurality of search targets, and annotation scores of the plurality of first data pairs corresponding to each of the plurality of search targets. Then, the first training unit 602 trains a first pre-trained language model for each search target based on the query in the plurality of first data pairs, the single-target features of the search results in the plurality of first data pairs corresponding to the search target, and the annotation scores of the plurality of first data pairs corresponding to the search target, to obtain a single-search target sorting model corresponding to the search target.

[0100] It should be understood that multiple single search target sorting models are obtained by the first training unit 602. Different single search target sorting models correspond to different search targets.

[0101] Specifically, when the first training unit 602 trains a first pre-trained language model for each search target based on queries in a plurality of first data pairs, single-target features in which search results in the plurality of first data pairs correspond to the search target, and annotation scores in which the plurality of first data pairs correspond to the search target, to obtain a single-search target sorting model corresponding to the search target, the first training unit 602 may adopt an optional implementation method of inputting, for each search target, the queries in the plurality of first data pairs and the single-target features in which search results in the plurality of first data pairs correspond to the search target into the first pre-trained language model to obtain a predicted score to be output for each first data pair, calculating a loss function value based on the predicted scores of the plurality of first data pairs and the annotation scores in which the plurality of first data pairs correspond to the search target, and adjusting parameters of the first pre-trained language model based on the calculated loss function value until the first pre-trained language model converges to obtain a single-search target sorting model corresponding to the search target.

[0102] That is, different single search target sorting models are obtained according to different search targets by the first training unit 602. The single search target sorting models corresponding to different search targets can output scoring results corresponding to different search targets based on the single target features of the input query and its search results corresponding to different search targets.

[0103] It should be understood that the first pre-trained language model trained by the first training unit 602 is a neural network model based on a transformer network, and may be an Ernie model.

[0104] In this embodiment, after the first training unit 602 obtains a single-search target sorting model corresponding to multiple search targets, the second acquisition unit 603 acquires multiple second data pairs and multi-target features in which each search result in the multiple second data pairs corresponds to all the search targets.

[0105] The second data pair acquired by the second acquisition unit 603 consists of a query and two corresponding search results, where the two search results in the second data pair may be an entire web page, an entire document, or an entire multimedia data, or may be one paragraph in the web page, document, or multimedia data.

[0106] When acquiring multiple second data pairs, the second acquisition unit 603 first acquires multiple queries, then acquires multiple search results corresponding to each query, and finally, for the same query, arbitrarily selects and combines two of the multiple search results corresponding to the query, thereby acquiring multiple second data pairs including the query and the two corresponding search results.

[0107] When the second acquisition unit 603 acquires multi-target features corresponding to all search targets for each search result in the plurality of second data pairs, the second acquisition unit 603 may adopt an optional realization method in which it acquires attribute information of each search result in the plurality of second data pairs and treats the attribute information of each search result as a multi-target feature corresponding to all search targets for each search result in the plurality of second data pairs.

[0108] That is, the multi-target features acquired by the second acquisition unit 603 are all attribute information of the search results, that is, the multi-target features include all single-target features corresponding to different search targets in the search results.

[0109] When the second acquisition unit 603 treats the attribute information of each search result in a plurality of second data pairs as a multi-target feature corresponding to all search targets for each search result, if it is determined that the attribute information of the search result is not text for each search result, the second acquisition unit 603 may further convert the attribute information of the search result into text and treat the text conversion result as a multi-target feature corresponding to all search targets for each search result.

[0110] That is, the second acquisition unit 603 can ensure that the acquired multi-target features are text, avoid the use of other complex forms of content, reduce the difficulty of model training, and improve the model training efficiency.

[0111] In this embodiment, the second acquisition unit 603 acquires multiple second data pairs and multi-target features in which each search result in the multiple second data pairs corresponds to all search targets. Then, the processing unit 604 scores each search result in the second data pair using multiple single-search target sorting models corresponding to multiple search targets for each second data pair, and determines the training search target for the second data pair based on the scores in which each search result corresponds to multiple search targets.

[0112] The processing unit 604 may adopt an optional implementation method in which, for each second data pair, it scores each search result in the second data pair using multiple single-search target sorting models corresponding to multiple search targets, and when determining a training search target for the second data pair based on the scores of each search result corresponding to the multiple search targets, it determines a scoring order of the multiple single-search target sorting models according to the priority of the search targets, and for each second data pair, it sequentially scores each search result in the second data pair using the single-search target sorting model according to the determined scoring order, and if it is determined that the score difference between the two search results exceeds the search target difference threshold corresponding to the current single-search target sorting model, it selects the search target corresponding to the current single-search target sorting model as the training search target for the second data pair.

[0113] That is, the processing unit 604 uses different single-search target sorting models to sequentially score the two search results in the second data pair according to a preset priority, and determines the training search target for the second data pair, thereby improving the accuracy of the obtained training search target. Furthermore, the determined training search target is used to train the search result sorting model in the next step, so there is no need to annotate the scores corresponding to the different search targets for the two search results in the second data pair, thereby reducing the difficulty and training cost of the model and improving the training efficiency of the model.

[0114] Here, the processing unit 604 presets priorities for different search targets. For example, if the preset priorities are "correlation target > timeliness target > authority target," the scoring order determined by the processing unit 604 is as follows: first, scoring is performed using a single search target sorting model corresponding to the correlation target, then scoring is performed using a single search target sorting model corresponding to the timeliness target, and finally scoring is performed using a single search target sorting model corresponding to the authority target.

[0115] When using the single-search target sorting model to score each search result in the second data pair, the processing unit 604 can input, for each search result in the second data pair, the query in the second data pair and the single-target feature for which the search result corresponds to the current search target into the single-search target sorting model corresponding to the current search target, and obtain a score output by the single-search target sorting model for the search result.

[0116] The processing unit 604 also presets difference thresholds corresponding to different search targets, and achieves the purpose of determining the training search target of the second data pair based on the result of comparing the score difference between the search results with the difference threshold.

[0117] When the processor 604 determines the training search target for the second data pair based on the score difference and the difference threshold, the processor 604 may determine that the score difference between the two search results does not exceed the difference threshold for the different search targets. In this case, the processor 604 may determine the highest priority search target as the training search target for the second data pair.

[0118] Furthermore, the processing unit 604 may adopt an optional implementation method in which, for each second data pair, it scores each search result in the second data pair using multiple single-search target sorting models corresponding to multiple search targets, and when determining the training search target for the second data pair based on the scores of each search result corresponding to the multiple search targets, it scores each search result in the second data pair using multiple single-search target sorting models corresponding to the multiple search targets for each second data pair, and the search target corresponding to the largest score difference between the two search results is the training search target for the second data pair.

[0119] In other words, the processing unit 604 further determines a training search target for each second data pair based on the scores of the search results output by all single search target sorting models for each second data pair, thereby ensuring that the two search results in the second data pair have the most significant difference in the determined training search target, thereby improving the training effect of the search result sorting model.

[0120] In this embodiment, after the processing unit 604 determines the training search targets of the multiple second data pairs, the second training unit 605 trains the second pre-trained language model to obtain a search result sorting model based on the queries in the multiple second data pairs, the multi-target features in which each search result in the multiple second data pairs corresponds to all the search targets, and the scores in which each search result in the multiple second data pairs corresponds to the training search target.

[0121] The second pre-trained language model used by the second training unit 605 may be a neural network model based on a Transformer network, for example the Ernie model.

[0122] When the second training unit 605 trains the second pre-trained language model to obtain a search result sorting model based on the queries in the plurality of second data pairs, the multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and the scores in which each search result in the plurality of second data pairs corresponds to the training search target, the second training unit 605 configures, for each second data pair, the query in the second data pair and the search result in the second data pair with a high score corresponding to the training search target as a positive example data pair, and the query in the second data pair and the search result in the second data pair with a low score corresponding to the training search target as a negative example data pair. and inputting multi-target features in which queries and search results in positive example data pairs correspond to all search targets and multi-target features in which queries and search results in negative example data pairs correspond to all search targets into a second pre-trained language model to obtain positive example prediction scores to be output for positive example data pairs and negative example prediction scores to be output for negative example data pairs from the second pre-trained language model, calculating loss function values ​​based on the positive example prediction scores and negative example prediction scores, and adjusting parameters of the second pre-trained language model based on the calculated loss function values ​​until the second pre-trained language model converges to obtain a search result sorting model.

[0123] That is, the second training unit 605 obtains positive example data pairs and negative example data pairs corresponding to the second data pairs based on the training search target of the second data pairs, and then uses the obtained positive example data pairs and negative example data pairs to complete the training of the second pre-trained language model, thereby eliminating the need to annotate each search result in the second data pairs, reducing the cost and difficulty of training the model and improving the efficiency of training the model.

[0124] The attention mechanism in the second pre-trained language model based on Transformer does not distinguish between features corresponding to different search goals when performing attention calculations, and the existing second pre-trained language model performs attention calculations on the features of search goal 1 and search goal 2, even if search goal 1 and search goal 2 are in completely unrelated dimensions.

[0125] However, different search goals actually correspond to different degrees of matching, e.g., the recency goal is used to describe the match between the time of the search result and the query, and the authoritative goal is used to describe the match between the publisher of the search result and the query, so there is no need to consider the authoritative goal when modeling recency.

[0126] In order to further improve the effectiveness of the pre-trained language model in learning multiple search targets, the search result sorting model training device 600 of this embodiment may further include an adjustment unit 606 that adjusts the attention mechanism in the second pre-trained language model so that the second pre-trained language model only performs attention calculation between the query and different target features, attention calculation between queries, and attention calculation between the same target features.

[0127] That is, the second pre-trained language model of this embodiment can avoid attention calculation between target features corresponding to different search goals during training, thereby reducing mutual interference between target features of different search goals and improving the sorting accuracy of the search result sorting model obtained by training.

[0128] 7 is a schematic diagram according to the seventh embodiment of the present disclosure. As shown in FIG. 7, a search result sorting device 700 of this embodiment includes the following units:

[0129] The third acquisition unit 701 acquires a query to be processed and a plurality of search results corresponding to the query.

[0130] The fourth acquiring unit 702 acquires, for each search result, a multi-goal feature corresponding to all search goals of the search result.

[0131] For each search result, the scoring unit 703 inputs the query to be processed and the multi-goal features of the search result corresponding to all search goals into a search result sorting model, and obtains a score to be output by the search result sorting model for the search result.

[0132] The sorting unit 704 sorts the search results based on the scores of the search results.

[0133] When acquiring a query to be processed, the third acquisition unit 701 may set a query input on the input side as the query to be processed, or may set a query selected on the input side over a network as the query to be processed. The search result corresponding to the query to be processed acquired by the third acquisition unit 701 may be an entire web page, an entire document, or an entire multimedia data, or may be one paragraph in the web page, document, or multimedia data.

[0134] In this embodiment, the third acquisition unit 701 acquires a query to be processed and multiple corresponding search results, and then the fourth acquisition unit 702 acquires, for each search result, a multi-target feature that corresponds to all search targets of the search result. Note that the multiple search targets in this embodiment may be preset relevance targets, timeliness targets, authoritative targets, etc.

[0135] The fourth acquisition unit 702 may adopt an optional realization method in which, for each search result, when the search result acquires a multi-target feature corresponding to all search targets, the fourth acquisition unit 702 acquires attribute information of the search result for each search result and sets the attribute information of the search result as a multi-target feature corresponding to all search targets.

[0136] The fourth acquisition unit 702 may further include, when the attribute information of the search result is determined to be not text, converting the attribute information of the search result into text and treating the text conversion result as a multi-target feature that corresponds to all search targets of the search result, if the attribute information of the search result is determined to be not text.

[0137] In this embodiment, the fourth acquisition unit 702 acquires multi-target features corresponding to all search targets for each of the multiple search results, and then the scoring unit 703 inputs the query to be processed and the multi-target features corresponding to all search targets for each search result into the search result sorting model, and obtains a score output by the search result sorting model for the search result.

[0138] When the scoring unit 703 inputs the query to be processed and the multi-target features of the search results corresponding to all search targets into the search result sorting model, it can input the splicing results between the two into the search result sorting model so that the search result sorting model scores the search results and obtains a score to be output from the search result sorting model for the search result.

[0139] In this embodiment, after the scoring unit 703 obtains scores for a plurality of search results, the sorting unit 704 sorts the plurality of search results according to the scores for the plurality of search results.

[0140] When sorting the search results according to the scores of the search results, the sorting unit 704 can sort the search results in descending order of scores.

[0141] In the technical solution disclosed herein, the acquisition, storage, application, etc. of personal information of the users involved are in accordance with the provisions of relevant laws and regulations and are not contrary to public order and morals.

[0142] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0143] 8 illustrates a block diagram of an electronic device for training a search result sorting model or a search result sorting method according to an embodiment of the present disclosure. The electronic device represents various forms of digital computers, such as laptops, desktop computers, workbenches, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as PDAs, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely examples and are not intended to limit the implementation of the present disclosure as described and / or claimed herein.

[0144] 8, device 800 includes a computing means 801 that can perform various appropriate operations and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or loaded from a storage means 808 into a random access memory (RAM) 803. The RAM 803 may store various programs and data necessary for the operation of device 800. The computing means 801, ROM 802, and RAM 803 are connected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0145] Several components of device 800 are connected to I / O interface 805, including input means 806, e.g., a keyboard, a mouse, etc., output means 807, e.g., various types of displays, speakers, etc., storage means 808, e.g., a magnetic disk, an optical disk, etc., and communication means 809, e.g., a network card, a modem, a wireless communication transceiver, etc. The communication means 809 enables device 800 to exchange information / data with other devices via computer networks, e.g., the Internet, and / or various telecommunication networks.

[0146] The computing means 801 may be various general-purpose and / or specialized processing components having processing and computing capabilities. Some examples of the computing means 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing means 801 executes various methods and processes described above, such as training a search result sorting model or a search result sorting method. For example, in some embodiments, training a search result sorting model or a search result sorting method may be implemented as a computer software program physically embodied in a machine-readable medium, such as the storage means 808.

[0147] In some embodiments, part or all of the computer program may be loaded and / or installed into device 800 via ROM 802 and / or communication means 809. When the computer program is loaded into RAM 803 and executed by computing means 801, it may perform one or more steps of said training of a search result sorting model or search result sorting method. Alternatively, in other embodiments, computing means 801 may be configured in any other suitable manner (e.g., via firmware) to train a search result sorting model or perform a search result sorting method.

[0148] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), field programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor. The programmable processor may be a special-purpose or general-purpose programmable processor that can receive data and instructions from, and transfer data and instructions to, a storage system, at least one input device, and at least one output device.

[0149] Program code for implementing the methods of the present disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the program code performs the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, as a standalone package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of this disclosure, a machine-readable medium is a tangible medium that can contain or store a program used by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of machine-readable storage media include one or more line-based electrical connections, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0151] To provide for user interaction, the systems and techniques described herein may be implemented on a computer that includes a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) for providing input by the user to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including sound input, speech input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and an internetwork.

[0153] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server is also called a cloud server, cloud computing server, or cloud host, and may be one of the host products of a cloud computing service system that solves the shortcomings of traditional physical hosts and VPS services (also known as "Virtual Private Servers" or "VPS"), such as difficulty in management and poor business scalability. The server may be a server in a distributed system or a server combined with blockchain.

[0154] It should be understood that steps can be rearranged, added, or deleted using the various types of flows shown above. For example, the steps described in this application can be performed in a parallel order, a sequential order, or can be performed in a different order, and are not limited thereto, as long as the desired results of the technical solution disclosed in this application can be achieved.

[0155] The above specific embodiments do not constitute limitations on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, partial combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application. (Other possible items) [Item 1] 1. A method for training a search result sorting model, comprising: Obtaining a plurality of first data pairs, each of which includes a query and one search result corresponding to the query, single-target features of the search results in the plurality of first data pairs corresponding to a plurality of search targets, and annotation scores of the plurality of first data pairs corresponding to the plurality of search targets; For each of the search targets included in the plurality of search targets, train a first pre-trained language model based on queries in the plurality of first data pairs, single-target features of search results in the plurality of first data pairs corresponding to the search target, and annotation scores of the plurality of first data pairs corresponding to the search target to obtain a single-search target sorting model corresponding to the search target; obtaining a plurality of second data pairs, each of which includes a query and two search results corresponding to the query, and a multi-target feature, each of which includes a search result corresponding to all search targets in the plurality of second data pairs; For each of the plurality of second data pairs, scoring each search result in the second data pair using a plurality of single-search-target sorting models corresponding to the plurality of search targets, and determining a training search target for the second data pair based on the scores of each search result corresponding to the plurality of search targets; training a second pre-trained language model to obtain a search result sorting model based on queries in the plurality of second data pairs, multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and scores in which each search result in the plurality of second data pairs corresponds to the training search target; How to train a search result sorting model. [Item 2] The acquisition of single-target features corresponding to the search results of the plurality of first data pairs, respectively, for the plurality of search targets includes: acquiring attribute information of search results in the plurality of first data pairs; For each search goal, attribute information corresponding to the search goal among the attribute information is set as a single-goal feature for which search results in the plurality of first data pairs correspond to the search goal. Item 1. A method for training a search result sorting model. [Item 3] For each search goal, the attribute information corresponding to the search goal among the attribute information is set as a single-goal feature in which search results in the plurality of first data pairs correspond to the search goal, For each search target, if it is determined that the attribute information corresponding to the search target among the attribute information is not text, converting the attribute information corresponding to the search target into text; and determining the text conversion result as a single-target feature corresponding to the search target in the plurality of first data pairs. Item 2. A method for training a search result sorting model. [Item 4] For each of the plurality of search targets, training the first pre-trained language model to obtain a single search target sorting model corresponding to the search target based on queries in the plurality of first data pairs, single target features of search results in the plurality of first data pairs corresponding to the search target, and annotation scores of the plurality of first data pairs corresponding to the search target, For each search goal, input a query in the plurality of first data pairs and a single-goal feature whose search result in the plurality of first data pairs corresponds to the search goal into a first pre-trained language model to obtain a predicted score output by the first pre-trained language model for each first data pair; Calculating a loss function value based on predicted scores of the first data pairs and annotation scores of the first data pairs corresponding to the search goal; and adjusting parameters of the first pre-trained language model based on the loss function value until the first pre-trained language model converges to obtain a single search target sorting model corresponding to the search target. 4. A method for training a search result sorting model according to any one of items 1 to 3. [Item 5] Obtaining multi-target features corresponding to all search targets for each search result in the plurality of second data pairs includes: acquiring attribute information of each search result in the plurality of second data pairs; and setting the attribute information of each search result as a multi-target feature corresponding to all search targets of each search result in the plurality of second data pairs. 4. A method for training a search result sorting model according to any one of items 1 to 3. [Item 6] The attribute information of each search result is a multi-target feature in which each search result in the plurality of second data pairs corresponds to all search targets, For each search result, if it is determined that the attribute information of the search result is not text, converting the attribute information into text; and converting the text conversion result into a multi-target feature whose search results correspond to all search targets. Item 5. A method for training a search result sorting model according to item 5. [Item 7] For each second data pair, scoring each search result in the second data pair using a plurality of single-search-target sorting models corresponding to the plurality of search targets, and determining a training search target for the second data pair based on scores of the search results corresponding to the plurality of search targets; determining a scoring order of the plurality of single search goal sorting models according to the priority of the search goals; for each second data pair, sequentially scoring each search result in that second data pair using the single search target sorting model according to the scoring order; When it is determined that the score difference between the two search results in the second data pair exceeds the search target difference threshold corresponding to the current single search target sorting model, setting the search target corresponding to the current single search target sorting model as the training search target of the second data pair. 4. A method for training a search result sorting model according to any one of items 1 to 3. [Item 8] training a second pre-trained language model to obtain a search result sorting model based on queries in the plurality of second data pairs, multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and scores in which each search result in the plurality of second data pairs corresponds to the training search target; For each second data pair, a query in the second data pair and a search result with a high score corresponding to the training search objective in the second data pair are configured as a positive example data pair, and a query in the second data pair and a search result with a low score corresponding to the training search objective in the second data pair are configured as a negative example data pair; inputting multi-target features in which the queries and search results in the positive example data pairs correspond to all search targets and multi-target features in which the queries and search results in the negative example data pairs correspond to all search targets into the second pre-trained language model, respectively, to obtain positive example prediction scores output for the positive example data pairs and negative example prediction scores output for the negative example data pairs, respectively; calculating a loss function value based on the positive example prediction score and the negative example prediction score; and adjusting parameters of the second pre-trained language model based on the loss function value until the second pre-trained language model converges to obtain the search result sorting model. 4. A method for training a search result sorting model according to any one of items 1 to 3. [Item 9] adjusting an attention mechanism in the second pre-trained language model so that the second pre-trained language model only performs attention calculations between the query and different target features, between queries, and between the same target features. 4. A method for training a search result sorting model according to any one of items 1 to 3. [Item 10] A search result sorting method, comprising: obtaining a query to be processed and a plurality of search results corresponding to the query; For each search result included in the plurality of search results, obtain a multi-goal feature corresponding to all search goals of the search result; For each of the search results, input the query to be processed and the multi-goal features corresponding to all search goals of the search result into a search result sorting model to obtain a score output by the search result sorting model for the search result; sorting the plurality of search results based on scores of the plurality of search results; The search result sorting model is trained by the search result sorting model training method described in any one of items 1 to 3. How search results are sorted. [Item 11] For each of the plurality of search results, obtaining a multi-goal feature corresponding to all search goals of the search result includes: For each search result, acquiring attribute information of the search result; The attribute information is a multi-target feature in which the search results correspond to all search targets. Item 10: Search result sorting method. [Item 12] A device for training a search result sorting model, comprising: a first acquisition unit that acquires a plurality of first data pairs, each of which includes a query and one search result corresponding to the query, single-target features of the search results in the plurality of first data pairs corresponding to a plurality of search targets, and annotation scores of the plurality of first data pairs corresponding to the plurality of search targets; a first training unit that, for each search goal included in the plurality of search goals, trains a first pre-trained language model based on queries in the plurality of first data pairs, single-target features of search results in the plurality of first data pairs corresponding to the search goal, and annotation scores of the plurality of first data pairs corresponding to the search goal, to obtain a single-search goal sorting model corresponding to the search goal; a second acquisition unit that acquires a plurality of second data pairs, each of which includes a query and two search results corresponding to the query, and a multi-target feature in which each search result in the plurality of second data pairs corresponds to all search targets; a processing unit that, for each of the plurality of second data pairs, scores each search result in the second data pair using a plurality of single-search-target sorting models corresponding to the plurality of search targets, and determines a training search target for the second data pair based on the scores of each search result corresponding to the plurality of search targets; a second training unit that trains a second pre-trained language model to obtain a search result sorting model based on queries in the plurality of second data pairs, multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and scores in which each search result in the plurality of second data pairs corresponds to the training search target; A training apparatus for a search result sorting model comprising: [Item 13] When the search results of the plurality of first data pairs acquire single-target features corresponding to a plurality of search targets, the first acquisition unit: acquiring attribute information of search results in the plurality of first data pairs; For each search goal, attribute information corresponding to the search goal among the attribute information is set as a single-goal feature for which search results in the plurality of first data pairs correspond to the search goal. Item 13. A training device for a search result sorting model according to item 12. [Item 14] When the first acquisition unit determines, for each search target, attribute information corresponding to the search target among the attribute information, as a single target feature corresponding to the search target in the search results of the plurality of first data pairs, For each search target, if it is determined that the attribute information corresponding to the search target among the attribute information is not text, convert the attribute information corresponding to the search target into text; The text conversion result is a single-target feature corresponding to the search target in the plurality of first data pairs. Item 14. A training device for a search result sorting model according to item 13. [Item 15] When the first training unit trains a first pre-trained language model for each of the plurality of search targets based on queries in the plurality of first data pairs, single-target features of search results in the plurality of first data pairs corresponding to the search target, and annotation scores of the plurality of first data pairs corresponding to the search target, to obtain a single-search target sorting model corresponding to the search target, For each search goal, input a query in the plurality of first data pairs and a single-goal feature whose search result in the plurality of first data pairs corresponds to the search goal into a first pre-trained language model to obtain a predicted score output by the first pre-trained language model for each first data pair; calculating a loss function value based on predicted scores of the plurality of first data pairs and annotation scores of the plurality of first data pairs corresponding to the search goal; and adjusting parameters of the first pre-trained language model based on the loss function value until the first pre-trained language model converges to obtain a single search target sorting model corresponding to the search target. 15. A device for training a search result sorting model according to any one of items 12 to 14. [Item 16] When each search result in the plurality of second data pairs acquires a multi-target feature corresponding to all search targets, the second acquisition unit: acquiring attribute information of each search result in the plurality of second data pairs; The attribute information of each search result is a multi-target feature in which each search result in the plurality of second data pairs corresponds to all search targets. 15. A device for training a search result sorting model according to any one of items 12 to 14. [Item 17] When the attribute information of each search result in the plurality of second data pairs is a multi-target feature corresponding to all search targets, the second acquisition unit: For each search result, if it is determined that the attribute information of the search result is not text, converting the attribute information into text; The text conversion result is a multi-target feature whose search results correspond to all search targets. Item 17. A training device for a search result sorting model according to item 16. [Item 18] The processing unit, for each second data pair, scores each search result in the second data pair using a plurality of single search target sorting models corresponding to the plurality of search targets, and determines a training search target for the second data pair based on the scores of each search result corresponding to the plurality of search targets: determining a scoring order of the plurality of single search goal sorting models according to the priority of the search goals; for each second data pair, sequentially scoring each search result in that second data pair using the single search target sorting model according to the scoring order; When it is determined that the score difference between the two search results in the second data pair exceeds the difference threshold of the search target corresponding to the current single search target sorting model, the search target corresponding to the current single search target sorting model is set as the training search target of the second data pair; 15. A device for training a search result sorting model according to any one of items 12 to 14. [Item 19] When the second training unit trains the second pre-trained language model to obtain a search result sorting model based on queries in the plurality of second data pairs, multi-target features in which each search result in the plurality of second data pairs corresponds to all search targets, and scores in which each search result in the plurality of second data pairs corresponds to the training search target, For each second data pair, a query in the second data pair and a search result with a high score corresponding to the training search target in the second data pair are configured as a positive example data pair, and a query in the second data pair and a search result with a low score corresponding to the training search target in the second data pair are configured as a negative example data pair; inputting multi-target features in which the query and search results in the positive example data pairs correspond to all search targets, and multi-target features in which the query and search results in the negative example data pairs correspond to all search targets, into the second pre-trained language model, to obtain positive example prediction scores output for the positive example data pairs and negative example prediction scores output for the negative example data pairs, calculating a loss function value based on the positive example prediction score and the negative example prediction score; and adjusting parameters of the second pre-trained language model based on the loss function value until the second pre-trained language model converges to obtain the search result sorting model. 15. A device for training a search result sorting model according to any one of items 12 to 14. [Item 20] an adjustment unit that adjusts an attention mechanism in the second pre-trained language model so that the second pre-trained language model only performs attention calculations between the query and different target features, between queries, and between the same target features. 15. A training device for a search result sorting model according to any one of items 12 to 14. [Item 21] A search result sorting device, comprising: a third acquisition unit that acquires a query to be processed and a plurality of search results corresponding to the query; a fourth acquisition unit for acquiring, for each search result, multi-target features corresponding to all search targets of the search result; a scoring unit that, for each search result, inputs the query to be processed and the multi-goal features corresponding to all search goals of the search result into a search result sorting model, and obtains a score output by the search result sorting model for the search result; a sorting unit that sorts the plurality of search results based on scores of the plurality of search results, The search result sorting model is trained by the search result sorting model training device according to any one of items 12 to 14. Search result sorting device. [Item 22] When the fourth acquisition unit acquires, for each search result, multi-target features corresponding to all search targets of the search result, For each search result, attribute information of the search result is obtained; The attribute information is a multi-target feature in which the search results correspond to all search targets. Item 22. The search result sorting device according to item 21. [Item 23] at least one processor; a memory communicatively coupled to the at least one processor; An electronic device, wherein the memory stores commands executable by the at least one processor, the commands, when executed by the at least one processor, causing the at least one processor to perform the method for training a search result sorting model described in any one of items 1 to 3. [Item 24] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method for training a search result sorting model described in any one of items 1 to 3. [Item 25] A computer program that, when executed by a processor, implements the method for training a search result sorting model according to any one of items 1 to 3.

Claims

1. A method for training a search result sorting model, The method involves obtaining multiple first data pairs, each of which contains a query and one search result corresponding to the query. Using a plurality of pre-configured search objectives, including a correlation objective representing the correlation between the search results and the query, an authority objective representing the authority of the search results, and an expiration objective representing the timeliness of the search results, to obtain single-objective features in the search results of the plurality of first data pairs such that the title and / or summary of the search results are attribute information corresponding to the correlation objective, the public site, publisher, and public address of the search results are attribute information corresponding to the authority objective, and the publication time of the search results is attribute information corresponding to the timeliness objective, For each of the aforementioned plurality of first data pairs, the score for each of the aforementioned plurality of search targets is obtained through user input, For each of the search targets which is at least one of the correlation target, authoritative target, and temporality target included in the plurality of search targets, a first pre-trained language model, which is a neural network model based on a transformation network, is trained to take queries in the plurality of first data pairs and single target features corresponding to the search target as inputs, and outputs scores based on user input corresponding to the search target. A first loss function value is calculated by a method for calculating a cross-entropy loss function based on a first predicted score output by inputting each first data pair different from the plurality of first data pairs, and the scores based on user input for each first data pair. The first pre-trained language model is trained by adjusting the parameters of the first pre-trained language model until the first pre-trained language model converges, thereby obtaining the trained first pre-trained language model as a single search target sort model that takes the plurality of first data pairs as inputs and outputs the first predicted score corresponding to the search target. The process involves obtaining multiple second data pairs, each of which includes a query and two search results selected from a plurality of search results corresponding to the query. To obtain multi-target features including the multiple search targets from the search results in the multiple second data pairs, For each of the plurality of second data pairs, using a plurality of single search target sort models corresponding to the plurality of search targets, output the first predicted score corresponding to the search target for each of the two search results included in the second data pair, calculate the score difference between the first predicted scores output for each of the plurality of search targets, and determine the search target corresponding to the largest of the calculated score differences as the training search target for the second data pair. For each second data pair, the query in the second data pair and the search result with a high first predicted score corresponding to the training search objective in the second data pair are configured as a positive example data pair, and the query in the second data pair and the search result with a low first predicted score corresponding to the training search objective in the second data pair are configured as a negative example data pair. By inputting the queries in the positive example data pair and the multi-target features in the search results of the positive example data pair that correspond to the multiple search targets into a second pre-trained language model, which is a neural network model based on a transformation network, the positive example prediction score output by the second pre-trained language model for the positive example data pair is obtained; and by inputting the queries in the negative example data pair and the multi-target features in the search results of the negative example data pair that correspond to the multiple search targets into the second pre-trained language model, the negative example prediction score output by the second pre-trained language model for the negative example data pair is obtained. Based on the positive example prediction points corresponding to the positive example data pair and the negative example prediction points corresponding to the negative example data pair, the second loss function value is calculated using the cross-entropy loss function calculation method. The method includes training the second pre-trained language model by adjusting its parameters based on the second loss function value until the second pre-trained language model converges, thereby obtaining the trained second pre-trained language model as a search result sorting model. Training methods for search result sorting models.

2. The method for training a search result sorting model according to claim 1, wherein the first pre-trained language model and the second pre-trained language model are Ernie models.

3. A method for sorting search results, To obtain the query to be processed and multiple search results corresponding to the said query, For each of the search results included in the aforementioned multiple search results, a multi-target feature corresponding to the aforementioned multiple search objectives is obtained, For each of the aforementioned search results, the query to be processed and the multi-target feature for which the search result corresponds to the multiple search objectives are input to the search result sorting model, and a second predicted score is obtained which the search result sorting model outputs for that search result. This includes sorting the plurality of search results based on the second predicted score of the plurality of search results, The search result sorting model is trained by the method for training a search result sorting model described in claim 1 or 2. How to sort search results.

4. A training device for a search result sorting model, Multiple first data pairs, each first data pair included in the multiple first data pairs includes a query and one search result corresponding to the query, and multiple first data pairs are obtained. Using a plurality of pre-configured search objectives, which include a correlation objective representing the correlation between the search results and the query, an authority objective representing the authority of the search results, and an expiration objective representing the timeliness of the search results, a single objective feature is obtained in the search results of the plurality of first data pairs such that the title and / or summary of the search results are attribute information corresponding to the correlation objective, the public site, publisher, and public address of the search results are attribute information corresponding to the authority objective, and the publication time of the search results is attribute information corresponding to the timeliness objective. A first acquisition unit acquires, for each of the plurality of first data pairs, the score for each of the plurality of search targets via user input. A first training unit trains a first pre-trained language model, which is a neural network model based on a transformation network, that is trained to take queries in the plurality of first data pairs and single target features corresponding to the search target as inputs for each of the plurality of search targets, and to output scores based on user input corresponding to the search target. This first pre-trained language model is trained to take queries in the plurality of first data pairs and single target features corresponding to the search target as inputs for each of the plurality of first data pairs, and to output scores based on user input corresponding to the search target. The first training unit trains the first pre-trained language model by adjusting the parameters of the first pre-trained language model until the first pre-trained language model converges, thereby obtaining the trained first pre-trained language model as a single search target sort model that takes the plurality of first data pairs as inputs and outputs the first predicted score corresponding to the search target. A second acquisition unit acquires multiple second data pairs, each of which includes a query and two search results selected from a plurality of search results corresponding to the query, and acquires multi-target features including the plurality of search targets for the search results in the plurality of second data pairs. A processing unit that, for each of the plurality of second data pairs, outputs the first predicted score corresponding to the search objective for each of the two search results included in the second data pair using a plurality of single search objective sort models corresponding to the plurality of search objectives, calculates the score difference between the first predicted scores output for each of the two search results for each of the plurality of search objectives, and determines the search objective corresponding to the largest of the calculated score differences as the training search objective for the second data pair. For each second data pair, the query in the second data pair and the search result with a high first predicted score corresponding to the training search objective in the second data pair are configured as a positive example data pair, and the query in the second data pair and the search result with a low first predicted score corresponding to the training search objective in the second data pair are configured as a negative example data pair. By inputting the queries in the positive example data pair and the multi-target features in the search results of the positive example data pair that correspond to the multiple search targets into a second pre-trained language model, which is a neural network model based on a transformation network, the second pre-trained language model outputs positive example prediction scores for the positive example data pair. By inputting the queries in the negative example data pair and the multi-target features in the search results of the negative example data pair that correspond to the multiple search targets into the second pre-trained language model, the second pre-trained language model outputs negative example prediction scores for the negative example data pair. A second training unit calculates a second loss function value using a cross-entropy loss function calculation method based on the positive example prediction score corresponding to the positive example data pair and the negative example prediction score corresponding to the negative example data pair, and trains the second pre-trained language model by adjusting the parameters of the second pre-trained language model based on the second loss function value until the second pre-trained language model converges, thereby obtaining the trained second pre-trained language model as a search result sorting model. A training device for a search result sorting model.

5. A search result sorting device, A third acquisition unit that acquires a query to be processed and multiple search results corresponding to the query, For each search result, a fourth acquisition unit acquires multi-target features corresponding to the multiple search targets, For each search result, a scoring unit inputs the query to be processed and the multi-target features corresponding to the multiple search targets into the search result sorting model and obtains a second predicted score output by the search result sorting model for that search result. The system includes a sorting unit that sorts the plurality of search results based on the second predicted score of the plurality of search results, The search result sorting model is trained by the search result sorting model training device described in claim 4. Search result sorting device.

6. At least one processor, The system comprises at least one processor and a memory that is communicably connected to it, An electronic device wherein a command executable by the at least one processor is stored in the memory, and when the command is executed by the at least one processor, the at least one processor is caused to execute the training method for the search result sorting model according to claim 1 or 2.

7. A non-temporary computer-readable storage medium storing computer commands for causing a computer to perform the method for training the search result sorting model described in Claim 1 or 2.

8. A computer program that, when executed by a processor, causes the processor to execute the method for training the search result sorting model described in Claim 1 or 2.

Images