A method and apparatus for generating a training set

Abstract

The specification discloses a method and device for generating a training set. According to actual requirements, a reference training sample is selected first, and then a reference feature representation corresponding to the reference training sample is determined, so that a training sample matched with the reference training sample is screened from each candidate sample according to the reference feature representation, and a training set for training a ranking model is generated. This greatly improves the construction efficiency of the training set, thereby reducing the cost of training the ranking model, improving the efficiency of screening the training sample, and improving the accuracy of the screened training sample.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for generating training sets. Background Technology

[0002] Search technology, as an important traffic portal and content distribution channel in the Internet field, plays an extremely important role in various websites and applications. Among them, the ranking model used to recommend search results related to the user's search terms is a core part of search technology.

[0003] In existing technologies, users' historical search data is usually acquired in batches, and then specific difficult cases are manually selected from the acquired historical search data as training samples to train the ranking model. However, this often leads to an increase in the cost of training the ranking model, reduces the efficiency of selecting training samples, and the accuracy of the selected training samples is also low, thus affecting the training of the ranking model.

[0004] Therefore, how to improve the efficiency of selecting training samples for ranking models, increase the accuracy of training sample selection, and reduce the cost of training ranking models is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a method and apparatus for generating training sets to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a method for generating a training set, including:

[0008] Obtain benchmark training samples;

[0009] Based on the search terms and search results contained in the benchmark training samples, the feature representation corresponding to the benchmark training samples is determined as the benchmark feature representation. The benchmark feature representation is used to characterize the word features of the search terms contained in the benchmark training samples, as well as the matching situation of each word sequence formed by the search terms and at least part of the text in the search results.

[0010] Based on the feature representations corresponding to each candidate sample, candidate samples that match the benchmark feature representations are selected from each candidate sample and used as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms.

[0011] The target samples are labeled, and a training set for training the ranking model is generated based on the labeled target samples.

[0012] Optionally, from each candidate sample, candidate samples that match the benchmark feature representation are selected, specifically including:

[0013] The weight of each candidate sample is determined based on the search time corresponding to each candidate sample.

[0014] Based on the feature representation and weight of each candidate sample, candidate samples that match the benchmark feature representation are selected from the candidate samples and used as target samples.

[0015] Optionally, the weight of each candidate sample is determined based on the search time corresponding to each candidate sample, specifically including:

[0016] The weight of each candidate sample is determined based on the chronological order of the search times corresponding to each candidate sample. Specifically, for each candidate sample, the closer the search time corresponding to the candidate sample is to the current time, the greater the weight of the candidate sample.

[0017] Optionally, the weight of each candidate sample is determined based on the search time corresponding to each candidate sample, specifically including:

[0018] For each candidate sample, determine whether the search time corresponding to the candidate sample belongs to the specified time.

[0019] If so, the preset first weight is used as the weight corresponding to the candidate sample; otherwise, the preset second weight is used as the weight corresponding to the candidate sample, and the first weight is greater than the second weight.

[0020] Optionally, based on the labeled target samples, a training set for model training is determined, specifically including:

[0021] Based on the labeled target samples, determine the test set and the sub-training set;

[0022] The ranking model is trained based on the sub-training set to obtain the trained ranking model;

[0023] The training samples contained in the test set are respectively input into the ranking model that has not been trained on the sub-training set and the ranking model that has been trained, so as to determine the accuracy of the ranking model that has not been trained on the sub-training set in determining the matching degree between the search terms contained in the test set and the historical search results, which is taken as the accuracy of the ranking model that has not been trained on the sub-training set, and the accuracy of the ranking model that has been trained in determining the matching degree between the search terms contained in the test set and the historical search results, which is taken as the accuracy of the ranking model that has been trained.

[0024] If it is determined that the accuracy of the trained ranking model is greater than the accuracy of the ranking model that was not trained on the sub-training set, the test set and the sub-training set are added to the preset full training set to obtain a training set for training the ranking model.

[0025] Optionally, the method further includes:

[0026] For each training sample in the training set, the training sample is input into the ranking model to be trained, and the matching degree between the historical search results and the search terms in the training sample output by the ranking model is obtained.

[0027] The ranking model is trained with the goal of minimizing the deviation between the matching degree and the label corresponding to the training sample.

[0028] Optionally, the benchmark training samples include at least one of the following: training samples in which the matching degree between the search results predicted by the ranking model and the search terms is inconsistent with the actual execution result of the user, and training samples in which the confidence degree of the matching degree between the search results predicted by the ranking model and the search terms is lower than a preset threshold.

[0029] This specification provides a training set generation apparatus, comprising:

[0030] The acquisition module is used to acquire benchmark training samples;

[0031] The determination module is used to determine the feature representation corresponding to the benchmark training sample based on the search terms contained in the benchmark training sample and the search results, as the benchmark feature representation. The benchmark feature representation is used to characterize the word features of the search terms contained in the benchmark training sample and the matching situation of each word sequence formed by the search terms and at least part of the text in the search results.

[0032] The filtering module is used to filter out candidate samples that match the baseline feature representation from each candidate sample based on the determined feature representation of each candidate sample, and use them as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms.

[0033] The generation module is used to label the target samples and generate a training set for training the ranking model based on the labeled target samples.

[0034] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating the training set.

[0035] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for generating a training set.

[0036] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0037] The training set generation method provided in this specification first obtains benchmark training samples. Based on the search terms and search results contained in the benchmark training samples, the feature representations corresponding to the benchmark training samples are determined as benchmark feature representations. The benchmark feature representations are used to characterize the word features of the search terms contained in the benchmark training samples, as well as the matching of each word sequence formed by the search terms and at least part of the text in the search results. Then, based on the feature representations corresponding to each candidate sample, candidate samples that match the benchmark feature representations are selected from each candidate sample as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms. Then, the target samples are labeled, and a training set for training the ranking model is generated based on the labeled target samples.

[0038] As can be seen from the above method, a benchmark training sample can be selected first according to actual needs, and then the benchmark feature representation corresponding to the benchmark training sample can be determined. Based on the benchmark feature representation, training samples that match the benchmark training sample can be selected from each candidate sample to generate a training set for training the ranking model. This will greatly improve the efficiency of training set construction, thereby reducing the cost of training the ranking model, improving the efficiency of selecting training samples, and improving the accuracy of the selected training samples. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart illustrating a method for generating a training set as provided in this specification.

[0041] Figure 2 This is an iterative diagram of the sorting model provided in this specification;

[0042] Figure 3 A schematic diagram of a training set generation device provided in this specification;

[0043] Figure 4 This specification provides a corresponding Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0045] In the existing technology, the following two methods are usually used to select difficult examples for training ranking models. The difficult examples referred to here are training samples that are not easy for ranking models to identify, such as skiing and skating. The selected difficult examples can usually be used to train the ranking model in a targeted manner, so as to significantly improve the training effect of the ranking model. The two methods mentioned above are explained in detail below.

[0046] The first method is to use rules to select a large amount of data from the database as training samples to train the ranking model. The rules can be random sampling or data selected within a certain period of time. This method selects a large number of training samples, and the proportion of hard examples in the selected training samples is low. Most of the training samples are training samples that the ranking model can already identify well, so it does not help much in training the ranking model.

[0047] The second method uses search results that are similar to the literal meaning of the search term but were not clicked by the user as hard examples to train the ranking model. However, this method only considers hard examples where the search results are literally similar to the search term. In practical applications, there are some hard examples where the search results are not similar to the search term. For example, the search term or search result may contain typos, the search term and the search result may contain homophones, or the search term and the search result may contain words that are hyponyms or hypernyms. This method is less effective at filtering these hard examples.

[0048] As can be seen from the above, neither of the two existing methods for selecting difficult examples for training a ranking model can accurately and comprehensively select difficult examples for training a ranking model. Therefore, this specification provides a method for generating a training set, which selects target samples from each candidate sample based on the benchmark feature representation, and generates a sample set for training a ranking model based on each target sample.

[0049] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0050] Figure 1The flowchart of a training set generation method provided in this specification is shown, including the following steps:

[0051] S101: Obtain benchmark training samples.

[0052] In this specification, the business platform may obtain abnormal training samples or a certain type of training samples determined by the R&D personnel based on actual needs as benchmark training samples. Abnormal training samples may include, for example: the matching degree between the search results predicted by the ranking model and the search terms, training samples that are inconsistent with the actual execution results of the user, training samples where the confidence of the matching degree between the search results predicted by the ranking model and the search terms is lower than a preset threshold, training samples that are easily confused by manual screening, and training samples where the matching between the search results predicted by the ranking model and the search terms is inconsistent with the matching between the manually labeled results, etc.

[0053] The training samples identified by the R&D personnel mentioned above can be determined based on actual needs. For example, the benchmark training samples can be identified by the R&D personnel from training samples with short search terms. Or, the benchmark training samples can be identified by the R&D personnel from training samples with food and beverage search terms based on actual needs.

[0054] Training samples where the matching degree between the search results predicted by the ranking model and the search terms is inconsistent with the actual results of user actions; that is, historical search results recommended to the user by the ranking model but not clicked by the user.

[0055] Training samples where the confidence level of the matching degree between the search results predicted by the ranking model and the search terms is lower than a preset threshold, such as training samples where the matching degree between the search results predicted by the ranking model and the search terms is between 0.4 and 0.6. Here, since a matching degree of 1 between the search results predicted by the ranking model and the search terms is considered a perfect match, and a matching degree of 0 is considered a complete mismatch, the closer the matching degree between the search results predicted by the ranking model and the search terms is to 0 or 1, the higher the confidence level of the matching degree between the search results predicted by the ranking model and the search terms. Conversely, the closer the matching degree between the search results predicted by the ranking model and the search terms is to 0.5, the lower the confidence level of the matching degree between the search results predicted by the ranking model and the search terms.

[0056] In this specification, the execution entity used to implement the training set generation method can refer to a specified device set up on the business platform, such as a server, or a specified device such as a desktop computer or a laptop computer. For ease of description, the following description will only use the server as the execution entity to illustrate the training set generation method provided in this specification.

[0057] S102: Based on the search terms and search results contained in the benchmark training samples, determine the feature representation corresponding to the benchmark training samples as the benchmark feature representation. The benchmark feature representation is used to characterize the word features of the search terms contained in the benchmark training samples and the matching situation of each word sequence formed by the search terms and at least part of the text in the search results.

[0058] The server can determine the feature representation corresponding to the benchmark training sample based on the word features of the search terms contained in the obtained benchmark training sample and the matching of the search terms with the word sequences formed by at least part of the text in the search results.

[0059] The word features of the search term can include, for example, whether it is a multi-entity term, whether it is a brand term, whether the suffix of the search term is a common word, whether the search term is a short word, and whether the search term is purely in English.

[0060] In the above context, whether it is a multi-entity search term refers to the presence of multiple entities, such as: "the same dress worn by a certain celebrity," which involves the entity of "the celebrity" and the entity of "dress."

[0061] Does the search term suffix indicate whether it is a common word? For example, in the phrase "XXX hot pot restaurant", "hot pot restaurant" is a common word suffix.

[0062] In the above context, the matching of the search term with the sequence of words formed by at least a portion of the text in the search results can be, for example, non-match, partial match, reversed match, subsequence match, matching the store name suffix, split match, etc.

[0063] Among them, partial matching can be, for example: the search term is AB and the search result is AB noodle shop; inverted matching can be, for example: the search term is AB and the search result is BA; subsequence matching can be, for example: the search term is AB noodle shop and the search result is A noodle shop; matching the suffix of the store name can be, for example: the search term is AB hot pot and the search result is ABC hot pot; and split matching can be, for example: the search term is orange cake and the search results are orange tea and cream cake.

[0064] To facilitate understanding of the method for determining the feature representation corresponding to the benchmark training sample, this specification provides an embodiment. Assuming the search term for a benchmark training sample is AB hotpot and the search result is CD hotpot, the word features of the search term can be determined as follows: not a multi-entity word, not a brand word, the suffix is ​​a common word, it is a short word, and it is not purely English. The matching situation of the word sequence formed by the search term and the search result is: it matches the store name suffix. Then the benchmark feature representation corresponding to the benchmark training sample can be determined as 00110000010, that is, it meets the above feature conditions (for example, whether it is a multi-entity word; if it is, it is considered to meet the above feature conditions). The corresponding dimension in the benchmark feature representation is represented as 1. If it does not meet the above feature conditions, the corresponding dimension in the benchmark feature representation is represented as 0.

[0065] In addition, the server can also respond to the candidate sample screening instructions sent by the R&D personnel and obtain the feature representation input by the R&D personnel as the baseline feature representation. This can be understood as the R&D personnel being able to make the server directly select each target sample from each candidate sample based on the baseline feature representation input by the R&D personnel according to actual needs.

[0066] S103: Based on the feature representations corresponding to each candidate sample, select candidate samples that match the benchmark feature representation from each candidate sample and use them as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms.

[0067] In this specification, the server can determine each candidate sample based on historical search terms and the corresponding historical search results, and determine the feature representation corresponding to each candidate sample. Then, it can select candidate samples that match the benchmark feature representation from each candidate sample as target samples.

[0068] Specifically, considering the coverage of each candidate sample determined by the server, and the timeliness of each historical search term and its corresponding historical search results (since the popularity of each search term varies at different times, each search term has a certain timeliness; therefore, in order for the trained ranking model to achieve better results in practical applications, the timeliness of search terms needs to be considered here), the server can obtain each recent historical search term and the top few historical search results corresponding to each historical search term (here, "top few" refers to the top few historical search results recommended by the ranking model, which may include historical search results that the user has not clicked). For example, it can obtain each historical search term within one month and the top 20 historical search results corresponding to each historical search term. The top 20 refers to the top 20 search results recommended by the ranking model to the user for a search term.

[0069] Furthermore, the server can determine each candidate sample based on the obtained historical search terms and the historical search results corresponding to the historical search terms, and determine the feature representation corresponding to each candidate sample using the method described above. Each candidate sample contains a search term and a search result corresponding to that search term.

[0070] It should be noted that the method for determining candidate samples can be performed after the method for determining the baseline feature representation described above, or before the method for determining the baseline feature representation described above, or both methods can be performed simultaneously.

[0071] In the above content, the server selects candidate samples that match the benchmark feature representation from each candidate sample and uses them as target samples. One method is to determine the candidate samples that match the benchmark feature representation based on the similarity between the feature representation of each candidate sample and the benchmark feature representation.

[0072] In addition, in order to improve the timeliness of the candidate samples obtained by the server, the method for the server to select candidate samples that match the benchmark feature representation from each candidate sample as target samples can be as follows: determine the weight of each candidate sample according to the order of the search time corresponding to each candidate sample, and then select candidate samples that match the benchmark feature representation from each candidate sample as target samples according to the feature representation of each candidate sample and the weight of each candidate sample.

[0073] In practical applications, considering that users of the business platform search more frequently during holidays than on weekdays, in order to optimize the user experience of performing corresponding business through the business platform, the above-mentioned method of the server selecting candidate samples that match the benchmark feature representation from each candidate sample as target samples can also be as follows: for each candidate sample, determine whether the search time corresponding to the candidate sample belongs to a preset specified time period (the specified time period can be, for example, national statutory holidays, Saturdays, Sundays, etc.). If so, the preset first weight is used as the weight corresponding to the candidate sample; otherwise, the preset second weight is used as the weight corresponding to the candidate sample. Then, based on the feature representation corresponding to each candidate sample and the weight corresponding to each candidate sample, the candidate samples that match the benchmark feature representation are selected from each candidate sample as target samples, wherein the first weight is greater than the second weight.

[0074] It should be noted that the two methods described above for selecting candidate samples that match the baseline feature representation from each candidate sample can be used individually or both methods can be used together.

[0075] S104: Label the target samples and generate a training set for training the ranking model based on the labeled target samples.

[0076] After the server selects each target sample, it can label the target samples and generate a training set for training the ranking model based on the labeled target samples.

[0077] Specifically, the server can display the selected target samples to the R&D personnel so that the R&D personnel can annotate each target sample and return the results. Then, the server can determine the test set and the sub-training set based on the returned annotated target samples. The method for determining the test set and the sub-training set can be to randomly divide each target sample according to a preset ratio to obtain the test set and the sub-training set.

[0078] Furthermore, the server can train the ranking model based on the sub-training set to obtain the trained ranking model. Then, the training samples contained in the test set are input into the ranking model that has not been trained on the sub-training set and the trained ranking model, respectively. This determines the accuracy of the ranking model that has not been trained on the sub-training set in determining the matching degree between the search terms contained in the test set and the historical search results, which is taken as the accuracy of the ranking model that has not been trained on the sub-training set. The accuracy of the ranking model that has been trained on the test set in determining the matching degree between the search terms contained in the test set and the historical search results is taken as the accuracy of the ranking model. The accuracy mentioned here can be determined based on the proportion of the training samples in the test set that match the results of the ranking model (the output results refer to the matching results between the search terms determined by the ranking model) and the labeled training samples.

[0079] If it is determined that the accuracy of the trained ranking model is greater than the accuracy of the ranking model trained without a sub-training set, the test set and the sub-training set are added to the preset full training set to obtain the training set used to train the ranking model.

[0080] Furthermore, in order to more comprehensively evaluate the effect of each target sample on the training improvement of the ranking model, the server can also test the ranking model that has not been trained on the sub-training set and the trained ranking model based on the preset full training set, and determine whether the accuracy of the trained ranking model is also greater than the accuracy of the ranking model that has not been trained on the sub-training set. If so, the test set and the sub-training set are added to the preset full training set to obtain the training set used to train the ranking model.

[0081] In this specification, the server can input each training sample in the training set into the ranking model to be trained, obtain the matching degree between the historical search results in the training sample and the search terms output by the ranking model, and then train the ranking model with the optimization objective of minimizing the deviation between the matching degree and the label corresponding to the training sample.

[0082] In practical applications, the server can also train the ranking model through multiple rounds of iteration, such as... Figure 2 As shown.

[0083] Figure 2 This is an iterative diagram of the sorting model provided in this specification.

[0084] Combination Figure 2 As can be seen, for each optimization iteration, the server can use the abnormal training samples encountered by the ranking model in the previous optimization iteration during testing or in actual application as the benchmark training samples for the current optimization iteration. Then, the ranking model can be trained using the benchmark training samples for the current optimization iteration through the above method until the accuracy of the ranking model in determining the matching degree between the search terms contained in the above abnormal training samples and the historical search results is greater than a preset threshold, or until the number of iterations reaches a preset number of iterations.

[0085] In practical applications, even after multiple rounds of optimization iterations, there may still be search results recommended to users by the ranking model that the users have not clicked. In this case, the server can send abnormal training samples containing these search results to the developers so that they can optimize and adjust the ranking model based on these abnormal training samples, thereby improving the accuracy of the search results recommended to users by the ranking model. Optimization and adjustment of the ranking model can include, for example, optimizing and adjusting the structure of the model or adjusting various parameters of the model.

[0086] As can be seen from the above method, a benchmark training sample can be selected first according to actual needs, and then the benchmark feature representation corresponding to the benchmark training sample can be determined. Based on the benchmark feature representation, training samples that match the benchmark training sample can be searched from a large number of candidate samples to generate a training set for training the ranking model. This will greatly improve the efficiency of training set construction, thereby reducing the cost of ranking model training, improving the efficiency of selecting training samples, and effectively improving the accuracy of the selected training samples, thus further ensuring the effect of model training.

[0087] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0088] The above describes one or more embodiments of the training set generation method provided in this specification. Based on the same idea, this specification also provides a corresponding training set generation apparatus, such as... Figure 3 As shown.

[0089] Figure 3 A schematic diagram of a training set generation device provided in this specification includes:

[0090] Module 301 is used to acquire benchmark training samples;

[0091] The determining module 302 is used to determine the feature representation corresponding to the benchmark training sample based on the search terms contained in the benchmark training sample and the search results, as the benchmark feature representation. The benchmark feature representation is used to characterize the word features of the search terms contained in the benchmark training sample and the matching situation of each word sequence formed by the search terms and at least part of the text in the search results.

[0092] The filtering module 303 is used to filter out candidate samples that match the benchmark feature representation from each candidate sample based on the determined feature representation of each candidate sample, and use them as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms.

[0093] The generation module 304 is used to label the target samples and generate a training set for training the ranking model based on the labeled target samples.

[0094] Optionally, the filtering module 303 is specifically used to: determine the weight of each candidate sample based on the search time corresponding to each candidate sample; and select candidate samples that match the benchmark feature representation from the candidate samples as target samples based on the feature representation of each candidate sample and the weight of each candidate sample.

[0095] Optionally, the filtering module 303 is specifically used to determine the weight of each candidate sample according to the order of the search times corresponding to each candidate sample, wherein, for each candidate sample, the closer the search time corresponding to the candidate sample is to the current time, the greater the weight of the candidate sample.

[0096] Optionally, the filtering module 303 is specifically used to determine, for each candidate sample, whether the search time corresponding to the candidate sample belongs to a specified time; if so, to use a preset first weight as the weight corresponding to the candidate sample; otherwise, to use a preset second weight as the weight corresponding to the candidate sample, wherein the first weight is greater than the second weight.

[0097] Optionally, the generation module 304 is specifically configured to: determine a test set and a sub-training set based on the labeled target samples; train the ranking model based on the sub-training set to obtain a trained ranking model; input the training samples contained in the test set into the ranking model that has not been trained on the sub-training set and the trained ranking model, respectively, to determine the accuracy of the ranking model that has not been trained on the sub-training set in determining the matching degree between the search terms contained in the test set and the historical search results, as the accuracy of the ranking model that has not been trained on the sub-training set, and the accuracy of the ranking model that has been trained on determining the matching degree between the search terms contained in the test set and the historical search results, as the accuracy of the ranking model that has been trained on the test set; if it is determined that the accuracy of the ranking model that has been trained on the training model is greater than the accuracy of the ranking model that has not been trained on the sub-training set, supplement the test set and the sub-training set into a preset full training set to obtain a training set for training the ranking model.

[0098] Optionally, the generation module 304 is specifically used to input each training sample in the training set into the ranking model to be trained, and obtain the matching degree between the historical search results and search terms in the training sample output by the ranking model; and train the ranking model with the optimization objective of minimizing the deviation between the matching degree and the label corresponding to the training sample.

[0099] Optionally, the benchmark training samples include at least one of the following: training samples in which the matching degree between the search results predicted by the ranking model and the search terms is inconsistent with the actual execution result of the user, and training samples in which the confidence degree of the matching degree between the search results predicted by the ranking model and the search terms is lower than a preset threshold.

[0100] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This provides a method for generating training sets.

[0101] This instruction manual also provides Figure 4 One of the corresponding Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile storage into memory and then runs it to achieve the above-mentioned functions. Figure 1The method for generating the training set is described above. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0102] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0103] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0105] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0106] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0117] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for generating a training set, characterized in that, include: Obtain benchmark training samples; Based on the search terms and search results contained in the benchmark training samples, the feature representation corresponding to the benchmark training samples is determined as the benchmark feature representation. The benchmark feature representation is used to characterize the word features of the search terms contained in the benchmark training samples, as well as the matching situation of each word sequence formed by the search terms and at least part of the text in the search results. Based on the feature representations corresponding to each candidate sample, candidate samples that match the benchmark feature representations are selected from each candidate sample and used as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms. The target samples are labeled, and a training set for training the ranking model is generated based on the labeled target samples. The benchmark training samples include at least one of the following: training samples in which the matching degree between the search results predicted by the ranking model and the search terms is inconsistent with the actual execution results of the user, and training samples in which the confidence degree of the matching degree between the search results predicted by the ranking model and the search terms is lower than a preset threshold.

2. The method as described in claim 1, characterized in that, From each candidate sample, candidate samples that match the baseline feature representation are selected, specifically including: The weight of each candidate sample is determined based on the search time corresponding to each candidate sample. Based on the feature representation and weight of each candidate sample, candidate samples that match the benchmark feature representation are selected from the candidate samples and used as target samples.

3. The method as described in claim 2, characterized in that, The weight of each candidate sample is determined based on the search time corresponding to each candidate sample, specifically including: The weight of each candidate sample is determined based on the chronological order of the search times corresponding to each candidate sample. Specifically, for each candidate sample, the closer the search time corresponding to the candidate sample is to the current time, the greater the weight of the candidate sample.

4. The method as described in claim 2, characterized in that, The weight of each candidate sample is determined based on the search time corresponding to each candidate sample, specifically including: For each candidate sample, determine whether the search time corresponding to the candidate sample belongs to the specified time. If so, the preset first weight is used as the weight corresponding to the candidate sample; otherwise, the preset second weight is used as the weight corresponding to the candidate sample, and the first weight is greater than the second weight.

5. The method as described in claim 1, characterized in that, Based on the labeled target samples, a training set for model training is determined, specifically including: Based on the labeled target samples, determine the test set and the sub-training set; The ranking model is trained based on the sub-training set to obtain the trained ranking model; The training samples contained in the test set are respectively input into the ranking model that has not been trained on the sub-training set and the ranking model that has been trained, so as to determine the accuracy of the ranking model that has not been trained on the sub-training set in determining the matching degree between the search terms contained in the test set and the historical search results, which is taken as the accuracy of the ranking model that has not been trained on the sub-training set, and the accuracy of the ranking model that has been trained in determining the matching degree between the search terms contained in the test set and the historical search results, which is taken as the accuracy of the ranking model that has been trained. If it is determined that the accuracy of the trained ranking model is greater than the accuracy of the ranking model that was not trained on the sub-training set, the test set and the sub-training set are added to the preset full training set to obtain a training set for training the ranking model.

6. The method as described in claim 1, characterized in that, The method further includes: For each training sample in the training set, the training sample is input into the ranking model to be trained, and the matching degree between the historical search results and the search terms in the training sample output by the ranking model is obtained. The ranking model is trained with the goal of minimizing the deviation between the matching degree and the label corresponding to the training sample.

7. A training set generation apparatus, characterized in that, include: The acquisition module is used to acquire benchmark training samples; The determination module is used to determine the feature representation corresponding to the benchmark training sample based on the search terms contained in the benchmark training sample and the search results, as the benchmark feature representation. The benchmark feature representation is used to characterize the word features of the search terms contained in the benchmark training sample and the matching situation of each word sequence formed by the search terms and at least part of the text in the search results. The filtering module is used to filter out candidate samples that match the baseline feature representation from each candidate sample based on the determined feature representation of each candidate sample, and use them as target samples. For each candidate sample, the candidate sample includes historical search terms and historical search results corresponding to the historical search terms. The generation module is used to label the target samples and generate a training set for training the ranking model based on the labeled target samples. The benchmark training samples include at least one of the following: training samples in which the matching degree between the search results predicted by the ranking model and the search terms is inconsistent with the actual execution results of the user, and training samples in which the confidence degree of the matching degree between the search results predicted by the ranking model and the search terms is lower than a preset threshold.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images