A model training method, an information retrieval method, a device, and an electronic device

Abstract

The specification discloses a model training method, an information recall method, a device and an electronic device, specifically comprising: determining positive sample data and negative sample data of a historical query request from a document database according to the historical query request and full historical click records. Determine the sample set according to the positive and negative sample data, and input it into the recall model to be trained, so that the recall model determines the matching result of each sample data in the sample set and the historical query request. According to the deviation between the matching result and the sample data matching label, the loss value is determined. According to the loss value, the recall model is trained. The recall model after training determines the target recall document according to the query request of the target user and returns it to the target user. The recall model trained by this method can effectively reduce the influence of click heat on whether the document is positive sample data, effectively ensure the accuracy of the recall result in the actual information retrieval process, and to some extent enhance the user experience.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of data retrieval, and in particular to a model training method, an information recall method, an apparatus and an electronic device. BACKGROUND

[0002] With the rapid development and progress of computer and network technology, information retrieval technology has become increasingly mature and widely popular. This technology has brought great convenience to people in data collection in learning or work, or information query and understanding in daily life.

[0003] At present, in the current technology, the mainstream information retrieval method is mostly to pre-train the recall model to obtain an intelligent computing model with accurate information retrieval capability. However, in the training process of the recall model at the present stage, the construction of positive and negative samples in the training sample is not comprehensive. In the current technology, the user's non-clicked document data is mainly used as the composition of the negative sample, but in the real situation, under the same display interface, the user's non-clicked negative sample data may not necessarily be the document content irrelevant to the user's query request, but also may be limited by the display position or exposure condition, so that the user cannot click or notice such document. The recall model trained based on the above sample data has poor precision, and the recall result generated according to the user input data in actual application may not meet the user's expectation, which affects the execution efficiency of the whole information retrieval task and also affects the user experience to some extent.

[0004] Therefore, how to train a recall model that can accurately retrieve information in actual application is crucial. SUMMARY

[0005] The present specification provides a model training method, an information recall method, an apparatus and an electronic device to partially solve the above problems existing in the prior art.

[0006] The present specification adopts the following technical solutions:

[0007] The present specification provides a model training method, comprising:

[0008] obtaining historical query requests and full historical click records of each historical document;

[0009] querying the historical documents corresponding to the historical query requests from a preset document database as positive sample data corresponding to the historical query requests, and determining the historical documents with click times exceeding a preset number threshold but not required by the historical query requests from the preset document database as negative sample data corresponding to the historical query requests according to the full historical click records.

[0010] constructing a sample set according to the positive sample data and the negative sample data;

[0011] inputting each sample data included in the sample set and a historical click record corresponding to each sample data into a recall model to be trained, so that the recall model determines, for each sample data, feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data, determines a matching result corresponding to the sample data and the historical query request according to the feature data, and determines a loss value according to a deviation between the matching result and a matching label corresponding to the sample data, the loss value being in a positive correlation with the deviation;

[0012] training the recall model according to the loss value.

[0013] Optionally, the step of determining, according to the full-amount historical click record, the historical documents with the number of clicks exceeding the preset number threshold but not required by the historical query request from the preset document database as the negative sample data corresponding to the historical query request specifically comprises:

[0014] determining, from the preset document database, the historical documents corresponding to other historical query requests as the positive sample data corresponding to the other historical query requests;

[0015] determining, according to the full-amount historical click record, the sample data with the number of clicks exceeding the preset number threshold from the positive sample data corresponding to the other historical query requests as the negative sample data corresponding to the historical query request.

[0016] Optionally, the step of determining, according to the full-amount historical click record, the historical documents with the number of clicks exceeding the preset number threshold but not required by the historical query request from the preset document database as the negative sample data corresponding to the historical query request specifically comprises:

[0017] randomly selecting a plurality of historical documents from the preset document database, and determining, as the negative sample data corresponding to the historical query request, the randomly selected plurality of historical documents and the historical documents with the number of clicks exceeding the preset number threshold but not required by the historical query request from the preset document database.

[0018] Optionally, the step of determining the matching label corresponding to the sample data specifically comprises:

[0019] If it is determined that the sample data is positive sample data, a weighted value corresponding to the sample data is determined according to a historical click record corresponding to the sample data, wherein the weighted value is smaller if the historical click record corresponding to the sample data indicates that the sample data has been clicked more times in the past.

[0020] The original matching label corresponding to the sample data is weighted according to the weighted value, to obtain a matching label corresponding to the sample data.

[0021] Optionally, the step of determining the matching label corresponding to the sample data specifically includes:

[0022] If it is determined that the sample data is negative sample data, an initial weighted value corresponding to the sample data is determined according to a historical click record corresponding to the sample data, wherein the initial weighted value is larger if the historical click record corresponding to the sample data indicates that the sample data has been clicked more times in the past.

[0023] The average number of clicks corresponding to each negative sample data is determined according to the historical click record corresponding to each negative sample data in the sample set.

[0024] The initial weighted value corresponding to the sample data is adjusted according to the average number of clicks, to obtain an adjusted weighted value corresponding to the sample data.

[0025] The original matching label corresponding to the sample data is weighted according to the adjusted weighted value, to obtain a matching label corresponding to the sample data.

[0026] Optionally, the recall model is a double tower model.

[0027] The present specification provides a method for information recall, comprising:

[0028] Obtaining a query request of a target user;

[0029] Inputting the query request into a pre-trained recall model, so that the recall model determines a target recall document corresponding to the query request from a pre-set document database according to the query request, and the recall model is trained by the above-mentioned model training method;

[0030] Returning the target recall document data to the target user.

[0031] The present specification provides a device for model training, comprising:

[0032] An obtaining module, configured to obtain historical query requests and full historical click records of historical documents;

[0033] The query module is configured to query, from a preset document database, a historical document corresponding to the historical query request as positive sample data corresponding to the historical query request, and determine, from the preset document database according to the full-history click record, a historical document that has a number of clicks exceeding a preset number threshold but is not required by the historical query request as negative sample data corresponding to the historical query request.

[0034] The construction module is configured to construct a sample set according to the positive sample data and the negative sample data.

[0035] The matching module is configured to input each sample data included in the sample set and a historical click record corresponding to each sample data into a to-be-trained recall model, so that the recall model determines, for each sample data, feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data, determines a matching result of the sample data and the historical query request according to the feature data, and determines a loss value according to a deviation between the matching result and a matching label corresponding to the sample data, the loss value being positively correlated with the deviation.

[0036] The training module is configured to train the recall model according to the loss value.

[0037] The present specification provides a computer-readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the above method.

[0038] The present specification provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the above method when executing the program.

[0039] The above at least one technical solution adopted by the present specification can achieve the following beneficial effects:

[0040] As can be seen from the above method, in the model training method and the information recall method provided in the specification, the full amount of historical click records of the historical query request and each historical document can be obtained. Then, the historical document corresponding to the historical query request is queried from the document database as the positive sample data. At the same time, according to the full amount of historical click records, the negative sample data corresponding to the historical query request is determined from the document database. The sample set constructed according to the positive sample data and the negative sample data is input into the recall model to be trained, so that the recall model to be trained determines the matching result of each sample data and the historical query request according to each sample data in the sample set. According to the bias between the matching result and the sample data matching label, the loss value of the training process is determined. The recall model is trained according to the loss value. The trained recall model can return the target recall document corresponding to the query request to the target user according to the query request of the target user.

[0041] As can be seen from the above content, the model training method and the information recall method provided in the specification can determine the positive sample data and the negative sample data corresponding to the historical query request according to the obtained historical query request and the full amount of historical click records, and train the recall model to be trained based on the sample set composed of the positive sample data and the negative sample data. The trained recall model can accurately match the target recall document corresponding to the query request from the database according to the query request of the target user. The recall model trained by the method in the specification can effectively ignore the influence of click heat on whether the document is positive sample data, effectively ensure the accuracy of the recall result in the actual information retrieval process, effectively meet the actual needs of the user, improve the task execution efficiency of the user, and effectively enhance the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0042] The drawings described herein are used to provide further understanding of the specification, and form a part of the specification. The illustrative embodiments of the specification and their descriptions serve to explain the specification, and do not constitute an improper limitation on the specification. In the drawings:

[0043] Figure 1 A flowchart of a model training method provided in the specification;

[0044] Figure 2 An example schematic diagram of an information recall method provided in the specification;

[0045] Figure 3 A schematic diagram of a model training device provided in the specification;

[0046] Figure 4 A schematic diagram of an information recall device provided in the specification;

[0047] Figure 5 A structure diagram of an electronic device corresponding to Figure 1 and Figure 2 provided in the present specification. DETAILED DESCRIPTION

[0048] For the purposes of the present specification, the technical solutions and advantages, the following will be combined with the embodiments of the present specification and the corresponding drawings to clearly and completely describe the technical solutions of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, not all. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present specification.

[0049] The technical solutions provided by the embodiments of the present specification will be described in detail below with reference to the drawings.

[0050] Figure 1 A flowchart of a model training method provided in the present specification, comprising the following steps:

[0051] S101: Obtain historical query requests and full historical click records of each historical document.

[0052] S102: Query the historical documents corresponding to the historical query requests from the preset document database as the positive sample data corresponding to the historical query requests, and determine the historical documents with click counts exceeding a preset number threshold but not required by the historical query requests from the preset document database as negative sample data corresponding to the historical query requests according to the full historical click records.

[0053] With the progress of science and technology, information retrieval technology has been perfected, and the wide popularity of this technology has provided great practical convenience for people's daily life, learning or work, etc. At the present stage, information retrieval means mostly uses pre-trained recall models to achieve, but in the current training process of the recall model, the construction method used when constructing negative samples of the sample set is not reasonable.

[0054] At the present stage, it is mainly believed that when users perform information retrieval, the documents in the retrieval results that are not clicked are considered as negative samples irrelevant to the search terms. However, in fact, this may not be true. It may be because of the display order or content blocking that the user cannot directly see such documents. Based on such negative samples, the recall model trained has poor retrieval accuracy when used formally, and there is a significant difference between the retrieved results and the user's expectations, thereby affecting the user experience. Therefore, how to train a recall model with high retrieval accuracy is a very important problem.

[0055] To this end, the specification provides a model training method and an information recall method, wherein the execution subject of the method provided by the specification can be a server, or a terminal device such as a desktop computer, a notebook computer, and the like. In addition, the execution subject of the specification can also be a software form of subject, such as a client installed in a terminal device. For ease of illustration, the specification below only takes the server as the execution subject to describe a model training method and an information recall method.

[0056] Based on this, the server applying the model training method and the information recall method provided by the specification can determine the positive sample data and the negative sample data corresponding to the historical query request according to the historical query request and the full amount of historical click records of each historical document. Then, the server can construct a sample set according to the positive and negative sample data, and optimize the model training of the recall model to be trained. The server can use the trained recall model to accurately query the corresponding target document according to the query request of the target user, and return it to the target user.

[0057] Wherein, the application scenario of the trained recall model can be determined according to actual application requirements, for example, in the e-commerce field, the server can obtain the query request input or clicked by the target user, and input the query request into the recall model trained by the method in the specification, so that the recall model can determine the recall result that exactly meets the user's expectation according to the query request. Effectively guarantee the efficiency of information query, maximize the actual needs of users, and improve the user experience.

[0058] For another example, in the field of intelligent question and answer, the server can input the query request corresponding to the target user's consultation question obtained into the recall model trained by the method in the specification, so that the recall model can recall the document content or related materials most matched with the query request from the database, and answer the question based on the matching result. Improve the efficiency of intelligent question and answer task execution, and the accurate matching result can also avoid the situation of answering the wrong question to a certain extent, effectively solve the actual problems of users, and optimize the user experience.

[0059] The method provided in the specification is mainly divided into two stages, one is the model training stage, and the other is the actual application stage. In the model training stage, the server can obtain the pre-stored historical query request and the full amount of historical click records of each historical document in the document database.

[0060] Then, the server can query the determined historical document corresponding to the obtained historical query request from the preset document database as the positive sample data corresponding to the historical query request. Meanwhile, the server can also determine the historical document corresponding to the historical query request from the preset document database according to the obtained full historical click record, which has a click number exceeding a preset number threshold but does not correspond to the historical query request, as the negative sample data corresponding to the historical query request.

[0061] Specifically, the server can determine the historical document clicked by the user according to the search result after the user inputs the historical query request in the historical process from the full historical click record according to the historical query request, find the corresponding historical document from the above-mentioned preset document database, and take it as the positive sample data corresponding to the historical query request. The above-mentioned positive sample data can be understood as the historical document in the positive sample data, which is the historical document that the user actively clicks in the query result fed back to the user when the historical query request is initiated in the historical query process.

[0062] Regarding the negative sample data, the server can first randomly select a number of historical document data from the preset document database, and then determine the historical document data that has a click number exceeding a preset number threshold but is not the historical document corresponding to the historical query request from the preset document database according to the obtained full historical click record. The above-mentioned two types of historical document data, i.e., the historical document data randomly selected from the document database and the historical document data irrelevant to the historical query request determined based on the full historical click record, are combined according to a preset negative sample composition ratio, so as to obtain the negative sample data corresponding to the historical query request.

[0063] The specific process of determining the historical document data irrelevant to the historical query request based on the full historical click record is not strictly limited in the specification. Since the constructed positive sample data and negative sample data need to be used for the subsequent training process of the recall model, the sample set corresponding to multiple historical query requests that can be used in the overall model training process. Further, after determining the positive sample data corresponding to each historical query request through the above-mentioned process, for each historical query request, the server can determine the sample data with a click number exceeding a preset number threshold from the positive sample data corresponding to other historical query requests except the historical query request according to the obtained full historical record, i.e., the historical document corresponding to the historical query request, as the negative sample data corresponding to the historical query request.

[0064] To facilitate understanding of the construction process of the above negative sample data, the following is illustrated by example. In this specification, when the number of historical query requests obtained is large, the server can divide the plurality of historical query requests into a plurality of training batches according to the preset number of requests in each batch, so as to reasonably plan the training process of the subsequent recall model. Assume that a training batch contains five historical query requests with no relevance: Q1, Q2, Q3, Q4, and Q5. The server determines that the corresponding positive sample data of the five historical query requests in the preset document database are: Q1-D1, Q2-D2, Q3-D3, Q4-D4, and Q5-D5 through the above steps.

[0065] When determining the corresponding negative sample data for the historical query request Q1, the server can directly use the historical documents corresponding to the positive sample data of other historical query data in the same training batch as the negative sample data corresponding to the historical query request Q1, that is, the negative sample data corresponding to the historical query request Q1 can include: D2, D3, D4, and D5. The same applies to the negative sample data corresponding to other historical query requests Q2, Q3, Q4, and Q5, which are not listed here. At this point, the server can determine the positive sample data and negative sample data corresponding to each historical query request in the training batch. The server can also randomly select a number of historical document data from the preset document database based on the above method, and combine the historical documents corresponding to the positive sample data of other historical query requests as the negative sample data corresponding to each historical query request.

[0066] It should be noted that the preset number threshold mentioned in the above process is mainly used to evaluate the click heat of the historical document in the historical query process, and the specific value and range are not strictly limited in this specification. The preset number threshold can be directly set to zero. Of course, considering that the negative sample data constructed through this step in the subsequent process reduces the attention of the recall model to the click heat of the historical document, the preset number threshold can also be flexibly increased according to the application scenario and requirements.

[0067] In addition, the specific value and range of the negative sample proportion mentioned in the above method are still not strictly limited in this specification, and the proportion of randomly selected historical document data in the document database and historical document data unrelated to the historical query request determined based on the full amount of historical click records can be 9:1. Of course, it can also be appropriately adjusted according to the actual application scenario and data volume.

[0068] In addition, in the present specification, the historical documents mentioned in the above process, the preset document database, and the document data in the positive sample data and the negative sample data are not strictly limited to article data represented in the form of text, but can also be any type of retrieval data represented in the form of pictures, videos, web pages, product connections, etc. In the present specification, all types of retrieval data are summarized as "documents" for ease of understanding.

[0069] S103: Constructing a sample set according to the positive sample data and the negative sample data.

[0070] S104: Inputting each sample data included in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to each sample data according to the sample data and the historical click record corresponding to the sample data, determines the matching result of the sample data corresponding to the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the matching label corresponding to the sample data.

[0071] In the present specification, the server can construct a sample set corresponding to the historical query request according to the positive sample data and the negative sample data corresponding to the historical query request determined by the above steps.

[0072] Specifically, the server can combine the positive sample data and the negative sample data corresponding to the historical query request into a sample set according to the corresponding proportion size according to the preset sample set composition proportion. It should be noted that the specific proportion in the preset sample set composition proportion is not strictly limited in the present specification, and can be flexibly set according to the actual application scenario and the data volume. In order to ensure the recall accuracy and training efficiency of the subsequent recall model, the proportion of negative sample data can be much larger than the proportion of positive sample data to construct the above sample set, for example, the proportion of positive sample data and negative sample data in the sample set can be 1:14, of course, it can also be adjusted appropriately according to the actual application scenario and the number of sample data.

[0073] After the sample set is constructed, the server can input the sample set and the historical click record data corresponding to each sample data in the sample set into the to-be-trained recall model, so that the recall model can determine the feature data corresponding to each sample data in the sample set, i.e., each historical document, according to the sample data and the historical click record data corresponding to the sample data. And according to the feature data, determine the matching result corresponding to the sample data and the historical query request, so as to determine the loss value of the recall model for the sample data according to the deviation between the matching result and the matching label corresponding to the sample data. Wherein, the loss value and the deviation between the above matching result and the matching label corresponding to the sample data are positively correlated.

[0074] It should be noted that the reason why the historical documents with a certain number of clicks are added in the construction process of the negative sample data in the present specification is to avoid the recall model paying too much attention to the relationship between the click heat of the historical document itself and the positive sample in the training process here, so as to ensure that the recall model can focus more on the retrieval matching of the query request content in actual application, rather than excessively considering the click heat of the document itself in the matching result.

[0075] In the current technology, due to the lack of comprehensive consideration in the construction of negative sample data, the recall model is easily affected by the click heat of the sample document itself in the training process, so that the recall model finally trained will also consider the click heat of the document itself as a retrieval factor in actual application. The phenomenon that may occur is that the recall model only performs content matching from the document data with high click heat when retrieving documents according to the user's query request, while the document with low click heat but high content matching degree may be considered as not matching the query request. The occurrence of such a situation affects the exposure and display of low-click-heat documents, and also makes users unable to obtain document data that truly meets their actual needs, affecting user experience, and also making users spend more time searching for the document they really need, seriously reducing the execution efficiency of related personnel.

[0076] In the present specification, by taking the historical documents with a certain click heat and irrelevant to the content of the historical query request as the negative sample data corresponding to the historical query request, the click heat problem mentioned in the above content can be effectively solved. In the training process of the above recall model, the historical documents in the positive sample data in the sample set are all historical documents with a certain click heat, and the negative sample data also contains historical documents with a certain click heat. In the case that there are historical documents with a certain click heat in both the positive sample data and the negative sample data, the recall model can not consider whether the click heat of the document itself is associated with the positive sample data in the training process, so as to focus more on whether the historical query content matches the content of the historical document in the sample set.

[0077] The feature data corresponding to each sample data mentioned in the above matching process contains not only the data content features of the corresponding historical document, but also the click heat features determined according to the historical click record of the sample data. In the above step, the click heat features of the sample data in the negative sample data are not completely low click heat, and are similar to the click heat features of the sample data in the positive sample data, so that the click heat features of the sample data in the overall sample set are relatively average, effectively reducing the attention degree of the recall model to the click heat in the training process.

[0078] In addition, in the present specification, it is considered that there may be high and low click heat in the positive sample data, and the document content with high click heat may not be highly matched with the query request, but because of the influence of high click heat, it is clicked by the user, causing false data record, so as to affect the training process of the recall model. Therefore, the server can perform click heat-based weighting processing on the positive sample data in the sample data.

[0079] Specifically, for each sample data in the sample set, when the server determines that the sample data is a positive sample data, the server can determine a weighting value corresponding to the sample data according to the historical click record corresponding to the sample data. The weighting value is used to represent the influence of the click heat of the sample data on the matching degree of the sample data and the historical query request. If the historical document corresponding to the sample data is clicked more frequently according to the historical click record corresponding to the sample data, that is, the click heat of the sample data is higher, then the influence of high click heat on the matching degree needs to be eliminated, which is specifically represented as the above weighting value is smaller.

[0080] Then, the server can perform weighting processing on the original matching label corresponding to the sample data according to the determined weighting value, so as to obtain the weighted matching label corresponding to the sample label. For example, if the server determines that the number of clicks of a certain sample data in the positive sample data in the sample set in the historical query process is the most among all positive sample data according to the historical click record corresponding to the sample data, that is, the click heat is extremely high. The server can weight the matching label of the sample data according to the preset weighting value ratio. Assuming that the original matching label of the sample data is 1, which specifically represents that the sample data is completely matched with the historical query request, and the weighting value determined according to the above process is 0.8, then the weighted matching label of the sample data can be 1*0.8, that is, 0.8.

[0081] Similarly, it can be known that, when the number of clicks of the sample data in the historical query process is determined to be the least among all positive sample data according to the historical click record corresponding to the sample data, it means that the click heat is low. The server can also preset a weighting value ratio to weight the matching label of the sample data, and increase the weighting of the original matching label, for example, from 1 to 1.2. The above-mentioned method of applying a weighting value based on historical click records can effectively reduce the influence of click heat on the matching degree between sample data and historical query requests.

[0082] In addition to the above weighting processing of the matching label of each sample data in the positive sample data, in this specification, the server can also perform similar weighting processing on the sample data in the negative sample data, so as to adjust the influence of the click heat of the sample data in the negative sample data on the model training process.

[0083] Specifically, for each sample data in the sample set, when the server determines that the sample data belongs to negative sample data, it can determine the initial weighting value corresponding to the sample data according to the historical click record corresponding to the sample data. Unlike the above method, when the number of clicks of the sample data in the historical query process is determined to be high according to the historical click record, that is, the click heat corresponding to the sample data is high, the initial weighting value corresponding to the sample data can be larger.

[0084] Then, the server can determine the average number of clicks corresponding to the negative sample data according to the historical click record corresponding to each sample data in the negative sample data. Next, the server can adjust the initial weighting value of each sample data in the negative sample data according to the average number of clicks, specifically, if the number of clicks corresponding to the sample data is higher than the determined average number of clicks, the server can appropriately reduce the initial weighting value, on the contrary, if the number of clicks corresponding to the sample data is lower than the determined average number of clicks, the server can appropriately increase the initial weighting value.

[0085] Then, the server can weight the original matching label of each sample data in the negative sample data according to the adjusted initial weighting value of each sample data in the negative sample data, to obtain the weighted matching label corresponding to each sample data in the negative sample data.

[0086] Through the above weighting processing of the sample data in the negative sample data, the high and low of the click heat in the negative sample data can be well averaged, effectively avoiding the uneven distribution of click heat in the negative sample data, resulting in the case of sample data with extremely high or low click heat, reducing the influence of click heat on the recall model training, and effectively improving the model training efficiency and model training accuracy.

[0087] S105: training the recall model according to the loss value.

[0088] In the present specification, the server can train and optimize the recall model to be trained according to the loss value determined through the above training process. In order to minimize the loss value, the model parameters in the recall model are appropriately adjusted to obtain the trained recall model.

[0089] It should be noted that the recall model mentioned in the present specification can be a dual-tower model (Dual-Tower Model). Based on the special dual-tower structure of the dual-tower model, the server can input the sample data in the sample set and the historical click record corresponding to the sample data into one tower, and input the historical query request into the other tower, and extract data features through the dual-tower model to obtain data features corresponding to the sample data and data features corresponding to the historical query request. The data features corresponding to the sample data not only include data content features corresponding to the data content of the sample data, but also include click heat features corresponding to the click heat of the sample data.

[0090] Through the construction of the above negative sample data and the weighting processing of the positive and negative sample data, the degree of attention of the recall model to the click heat of the sample data in the training process can be effectively removed, so as to mainly determine the matching result according to the data content features of the sample data and the data features corresponding to the historical query request. In addition, the above-mentioned training and optimization process can also use the pointwise type loss (Pointwise Loss Function) with faster training speed to optimize the recall model to be trained.

[0091] The trained recall model can be tested online at the same time as the old recall model. If it is determined according to the test result that the user click rate is improved compared with the old recall model after the recall task is performed by the recall model trained through the above method, the trained recall model can be directly replaced by the old recall model to perform the recall task. If the user click rate is not improved or even decreased compared with the old recall model after the recall task is performed by the recall model trained through the above method, the configuration of the model parameters in the training process of the recall model, and the proportion distribution and data source of each sample in the above model training process need to be adjusted.

[0092] The method provided in the present specification is mainly divided into two stages, a model training stage and an actual application stage. The above-mentioned model training stage is mainly used to obtain a recall model that can accurately recall corresponding document data according to the query request after the above-mentioned model training process, so that in the actual application stage, the trained recall model can be applied to the server or terminal device used to perform the recall task. In order to facilitate the introduction of the information recall method, the following will be described with reference to a flowchart of an information recall method, as shown inFigure 2 as shown.

[0093] Figure 2 A flowchart of a method of information recall provided in the present specification includes the following steps:

[0094] S201: Obtain a query request of a target user.

[0095] At present, the increasing maturity and wide application of information retrieval technology have brought high convenience to various fields. However, there are obvious defects in the training process of the recall model in the information retrieval technology at the present stage. The construction of negative sample data is too absolute, and there is a sample selection bias (SSB) problem. The recall model after training lacks accuracy in the retrieved results when performing the information retrieval task, cannot effectively meet the real needs of users, and further affects the task execution efficiency and user experience of users.

[0096] Therefore, the present specification provides a method of information recall, wherein the execution subject of the method provided by the present specification can be a server, or a terminal device such as a desktop computer, a notebook computer, and the like. In addition, the execution subject of the present specification can also be a software subject, such as a client installed in a terminal device. In order to facilitate the description, the present specification will only take the server as the execution subject to explain the method of information recall provided.

[0097] Based on this, the server applying the method of information recall provided by the present specification can input the query request of the user into the pre-trained recall model, so that the recall model can accurately match the corresponding target document according to the query request and return it to the target user.

[0098] Wherein, the application scenario of the information recall method can be determined according to the actual application requirement, for example, in the field of search engine, the server can obtain the query request manually input or clicked by the target user in the search box part, input the obtained query request into the pre-trained recall model, so that the recall model can accurately retrieve the recall result matched with the query request according to the query request, and display it to the target user. Effectively guarantee the query efficiency, and the highly matched recall result can also maximize the user demand.

[0099] For another example, in the field of data statistics, the server can obtain the query request of the target user input for the data, and input the query request into the pre-trained recall model, so that the recall model can query the corresponding recall result from the database according to the query request. The server can display the recall result to the target user, effectively guarantee the user to obtain the real matched data, improve the data retrieval efficiency, and optimize the user experience.

[0100] In the present specification, the server can obtain a query request of a target user. It should be noted that the specific data form of the query request of the target user is not strictly limited in the present specification, which can be a query request in the form of text, or can also be in the form of a picture, or even in the form of a video, etc. Different types of query requests can be set and adjusted according to actual application scenarios.

[0101] S202: inputting the query request into a pre-trained recall model, so that the recall model determines a target recall document corresponding to the query request from a preset document database according to the query request, and the recall model is obtained by training the model according to the above method.

[0102] S203: returning the target recall document data to the target user.

[0103] In the present specification, the server can input the obtained query request into a pre-trained recall model, so that the recall model can determine a target document corresponding to the query request from a preset document database according to the received query request, as a target recall document. The pre-training process of the recall model is realized based on the above model training method.

[0104] After determining the corresponding target recall document of the query request, the server can send the target recall document to the terminal device used by the target user, so as to display the target recall document to the target user through the terminal device.

[0105] From the above content, it can be seen that the model training method and the information recall method provided in the present specification can determine positive sample data and negative sample data corresponding to the historical query request according to the obtained historical query request and full historical click record, and train the recall model to be trained based on the sample set composed of the positive sample data and the negative sample data. The trained recall model can accurately match the target recall document corresponding to the query request from the database according to the query request of the target user. The recall model trained by the method in the present specification can effectively ignore the influence of click heat on whether the document is positive sample data, effectively guarantee the accuracy of the recall result in the actual information retrieval process, effectively meet the actual needs of the user, improve the task execution efficiency of the user, and effectively enhance the user experience.

[0106] The above is the method of one or more embodiments of the present specification. Based on the same idea, the present specification also provides a corresponding model training device, as shown in Figure 3 .

[0107] Figure 3 FIG. 1 is a schematic diagram of a model training device provided in the present specification, which comprises:

[0108] The acquisition module 301 is configured to acquire a historical query request and full-volume historical click records of each historical document.

[0109] The query module 302 is configured to query, from a preset document database, historical documents corresponding to the historical query request as positive sample data corresponding to the historical query request, and determine, from the preset document database according to the full-volume historical click records, historical documents with click counts exceeding a preset count threshold but not required by the historical query request as negative sample data corresponding to the historical query request.

[0110] The construction module 303 is configured to construct a sample set according to the positive sample data and the negative sample data.

[0111] The matching module 304 is configured to input each sample data included in the sample set and historical click records corresponding to each sample data into a to-be-trained recall model, so that the recall model determines, for each sample data, feature data corresponding to the sample data according to the sample data and the historical click records corresponding to the sample data, determines a matching result of the sample data corresponding to the historical query request according to the feature data, and determines a loss value according to a deviation between the matching result and a matching label corresponding to the sample data, the loss value and the deviation being in a positive correlation relationship.

[0112] The training module 305 is configured to train the recall model according to the loss value.

[0113] Optionally, the query module 302 is specifically configured to determine, from the preset document database, historical documents corresponding to other historical query requests as respective positive sample data corresponding to the other historical query requests, and determine, from the respective positive sample data corresponding to the other historical query requests according to the full-volume historical click records, sample data with click counts exceeding a preset count threshold as negative sample data corresponding to the historical query request.

[0114] Optionally, the query module 302 is specifically configured to randomly select a plurality of historical documents from the preset document database, and determine, from the preset document database, historical documents with click counts exceeding a preset count threshold but not required by the historical query request as negative sample data corresponding to the historical query request.

[0115] Optionally, the matching module 304 is specifically configured to, if it is determined that the sample data is positive sample data, determine a weighted value corresponding to the sample data according to a historical click record corresponding to the sample data, wherein the weighted value is smaller if the historical click record corresponding to the sample data indicates that the sample data has been clicked more times in the past; and weight the original matching label corresponding to the sample data according to the weighted value to obtain the matching label corresponding to the sample data.

[0116] Optionally, the matching module 304 is specifically configured to, if it is determined that the sample data is negative sample data, determine an initial weighted value corresponding to the sample data according to a historical click record corresponding to the sample data, wherein the initial weighted value is larger if the historical click record corresponding to the sample data indicates that the sample data has been clicked more times in the past; determine a mean value of the number of clicks corresponding to each negative sample data according to the historical click record corresponding to each negative sample data in the sample set; adjust the initial weighted value corresponding to the sample data according to the mean value of the number of clicks to obtain an adjusted weighted value corresponding to the sample data; and weight the original matching label corresponding to the sample data according to the adjusted weighted value to obtain the matching label corresponding to the sample data.

[0117] Optionally, the recall model is a dual tower model.

[0118] Based on the same idea, the present specification also provides a corresponding information recall device, as shown in Figure 4 .

[0119] Figure 4 FIG. 1 is a schematic diagram of an information recall device provided in the present specification, which comprises:

[0120] The acquisition module 401 is configured to acquire a query request of a target user.

[0121] The recall module 402 is configured to input the query request into a pre-trained recall model, so that the recall model determines a target recall document corresponding to the query request from a pre-set document database according to the query request, and the recall model is obtained by training the above-mentioned model training method.

[0122] The return module 403 is configured to return the target recall document data to the target user.

[0123] The present specification also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the above-mentioned Figure 1 and Figure 2 providing a model training method and an information recall method.

[0124] The present specification also providesFigure 5 An electronic device corresponding to Figure 1 and Figure 2 An illustrative structural diagram of an electronic device. As Figure 5 shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course can also include other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the model training method shown in Figure 1 or the information recall method shown in Figure 2 .

[0125] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0126] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0127] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0128] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in one or more software and / or hardware in implementing the present specification.

[0129] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0130] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0131] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. ​ The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0133] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0134] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer-readable media.

[0135] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0136] It should also be noted that the terms "comprising", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0137] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The present specification can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0139] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different but related aspects of the description. Each of the various embodiments can stand on its own, and each can be combined with the subject matter of other embodiments to produce further embodiments. Where appropriate, therefore, the contents of the specification can be regarded as incorporating text of the detailed description under the heading "Embodiments."

[0140] The above description is embodied only by the embodiments of the specification, and is not intended to limit the specification. The specification can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the specification shall be included in the scope of claims of the specification.

Claims

1. A method for training a model, characterized in that, include: Retrieve multiple unrelated historical query requests and the full historical click records of each historical document; For any of the aforementioned historical query requests, the historical documents corresponding to the historical query request are retrieved from the preset document database and used as positive sample data corresponding to the historical query request. Additionally, the historical documents corresponding to other historical query requests are determined from the preset document database and used as positive sample data corresponding to the other historical query requests. Based on the full historical click records, sample data whose click count exceeds a preset threshold is determined from the positive sample data corresponding to the other historical query requests, and used as the negative sample data corresponding to the historical query requests; Construct a sample set based on the positive sample data and the negative sample data; Each sample data and its corresponding historical click record in the sample set are input into the recall model to be trained. The recall model, for each sample data, determines the feature data corresponding to that sample data based on the sample data and its corresponding historical click record. Based on the feature data, it determines the matching result between the sample data and the historical query request. Furthermore, it determines the loss value based on the deviation between the matching result and the determined matching tag corresponding to the sample data, wherein the loss value and the deviation are positively correlated. The feature data corresponding to the sample data includes the data content features of the corresponding historical document and the click popularity features determined based on the historical click records of the sample data. The recall model is trained based on the loss value.

2. The method as described in claim 1, characterized in that, The step of determining, based on the full historical click records, historical documents from the preset document database whose click count exceeds a preset threshold but are not required by the historical query request, and using these documents as negative sample data corresponding to the historical query request, specifically includes: Several historical documents are randomly selected from the preset document database, and the randomly selected historical documents, as well as historical documents from the preset document database whose click count exceeds a preset threshold but are not required by the historical query request, are used as negative sample data corresponding to the historical query request.

3. The method as described in claim 1, characterized in that, The step of determining the matching label corresponding to the sample data specifically includes: If the sample data is determined to be positive sample data, the weighting value corresponding to the sample data is determined based on the historical click records corresponding to the sample data. The more times the sample data has been clicked in history, the smaller the weighting value will be. Based on the weighted value, the original matching labels corresponding to the sample data are weighted to obtain the matching labels corresponding to the sample data.

4. The method as described in claim 1, characterized in that, The step of determining the matching label corresponding to the sample data specifically includes: If the sample data is determined to be negative sample data, then the initial weighting value corresponding to the sample data is determined based on the historical click records corresponding to the sample data. The more times the sample data has been clicked in history, the larger the initial weighting value will be. Based on the historical click records corresponding to each negative sample data in the sample set, determine the average number of clicks for each negative sample data. Based on the average number of clicks, the initial weighting value corresponding to the sample data is adjusted to obtain the adjusted weighting value corresponding to the sample data. Based on the adjusted weighted value, the original matching label corresponding to the sample data is weighted to obtain the matching label corresponding to the sample data.

5. The method according to any one of claims 1 to 4, characterized in that, The recall model is a dual-tower model.

6. A method for information retrieval, characterized in that, include: Obtain the query request from the target user; The query request is input into a pre-trained recall model, which determines the target recall document corresponding to the query request from a preset document database based on the query request. The recall model is trained by the method described in any one of claims 1 to 5. The target recall document data is returned to the target user.

7. A device for model training, characterized in that, include: The acquisition module is used to acquire historical query requests and the full historical click records of each historical document; The query module is used to query the historical documents corresponding to the historical query request from the preset document database as positive sample data corresponding to the historical query request, and to determine the historical documents whose click count exceeds a preset threshold but are not required by the historical query request from the preset document database based on the full historical click records as negative sample data corresponding to the historical query request. The construction module is used to construct a sample set based on the positive sample data and the negative sample data; The matching module is used to input each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to each sample data based on the sample data and the historical click record corresponding to the sample data, determines the matching result between the sample data and the historical query request based on the feature data, and determines the loss value based on the deviation between the matching result and the determined matching label corresponding to the sample data, wherein the loss value and the deviation are positively correlated. The training module is used to train the recall model based on the loss value.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images