Model training method and device, information recall method and device and electronic equipment

Abstract

The invention discloses a model training method and device, an information recall method and device and electronic equipment, and the method specifically comprises the steps: determining positive sample data and negative sample data of a historical query request from a document database according to the historical query request and a total historical click record; and determining a sample set according to the positive and negative sample data, and inputting the sample set into a to-be-trained recall model, so that the recall model determines a matching result of each sample data in the sample set and the historical query request. And determining a loss value according to the deviation between the matching result and the sample data matching label. And performing model training on the recall model according to the loss value. And the trained recall model determines a target recall document according to a query request of the target user and returns the target recall document to the target user. The recall model trained through the method can effectively reduce the influence of the click popularity on whether the document is positive sample data or not, the accuracy of the recall result in the actual information retrieval process is effectively guaranteed, and the use experience of a user is enhanced to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and in particular to a model training method, an information recall method, a device and an electronic device. Background Art

[0002] With the rapid development and progress of computer and network technology, information retrieval technology has become increasingly mature and popular. This technology has brought great convenience to people in data collection in study or work, as well as information query and understanding in daily life.

[0003] At present, in the current technology, most mainstream information retrieval methods pre-train the recall model to obtain an intelligent computing model with accurate information retrieval capabilities. However, in the current training process for the recall model, when constructing positive and negative samples in the training samples, the construction of negative samples is not fully considered. In the current technology, document data that users have not clicked on is mainly used as the composition of negative samples, but in real situations, under the same display interface, negative sample data that users have not clicked on may not necessarily be document content that is irrelevant to the user's query request, and may also be restricted due to display location or exposure conditions, resulting in users being unable to click or notice such documents. The recall model trained based on the above sample data has poor accuracy. In actual applications, the recall results generated based on user input data may not meet user expectations, affecting the execution efficiency of the overall information retrieval task, and will also affect the user experience to a certain extent.

[0004] Therefore, it is crucial to train a recall model that can perform information retrieval with high accuracy in practical applications. Summary of the invention

[0005] This specification provides a model training method, an information recall method, a device and an electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides a model training method, including:

[0008] Obtain historical query requests and the full historical click records of each historical document;

[0009] Querying historical documents corresponding to the historical query request from a preset document database as positive sample data corresponding to the historical query request, and determining historical documents whose click counts exceed a preset threshold but are not required by the historical query request from the preset document database based on the full amount of historical click records as negative sample data corresponding to the historical query request;

[0010] Constructing a sample set according to the positive sample data and the negative sample data;

[0011] Input each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data for each sample data, determines the matching result corresponding to the sample data and the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the determined matching label corresponding to the sample data, and the loss value is positively correlated with the deviation;

[0012] The recall model is trained according to the loss value.

[0013] Optionally, the step of determining, from the preset document database, historical documents whose click counts exceed a preset threshold but are not required by the historical query request, as negative sample data corresponding to the historical query request, based on the full amount of historical click records, specifically includes:

[0014] Determining historical documents corresponding to other historical query requests from the preset document database as positive sample data corresponding to the other historical query requests;

[0015] According to the full amount of historical click records, sample data whose click times exceed a preset threshold are determined from each positive sample data corresponding to the other historical query requests as negative sample data corresponding to the historical query requests.

[0016] Optionally, the step of determining, from the preset document database, historical documents whose click counts exceed a preset threshold but are not required by the historical query request, as negative sample data corresponding to the historical query request, based on the full amount of historical click records, specifically includes:

[0017] A number of historical documents are randomly selected from the preset document database, and the randomly selected historical documents and historical documents determined from the preset document database whose click counts exceed a preset threshold but are not required by the historical query request are used as negative sample data corresponding to the historical query request.

[0018] Optionally, the step of determining a matching tag corresponding to the sample data specifically includes:

[0019] If the sample data is determined to be positive sample data, a weighted value corresponding to the sample data is determined according to the historical click records corresponding to the sample data, wherein if the number of times the sample data has been clicked in history is determined to be greater through the historical click records corresponding to the sample data, the weighted value is smaller;

[0020] The original matching label corresponding to the sample data is weighted according to the weighted value to obtain the matching label corresponding to the sample data.

[0021] Optionally, the step of determining a matching tag corresponding to the sample data specifically includes:

[0022] If it is determined that the sample data is negative sample data, then according to the historical click records corresponding to the sample data, an initial weighted value corresponding to the sample data is determined, wherein if it is determined through the historical click records corresponding to the sample data that the sample data has been clicked more times in history, the initial weighted value is greater;

[0023] Determine the average number of clicks corresponding to each negative sample data according to the historical click records corresponding to each negative sample data in the sample set;

[0024] According to the average number of clicks, the initial weighted value corresponding to the sample data is adjusted to obtain an adjusted weighted value corresponding to the sample data;

[0025] The original matching label corresponding to the sample data is weighted according to the adjusted weighted value to obtain the matching label corresponding to the sample data.

[0026] Optionally, the recall model is a double-tower model.

[0027] This manual provides a method for information recall, including:

[0028] Get the query request of the target user;

[0029] Inputting the query request into a pre-trained recall model, so that the recall model determines the target recall document corresponding to the query request from a preset document database according to the query request, and the recall model is trained by the above-mentioned model training method;

[0030] The target recalled document data is returned to the target user.

[0031] This specification provides a model training device, including:

[0032] The acquisition module is used to obtain historical query requests and the full historical click records of each historical document;

[0033] A query module, configured to query a preset document database for historical documents corresponding to the historical query request as positive sample data corresponding to the historical query request, and to determine, based on the full amount of historical click records, historical documents whose click counts exceed a preset threshold but are not required by the historical query request from the preset document database as negative sample data corresponding to the historical query request;

[0034] A construction module, used to construct a sample set according to the positive sample data and the negative sample data;

[0035] A matching module, used for inputting each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data for each sample data, and determines the matching result corresponding to the sample data and the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the determined matching label corresponding to the sample data, and the loss value is positively correlated with the deviation;

[0036] A training module is used to train the recall model according to the loss value.

[0037] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method is implemented.

[0038] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above method is implemented when the processor executes the program.

[0039] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0040] It can be seen from the above method that in the model training method and information recall method provided in this specification, the historical query requests and the full historical click records of each historical document can be obtained. Then, the historical documents corresponding to the historical query requests are queried from the document database as positive sample data. At the same time, based on the full historical click records, the negative sample data corresponding to the historical query requests are determined from the document database. The sample set constructed based on the positive sample data and the negative sample data is input into the recall model to be trained, so that the recall model to be trained determines the matching result corresponding to each sample data and the historical query request based on each sample data in the sample set. The loss value of the training process is determined based on the partial check between the matching result and the sample data matching label. The recall model is trained based on the loss value. The trained recall model can return the target recall document corresponding to the query request to the target user based on the query request of the target user.

[0041] From the above content, it can be seen that the model training method and information recall method provided in this specification can determine the positive sample data and negative sample data corresponding to the historical query request based on the acquired historical query request and the full amount of historical click records, and perform model training on the recall model to be trained based on the sample set composed of the positive sample data and the negative sample data. The trained recall model can accurately match the target recall document corresponding to the query request from the database according to the query request of the target user. The recall model trained by the method in this specification can effectively ignore the influence of click heat on whether the document is positive sample data, effectively ensure the accuracy of the recall results in the actual information retrieval process, effectively meet the actual needs of users, improve the user's task execution efficiency, and effectively enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0043] Figure 1 A flowchart of a model training method provided in this specification;

[0044] Figure 2 This is a schematic diagram of an example of an information recall method provided in this manual;

[0045] Figure 3 A schematic diagram of a model training device provided in this specification;

[0046] Figure 4 A schematic diagram of an information recall device provided in this manual;

[0047] Figure 5 A method provided in this specification corresponding to Figure 1 and Figure 2 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0049] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0050] Figure 1 A flow chart of a model training method provided in this specification includes the following steps:

[0051] S101: Obtain historical query requests and all historical click records of each historical document.

[0052] S102: Query the historical documents corresponding to the historical query request from a preset document database as the positive sample data corresponding to the historical query request, and determine the historical documents whose click times exceed a preset threshold but are not required by the historical query request from the preset document database based on the full historical click records, as the negative sample data corresponding to the historical query request.

[0053] With the advancement of science and technology, information retrieval technology has become more and more perfect. The widespread popularization of this technology has provided extremely practical convenience for people's daily life, study or work. At the current stage of technology, most information retrieval methods are implemented using pre-trained recall models, but in the current training process of recall models, the construction method used to construct negative samples of sample sets is not reasonable.

[0054] At present, it is generally believed that when users perform information retrieval, they regard documents that are not clicked in the search results as negative samples that are irrelevant to the search terms. However, this is not necessarily true. It may be due to problems such as display order or content occlusion that prevent users from directly seeing such documents. When the recall model trained based on such negative samples is officially used, the retrieval accuracy is poor, and the retrieved results are significantly different from the user's expectations, which in turn affects the user experience. Therefore, how to train a recall model with high retrieval accuracy is an extremely important issue.

[0055] To this end, this specification provides a model training method and an information recall method, wherein the execution subject used by the method provided in this specification can be a server, or a terminal device such as a desktop computer or a laptop computer. In addition, the execution subject of this specification can also be a subject in the form of software, such as a client installed in a terminal device. For the sake of ease of explanation, this specification below only uses the server as the execution subject to explain a model training method and an information recall method provided.

[0056] Based on this, a server that applies a model training method and an information recall method provided in this specification can determine the positive sample data and negative sample data corresponding to the historical query request based on the historical query request and the full historical click records of each historical document. Then the server can construct a sample set based on the positive and negative sample data, and perform model training optimization on the recall model to be trained. The server can use the trained recall model to accurately query the corresponding target document according to the query request of the target user, and return it to the target user.

[0057] The application scenario of the trained recall model can be determined according to actual application needs. For example, in the field of e-commerce, the server can obtain the query request input or clicked by the target user, and input the query request into the recall model trained by the method in this specification, so that the recall model can determine the recall result that accurately meets the user's expectations according to the query request. This effectively ensures the efficiency of information query, meets the actual needs of users to the greatest extent, and improves the user experience.

[0058] For another example, in the field of intelligent question answering, the server can input the query request corresponding to the target user's inquiry question into the recall model trained by the method in this specification, so that the recall model can recall the document content or related materials that best match the query request from the database according to the query request, and answer the question based on the matching results. While improving the execution efficiency of intelligent question answering tasks, accurate and correct matching results can also avoid the situation of answering questions that are not asked to a certain extent, effectively solving users' actual problems and optimizing user experience.

[0059] The method provided in this specification is mainly divided into two stages, one is the model training stage, and the other is the actual application stage. In the model training stage, the server can obtain the pre-stored historical query requests and the full historical click records of each historical document in the document database.

[0060] Then, the server can query and determine the historical documents corresponding to the acquired historical query requests from the preset document database as the positive sample data corresponding to the historical query requests. At the same time, the server can also determine the historical documents whose click counts exceed the preset number threshold but do not correspond to the historical query requests from the preset document database based on the acquired full historical click records as the negative sample data corresponding to the historical query requests.

[0061] Specifically, the server can determine from the full amount of historical click records which historical document or documents the user clicked on based on the search results after inputting the historical query request in the historical process, and find the corresponding historical document from the above-mentioned preset document database as the positive sample data corresponding to the historical query request. The above-mentioned positive sample data can be understood as follows: the historical documents in the positive sample data are all historical documents that the user actively clicked on in the query results fed back to the user when the historical query request was initiated during the historical query process.

[0062] Regarding negative sample data, the server may first randomly select a certain amount of historical document data from a preset document database, and then determine historical documents from the preset document library whose click counts exceed a preset threshold but are not corresponding to historical query requests based on the acquired full historical click records. The above two types of historical document data, i.e., historical document data randomly selected from the document database and historical document data irrelevant to historical query requests determined based on the full historical click records, are combined according to a preset negative sample composition ratio to obtain negative sample data corresponding to historical query requests.

[0063] Among them, the specific process of determining the historical document data that is not related to the historical query request based on the full amount of historical click records is not strictly limited in this specification. Since the constructed positive sample data and negative sample data need to be used in the subsequent recall model training process, the sample sets corresponding to multiple historical query requests may be used in the overall model training process. After the positive sample data corresponding to each historical query request is determined through the above process, for each historical query request, the server can determine the sample data with a click count exceeding a preset threshold from the positive sample data corresponding to other historical query requests except the historical query request based on the acquired full amount of historical records, that is, the historical document corresponding to its historical query request, as the negative sample data corresponding to the historical query request.

[0064] In order to facilitate understanding of the construction process of the above-mentioned negative sample data, an example is given below. In this specification, when the number of historical query requests obtained is relatively large, the server can divide multiple historical query requests into multiple training batches (batches) according to the number of requests in the preset batches, so as to rationally plan the training process of the subsequent recall model. Suppose a training batch contains five historical query requests with irrelevant content: Q1, Q2, Q3, Q4, and Q5. Through the above steps, the server determines that the positive sample data corresponding to these five historical query requests in the preset document database are: Q1-D1, Q2-D2, Q3-D3, Q4-D4, and Q5-D5.

[0065] When determining the corresponding negative sample data for the historical query request Q1, the server can directly use the historical documents in the positive sample data corresponding to other historical query data in the same training batch as the negative sample data corresponding to the historical query request Q1, that is, the negative sample data corresponding to the historical query request Q1 can include: D2, D3, D4, D5. The same is true for the negative sample data corresponding to other historical query requests Q2, Q3, Q4, and Q5, so they are not listed one by one. At this point, the server can determine the positive sample data and negative sample data corresponding to each historical query request in the training batch. Based on the above method, the server can also randomly select a number of historical document data from a preset document database, and combine them with the historical documents in the positive sample data corresponding to other historical query requests as the negative sample data corresponding to each historical query request.

[0066] It should be noted that the preset number of times threshold mentioned in the above process is mainly used to evaluate the corresponding click heat of historical documents in the historical query process. The specific value size and range are not strictly limited in this specification. The preset number of times threshold can be directly set to zero. Of course, considering that the negative sample data constructed by the steps here in the subsequent process reduces the recall model's attention to the click heat of historical documents, the preset number of times threshold can also be flexibly increased according to the application scenario and needs.

[0067] In addition, regarding the proportion of negative samples mentioned in the above method, the specific numerical size and range are still not strictly limited in this specification. It can be that the proportion of historical document data randomly selected from the document database and historical document data irrelevant to historical query requests determined based on the full historical click records is 9:1. Of course, it can also be flexibly adjusted according to the actual application scenario and data volume.

[0068] In addition, in this specification, whether it is the historical documents, the preset document database, or the document data in the positive and negative sample data mentioned in the above process, it is not strictly limited to article data represented in text form, but can also be retrieval data of any representation type such as pictures, videos, web pages, product links, etc. Just for the sake of ease of understanding, this specification uses "document" to summarize retrieval data of all data types.

[0069] S103: Construct a sample set according to the positive sample data and the negative sample data.

[0070] S104: Input each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to the sample data for each sample data according to the sample data and the historical click record corresponding to the sample data, and determines the matching result corresponding to the sample data and the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the determined matching label corresponding to the sample data.

[0071] In this specification, the server may construct a sample set corresponding to the historical query request based on the positive sample data and negative sample data corresponding to the historical query request determined through the above steps.

[0072] Specifically, the server can combine the positive sample data and negative sample data corresponding to the historical query request into a sample set according to the corresponding proportions according to the preset sample set composition ratio. It should be noted that the specific proportion distribution in the preset sample set composition ratio is not strictly limited in this specification and can be flexibly set according to the actual application scenario and data volume. In order to ensure that the recall accuracy and training efficiency of the subsequent recall model are improved, the above sample set can be constructed with the proportion of negative sample data much greater than the proportion of positive samples. For example, the ratio of positive sample data to negative sample data in the sample set can be 1:14. Of course, it can also be appropriately adjusted according to the actual application scenario and the amount of sample data.

[0073] After constructing the sample set, the server can input the sample set and the historical click record data corresponding to each sample data in the sample set into the recall model to be trained, so that the recall model can determine the feature data corresponding to each sample data in the sample set, that is, each historical document, based on the sample data and the historical click record data corresponding to the sample data. And based on the feature data, determine the matching result corresponding to the sample data and the historical query request, and then determine the loss value of the recall model for the sample data based on the deviation between the matching result and the matching label corresponding to the sample data. Among them, the loss value is positively correlated with the deviation between the above matching result and the matching label corresponding to the sample data.

[0074] It should be noted that the reason why historical documents with a certain number of clicks are added in the process of constructing negative sample data in this manual is to avoid the recall model paying too much attention to the connection between the click popularity of the historical documents themselves and the positive samples during the training process here, thereby ensuring that the recall model can be more focused on the retrieval and matching of the query request content in actual applications, rather than excessively considering the click popularity of the document itself in the matching results.

[0075] In the current technology, due to the lack of comprehensive consideration when constructing negative sample data, the recall model is easily affected by the click heat of the sample document itself during the training process, so that the recall model finally trained will also include the click heat of the document itself in the retrieval factor when it is actually used. The corresponding phenomenon may be that when the recall model retrieves and recalls documents according to the user's query request, it only matches the content from the document data with high click heat, while documents with low click heat but high content matching may be considered as not matching the query request. The emergence of this situation affects the exposure and display of documents with low click heat, and also makes users unable to obtain document data that truly meets their actual needs. While affecting the user experience, it also makes users spend more time to retrieve the documents they really need, seriously reducing the execution efficiency of related people.

[0076] In this specification, by using historical documents with a certain click heat and irrelevant to the content of historical query requests as negative sample data corresponding to historical query requests, the click heat problem mentioned in the above content can be effectively solved. During the training process of the above recall model, the historical documents in the positive sample data in the sample set are all historical documents with a certain click heat, and there are also historical documents with a certain click heat in the negative sample data. In the case that both the positive sample data and the negative sample data have a certain click heat, the recall model does not need to consider whether there is a correlation between the click heat of the document itself and the positive sample data during the training process, so as to focus more on whether the historical query content matches the content of the historical documents in the sample set.

[0077] The feature data corresponding to each sample data mentioned in the above matching process includes not only the data content features of the corresponding historical documents, but also the click heat features determined based on the historical click records of the sample data. In the above steps, the click heat features of each sample data in the negative sample data are not completely low click heat, but are similar to the click heat features of each sample data in the positive sample data, which makes the click heat features corresponding to each sample data in the overall sample set more average, effectively reducing the recall model's attention to click heat during the training process.

[0078] In addition, this specification takes into account that there may be problems with click heat in positive sample data. The content of a document with high click heat may not be highly matched with the query request, but because of the influence of high click heat, it is clicked by the user, resulting in false data records, which affects the training process of the recall model. Therefore, the server can perform weighted processing based on click heat for the positive sample data in the sample data.

[0079] Specifically, for each sample data in the sample set, the server can determine the weighted value corresponding to the sample data according to the historical click records corresponding to the sample data when determining that the sample data is positive sample data. The weighted value is used to indicate the influence of the click heat of the sample data on the degree of matching between the sample data and the historical query request. If it is determined according to the historical click records corresponding to the sample data that the historical document corresponding to the sample data is clicked more and more frequently during the historical query process, it means that the click heat of the sample data is high, and then it is necessary to eliminate the influence of the high click heat on the matching degree, which is specifically expressed as the smaller the above weighted value will be.

[0080] Then, the server can weight the original matching label corresponding to the sample data according to the determined weighted value, so as to obtain the weighted matching label corresponding to the sample label. For example, suppose that, based on the historical click record corresponding to a certain sample data of the positive sample data in the sample set, the server determines that the number of clicks on the sample data in the historical query process is the largest among all the positive sample data, which means that the click heat is extremely high. The server can weight the matching label of the sample data according to the preset weighted value ratio. Assuming that the original matching label of the sample data is 1, which specifically indicates that the sample data completely matches the historical query request, and the weighted value determined according to the above process is 0.8, the weighted matching label of the sample data can be 1*0.8, that is, 0.8.

[0081] Similarly, when it is determined based on the historical click records corresponding to a certain sample data that the number of clicks on the sample data during the historical query process is the least among all the positive sample data, it means that the click heat is low. The server can also weight the matching label of the sample data at a preset weighting ratio, and increase the weight of the original matching label, for example, from 1 to 1.2. The method of applying weighted values ​​based on historical click records in the above content can effectively reduce the impact of click heat on the matching degree between sample data and historical query requests.

[0082] In addition to the above-mentioned weighted processing of the matching labels of each sample data in the positive sample data, in this specification, the server can also perform similar weighted processing on the sample data in the negative sample data, thereby adjusting the impact of the click popularity of the sample data in the negative sample data on the model training process.

[0083] Specifically, for each sample data in the sample set, when the server determines that the sample data belongs to negative sample data, it can determine the initial weighted value corresponding to the sample data according to the historical click record corresponding to the sample data. Different from the above method, when it is determined according to the historical click record that the sample data has been clicked more times in the historical query process, that is, the click heat corresponding to the sample data is higher, the initial weighted value corresponding to the sample data can be larger.

[0084] Then, the server can determine the average number of clicks corresponding to the negative sample data according to the historical click records corresponding to each sample data in the negative sample data. Next, the server can adjust the initial weighted value of each sample data in the negative sample data according to the average number of clicks. Specifically, if the number of clicks corresponding to the sample data is higher than the determined average number of clicks, the server can appropriately reduce the initial weighted value. On the contrary, if the number of clicks corresponding to the sample data is lower than the determined average number of clicks, the server can appropriately increase the initial weighted value.

[0085] Then, the server can perform weighted processing on the original matching label of each sample data according to the adjusted initial weighted value of each sample data in the negative sample data, so as to obtain the weighted matching label corresponding to each sample data in the negative sample data.

[0086] Through the above-mentioned process of weighted processing of sample data in negative sample data, the click heat in negative sample data can be well averaged, effectively avoiding uneven distribution of click heat in negative sample data, resulting in sample data with extremely high or extremely low click heat. While reducing the impact of click heat on recall model training, it also effectively improves model training efficiency and model training accuracy.

[0087] S105: Training the recall model according to the loss value.

[0088] In this specification, the server can perform training optimization on the recall model to be trained according to the loss value determined through the above training process. With the purpose of minimizing the loss value, the model parameters in the recall model are appropriately adjusted to obtain the trained recall model.

[0089] It should be noted that the recall model mentioned in this specification can be a dual-tower model. Based on the special dual-tower structure of the dual-tower model, the server can input the sample data in the sample set and the historical click records corresponding to the sample data into one tower, and input the historical query request into another tower. The dual-tower model can simultaneously extract data features to obtain data features corresponding to the sample data and data features corresponding to the historical query request. The data features corresponding to the sample data include both the data content features corresponding to the data content of the sample data itself and the click heat features corresponding to the click heat of the sample data.

[0090] Through the construction of negative sample data and weighted processing in positive and negative sample data, the recall model can effectively remove the focus on the click popularity of sample data during training, so as to determine the matching results mainly based on the data content characteristics of the sample data and the data characteristics corresponding to the historical query request. In addition, the training optimization process mentioned above can also use the pointwise loss function with faster training speed to optimize the recall model to be trained.

[0091] The trained recall model can be tested online at the same time as the old recall model. If the test results show that the user click rate after the recall task is performed by the trained recall model is higher than that of the old recall model, the trained recall model can be directly replaced with the old recall model to perform the recall task. If the user click rate after the recall task is performed by the trained recall model is not higher or lower than that of the old recall model, it is necessary to adjust the configuration of the model parameters during the recall model training process, as well as the proportion distribution and data source between the samples during the model training process.

[0092] The method provided in this specification is mainly divided into two stages, the model training stage and the actual application stage. The model training stage introduced above is mainly used to obtain a recall model that can accurately recall corresponding document data according to query requests after the above model training process, so that in the actual application stage, the trained recall model can be applied to the server or terminal device used to perform the recall task. In order to facilitate the introduction of the information recall method, the following will be explained with a flow chart of the information recall method, such as Figure 2 shown.

[0093] Figure 2 The following is a flowchart of a method for information recall provided in this specification, comprising the following steps:

[0094] S201: Obtain a query request from a target user.

[0095] At present, the increasing maturity and widespread application of information retrieval technology has brought great convenience to various fields. However, there are obvious defects in the training process of recall models in information retrieval technology at this stage. The construction of negative sample data is too absolute and there is a problem of sample selection bias (SSB). When the trained recall model performs information retrieval tasks, the retrieved results lack accuracy and cannot effectively meet the real needs of users, thus affecting the user's task execution efficiency and usage experience.

[0096] To this end, this specification provides a method for information recall, wherein the execution subject used in the method provided in this specification can be a server, or a terminal device such as a desktop computer or a laptop computer. In addition, the execution subject of this specification can also be a subject in the form of software, such as a client installed in a terminal device. For the sake of convenience, this specification below only uses the server as the execution subject to explain a method for information recall provided.

[0097] Based on this, a server that uses the information recall method provided by this specification can input the user's query request into a pre-trained recall model, so that the recall model accurately matches the corresponding target document according to the query request and returns it to the target user.

[0098] The application scenario of the information recall method can be determined according to actual application needs. For example, in the field of search engines, the server can obtain the query request manually entered or clicked by the target user in the search box, and input the obtained query request into the pre-trained recall model, so that the recall model can accurately retrieve the recall results that match the query request according to the query request and display them to the target user. While effectively ensuring the query efficiency, the highly matched recall results can also meet the user needs to the greatest extent.

[0099] For another example, in the field of data statistics, the server can obtain the query request for data input by the target user, and input the query request into the pre-trained recall model, so that the recall model can query the corresponding recall result from the database according to the query request. The server can display the recall result to the target user, effectively ensuring that the user obtains real matching data, improving data retrieval efficiency, and optimizing user experience.

[0100] In this specification, the server can obtain the query request of the target user. It should be noted that the specific data form of the query request of the target user is not strictly limited in this specification, and the query request can be in text form, or in image form, or even in video form, etc., and can be set and adjusted for different types of query requests according to actual application scenarios.

[0101] S202: Input the query request into a pre-trained recall model, so that the recall model determines the target recall document corresponding to the query request from a preset document database according to the query request, and the recall model is trained by the above-mentioned model training method.

[0102] S203: Return the target recalled document data to the target user.

[0103] In this specification, the server can input the obtained query request into the pre-trained recall model, so that the recall model can determine the target document corresponding to the query request from the preset document database according to the received query request as the target recall document. The pre-training process for the recall model is implemented based on the above-mentioned model training method.

[0104] After determining the target recall document corresponding to the query request, the server may send the target recall document to the terminal device used by the target user, so as to display the target recall document to the target user through the terminal device.

[0105] From the above content, it can be seen that the model training method and information recall method provided in this specification can determine the positive sample data and negative sample data corresponding to the historical query request based on the acquired historical query request and the full amount of historical click records, and perform model training on the recall model to be trained based on the sample set composed of the positive sample data and the negative sample data. The trained recall model can accurately match the target recall document corresponding to the query request from the database according to the query request of the target user. The recall model trained by the method in this specification can effectively ignore the influence of click heat on whether the document is positive sample data, effectively ensure the accuracy of the recall results in the actual information retrieval process, effectively meet the actual needs of users, improve the user's task execution efficiency, and effectively enhance the user experience.

[0106] The above are one or more implementation methods of this specification. Based on the same idea, this specification also provides a corresponding model training device, such as Figure 3 shown.

[0107] Figure 3 A schematic diagram of a model training device provided in this specification, including:

[0108] The acquisition module 301 is used to acquire the historical query request and the full amount of historical click records of each historical document;

[0109] A query module 302 is used to query the historical documents corresponding to the historical query request from a preset document database as positive sample data corresponding to the historical query request, and to determine, based on the full amount of historical click records, historical documents whose click counts exceed a preset number threshold but are not required by the historical query request from the preset document database as negative sample data corresponding to the historical query request;

[0110] A construction module 303 is used to construct a sample set according to the positive sample data and the negative sample data;

[0111] The matching module 304 is used to input each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data for each sample data, and determines the matching result corresponding to the sample data and the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the determined matching label corresponding to the sample data, and the loss value is positively correlated with the deviation;

[0112] The training module 305 is used to train the recall model according to the loss value.

[0113] Optionally, the query module 302 is specifically used to determine historical documents corresponding to other historical query requests from the preset document database as positive sample data corresponding to the other historical query requests; based on the full historical click records, determine sample data whose number of clicks exceeds a preset threshold from the positive sample data corresponding to the other historical query requests as negative sample data corresponding to the historical query requests.

[0114] Optionally, the query module 302 is specifically used to randomly select a number of historical documents from the preset document database, and use the randomly selected number of historical documents and the historical documents determined from the preset document database whose click counts exceed a preset threshold but are not required by the historical query request as negative sample data corresponding to the historical query request.

[0115] Optionally, the matching module 304 is specifically used to, if it is determined that the sample data is positive sample data, determine a weighted value corresponding to the sample data according to the historical click records corresponding to the sample data, wherein if it is determined through the historical click records corresponding to the sample data that the sample data has been clicked more times in history, the weighted value is smaller; according to the weighted value, weight the original matching label corresponding to the sample data to obtain the matching label corresponding to the sample data.

[0116] Optionally, the matching module 304 is specifically used to, if it is determined that the sample data is negative sample data, determine an initial weighted value corresponding to the sample data according to the historical click records corresponding to the sample data, wherein if it is determined through the historical click records corresponding to the sample data that the sample data has been clicked more times in history, the initial weighted value is larger; determine an average number of clicks corresponding to each negative sample data according to the historical click records corresponding to each negative sample data in the sample set; adjust the initial weighted value corresponding to the sample data according to the average number of clicks to obtain an adjusted weighted value corresponding to the sample data; and weight the original matching label corresponding to the sample data according to the adjusted weighted value to obtain a matching label corresponding to the sample data.

[0117] Optionally, the recall model is a double-tower model.

[0118] Based on the same idea, this manual also provides a corresponding information recall device, such as Figure 4 shown.

[0119] Figure 4 This is a schematic diagram of an information recall device provided in this manual, including:

[0120] The acquisition module 401 is used to acquire the query request of the target user;

[0121] A recall module 402 is used to input the query request into a pre-trained recall model, so that the recall model determines the target recall document corresponding to the query request from a preset document database according to the query request, and the recall model is trained by the above-mentioned model training method;

[0122] The returning module 403 is used to return the target recalled document data to the target user.

[0123] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 and Figure 2 Provides model training methods and information recall methods.

[0124] This manual also provides Figure 5 The one shown corresponds to Figure 1 and Figure 2 A schematic diagram of the electronic device. Figure 5 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training method shown or Figure 2 The method of information recall is shown.

[0125] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0126] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0127] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0128] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0129] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0130] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0139] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A model training method, characterized in that: include: Obtain historical query requests and the full historical click records of each historical document; Querying historical documents corresponding to the historical query request from a preset document database as positive sample data corresponding to the historical query request, and determining historical documents whose click counts exceed a preset threshold but are not required by the historical query request from the preset document database based on the full amount of historical click records as negative sample data corresponding to the historical query request; Constructing a sample set according to the positive sample data and the negative sample data; Input each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data for each sample data, determines the matching result corresponding to the sample data and the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the determined matching label corresponding to the sample data, and the loss value is positively correlated with the deviation; The recall model is trained according to the loss value.

2. The method according to claim 1, characterized in that The step of determining, from the preset document database, historical documents whose click counts exceed a preset threshold but are not required by the historical query request according to the full amount of historical click records as negative sample data corresponding to the historical query request specifically includes: Determining historical documents corresponding to other historical query requests from the preset document database as positive sample data corresponding to the other historical query requests; According to the full amount of historical click records, sample data whose click times exceed a preset threshold are determined from each positive sample data corresponding to the other historical query requests as negative sample data corresponding to the historical query requests.

3. The method according to claim 1, characterized in that The step of determining, from the preset document database, historical documents whose click counts exceed a preset threshold but are not required by the historical query request according to the full amount of historical click records as negative sample data corresponding to the historical query request specifically includes: A number of historical documents are randomly selected from the preset document database, and the randomly selected historical documents and historical documents determined from the preset document database whose click counts exceed a preset threshold but are not required by the historical query request are used as negative sample data corresponding to the historical query request.

4. The method according to claim 1, characterized in that The step of determining the matching tag corresponding to the sample data specifically includes: If the sample data is determined to be positive sample data, a weighted value corresponding to the sample data is determined according to the historical click records corresponding to the sample data, wherein if the number of times the sample data has been clicked in history is determined to be greater through the historical click records corresponding to the sample data, the weighted value is smaller; The original matching label corresponding to the sample data is weighted according to the weighted value to obtain the matching label corresponding to the sample data.

5. The method according to claim 1, characterized in that The step of determining the matching tag corresponding to the sample data specifically includes: If it is determined that the sample data is negative sample data, then according to the historical click records corresponding to the sample data, an initial weighted value corresponding to the sample data is determined, wherein if it is determined through the historical click records corresponding to the sample data that the sample data has been clicked more times in history, the initial weighted value is greater; Determine the average number of clicks corresponding to each negative sample data according to the historical click records corresponding to each negative sample data in the sample set; According to the average number of clicks, the initial weighted value corresponding to the sample data is adjusted to obtain an adjusted weighted value corresponding to the sample data; The original matching label corresponding to the sample data is weighted according to the adjusted weighted value to obtain the matching label corresponding to the sample data.

6. The method according to any one of claims 1 to 4, characterized in that: The recall model is a double tower model.

7. A method for information recall, characterized in that: include: Get the query request of the target user; Inputting the query request into a pre-trained recall model, so that the recall model determines the target recall document corresponding to the query request from a preset document database according to the query request, wherein the recall model is trained by the method according to any one of claims 1 to 6; The target recalled document data is returned to the target user.

8. A device for model training, characterized in that: include: The acquisition module is used to obtain historical query requests and the full historical click records of each historical document; A query module, configured to query a preset document database for historical documents corresponding to the historical query request as positive sample data corresponding to the historical query request, and to determine, based on the full amount of historical click records, historical documents whose click counts exceed a preset threshold but are not required by the historical query request from the preset document database as negative sample data corresponding to the historical query request; A construction module, used to construct a sample set according to the positive sample data and the negative sample data; A matching module, used for inputting each sample data contained in the sample set and the historical click record corresponding to each sample data into the recall model to be trained, so that the recall model determines the feature data corresponding to the sample data according to the sample data and the historical click record corresponding to the sample data for each sample data, and determines the matching result corresponding to the sample data and the historical query request according to the feature data, and determines the loss value according to the deviation between the matching result and the determined matching label corresponding to the sample data, and the loss value is positively correlated with the deviation; A training module is used to train the recall model according to the loss value.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images