Recall model optimization methods, devices, equipment and storage media

By performing relevance scoring and generating label information on the ranking results of existing recall models, optimizing sample data, and retraining the model, the problem of low discrimination in recall models was solved, and more efficient retrieval results were achieved.

CN116501950BActive Publication Date: 2026-11-10BEIJING QIHOOD TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210057510.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-18
Publication Date
2026-11-10
Estimated Expiration
2042-01-18

AI Technical Summary

Technical Problem

Existing semantic recall models have low text discrimination capabilities, resulting in poor retrieval performance.

Method used

By obtaining the current ranking results, relevance scoring and label information are generated, sample data is optimized, the recall model is retrained, and the optimized sample data is used to further optimize the recall model.

Benefits of technology

It improves the discrimination and recall rate of the recall model, thereby increasing the accuracy and efficiency of retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116501950B_ABST
    Figure CN116501950B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of artificial intelligence technology and discloses a method, apparatus, device, and storage medium for optimizing a recall model. The method includes: obtaining the current ranking result of a first preset model; performing a relevance score on the current ranking result to obtain label information corresponding to the current ranking result; generating optimized sample data based on the current ranking result and the corresponding label information; and optimizing the recall model to be optimized based on the optimized sample data. Through this method, the existing recall model is optimized. Since ranking results obtained from other models have a certain degree of similarity, by re-evaluating the relevance based on the ranking results, re-labeling the ranking results, and further optimizing the existing recall model, the discriminative power and recall rate of the recall model are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a recall model optimization method, apparatus, device, and storage medium. Background Technology

[0002] The entire search process can be roughly divided into three parts: "recall -> coarse ranking -> fine ranking". The semantic recall model calculates the semantic similarity score between the query and the title, and selects the result with the highest score. This is done to make retrieval more efficient.

[0003] However, the training data for current semantic recall models are all raw data mined directly from web pages. Because the sample data collection spans a very wide range, the similarity and semantic differences between positive and negative samples are very large. Therefore, the recall models trained with this raw data have low text discrimination.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a recall model optimization method, apparatus, device, and storage medium, aiming to solve the technical problem of low text discrimination in existing recall models.

[0006] To achieve the above objectives, the present invention provides a recall model optimization method, the method comprising the following steps:

[0007] Obtain the current ranking result of the first preset model;

[0008] The relevance score is calculated for the current ranking result to obtain the label information corresponding to the current ranking result;

[0009] Optimized sample data is generated based on the current sorting result and the corresponding label information;

[0010] The optimization of the recall model to be optimized is completed based on the optimized sample data.

[0011] Optionally, the search engine address is determined according to a preset address table, wherein the preset address table is the search engine address used for searching using a first preset model;

[0012] The finely ranked data is retrieved from the search engine address using a web crawler;

[0013] The current sorting result is obtained by filtering the finely sorted data.

[0014] Optionally, the relevance scores are calculated for the recommended questions in the current ranking results to obtain relevance scores, wherein the current ranking results contain multiple recommended questions;

[0015] When the relevance score is greater than or equal to a preset score, the corresponding recommended question will be labeled with a positive label;

[0016] When the relevance score is less than a preset score, the corresponding recommendation question will be labeled with a negative label;

[0017] The label information corresponding to the current sorting result is determined based on the positive and negative labels described.

[0018] Optionally, obtain the query question corresponding to the recommendation question;

[0019] Determine the corresponding first target semantic vector based on the query question;

[0020] Determine the corresponding second target semantic vector based on the recommendation question;

[0021] Calculate the similarity between the first target semantic vector and the second target semantic vector;

[0022] The relevance score is determined based on the similarity.

[0023] Optionally, obtain the user log information corresponding to the current sorting result;

[0024] The number of times each of the recommended questions is displayed is determined based on the user log information.

[0025] The relevance score is determined based on the number of times each of the recommended questions is displayed.

[0026] Optionally, the total number of times the current sorting result is displayed can be determined based on the number of displays.

[0027] The target weighting coefficient is determined based on the total number of displays.

[0028] The relevance score for each recommendation question is determined based on the target weighting coefficient and the number of times each question is displayed.

[0029] Optionally, a query sample is determined based on the current ranking result, wherein the current ranking result contains multiple recommendation questions;

[0030] The first positive sample and the first negative sample are determined based on the recommendation questions and label information described above;

[0031] Optimized sample data is generated based on the query sample, the first positive sample, and the first negative sample.

[0032] Optionally, a second positive sample and a second negative sample are obtained, wherein the second positive sample and the second negative sample are positive and negative samples corresponding to sorting results that are not in the same batch as the current sorting result;

[0033] The third negative sample is obtained based on the second positive sample and the second negative sample;

[0034] Optimized sample data is generated based on the query sample, the first positive sample, the first negative sample, and the third negative sample.

[0035] Optionally, training samples and test samples are determined based on the optimized sample data;

[0036] The recall model to be optimized is trained based on the training samples to obtain the recall model to be tested;

[0037] The recall model to be tested is tested based on the test samples to obtain test results;

[0038] When the test result is successful, the optimization of the recall model to be optimized is completed.

[0039] Optionally, the training samples are input into the recall model to be optimized to determine the model representation parameters;

[0040] Calculate the loss value based on the model characterization parameters;

[0041] The recall model to be optimized is adjusted based on the loss value until the model converges, thus obtaining the recall model to be tested.

[0042] Furthermore, to achieve the above objectives, the present invention also proposes a recall model optimization apparatus comprising:

[0043] The acquisition module is used to acquire the current sorting result of the first preset model;

[0044] The processing module is used to perform a relevance score on the current ranking result and obtain the label information corresponding to the current ranking result;

[0045] The processing module is also used to generate optimized sample data based on the current sorting result and the corresponding label information;

[0046] The processing module is also used to optimize the recall model to be optimized based on the optimized sample data.

[0047] Optionally, the acquisition module is further configured to determine the search engine address according to a preset address table, wherein the preset address table is the search engine address used for searching using a first preset model;

[0048] The finely ranked data is retrieved from the search engine address using a web crawler;

[0049] The current sorting result is obtained by filtering the finely sorted data.

[0050] Optionally, the processing module is further configured to perform a relevance score on the recommended questions in the current ranking result to obtain a relevance score, wherein the current ranking result contains multiple recommended questions;

[0051] When the relevance score is greater than or equal to a preset score, the corresponding recommended question will be labeled with a positive label;

[0052] When the relevance score is less than a preset score, the corresponding recommendation question will be labeled with a negative label;

[0053] The label information corresponding to the current sorting result is determined based on the positive and negative labels described.

[0054] Optionally, the processing module is further configured to obtain the query question corresponding to the recommendation question;

[0055] Determine the corresponding first target semantic vector based on the query question;

[0056] Determine the corresponding second target semantic vector based on the recommendation question;

[0057] Calculate the similarity between the first target semantic vector and the second target semantic vector;

[0058] The relevance score is determined based on the similarity.

[0059] Optionally, the processing module is further configured to obtain user log information corresponding to the current sorting result;

[0060] The number of times each of the recommended questions is displayed is determined based on the user log information.

[0061] The relevance score is determined based on the number of times each of the recommended questions is displayed.

[0062] Optionally, the processing module is further configured to determine the total number of times the current sorting result is displayed based on the number of displays;

[0063] The target weighting coefficient is determined based on the total number of displays.

[0064] The relevance score for each recommendation question is determined based on the target weighting coefficient and the number of times each question is displayed.

[0065] Optionally, the processing module is further configured to determine a query sample based on the current ranking result, wherein the current ranking result contains multiple recommendation questions;

[0066] The first positive sample and the first negative sample are determined based on the recommendation questions and label information described above;

[0067] Optimized sample data is generated based on the query sample, the first positive sample, and the first negative sample.

[0068] Optionally, the processing module is further configured to obtain a second positive sample and a second negative sample, wherein the second positive sample and the second negative sample are positive and negative samples that are not in the same batch as the current sorting result;

[0069] The third negative sample is obtained based on the second positive sample and the second negative sample;

[0070] Optimized sample data is generated based on the query sample, the first positive sample, the first negative sample, and the third negative sample.

[0071] Furthermore, to achieve the above objectives, the present invention also proposes a recall model optimization device, which includes: a memory, a processor, and a recall model optimization program stored in the memory and executable on the processor, wherein the recall model optimization program is configured to implement the steps of the recall model optimization method described above.

[0072] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a recall model optimization program, wherein when the recall model optimization program is executed by a processor, it implements the steps of the recall model optimization method as described above.

[0073] This invention obtains the current ranking result of a first preset model; scores the relevance of the current ranking result to obtain the label information corresponding to the current ranking result; generates optimized sample data based on the current ranking result and the corresponding label information; and optimizes the recall model to be optimized based on the optimized sample data. Through the above method, the existing recall model is optimized. Since the ranking results obtained through other models have a certain degree of similarity, by re-evaluating the relevance based on the ranking results, re-labeling the ranking results, and further optimizing the existing recall model, the discrimination and recall rate of the recall model are improved. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the structure of the recall model optimization device for the hardware operating environment involved in the embodiments of the present invention;

[0075] Figure 2 This is a flowchart illustrating the first embodiment of the recall model optimization method of the present invention;

[0076] Figure 3 This is a flowchart illustrating the second embodiment of the recall model optimization method of the present invention;

[0077] Figure 4 This is a structural block diagram of the first embodiment of the recall model optimization device of the present invention.

[0078] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0079] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0080] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure for the recall model optimization of the hardware operating environment involved in the embodiments of the present invention.

[0081] like Figure 1 As shown, the recall model optimization device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0082] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the recall model optimization device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0083] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a recall model optimization program.

[0084] exist Figure 1In the recall model optimization device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the recall model optimization device of the present invention can be set in the recall model optimization device, and the recall model optimization device calls the recall model optimization program stored in the memory 1005 through the processor 1001 and executes the recall model optimization method provided in the embodiment of the present invention.

[0085] This invention provides a recall model optimization method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a recall model optimization method according to the present invention.

[0086] In this embodiment, the recall model optimization method includes the following steps:

[0087] Step S10: Obtain the current sorting result of the first preset model.

[0088] It should be noted that the execution subject of this embodiment is the model optimization system. The model optimization system can be set in a server or in other devices with the same or similar functions as the server. This embodiment does not limit this.

[0089] It should be noted that in the process of applying this embodiment to model optimization, it is necessary to first mine data with a certain degree of similarity to optimize the model. Therefore, the training data for optimization is extracted based on the current ranking result of the first preset model. Since the ranking result comes from the fine ranking scenario, for example, the Top 10 results of the existing search engine are selected for labeling. The data distribution is quite different from the coarse ranking and recall scenarios. Using this data for training will increase the semantic discriminativeness of the model.

[0090] It is understandable that the entire search process can be roughly divided into three parts: "recall -> coarse ranking -> fine ranking". The first preset model is the model that performs the ranking in the fine ranking process. The current ranking result is actually the fine ranking result. Obtaining the fine ranking result only means that the current ranking result comes from the first preset model. It does not limit it to be obtained directly from the first preset model. It can also be the ranking result that has undergone the fine ranking process obtained by directly searching on the search website.

[0091] In this embodiment, the search engine address is determined according to a preset address table, which is the search engine address used for searching using a first preset model; finely ranked data is crawled from the search engine address by a web crawler; and the finely ranked data is filtered to obtain the current ranking result.

[0092] In its specific implementation, this embodiment proposes a preferred method for obtaining ranking results, as follows: First, a preset address table is created for web crawlers or manual data scraping. This is because different search engines may provide different recommended questions when the query questions are the same, and the similarity between the same query question and the recommended questions varies. Therefore, if the similarity is too high, it cannot be effectively used to improve the differentiation process; if the similarity is too low, all recommended questions may be truly relevant, resulting in too few negative samples that meet the requirements, which cannot satisfy the optimization process. Therefore, it is necessary to predetermine a preset address table so that the data can obtain the ranking results given by the corresponding search engine from the addresses that meet the requirements.

[0093] It's important to note that the relationship between a query and recommended questions (titles) is the same as the relationship between a question and similar questions. For example, if you enter the query "My monitor keeps flashing the light and won't turn on" into a search engine, the search engine might provide multiple options, or recommended questions, such as "Monitor light flashing but screen not lit?", "Computer screen not lit, display indicator light flashing", "Xiaomi tablet screen won't turn on but the flashing light is always on", "How to fix the flashing light on my Misi electric water heater_DesignBen Q&A", "Little Bear yogurt maker monitor flashing light and beeping and not working", etc. Each title has a certain degree of relevance to the user's query, allowing users to choose the answer to their desired question.

[0094] The process of generating the preset address table can be as follows: obtaining a set of target addresses; sampling the sorting results of each target address in the set of target addresses to obtain the corresponding sampling sorting results; calculating the relevance index of each sampling sorting result; and determining the preset address table based on each relevance index.

[0095] It is understandable that the target address set is the address of any search engine collected in advance. These addresses are filtered, and suitable addresses are selected and placed into a preset address table. The filtering process involves determining the relevance between the ranking results and the query question among these addresses. The relevance between the ranking results and the query question can be calculated by calculating the average similarity of each recommended question in the ranking results, or the similarity of the question with the lowest similarity. This embodiment does not limit this. The sampling process of the ranking results can be random sampling, that is, randomly inputting the query question into the selected search engine and obtaining the ranking results as samples. This process is repeated to accumulate samples to a certain extent.

[0096] Step S20: Perform a relevance score on the current ranking result to obtain the label information corresponding to the current ranking result.

[0097] It's important to note that relevance scoring is essentially the process of labeling the ranking results. The labels inform the retrieval model being trained whether each recommendation question in the ranking results is a positive or negative sample. For example, the training data can be designed as follows: "Query (query question), positive sample Title+ (recommendation question for positive samples), negative sample Title- (recommendation question for negative samples)". If the labels are 3, 2, 1, and 0, then positive samples are titles labeled with Label1 (label 1) of 3 and 2, while negative samples are titles manually labeled with Label1 and 0. This data distribution is then incorporated into the training data.

[0098] In this ranking, each recommendation question is treated as a title. Each title is tagged based on its relevance score. The tags are used to provide a reference for generating positive or negative samples in subsequent processes.

[0099] In practical implementation, manual labeling can be used to label the ranking results. Although manually labeled data is very time-consuming and labor-intensive, the quality is high. To make the model more similar to human discrimination ability, manual labeling is undoubtedly very accurate. To solve the problem of high time cost of manual labeling, an automatic labeling combined with manual review can be used. Alternatively, titles with low differentiation can be manually labeled, and then data with high differentiation can be labeled automatically as a supplement to the sample.

[0100] Step S30: Generate optimized sample data based on the current sorting result and the corresponding label information.

[0101] It should be noted that the sample data required for optimization can be obtained based on the current sorting results and the labeled information.

[0102] In this embodiment, a query sample is determined based on the current ranking result, which includes multiple recommendation questions; a first positive sample and a first negative sample are determined based on each recommendation question and tag information; and optimized sample data is generated based on the query sample, the first positive sample, and the first negative sample.

[0103] Understandably, determining the query sample based on the current ranking result means identifying the query question corresponding to that ranking result, using the query question as the query sample, and distinguishing positive and negative samples based on label information to determine the first positive sample and the first negative sample. Finally, optimized sample data is generated based on the query sample, the first positive sample, and the first negative sample. The first positive sample and the first negative sample are sample data directly obtained from the current ranking result based on the label information. For example, if all recommended questions retrieved for a current query question are considered as a single batch of data, and each recommended question in this batch has corresponding label information, the positive and negative samples corresponding to the current query question generated based on this batch of label information are the first positive sample and the first negative sample. However, the sample data corresponding to a query question does not necessarily all come from the recommended questions retrieved for the current query question; it may also be samples supplemented from other batches of recommended questions. Therefore, the samples obtained from the same batch of recommended questions as the query question are either the first positive sample or the first negative sample.

[0104] In this embodiment, a second positive sample and a second negative sample are obtained, wherein the second positive sample and the second negative sample are positive samples and negative samples that are not in the same batch as the current sorting results; a third negative sample is obtained based on the second positive sample and the second negative sample; and optimized sample data is generated based on the query sample, the first positive sample, the first negative sample and the third negative sample.

[0105] Understandably, the sample data corresponding to a query question is not necessarily from the same batch as the query question itself. Sample data generated from other batches of recommendation questions serves as the second positive sample and the second negative sample. The third negative sample is a mixture of the first and second negative samples. This is done because a portion of the titles needs to be extracted as supplementary negative samples. By introducing a massive amount of weighted negative samples, the data distribution for both coarse ranking and recall scenarios is incorporated into the training data. Simultaneously, we designed the model's loss function. Common methods like Triplet Loss and MSE Loss can only optimize the model by comparing the good and bad relationships between a few samples. However, the cross-entropy loss function can introduce more negative examples. During training, positive or negative samples from all other queries within the same batch of training data are used as negative samples for the current query question. This achieves sufficient interaction between the current query and a large number of negative samples, ensuring that the distribution of the training data is consistent with that in coarse ranking and recall scenarios. This prevents the model from focusing excessively on the semantic relationships of the top ten results when selecting the top ten recommendation questions for ranking.

[0106] Step S40: Optimize the recall model to be optimized based on the optimized sample data.

[0107] It should be noted that optimizing the recall model to be optimized based on the optimized sample data is essentially a process of retraining the recall model to be optimized. The recall model to be optimized can be a model that has been trained on ordinary data, but due to the nature of its training data, the recall model's discrimination of the recommendation question (title) is not strong. Therefore, the recall model to be optimized can be retrained based on the optimized sample data to achieve the purpose of optimization.

[0108] In this embodiment, training samples and test samples are determined based on the optimized sample data; the recall model to be optimized is trained using the training samples to obtain a recall model to be tested; the recall model to be tested is tested using the test samples to obtain test results; when the test result is successful, the optimization of the recall model to be optimized is completed. For example, approximately 100,000 optimized sample data are extracted from the optimized sample data for testing, and the remainder is used for model training.

[0109] In this embodiment, the training samples are input into the recall model to be optimized to determine the model representation parameters; the loss value is calculated based on the model representation parameters; the recall model to be optimized is adjusted based on the loss value until the model converges, thus obtaining the recall model to be tested.

[0110] In practical implementation, the model representation parameters are the training parameters needed to calculate the loss value. Different loss functions result in different model representation parameters. This embodiment does not limit the type of loss function. This embodiment proposes an implementation method, for example: the loss function of the recall model to be optimized in this scheme has been designed. Common Triplet Loss and MSE Loss can only optimize the model by comparing the good and bad relationships between a few samples. However, the cross-entropy loss function can introduce more negative examples. During training, the titles of all other samples in the same batch are used as negative examples, realizing sufficient interaction between the current query and a large number of negative samples. Therefore, the loss value can be calculated using the cross-entropy loss function, and the recall model to be optimized is adjusted according to the loss value until the model converges, resulting in the recall model to be tested.

[0111] This embodiment obtains the current ranking result of the first preset model; performs a relevance score on the current ranking result to obtain the label information corresponding to the current ranking result; generates optimized sample data based on the current ranking result and the corresponding label information; and completes the optimization of the recall model to be optimized based on the optimized sample data. Through the above method, the existing recall model is optimized. Since the ranking results obtained through other models have a certain degree of similarity, by re-evaluating the relevance based on the ranking results, re-labeling the ranking results, and further optimizing the existing recall model, the discrimination and recall rate of the recall model are improved.

[0112] refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of a recall model optimization method according to the present invention.

[0113] Based on the first embodiment described above, the recall model optimization method of this embodiment further includes, in step S20:

[0114] Step S21: Assess the relevance of the recommended questions in the current ranking result to obtain a relevance score. The current ranking result contains multiple recommended questions.

[0115] It should be noted that the current ranking result is the ranking result currently being processed. The process of scoring each recommendation question in the current ranking result can be done manually. Although manually labeled data is very time-consuming and labor-intensive, the quality is high, and the scores obtained by manual scoring are often the best sample data. Alternatively, the scores can be obtained based on the number of times the recommendation question is displayed, the click-through rate, and other similarity calculation methods. This embodiment does not limit this approach.

[0116] In this embodiment, the query question corresponding to the recommendation question is obtained; a first target semantic vector is determined based on the query question; a second target semantic vector is determined based on the recommendation question; the similarity between the first target semantic vector and the second target semantic vector is calculated; and a relevance score is determined based on the similarity.

[0117] Understandably, semantic vectors for the query question and the recommendation question can be extracted separately based on the semantic vector model. The first target semantic vector is the semantic vector for the query question, and the second target semantic vector is the semantic vector for the recommendation question. The similarity between the two vectors can be obtained by calculating the cosine of the angle between them. Then, a relevance score is given based on the similarity. For example, the relevance is 1 if the similarity is greater than a preset similarity threshold, and 0 otherwise.

[0118] In this embodiment, user log information corresponding to the current ranking result is obtained; the number of times each of the recommended questions is displayed is determined based on the user log information; and a relevance score is determined based on the number of times each of the recommended questions is displayed.

[0119] It should be noted that relevance can also be determined based on the number of times the recommended question is displayed. The number of times a recommended question is displayed is the number of times a user selects to display the recommended question when the user makes a search. This can be obtained from user log information. The more times a recommended question is displayed, the higher the degree of relevance the user believes that the recommended question is related to the query question. Therefore, the degree of relevance can be determined based on the number of times it is displayed, and recommended questions with low relevance can be directly added to the optimization sample data as negative samples.

[0120] In this embodiment, the total number of times the current ranking result is displayed is determined based on the number of times it is displayed; a target weighting coefficient is determined based on the total number of times it is displayed; and a relevance score corresponding to each of the recommended questions is determined based on the target weighting coefficient and the number of times each is displayed.

[0121] Understandably, popular questions will have several orders of magnitude more impressions than less popular ones. Therefore, using a constant number of impressions to score the impressions of recommended questions for different queries becomes highly inaccurate due to the popularity of the questions. Thus, a weighting coefficient should be determined based on the total number of impressions for each query, proportionally increasing or decreasing the impressions of each recommended question. Recommended questions with impressions below a threshold are then treated as negative samples. Specifically, the total number of impressions for the current ranking is the sum of the impressions of all recommended questions in the current ranking. For example, if the current ranking consists of the top 10 recommended questions, then the total number of impressions is the sum of the impressions of these 10 recommended questions.

[0122] Step S22: When the relevance score is greater than or equal to the preset score, the corresponding recommended question is labeled with a positive label.

[0123] It should be noted that when the relevance score is greater than or equal to the preset score, it indicates that the query question and the recommendation question are more related. Therefore, the corresponding recommendation question should be labeled with a positive label, thus determining that this recommendation question will be used as a positive sample to optimize the model.

[0124] Step S23: When the relevance score is less than the preset score, the corresponding recommended question is labeled with a negative label.

[0125] It should be noted that when the relevance score is less than the preset score, it indicates that the relevance between the query question and the recommendation question is very low. Therefore, the corresponding recommendation question should be labeled negatively, thus determining that this recommendation question will be used as a negative sample to optimize the model.

[0126] Step S24: Determine the label information corresponding to the current sorting result based on the positive and negative labels.

[0127] It is understandable that the label information corresponding to the current ranking result is determined based on the positive and negative labels. Here, the ranking result recommended by a query question contains multiple recommendation questions, and each recommendation question corresponds to a label information. Based on the label information, it can be determined whether the recommendation question should be divided into positive or negative samples for this query question when generating optimization sample data.

[0128] This embodiment scores the relevance of recommended questions in the current ranking result, obtaining a relevance score. The current ranking result contains multiple recommended questions. When the relevance score is greater than or equal to a preset score, the corresponding recommended question is labeled with a positive label; when the relevance score is less than the preset score, the corresponding recommended question is labeled with a negative label. The label information corresponding to the current ranking result is determined based on the positive and negative labels. This method achieves the acquisition of label information. Recommended questions are labeled with positive or negative labels through manual or automatic labeling. Since the recommended questions originate from the ranking results of the search engine, the positive and negative labels obtained based on the overall high similarity of the recommended questions are conducive to generating high-quality sample data, further improving the discriminative power of the recall model to be optimized.

[0129] Furthermore, this embodiment of the invention also proposes a storage medium storing a recall model optimization program, which, when executed by a processor, implements the steps of the recall model optimization method described above.

[0130] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the recall model optimization device of the present invention.

[0131] like Figure 4 As shown, the recall model optimization device proposed in this embodiment of the invention includes:

[0132] The acquisition module 10 is used to acquire the current sorting result of the first preset model;

[0133] Processing module 20 is used to perform relevance scoring on the current ranking result and obtain the label information corresponding to the current ranking result;

[0134] The processing module 20 is also used to generate optimized sample data based on the current sorting result and the corresponding label information;

[0135] The processing module 20 is also used to optimize the recall model to be optimized based on the optimized sample data.

[0136] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0137] In this embodiment, the acquisition module 10 acquires the current ranking result of the first preset model; the processing module 20 performs a relevance score on the current ranking result to obtain the label information corresponding to the current ranking result; the processing module 20 generates optimized sample data based on the current ranking result and the corresponding label information; and the processing module 20 completes the optimization of the recall model to be optimized based on the optimized sample data. Through the above method, the existing recall model is optimized. Since the ranking results obtained through other models have a certain degree of similarity, by re-evaluating the relevance based on the ranking results, re-labeling the ranking results, and further optimizing the existing recall model, the discrimination and recall rate of the recall model are improved.

[0138] In one embodiment, the acquisition module 10 is further configured to determine the search engine address according to a preset address table, wherein the preset address table is the search engine address used for searching using a first preset model;

[0139] The finely ranked data is retrieved from the search engine address using a web crawler;

[0140] The current sorting result is obtained by filtering the finely sorted data.

[0141] In one embodiment, the processing module 20 is further configured to perform a relevance score on the recommended questions in the current ranking result to obtain a relevance score, wherein the current ranking result contains multiple recommended questions;

[0142] When the relevance score is greater than or equal to a preset score, the corresponding recommended question will be labeled with a positive label;

[0143] When the relevance score is less than a preset score, the corresponding recommendation question will be labeled with a negative label;

[0144] The label information corresponding to the current sorting result is determined based on the positive and negative labels described.

[0145] In one embodiment, the processing module 20 is further configured to obtain the query question corresponding to the recommendation question;

[0146] Determine the corresponding first target semantic vector based on the query question;

[0147] Determine the corresponding second target semantic vector based on the recommendation question;

[0148] Calculate the similarity between the first target semantic vector and the second target semantic vector;

[0149] The relevance score is determined based on the similarity.

[0150] In one embodiment, the processing module 20 is further configured to obtain user log information corresponding to the current sorting result;

[0151] The number of times each of the recommended questions is displayed is determined based on the user log information.

[0152] The relevance score is determined based on the number of times each of the recommended questions is displayed.

[0153] In one embodiment, the processing module 20 is further configured to determine the total number of times the current sorting result is displayed based on the number of displays;

[0154] The target weighting coefficient is determined based on the total number of displays.

[0155] The relevance score for each recommendation question is determined based on the target weighting coefficient and the number of times each question is displayed.

[0156] In one embodiment, the processing module 20 is further configured to determine a query sample based on the current ranking result, wherein the current ranking result contains multiple recommendation questions;

[0157] The first positive sample and the first negative sample are determined based on the recommendation questions and label information described above;

[0158] Optimized sample data is generated based on the query sample, the first positive sample, and the first negative sample.

[0159] In one embodiment, the processing module 20 is further configured to obtain a second positive sample and a second negative sample, wherein the second positive sample and the second negative sample are positive and negative samples corresponding to sorting results that are not in the same batch as the current sorting result;

[0160] The third negative sample is obtained based on the second positive sample and the second negative sample;

[0161] Optimized sample data is generated based on the query sample, the first positive sample, the first negative sample, and the third negative sample.

[0162] In one embodiment, the processing module 20 is further configured to determine training samples and test samples based on the optimized sample data;

[0163] The recall model to be optimized is trained based on the training samples to obtain the recall model to be tested;

[0164] The recall model to be tested is tested based on the test samples to obtain test results;

[0165] When the test result is successful, the optimization of the recall model to be optimized is completed.

[0166] In one embodiment, the processing module 20 is further configured to input the training samples into the recall model to be optimized to determine the model representation parameters;

[0167] Calculate the loss value based on the model characterization parameters;

[0168] The recall model to be optimized is adjusted based on the loss value until the model converges, thus obtaining the recall model to be tested.

[0169] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0170] In addition, for technical details not described in detail in this embodiment, please refer to the recall model optimization method provided in any embodiment of the present invention, which will not be repeated here.

[0171] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0172] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0174] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A recall model optimization method, characterized in that, The recall model optimization method includes: Obtain the current ranking result of the first preset model, where the first preset model is the model that is ranked in the fine ranking process during the search process, and the current ranking result is the fine ranking result; The relevance score is calculated for the current ranking result to obtain the label information corresponding to the current ranking result; Optimized sample data is generated based on the current sorting result and the corresponding label information; The optimization of the recall model to be optimized is completed based on the optimized sample data; The step of generating optimized sample data based on the current sorting result and the corresponding label information includes: The query sample is determined based on the current ranking result, which contains multiple recommendation questions. The first positive sample and the first negative sample are determined based on the recommendation questions and the label information. Obtain the second positive sample and the second negative sample, where the second positive sample and the second negative sample are positive and negative samples that are not in the same batch as the current sorting result; A third negative sample is obtained based on the second positive sample and the second negative sample; The optimized sample data is generated based on the query sample, the first positive sample, the first negative sample, and the third negative sample.

2. The method as described in claim 1, characterized in that, The step of obtaining the current ranking result of the first preset model includes: The search engine address is determined according to a preset address table, wherein the preset address table is the search engine address used for searching using a first preset model; The finely ranked data is retrieved from the search engine address using a web crawler; The current sorting result is obtained by filtering the finely sorted data.

3. The method as described in claim 1, characterized in that, The step of performing a relevance score on the current ranking result to obtain the label information corresponding to the current ranking result includes: The relevance scores are calculated for the recommended questions in the current ranking results, and the current ranking results contain multiple recommended questions. When the relevance score is greater than or equal to a preset score, the corresponding recommended question will be labeled with a positive label; When the relevance score is less than a preset score, the corresponding recommendation question will be labeled with a negative label; The label information corresponding to the current sorting result is determined based on the positive and negative labels described.

4. The method as described in claim 3, characterized in that, The process of scoring the relevance of the recommended questions in the current ranking results to obtain a relevance score includes: Obtain the query question corresponding to the recommendation question; Determine the corresponding first target semantic vector based on the query question; Determine the corresponding second target semantic vector based on the recommendation question; Calculate the similarity between the first target semantic vector and the second target semantic vector; The relevance score is determined based on the similarity.

5. The method as described in claim 3, characterized in that, The process of scoring the relevance of the recommended questions in the current ranking results to obtain a relevance score includes: Obtain the user log information corresponding to the current sorting result; The number of times each of the recommended questions is displayed is determined based on the user log information. The relevance score is determined based on the number of times each of the recommended questions is displayed.

6. The method as described in claim 5, characterized in that, The determination of relevance scores based on the number of times each of the recommended questions is displayed includes: The total number of times the current sorting result is displayed is determined based on the number of displays. The target weighting coefficient is determined based on the total number of displays. The relevance score for each recommendation question is determined based on the target weighting coefficient and the number of times each question is displayed.

7. The method as described in claim 1, characterized in that, The step of optimizing the recall model to be optimized based on the optimized sample data includes: The training samples and test samples are determined based on the optimized sample data; The recall model to be optimized is trained based on the training samples to obtain the recall model to be tested; The recall model to be tested is tested based on the test samples to obtain test results; When the test result is successful, the optimization of the recall model to be optimized is completed.

8. The method as described in claim 7, characterized in that, The step of training the recall model to be optimized based on the training samples to obtain the recall model to be tested includes: The training samples are input into the recall model to be optimized to determine the model representation parameters; Calculate the loss value based on the model characterization parameters; The recall model to be optimized is adjusted based on the loss value until the model converges, thus obtaining the recall model to be tested.

9. A recall model optimization device, characterized in that, The recall model optimization device includes: The acquisition module is used to acquire the current ranking result of the first preset model, where the first preset model is a model that is ranked in the fine ranking process in the search process, and the current ranking result is the fine ranking result. The processing module is used to perform a relevance score on the current ranking result and obtain the label information corresponding to the current ranking result; The processing module is also used to generate optimized sample data based on the current sorting result and the corresponding label information; The processing module is also used to optimize the recall model to be optimized based on the optimized sample data; The processing module is further configured to: determine a query sample based on the current ranking result, wherein the current ranking result contains multiple recommendation questions; determine a first positive sample and a first negative sample based on each recommendation question and the tag information; obtain a second positive sample and a second negative sample, wherein the second positive sample and the second negative sample are positive and negative samples corresponding to ranking results that are not in the same batch as the current ranking result; obtain a third negative sample based on the second positive sample and the second negative sample; and generate the optimized sample data based on the query sample, the first positive sample, the first negative sample, and the third negative sample.

10. The apparatus as claimed in claim 9, characterized in that, The acquisition module is further configured to determine the search engine address according to a preset address table, wherein the preset address table is the search engine address used for searching using a first preset model; The finely ranked data is retrieved from the search engine address using a web crawler; The current sorting result is obtained by filtering the finely sorted data.

11. The apparatus as claimed in claim 9, characterized in that, The processing module is also used to score the relevance of the recommended questions in the current ranking result to obtain a relevance score, wherein the current ranking result contains multiple recommended questions; When the relevance score is greater than or equal to a preset score, the corresponding recommended question will be labeled with a positive label; When the relevance score is less than a preset score, the corresponding recommendation question will be labeled with a negative label; The label information corresponding to the current sorting result is determined based on the positive and negative labels described.

12. The apparatus as claimed in claim 11, characterized in that, The processing module is also used to obtain the query question corresponding to the recommendation question; Determine the corresponding first target semantic vector based on the query question; Determine the corresponding second target semantic vector based on the recommendation question; Calculate the similarity between the first target semantic vector and the second target semantic vector; The relevance score is determined based on the similarity.

13. The apparatus as claimed in claim 11, characterized in that, The processing module is also used to obtain user log information corresponding to the current sorting result; The number of times each of the recommended questions is displayed is determined based on the user log information. The relevance score is determined based on the number of times each of the recommended questions is displayed.

14. The apparatus as claimed in claim 13, characterized in that, The processing module is further configured to determine the total number of times the current sorting result is displayed based on the number of displays; The target weighting coefficient is determined based on the total number of displays. The relevance score for each recommendation question is determined based on the target weighting coefficient and the number of times each question is displayed.

15. A recall model optimization device, characterized in that, The device includes: a memory, a processor, and a recall model optimization program stored in the memory and executable on the processor, the recall model optimization program being configured to implement the steps of the recall model optimization method as described in any one of claims 1 to 8.

16. A storage medium, characterized in that, The storage medium stores a recall model optimization program, which, when executed by a processor, implements the steps of the recall model optimization method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Semantic matching method and system, equipment and storage medium

    CN112287085A

  • Model training method and device

    CN113344078A