Method and apparatus for verifying search recall quality, computer device
By establishing use case files during information search and conducting two rounds of search comparison, the problem of difficulty in verifying the quality of recall results was solved, search efficiency and the accuracy of recall information were improved, and the reliability of recall quality was ensured.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU LANGDONG NET TEC CO LTD
- Filing Date
- 2022-07-28
- Publication Date
- 2026-05-19
AI Technical Summary
In information search scenarios, existing technologies struggle to quantify and evaluate the quality of recall results, especially with massive amounts of search terms and diverse testing scenarios, making it difficult to verify the recall results.
By establishing use case files, two rounds of searches are conducted based on search dimensions and target search terms to obtain first and second recall information. The recall quality is evaluated through comparison rules, including the order and quantity of recall results and the overlap rate of hit dimensions, and validation results are generated.
It improved search efficiency and the ability to organize recall information, ensured the consistency of search dimensions and target search terms, and achieved accurate verification of recall quality.
Smart Images
Figure CN115203000B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of information retrieval technology, and in particular to a method, apparatus, and computer device for verifying the quality of search recall. Background Technology
[0002] With the development of information technology, the internet has broken through the limitations of time and space, gradually becoming an important channel for information search. Driven by technologies such as big data, artificial intelligence, and social networks, information search technology has been greatly improved, and the search scope has become wider. However, information overload has also increased the difficulty of information discernment, making it more difficult to find accurate and comprehensive information. Taking Elasticsearch (hereinafter referred to as ES) as an example, ES is a real-time distributed storage, search, and analysis engine with powerful data retrieval capabilities. ES can use a user's search terms as query keywords, searching all dimensions of the keywords for content that matches the query keywords. It then calculates the relevance score by multiplying the matching score by the weight of that dimension and summing the results. Internally, ES sorts the results by relevance and returns the information with the highest scores.
[0003] However, in real-world search scenarios, users' search terms are not only wide-ranging and numerous, but search engines also store a vast amount of information, making it difficult to quantify and evaluate the retrieved results. Furthermore, the diverse testing scenarios and the difficulty in covering a massive number of search terms make it challenging to verify the quality of the retrieved results. Summary of the Invention
[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, storage medium, and computer program product for verifying the quality of search and recall in response to the above-mentioned technical problems.
[0005] Firstly, this disclosure provides a method for verifying search recall quality. The method includes:
[0006] The target search terms and search dimensions are determined based on historical search information, and a use case file corresponding to the search dimensions is created based on the search dimensions. The use case file includes the target search terms corresponding to the search dimensions.
[0007] A first round of search is performed based on the target search terms in the use case file to obtain the first recall information;
[0008] According to a preset frequency, a second round of search is performed based on the target search terms of the first round of search to obtain second recall information;
[0009] By comparing the first recall information with the second recall information, the verification result of the recall quality is obtained based on the comparison result.
[0010] In one embodiment, the step of determining the target search term and search dimension based on historical search information, and establishing a use case file corresponding to the search dimension, includes:
[0011] The historical search information is retrieved from the search engine logs at a preset period, and the target search terms are extracted based on the historical search information.
[0012] Perform part-of-speech tagging on the target search term to obtain the search dimension of the target search term;
[0013] Create use case files corresponding to the search dimensions.
[0014] In one embodiment, the first round of searching based on the target search terms in the use case file to obtain the first recall information includes:
[0015] Read the target search terms from the use case file corresponding to the search dimension, and send search requests in batches based on the target search terms;
[0016] The search request results are received to obtain the first recall information.
[0017] In one embodiment, the comparison of the first recall information and the second recall information includes:
[0018] The first recall information and the second recall information are compared according to the comparison rules, which include the comparison of the order of recall results, the comparison of the number of recall results, and the comparison of the overlap rate between the hit dimension and the search dimension of the recall results.
[0019] In one embodiment, obtaining the verification result of recall quality based on the comparison result includes:
[0020] The sub-scores and total scores of the comparison results are obtained according to the comparison results for each of the comparison rules. The verification results include the sub-scores and the total score.
[0021] If the sub-score or the total score does not meet the preset threshold, the verification result includes alarm information.
[0022] In one embodiment, the method includes:
[0023] The first recall information is stored in the first storage space, and the second recall information is stored in the second storage space.
[0024] Secondly, this disclosure also provides an apparatus for verifying the quality of search recall. The apparatus includes:
[0025] The use case file module is used to determine the target search terms and search dimensions based on historical search information, and to create a use case file corresponding to the search dimensions. The use case file includes the target search terms corresponding to the search dimensions.
[0026] The first recall module is used to perform a first round of search based on the target search terms in the use case file to obtain the first recall information.
[0027] The second recall module is used to perform a second round of search based on the target search terms of the first round of search according to a preset frequency, and obtain second recall information.
[0028] The recall quality module is used to compare the first recall information and the second recall information, and obtain the verification result of the recall quality based on the comparison result.
[0029] In one embodiment, the use case file module includes:
[0030] The update unit is used to obtain the historical search information from the search engine logs according to a preset period, and extract the target search terms based on the historical search information;
[0031] The part-of-speech tagging unit is used to perform part-of-speech tagging on the target search term and obtain the search dimension of the target search term;
[0032] The file creation unit is used to create use case files corresponding to the search dimensions based on the search dimensions.
[0033] In one embodiment, the first recall module includes:
[0034] The search request unit is used to read the target search terms in the use case file corresponding to the search dimension, and send search requests in batches based on the target search terms;
[0035] The result return unit is used to receive the return results of the search request and obtain the first recall information.
[0036] In one embodiment, the recall quality module includes:
[0037] The comparison unit is used to compare the first recall information and the second recall information according to the comparison rules, which include the comparison of the order of recall results, the comparison of the number of recall results, and the comparison of the overlap rate between the hit dimension and the search dimension of the recall results.
[0038] In one embodiment, the recall quality module further includes:
[0039] The scoring unit is used to obtain the sub-scores and total scores of the comparison results corresponding to each of the comparison rules based on the comparison results. The verification results include the sub-scores and the total scores.
[0040] An alarm unit is used to include alarm information in the verification result when the sub-score or the total score does not meet a preset threshold.
[0041] In one embodiment, the device further includes:
[0042] The storage module is used to store the first recall information in the first storage space and the second recall information in the second storage space.
[0043] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for verifying search recall quality described above.
[0044] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method for verifying search recall quality described above.
[0045] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method for verifying search recall quality described above.
[0046] The methods, apparatus, computer equipment, storage media, and computer program products described above for verifying the quality of search and recall have at least the following beneficial effects:
[0047] This disclosure establishes a use case file containing target search terms under the search dimensions, and performs a first round of search based on the target search terms in the use case file to obtain first recall information. This allows for orderly batch searches, greatly improving search efficiency, and the obtained first recall information is easy to summarize and organize. A second round of search is performed at a preset frequency, using the same target search terms and search dimensions as the first search, to obtain second recall information. The first and second recall information are compared, and the final verification result of the recall quality is obtained based on the comparison result. This ensures the consistency of search dimensions and target search terms between the two searches, making the comparison result of the first and second recall information more orderly and accurate, and more accurately verifying the recall quality. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments or conventional technologies of this disclosure, the accompanying drawings used in the description of the embodiments or conventional technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 This is an application environment diagram of a method for verifying search and recall quality in one embodiment;
[0050] Figure 2 This is a flowchart illustrating a method for verifying search recall quality in one embodiment;
[0051] Figure 3 This is a flowchart illustrating a method for verifying search recall quality in one embodiment;
[0052] Figure 4 This is a flowchart illustrating a method for verifying search recall quality in one embodiment;
[0053] Figure 5 This is a flowchart illustrating a method for verifying search recall quality in one embodiment;
[0054] Figure 6 This is a structural block diagram of an apparatus for verifying search and recall quality in one embodiment;
[0055] Figure 7 This is a structural block diagram of an apparatus for verifying search and recall quality in one embodiment;
[0056] Figure 8 This is a structural block diagram of an apparatus for verifying search and recall quality in one embodiment;
[0057] Figure 9 This is a structural block diagram of an apparatus for verifying search and recall quality in one embodiment;
[0058] Figure 10 This is a structural block diagram of an apparatus for verifying search and recall quality in one embodiment;
[0059] Figure 11 This is a block diagram of the internal structure of a computer device in one embodiment. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.
[0062] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any specific order.
[0063] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, in this specification, the term “and / or” includes any and all combinations of the associated listed items.
[0064] The method for verifying search recall quality provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 is equipped with a search engine, which can send search requests to server 104 and receive the results returned by server 104. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0065] In some embodiments of this disclosure, such as Figure 2 As shown, a method for verifying search recall quality is provided, which is then applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:
[0066] Step S10: Determine the target search terms and search dimensions based on historical search information, and establish a use case file corresponding to the search dimensions, wherein the use case file includes the target search terms corresponding to the search dimensions.
[0067] Specifically, historical search information is obtained, and target search terms and search dimensions are extracted from this information. The identified target search terms and search dimensions are then compiled and organized, and a corresponding use case file is created for each search dimension. The use case file includes the target search terms for that search dimension. Historical search information must at least include the search terms from previous searches. Historical search terms can be organized by frequency, and search terms with a frequency exceeding a set threshold can be selected as target search terms. Alternatively, user-specified search terms can be received via the terminal as target search terms. Search dimensions can refer to the category of search terms, and must be categories that the search engine can recognize and support for searching. For example, when searching for a company, search dimensions could include company name, former name, legal representative, shareholders, unified social credit code, telephone number, email address, official website, address, stocks, bonds, products, trademarks, patents, legal cases, etc.
[0068] Step S20: Perform a first round of search based on the target search terms in the use case file to obtain the first recall information.
[0069] Specifically, the target search terms in the use case file are read, a first-round search is performed using a search engine, and the first recall information is obtained. The first recall information may include the recall results corresponding to the target search terms, the number of recalls, and the recall order. Furthermore, based on the recall results and the search dimensions corresponding to the target search terms, the overlap rate between the hit dimension and the search dimension of the recall results can be obtained. For example, when searching for a company as the target search term and the search dimension is "product," the hit rate of recall results that match the company's products is the overlap rate between the hit dimension and the search dimension of the recall results.
[0070] Step S30: Perform a second round of search based on the target search terms of the first round of search according to a preset frequency to obtain second recall information.
[0071] Specifically, at a preset frequency, or at a set time interval after obtaining the first recall information, a second recall information is obtained by searching based on the target search terms of the first round of searches. It is important to note that the target search terms and search dimensions remain consistent in both searches.
[0072] In some embodiments, before performing a search based on the target search terms from the first round of search, first recall information is read, and a second round of search is performed on the target search terms that have recall results in the first round of search. That is, when performing a second round of search based on the target search terms from the first round of search, it is necessary to determine whether the target search term has a corresponding recall result in the first recall information. If the target search term does not have a recall result in the first recall information, the search for the target search term is skipped in the second round of search, thereby improving operational efficiency.
[0073] Step S40: Compare the first recall information and the second recall information, and obtain the verification result of the recall quality based on the comparison result.
[0074] Specifically, the obtained first and second recall information are compared, and the comparison can be performed on a per-target search term basis. The comparison results are then summarized to obtain a verification result of the recall quality. The verification result can include the quality of the search recall, such as its stability and accuracy. The verification result can be displayed through charts, quality trend graphs, text reports, etc. In some embodiments, the search dimensions and search engine rules can be optimized based on the verification result.
[0075] In the above method for verifying search recall quality, a use case file is established based on the search dimensions, including the target search terms under the search dimensions. A first round of search is conducted based on the target search terms in the use case file to obtain the first recall information. This allows for orderly batch searches, greatly improving search efficiency, and the obtained first recall information is easy to summarize and organize. A second round of search is conducted at a preset frequency, using the same target search terms and search dimensions as the first search, to obtain the second recall information. The first and second recall information are compared, and the final verification result of recall quality is obtained based on the comparison result. This ensures the consistency of search dimensions and target search terms between the two searches, making the comparison result of the first and second recall information more orderly and accurate, and more accurately verifying the recall quality.
[0076] In some embodiments of this disclosure, such as Figure 3 As shown, step S10 includes:
[0077] Step S12: Obtain the historical search information from the search engine logs according to a preset period, and extract the target search terms based on the historical search information.
[0078] Specifically, historical search information can be extracted from search engine logs. Search engine logs record user search behavior, including search time, search terms, and click information for retrieved results. Historical search information is retrieved from search engine logs at preset intervals, for example, monthly, extracting search records from the most recent month each month. The search terms and search dimensions from the selected historical search information are then organized to extract target search terms and search dimensions. For example, historical search terms can be organized by frequency, selecting those exceeding a set threshold as target search terms.
[0079] Step S14: Perform part-of-speech determination on the target search term to obtain the search dimension of the target search term.
[0080] Specifically, a natural language processing interface can be called to perform part-of-speech tagging on the target search term and obtain the search dimension information to which the target search term belongs. For example, if a company's product name is used as the target search term, after part-of-speech tagging, the search dimension of the target search term is found to be the company's products.
[0081] Step S16: Create a use case file corresponding to the search dimension based on the search dimension.
[0082] Specifically, the target search terms and search dimensions after part-of-speech tagging are summarized. A use case file is created for each search dimension, and the target search terms corresponding to that search dimension are associated with the use case file, thus completing the use case file creation. Since historical search information is acquired according to a preset period, the use case file is updated periodically.
[0083] This embodiment extracts historical search information from search engine logs according to a preset period, ensuring the timeliness of target search terms and their relevance to user behavior; at the same time, it improves the search and storage efficiency of target search terms and search dimensions by establishing use case files.
[0084] In some embodiments of this disclosure, such as Figure 4 As shown, step S20 includes:
[0085] Step S22: Read the target search terms in the use case file corresponding to the search dimension, and send search requests in batches based on the target search terms.
[0086] Step S24: Receive the return result of the search request and obtain the first recall information.
[0087] Specifically, the system sequentially reads the use case files corresponding to the search dimensions and sends search requests in batches for the target search terms in each use case file. Search requests can be sent using a pre-written search engine request script. The search engine request script can include the request protocol, request method, request headers, and request address. The system then receives the returned search results to obtain the first recall information.
[0088] This embodiment sends search requests in batches based on the target search terms in the use case file, which can efficiently and systematically search for the target search terms.
[0089] In some embodiments of this disclosure, step S40 includes:
[0090] Step S42: Compare the first recall information and the second recall information according to the comparison rules, which include the comparison of the order of recall results, the comparison of the number of recall results, and the comparison of the overlap rate between the hit dimension and the search dimension of the recall results.
[0091] Specifically, the first and second recall information are compared according to preset comparison rules. The comparison rules may include, but are not limited to, comparison of the order of recall results, comparison of the number of recall results, and comparison of the overlap rate between the hit dimension and the search dimension of the recall results.
[0092] The order comparison of recall results can refer to the comparison of the order of recall results returned by the first and second recall information based on the same target search term and search dimension. In some embodiments, the order range can be determined by the cumulative click-through rate (CTR). That is, starting from the CTR of the first-ranked recall result in the recall results, the CTR is accumulated sequentially. When the cumulative CTR meets a preset threshold, the recall results currently participating in the CTR accumulation are determined as the order range for comparison. Based on the user's click information on the recall results in historical search information, the CTR of the top few results in the recall results is calculated. For example, the preset cumulative CTR threshold is 80%. Based on the summary calculation of historical search information, if the cumulative CTR of the first to fourth ranked results in the recall results does not meet 80%, while the cumulative CTR of the first to fifth ranked results in the recall results does meet 80%, then the order range for comparison is determined to be the first to fifth ranked results. That is, the order comparison rule of recall results is used to compare the order of the first to fifth ranked results in the recall results returned by the first and second recall information based on the same target search term and search dimension.
[0093] The comparison of the number of recall results can refer to the comparison of the number of recall results returned by the first recall information and the second recall information based on the same target search term and search dimension.
[0094] The comparison of the overlap rate between the hit dimension and the search dimension in the recall results can refer to the comparison of the overlap rate between the hit dimension and the search dimension of the recall results returned by the first and second recall information based on the same target search term and search dimension. Specifically, the overlap rate between the hit dimension and the search dimension can be the ratio of the number of recall results with the same hit dimension and search dimension to the total number of recall results. For example, when the target search term is a company brand and the search dimension is product, and N recall results are obtained, of which M recall results have a hit dimension of product, then the overlap rate between the hit dimension and the search dimension for this search request is (M / N)*100%.
[0095] In this embodiment, the first recall information and the second recall information are compared according to the comparison rules. The comparison rules compare the recall results from multiple perspectives, such as the order of the recall results, the number of recall results, and the overlap rate between the hit dimension and the search dimension of the recall results, which provides sufficient data support for subsequent verification of recall quality.
[0096] In some embodiments of this disclosure, such as Figure 5 As shown, step S40 further includes:
[0097] Step S44: Obtain the sub-scores and total score of each comparison rule according to the comparison results. The verification results include the sub-scores and total score.
[0098] Specifically, the first recall information and the second recall information are compared according to the comparison rules to obtain the comparison results. The comparison rules can derive their respective sub-scores and the total comparison result score based on the comparison results. The sub-score for each comparison rule can refer to a score assigned to each rule, and the total comparison result score can refer to the overall comparison score representing the first and second recall information; the total score can be calculated from the sub-scores. The sub-scores and the total score can be specific scores or level scores, and are not limited in this embodiment.
[0099] In some embodiments, when obtaining the sub-score of the order comparison rule of the recall results, the sub-score can be calculated according to the following criteria: (1) If the first recall information includes a certain recall result, but the second recall information does not include the recall result, the order comparison rule of the recall results will be judged as poor level, and the sub-score corresponding to the poor level can be calculated; (2) If the first recall information includes a certain recall result, and the second recall information also includes the recall result, but the order of the two recall results is inconsistent, the comparison rule of the recall results will be judged as medium level, and the sub-score corresponding to the medium level can be calculated; (3) If the first recall information includes a certain recall result, and the second recall information also includes the recall result, and the order of the two recall results is consistent, the comparison rule of the recall results will be judged as excellent level, and the sub-score corresponding to the excellent level can be calculated.
[0100] In some embodiments, the sub-scores of the order comparison rule for obtaining recall results described above can be implemented by writing a script. A first-level loop is written based on the order of recall results in the first recall information for traversal, and then a second-level loop is nested to traverse the order of recall results in the second recall information. If the same recall result as the first-level result is encountered in the second-level loop, it is then determined whether the counters of the two loops are consistent. For example, if the counter of the first-level loop is 1 and the counter of the second-level loop is 2, it means that the recall result appeared in both the first and second recall information, but the order is inconsistent and the ranking has decreased by one position. In this case, the order comparison rule for recall results is judged as medium-level, and a medium-level score is output. If the counter of the first-level loop is 1 and the counter of the second-level loop is 1, it means that the recall result appeared in both the first and second recall information and the order is consistent. In this case, the order comparison rule for recall results is judged as relatively stable, judged as excellent-level, and an excellent-level score is output. If no recall result matching the first loop is found after the second loop iteration, the recall result order comparison rule judges it as poor, outputs the poor grade score, and records the corresponding target search term, search dimension, and the recall result and hit dimension hit in the first recall information to facilitate subsequent analysis of the cause and assist R&D personnel in following up and solving the problem.
[0101] In some embodiments, when obtaining the sub-score of the recall result comparison rule, the sub-score can be calculated according to the following criteria: the score is output based on the difference in the number of recall results obtained from the same target search term and search dimension in the first and second recall information. The larger the difference, the lower the sub-score output by the recall result comparison rule. For example, in the first recall information, the number of companies (i.e., the number of recall results) obtained by searching based on company name, business scope, and other search dimensions using "construction" as the target search term is 20 million, but in the second recall information, the number of companies (i.e., the number of recall results) obtained by searching based on company name, business scope, and other search dimensions using "construction" as the target search term is 2 million. This difference in number is large, and the sub-score output by the recall result comparison rule is low.
[0102] In some embodiments, when obtaining the sub-score of the rule comparing the overlap rate of the hit dimension and search dimension of the recall results, the sub-score can be calculated according to the following criteria: determining whether the overlap rate of the hit dimension and search dimension of the recall results in the first recall information meets the threshold condition, and determining whether the overlap rate of the hit dimension and search dimension of the recall results in the second recall information that are the same as those in the first recall information meets the threshold condition. If the overlap rate in both the first and second recall information meets the threshold condition, the rule comparing the overlap rate of the hit dimension and search dimension of the recall results outputs a higher score; otherwise, it outputs a lower score or even zero.
[0103] Step S46: If the sub-score or the total score does not meet the preset threshold, the verification result includes alarm information.
[0104] Specifically, thresholds can be preset for sub-scores and total scores based on comparison rules. When a sub-score output according to the comparison rules does not meet the preset threshold, the verification result includes an alarm message, which can be pushed through reports, alarm log emails, etc. The preset threshold can be set directly on the terminal or automatically obtained by calculation from historical recall information.
[0105] This embodiment generates verification results by quantifying the comparison results, which can clearly and explicitly characterize the search recall quality and provide multi-dimensional data references for recall quality. At the same time, based on the quantified score, an alarm message can be issued when the comparison result exceeds a preset threshold, which facilitates timely tracking and processing and improves the stability of the search engine.
[0106] In some embodiments of this disclosure, the method further includes:
[0107] The first recall information is stored in the first storage space, and the second recall information is stored in the second storage space.
[0108] Specifically, the first recall information and the second recall information are stored in separate partitions, that is, the first recall information is stored in the first storage space and the second recall information is stored in the second storage space.
[0109] In this embodiment, the first recall information and the second recall information are stored in separate partitions, which facilitates the rapid retrieval of the target search terms in the first recall information during the second round of search, and also facilitates the rapid retrieval and comparison of the first recall information and the second recall information.
[0110] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0111] Based on the same inventive concept, this disclosure also provides an apparatus for verifying search recall quality to implement the method for verifying search recall quality described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, specific limitations in one or more embodiments of the apparatus for verifying search recall quality provided below can be found in the limitations of the method for verifying search recall quality described above, and will not be repeated here.
[0112] The apparatus may include a system (including a distributed system), software (application), module, component, server, client, etc., that uses the methods described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the apparatus are similar, the implementation of the specific apparatus in the embodiments of this specification can refer to the implementation of the foregoing methods, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.
[0113] In some embodiments of this disclosure, such as Figure 6As shown, an apparatus for verifying search recall quality is provided. The apparatus can be the aforementioned terminal, a server, or a module, component, device, or unit integrated into the terminal. The apparatus Z00 may include:
[0114] The use case file module Z10 is used to determine the target search terms and search dimensions based on historical search information, and to create a use case file corresponding to the search dimensions based on the search dimensions. The use case file includes the target search terms corresponding to the search dimensions.
[0115] The first recall module Z20 is used to perform a first round of search based on the target search terms in the use case file to obtain the first recall information.
[0116] The second recall module Z30 is used to perform a second round of search based on the target search terms of the first round of search according to a preset frequency, and obtain second recall information.
[0117] The recall quality module Z40 is used to compare the first recall information and the second recall information, and obtain the verification result of the recall quality based on the comparison result.
[0118] In some embodiments of this disclosure, such as Figure 7 As shown, the use case file module Z10 includes:
[0119] Update unit Z12 is used to obtain the historical search information from the search engine log according to a preset period, and extract the target search terms based on the historical search information;
[0120] The part-of-speech tagging unit Z14 is used to perform part-of-speech tagging on the target search term and obtain the search dimension of the target search term;
[0121] The file creation unit Z16 is used to create use case files corresponding to the search dimension based on the search dimension.
[0122] In some embodiments of this disclosure, such as Figure 8 As shown, the first recall module Z20 includes:
[0123] Search request unit Z22 is used to read the target search terms in the use case file corresponding to the search dimension, and send search requests in batches based on the target search terms;
[0124] The return result unit Z24 is used to receive the return result of the search request and obtain the first recall information.
[0125] In some embodiments of this disclosure, such as Figure 9 As shown, the recall quality module Z40 includes:
[0126] The comparison unit Z42 is used to compare the first recall information and the second recall information according to the comparison rules, which include the comparison of the order of recall results, the comparison of the number of recall results, and the comparison of the overlap rate between the hit dimension and the search dimension of the recall results.
[0127] In some embodiments of this disclosure, such as Figure 9 As shown, the recall quality module Z40 also includes:
[0128] Score unit Z44 is used to obtain the sub-scores and total score of each comparison rule according to the comparison results. The verification results include the sub-scores and the total score.
[0129] Alarm unit Z46 is used to include alarm information in the verification result when the sub-score or the total score does not meet a preset threshold.
[0130] In some embodiments of this disclosure, such as Figure 10 As shown, the device Z00 further includes:
[0131] Storage module Z50 is used to store the first recall information in the first storage space and the second recall information in the second storage space.
[0132] Each module in the aforementioned apparatus for verifying search and recall quality can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; in actual implementation, other division methods may be used.
[0133] Based on the foregoing description of embodiments of the method for verifying search recall quality, in another embodiment provided in this disclosure, a computer device is provided. This computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for verifying search recall quality. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0134] Those skilled in the art will understand that the structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0135] Based on the foregoing description of embodiments of the method for verifying search recall quality, in another embodiment provided in this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.
[0136] Based on the foregoing description of embodiments of the method for verifying search recall quality, in another embodiment provided in this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.
[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0139] In the description of this specification, references to terms such as "some embodiments," "other embodiments," and "ideal embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.
[0140] It is understood that the various embodiments of the methods described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. Relevant details can be found in the descriptions of other method embodiments.
[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features of the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0142] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the scope of protection of this disclosure. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A method for verifying search recall quality, characterized in that, The method includes: Historical search information is retrieved from the search engine logs at a preset period, and target search terms are extracted based on the historical search information. Perform part-of-speech tagging on the target search term to obtain the search dimension of the target search term; Create use case files corresponding to the search dimensions; A first round of search is performed based on the target search terms in the use case file to obtain the first recall information; According to a preset frequency, a second round of search is performed based on the target search terms of the first round of search to obtain second recall information; The first recall information and the second recall information are compared, and the verification result of the recall quality is obtained based on the comparison result. The comparison of the first recall information and the second recall information includes: comparing the first recall information and the second recall information according to the comparison rules, which include the sequential comparison of recall results, the comparison of the number of recall results, and the comparison of the overlap rate between the hit dimension and the search dimension of the recall results. The range of the sequential comparison of recall results is determined based on the cumulative click-through rate of historical search information, starting from the first position of the recall result ranking. When the cumulative click-through rate meets a preset threshold, the recall results currently participating in the accumulation are the range of sequential comparison. The verification result of recall quality obtained based on the comparison results includes: obtaining the sub-scores and total scores of the comparison results corresponding to each of the comparison rules based on the comparison results, and the verification result includes the sub-scores and total scores; if the sub-scores or the total scores do not meet the preset threshold, the verification result includes alarm information; When obtaining the sub-scores for the order comparison rules of the recall results, the sub-scores are calculated according to the following criteria: If a recall result is included in the first recall information but not in the second recall information, the order comparison rule for the recall results is judged as poor, and the sub-score corresponding to the poor level is calculated and output; If a recall result is included in both the first and second recall information, but the order of the two recall results is inconsistent, the comparison rule for the recall results is judged as medium, and the sub-score corresponding to the medium level is calculated and output; If a recall result is included in both the first and second recall information, and the order of the two recall results is consistent, the comparison rule for the recall results is judged as excellent, and the sub-score corresponding to the excellent level is calculated and output. The first round of searching based on the target search terms in the use case file to obtain the first recall information includes: Read the target search terms from the use case file corresponding to the search dimension, and send search requests in batches based on the target search terms; The search request results are received to obtain the first recall information.
2. The method according to claim 1, characterized in that, The method includes: The first recall information is stored in the first storage space, and the second recall information is stored in the second storage space.
3. An apparatus for verifying the quality of search recall, characterized in that, The device includes: The use case file module includes: an update unit, used to obtain historical search information from the search engine logs at a preset period and extract target search terms based on the historical search information; a part-of-speech tagging unit, used to perform part-of-speech tagging on the target search terms and obtain the search dimensions of the target search terms; and a file creation unit, used to create use case files corresponding to the search dimensions. The first recall module is used to perform a first round of search based on the target search terms in the use case file to obtain the first recall information. The second recall module is used to perform a second round of search based on the target search terms of the first round of search according to a preset frequency, and obtain second recall information. The recall quality module is used to compare the first recall information and the second recall information, and obtain the verification result of the recall quality based on the comparison result; The recall quality module includes a comparison unit, used to compare the first recall information and the second recall information according to comparison rules. The comparison rules include the sequential comparison of recall results, the comparison of the number of recall results, and the comparison of the overlap rate between the hit dimension and the search dimension of the recall results. The range of the sequential comparison of recall results is determined based on the cumulative click-through rate of historical search information, starting from the first position in the recall result ranking. When the cumulative click-through rate meets a preset threshold, the recall results currently participating in the accumulation are the range of sequential comparison. The recall quality module further includes: a scoring unit, used to obtain the sub-scores and total scores of the comparison results corresponding to each of the comparison rules based on the comparison results, wherein the verification results include the sub-scores and the total scores; and an alarm unit, used to include alarm information in the verification results if the sub-scores or the total scores do not meet a preset threshold. When obtaining the sub-scores for the order comparison rules of the recall results, the sub-scores are calculated according to the following criteria: If a recall result is included in the first recall information but not in the second recall information, the order comparison rule for the recall results is judged as poor, and the sub-score corresponding to the poor level is calculated and output; If a recall result is included in both the first and second recall information, but the order of the two recall results is inconsistent, the comparison rule for the recall results is judged as medium, and the sub-score corresponding to the medium level is calculated and output; If a recall result is included in both the first and second recall information, and the order of the two recall results is consistent, the comparison rule for the recall results is judged as excellent, and the sub-score corresponding to the excellent level is calculated and output. The first recall module includes: The search request unit is used to read the target search terms in the use case file corresponding to the search dimension, and send search requests in batches based on the target search terms; The result return unit is used to receive the return results of the search request and obtain the first recall information.
4. The apparatus according to claim 3, characterized in that, The device further includes: The storage module is used to store the first recall information in the first storage space and the second recall information in the second storage space.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 1 or 2.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 1 or 2.
7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 1 or 2.