Retrieval result quality evaluation method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a retrieval result quality evaluation method and device, equipment, a medium and a program product, and particularly relates to the technical field of interest point retrieval. The method comprises the steps that a keyword sample is sent to a plurality of retrieval service providers, interest point retrieval results for the keyword sample returned by the retrieval service providers are obtained, and the interest point retrieval results comprise interest points obtained by the retrieval service providers through retrieval based on the keyword sample; the interest point retrieval results corresponding to different retrieval service providers are compared, an interest point evaluation value of the keyword sample is determined based on the comparison result, and the interest point evaluation value corresponds to a target interest point expected to be retrieved by the user; and on the basis of the interest point evaluation values, performing quality evaluation on interest point retrieval results corresponding to different retrieval service providers, and generating retrieval service evaluation results of the different retrieval service providers for the keyword sample. The method is used for achieving the effect of improving the evaluation efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of interest point retrieval, and in particular to a retrieval result quality assessment method, device, equipment, medium and program product. Background Art

[0002] Point of interest retrieval, also known as interest point query or POI (Point of Interest) search, is an important function in geographic information systems and location information services. Point of interest (POI) is a term in geographic information systems, referring to all geographical objects that can be abstracted as points, especially some geographical entities closely related to people's lives, such as schools, banks, restaurants, gas stations, hospitals, supermarkets, etc. Point of interest retrieval is to query and return information about relevant points of interest in the geographic information system according to user needs.

[0003] The quality assessment of POI retrieval results is crucial to ensure that users obtain accurate and useful information. Currently, manual evaluation methods are still widely used, which require a lot of manpower and time, and require high professional level and experience of evaluators. The evaluation results may be subjective. Summary of the invention

[0004] The embodiments of the present application provide a search result quality assessment method, apparatus, device, medium and program product for realizing automated assessment of interest point retrieval quality, thereby achieving the effect of improving assessment efficiency and accuracy.

[0005] In a first aspect, an embodiment of the present application provides a method for evaluating the quality of a search result, comprising:

[0006] Sending keyword samples to multiple search service providers, obtaining interest point search results for the keyword samples returned by the search service providers, the interest point search results including interest points retrieved by the search service providers based on the keyword samples;

[0007] Comparing the interest point search results corresponding to different search service providers, and determining the interest point evaluation value of the keyword sample based on the comparison result, wherein the interest point evaluation value corresponds to the target interest point that the user expects to retrieve;

[0008] Based on the interest point evaluation values, quality evaluation is performed on interest point search results corresponding to different search service providers to generate search service evaluation results of different search service providers for the keyword samples.

[0009] In a possible implementation manner, comparing the interest point search results corresponding to different search service providers, and determining the interest point evaluation value of the keyword sample based on the comparison result, includes:

[0010] Comparing the interest point search results corresponding to different search service providers, identifying key interest points from multiple interest points in the different interest point search results, wherein the key interest points are interest points whose frequencies of appearing in the multiple interest point search results meet a preset frequency condition;

[0011] The identified key interest points are determined as interest point evaluation values ​​of the keyword samples.

[0012] In a possible implementation, the step of performing quality assessment on the interest point search results corresponding to different search service providers based on the interest point evaluation values, and generating search service evaluation results of different search service providers for the keyword sample includes:

[0013] For any search service provider,

[0014] Filtering out records whose sorting positions meet preset sorting position conditions from the multiple records included in the interest point search results corresponding to the search service provider;

[0015] Determine whether the filtered records contain the interest point evaluation value, and generate a first evaluation result based on the determination result;

[0016] If the judgment result is that the record contains the interest point evaluation value, generating a second evaluation result according to the sorting position of the interest point evaluation value in the filtered record;

[0017] Based on the first evaluation result and the second evaluation result, a search service evaluation result of the search service provider for the keyword sample is generated.

[0018] In a possible implementation, the method further includes:

[0019] For any search service provider, based on the search service evaluation result of the search service provider for the keyword sample and the frequency of occurrence of the keyword sample in the target sample set, a comprehensive search service evaluation result of the search service provider for the target sample set is generated, and the target sample set is a sample set selected according to the search service evaluation instruction.

[0020] In a possible implementation manner, for any search service provider, based on the search service evaluation result of the search service provider for the keyword sample and the occurrence frequency of the keyword sample in the target sample set, generating a comprehensive search service evaluation result of the search service provider for the target sample set includes:

[0021] Performing a natural logarithm transformation on the occurrence frequency of the keyword sample in the target sample set to obtain a weight value corresponding to the keyword sample;

[0022] For any search service provider, based on the search service evaluation results of the search service provider for different keyword samples and the weight values ​​corresponding to different keyword samples, the comprehensive search service evaluation results of the search service provider for the target sample set are calculated.

[0023] In a possible implementation, sending the keyword sample to multiple search service providers and obtaining the interest point search results for the keyword sample returned by the search service providers includes:

[0024] Retrieve log data from multiple data sources;

[0025] Classifying the search keywords in the search log data to obtain sample sets corresponding to different search scenarios, wherein the sample sets include keyword samples corresponding to the search scenarios;

[0026] In response to a retrieval service evaluation instruction for a target retrieval scenario, a target sample set corresponding to the target retrieval scenario is extracted from sample sets corresponding to different retrieval scenarios;

[0027] The keyword samples in the target sample set are sent to multiple search service providers, and the interest point search results for the keyword samples returned by the search service providers are obtained.

[0028] In a possible implementation manner, the classifying and processing the search keywords in the search log data to obtain sample sets corresponding to different search scenarios includes:

[0029] If the search keyword in the retrieval log data is a location keyword, classifying the search keyword into a location retrieval scenario;

[0030] If the search keyword is a brand keyword, classifying the search keyword into a brand search scenario;

[0031] If the search keyword is a gourmet keyword, determining the target gourmet food category to which the search keyword belongs, and classifying the search keyword into the search scenario of the target gourmet food category;

[0032] According to the classification results, a sample set corresponding to the retrieval scenario is generated.

[0033] In a second aspect, an embodiment of the present application provides a search result quality assessment device, comprising:

[0034] An acquisition module, configured to send keyword samples to multiple search service providers, and acquire interest point search results for the keyword samples returned by the search service providers, wherein the interest point search results include interest points retrieved by the search service providers based on the keyword samples;

[0035] A determination module, configured to compare the interest point search results corresponding to different search service providers, and determine the interest point evaluation value of the keyword sample based on the comparison result, wherein the interest point evaluation value corresponds to the target interest point that the user expects to retrieve;

[0036] A generating module is used to perform quality evaluation on the interest point search results corresponding to different search service providers based on the interest point evaluation values, and generate search service evaluation results of different search service providers for the keyword samples.

[0037] In a possible implementation manner, the determining module is specifically configured to:

[0038] Comparing the interest point search results corresponding to different search service providers, identifying key interest points from multiple interest points in the different interest point search results, wherein the key interest points are interest points whose frequencies of appearing in the multiple interest point search results meet a preset frequency condition;

[0039] The identified key interest points are determined as interest point evaluation values ​​of the keyword samples.

[0040] In a possible implementation manner, the generating module is specifically used for:

[0041] For any search service provider,

[0042] Filtering out records whose sorting positions meet preset sorting position conditions from the multiple records included in the interest point search results corresponding to the search service provider;

[0043] Determine whether the filtered records contain the interest point evaluation value, and generate a first evaluation result based on the determination result;

[0044] If the judgment result is that the record contains the interest point evaluation value, generating a second evaluation result according to the sorting position of the interest point evaluation value in the filtered record;

[0045] Based on the first evaluation result and the second evaluation result, a search service evaluation result of the search service provider for the keyword sample is generated.

[0046] In a possible implementation manner, the retrieval result quality assessment device is further configured to:

[0047] For any search service provider, based on the search service evaluation result of the search service provider for the keyword sample and the frequency of occurrence of the keyword sample in the target sample set, a comprehensive search service evaluation result of the search service provider for the target sample set is generated, and the target sample set is a sample set selected according to the search service evaluation instruction.

[0048] In a possible implementation manner, the retrieval result quality assessment device is further configured to:

[0049] Performing a natural logarithm transformation on the occurrence frequency of the keyword sample in the target sample set to obtain a weight value corresponding to the keyword sample;

[0050] For any search service provider, based on the search service evaluation results of the search service provider for different keyword samples and the weight values ​​corresponding to different keyword samples, the comprehensive search service evaluation results of the search service provider for the target sample set are calculated.

[0051] In a possible implementation manner, the acquisition module is specifically used to:

[0052] Retrieve log data from multiple data sources;

[0053] Classifying the search keywords in the search log data to obtain sample sets corresponding to different search scenarios, wherein the sample sets include keyword samples corresponding to the search scenarios;

[0054] In response to a retrieval service evaluation instruction for a target retrieval scenario, a target sample set corresponding to the target retrieval scenario is extracted from sample sets corresponding to different retrieval scenarios;

[0055] The keyword samples in the target sample set are sent to multiple search service providers, and the interest point search results for the keyword samples returned by the search service providers are obtained.

[0056] In a possible implementation manner, the acquisition module is specifically used to:

[0057] If the search keyword in the retrieval log data is a location keyword, classifying the search keyword into a location retrieval scenario;

[0058] If the search keyword is a brand keyword, classifying the search keyword into a brand search scenario;

[0059] If the search keyword is a gourmet keyword, determining the target gourmet food category to which the search keyword belongs, and classifying the search keyword into the search scenario of the target gourmet food category;

[0060] According to the classification results, a sample set corresponding to the retrieval scenario is generated.

[0061] In a third aspect, an embodiment of the present application provides a retrieval result quality assessment device, including: a memory, a processor;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0065] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0066] The retrieval result quality assessment method, device, equipment, medium and program product provided in the embodiment of the present application can send keyword samples to multiple retrieval service providers, collect the interest point retrieval results for the keyword sample returned by each retrieval service provider, compare the interest point retrieval results corresponding to different retrieval service providers, determine the target interest point that the user expects to retrieve based on the comparison results, and assign it to the keyword sample as the interest point evaluation value. The present application assigns an interest point evaluation value to each keyword sample. In other words, the present application determines the interest point evaluation value of each keyword sample based on a voting mechanism. The voting mechanism determines the interest point evaluation value based on the consensus of multiple retrieval service providers, which is usually closer to the user's real expectations. The determined interest point evaluation value will be used as the basis for subsequent evaluation of the retrieval result quality to ensure the reliability of the evaluation result. In the quality assessment process, an evaluation index for evaluating the quality of the interest point retrieval result is defined based on the interest point evaluation value, the interest point retrieval result returned by each retrieval service provider is evaluated, and a retrieval service evaluation result for the keyword sample is generated for each retrieval service provider. The technical solution of the present application realizes the automated evaluation of the interest point retrieval quality, thereby improving the evaluation efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram of a scenario of a search result quality assessment method provided for this application;

[0069] Figure 2 Schematic diagram of the process of the search result quality assessment method provided for this application Figure 1 ;

[0070] Figure 3 Schematic diagram of the process of the search result quality assessment method provided for this application Figure 2 ;

[0071] Figure 4 A schematic diagram of the process of voting to determine the evaluation value of a point of interest provided for this application;

[0072] Figure 5 A schematic diagram of the process of quality assessment of POI search results corresponding to different search service providers provided by this application;

[0073] Figure 6 A schematic diagram of the overall process of the search result quality assessment method provided for this application;

[0074] Figure 7 A schematic diagram of the structure of the search result quality assessment device provided in this application;

[0075] Figure 8 A schematic diagram of the structure of the retrieval result quality assessment device provided in this application.

[0076] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] The quality assessment of current POI retrieval results is a key link to ensure that users can obtain accurate and valuable information. At present, manual evaluation methods are still widely used. Manual evaluation methods require a lot of manpower and time, especially when dealing with large-scale retrieval results, this limitation is particularly obvious. This may lead to inefficient evaluation and difficulty in meeting rapidly changing user needs. The professional level and experience of evaluators are crucial to the accuracy of evaluation results. However, it takes time and cost to cultivate personnel with high-level evaluation capabilities, and it is difficult to ensure that all evaluators can meet the same evaluation standards. Although evaluators will try their best to remain objective and fair, the evaluation results may still be affected by factors such as personal bias, knowledge background and experience, resulting in a certain degree of subjectivity. This may lead to inconsistency and difficulty in reproducibility of evaluation results.

[0079] The retrieval result quality assessment method provided by the present application can send keyword samples to multiple retrieval service providers, collect the interest point retrieval results for the keyword sample returned by each retrieval service provider, compare the interest point retrieval results corresponding to different retrieval service providers, and determine the target interest point that the user expects to retrieve based on the comparison results, and assign it to the keyword sample as an interest point evaluation value. The present application assigns an interest point evaluation value to each keyword sample. In other words, the present application determines the interest point evaluation value of each keyword sample based on a voting mechanism. The voting mechanism determines the interest point evaluation value based on the consensus of multiple retrieval service providers, which is usually closer to the user's true expectations. The determined interest point evaluation value will be used as the basis for subsequent evaluation of the quality of the retrieval results to ensure the reliability of the evaluation results. In the quality assessment process, an evaluation index for evaluating the quality of the interest point retrieval results is defined based on the interest point evaluation value, the interest point retrieval results returned by each retrieval service provider are evaluated, and a retrieval service evaluation result for the keyword sample is generated for each retrieval service provider, which solves the technical problem of low efficiency in quality assessment of the interest point retrieval results.

[0080] Figure 1 A schematic diagram of a scenario of a search result quality assessment method provided for this application, such as Figure 1 As shown, the quality assessment platform 101 stores multiple sample sets, each sample set corresponds to multiple keyword samples. The quality assessment platform can send the keyword samples in the sample set to multiple retrieval service providers, for example, to the retrieval service provider 102, the retrieval service providing method 103 and the retrieval service provider 104. The retrieval service provider can perform interest point retrieval based on the received keyword samples and generate interest point retrieval results. The retrieval service provider can send the interest point retrieval results to the quality assessment platform for retrieval result quality assessment for different retrieval service providers.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 2 Schematic diagram of the process of the search result quality assessment method provided for this application Figure 1 ,like Figure 2 As shown, the method includes:

[0083] S201. Send keyword samples to multiple search service providers, and obtain interest point search results for the keyword samples returned by the search service providers, where the interest point search results include interest points retrieved by the search service providers based on the keyword samples.

[0084] In an embodiment of the present application, one or more keyword samples may be sent to multiple different search service providers. The search service provider may include various map services, search engines or other services that can provide information about points of interest. Each search service provider will return corresponding search results for points of interest based on the received keyword samples. Specifically, the search service provider will analyze the keyword samples to understand the query intent and requirements. Next, the search service provider will query its geographic information database based on the keyword samples. This database may contain a large amount of point of interest data, including information such as name, address, category, coordinates, user evaluation, etc. After obtaining the preliminary search results, the search service provider may filter and sort the preliminary search results according to certain criteria (such as distance, rating, user preference, etc.) to form the final search results for points of interest. Finally, the search service provider will present the filtered and sorted points of interest to the user in an appropriate manner.

[0085] Since different search service providers may use different algorithms and data sources, even for the same keyword sample, different search service providers may return different POI search results.

[0086] S202: Compare the interest point search results corresponding to different search service providers, and determine the interest point evaluation value of the keyword sample based on the comparison result, where the interest point evaluation value corresponds to the target interest point that the user expects to retrieve.

[0087] Compare the POI search results from different search service providers. Based on the comparison results, determine the POI evaluation value of the keyword sample. The POI evaluation value represents the target POI that the user expects to retrieve, which may be the most interesting or relevant result for the user. The name emphasizes the function of this value as an evaluation criterion, which is used to evaluate the quality of the search results. By integrating the results from multiple search service providers, richer information can be obtained, which helps to identify more representative POIs. User interests and needs may change over time. By continuously updating the POI evaluation value, it can be dynamically adjusted to meet the user's latest needs.

[0088] S203: Based on the interest point evaluation values, quality evaluation is performed on interest point search results corresponding to different search service providers, and search service evaluation results of different search service providers for keyword samples are generated.

[0089] Using the determined point of interest evaluation value as a benchmark, quality evaluation is performed on the point of interest retrieval results of different retrieval service providers to generate a retrieval service evaluation result for each retrieval service provider.

[0090] The retrieval result quality evaluation method provided in the embodiment of the present application can send a keyword sample to multiple retrieval service providers, collect the interest point retrieval results for the keyword sample returned by each retrieval service provider, compare the interest point retrieval results corresponding to different retrieval service providers, and determine the target interest point that the user expects to retrieve based on the comparison result, and assign it to the keyword sample as an interest point evaluation value. The present application assigns an interest point evaluation value to each keyword sample. In other words, the present application determines the interest point evaluation value of each keyword sample based on a voting mechanism. The voting mechanism determines the interest point evaluation value based on the consensus of multiple retrieval service providers, which is usually closer to the user's real expectations. The determined interest point evaluation value will serve as the basis for subsequent evaluation of the retrieval result quality to ensure the reliability of the evaluation result. In the quality evaluation process, an evaluation index for evaluating the quality of the interest point retrieval result is defined based on the interest point evaluation value, the interest point retrieval result returned by each retrieval service provider is evaluated, and a retrieval service evaluation result for the keyword sample is generated for each retrieval service provider. The technical solution of the present application realizes the automated evaluation of the interest point retrieval quality, thereby improving the evaluation efficiency and accuracy.

[0091] Figure 3 Schematic diagram of the process of the search result quality assessment method provided for this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, a method for evaluating the quality of search results is described in detail. The method includes:

[0092] S301: Send keyword samples to multiple search service providers, and obtain interest point search results for the keyword samples returned by the search service providers, where the interest point search results include interest points retrieved by the search service providers based on the keyword samples.

[0093] In a possible implementation, sending keyword samples to multiple search service providers and obtaining interest point search results for the keyword samples returned by the search service providers may specifically include:

[0094] Retrieve log data from multiple data sources;

[0095] Classify the search keywords in the retrieval log data to obtain sample sets corresponding to different retrieval scenarios, where the sample sets contain keyword samples corresponding to the retrieval scenarios;

[0096] In response to a retrieval service evaluation instruction for a target retrieval scenario, a target sample set corresponding to the target retrieval scenario is extracted from sample sets corresponding to different retrieval scenarios;

[0097] The keyword samples in the target sample set are sent to multiple search service providers, and the interest point search results for the keyword samples returned by the search service providers are obtained.

[0098] The retrieval log data is collected from multiple data sources (such as different search engines, application logs, etc.), and the process is automated. The retrieval log data contains the user's search behavior and the search keywords used. Then, the search keywords in the collected retrieval log data are classified. The retrieval scenarios that need to be classified can be clearly identified, and specific classification conditions can be set for each retrieval scenario. The search keywords can be classified based on the preset classification conditions, which may involve natural language processing technology to identify and classify search keywords belonging to different retrieval scenarios. The result of the classification is a sample set of different retrieval scenarios, each of which contains keyword samples related to a specific retrieval scenario.

[0099] Exemplarily, the search scenarios can be divided into two categories: precise search and extensive search, and further subdivided into multiple demand scenarios. Among them, precise search can be divided into single demand search, multiple demand search, total demand search, road demand search, administrative area demand search, etc., and sample sets of multiple demand scenarios can be made. Extensive search includes brand search, type search, etc., and corresponding sample sets can be made. Single demand search means that the user's intention is very clear, usually looking for a specific place. For example, when searching for "Forbidden City", the expected result is "Forbidden City Museum". Multiple demand search means that the user's intention is relatively vague, and the user may have multiple related needs. The search results cover multiple aspects. For example, when searching for "Natural Residence", the search results include "Natural Residence Tea House", "Natural Residence Iron Pot Stew", "Natural Residence Decoration", "Natural Residence Apartment" and other related types of results. Total demand search means searching for a general concept, and the user expects to obtain multiple branches or specific details related to it in the search results. Taking the search for "Peking University" in Beijing as an example, the user may not only want to understand the overall concept of Peking University, but also want to obtain specific information about its various branches or campuses. When searching for "Peking University" in Beijing, the search results should include relevant information about multiple branches such as "Peking University (Xueyuan Road Campus)", "Peking University (Changping Campus)", and "Peking University (Yuanmingyuan Campus)". Road demand search means that the user's intention is to find information about a specific road or route. For example, when searching for "Chang'an Avenue", the expected result includes detailed information about the road. Administrative area demand search means that the user's intention is to find information about a specific administrative area. For example, when searching for "Haidian District", the expected result includes an overview of the area. Brand search means that the user's intention is to find relevant information about a certain brand. For example, when searching for "McDonald's", the expected result is a McDonald's store near the user's location. Type search means that the user's intention is to find a certain type of demand. For example, when searching for "hot pot", the expected result is a hot pot restaurant near the user's location.

[0100] In this embodiment, when a retrieval service evaluation instruction for a specific target retrieval scenario is received, a target sample set corresponding to the target retrieval scenario is extracted from the classified sample set. The target sample set contains keyword samples related to the target retrieval scenario. The keyword samples in the target sample set can be sent to multiple retrieval service providers.

[0101] In a possible implementation, the search keywords in the search log data are classified to obtain sample sets corresponding to different search scenarios, which may specifically include:

[0102] If the search keyword in the retrieval log data is a location keyword, the search keyword is classified into a location retrieval scenario;

[0103] If the search keyword is a brand keyword, the search keyword is classified into the brand search scenario;

[0104] If the search keyword is a gourmet keyword, determine the target gourmet category to which the search keyword belongs, and classify the search keyword into the search scenario of the target gourmet category;

[0105] According to the classification results, a sample set corresponding to the retrieval scenario is generated.

[0106] In this embodiment, the search scene includes a location search scene, a brand search scene and a food search scene, wherein the food search scene can be further subdivided into search scenes of multiple food categories. If the search keyword is a location keyword, the search keyword can be classified into the location search scene to generate a sample set of the location search scene, and the sample set of the location search scene includes all search keywords related to the location. If the search keyword is a brand keyword, the search keyword can be classified into the brand search scene to generate a sample set of the brand search scene, and the sample set of the brand search scene includes all search keywords related to the brand. If the search keyword is a food keyword, the specific food category to which it belongs, that is, the target food category, can be further determined, and it can be classified into the search scene of the target food category to generate a sample set of the target food category search scene, wherein the sample set of the target food category search scene includes all search keywords related to the target food category.

[0107] S302: Compare the interest point search results corresponding to different search service providers, and identify key interest points from multiple interest points in different interest point search results, where the key interest points are interest points whose frequencies of appearance in the multiple interest point search results meet a preset frequency condition.

[0108] A preset frequency condition may be pre-set to determine whether a point of interest is a key point of interest. For example, a point of interest needs to appear in at least three different points of interest search results to be considered a key point of interest. Furthermore, different search service providers have different search qualities, and different weights may be assigned to them to affect the frequency calculation.

[0109] Traverse all the interest point retrieval results, record the number of times each interest point appears in different interest point retrieval results, and record the total frequency of each interest point, and identify the key interest points according to the preset frequency conditions. Alternatively, extract the top N interest points from the interest point retrieval results of each retrieval service provider, where the specific value of N can be set before the quality assessment task is started, such as N=3. Collect the top N interest points of each retrieval service provider to form a set of candidate interest points. Record the number of times each interest point in the candidate interest point set appears in different interest point retrieval results, and record the total frequency of these interest points, and determine the interest points that meet the preset frequency conditions as key interest points. Through the above method, key interest points that are consistent and important in multiple retrieval service providers can be effectively identified.

[0110] S303: Determine the identified key interest points as interest point evaluation values ​​of the keyword sample, where the interest point evaluation values ​​correspond to target interest points that the user expects to retrieve.

[0111] The interest point evaluation value is applied to the keyword samples, and each keyword sample is labeled with its corresponding interest point evaluation value.

[0112] Reference Figure 4 As shown, it is a schematic diagram of the process of voting to determine the evaluation value of the point of interest provided by the present application, wherein the keyword sample is "KFC Xisanqi Bridge Store", and the keyword sample is sent to multiple search service providers, and each search service provider returns the corresponding point of interest search result. The point of interest search results corresponding to different search service providers are compared, and the point of interest evaluation value is determined by a voting mechanism. Since "KFC (Xisanqi Bridge East Store)" appears in the first three records of the three points of interest search results returned, the point of interest "KFC (Xisanqi Bridge East Store)" can be determined as the point of interest evaluation value of the keyword sample "KFC Xisanqi Bridge Store", and the voting process is automatically completed by the quality assessment platform. It should be noted that a keyword sample can correspond to one or more point of interest evaluation values. The point of interest evaluation value corresponding to the keyword sample determined by the automatic voting mechanism is used as a benchmark value or reference value for subsequent analysis.

[0113] As can be seen from the above, the key interest points may also be interest points whose ranking positions in the different interest point search results meet the preset ranking position conditions.

[0114] In addition, when the automatic voting mechanism cannot determine the interest point evaluation value corresponding to the keyword sample, the keyword sample and the interest point retrieval results returned by each retrieval service provider can be sent to the manual annotation platform for manual judgment. After the interest point evaluation value corresponding to the keyword sample is manually determined, the interest point evaluation value can be sent to the quality assessment platform as a basis for subsequent quality assessment.

[0115] S304: Based on the interest point evaluation values, quality evaluation is performed on interest point search results corresponding to different search service providers, and search service evaluation results of different search service providers for keyword samples are generated.

[0116] In a possible implementation, based on the interest point evaluation value, quality evaluation is performed on interest point search results corresponding to different search service providers, and search service evaluation results of different search service providers for keyword samples are generated, which may specifically include:

[0117] For any search service provider,

[0118] Filtering out records whose sorting positions meet preset sorting position conditions from the multiple records included in the interest point search results corresponding to the search service provider;

[0119] Determine whether the filtered records contain the interest point evaluation value, and generate a first evaluation result based on the determination result;

[0120] If the judgment result is that the record contains the interest point evaluation value, generating a second evaluation result according to the ranking position of the interest point evaluation value in the filtered records;

[0121] Based on the first evaluation result and the second evaluation result, a search service evaluation result of the search service provider for the keyword sample is generated.

[0122] An evaluation index for evaluating the quality of the retrieval results of points of interest is defined based on the evaluation values ​​of the points of interest. In the present embodiment, two evaluation indexes are set, one is an inclusive index, and the other is a ranking position index. Specifically, for any search service provider, the records whose ranking positions meet the preset ranking position conditions are screened out from the multiple records contained in the search results of the points of interest corresponding to the search service provider. For example, the records in the top 3 positions can be screened out. It is determined whether the evaluation values ​​of the points of interest are contained in the filtered records, and a first evaluation result is generated based on the judgment result. For example, it is determined whether the evaluation values ​​of the points of interest are contained in the top 3 records. If the evaluation values ​​of the points of interest are contained, a score of 50 can be obtained; otherwise, the score is 0. If the judgment result is that the evaluation values ​​of the points of interest are contained in the filtered records, a second evaluation result can be generated according to the specific ranking position of the evaluation values ​​of the points of interest in the filtered records. For example, if it ranks first among the top 3 records, a score of 50 can be obtained; otherwise, the score is 0.

[0123] Reference Figure 5As shown, it is a schematic diagram of the process of quality evaluation of interest point search results corresponding to different search service providers provided by this application. As shown in the figure, the evaluation scores of the search service evaluation results of different search service providers are 100 points, 100 points, 0 points and 50 points respectively.

[0124] The above are the retrieval service evaluation results of different retrieval service providers for keyword samples. The retrieval service evaluation results of different retrieval service providers for the entire sample set need to be determined later.

[0125] In a possible implementation manner, the search result quality assessment method provided in the embodiment of the present application may further include:

[0126] For any search service provider, based on the search service evaluation results of the search service provider for the keyword sample and the frequency of occurrence of the keyword sample in the target sample set, a comprehensive search service evaluation result of the search service provider for the target sample set is generated. The target sample set is the sample set selected according to the search service evaluation instruction.

[0127] Since keyword samples are extracted from retrieval log data from multiple data sources, there may be the same search behavior in the same time period, that is, different users use the same search keyword in the same time period. For example, if multiple users enter "hot pot" as a search keyword in the same time period, these same search keywords will be regarded as multiple instances and included in the same sample set. Therefore, the frequency of occurrence of each keyword sample in the target sample set can be calculated.

[0128] In a possible implementation, for any search service provider, based on the search service evaluation result of the search service provider for the keyword sample and the occurrence frequency of the keyword sample in the target sample set, a comprehensive search service evaluation result of the search service provider for the target sample set is generated, which may specifically include:

[0129] Perform natural logarithm transformation on the occurrence frequency of keyword samples in the target sample set to obtain the weight value corresponding to the keyword sample;

[0130] For any search service provider, based on the search service evaluation results of the search service provider for different keyword samples and the weight values ​​corresponding to different keyword samples, the comprehensive evaluation results of the search service of the search service provider for the target sample set are calculated.

[0131] In this embodiment, for each keyword sample, its search service evaluation result is combined with the corresponding weight value to generate a search service weighted evaluation result, and the search service weighted evaluation results of all keyword samples are summarized to calculate the search service comprehensive evaluation result of the search service provider for the target sample set. The search service comprehensive evaluation result can be used to compare the overall performance of different search service providers.

[0132] For any search service provider, the calculation formula of its comprehensive evaluation result P of the search service for a certain sample set can be as follows:

[0133]

[0134] Wherein, i=1, 2, 3, ..., n; n is the sample size.

[0135] In the above formula, the frequency of occurrence of each keyword sample in the sample set is transformed into a natural logarithm and used as the weight value. This method uses the natural logarithm function to perform nonlinear transformation, making the weight distribution more flexible. In the case of a large difference in the order of magnitude of the retrieval frequency, it more accurately reflects the evaluation of each sample in the sample set.

[0136] Reference Figure 6As shown, it is a schematic diagram of the overall process of the search result quality assessment method provided by the present application. In step S601, the sample data is automatically collected, analyzed and classified. This process is completed automatically, and a sample set corresponding to different search scenarios can be obtained. In step S602, based on the classified sample set, a quality assessment task for the search service of each search service provider is initiated. In this step, the quality assessment platform can obtain the interest point search results for the keyword sample returned by each search service provider. In step S603, the interest point search results corresponding to different search service providers are compared, and the interest point evaluation value of the keyword sample is determined based on the comparison result. If the interest point evaluation value corresponding to the keyword sample can be determined based on the voting mechanism, step S605 can be executed. When the automatic voting mechanism cannot determine the interest point evaluation value corresponding to the keyword sample, step S604 is executed to send the keyword sample and the interest point search results returned by each search service provider to the manual annotation platform to manually determine the interest point evaluation value. In step S605, based on the interest point evaluation value, the quality of the interest point search results corresponding to different search service providers is evaluated, and the search service evaluation results of different search service providers for keyword samples are generated. And for the sample sets corresponding to different search scenarios, the search service comprehensive evaluation results corresponding to different search service providers are generated. In step S607, the search service logic is adjusted based on the search service comprehensive evaluation results and continuously monitored to verify the quality improvement of the interest point search results after adjustment. Specifically, after determining the search service comprehensive evaluation results, the search service logic of the corresponding search service provider can be adjusted based on the search service comprehensive evaluation results. If the evaluation score of the search service comprehensive evaluation result of a search service provider is low, a keyword sample with a low evaluation score can be extracted from the sample set, and the gap between the interest point search results for the keyword sample and other search service providers can be identified. The defects and errors in the search service are adjusted, and the corresponding quality evaluation task of the sample set is re-executed after the problem is fixed, so as to verify the quality improvement of the interest point search results after adjustment. In addition, in step S606, the sample collection and classification logic is adjusted based on the search service comprehensive evaluation results to continuously improve the effectiveness of the quality evaluation. By building an automatic monitoring regression mechanism. Automatically carry out quality assessment tasks. When the evaluation score in the comprehensive evaluation results of the retrieval service meets the preset alarm conditions, it will automatically alarm and block the online access to ensure the online effect. With the expansion of the POI retrieval scenarios and the update of POI data, the sample set needs to be updated regularly to ensure that the sample set can reflect the latest user needs and market changes. When the comprehensive evaluation results of the retrieval service are obviously abnormal or the comprehensive evaluation results of the retrieval services of all parties are close, it means that the classification logic of the current sample set is wrong or tends to fail. It is necessary to re-check and adjust the collection and classification logic of the sample set to ensure the effectiveness of the quality assessment.

[0137] The retrieval result quality evaluation method provided in the embodiment of the present application can send a keyword sample to multiple retrieval service providers, collect the interest point retrieval results for the keyword sample returned by each retrieval service provider, compare the interest point retrieval results corresponding to different retrieval service providers, and determine the target interest point that the user expects to retrieve based on the comparison result, and assign it to the keyword sample as an interest point evaluation value. The present application assigns an interest point evaluation value to each keyword sample. In other words, the present application determines the interest point evaluation value of each keyword sample based on a voting mechanism. The voting mechanism determines the interest point evaluation value based on the consensus of multiple retrieval service providers, which is usually closer to the user's real expectations. The determined interest point evaluation value will serve as the basis for subsequent evaluation of the retrieval result quality to ensure the reliability of the evaluation result. In the quality evaluation process, an evaluation index for evaluating the quality of the interest point retrieval result is defined based on the interest point evaluation value, the interest point retrieval result returned by each retrieval service provider is evaluated, and a retrieval service evaluation result for the keyword sample is generated for each retrieval service provider. The technical solution of the present application realizes the automated evaluation of the interest point retrieval quality, thereby improving the evaluation efficiency and accuracy.

[0138] Figure 7 The schematic diagram of the structure of the search result quality evaluation device provided in this application is as follows: Figure 7 As shown, the retrieval result quality assessment device 70 provided in this embodiment includes:

[0139] An acquisition module 701 is used to send keyword samples to multiple search service providers, and obtain interest point search results for the keyword samples returned by the search service providers, where the interest point search results include interest points retrieved by the search service providers based on the keyword samples;

[0140] A determination module 702 is used to compare the interest point search results corresponding to different search service providers, and determine the interest point evaluation value of the keyword sample based on the comparison result, where the interest point evaluation value corresponds to the target interest point that the user expects to retrieve;

[0141] The generating module 703 is used to perform quality evaluation on the interest point search results corresponding to different search service providers based on the interest point evaluation values, and generate search service evaluation results of different search service providers for keyword samples.

[0142] In a possible implementation, the determination module is specifically used for:

[0143] Comparing the POI search results corresponding to different search service providers, identifying key POIs from multiple POIs in different POI search results, where the key POIs are POIs whose frequencies of appearance in the multiple POI search results meet a preset frequency condition;

[0144] The identified key interest points are determined as interest point evaluation values ​​of the keyword sample.

[0145] In a possible implementation, the generation module is specifically used for:

[0146] For any search service provider,

[0147] Filtering out records whose sorting positions meet preset sorting position conditions from the multiple records included in the interest point search results corresponding to the search service provider;

[0148] Determine whether the filtered records contain the interest point evaluation value, and generate a first evaluation result based on the determination result;

[0149] If the judgment result is that the record contains the interest point evaluation value, generating a second evaluation result according to the sorting position of the interest point evaluation value in the filtered record;

[0150] Based on the first evaluation result and the second evaluation result, a search service evaluation result of the search service provider for the keyword sample is generated.

[0151] In a possible implementation, the retrieval result quality assessment device is further configured to:

[0152] For any search service provider, based on the search service evaluation results of the search service provider for the keyword sample and the frequency of occurrence of the keyword sample in the target sample set, a comprehensive search service evaluation result of the search service provider for the target sample set is generated. The target sample set is the sample set selected according to the search service evaluation instruction.

[0153] In a possible implementation, the retrieval result quality assessment device is further configured to:

[0154] Perform natural logarithm transformation on the occurrence frequency of keyword samples in the target sample set to obtain the weight value corresponding to the keyword sample;

[0155] For any search service provider, based on the search service evaluation results of the search service provider for different keyword samples and the weight values ​​corresponding to different keyword samples, the comprehensive evaluation results of the search service of the search service provider for the target sample set are calculated.

[0156] In a possible implementation, the acquisition module is specifically used to:

[0157] Retrieve log data from multiple data sources;

[0158] Classify the search keywords in the retrieval log data to obtain sample sets corresponding to different retrieval scenarios, where the sample sets contain keyword samples corresponding to the retrieval scenarios;

[0159] In response to a retrieval service evaluation instruction for a target retrieval scenario, a target sample set corresponding to the target retrieval scenario is extracted from sample sets corresponding to different retrieval scenarios;

[0160] The keyword samples in the target sample set are sent to multiple search service providers, and the interest point search results for the keyword samples returned by the search service providers are obtained.

[0161] In a possible implementation, the acquisition module is specifically used to:

[0162] If the search keyword in the retrieval log data is a location keyword, the search keyword is classified into a location retrieval scenario;

[0163] If the search keyword is a brand keyword, the search keyword is classified into the brand search scenario;

[0164] If the search keyword is a gourmet keyword, determine the target gourmet category to which the search keyword belongs, and classify the search keyword into the search scenario of the target gourmet category;

[0165] According to the classification results, a sample set corresponding to the retrieval scenario is generated.

[0166] The retrieval result quality assessment device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0167] Figure 8 This is a schematic diagram of the structure of the search result quality assessment device provided in this application. Figure 8 As shown, the retrieval result quality assessment device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus.

[0168] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0169] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0170] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0171] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0172] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0173] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0174] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0175] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0176] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0177] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0181] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0182] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for evaluating the quality of a search result, characterized in that: include: Sending keyword samples to multiple search service providers, obtaining interest point search results for the keyword samples returned by the search service providers, wherein the interest point search results include interest points retrieved by the search service providers based on the keyword samples; Comparing the interest point search results corresponding to different search service providers, and determining the interest point evaluation value of the keyword sample based on the comparison result, wherein the interest point evaluation value corresponds to the target interest point that the user expects to retrieve; Based on the interest point evaluation values, quality evaluation is performed on interest point search results corresponding to different search service providers to generate search service evaluation results of different search service providers for the keyword samples.

2. The method according to claim 1, characterized in that The step of comparing the interest point search results corresponding to different search service providers and determining the interest point evaluation value of the keyword sample based on the comparison result includes: Comparing the interest point search results corresponding to different search service providers, identifying key interest points from multiple interest points in the different interest point search results, wherein the key interest points are interest points whose frequencies of appearing in the multiple interest point search results meet a preset frequency condition; The identified key interest points are determined as interest point evaluation values ​​of the keyword samples.

3. The method according to claim 1, characterized in that The method of performing quality evaluation on the interest point search results corresponding to different search service providers based on the interest point evaluation values, and generating search service evaluation results of different search service providers for the keyword samples, includes: for any search service provider, Filtering out records whose sorting positions meet preset sorting position conditions from the multiple records included in the interest point search results corresponding to the search service provider; Determine whether the filtered records contain the interest point evaluation value, and generate a first evaluation result based on the determination result; If the judgment result is that the record contains the interest point evaluation value, generating a second evaluation result according to the sorting position of the interest point evaluation value in the filtered record; Based on the first evaluation result and the second evaluation result, a search service evaluation result of the search service provider for the keyword sample is generated.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: For any search service provider, based on the search service evaluation result of the search service provider for the keyword sample and the frequency of occurrence of the keyword sample in the target sample set, a comprehensive search service evaluation result of the search service provider for the target sample set is generated, and the target sample set is a sample set selected according to the search service evaluation instruction.

5. The method according to claim 4, characterized in that The step of generating, for any search service provider, a comprehensive search service evaluation result of the search service provider for the target sample set based on the search service evaluation result of the search service provider for the keyword sample and the occurrence frequency of the keyword sample in the target sample set, includes: Performing a natural logarithm transformation on the occurrence frequency of the keyword sample in the target sample set to obtain a weight value corresponding to the keyword sample; For any search service provider, based on the search service evaluation results of the search service provider for different keyword samples and the weight values ​​corresponding to different keyword samples, the comprehensive search service evaluation results of the search service provider for the target sample set are calculated.

6. The method according to any one of claims 1 to 3, characterized in that The sending of the keyword sample to multiple search service providers and obtaining the interest point search results for the keyword sample returned by the search service providers include: Retrieve log data from multiple data sources; Classifying the search keywords in the search log data to obtain sample sets corresponding to different search scenarios, wherein the sample sets include keyword samples corresponding to the search scenarios; In response to a retrieval service evaluation instruction for a target retrieval scenario, a target sample set corresponding to the target retrieval scenario is extracted from sample sets corresponding to different retrieval scenarios; The keyword samples in the target sample set are sent to multiple search service providers, and the interest point search results for the keyword samples returned by the search service providers are obtained.

7. The method according to claim 6, characterized in that The classifying and processing the search keywords in the search log data to obtain sample sets corresponding to different search scenarios includes: If the search keyword in the retrieval log data is a location keyword, classifying the search keyword into a location retrieval scenario; If the search keyword is a brand keyword, classifying the search keyword into a brand search scenario; If the search keyword is a gourmet keyword, determining the target gourmet food category to which the search keyword belongs, and classifying the search keyword into the search scenario of the target gourmet food category; According to the classification results, a sample set corresponding to the retrieval scenario is generated.

8. A search result quality assessment device, characterized in that: include: An acquisition module, configured to send keyword samples to multiple search service providers, and acquire interest point search results for the keyword samples returned by the search service providers, wherein the interest point search results include interest points retrieved by the search service providers based on the keyword samples; A determination module, configured to compare the interest point search results corresponding to different search service providers, and determine the interest point evaluation value of the keyword sample based on the comparison result, wherein the interest point evaluation value corresponds to the target interest point that the user expects to retrieve; A generating module is used to perform quality evaluation on the interest point search results corresponding to different search service providers based on the interest point evaluation values, and generate search service evaluation results of different search service providers for the keyword samples.

9. A search result quality assessment device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor; and / or, The computer program product comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images