Search ranking performance evaluation methods, devices, electronic equipment, and readable storage media

Abstract

This application provides a method, apparatus, electronic device, and computer-readable storage medium for evaluating search ranking performance. The method includes: obtaining the actual recall ranking of each sub-recall result in the target recall result for a target search term, wherein the target recall result includes M sub-recall results, M≥1; obtaining the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term; obtaining the ideal recall ranking of each sub-recall result based on the relevance of each sub-recall result; and determining the ranking performance of the target recall result based on the actual recall ranking, the ideal recall ranking, and the relevance of each sub-recall result. This application can effectively evaluate the search ranking performance when the relevance of the recall results has two or more degrees.

Description

Technical Field

[0001] This application relates to the field of data search technology, specifically to a method, apparatus, electronic device, and computer-readable storage medium for evaluating search ranking performance. Background Technology

[0002] To improve the recall performance of search tools (such as map search engines and website search engines), it is usually necessary to evaluate metrics such as the overall recall accuracy and recall ranking (i.e., search ranking performance) of the search results.

[0003] In existing technologies, Mean Average Precision (MAP) is typically used to evaluate the ranking effect of search results (referred to as search ranking effect). However, since evaluating search ranking effect based on MAP requires that the recalled results be either completely relevant or completely irrelevant, it is not suitable for evaluating search ranking effect when the relevance of the recalled results is divided into more than two search scenarios. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and computer-readable storage medium for evaluating search ranking performance, aiming to solve the problem of evaluating search ranking performance when the relevance of recall results is two or more.

[0005] Firstly, this application provides a method for evaluating search ranking performance, the method comprising:

[0006] Obtain the actual recall ranking of each sub-recall result in the target recall result of the target search term, wherein the target recall result includes M sub-recall results, M≥1;

[0007] Based on the matching degree between each sub-recall result and the target search term, the relevance of each sub-recall result is obtained;

[0008] Based on the relevance of each sub-recall result, the ideal recall ranking of each sub-recall result is obtained;

[0009] The ranking effect of the target recall result is determined based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result.

[0010] Secondly, this application provides a search ranking effect evaluation device, the search ranking effect evaluation device comprising:

[0011] The first acquisition unit is used to acquire the actual recall ranking of each sub-recall result in the target recall result of the target search term, wherein the target recall result includes M sub-recall results, M≥1;

[0012] The second acquisition unit is used to acquire the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term;

[0013] The third acquisition unit is used to acquire the ideal recall ranking of each sub-recall result based on the relevance of each sub-recall result;

[0014] An evaluation unit is used to determine the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result.

[0015] Thirdly, this application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes the steps in any of the search ranking effect evaluation methods provided in this application.

[0016] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the search ranking effect evaluation method.

[0017] This application obtains the actual recall ranking of each sub-recall result in the target recall result of the target search term; obtains the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term; obtains the ideal recall ranking of each sub-recall result based on the relevance of each sub-recall result; and determines the ranking effect of the target recall result based on the actual recall ranking, the ideal recall ranking, and the relevance of each sub-recall result. Therefore, even when the recall result includes multiple relevance levels, the ranking effect of the recall result can still be effectively evaluated without requiring the recall result to be completely relevant or completely irrelevant. Thus, dividing the relevance of the recall result into two or more search scenarios can effectively evaluate the search ranking effect, solving the problem that the mean and average precision cannot effectively evaluate the search ranking effect (such as when the relevance of the recall result is divided into two or more search scenarios). Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of a search ranking effect evaluation system provided in an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a search ranking effect evaluation method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic flowchart of an embodiment of step 202 provided in this application;

[0022] Figure 4 This is a schematic flowchart of another embodiment of step 202 provided in this application;

[0023] Figure 5 This is a flowchart illustrating an example of adjusting preset sorting parameters provided in this application embodiment;

[0024] Figure 6 This is a schematic diagram of a process for adjusting the actual recall order provided in the embodiments of this application;

[0025] Figure 7 This is a schematic diagram of an embodiment of the search ranking effect evaluation device provided in this application.

[0026] Figure 8 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0029] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0030] Based on the aforementioned deficiencies in existing related technologies, embodiments of this application provide a method for evaluating search ranking performance, which at least to some extent overcomes the deficiencies in existing related technologies.

[0031] The execution subject of the search ranking effect evaluation method in this application embodiment can be the search ranking effect evaluation device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the search ranking effect evaluation device. The search ranking effect evaluation device can be implemented in hardware or software. The UE can be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0032] The electronic device can operate independently or in a cluster.

[0033] See Figure 1 , Figure 1 This is a schematic diagram of a search ranking effect evaluation system provided in this application embodiment. The system may include an electronic device 100, which integrates a search ranking effect evaluation device. For example, the electronic device can obtain the actual recall ranking of each sub-recall result in the target recall result of a target search term, wherein the target recall result includes M sub-recall results, M≥1; based on the matching degree between each sub-recall result and the target search term, obtain the relevance of each sub-recall result; based on the relevance of each sub-recall result, obtain the ideal recall ranking of each sub-recall result; and determine the ranking effect of the target recall result based on the actual recall ranking, the ideal recall ranking, and the relevance of each sub-recall result.

[0034] In addition, such as Figure 1As shown, the search ranking effect evaluation system may also include a memory 200 for storing data, such as the recall results of search terms, the actual recall ranking of each sub-recall result in the recall results, and other data.

[0035] It should be noted that, Figure 1 The schematic diagram of the search ranking effect evaluation system shown is merely an example. The search ranking effect evaluation system and scenario described in this application embodiment are for the purpose of more clearly illustrating the technical solutions of this application embodiment and do not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the evolution of search ranking effect evaluation systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.

[0036] The following describes the search ranking effect evaluation method provided in the embodiments of this application. In the embodiments of this application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0037] Reference Figure 2 , Figure 2 This is a flowchart illustrating a search ranking effect evaluation method provided in an embodiment of this application. It should be noted that, although in Figure 2 The logical order is shown in the flowcharts of other accompanying figures, but in some cases, the steps shown or described may be performed in a different order than that shown here; for example, step 201 may be performed before step 202, or step 202 may be performed before step 201, or steps 201 and 202 may be performed simultaneously. This search ranking effect evaluation method includes steps 201–204, wherein:

[0038] 201. Obtain the actual recall ranking of each sub-recall result in the target recall result of the target search term.

[0039] The target recall result includes M sub-recall results, where M≥1.

[0040] The target search term refers to the search keywords entered during a search, such as the place name entered in a map search engine, or the search keywords entered on a web search page.

[0041] Specifically, target recall results are the search results returned by the search tool after searching with the target search term. Target recall results include at least one sub-recall result. For example, if the target search term "No. 100 Bailu Street" is entered into a map search engine, the target recall results returned by the map search engine include: sub-recall result a (Aquatic Products Bureau Community | No. 10 Donghu Road, Shuiguohu Area), sub-recall result b (Fangyingtai | No. 100 Bailu Street), and sub-recall result c (New Millennium Gate | No. 102 Bailu Street).

[0042] Each sub-recall result represents a search result returned by the search tool based on the target search term. For example, sub-recall result a, sub-recall result b, or sub-recall result c for the target search term "100 Bailu Street".

[0043] The recall ranking of each sub-recall result refers to the ranking of each sub-recall result in the target recall result. For example, if the map search engine returns the above sub-recall result a, sub-recall result b, and sub-recall result c for the target search term "Bailu Street No. 100", and assumes that the ranking of sub-recall results a, b, and c is 1st, 2nd, and 3rd respectively, then the recall ranking of sub-recall result a is 1st.

[0044] The actual recall ranking of each sub-recall result refers to the recall ranking of each sub-recall result returned by the search tool (such as map search engine, website search engine, etc.).

[0045] There are several ways to obtain the actual recall ranking of each sub-recall result in step 201, including, for example:

[0046] (1) Based on the real-time return of the search tool. At this time, step 201 may specifically include: using the search tool, searching based on the target search term to obtain the target recall result and the actual recall ranking of each sub-recall result in the target recall result.

[0047] (2) Read directly from the preset database. Specifically, each search term is searched in advance using a search tool to obtain the recall results of each search term and the actual recall ranking of each sub-recall result in the recall results of each search term; and the recall results of each search term and the actual recall ranking of each sub-recall result in the recall results of each search term are stored in the preset database; in step 201, the target recall results of the target search term and the actual recall ranking of each sub-recall result in the target recall results are read directly from the preset database.

[0048] 202. Based on the matching degree between each sub-recall result and the target search term, obtain the relevance of each sub-recall result.

[0049] The relevance of each sub-recall result refers to the relevance between each sub-recall result and the target search term. The relevance of each sub-recall result can be determined based on factors such as intent matching and keyword matching between the sub-recall result and the target search term. An example is provided below for illustration:

[0050] First, based on the intent matching degree between each sub-recall result and the target search term, determine the relevance of each sub-recall result. At this point, such as... Figure 3 As shown, step 202 may specifically include the following steps 2021A to 2022A, wherein:

[0051] 2021A. Obtain the intent matching degree between each sub-recall result and the target search term, and use it as the intent matching degree of each sub-recall result.

[0052] Among them, the intent matching degree of each sub-recall result refers to the intent matching degree between each sub-recall result and the target search term, specifically the matching degree between each sub-recall result and the referential intent of the target search term.

[0053] For example, in step 2021A, the intent matching degree of each sub-recall result can be obtained by comparing the proximity between the referential intent of each sub-recall result and the referential intent of the target search term as the intent matching degree between each sub-recall result and the target search term.

[0054] 2022A. Based on the intent matching degree of each sub-recall result, determine the relevance of each sub-recall result.

[0055] In step 2022A, there are several ways to determine the relevance of each sub-recall result, including, for example:

[0056] (1) Directly use the intent matching degree of each sub-recall result as the relevance of each sub-recall result. Alternatively, use the result of multiplying the intent matching degree of each sub-recall result by a preset coefficient as the relevance of each sub-recall result.

[0057] (2) Combine the exact intent type of the target search term and the intent matching degree of each sub-recall result to determine the relevance of each sub-recall result.

[0058] Because the input objects of target search terms (such as map search users) are subjective, the clarity of the referential intent of the target search terms entered by different input objects varies. Therefore, target search terms may have clear referential intent or vague referential intent.

[0059] When the referential intent of the target search term is clear, the relevance of the sub-recall results can be determined by directly comparing whether the referential intent of the sub-recall results is the same as or similar to that of the target search term. That is, the relevance of the sub-recall results is directly positively correlated with the intent matching degree of the sub-recall results.

[0060] When the referential intent of the target search term is ambiguous, since the referential intent of the target search term (such as a place name) is not unique and the degree of similarity between different referential intents and user intent is inconsistent, in addition to comparing whether the referential intent of the sub-recall results is the same or similar to the referential intent of the target search term, it is also necessary to combine the degree of clarity of the intent of the target search term (i.e., the degree of clarity of the referential intent of the target search term, specifically the degree of similarity between the referential intent of the target search term and user intent) to determine the intent matching degree of each sub-recall result.

[0061] At this point, prior to step 2021A, the process may further include: obtaining the intent precision type of the target search term. For example, the intent precision type of the target search term can be determined by detecting whether the referential intent of the target search term is unique. For instance, when the referential intent of the target search term is unique, such as when the number of referential intents of the target search term is equal to 1, the intent precision type of the target search term is determined to be explicit; when the referential intent of the target search term is not unique, such as when the number of referential intents of the target search term is greater than 1, the intent precision type of the target search term is determined to be ambiguous. Alternatively, the intent precision type of the target search term can also be determined by detecting the word type of the target search term. For example, when searching for ground in a map, when the word type of the target search term is an exact word, the intent precision type of the target search term can be directly determined to be explicit; when the word type of the target search term is a chain word, a generic word, an abbreviation, etc., the intent precision type of the target search term can be directly determined to be ambiguous.

[0062] Among them, the intent precision type is used to indicate the degree of explicitness of the referential intent of the target search term.

[0063] Please refer to Table 1. Table 1 is an example of determining the relevance of each sub-recall result by combining the precise intent type of the target search term. The following examples illustrate how to determine the intent matching degree of each sub-recall result when the referential intent of the target search term is clear and when the referential intent of the target search term is ambiguous.

[0064] Table 1

[0065]

[0066] ① The referential intent of the target search term is clear.

[0067] When the intent type is explicit, meaning the target search term is an exact word, the target recall results fall into two categories: first, the referential intent of the sub-recall results is the same as or similar to the referential intent of the target search term; second, the referential intent of the sub-recall results is different from or not similar to the referential intent of the target search term. For example, the intent matching degree of each sub-recall result can be directly set as the relevance of each sub-recall result; alternatively, the product of the intent matching degree of each sub-recall result and a preset coefficient can be set as the relevance of each sub-recall result; or, a preset relevance value corresponding to the explicit intent type of the target search term and the intent matching degree of each sub-recall result can be set as the relevance of each sub-recall result.

[0068] When setting the relevance of each sub-recall result to the pre-defined relevance values ​​corresponding to the intent type of the target search term and the intent matching degree of each sub-recall result, the target recall results can be divided into two categories (namely: the first sub-recall result and the second sub-recall result) to indicate the closeness between the referential intent of the sub-recall result and the referential intent of the target search term. In this case, as shown in the exact word-to-recall results in Table 1, the relevance of the first sub-recall result can be set to the first relevance pre-defined value (e.g., 2), and the relevance of the second sub-recall result can be set to the second relevance pre-defined value (e.g., 0).

[0069] Specifically, step 2022A may include: obtaining the exact intent type of the target search term; if the exact intent type is explicit, then setting the relevance of the first sub-recall result to a first relevance preset value; or, setting the relevance of the second sub-recall result to a second relevance preset value.

[0070] Among them, the first relevance preset value is greater than the second relevance preset value.

[0071] The first relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is explicit and the referential intent is the same as or similar to the referential intent of the search term.

[0072] The second relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is clear and the referential intent is different from or not similar to the referential intent of the search term. That is, for sub-recall results that are different from or not similar to each other, other preset values ​​are used to represent them.

[0073] The first sub-recall result is the sub-recall result with the highest intent match among the target recall results. Specifically, the first sub-recall result indicates the sub-recall result whose referential intent is the same as or similar to the referential intent of the target search term when the exact intent type of the target search term is explicit.

[0074] The second sub-recall result is each sub-recall result in the target recall result other than the first sub-recall result. Specifically, the second sub-recall result is used to indicate: when the exact intent type of the target search term is clear, the sub-recall result whose referential intent is different from or not similar to the referential intent of the target search term.

[0075] Each sub-recall result can be either the first sub-recall result or the second sub-recall result.

[0076] Since the first relevance preset value is greater than the second relevance preset value, and the ideal recall ranking of each subsequent sub-recall result is positively correlated with the relevance of each sub-recall result, when the intent precise type of the target search term is clear, on the one hand, the ideal recall ranking of the first sub-recall result with the highest intent match can be determined as the highest, thus improving the rationality of the ideal recall ranking of the sub-recall results when the intent precise type of the target search term is clear; on the other hand, the ideal recall ranking of other sub-recall results (i.e., the second sub-recall results) can be ignored. By directly determining the relevance of the second sub-recall result as the second relevance preset value which is less than the first relevance preset value, the amount of data processing when determining the relevance of each sub-recall result can be reduced, thereby improving the evaluation speed of the ranking effect of the target recall results.

[0077] ②The referential intent of the target search term is vague.

[0078] When the intent type is ambiguous, meaning the target search term is an abbreviation, generic term, or chain term, the referential intent of the target search term is vague. Therefore, it is difficult to directly determine whether the referential intent of the sub-recall results is the same as or similar to that of the target search term. To improve the reasonableness of the specific relevance values ​​of each sub-recall result and thus the ranking effect of the determined target recall results, the degree of closeness between the referential intent of each sub-recall result and the referential intent of the target search term can be indicated based on the range of intent matching degree of each sub-recall result, thereby determining the specific relevance value of each sub-recall result. Furthermore, the relevance of each determined sub-recall result should be positively correlated with its intent matching degree; that is, the higher the intent matching degree of each sub-recall result, the closer its referential intent is to that of the target search term, and the higher the specific relevance value of each recall result.

[0079] In cases where the referential intent of the target search term is ambiguous, multiple intent matching intervals can be pre-defined to indicate the degree of closeness between the referential intent of each sub-recall result and the referential intent of the target search term (e.g., intervals 1, 2, and 3 represent medium, high, and low closeness, respectively). This allows for the determination of the specific value of each sub-recall result. Thus, the relevance value of each sub-recall result can be determined based on which of the pre-defined intent matching intervals it falls into. The number of pre-defined intent matching intervals can be adjusted according to actual application needs; for example, two or three intervals can be used. Examples using two and three pre-defined intent matching intervals are provided below.

[0080] In some embodiments, the number of pre-divided intent matching intervals is two. One intent matching interval (e.g., a first preset range) indicates a high degree of similarity, and the other intent matching interval (e.g., outside the first preset range) indicates a low degree of similarity. That is, the target recall results can be divided into two categories (a third sub-recall result and a fourth sub-recall result) to indicate the closeness between the referential intent of the sub-recall result and the referential intent of the target search term. The closeness between the referential intent of the third sub-recall result and the referential intent of the fourth sub-recall result and the referential intent of the target search term is, in order, higher and lower. In this case, the relevance of the third sub-recall result can be set to a third relevance preset value (e.g., set to 1), and the relevance of the fourth sub-recall result can be set to a fourth relevance preset value (e.g., set to 0).

[0081] Specifically, step 2022A may include: obtaining the intent precision type of the target search term; if the intent precision type is fuzzy, then setting the relevance of the third sub-recall result to a third relevance preset value; or, setting the relevance of the fourth sub-recall result to a fourth relevance preset value.

[0082] Among them, the third correlation preset value is greater than the fourth correlation preset value.

[0083] Among them, the intent matching degree of the fourth sub-recall result is less than the intent matching degree of any one within the first preset range.

[0084] The third relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is fuzzy and the referential intent is relatively close to the referential intent of the search term (i.e., the intent matching degree is within the first preset range).

[0085] The fourth relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is fuzzy and the referential intent is relatively close to the referential intent of the search term (i.e., the intent matching degree is less than any intent matching degree within the first preset range).

[0086] The third sub-recall result refers to each sub-recall result in the target recall result whose intent matching degree is within a first preset range. For example, each sub-recall result whose intent matching degree is greater than a preset matching degree threshold. Specifically, the third sub-recall result is used to indicate: when the intent precision type of the target search term is ambiguous, the sub-recall result with a high degree of similarity between the referential intent and the referential intent of the target search term.

[0087] The fourth sub-recall result refers to each sub-recall result in the target recall result other than the third sub-recall result. For example, sub-recall results with intent matching degree less than or equal to a preset matching degree threshold. The second sub-recall result is specifically used to indicate: when the intent precision type of the target search term is ambiguous, sub-recall results with a low degree of similarity between the referential intent and the referential intent of the target search term.

[0088] Each sub-recall result can be either the third sub-recall result or the fourth sub-recall result.

[0089] Since the third relevance preset value is greater than the fourth relevance preset value, and the ideal recall ranking of each subsequent sub-recall result is positively correlated with the relevance of each sub-recall result, when the intent precision type of the target search term is fuzzy, on the one hand, the ideal recall ranking of the third sub-recall result with intent matching degree within the first preset range (such as greater than the preset matching degree threshold) can be determined to be relatively high, improving the rationality of the ideal recall ranking of the sub-recall results; on the other hand, the ideal recall ranking of the sub-recall result (i.e., the fourth sub-recall result) with intent matching degree outside the first preset range can be ignored. By directly determining the relevance of the fourth sub-recall result to the fourth relevance preset value which is less than the third relevance preset value, the amount of data processing when determining the relevance of each sub-recall result can be reduced, thereby improving the evaluation speed of the ranking effect of the target recall result.

[0090] In some embodiments, the number of pre-divided intent matching intervals is three. The first intent matching interval (e.g., the highest intent matching value) indicates a high degree of similarity, the second intent matching interval (e.g., a second preset range) indicates a medium degree of similarity, and the third intent matching interval (e.g., a third preset range) indicates a low degree of similarity. This divides the target recall results into three categories (the fifth sub-recall result, the sixth sub-recall result, and the seventh sub-recall result) to indicate the closeness between the referential intent of the sub-recall result and the referential intent of the target search term. The closeness between the referential intent of the fifth, sixth, and seventh sub-recall results and the referential intent of the target search term is high, medium, and low, respectively. In this case, as shown in Table 1 for the recall results corresponding to abbreviations, the relevance of the fifth sub-recall result can be set to the fifth relevance preset value (e.g., 2), the relevance of the sixth sub-recall result can be set to the sixth relevance preset value (e.g., 1), and the relevance of the seventh sub-recall result can be set to the seventh relevance preset value (e.g., 0).

[0091] Specifically, step 2022A may include: obtaining the intent precision type of the target search term; if the intent precision type is fuzzy, then setting the relevance of the fifth sub-recall result to the fifth relevance preset value; or, setting the relevance of the sixth sub-recall result to the sixth relevance preset value; or, setting the relevance of the seventh sub-recall result to the seventh relevance preset value.

[0092] Among them, the fifth relevance preset value is greater than the sixth relevance preset value, and the sixth relevance preset value is greater than the seventh relevance preset value.

[0093] Among them, the intent matching degree of any one within the second preset range is greater than that of any one within the third preset range. The intent matching degree of the fifth sub-recall result is greater than that of any one within the second preset range.

[0094] Among them, the fifth relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is fuzzy and the referential intent is relatively close to the referential intent of the search term (i.e. the sub-recall results with the highest intent matching degree).

[0095] The sixth relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is fuzzy and the degree of similarity between the referential intent and the referential intent of the search term is moderate (i.e., the intent matching degree is within the second preset range).

[0096] The seventh relevance preset value is the relevance preset value corresponding to the sub-recall results where the intent of the search term is ambiguous and the referential intent is not close to the referential intent of the search term (i.e., the intent matching degree is within the third preset range).

[0097] The fifth sub-recall result is the sub-recall result with the highest intent match among the target recall results. Specifically, the fifth sub-recall result indicates the sub-recall result with a high degree of similarity to the referential intent of the target search term when the exact intent type of the target search term is ambiguous.

[0098] The sixth sub-recall result is each sub-recall result whose intent matching degree falls within the second preset range. For example, each sub-recall result whose intent matching degree is greater than the first preset threshold and less than the second preset threshold (such as less than the intent matching degree of the fifth sub-recall result). Specifically, the sixth sub-recall result is used to indicate: when the exact intent type of the target search term is ambiguous, the sub-recall result with a moderate degree of closeness between the referential intent and the referential intent of the target search term.

[0099] The seventh sub-recall result is each sub-recall result whose intent matching degree is within the third preset range. For example, each sub-recall result whose intent matching degree is less than or equal to the first preset threshold. Specifically, the seventh sub-recall result is used to indicate: when the intent exact type of the target search term is ambiguous, the sub-recall result with a low degree of similarity between the referential intent and the referential intent of the target search term.

[0100] The results of each sub-recall are the results of the fifth sub-recall, the sixth sub-recall, or the seventh sub-recall.

[0101] Since the fifth relevance preset value is greater than the sixth relevance preset value, the sixth relevance preset value is greater than the seventh relevance preset value, and the intent matching degree within the second preset range is greater than the intent matching degree within the third preset range, and the ideal recall ranking of each subsequent sub-recall result is positively correlated with the relevance of each sub-recall result, when the intent precision type of the target search term is fuzzy, the ideal recall ranking of the fifth sub-recall result with the highest intent matching degree can be determined as the sixth sub-recall result, which is ranked first and has an intent matching degree within the second preset range (e.g., greater than the first preset threshold and less than the second preset threshold). The ideal recall ranking of the results is determined to be first, followed by the fifth sub-recall result. The ideal recall ranking of the seventh sub-recall result, whose intent matching degree is within the third preset range (such as sub-recall results less than or equal to the first preset threshold), is determined to be first, followed by the sixth sub-recall result. Thus, when the exact type of the target search term intent is an abbreviation, the full name result corresponding to the abbreviation (such as the address name of the full name "XX Company") and the corresponding sub-results of the full name (the detailed address name of the full name "XX Company XX Branch") can be effectively filtered out and ranked in order, thereby improving the rationality of the ideal recall ranking of the sub-recall results.

[0102] In other embodiments, since the intent of the target search term is fuzzy, each sub-recall result in the target recall results returned by the search tool can be considered to have the same referential intent as the target search term. That is, the number of pre-divided intent matching intervals is 1. In other words, the target recall results can be divided into only one category to indicate the closeness of the referential intent of the sub-recall results to the referential intent of the target search term (e.g., all are considered the same). In this case, as shown in Table 1, where broad search terms or chain terms correspond to corresponding recall results, the relevance of each sub-recall result in the target recall results, i.e., each sub-recall result, can be directly determined to a certain preset relevance value (e.g., set to 1).

[0103] Second, based on the keyword matching degree between each sub-recall result and the target search term, determine the relevance of each sub-recall result. At this point, step 202 may specifically include the following steps 2021B to 2022B, wherein:

[0104] 2021B. Obtain the keyword matching degree between each sub-recall result and the target search term, and use it as the keyword matching degree of each sub-recall result.

[0105] In step 2021B, there are multiple ways to determine the keyword matching degree of each sub-recall result. For example, these include:

[0106] 1) First, the target search term can be segmented according to its targeting type to obtain multiple intent-targeting keywords. Then, each sub-recall result can be segmented into multiple intent-targeting keywords. Next, from the multiple intent-targeting keywords in each sub-recall result, a target keyword that matches any one of the multiple intent-targeting keywords of the target search term is selected. Finally, based on the preset matching score for each targeting type and the targeting type of the target keywords in each sub-recall result, the keyword matching score for each sub-recall result is calculated.

[0107] For example, the preset keyword targeting types include street address, road section, county or district, etc., and the preset matching scores for street address, road section, and county or district are 10 points, 5 points, and 2 points, respectively.

[0108] Suppose the target search term is the place name "No. 100 Bailu Street" entered into the map search engine. Since the target search term only contains one type of keyword, the target search term "No. 100 Bailu Street" is divided into keywords, resulting in only one intent-pointing keyword "No. 100 Bailu Street".

[0109] By segmenting the sub-recall result b, “Fangyingtai|No. 100 Bailu Street”, we obtained two intent-pointing keywords for sub-recall result b: “Fangyingtai” and “No. 100 Bailu Street”.

[0110] Since the two intent keywords of sub-recall result b, "No. 100 Bailu Street", are the same as the intent keywords of the target search term, based on the targeting type (i.e., the street address) of the intent keyword "No. 100 Bailu Street" in sub-recall result b and the preset matching score (i.e., 10 points) of the keyword of the street address, it can be determined that the keyword matching degree of sub-recall result b is 10.

[0111] 2) First, the target search term can be segmented according to its targeting type to obtain multiple intent-targeting keywords for the target search term. Then, each sub-recall result can be segmented to obtain multiple intent-targeting keywords for each sub-recall result. Next, the number and percentage of keyword overlap between the multiple intent-targeting keywords of each sub-recall result and the multiple intent-targeting keywords of the target search term are compared. Finally, based on the number and percentage of keyword overlap, the preset matching score for each range of keyword overlap, and the preset matching score for each range of keyword overlap, the keyword matching degree between each sub-recall result and the target search term is determined, serving as the keyword matching degree for each sub-recall result. Specifically, the keyword overlap ratio can be the ratio between the number of target keywords and the number of intent-targeting keywords for the target search term. Target keywords can specifically refer to any keyword selected from the multiple intent-targeting keywords of each sub-recall result that is identical to any one of the multiple intent-targeting keywords of the target search term.

[0112] For example, the preset keyword targeting types include street address, road section, county or district, etc. Assuming the keyword overlap ratio is 100%, 70% ≤ keyword overlap ratio < 100%, and keyword overlap ratio < 70%, the corresponding preset matching scores are 10 points, 5 points, and 1 point, respectively.

[0113] Suppose the target search term is the place name "No. 100 Bailu Street" entered into the map search engine. Since the target search term only contains one type of keyword, the target search term "No. 100 Bailu Street" is divided into keywords, resulting in only one intent-pointing keyword "No. 100 Bailu Street".

[0114] By segmenting the sub-recall result b, “Fangyingtai|No. 100 Bailu Street”, we obtained two intent-pointing keywords for sub-recall result b: “Fangyingtai” and “No. 100 Bailu Street”.

[0115] Since the keyword overlap between the intent-targeting keywords of sub-recall result b and the intent-targeting keywords of the target search term is 100%, the keyword matching degree of sub-recall result b can be determined to be 10.

[0116] 2022B. Based on the keyword matching degree of each sub-recall result, determine the relevance of each sub-recall result.

[0117] For example, the keyword matching degree of each sub-recall result can be directly used as the relevance of each sub-recall result. Alternatively, the keyword matching degree of each sub-recall result multiplied by a preset coefficient can be used as the relevance of each sub-recall result.

[0118] The above examples illustrate how to determine the relevance of each sub-recall result, using both intent matching and keyword matching as examples. It's understandable that the relevance of each sub-recall result can also be determined by combining both intent matching and keyword matching; for example, the sum of intent relevance and keyword relevance of each sub-recall result can be used as the relevance of that sub-recall result. Intent relevance is obtained by multiplying the intent matching of each sub-recall result by a first preset coefficient, and keyword relevance is obtained by multiplying the keyword matching of each sub-recall result by a second preset coefficient.

[0119] Furthermore, the relevance of each sub-recall result can be determined by combining one, two, or more of the following: the intent matching degree between each sub-recall result and the target search term; the keyword matching degree between each sub-recall result and the target search term; examples will not be given here.

[0120] 203. Based on the relevance of each sub-recall result, obtain the ideal recall ranking of each sub-recall result.

[0121] The ideal recall ranking of each sub-recall result refers to the recall ranking of each sub-recall result determined based on the relevance of each sub-recall result.

[0122] Among them, the order of recall of each sub-recall result is positively correlated with the relevance of each sub-recall result. The higher the relevance of each sub-recall result, the higher the recall order of each sub-recall result. The lower the relevance of each sub-recall result, the lower the recall order of each sub-recall result.

[0123] For example, as shown in Table 5, assuming the relevance of the sub-recall results of the target search term "661 Youyi Avenue Dongrun Shangyu" (in order: "Dongrun Shangyu | 661 Youyi Avenue", "Dongrun Shangyu Building B1 | 661 Youyi Avenue Dongrun Shangyu", "Dongrun Shangyu (Parking Lot) | 661 Youyi Avenue", "A Certain Express (Youyi Avenue) | 661 Youyi Avenue Dongrun Shangyu", "Dongrun Shangyu Bus Station | Hongshan Xudong Business District Youyi Avenue") are 1, 0.9, 0.8, 0.7, and 0 respectively, then the ideal recall ranking of the sub-recall results (in order: "Dongrun Shangyu | 661 Youyi Avenue", "Dongrun Shangyu Building B1 | 661 Youyi Avenue Dongrun Shangyu", "Dongrun Shangyu (Parking Lot) | 661 Youyi Avenue", "A Certain Express (Youyi Avenue) | 661 Youyi Avenue Dongrun Shangyu", "Dongrun Shangyu Bus Station | Hongshan Xudong Business District Youyi Avenue") is 1, 2, 3, 4, and 5 respectively.

[0124] 204. Determine the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result.

[0125] The target recall result includes at least one sub-recall result.

[0126] The ranking effect of the target recall results is used to indicate the search ranking effect of the search tool. In step 204, the ranking effect of the search tool can be reflected directly based on the ranking effect of the recall results of a single search term, or it can be reflected based on the ranking effect of the recall results of two or more search terms. An example is given below:

[0127] (i) To reflect the search ranking effect of the search tool based on the ranking effect of the recall results of a search term, step 204 may specifically include: substituting the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result into the preset ranking effect formula shown in the following formula (1), to calculate the ranking effect of the target recall result, wherein the preset ranking effect formula is:

[0128]

[0129] In formula (1), 1 ≤ k ≤ M, k is a positive integer, P represents the ranking effect of the target recall result, M represents the total number of sub-recall results of the target recall result, k represents the k-th sub-recall result among the M sub-recall results of the target search term, and R k IdealR represents the actual recall ranking of the k-th sub-recall result. k Represents the ideal recall order of the k-th sub-recall result, rel kThis represents the relevance of the k-th sub-recall result.

[0130] The ideal recall ranking of the k-th sub-recall result refers to the recall ranking of the k-th sub-recall result determined based on its relevance.

[0131] The actual recall ranking of the k-th sub-recall result refers to the recall ranking of the k-th sub-recall result returned by the search tool (such as a map search engine, website search engine, etc.).

[0132] Alternatively, the ranking effect of the target recall result can be calculated based on the preset ranking effect formula shown in formula (2). In this case, the target search terms include N=1. The ranking effect of the target recall result can be calculated and can also be directly used to evaluate the search ranking effect of the search tool.

[0133] For example, please refer to Tables 2 and 3. Taking the number of target search terms N=1, the target search term as “Friendship Avenue 661 Dongrun Shangyu”, and the target recall results as containing M=5 sub-recall results, assume that the actual recall ranking of each sub-recall result is as shown in Table 2 and the actual recall ranking of each sub-recall result is as shown in Table 3.

[0134] Among them, the M sub-recall results are: "Dongrun Shangyu | No. 661 Youyi Avenue" (referred to as result a), "Dongrun Shangyu Building B1 | No. 661 Youyi Avenue Dongrun Shangyu" (referred to as result b), "Dongrun Shangyu Bus Station | Hongshan Xudong Business District Youyi Avenue" (referred to as result c), "Dongrun Shangyu (Parking Lot) | No. 661 Youyi Avenue" (referred to as result d), "A Certain Express (Youyi Avenue) | No. 661 Youyi Avenue Dongrun Shangyu" (referred to as result e).

[0135] The relevance of results a, b, c, d, and e are 2, 1, 0, 1, and 1, respectively. The actual recall rankings of results a, b, c, d, and e are 1, 2, 3, 4, and 5, respectively. The ideal recall rankings of results a, b, c, d, and e are 1, 2, 5, 3, and 4, respectively.

[0136] Then, the actual recall ranking and ideal recall ranking of results a, b, c, d, and e can be substituted into formula (1) to calculate the ranking effect of the target recall results. The specific calculation is as follows:

[0137] P = (Relevance of result a * Ideal recall ranking of result a + Relevance of result b * Ideal recall ranking of result b + Relevance of result c * Ideal recall ranking of result c + Relevance of result d * Ideal recall ranking of result d + Relevance of result e * Ideal recall ranking of result e) / (Relevance of result a * Actual recall ranking of result a + Relevance of result b * Actual recall ranking of result b + Relevance of result c * Actual recall ranking of result c + Relevance of result d * Actual recall ranking of result d + Relevance of result e * Actual recall ranking of result e) = (2*1 + 1*2 + 0*5 + 1*3 + 1*4) / (2*1 + 1*2 + 0*3 + 1*4 + 1*5) = 11 / 13 = 0.846.

[0138] Table 2

[0139]

[0140] Table 3

[0141]

[0142] (ii) The ranking effect of the search tool is reflected by the ranking effect of the recall results based on two or more search terms, that is, the target search terms include N, where N is an integer ≥ 2. Step 204 may specifically include: substituting the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result into the preset ranking effect formula shown in the following formula (2), to calculate the ranking effect of the target recall result. The preset ranking effect formula is:

[0143]

[0144] In formula (2), 1≤k≤M, 1≤i≤N, i and k are positive integers, P represents the ranking effect of the target recall result, n represents the total number of sub-recall results of the target recall result, k represents the k-th sub-recall result of the target search term, and R k IdealR represents the actual recall ranking of the k-th sub-recall result. k Represents the ideal recall order of the k-th sub-recall result, rel k Let N represent the relevance of the k-th sub-recall result, N represent the number of target search terms, and i represent the i-th target search term.

[0145] Since evaluating the search ranking effect of a search tool based on the recall results of a single target search term may be somewhat one-sided, combining the ranking results of N (N≥2) target recall results to evaluate the search ranking effect of the search tool makes the data on which the search ranking effect is based more comprehensive. This can improve the accuracy of the evaluation of the search ranking effect of the search tool to a certain extent, and provide more accurate data basis for subsequent adjustments or improvements to the search tool.

[0146] Furthermore, to improve search ranking results, after determining the ranking effect of the target recall results, the search ranking parameters (such as the ranking parameters of the search tool) can be adjusted. The actual recall ranking of each sub-recall result within the target recall results is determined based on preset search parameters, such as... Figure 5 As shown, after step 204, the following steps A1 to A2 may be further included, wherein:

[0147] A1. If the ranking effect is lower than the preset effect threshold, then obtain the ranking error between the actual recall ranking of each sub-recall result and the ideal recall ranking of each sub-recall result.

[0148] The specific value of the preset effect threshold can be set according to the actual business scenario requirements, and there is no restriction on the specific value of the preset effect threshold here. For example, when the ranking effect of the target recall result is in the range of [0,1], the ranking effect below 60% of the maximum value in the range can be considered as below the preset effect threshold, that is, the specific value of the preset threshold can be set to 60%*1=0.6.

[0149] For example, the difference between the actual recall ranking and the ideal recall ranking of each sub-recall result can be directly used as the ranking error of each sub-recall result. For instance, as shown in Tables 4 and 5 below, in the recall results for the target search term "661 Youyi Avenue Dongrun Shangyu", the difference between the actual recall ranking (i.e., 1) and the ideal recall ranking (i.e., 1-1=0) of the sub-recall result "Dongrun Shangyu|661 Youyi Avenue" can be directly used as the ranking error of the sub-recall result (i.e., "Dongrun Shangyu|661 Youyi Avenue"). For example, in the recall results for the target search term "661 Youyi Avenue Dongrun Shangyu", the sub-recall result "Dongrun Shangyu Bus Station | Hongshan Xudong Business Circle Youyi Avenue" can be directly used as the ranking error of the sub-recall result ("Dongrun Shangyu Bus Station | Hongshan Xudong Business Circle Youyi Avenue") because the difference between the actual recall ranking (3) and the ideal recall ranking (5) is 3-5=-2. Similarly, in the recall results for the target search term "661 Youyi Avenue Dongrun Shangyu", the sub-recall result "Dongrun Shangyu (Parking Lot) | 661 Youyi Avenue" can be directly used as the ranking error of the sub-recall result ("Dongrun Shangyu (Parking Lot) | 661 Youyi Avenue").

[0150] Alternatively, the difference between the ideal recall ranking and the actual recall ranking of each sub-recall result can be used as the ranking error of each sub-recall result. For example, as shown in Tables 4 and 5 below, for the sub-recall result "Dongrun Shangyu (Parking Lot) | No. 661 Youyi Avenue" in the recall results for the target search term "661 Youyi Avenue Dongrun Shangyu", the difference between the ideal recall ranking (i.e., 3) and the actual recall ranking (i.e., 4) of this sub-recall result (i.e., 3-4=-1) can be directly used as the ranking error of this sub-recall result (i.e., "Dongrun Shangyu (Parking Lot) | No. 661 Youyi Avenue").

[0151] Table 4

[0152]

[0153] Table 4 shows the actual recall ranking of each sub-recall result in the recall results of the target search term "Friendship Avenue 661 Dongrun Shangyu".

[0154] Table 5

[0155]

[0156] Table 5 shows the ideal recall ranking of each sub-recall result in the recall results of the target search term "Friendship Avenue 661 Dongrun Shangyu".

[0157] Furthermore, the ranking error of each sub-recall result can be determined by combining the relevance of each sub-recall result, the actual recall ranking of each sub-recall result, and the ideal recall ranking of each sub-recall result. The following two specific examples illustrate how to determine the ranking error of each sub-recall result by combining the relevance of each sub-recall result, the actual recall ranking of each sub-recall result, and the ideal recall ranking of each sub-recall result.

[0158] ① The difference between the actual recall ranking and the ideal recall ranking of each sub-recall result can be used as the initial error of each sub-recall result; then, the product of the initial error and the relevance of each sub-recall result can be used as the ranking error of each sub-recall result. For example, as shown in Tables 4 and 5, for the sub-recall result "Dongrun Shangyu Bus Station | Hongshan Xudong Business Circle Friendship Avenue" in the recall results for the target search term "Youyi Avenue 661 Dongrun Shangyu", the difference between the actual recall ranking (i.e., 3) and the ideal recall ranking (i.e., 5) of this sub-recall result (i.e., 3-5=-2) can be used as the initial error of this sub-recall result; then, the product of the initial error and the relevance of this sub-recall result (i.e., =-2*0=0) can be used as the ranking error of this sub-recall result (i.e., "Dongrun Shangyu Bus Station | Hongshan Xudong Business Circle Friendship Avenue").

[0159] ② The difference between the ideal recall ranking and the actual recall ranking of each sub-recall result can be used as the initial error of each sub-recall result; then the product of the initial error and the relevance of each sub-recall result can be used as the ranking error of each sub-recall result. For example, as shown in Tables 4 and 5, for the sub-recall result "Dongrun Shangyu (Parking Lot) | No. 661 Youyi Avenue" in the recall results of the target search term "Youyi Avenue 661 Dongrun Shangyu", the difference between the ideal recall ranking (i.e., 3) and the actual recall ranking (i.e., 4) of the sub-recall result (i.e., 3-4=-1) can be used as the initial error of the sub-recall result; then the product of the initial error and the relevance of the sub-recall result (i.e., =-1*0.8=-0.8) can be used as the ranking error of the sub-recall result (i.e., "Dongrun Shangyu (Parking Lot) | No. 661 Youyi Avenue").

[0160] It is evident that by combining the relevance of each sub-recall result to determine the ranking error of each sub-recall result, the ranking error of the highly relevant sub-recall results can be given a higher weight. Consequently, when adjusting the preset ranking parameters based on the ranking error of each sub-recall result, the recall ranking of the highly relevant sub-recall results can be given priority. This ensures that the updated ranking parameters prioritize ensuring that the actual recall ranking of the highly relevant sub-recall results is more consistent with the actual recall ranking, thereby improving the ranking accuracy of the updated ranking parameters.

[0161] A2. Based on the sorting error of each sub-recall result, adjust the preset sorting parameters to obtain the updated sorting parameters.

[0162] Wherein, the error between the recall order of each sub-recall result determined based on the updated sorting parameters and the ideal recall order of each sub-recall result is less than a preset error.

[0163] The preset sorting parameters are used to sort the sub-recall results of the target search term to obtain the actual recall ranking of each sub-recall result. For example, the preset sorting parameters can be the search sorting parameters of a map search engine.

[0164] For example, in step A2, the sum of the ranking errors of each sub-recall result of the target search term is used as the ranking error of the preset ranking parameter. Then, according to the preset adjustment target (e.g., the preset adjustment target could be that the ranking error of the adjusted preset ranking parameter is minimized, or that the ranking error of the adjusted preset ranking parameter is less than a preset ranking error threshold, etc.), such as taking the minimum ranking error of the adjusted preset ranking parameter as the preset adjustment target, the preset ranking parameter is adjusted until the ranking error of the adjusted preset ranking parameter is minimized, and then the adjusted preset ranking parameter can be used as the updated ranking parameter.

[0165] Since the updated ranking parameters are adjusted based on the ranking errors of the recall results of historical search terms (such as the target search term being used as a historical search term), the ranking accuracy of the search term recall results can be effectively improved when the updated ranking parameters are subsequently applied to rank the search term recall results.

[0166] The sorting error of the adjusted preset sorting parameters is determined in a similar way to the sorting error of the preset sorting parameters. For details, please refer to the relevant explanations above. For the sake of simplicity, it will not be repeated here.

[0167] Furthermore, to improve search ranking results, after determining the ranking effect of the target recall results, the actual recall ranking of each sub-recall result can be adjusted to make the output of each sub-recall result more in line with the user's intent. For example... Figure 6As shown, after step 204, the following steps B1 to B3 may be further included, wherein:

[0168] B1. If the ranking effect is lower than the preset effect threshold, then obtain the sub-recall result ranked first in the ideal recall ranking from the target recall result, and use it as a reference sub-recall result.

[0169] The reference sub-recall result refers to the sub-recall result that ranks first in the ideal recall order among the target recall results.

[0170] B2. Obtain the topic type of the target search term and the topic type of each sub-recall result.

[0171] The topic types can be preset in multiple ways. Each topic type indicates the content category of the sub-recall results. The topic types can be preset based on the content categories involved in the sub-recall results in the actual business scenario. For example, in a map search scenario, the sub-recall results can be preset with multiple topic types such as store name, building name, road name, and company name.

[0172] The topic type of the target search term refers to the content category to which the target search term belongs. The topic type of the target search term belongs to at least one of several preset topic types. For example, in a map search scenario, the sub-recall results can be preset with multiple topic types such as business district name, shop name, building name, road name, company name, and parking lot name. In this case, the topic type of the target search term "Friendship Avenue 661 Dongrun Shangyu" is "business district name".

[0173] The topic type of each sub-recall result refers to the content category to which each sub-recall result belongs. Each topic type of sub-recall result belongs to at least one of the preset topic types. For example, in a map search scenario, the sub-recall results can be preset with multiple topic types such as store name, building name, road name, company name, and parking lot name. In this case, the topic type of the sub-recall result (such as the sub-recall result "Dongrun Shangyu (Parking Lot) | No. 661 Youyi Avenue" in Table 4) is "Parking Lot Name".

[0174] B3. Adjust the actual recall ranking of each sub-recall result according to the topic type of the target search term and the topic type of each sub-recall result to obtain the updated recall ranking of each sub-recall result.

[0175] The ranking of each sub-recall result after the update is positively correlated with the similarity between the topic type of each sub-recall result and the topic type of the target search term.

[0176] Taking a map search engine as an example, the ranking effect of the target recall results returned by the map search engine for the target search term can be determined by referring to steps 201 to 204 above. Assuming that the ranking effect of the target recall results returned by the map search engine for the target search term is lower than the preset effect threshold, then firstly, the sub-recall result ranked first in the ideal recall ranking can be obtained from the target recall results as a reference sub-recall result.

[0177] Then, the topic types of the target search term and the topic types of each sub-recall result are detected. For example, the topic types of each sub-recall result for the target search term "Friendship Avenue 661 Dongrun Shangyu" are shown in Table 6 below.

[0178] Table 6

[0179]

[0180] Finally, based on the principle that the closer the topic type of each sub-recall result is to the topic type of the target search term, the higher the ranking of each sub-recall result, the actual recall ranking of each sub-recall result is adjusted to obtain the updated recall ranking of each sub-recall result. This is to make the updated recall ranking of each sub-recall result more in line with the ideal recall ranking of each sub-recall result, that is, more in line with the user's search intent.

[0181] Furthermore, when there are multiple sub-recall results with the same degree of similarity to the topic type of the target search term, the actual recall ranking of each sub-recall result can be adjusted according to the rule that the higher the relevance of each sub-recall result, the higher its ranking. This results in an updated recall ranking for each sub-recall result, making the updated recall ranking of each sub-recall result more in line with the ideal recall ranking of each sub-recall result, that is, more in line with the user's search intent.

[0182] As can be seen from the above, this embodiment obtains the actual recall ranking of each sub-recall result in the target recall result of the target search term; obtains the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term; obtains the ideal recall ranking of each sub-recall result based on the relevance of each sub-recall result; and determines the ranking effect of the target recall result based on the actual recall ranking, the ideal recall ranking, and the relevance of each sub-recall result. Therefore, even when the recall result includes multiple relevance levels, the ranking effect of the recall result can still be effectively evaluated without requiring the recall result to be either completely relevant or completely irrelevant. Thus, dividing the relevance of the recall result into two or more search scenarios can effectively evaluate the search ranking effect, solving the problem that the average precision of the mean cannot effectively evaluate the search ranking effect (such as when the relevance of the recall result is divided into two or more search scenarios).

[0183] To better implement the search ranking effect evaluation method in the embodiments of this application, based on the search ranking effect evaluation method, the embodiments of this application also provide a search ranking effect evaluation device, such as... Figure 7 The diagram shown is a structural schematic of one embodiment of the search ranking effect evaluation device in this application. The search ranking effect evaluation device 700 includes:

[0184] The first acquisition unit 701 is used to acquire the actual recall ranking of each sub-recall result in the target recall result of the target search term, wherein the target recall result includes M sub-recall results, M≥1;

[0185] The second acquisition unit 702 is used to acquire the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term;

[0186] The third acquisition unit 703 is used to acquire the ideal recall ranking of each sub-recall result based on the relevance of each sub-recall result;

[0187] Evaluation unit 704 is used to determine the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result.

[0188] In some embodiments of this application, the second acquisition unit 702 is specifically used for:

[0189] Obtain the intent matching degree between each sub-recall result and the target search term, and use it as the intent matching degree of each sub-recall result;

[0190] Based on the intent matching degree of each sub-recall result, the relevance of each sub-recall result is determined.

[0191] In some embodiments of this application, each sub-recall result is either a first sub-recall result or a second sub-recall result. The first sub-recall result is the sub-recall result with the highest intent matching degree among the target recall results, and the second sub-recall result is each sub-recall result in the target recall result except for the first sub-recall result. The second acquisition unit 702 is specifically used for:

[0192] Obtain the precise intent type of the target search term;

[0193] If the intent is of the exact type, then the relevance of the first sub-recall result is set to the first relevance preset value;

[0194] Alternatively, the relevance of the second sub-recall result can be set to a second relevance preset value, wherein the first relevance preset value is greater than the second relevance preset value.

[0195] In some embodiments of this application, each sub-recall result is a third sub-recall result or a fourth sub-recall result. The third sub-recall result is each sub-recall result in the target recall result whose intent matching degree is within a first preset range. The fourth sub-recall result is each sub-recall result in the target recall result other than the third sub-recall result. The intent matching degree of the fourth sub-recall result is less than any intent matching degree within the first preset range. The second acquisition unit 702 is specifically used for:

[0196] Obtain the precise intent type of the target search term;

[0197] If the intent is of the exact type, then the relevance of the third sub-recall result is set to the third relevance preset value;

[0198] Alternatively, the relevance of the fourth sub-recall result can be set as a fourth relevance preset value, wherein the third relevance preset value is greater than the fourth relevance preset value.

[0199] In some embodiments of this application, each sub-recall result is a fifth sub-recall result, a sixth sub-recall result, or a seventh sub-recall result. The fifth sub-recall result is the sub-recall result with the highest intent matching degree among the target recall results. The sixth sub-recall result is each sub-recall result among the target recall results whose intent matching degree is within a second preset range. The seventh sub-recall result is each sub-recall result among the target recall results whose intent matching degree is within a third preset range. The intent matching degree of the fifth sub-recall result is greater than any intent matching degree within the second preset range, and any intent matching degree within the second preset range is greater than any intent matching degree within the third preset range. The second acquisition unit 702 is specifically used for:

[0200] Obtain the precise intent type of the target search term;

[0201] If the intent is of the exact type, then the relevance of the fifth sub-recall result is set to the fifth relevance preset value;

[0202] Alternatively, the relevance of the sixth sub-recall result can be set to a sixth relevance preset value, wherein the fifth relevance preset value is greater than the sixth relevance preset value;

[0203] Alternatively, the relevance of the seventh sub-recall result can be set as a seventh relevance preset value, wherein the sixth relevance preset value is greater than the seventh relevance preset value.

[0204] In some embodiments of this application, the search ranking effect evaluation device 700 further includes a first adjustment unit (not shown in the figure). The actual recall ranking of each sub-recall result is determined based on preset ranking parameters. After determining the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result, the first adjustment unit is specifically used for:

[0205] If the ranking effect is lower than the preset effect threshold, then the ranking error between the actual recall ranking of each sub-recall result and the ideal recall ranking of each sub-recall result is obtained;

[0206] Based on the sorting error of each sub-recall result, the preset sorting parameters are adjusted to obtain updated sorting parameters, wherein the error between the recall sorting of each sub-recall result determined based on the updated sorting parameters and the ideal recall sorting of each sub-recall result is less than the preset error.

[0207] In some embodiments of this application, the search ranking effect evaluation device 700 further includes a second adjustment unit (not shown in the figure). After determining the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result, the second adjustment unit is specifically used for:

[0208] If the ranking effect is lower than the preset effect threshold, then the sub-recall result ranked first in the ideal recall ranking is obtained from the target recall result and used as a reference sub-recall result.

[0209] Obtain the topic type of the target search term and the topic type of each sub-recall result;

[0210] Based on the topic type of the target search term and the topic type of each sub-recall result, the actual recall ranking of each sub-recall result is adjusted to obtain the updated recall ranking of each sub-recall result; wherein, the degree to which the updated recall ranking of each sub-recall result ranks higher is positively correlated with the degree of similarity between the topic type of each sub-recall result and the topic type of the target search term.

[0211] In some embodiments of this application, the target search terms include N, and the evaluation unit 704 is specifically used for:

[0212] The actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result are substituted into a preset ranking effect formula to calculate the ranking effect of the target recall result. The preset ranking effect formula is as follows:

[0213]

[0214] Where P represents the ranking effect of the target recall result, M represents the total number of sub-recall results of the target recall result, k represents the k-th sub-recall result among the M sub-recall results of the target search term, and R k IdealR represents the actual recall ranking of the k-th sub-recall result. k Represents the ideal recall order of the k-th sub-recall result, rel k Let N represent the relevance of the k-th sub-recall result, N represent the number of target search terms, and i represent the i-th target search term.

[0215] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0216] Since this search ranking effect evaluation device can perform the functions described in this application, Figures 1 to 6 Corresponding to the steps in the search ranking effect evaluation method in any embodiment, this application can achieve the following: Figures 1 to 6 For details on the beneficial effects that the search ranking effect evaluation method can achieve in any embodiment, please refer to the preceding description, which will not be repeated here.

[0217] Furthermore, to better implement the search ranking effect evaluation method in the embodiments of this application, based on the search ranking effect evaluation method, the embodiments of this application also provide an electronic device, see below. Figure 8 , Figure 8 This illustration shows a structural diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 801, which executes a computer program stored in a memory 802 to implement, for example... Figures 1 to 6 Corresponding to each step of the search ranking effect evaluation method in any embodiment; or, when the processor 801 executes the computer program stored in the memory 802, it implements as follows: Figure 7 The functions of each unit in the corresponding embodiment.

[0218] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 802 and executed by processor 801 to complete the embodiments of this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0219] The electronic device may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc., with processor 801, memory 802, input / output devices, and network access devices connected via a bus.

[0220] The processor 801 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0221] The memory 802 can be used to store computer programs and / or modules. The processor 801 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 802 and by calling the data stored in the memory 802. The memory 802 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0222] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the search ranking effect evaluation device, electronic equipment, and their corresponding units described above can be found in, for example... Figures 1 to 6 The description of the search ranking effect evaluation method corresponding to any embodiment will not be repeated here.

[0223] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0224] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figures 1 to 6 For the steps in the search ranking effect evaluation method corresponding to any embodiment, please refer to the following for specific operations: Figures 1 to 6 The description of the search ranking effect evaluation method corresponding to any embodiment will not be repeated here.

[0225] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0226] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figures 1 to 6 Corresponding to the steps in the search ranking effect evaluation method in any embodiment, this application can achieve the following: Figures 1 to 6 For details on the beneficial effects that the search ranking effect evaluation method can achieve in any embodiment, please refer to the preceding description, which will not be repeated here.

[0227] The above provides a detailed description of a search ranking effect evaluation method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for evaluating search ranking performance, characterized in that, The method includes: Obtain the actual recall ranking of each sub-recall result in the target recall result of the target search term, wherein the target recall result includes M sub-recall results, M≥1; Based on the matching degree between each sub-recall result and the target search term, the relevance of each sub-recall result is obtained; Based on the relevance of each sub-recall result, the ideal recall ranking of each sub-recall result is obtained; The ranking effect of the target recall result is determined based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result; After determining the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result, the method further includes: If the ranking effect is lower than the preset effect threshold, then the sub-recall result ranked first in the ideal recall ranking is obtained from the target recall result and used as a reference sub-recall result. Obtain the topic type of the target search term and the topic type of each sub-recall result; Based on the topic type of the target search term and the topic type of each sub-recall result, the actual recall ranking of each sub-recall result is adjusted to obtain the updated recall ranking of each sub-recall result; wherein, the degree to which the updated recall ranking of each sub-recall result ranks higher is positively correlated with the degree of similarity between the topic type of each sub-recall result and the topic type of the target search term.

2. The search ranking effect evaluation method according to claim 1, characterized in that, The process of obtaining the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term includes: Obtain the intent matching degree between each sub-recall result and the target search term, and use it as the intent matching degree of each sub-recall result; Based on the intent matching degree of each sub-recall result, the relevance of each sub-recall result is determined.

3. The search ranking effect evaluation method according to claim 2, characterized in that, Each sub-recall result is either a first sub-recall result or a second sub-recall result. The first sub-recall result is the sub-recall result with the highest intent matching degree among the target recall results, and the second sub-recall result is each sub-recall result in the target recall result except for the first sub-recall result. The determination of the relevance of each sub-recall result based on the intent matching degree of each sub-recall result includes: Obtain the precise intent type of the target search term; If the intent is of the exact type, then the relevance of the first sub-recall result is set to the first relevance preset value; Alternatively, the relevance of the second sub-recall result can be set to a second relevance preset value, wherein the first relevance preset value is greater than the second relevance preset value.

4. The search ranking effect evaluation method according to claim 2, characterized in that, Each sub-recall result is either a third sub-recall result or a fourth sub-recall result. The third sub-recall result is each sub-recall result in the target recall result whose intent matching degree is within a first preset range. The fourth sub-recall result is each sub-recall result in the target recall result other than the third sub-recall result. The intent matching degree of the fourth sub-recall result is less than any intent matching degree within the first preset range; The determination of the relevance of each sub-recall result based on the intent matching degree of each sub-recall result includes: Obtain the precise intent type of the target search term; If the intent is of the exact type, then the relevance of the third sub-recall result is set to the third relevance preset value; Alternatively, the relevance of the fourth sub-recall result can be set as a fourth relevance preset value, wherein the third relevance preset value is greater than the fourth relevance preset value.

5. The search ranking effect evaluation method according to claim 2, characterized in that, Each sub-recall result is a fifth sub-recall result, a sixth sub-recall result, or a seventh sub-recall result. The fifth sub-recall result is the sub-recall result with the highest intent matching degree among the target recall results. The sixth sub-recall result is each sub-recall result among the target recall results whose intent matching degree is within a second preset range. The seventh sub-recall result is each sub-recall result among the target recall results whose intent matching degree is within a third preset range. The intent matching degree of the fifth sub-recall result is greater than any intent matching degree within the second preset range, and any intent matching degree within the second preset range is greater than any intent matching degree within the third preset range. The determination of the relevance of each sub-recall result based on the intent matching degree of each sub-recall result includes: Obtain the precise intent type of the target search term; If the intent is of the exact type, then the relevance of the fifth sub-recall result is set to the fifth relevance preset value; Alternatively, the relevance of the sixth sub-recall result can be set to a sixth relevance preset value, wherein the fifth relevance preset value is greater than the sixth relevance preset value; Alternatively, the relevance of the seventh sub-recall result can be set as a seventh relevance preset value, wherein the sixth relevance preset value is greater than the seventh relevance preset value.

6. The search ranking effect evaluation method according to claim 1, characterized in that, The actual recall ranking of each sub-recall result is determined based on preset ranking parameters. After determining the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result, the method further includes: If the ranking effect is lower than the preset effect threshold, then the ranking error between the actual recall ranking of each sub-recall result and the ideal recall ranking of each sub-recall result is obtained; Based on the sorting error of each sub-recall result, the preset sorting parameters are adjusted to obtain updated sorting parameters, wherein the error between the recall sorting of each sub-recall result determined based on the updated sorting parameters and the ideal recall sorting of each sub-recall result is less than the preset error.

7. The method for evaluating search ranking effectiveness according to any one of claims 1-6, characterized in that, The target search terms include N terms. Determining the ranking effect of the target recall results based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result includes: The actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result are substituted into a preset ranking effect formula to calculate the ranking effect of the target recall result. The preset ranking effect formula is as follows: Where P represents the ranking effect of the target recall result, M represents the total number of sub-recall results of the target recall result, k represents the k-th sub-recall result among the M sub-recall results of the target search term, and R k IdealR represents the actual recall ranking of the k-th sub-recall result. k Represents the ideal recall order of the k-th sub-recall result, rel k Let N represent the relevance of the k-th sub-recall result, N represent the number of target search terms, and i represent the i-th target search term.

8. A search ranking effect evaluation device, characterized in that, The search ranking effect evaluation device includes: The first acquisition unit is used to acquire the actual recall ranking of each sub-recall result in the target recall result of the target search term, wherein the target recall result includes M sub-recall results, M≥1; The second acquisition unit is used to acquire the relevance of each sub-recall result based on the matching degree between each sub-recall result and the target search term; The third acquisition unit is used to acquire the ideal recall ranking of each sub-recall result based on the relevance of each sub-recall result; An evaluation unit is used to determine the ranking effect of the target recall result based on the actual recall ranking of each sub-recall result, the ideal recall ranking of each sub-recall result, and the relevance of each sub-recall result; The second adjustment unit is used for: If the ranking effect is lower than the preset effect threshold, then the sub-recall result ranked first in the ideal recall ranking is obtained from the target recall result and used as a reference sub-recall result. Obtain the topic type of the target search term and the topic type of each sub-recall result; Based on the topic type of the target search term and the topic type of each sub-recall result, the actual recall ranking of each sub-recall result is adjusted to obtain the updated recall ranking of each sub-recall result; wherein, the degree to which the updated recall ranking of each sub-recall result ranks higher is positively correlated with the degree of similarity between the topic type of each sub-recall result and the topic type of the target search term.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the search ranking effect evaluation method as described in any one of claims 1 to 7 when it invokes the computer program in the memory.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps in the search ranking effect evaluation method according to any one of claims 1 to 7.

Images