Search evaluation methods, devices, equipment, storage media and products

Abstract

This application provides a search evaluation method, apparatus, device, storage medium, and product, belonging to the field of Internet technology, and applicable to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: determining the search business data included in multiple sampling layers based on the search frequency corresponding to search keywords in the search business data; extracting first sample data from multiple sampling layers based on a total sampling number determined by the minimum sampling number corresponding to each of the multiple sampling layers and a first sampling ratio for each of the multiple sampling layers; extracting second sample data from the first sample data corresponding to each of the multiple sampling layers based on a second sampling ratio determined by the minimum sampling number corresponding to each of the multiple sampling layers; and determining target evaluation parameters for the target search business based on first evaluation parameters corresponding to search keywords in the second sample data, thereby ensuring high accuracy of the obtained evaluation parameters for the target search business.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a search evaluation method, apparatus, device, storage medium, and product. Background Technology

[0002] With the development of internet technology, search services have been widely adopted. A primary concern in search services is accuracy. To improve accuracy, it's necessary to evaluate search performance and optimize the service based on the evaluation results.

[0003] In related technologies, when evaluating search services, due to the large amount of search service data, the data is sampled, and the evaluation results of the sampled data are used as the overall evaluation results of the search service. However, during sampling, the uneven number of samples can lead to low accuracy of the obtained evaluation results. Summary of the Invention

[0004] This application provides a search evaluation method, apparatus, device, storage medium, and product, resulting in high accuracy of the evaluation parameters for the target search service. The technical solution is as follows:

[0005] On the one hand, a method for evaluating search is provided, the method comprising:

[0006] Based on the number of searches corresponding to search keywords in multiple sets of search business data of the target search business, the search business data included in multiple sampling layers are determined, and each set of search business data includes search keywords and search results for the search keywords;

[0007] Based on the minimum sampling number corresponding to each of the multiple sampling layers, the total sampling number of the multiple sampling layers is determined, wherein the minimum sampling number is determined based on the accuracy parameter of the evaluation.

[0008] Based on the total number of samples from the multiple sampling layers and the first sampling ratio of the multiple sampling layers, the first sample data of each of the multiple sampling layers is extracted from the search business data included in the multiple sampling layers.

[0009] Based on the minimum number of samples corresponding to each of the plurality of sampling strata, the second sampling ratio corresponding to each of the plurality of sampling strata is determined;

[0010] Based on the second sampling ratio corresponding to each of the plurality of sampling layers, the second sample data of each of the plurality of sampling layers are extracted from the first sample data corresponding to each of the plurality of sampling layers.

[0011] Based on the first evaluation parameters corresponding to the search keywords in the second sample data, the target evaluation parameters of the target search service are determined. The first evaluation parameters are obtained by evaluating the search results in the second sample data.

[0012] On the other hand, a search evaluation device is provided, the device comprising:

[0013] The first determining module is used to determine the search business data included in multiple sampling layers based on the number of searches corresponding to search keywords in multiple sets of search business data of the target search business. Each set of search business data includes search keywords and search results for the search keywords.

[0014] The second determining module is used to determine the total number of samples for the plurality of sampling layers based on the minimum number of samples corresponding to each of the plurality of sampling layers, wherein the minimum number of samples is determined based on the accuracy parameters of the evaluation.

[0015] The first extraction module is used to extract the first sample data of the multiple sampling layers from the search business data included in the multiple sampling layers, based on the total number of samples in the multiple sampling layers and the first sampling ratio of the multiple sampling layers.

[0016] The third determining module is used to determine the second sampling ratio corresponding to each of the multiple sampling layers based on the minimum sampling number corresponding to each of the multiple sampling layers.

[0017] The second extraction module is used to extract the second sample data of the multiple sampling layers from the first sample data corresponding to the multiple sampling layers, based on the second sampling ratio corresponding to the multiple sampling layers respectively.

[0018] The fourth determining module is used to determine the target evaluation parameters of the target search service based on the first evaluation parameters corresponding to the search keywords in the second sample data. The first evaluation parameters are obtained by evaluating the search results in the second sample data.

[0019] In some embodiments, the accuracy parameters of the evaluation include the target evaluation accuracy, the target error, and the first confidence interval. The device further includes a fifth determining module, used to determine the minimum sampling number based on the target evaluation accuracy, the target error, and the first confidence interval. The target evaluation accuracy is used to represent the expected accuracy of the first evaluation parameter obtained from evaluating the evaluation object. The target error is used to represent the error between the target evaluation parameter and the actual evaluation parameter of the target search service. The first confidence interval is used to represent the confidence level of the target error.

[0020] In some embodiments, the third determining module is configured to, for each sampling stratum, determine a third sampling ratio between the minimum sampling number corresponding to the sampling stratum and the first sampling number, wherein the first sampling number is the number of first sample data in the sampling stratum; and multiply the third sampling ratio by a first multiple to obtain the second sampling ratio.

[0021] In some embodiments, the second determining module is configured to, for each sampling layer, multiply the minimum sampling number corresponding to the sampling layer by a second multiple to obtain the second sampling number corresponding to the sampling layer, wherein the second multiple is positively correlated with the first sampling ratio; and to use the sum of the second sampling numbers corresponding to the plurality of sampling layers as the total sampling number.

[0022] In some embodiments, the fourth determining module is configured to sort the multiple search keywords based on first evaluation parameters corresponding to the multiple search keywords respectively, wherein the sorting position of each search keyword is positively correlated with the first evaluation parameter of the search keyword; determine at least one target search keyword among the sorted multiple search keywords that is located within the second confidence interval based on a second confidence interval; and determine the target evaluation parameter based on the first evaluation parameter corresponding to the at least one target search keyword.

[0023] In some embodiments, if the first evaluation parameters corresponding to the plurality of search keywords are obtained based on an evaluation object, the fourth determining module is further configured to use the parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to the at least one target search keyword as the target evaluation parameter.

[0024] In some embodiments, if the first evaluation parameters corresponding to the plurality of search keywords are obtained based on multiple evaluation objects, and each evaluation object is used to evaluate the plurality of search keywords, then the fourth determining module is further configured to, for each evaluation object, take the parameter interval determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to the at least one target search keyword as the parameter interval corresponding to the evaluation object; and take the intersection of the parameter intervals corresponding to the plurality of evaluation objects as the target evaluation parameter.

[0025] In some embodiments, the apparatus further includes:

[0026] The first acquisition module is used to acquire the average value of the parameters corresponding to the evaluation object, wherein the average value of the parameters is the average value of the first evaluation parameters corresponding to the plurality of search keywords respectively;

[0027] The sixth determination module is used to determine the difference between the average values ​​of the parameters corresponding to each pair of evaluation objects, and obtain multiple differences;

[0028] The second acquisition module is used to acquire, if the plurality of differences include a target difference that exceeds a difference threshold, the second evaluation parameters for re-evaluating the plurality of search keywords by two target evaluation objects, wherein the two target evaluation objects are the evaluation objects corresponding to the target difference, and the second evaluation parameters are used to redetermine the target evaluation parameters of the target search business.

[0029] In some embodiments, the fourth determining module is configured to, for each sampling layer, determine a third evaluation parameter corresponding to the sampling layer based on the first evaluation parameter corresponding to the search keywords in the second sample data of the sampling layer; and to obtain the target evaluation parameter by weighted summation of the third evaluation parameters corresponding to the multiple sampling layers based on the first sampling ratios corresponding to the multiple sampling layers.

[0030] In some embodiments, the apparatus further includes:

[0031] The third acquisition module is used to acquire the recall parameter of each search keyword, wherein the recall parameter is used to indicate whether an item is found based on the search keyword;

[0032] The fourth acquisition module is used to acquire the item association parameter of the item if the recall parameter is used to indicate that an item was found based on the search keyword. The item association parameter is used to indicate the degree of association between the item and the search keyword.

[0033] The seventh determining module is used to determine the first evaluation parameter of the search keyword based on the recall parameter and the item association parameter.

[0034] In some embodiments, the search keyword retrieves at least one item, and the at least one item is sorted in a first order. The seventh determining module is used to determine the sorting parameter of the search keyword based on the first order, item association parameters, and a second order of the at least one item. The sorting position of each item in the second order is positively correlated with the item association parameters of the item. The sorting parameter is used to indicate the rationality of the sorting of the at least one item. The recall parameter and the sorting parameter are weighted and summed to obtain the first evaluation parameter of the search keyword.

[0035] In some embodiments, the apparatus further includes:

[0036] The matching module is used to match the item with the search keywords;

[0037] The eighth determining module is used to determine the item association parameter as a first association parameter if the subject of the item does not match the subject of the search keyword. The first association parameter is used to indicate that the item is not related to the search keyword. The subject includes at least one of the product words and brand words of the item.

[0038] The ninth determining module is used to determine the item association parameter as a second association parameter if the main body of the item matches the main body of the search keyword, but the modification attribute of the item does not match the modification attribute of the search keyword. The second association parameter is used to indicate that the item and the search keyword are not completely related.

[0039] The tenth determining module is used to determine the item association parameter as a third association parameter if the main body of the item matches the main body of the search keyword and the modification attribute of the item matches the modification attribute of the search keyword. The third association parameter is used to indicate that the item is related to the search keyword.

[0040] In some embodiments, the apparatus further includes:

[0041] The eleventh determination module is used to determine the first evaluation accuracy of each sampling layer based on the accuracy of the first evaluation parameters of the multiple search keywords corresponding to the sampling layer.

[0042] The weighted summation module is used to perform a weighted summation of the first evaluation accuracy rates corresponding to the multiple sampling layers based on the first sampling ratios corresponding to the multiple sampling layers, so as to obtain the second evaluation accuracy rate of the target search service.

[0043] The update module is used to update the minimum number of samples corresponding to the plurality of sampling layers if the accuracy of the second evaluation is lower than the accuracy threshold.

[0044] The evaluation module is used to re-evaluate the target search service based on the updated minimum sample size.

[0045] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor to implement the search evaluation method in the embodiments of this application.

[0046] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to implement the search evaluation method as described in the embodiments of this application.

[0047] On the other hand, a computer program product is provided, the computer program product including computer program code stored in a computer-readable storage medium, a processor of a computer device reading the computer program code from the computer-readable storage medium, the processor executing the computer program code, causing the computer device to perform the search evaluation method described in any of the above implementations.

[0048] This application provides a search evaluation method. This method, based on stratified sampling, determines a secondary sampling ratio based on the minimum sample size corresponding to each of the multiple sampling strata. Then, based on this secondary sampling ratio, secondary sampling is performed on the multiple sampling strata, ensuring that the sample data from each stratum meets the minimum sample size requirement, i.e., the sample size is evenly distributed across the multiple sampling strata. Because the sample size is evenly distributed across the multiple sampling strata, the evaluation parameters of the target search service are obtained based on the evaluation parameters of the evenly distributed sample data, resulting in high accuracy of the obtained evaluation parameters for the target search service. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart of a search evaluation method provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart of another search evaluation method provided in the embodiments of this application;

[0053] Figure 4 This is a flowchart of another search evaluation method provided in the embodiments of this application;

[0054] Figure 5 This is a block diagram of a search evaluation device provided in an embodiment of this application;

[0055] Figure 6 This is a block diagram of a terminal provided in an embodiment of this application;

[0056] Figure 7 This is a block diagram of a server provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0058] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0059] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the search business data involved in this application were obtained with full authorization.

[0060] The following is an explanation of the terms used in this application.

[0061] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0062] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0063] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0064] This application provides a search evaluation method that can obtain evaluation parameters for search services based on the evaluation of search services, and then provide optimization targets for search services based on these evaluation parameters. The search evaluation method of this application includes the following application scenarios.

[0065] For example, the search evaluation method provided in this application is applied to an APP (Application) that provides search services, for evaluating the search services in the APP, and then optimizing the search services of the APP based on the evaluation results.

[0066] In some embodiments, the APP can be an e-commerce APP, and correspondingly, the search service is an e-commerce search service. The e-commerce search service is used to find search results corresponding to search keywords. Optionally, the search results are the items found. When evaluating the e-commerce search service, multiple sets of search service data consisting of search keywords and search results are obtained. Then, stratified sampling is performed on the multiple sets of search service data to obtain sample data from multiple sampling layers. Then, based on the minimum sampling number corresponding to the multiple sampling layers, secondary sampling is performed on the multiple sampling layers on the basis of stratified sampling to obtain sample data with a uniform sampling number distribution. Therefore, the evaluation parameters obtained based on the evaluation parameters of the uniformly distributed sample data have high accuracy.

[0067] In some embodiments, the APP can be a transportation-related APP, and correspondingly, the search service is a transportation search service. The transportation search service is used to find search results corresponding to search keywords. Optionally, the search results are the locations found. When evaluating the transportation search service, multiple sets of search service data consisting of search keywords and search results are obtained. Then, stratified sampling is performed on the multiple sets of search service data to obtain sample data for multiple sampling layers. Then, based on the minimum sampling number corresponding to the multiple sampling layers, secondary sampling is performed on the multiple sampling layers on the basis of stratified sampling to obtain sample data with a uniform sampling number distribution. Therefore, the evaluation parameters obtained based on the evaluation parameters of the sample data with a uniform sampling number distribution have high accuracy.

[0068] The following section describes the implementation environment involved in this application. Please refer to it. Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of the search evaluation method provided in this application embodiment. This implementation environment involves a terminal 101 and a server 102. The terminal 101 and server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The search evaluation method provided in this application embodiment can be implemented by the terminal 101 alone, by the server 102, or by the terminal 101 and server 102 through data interaction; this application embodiment does not limit this approach.

[0069] In some embodiments, terminal 101 is a first terminal capable of performing searches based on search services. Accordingly, the first terminal can generate search service data, and the first terminal sends the generated search service data to server 102, which records the search service data. In one implementation, server 102 performs stratified sampling on the recorded search service data, and then performs secondary sampling based on the minimum sampling number corresponding to each of the multiple sampling layers. Based on the evaluation parameters of the search keywords in the secondary sample data, the evaluation parameters of the search service are determined to evaluate the search service. Optionally, server 102 feeds back the evaluation parameters to the first terminal for display.

[0070] In other embodiments, terminal 101 includes a first terminal and a second terminal, the second terminal being a terminal used by a technician; the second terminal obtains search service data recorded by server 102 from server 102, performs stratified sampling on the search service data, and then performs secondary sampling based on the minimum sampling number corresponding to each of the multiple sampling layers; based on the evaluation parameters of the search keywords in the secondary sample data, the evaluation parameters of the search service are determined, thereby realizing the evaluation of the search service.

[0071] It should be noted that the aforementioned terminal 101 includes mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc., but is not limited to these. The aforementioned server 102 can be an independent server, a server cluster of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 provides background services for the target applications installed on terminal 101. In some embodiments, server 102 primarily undertakes computational work, while terminal 101 undertakes secondary computational work; or, server 102 undertakes secondary computational services, while terminal 101 undertakes primary computational work; or, server 102 and terminal 101 collaborate on computation using a distributed computing architecture. This invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0072] Figure 2 This is a flowchart of a search evaluation method provided according to an embodiment of this application. See also... Figure 2 In this embodiment, the method is described using a computer device as an example. The evaluation method for this search includes the following steps:

[0073] 201. Based on the number of searches corresponding to search keywords in multiple sets of search business data of target search business, computer equipment determines the search business data included in multiple sampling layers.

[0074] In this embodiment, each set of search business data includes search keywords and search results for those keywords. Multiple sets of search business data may include different search keywords. Each set of search business data is obtained by combining at least one search process using search keywords. If the search business data is obtained by combining multiple search processes using search keywords, the computer device combines the original search business data corresponding to the search keywords with the search business data from the multiple search processes to obtain the set of search business data. The target search business is a search service that can be provided in a certain app; optionally, the search business is a search business in the e-commerce field. The search results are the results obtained by inputting the search keywords into the target search business; if the target search business is a search business in the e-commerce field, the search results are multiple items found. The search count corresponding to the search keyword is the number of times the search keyword has been searched historically.

[0075] 202. The computer equipment determines the total number of samples for multiple sampling strata based on the minimum number of samples corresponding to each stratum.

[0076] In this embodiment, the minimum sample size is determined based on the accuracy parameter of the evaluation. The accuracy parameter represents the precision of the first evaluation parameter of the search keyword. The accuracy parameter includes the target evaluation accuracy, the target error, and the first confidence interval; the target evaluation accuracy represents the expected accuracy of the first evaluation parameter obtained from evaluating the evaluation object, the target error represents the error between the target evaluation parameter and the actual evaluation parameter of the target search business, and the first confidence interval represents the confidence level of the target error.

[0077] 203. The computer equipment extracts the first sample data of each of the multiple sampling layers from the search business data included in the multiple sampling layers, based on the total number of samples in the multiple sampling layers and the first sampling ratio of the multiple sampling layers.

[0078] In this embodiment, the first sampling ratio of each sampling layer is proportional to the layer proportion of that sampling layer. The layer proportion refers to the proportion of the sum of the search counts corresponding to multiple search keywords in the search business data included in that sampling layer to the total number of searches. The total number of searches refers to the sum of the search counts corresponding to multiple search keywords in multiple sets of business data of the target search business. Thus, based on the total number of samples from multiple sampling layers and the first sampling ratio of multiple sampling layers, stratified sampling of multiple sets of search business data is achieved.

[0079] 204. The computer equipment determines the second sampling ratio corresponding to each of the multiple sampling strata based on the minimum sampling number corresponding to each of the multiple sampling strata.

[0080] In this embodiment of the application, the second sampling ratio is greater than or equal to the minimum sampling ratio.

[0081] 205. The computer device extracts second sample data from the first sample data corresponding to the multiple sampling layers based on the second sampling ratio corresponding to the multiple sampling layers.

[0082] In this embodiment of the application, secondary sampling based on stratified sampling is achieved by extracting second sample data.

[0083] 206. The computer equipment determines the target evaluation parameters of the target search business based on the first evaluation parameters corresponding to the search keywords in the second sample data.

[0084] In this embodiment of the application, the first evaluation parameter is obtained based on the search results in the second sample data.

[0085] This application provides a search evaluation method. This method, based on stratified sampling, determines a secondary sampling ratio based on the minimum sample size corresponding to each of the multiple sampling strata. Then, it performs secondary sampling on the multiple sampling strata based on this ratio, ensuring that the sample data from each stratum meet the minimum sample size requirement, i.e., ensuring a uniform distribution of sample sizes across the multiple sampling strata. Because the sample sizes are uniformly distributed across the multiple strata, the evaluation parameters of the target search service are obtained based on the evaluation parameters of the uniformly distributed sample data, resulting in high accuracy of the obtained evaluation parameters for the target search service.

[0086] Figure 3 This is a flowchart of a search evaluation method provided according to an embodiment of this application. The search evaluation method includes the following steps:

[0087] 301. Based on the number of searches corresponding to search keywords in multiple sets of search business data of target search business, computer equipment determines the search business data included in multiple sampling layers.

[0088] In this embodiment, the computer equipment divides multiple sets of search business data into multiple sampling layers based on the number of searches. In some embodiments, the proportion of the sum of the search counts corresponding to multiple search keywords in the search business data included in each sampling layer is different, that is, the layer proportions of the multiple sampling layers are different. In other embodiments, the proportion of the sum of the search counts corresponding to multiple search keywords in the search business data included in each sampling layer is the same, that is, the layer proportions of the multiple sampling layers are the same. In this embodiment, multiple sets of business data are divided into multiple sampling layers based on the search count of search keywords. Then, based on the sampling proportion corresponding to the layer proportion of each sampling layer, the first sample data of each sampling layer is extracted. Compared with the method of randomly extracting sample data from the overall data, the first sample data extracted according to the layer proportion is closer to the data structure distribution of the overall data, that is, the extracted sample data is more representative and the sampling error is smaller.

[0089] In some embodiments, the search count corresponding to the search keyword is the average search count within a target time period; optionally, the average search count is the search count per hour, per day, or per week. The computer device retrieves the total search count of the search keyword within the target time period from the database, and uses the quotient of this total search count and the target time period as the average search count; the target time period is a time period set as needed and is not specifically limited here. In this embodiment, by using the average search count of the search keyword within the target time period as the search count corresponding to the search keyword, the search count corresponding to the search keyword is reduced. Therefore, when determining the search business data included in multiple sampling layers based on the reduced search count, computational efficiency can be improved, thereby improving the efficiency of determining the search business data included in multiple sampling layers.

[0090] In some embodiments, the computer device determines the search threshold corresponding to each sampling layer based on the number of searches corresponding to search keywords in multiple sets of search business data of the target search business, and then obtains the search business data included in each sampling layer based on the search threshold corresponding to each sampling layer, wherein the number of searches of search keywords in the search business data matches the search threshold.

[0091] The process of determining the search threshold includes: The computer device sorts multiple search keywords in descending order based on the search counts corresponding to each keyword; the computer device sums the search counts of the descendingly sorted keywords to obtain the total search count for each keyword. The computer device determines a target proportion of the sum of the search counts of the keywords in each sampling layer to the total search count; then, the computer device multiplies this target proportion by the total search count as the total search count for each sampling layer; the computer device determines the search threshold for each sampling layer based on the total search count for each sampling layer and the search counts of the descendingly sorted keywords. Since each keyword in the descendingly sorted search keywords corresponds to one search count, for the first sampling layer, if the computer device obtains the total search count for the first sampling layer based on the sum of the search counts of the first-ranked keywords, then the search count of the last-ranked keyword among these keywords is the search threshold for that first sampling layer. For the second sampling layer, if the computer device obtains the total number of searches for the second sampling layer by summing the search counts of multiple search keywords after the last search keyword in the first sampling layer, then the search count of the last search keyword among these multiple search keywords is the search threshold corresponding to the second sampling layer; similarly, the computer device can obtain the search thresholds corresponding to multiple sampling layers.

[0092] It should be noted that by sorting multiple search keywords in descending order, the search frequency of the search keywords ranked higher is greater than that of the search keywords ranked lower. Therefore, when subsequent computer equipment retrieves search business data for any sampling layer based on search thresholds, it only needs to retrieve the search business data belonging to the search keywords ranked between the first and second search keywords. The first search keyword is the search keyword corresponding to the upper limit search threshold of the sampling layer, and the second search keyword is the search keyword corresponding to the lower limit search threshold of the sampling layer. This improves the efficiency of determining the search business data included in multiple sampling layers.

[0093] In some embodiments, the number of sampling layers is three, and the computer device determines the target proportions corresponding to the multiple sampling layers as 80%, 19%, and 1%, respectively. Then, the computer device multiplies the target proportion corresponding to each sampling layer by the total number of searches, which is taken as the total number of searches for each sampling layer. Furthermore, based on the total number of searches for each sampling layer and the search counts corresponding to multiple search keywords in descending order, the computer device determines the search threshold for each sampling layer. Specifically, the computer device designates the search keywords accounting for the top 80% of the total search counts as head keywords, the search keywords accounting for 80%-99% of the total search counts as middle keywords, and the search keywords accounting for 1% of the total search counts as tail keywords. The computer device rounds down the search counts to determine the search count x1 corresponding to the tail keyword in the head keyword category and the search count x2 corresponding to the tail keyword in the middle keyword category. Then, x1 is used as the lower limit search threshold for head keywords and the upper limit search threshold for middle keywords, and x2 is used as the lower limit search threshold for middle keywords and the upper limit search threshold for tail keywords. As shown in Table 1, the search count for keywords in the first-level terms is greater than or equal to x1, the search count for keywords in the middle-level terms is greater than or equal to x2 and less than x1, and the search count for keywords in the last-level terms is less than x2. Accordingly, the computer equipment forms a first sampling layer with the search data corresponding to the keywords in the first-level terms, a second sampling layer with the search data corresponding to the keywords in the middle-level terms, and a third sampling layer with the search data corresponding to the keywords in the last-level terms.

[0094] Table 1

[0095] Sampling layer Percentage of search volume Search frequency threshold Head words Top 80% ≥x1 Middle Part 80％～99％ x2≤x<x1 End words The last 1% ＜x2

[0096] 302. The computer equipment determines the total number of samples for multiple sampling strata based on the minimum number of samples corresponding to each stratum.

[0097] In some embodiments, the minimum sampling number is the same for each sampling stratum, and the computer device obtains the minimum sampling number based on the target evaluation accuracy, the target error, and the first confidence interval. Optionally, the computer device obtains the minimum sampling number based on the target evaluation accuracy, the target error, and the first confidence interval using the following formula (1).

[0098]

[0099] Where p represents the target evaluation accuracy, e represents the target error, z represents the first confidence interval, and n represents the minimum number of samples.

[0100] The target evaluation accuracy rate is derived from historical experience, ensuring its reliability as it is not arbitrarily determined. The target error is determined based on the 3 sigma principle of the normal distribution. The 3 sigma principle states that under a normal distribution, the probability of a point falling within μ+σ is 65.26%, within μ+2σ is 95.44%, and within μ±3σ is 99.74%, where μ represents the expected value and σ represents the standard deviation. Accordingly, the computer determines a first confidence interval based on probability, and the target error is obtained from the standard normal distribution table based on the first confidence interval and the 3 sigma principle. The target error and the first confidence interval can be set and changed as needed, such as setting the first confidence interval to 95%.

[0101] In one implementation, formula (1) is derived by modifying the following formula (2).

[0102]

[0103] Where p represents the target measurement accuracy, e represents the target error, z represents the first confidence interval, and σ represents the standard deviation. 2 =p(1-p), where σ represents the standard deviation and p represents the target measurement accuracy.

[0104] In other embodiments, the minimum number of samples corresponding to the multiple sampling layers is different. Accordingly, the computer device obtains the minimum number of samples corresponding to the multiple sampling layers by using the different target evaluation accuracy, target error and first confidence interval corresponding to the multiple sampling layers.

[0105] It should be noted that, since the stratum percentages of each sampling stratum may be different or the same, optionally, when the stratum percentages are the same, the computer device obtains the minimum sample size corresponding to each sampling stratum based on the same target evaluation accuracy, target error, and first confidence interval, and the minimum sample size corresponding to each sampling stratum is the same; when the stratum percentages are different, the computer device obtains the minimum sample size corresponding to each of the multiple sampling stratums based on different target evaluation accuracy, target error, and first confidence intervals. The different stratum percentages of different sampling stratums represent different impacts of the evaluation parameters of the sample data in different sampling stratums on the target evaluation parameters, and thus different accuracy requirements for the evaluation of different sampling stratums. Therefore, for sampling stratums with different stratum percentages, different target evaluation accuracy, target error, and first confidence intervals can be set to obtain the minimum sample size corresponding to each sampling stratum, and the minimum sample size corresponding to each sampling stratum is different. It should be noted that when the minimum sampling numbers corresponding to multiple sampling layers are different, the computer device sets the difference between the minimum sampling numbers corresponding to multiple sampling layers to not exceed the difference threshold, so as to ensure that the minimum sampling numbers corresponding to multiple sampling layers are distributed as evenly as possible. This ensures that when the second sample data is subsequently drawn based on the minimum sampling number, the number of second sample data drawn from the multiple sampling layers is as even as possible.

[0106] It should be noted that in related technologies, the number of sample data extracted is too arbitrary and difficult to determine. Furthermore, since the number is not based on the accuracy of the evaluation, the number of sample data extracted is difficult to meet the accuracy requirements of the evaluation. However, in the embodiments of this application, the minimum sampling number is determined by an accuracy parameter that can represent the accuracy requirements of this evaluation. Subsequently, sample data is extracted based on the minimum sampling number, making the number of sample data quantifiable and meeting the accuracy requirements of the evaluation.

[0107] In some embodiments, the computer device determines the total number of samples for multiple sampling strata based on the minimum number of samples corresponding to each sampling stratum, including the following steps: for each sampling stratum, the computer device multiplies the minimum number of samples corresponding to the sampling stratum by a second multiple to obtain the second number of samples corresponding to the sampling stratum; the computer device uses the sum of the second number of samples corresponding to each of the multiple sampling strata as the total number of samples.

[0108] The second multiple is positively correlated with the first sampling ratio, and the second multiple is greater than or equal to 1. Optionally, if the first sampling ratios of multiple sampling strata are 80%, 19%, and 1%, the second multiples of multiple sampling strata are 80, 19, and 1, respectively. Accordingly, the computer device multiplies the minimum sampling number n of each stratum by 80, 19, and 1, respectively, to obtain the second sampling number 80*n, 19*n, and n corresponding to each sampling stratum. Optionally, if the minimum sampling number corresponding to multiple sampling strata is 300, the second sampling number of each sampling stratum is 24000, 5700, and 300, respectively, and the total sampling number is 30000.

[0109] In this embodiment, the minimum sampling number corresponding to each sampling layer is multiplied by a second factor to expand the minimum sampling number corresponding to each sampling layer. The total sampling number is obtained based on the expanded second sampling number. Subsequently, when the computer device extracts the first sample data of each sampling layer based on the total sampling number and the first sampling ratio, it can obtain a large number of first sample data. Then, the second sample data is extracted from the first sample data. This realizes the extraction of second sample data from a large number of sample data, thereby improving the selectivity and randomness of the extracted second sample data and reducing the error of extracting the second sample data.

[0110] 303. The computer equipment extracts the first sample data of each of the multiple sampling layers from the search business data included in the multiple sampling layers, based on the total number of samples in the multiple sampling layers and the first sampling ratio of the multiple sampling layers.

[0111] In some embodiments, if the total number of samples is 30,000 and the first sampling ratios of the multiple sampling layers are 80%, 19% and 1% respectively, then the computer device extracts 24,000, 5,700 and 300 first sample data from the search business data included in the multiple sampling layers respectively.

[0112] In this embodiment, after the total number of samples is determined, the first sample data is extracted using stratified sampling. Stratified sampling involves dividing the overall data into different sampling strata according to specific rules, then independently and randomly extracting sample data from each stratum, and finally combining the values ​​of the sample data from different strata to estimate the target value of the overall data. Compared to random sampling, the sample data extracted by stratified sampling has a more similar data structure distribution to the overall data, thus the results obtained by stratified sampling in this embodiment have higher accuracy.

[0113] 304. The computer equipment determines the second sampling ratio corresponding to each of the multiple sampling strata based on the minimum sampling number corresponding to each of the multiple sampling strata.

[0114] In some embodiments, the computer device determines a second sampling ratio corresponding to each of the multiple sampling strata based on the minimum sampling number corresponding to each of the multiple sampling strata, including the following steps: for each sampling stratum, the computer device determines a third sampling ratio between the minimum sampling number corresponding to the sampling stratum and the first sampling number, where the first sampling number is the number of first sample data in the sampling stratum; the computer device multiplies the third sampling ratio by a first multiple to obtain the second sampling ratio.

[0115] The third sampling ratio between the minimum sampling number and the first sampling number is the minimum sampling ratio. Optionally, the first multiple is greater than or equal to 1, so that the second sampling ratio is greater than the minimum sampling ratio. Then, the second sample data is drawn based on the second sampling ratio, which can ensure that the number of the second sample data meets the minimum sampling number. Thus, the evaluation parameters based on the second sample data can accurately represent the evaluation parameters of the overall sampling layer.

[0116] Optionally, if the first sample sizes of multiple sampling strata are 24000, 5700, and 300 respectively, and the minimum sample size corresponding to each sampling stratum is 300, then the computer device determines the ratios between 300 and 24000, 300 and 5700, and 300 and 300 respectively, obtaining the third sampling ratios of 1:80, 1:19, and 1:1 corresponding to the multiple sampling strata. Since both the first and minimum sample sizes of the third sampling stratum are 300, a first multiplier of 1 is set to retain 300 sample data points from the third stratum, meaning the third sampling stratum is sampled at 100%. The first multipliers corresponding to the first and second sampling strata can be set and changed as needed; optionally, if the first multipliers for both the first and second sampling strata are also 1, then the second sampling ratios for the first and second sampling strata are 1:80 and 1:19 respectively.

[0117] 305. The computer equipment extracts second sample data from the first sample data corresponding to the multiple sampling layers based on the second sampling ratio corresponding to the multiple sampling layers.

[0118] Optionally, if the first sample numbers of the multiple sampling layers are 24,000, 5,700 and 300 respectively, and the second sampling ratios of the multiple sampling layers are 1:80, 1:19 and 1:1 respectively, then the computer device will extract 300 second sample data from each sampling layer.

[0119] It should be noted that in related technologies, the small number of samples and the arbitrary nature of the sampling process, coupled with a lack of clear guidelines, lead to evaluation results based on these samples that fail to accurately reflect the true distribution of the overall data. Stratified sampling, which extracts data from each stratum based on its proportion of the total data, results in strata with smaller data percentages having very low sample sizes, making it difficult for evaluation parameters based on these samples to scientifically explain the true search quality of those strata. In this embodiment, however, by using stratified sampling and then further extracting samples from each stratum based on its minimum sample size, the distribution of sample data becomes more even. This ensures that subsequent evaluation parameters based on these samples accurately represent the evaluation parameters of the target search service, thus improving the accuracy of the evaluation.

[0120] It should be noted that traditional search evaluation methods primarily rely on subjective intuition and the number of evaluation objects when determining the sampling size, without considering the inaccuracy of evaluation results caused by an unreasonable sampling size. In this embodiment, however, a secondary sampling is performed based on stratified sampling and the minimum sampling size. This sampling method, while ensuring the accuracy of the overall sampling conclusion, reduces the sampling size of the sampling layer with a high data proportion and increases the sampling size of the sampling layer with a low data proportion. This ensures that the sampling size of different sampling layers meets the lower limit of the sampling size, thus clarifying the scientific nature of the sampling size selection in a statistical sense. Furthermore, the evaluation parameters based on the final sample data can scientifically explain the specific performance of different sampling layers and the overall performance of the search business, thereby improving the effectiveness of the search business evaluation.

[0121] 306. The computer equipment sorts multiple search keywords based on the first evaluation parameters corresponding to each of the multiple search keywords.

[0122] The ranking position of each search keyword is positively correlated with the first evaluation parameter of the search keyword. The determination process of the first evaluation parameter includes the following steps (1)-(3):

[0123] (1) For each search keyword, the computer device obtains the recall parameter of the search keyword. The recall parameter is used to indicate whether an item is found based on the search keyword.

[0124] In some embodiments, the process of determining the recall parameter includes the following steps: if an item is found based on the search keyword, the computer device determines the recall parameter as a first recall parameter, which indicates that an item was found based on the search keyword, i.e., there is a search result; if no item is found based on the search keyword, the computer device determines the recall parameter as a second recall parameter, which indicates that no item was found based on the search keyword, i.e., there is no search result.

[0125] Optionally, the recall parameter is a natural value, and the size of the natural value can be set and changed as needed; see Table 2, which shows one way to define the recall parameter, where the first recall parameter and the second recall parameter are 1 and 0 respectively, and 1 and 0 represent search results with and without search results respectively.

[0126] Table 2

[0127] Search results Recall parameters illustrate Search results are available. 1 Items found No search results found 0 No items found

[0128] (2) If the recall parameter is used to represent the items found based on the search keywords, the computer device obtains the item association parameter of the item, which is used to represent the degree of association between the item and the search keywords.

[0129] It should be noted that both items and search keywords have a subject. In some cases, items and search keywords may also include descriptive attributes, such as color, region, and size. Accordingly, to further improve the representation of the relevance between items and search keywords by item association parameters, this embodiment determines item association parameters from two dimensions: subject and descriptive attributes. The process of determining item association parameters includes the following steps: a computer device matches items with search keywords; if the subject of the item does not match the subject of the search keyword, the computer device determines the item association parameter as a first association parameter, which indicates that the item and search keyword are not related, and the subject includes at least one of the item's product term and brand term; if the subject of the item matches the subject of the search keyword, and the descriptive attributes of the item do not match the descriptive attributes of the search keyword, the computer device determines the item association parameter as a second association parameter, which indicates that the item and search keyword are not completely related; if the subject of the item matches the subject of the search keyword, and the descriptive attributes of the item match the descriptive attributes of the search keyword, the computer device determines the item association parameter as a third association parameter, which indicates that the item and search keyword are related.

[0130] The product terms for an item include search keywords indicating the type of product, the major service category, and the sub-category of the service industry. Major service categories include clothing, catering, and entertainment, while sub-categories include short-sleeved clothing and hot pot catering. The brand terms refer to the company brand name to which the item belongs, such as Li Ning or Yili. The descriptive attributes refer to the attributes used to describe the item, including descriptive words such as color, region, and size. These descriptive words can be phrases, such as "blue" for color and "southern" for region. See Table 3, which illustrates one way to define the subject and descriptive attributes.

[0131] Table 3

[0132]

[0133] Optionally, the item association parameter is a natural value, the size of which can be set and changed as needed; see Table 4, which shows one way to define the item association parameter, where the first association parameter, the second association parameter, and the third association parameter are 0, 1, and 2, respectively.

[0134] Table 4

[0135]

[0136] In this embodiment, by matching the main and modifier attributes of items with search keywords, and assigning corresponding item association parameters to items based on the matching results, the level of item association parameters is simplified, thereby shortening the process of determining item association parameters and improving the efficiency of determining item association parameters.

[0137] It should be noted that if the recall parameter is used to indicate that no item was found based on the search keyword, the computer device directly determines that the first evaluation parameter of the search keyword is 0.

[0138] (3) The computer equipment determines the first evaluation parameter of the search keyword based on the recall parameter and the item association parameter.

[0139] In some embodiments, the search keyword retrieves at least one item, and the at least one item is sorted in a first order. The recall parameter only indicates whether the search keyword retrieves an item, and the item association parameter only indicates the correlation between the retrieved item and the search keyword. Neither parameter indicates the rationality of the sorting of the at least one retrieved item. Therefore, determining the first evaluation parameter of the search keyword solely based on the recall parameter and the item association parameter does not reflect the impact of the rationality of the item sorting on the search business. To enhance the representativeness of the first evaluation parameter, the computer device determines the first evaluation parameter of the search keyword based on the recall parameter and the item association parameter, implemented as follows: The computer device determines the sorting parameter of the search keyword based on the first order of at least one item, the item association parameter, and the second order. The sorting position of each item in the second order is positively correlated with the item association parameter, and the sorting parameter is used to represent the rationality of the sorting of at least one item. The computer device performs a weighted sum of the recall parameter and the sorting parameter to obtain the first evaluation parameter of the search keyword.

[0140] In one implementation, the ranking parameter is the Normalized Discounted Cumulative Gain (NDCG). Accordingly, the computer device determines the ranking parameter of the search keyword based on the first order of at least one item, the item association parameter, and the second order, including the following steps: the computer device obtains the Discounted Cumulative Gain (DCG) based on the first order of at least one item and the item association parameter using the following formula (3); the computer device obtains the Ideal Discounted Cumulative Gain (IDCG) based on the second order of at least one item and the item association parameter using the following formula (4); and then the computer device uses the ratio between DCG and IDCG as the ranking parameter.

[0141]

[0142] Among them, DCG k rel1 represents the cumulative gain of the k-th search keyword; rel1 represents the item association parameter of the first item; i represents the i-th item in at least one item; p represents the number of at least one item, or p represents the first p items in at least one item.

[0143]

[0144] Among them, IDCG kRepresents the ideal cumulative gain of the k-th search keyword; i represents the i-th item among at least one item; rel i This represents the item association parameter for the i-th item; || p This represents p items arranged in the second order; p represents the number of at least one item, or p represents the first p items of at least one item.

[0145] The computer equipment determines the ratio between DCG and IDCG using the following formula (5).

[0146]

[0147] Among them, NDCG k DCG represents the normalized loss cumulative gain of the k-th search keyword, i.e., the ranking parameter. k IdCG represents the cumulative gain of the k-th search keyword; k This represents the ideal cumulative gain of the k-th search keyword. In one implementation, the computer device obtains the first evaluation parameter of the search keyword by weighted summing of the recall parameter and the ranking parameter using the following formula (6).

[0148] s k =w1*r k +w2*NDCG k (6);

[0149] Where w1 + w2 = 1, w1 represents the weight of the recall parameter, and w2 represents the weight of the ranking parameter; s k NDCG represents the first evaluation parameter for the k-th search keyword. k The sorting parameter for the k-th search keyword; r k This represents the recall parameter for the k-th search keyword.

[0150] It should be noted that if the recall parameter is used to indicate that no items were found based on the search keyword, then the first evaluation parameter of the search keyword is 0. If the recall parameter is used to indicate that items were found based on the search keyword, and the first recall parameter is represented by the natural value 1, then the computer device obtains the first evaluation parameter of the search keyword by weighted summing of the recall parameter and the ranking parameter using the following formula (7).

[0151] s k =w1+w2*NDCG k (7);

[0152] Where w1 + w2 = 1, w1 represents the weight of the recall parameter, and w2 represents the weight of the ranking parameter; s k NDCG represents the first evaluation parameter for the k-th search keyword. kThis represents the sorting parameter for the k-th search keyword.

[0153] It should be noted that the method provided in this application embodiment can offer a breakthrough for optimizing search services. Specifically, it provides optimization targets for further improving search services when current evaluation metrics are already satisfactory. For example, in supermarket apps, a large number of new products are added daily. After a period of time, the new product catalog may be vastly different from the original. While relevant performance metrics, such as Gross Merchandise Volume (GMV), may be relatively stable, it's unclear whether the existing search functionality can find products related to the search keywords, or whether the ranking of those products is reasonable. In this application embodiment, by determining evaluation parameters for search keywords based on their ranking parameters, it's determined that these evaluation parameters reflect the reasonableness of the item ranking. Based on these evaluation parameters, optimization space can be found for the search service to improve its revenue.

[0154] 307. A computer device determines at least one target search keyword that falls within the second confidence interval from a plurality of ranked search keywords.

[0155] The second confidence interval is used to represent the confidence level of the obtained target evaluation parameter. The size of the interval corresponding to the second confidence interval is taken as the confidence level of the target evaluation parameter. For example, if the second confidence interval is [2.5%, 97.5%], then the size of the interval corresponding to the second confidence interval, 95%, is the confidence level of the target evaluation parameter.

[0156] The second confidence interval corresponds to the maximum confidence interval [0%, 100%]. The second confidence interval is a sub-interval within the maximum confidence interval and is the middle interval of the maximum confidence interval. For example, if the interval size corresponding to the second confidence interval is 95%, then the second confidence interval is the interval between 2.5% and 97.5%; or if the interval size corresponding to the second confidence interval is 90%, then the second confidence interval is the interval between 5% and 95%. Since multiple search keywords are sorted according to evaluation parameters, this method of determining target keywords within the second confidence interval removes search keywords with excessively large or small evaluation parameters. In other words, it removes unstable search keywords with excessively large or small evaluation parameters obtained from the evaluation of the evaluation object. The remaining target keywords within the second confidence interval are search keywords with stable evaluation parameters, thus making the target evaluation parameters obtained from subsequent evaluation based on the target keywords more accurate.

[0157] In one implementation, the computer device determines the difference interval between the maximum confidence interval and the second confidence interval, and uses half of this difference interval as the discard interval. Then, it subtracts the discard interval from the maximum confidence interval before and after the sorted search keywords, and uses the search keywords within the remaining second confidence interval as the target search keywords. Optionally, the second confidence interval is represented as [lower percentile value after sorting, upper percentile value after sorting], where lower is the interval size corresponding to half of the difference interval, and upper is the interval size corresponding to the sum of half of the difference interval and the second confidence interval. For example, if the interval size of the second confidence interval is 95%, then lower = 2.5, upper = 97.5. In this embodiment, by determining the target keywords located within the second confidence interval, search keywords with excessively large or small evaluation parameters are removed from the multiple search keywords, i.e., unstable search keywords with excessively large or small evaluation parameters obtained from the evaluation of the evaluation object are removed, thus making the target evaluation parameters obtained from the subsequent evaluation based on the target keywords more accurate.

[0158] 308. The computer device determines the target evaluation parameters based on the first evaluation parameters corresponding to at least one target search keyword.

[0159] The target evaluation parameters are used to represent the quality of the target search service. Optionally, the target evaluation parameters are also used to determine the impact of changes on the search quality of the target search service based on the target evaluation parameters before and after the changes. These changes include updating a function within the target search service, adding new functions to the target search service, updating the search model corresponding to the target search service, etc., without specific limitations.

[0160] In related technologies, the target evaluation parameter is generally obtained by using the average value of multiple evaluation parameters as the point estimation method. This makes the evaluation results difficult to analyze and the evaluation conclusions difficult to quantify. The evaluation results obtained by point estimation alone are difficult to measure the difference between the evaluation parameters obtained by sample statistics and the overall evaluation parameters. In order to improve the interpretability of the target evaluation parameter, the computer equipment obtains the target evaluation parameter in the following way.

[0161] It should be noted that the first evaluation parameters corresponding to multiple search keywords can be obtained independently based on a single evaluation object, or they can be obtained independently based on multiple evaluation objects. In one implementation, if the first evaluation parameters corresponding to multiple search keywords are obtained based on a single evaluation object, the computer device determines the target evaluation parameter based on the first evaluation parameters corresponding to at least one target search keyword. This includes the following steps: the computer device uses the parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to at least one target search keyword as the target evaluation parameter. In this embodiment, by using the parameter range determined based on the evaluation parameters of the search keywords within the confidence interval as the target evaluation parameter, the adverse effects on the evaluation parameters caused by instability due to the evaluation parameters of different evaluation objects being too large or too small, which are avoided by the target evaluation parameter obtained by point estimation, are prevented. Furthermore, the evaluation parameter is quantified based on this confidence interval, and the difference between the evaluation parameters obtained from the sample statistics and the overall evaluation parameters can be measured through this confidence interval, thus achieving an effective interpretation of the obtained evaluation parameters.

[0162] It should be noted that with the development of e-commerce, search services are widely used in the e-commerce field. The more products a store has, the more important search becomes; in leading e-commerce platforms, search contributes more than half of the GMV, highlighting its undeniable importance. Traditionally, evaluations of search services rely solely on model evaluation metrics and A / B testing metrics to numerically assess performance, neglecting the actual buyer experience. Model evaluation metrics refer to the accuracy and recall of the search model, while A / B testing metrics determine the effectiveness of adding a new feature or updating the search model by comparing test parameters before and after the update. In this embodiment, however, evaluation parameters are obtained by evaluating sample data using the evaluation subjects. These target evaluation parameters are then used to derive the target evaluation parameters for the target search service, ensuring that they reflect the actual user experience and thus are more accurate.

[0163] In another implementation, if the first evaluation parameters corresponding to multiple search keywords are obtained based on multiple evaluation objects, and each evaluation object is used to evaluate multiple search keywords, then the computer device determines the target evaluation parameter based on the first evaluation parameter corresponding to at least one target search keyword, including the following steps: For each evaluation object, the computer device takes the parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to at least one target search keyword as the parameter range corresponding to the evaluation object; the computer device takes the intersection of the parameter ranges corresponding to multiple evaluation objects as the target evaluation parameter.

[0164] It should be noted that, according to the law of large numbers, when multiple evaluation objects are evaluated independently at the same time, the average value of the evaluations of multiple evaluation objects will be closer to the expected value when the number of multiple evaluation objects is large enough. In this embodiment, multiple evaluation objects are used to independently evaluate multiple search keywords, and the intersection of the parameter ranges of multiple evaluation objects is used as the target evaluation parameter. This avoids the situation where the target evaluation parameter obtained based on the evaluation parameter of only one evaluation object is inaccurate, and further improves the accuracy of the target evaluation parameter.

[0165] In the embodiments of this application, the computer device determines the target evaluation parameters of the target search service based on the first evaluation parameters of the multiple search keywords corresponding to the multiple sampling layers respectively. In some embodiments, the computer device also determines the evaluation parameters of each sampling layer based on the first evaluation parameters of the multiple search keywords corresponding to the multiple sampling layers respectively. The implementation method is the same as the implementation method of determining the target evaluation parameters of the target search service in steps 306-308, and will not be described again here.

[0166] It should be noted that in the daily application of search services, there may be situations where all relevant evaluation indicators are normal, but the search click-through rate drops. In this embodiment, small-scale evaluations are conducted on multiple sampling layers to assess whether there are fluctuations in different sampling layers based on the evaluation parameters of each sampling layer. This allows for the determination of whether bad cases discovered based on fluctuations are common problems. By investigating step by step, the root cause of the problem can be found, enabling the search service to be optimized based on the root cause.

[0167] In this embodiment, steps 306-308 represent one implementation of a computer device determining target evaluation parameters for a target search service based on the first evaluation parameters corresponding to the search keywords in the second sample data. In another implementation, the computer device determining target evaluation parameters for a target search service based on the first evaluation parameters corresponding to the search keywords in the second sample data includes the following steps: For each sampling layer, the computer device determines a third evaluation parameter corresponding to the sampling layer based on the first evaluation parameters corresponding to the search keywords in the second sample data of that sampling layer; the computer device weighted sums the third evaluation parameters corresponding to the multiple sampling layers based on the first sampling ratios corresponding to the multiple sampling layers to obtain the target evaluation parameters.

[0168] In this process, for each sampling layer, the computer device uses the average of the first evaluation parameters of the multiple search keywords corresponding to that sampling layer as the evaluation parameter of that sampling layer.

[0169] In this implementation, the computer device calculates the weighted sum of the evaluation parameters corresponding to the multiple sampling layers based on the first sampling ratio of each sampling layer to obtain the target evaluation parameters. This ensures that the data volume of different sampling layers has a certain impact on the target evaluation parameters, making the target evaluation parameters more compatible with the overall data structure distribution of the target search business, thereby further improving the accuracy of the target evaluation parameters.

[0170] It's important to note that sampling can reduce evaluation costs by assessing overall performance with a small amount of data; that is, the evaluation results of sampled data can represent the overall evaluation results. However, sampling introduces sampling error. To improve the accuracy of evaluation results, it's necessary to reduce sampling error. Sampling error includes systematic error and random error. Systematic error is caused by incorrect sampling methods, while random error is caused by drawing different samples. Random error is difficult to measure; therefore, reducing systematic error is necessary to reduce sampling error.

[0171] In some embodiments, the computer device reduces system errors by comprising the following steps: For each sampling layer, the computer device determines a first evaluation accuracy rate of the sampling layer based on the accuracy rates of first evaluation parameters of multiple search keywords corresponding to the sampling layer. The computer device then performs a weighted summation of the first evaluation accuracy rates corresponding to the multiple sampling layers based on the first sampling ratios corresponding to each of the multiple sampling layers to obtain a second evaluation accuracy rate for the target search service. If the second evaluation accuracy rate is lower than an accuracy threshold, the computer device updates the minimum sampling number corresponding to each of the multiple sampling layers. The computer device then re-evaluates the target search service based on the updated minimum sampling number.

[0172] The accuracy threshold can be set and changed as needed, and is not specifically limited here. In some embodiments, if the evaluation accuracy of multiple sampling layers is 90%, 97%, and 91%, respectively, and the first sampling proportions of multiple sampling layers are 80%, 19%, and 1%, respectively, then the evaluation accuracy of the target search service is: (90%*80+97%*19+91%*1) / (80+19+1)=91.34%.

[0173] In this embodiment, when the evaluation accuracy of the target search service is lower than the accuracy threshold, the minimum sampling number corresponding to each of the multiple sampling layers is updated, and the target search service is re-evaluated based on the updated minimum sampling number. This effectively reduces the system error caused by incorrect sampling number, thereby reducing the problem of inaccurate evaluation parameters caused by system error.

[0174] In some embodiments, the accuracy of the evaluation parameters used for evaluation through evaluation objects cannot be guaranteed, i.e., there are fluctuation errors in the evaluation parameters between different evaluation objects, indicating that the evaluation standards are unreasonable. In this embodiment, the computer device further improves the accuracy of the target evaluation parameters through the following steps: the computer device obtains the average value of the parameters corresponding to the evaluation objects, where the average value is the average value of the first evaluation parameters corresponding to multiple search keywords; the computer device determines the difference between the average values ​​of the parameters corresponding to each pair of evaluation objects, obtaining multiple differences; if the multiple differences include a target difference exceeding a difference threshold, the computer device obtains second evaluation parameters for re-evaluating the multiple search keywords using two target evaluation objects, where the two target evaluation objects are the evaluation objects corresponding to the target difference, and the second evaluation parameters are used to redetermine the target evaluation parameters for the target search business.

[0175] In this embodiment, by re-evaluating the evaluation parameters whose average value among the evaluation objects exceeds the difference threshold, and redetermining the target evaluation parameters of the target search business based on the re-evaluated evaluation parameters, the accuracy of the evaluation parameters obtained from the evaluation of the evaluation objects is ensured. Furthermore, the target evaluation parameters of the target search business are determined based on the accurate evaluation parameters, thereby improving the accuracy of the target evaluation parameters.

[0176] It should be noted that the processes and techniques for evaluating search services in related technologies lack scientific basis, resulting in evaluation results that can only achieve a rough assessment of the search service and cannot deeply and accurately measure the search performance of the target service. However, in this application embodiment, by optimizing the three stages of sampling, evaluation, and evaluation parameter output, the evaluation method provided by this application embodiment is more scientific and interpretable in a statistical sense, thereby improving the accuracy of the evaluation results obtained.

[0177] See Figure 4 , Figure 4 This is a flowchart illustrating a search evaluation method provided according to an embodiment of this application. The method optimizes the evaluation process in three stages: sampling, evaluation, and evaluation parameter output. In the sampling stage, the computer device samples head words, middle words, and tail words, achieving stratified sampling. In the evaluation stage, the computer device performs evaluations from both the item dimension and the search keyword dimension. The relevance between items and search keywords is evaluated at the item dimension, while the presence of search results for items is evaluated at the search keyword dimension. Finally, in the evaluation parameter output stage, evaluation parameters for the search service are output through parameter ranges.

[0178] It should be noted that the method provided in this application can determine the search quality of the search service, thus providing effective support for subsequent adjustments to the search service. For example, for a newly launched app version, the search model will be optimized on the algorithm side based on a series of questions raised about the product. Even if the North Star indicator (the only key indicator) performs well in A / B testing, some related indicators may still perform poorly. It is difficult to determine whether this will affect the user experience after the new search model is launched. However, in this application embodiment, the search keywords are evaluated using evaluation objects, and the evaluation parameters for the search service are subsequently determined. Based on these evaluation parameters, the search quality of the new search model in the search service can be determined, thus providing more reasonable guidance for adjustments to the new search model.

[0179] This application provides a search evaluation method. This method, based on stratified sampling, determines a secondary sampling ratio based on the minimum sample size corresponding to each of the multiple sampling strata. Then, it performs secondary sampling on the multiple sampling strata based on this ratio, ensuring that the sample data from each stratum meet the minimum sample size requirement, i.e., ensuring a uniform distribution of sample sizes across the multiple sampling strata. Because the sample sizes are uniformly distributed across the multiple strata, the evaluation parameters of the target search service are obtained based on the evaluation parameters of the uniformly distributed sample data, resulting in high accuracy of the obtained evaluation parameters for the target search service.

[0180] Figure 5 This is a block diagram of a search evaluation device according to an embodiment of this application. The device includes:

[0181] The first determining module 501 is used to determine the search business data included in multiple sampling layers based on the number of searches corresponding to search keywords in multiple sets of search business data of the target search business. Each set of search business data includes search keywords and search results for search keywords.

[0182] The second determining module 502 is used to determine the total number of samples for multiple sampling layers based on the minimum number of samples corresponding to each of the multiple sampling layers. The minimum number of samples is determined based on the accuracy parameters of the evaluation.

[0183] The first extraction module 503 is used to extract the first sample data of multiple sampling layers from the search business data included in multiple sampling layers based on the total number of samples in multiple sampling layers and the first sampling ratio of multiple sampling layers.

[0184] The third determining module 504 is used to determine the second sampling ratio corresponding to each of the multiple sampling layers based on the minimum number of samples corresponding to each of the multiple sampling layers.

[0185] The second extraction module 505 is used to extract second sample data of multiple sampling layers from the first sample data corresponding to multiple sampling layers based on the second sampling ratio corresponding to each of the multiple sampling layers.

[0186] The fourth determining module 506 is used to determine the target evaluation parameters of the target search business based on the first evaluation parameters corresponding to the search keywords in the second sample data. The first evaluation parameters are obtained by evaluating the search results in the second sample data.

[0187] In some embodiments, the accuracy parameters of the evaluation include the target evaluation accuracy, the target error, and the first confidence interval. The apparatus further includes a fifth determining module, used to determine the minimum sampling number based on the target evaluation accuracy, the target error, and the first confidence interval. The target evaluation accuracy is used to represent the expected accuracy of the first evaluation parameter obtained from evaluating the evaluation object. The target error is used to represent the error between the target evaluation parameter and the actual evaluation parameter of the target search service. The first confidence interval is used to represent the confidence level of the target error.

[0188] In some embodiments, the third determining module 504 is used to determine, for each sampling layer, a third sampling ratio between the minimum number of samples corresponding to the sampling layer and the first number of samples, wherein the first number of samples is the number of first sample data in the sampling layer; and multiply the third sampling ratio by the first multiple to obtain a second sampling ratio.

[0189] In some embodiments, the second determining module 502 is used to multiply the minimum number of samples corresponding to each sampling layer by a second multiple to obtain the second number of samples corresponding to the sampling layer, wherein the second multiple is positively correlated with the first sampling ratio; and to use the sum of the second number of samples corresponding to multiple sampling layers as the total number of samples.

[0190] In some embodiments, the fourth determining module 506 is used to sort multiple search keywords based on first evaluation parameters corresponding to the multiple search keywords respectively, wherein the sorting position of each search keyword is positively correlated with the first evaluation parameter of the search keyword; to determine at least one target search keyword that is located within the second confidence interval among the sorted multiple search keywords based on the second confidence interval; and to determine the target evaluation parameter based on the first evaluation parameter corresponding to the at least one target search keyword.

[0191] In some embodiments, if the first evaluation parameters corresponding to multiple search keywords are obtained based on an evaluation object, the fourth determining module 506 is further used to take the parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to at least one target search keyword as the target evaluation parameter.

[0192] In some embodiments, if the first evaluation parameters corresponding to multiple search keywords are obtained based on multiple evaluation objects, and each evaluation object is used to evaluate multiple search keywords, then the fourth determining module 506 is further configured to, for each evaluation object, take the parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to at least one target search keyword as the parameter range corresponding to the evaluation object; and take the intersection of the parameter ranges corresponding to multiple evaluation objects as the target evaluation parameter.

[0193] In some embodiments, the apparatus further includes:

[0194] The first acquisition module is used to acquire the average value of the parameters corresponding to the evaluation object. The average value of the parameters is the average value of the first evaluation parameters corresponding to multiple search keywords.

[0195] The sixth determination module is used to determine the difference between the average values ​​of the parameters corresponding to each pair of evaluation objects, and obtain multiple differences;

[0196] The second acquisition module is used to acquire the second evaluation parameters of two target evaluation objects for re-evaluating multiple search keywords if multiple differences include target differences that exceed the difference threshold. The two target evaluation objects are the evaluation objects corresponding to the target differences. The second evaluation parameters are used to redetermine the target evaluation parameters of the target search business.

[0197] In some embodiments, the fourth determining module 506 is used to determine, for each sampling layer, a third evaluation parameter corresponding to the sampling layer based on the first evaluation parameter corresponding to the search keywords in the second sample data of the sampling layer; and to obtain the target evaluation parameter by weighted summation of the third evaluation parameters corresponding to the multiple sampling layers based on the first sampling ratios corresponding to the multiple sampling layers.

[0198] In some embodiments, the apparatus further includes:

[0199] The third acquisition module is used to acquire the recall parameters for each search keyword. The recall parameters are used to indicate whether an item is found based on the search keyword.

[0200] The fourth acquisition module is used to acquire the item association parameters of the item if the recall parameter is used to indicate that the item was found based on the search keyword. The item association parameters are used to indicate the degree of association between the item and the search keyword.

[0201] The seventh determination module is used to determine the first evaluation parameters of the search keywords based on the recall parameters and item association parameters.

[0202] In some embodiments, the search keywords retrieve at least one item, and the at least one item is sorted in a first order. The seventh determining module is used to determine the sorting parameters of the search keywords based on the first order of the at least one item, the item association parameters, and the second order. The sorting position of each item in the second order is positively correlated with the item association parameters of the item. The sorting parameters are used to indicate the rationality of the sorting of the at least one item. The recall parameters and the sorting parameters are weighted and summed to obtain the first evaluation parameters of the search keywords.

[0203] In some embodiments, the apparatus further includes:

[0204] The matching module is used to match items with search keywords;

[0205] The eighth determination module is used to determine the item association parameter as the first association parameter if the subject of the item does not match the subject of the search keyword. The first association parameter is used to indicate that the item is not related to the search keyword, and the subject includes at least one of the product words and brand words of the item.

[0206] The ninth determination module is used to determine the item association parameter as the second association parameter if the main body of the item matches the main body of the search keyword, but the item's modifier attribute does not match the modifier attribute of the search keyword. The second association parameter is used to indicate that the item and the search keyword are not completely related.

[0207] The tenth determination module is used to determine the item association parameter as the third association parameter if the main body of the item matches the main body of the search keyword and the modifier attribute of the item matches the modifier attribute of the search keyword. The third association parameter is used to indicate that the item is related to the search keyword.

[0208] In some embodiments, the apparatus further includes:

[0209] The eleventh determination module is used to determine the first evaluation accuracy of each sampling layer based on the accuracy of the first evaluation parameters of multiple search keywords corresponding to the sampling layer.

[0210] The weighted summation module is used to sum the first evaluation accuracy rates corresponding to multiple sampling layers based on the first sampling ratios corresponding to multiple sampling layers, and obtain the second evaluation accuracy rate of the target search business.

[0211] The update module is used to update the minimum number of samples corresponding to each of the multiple sampling layers if the accuracy of the second evaluation is lower than the accuracy threshold.

[0212] The evaluation module is used to re-evaluate the target search business based on the updated minimum sample size.

[0213] This application provides a search evaluation device. Based on stratified sampling, a secondary sampling ratio is determined according to the minimum sample size corresponding to each of the multiple sampling strata. Then, secondary sampling is performed on the multiple sampling strata based on this ratio, ensuring that the sample data from each stratum meets the minimum sample size requirement, i.e., the sample size is evenly distributed across the multiple sampling strata. Because the sample size is evenly distributed across the multiple sampling strata, the evaluation parameters of the target search service are obtained based on the evaluation parameters of the evenly distributed sample data, resulting in high accuracy of the obtained evaluation parameters.

[0214] In the embodiments of this application, the computer device can be a terminal or a server. When the computer device is a terminal, the terminal acts as the execution subject to implement the technical solution provided in the embodiments of this application; when the computer device is a server, the server acts as the execution subject to implement the technical solution provided in the embodiments of this application; or, the technical solution provided in this application can be implemented through the interaction between the terminal and the server. The embodiments of this application do not limit this.

[0215] When the computer device is a terminal, Figure 6 This is a structural block diagram of a terminal 600 according to an embodiment of this application. The terminal 600 includes a processor 601 and a memory 602.

[0216] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0217] Memory 602 may include one or more computer-readable storage media, which may be non-transitory. Memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 602 is used to store at least one program code, which is executed by processor 601 to implement the search evaluation method provided in the method embodiments of this application.

[0218] In some embodiments, the terminal 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0219] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0220] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0221] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of terminal 600; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 600 or in a folded design; in other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0222] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0223] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0224] Power supply 608 is used to power the various components in terminal 600. Power supply 608 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 608 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0225] In some embodiments, the terminal 600 further includes one or more sensors 609. The one or more sensors 609 include, but are not limited to, an accelerometer 610, a gyroscope 611, a pressure sensor 612, an optical sensor 613, and a proximity sensor 614.

[0226] Accelerometer 610 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 600. For example, accelerometer 610 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 601 can control display screen 605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 610. Accelerometer 610 can also be used for games or for acquiring user motion data.

[0227] The gyroscope sensor 611 can detect the orientation and rotation angle of the terminal 600. The gyroscope sensor 611 can work in conjunction with the accelerometer sensor 610 to collect the user's 3D movements on the terminal 600. Based on the data collected by the gyroscope sensor 611, the processor 601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0228] The pressure sensor 612 can be disposed on the side bezel of the terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 612 is disposed on the side bezel of the terminal 600, it can detect the user's grip signal on the terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 612. When the pressure sensor 612 is disposed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0229] An optical sensor 613 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 613. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 613.

[0230] The proximity sensor 614, also known as a distance sensor, is typically mounted on the front panel of the terminal 600. The proximity sensor 614 is used to detect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 614 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 614 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0231] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0232] When the computer device is a server, Figure 7This is a schematic diagram of a server structure according to an embodiment of this application. The server 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The memories 702 are used to store executable program code, and the processors 701 are configured to execute the executable program code to implement the search evaluation methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0233] This application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the search evaluation method of any of the above implementation methods.

[0234] This application also provides a computer program product, which includes computer program code stored in a computer-readable storage medium. The processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the search evaluation method of any of the above implementations.

[0235] In some embodiments, the computer program product involved in the present application can be deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0236] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for evaluating search, characterized in that, The method includes: Based on the number of searches corresponding to search keywords in multiple sets of search business data of the target search business, the search business data included in multiple sampling layers are determined, and each set of search business data includes search keywords and search results for the search keywords; Based on the minimum sampling number corresponding to each of the multiple sampling layers, the total sampling number of the multiple sampling layers is determined, wherein the minimum sampling number is determined based on the accuracy parameter of the evaluation. Based on the total number of samples from the multiple sampling layers and the first sampling ratio of the multiple sampling layers, the first sample data of each of the multiple sampling layers is extracted from the search business data included in the multiple sampling layers. Based on the minimum number of samples corresponding to each of the plurality of sampling strata, the second sampling ratio corresponding to each of the plurality of sampling strata is determined; Based on the second sampling ratio corresponding to each of the plurality of sampling layers, the second sample data of each of the plurality of sampling layers are extracted from the first sample data corresponding to each of the plurality of sampling layers. Based on the first evaluation parameters corresponding to the search keywords in the second sample data, the target evaluation parameters of the target search service are determined. The first evaluation parameters are obtained by evaluating the search results in the second sample data.

2. The method according to claim 1, characterized in that, The accuracy parameters of the evaluation include the target evaluation accuracy, target error, and first confidence interval. The process of determining the minimum sample size includes: Based on the target evaluation accuracy, the target error, and the first confidence interval, the minimum sampling number is determined. The target evaluation accuracy is used to represent the expected accuracy of the first evaluation parameter obtained from evaluating the evaluation object. The target error is used to represent the error between the target evaluation parameter and the actual evaluation parameter of the target search service. The first confidence interval is used to represent the confidence level of the target error.

3. The method according to claim 1, characterized in that, The step of determining the second sampling ratio corresponding to each of the plurality of sampling strata based on the minimum sampling number corresponding to each of the plurality of sampling strata includes: For each sampling stratum, a third sampling ratio is determined between the minimum sampling number corresponding to the sampling stratum and the first sampling number, where the first sampling number is the number of first sample data in the sampling stratum. The second sampling ratio is obtained by multiplying the third sampling ratio by the first multiple.

4. The method according to claim 1, characterized in that, The step of determining the total number of samples for each of the multiple sampling strata based on the minimum sampling number corresponding to each of the multiple sampling strata includes: For each sampling stratum, the minimum sampling number corresponding to the sampling stratum is multiplied by a second multiple to obtain the second sampling number corresponding to the sampling stratum. The second multiple is positively correlated with the first sampling ratio. The sum of the second sampling numbers corresponding to the multiple sampling layers is taken as the total sampling number.

5. The method according to claim 1, characterized in that, The determination of target evaluation parameters for the target search service based on the first evaluation parameters corresponding to the search keywords in the second sample data includes: Based on the first evaluation parameters corresponding to multiple search keywords, the multiple search keywords are sorted, and the sorting position of each search keyword is positively correlated with the first evaluation parameter of the search keyword; Based on the second confidence interval, at least one target search keyword that falls within the second confidence interval is identified from among the sorted search keywords; The target evaluation parameters are determined based on the first evaluation parameters corresponding to the at least one target search keyword.

6. The method according to claim 5, characterized in that, If the first evaluation parameters corresponding to the plurality of search keywords are obtained based on an evaluation object, then determining the target evaluation parameters based on the first evaluation parameters corresponding to the at least one target search keyword includes: The parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to the at least one target search keyword shall be used as the target evaluation parameters.

7. The method according to claim 5, characterized in that, If the first evaluation parameters corresponding to the plurality of search keywords are obtained based on multiple evaluation objects, and each evaluation object is used to evaluate the plurality of search keywords, then determining the target evaluation parameters based on the first evaluation parameters corresponding to the at least one target search keyword includes: For each evaluation object, the parameter range determined by the maximum and minimum values ​​of the first evaluation parameters corresponding to the at least one target search keyword is taken as the parameter range corresponding to the evaluation object. The intersection of the parameter intervals corresponding to the multiple evaluation objects is taken as the target evaluation parameter.

8. The method according to claim 7, characterized in that, The method further includes: Obtain the average value of the parameters corresponding to the evaluation object, wherein the average value of the parameters is the average value of the first evaluation parameters corresponding to the plurality of search keywords respectively; Determine the difference between the average parameters of each pair of evaluation objects to obtain multiple differences; If the plurality of differences include a target difference that exceeds the difference threshold, then the second evaluation parameters for re-evaluating the plurality of search keywords by two target evaluation objects are obtained respectively. The two target evaluation objects are the evaluation objects corresponding to the target difference. The second evaluation parameters are used to redetermine the target evaluation parameters of the target search business.

9. The method according to claim 1, characterized in that, The determination of target evaluation parameters for the target search service based on the first evaluation parameters corresponding to the search keywords in the second sample data includes: For each sampling layer, based on the first evaluation parameters corresponding to the search keywords in the second sample data of the sampling layer, the third evaluation parameter corresponding to the sampling layer is determined. Based on the first sampling ratio corresponding to each of the multiple sampling layers, the third evaluation parameters corresponding to each of the multiple sampling layers are weighted and summed to obtain the target evaluation parameters.

10. The method according to claim 1, characterized in that, The method further includes: For each search keyword, a recall parameter is obtained for the search keyword, which indicates whether an item is found based on the search keyword; If the recall parameter is used to indicate that an item was found based on the search keyword, then the item association parameter of the item is obtained, and the item association parameter is used to indicate the degree of association between the item and the search keyword; Based on the recall parameters and the item association parameters, the first evaluation parameters of the search keywords are determined.

11. The method according to claim 10, characterized in that, The search keyword retrieves at least one item, and the at least one item is sorted in a first order. The determination of the first evaluation parameter for the search keyword based on the recall parameter and the item association parameter includes: Based on the first order, item association parameters, and second order of the at least one item, the sorting parameters of the search keywords are determined. The sorting position of each item in the second order is positively correlated with the item association parameters of the item. The sorting parameters are used to indicate the rationality of the sorting of the at least one item. The first evaluation parameter of the search keyword is obtained by weighted summation of the recall parameter and the ranking parameter.

12. The method according to claim 10, characterized in that, The process of determining the item association parameters includes: Match the item with the search keywords; If the subject of the item does not match the subject of the search keyword, then the item association parameter is determined to be the first association parameter. The first association parameter is used to indicate that the item is not related to the search keyword. The subject includes at least one of the product term and brand term of the item. If the main body of the item matches the main body of the search keyword, but the modifying attributes of the item do not match the modifying attributes of the search keyword, then the item association parameter is determined to be the second association parameter, which indicates that the item and the search keyword are not completely related. If the main body of the item matches the main body of the search keyword, and the modifying attributes of the item match the modifying attributes of the search keyword, then the item association parameter is determined to be a third association parameter, which indicates that the item is related to the search keyword.

13. The method according to claim 1, characterized in that, The method further includes: For each sampling layer, the first evaluation accuracy of the sampling layer is determined based on the accuracy of the first evaluation parameters of the multiple search keywords corresponding to the sampling layer. Based on the first sampling ratio corresponding to each of the multiple sampling layers, the first evaluation accuracy corresponding to each of the multiple sampling layers is weighted and summed to obtain the second evaluation accuracy of the target search service; If the accuracy of the second evaluation is lower than the accuracy threshold, then update the minimum number of samples corresponding to the multiple sampling layers respectively. The target search service was re-evaluated based on the updated minimum sample size.

14. A search evaluation device, characterized in that, The device includes: The first determining module is used to determine the search business data included in multiple sampling layers based on the number of searches corresponding to search keywords in multiple sets of search business data of the target search business. Each set of search business data includes search keywords and search results for the search keywords. The second determining module is used to determine the total number of samples for the plurality of sampling layers based on the minimum number of samples corresponding to each of the plurality of sampling layers, wherein the minimum number of samples is determined based on the accuracy parameters of the evaluation. The first extraction module is used to extract the first sample data of the multiple sampling layers from the search business data included in the multiple sampling layers, based on the total number of samples in the multiple sampling layers and the first sampling ratio of the multiple sampling layers. The third determining module is used to determine the second sampling ratio corresponding to each of the multiple sampling layers based on the minimum sampling number corresponding to each of the multiple sampling layers. The second extraction module is used to extract the second sample data of the multiple sampling layers from the first sample data corresponding to the multiple sampling layers, based on the second sampling ratio corresponding to the multiple sampling layers respectively. The fourth determining module is used to determine the target evaluation parameters of the target search service based on the first evaluation parameters corresponding to the search keywords in the second sample data. The first evaluation parameters are obtained by evaluating the search results in the second sample data.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executed as the search evaluation method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one computer program for performing the search evaluation method according to any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes computer program code stored in a computer-readable storage medium, a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code to cause the computer device to perform the search evaluation method as described in any one of claims 1 to 13.

Images