A search quality real-time evaluation method, device, equipment and storage medium

Abstract

The application discloses a search quality real-time evaluation method and device, equipment and a storage medium, and relates to the technical field of Internet, and comprises the following steps: a sample pool construction module is used for online search based on each preset search vocabulary and corresponding prediction category and entity recognition result to determine a corresponding number of positive and negative samples to obtain a target commodity sample pool; a vocabulary routing module is used for judging whether a target search vocabulary currently input by a user belongs to a preset search vocabulary, and if yes, the target search vocabulary is routed to the target commodity sample pool to perform a search operation; a data uploading module is used for storing detailed data of each search stage into a preset persistent database based on a preset burying point when the search operation is performed; and a quality evaluation module is used for determining a quality evaluation result based on the detailed data and each preset quality index formula. The application determines the search quality by evaluating the detailed data of each search stage when the preset search vocabulary is input, and effectively guarantees the real-time performance of the evaluation.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a method, apparatus, device, and storage medium for real-time evaluation of search quality. Background Technology

[0002] Currently, existing technical solutions primarily rely on indirect feedback from online users to illustrate metrics such as search result page click-through rate, page turn rate, referral rate, conversion rate, dwell time, page views (PV), and unique visitors (UV). In this approach, evaluation results can only be provided retrospectively; that is, an assessment can only be conducted after a user has completed their search. This not only fails to provide a detailed analysis of each stage of the search system (multi-path recall, ranking, etc.) but also makes it susceptible to deviations in search quality due to external factors. The same search system may yield different search quality results; for example, referral and conversion rates are much higher during peak shopping periods than during off-peak periods; and click-through rates are much higher when users urgently want a particular product than when they are just browsing. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for real-time search quality assessment, capable of assessing the quality of each search stage and effectively ensuring the real-time nature and accuracy of the assessment. The specific solution is as follows:

[0004] In a first aspect, this application provides a real-time search quality evaluation device, comprising:

[0005] The product sample pool construction module is used to perform online searches based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term, and determine a first preset number of positive samples and a second preset number of negative samples to obtain the corresponding target product sample pool.

[0006] The search term routing module is used to obtain the target search term currently input by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool.

[0007] The search data upload module is used to asynchronously store detailed data of each search stage into a preset persistent database through a message queue when the search operation is performed, based on preset tracking points.

[0008] The search quality assessment module is used to determine the corresponding quality assessment results based on the detailed data stored in the preset persistent database and the calculation formulas of each preset quality indicator.

[0009] Optionally, the real-time search quality evaluation device further includes:

[0010] The evaluation result display module is used to generate corresponding search quality evaluation reports based on the detailed data stored in the preset persistent database; wherein, the search quality evaluation reports include search accuracy trend reports, search performance trend reports, and search diversity trend reports.

[0011] Optionally, the product sample pool construction module further includes:

[0012] The preset search term determination unit is used to determine a number of terms on the current platform that meet the preset search frequency as preset search terms;

[0013] The vocabulary category prediction unit is used to predict the category of each preset search term and obtain the predicted category corresponding to each preset search term.

[0014] The first entity recognition unit is used to perform entity recognition on each of the preset search terms to obtain entity recognition results corresponding to each of the preset search terms.

[0015] Optionally, the product sample pool construction module includes:

[0016] The first online search unit is used to perform online search by using each preset search term and the predicted category and entity recognition result corresponding to each preset search term as search conditions to obtain a first preset number of positive samples.

[0017] The second online search unit is used to perform an online search by using each preset search term and the entity recognition result corresponding to each preset search term as search conditions, and using the corresponding prediction category as an exclusion condition, to obtain a second preset number of negative samples.

[0018] The sample annotation submodule is used to perform corresponding annotation operations on each of the positive samples and each of the negative samples based on preset annotation rules.

[0019] Optionally, the sample annotation submodule includes:

[0020] The first annotation unit is used to annotate the corresponding positive samples based on the entity recognition results, and to determine the annotation status of the corresponding samples as annotated.

[0021] The second annotation unit is used to annotate the corresponding negative samples based on the entity recognition results, and to determine the annotation status of the corresponding samples as needing to be manually annotated again, so that subsequent manual review can determine whether to correct the annotation.

[0022] Optionally, the search term routing module further includes:

[0023] The normal product pool routing module is used to route the target search term to the normal online product pool to perform the corresponding search operation when the target search term does not belong to the preset search term.

[0024] Optionally, the search data upload module includes:

[0025] The first data uploading unit is used to asynchronously store the first search detail data of the current stage into a preset persistent database through a message queue based on the first preset tracking point during the multi-channel recall stage; wherein, the first search detail data includes the first true positive data, the first false negative data, the first false positive data, and the first true negative data corresponding to the multi-channel recall stage;

[0026] The second data upload unit is used to asynchronously store the second search detail data of the current stage into the preset persistent database through a message queue based on the second preset tracking point during the sorting stage.

[0027] Secondly, this application provides a method for real-time evaluation of search quality, including:

[0028] Online search is performed based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term to determine a first preset number of positive samples and a second preset number of negative samples, so as to obtain the corresponding target product sample pool.

[0029] Obtain the target search term currently entered by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool;

[0030] When performing the search operation, the step of asynchronously storing the detailed data of each search stage into a preset persistent database through a message queue is triggered based on preset tracking points.

[0031] The corresponding quality assessment results are determined based on the detailed data already stored in the preset persistent database and the calculation formulas for each preset quality indicator.

[0032] Thirdly, this application provides an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the steps of the aforementioned real-time search quality evaluation method.

[0035] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps of the aforementioned real-time search quality evaluation method.

[0036] As can be seen, in this application, the product sample pool construction module is used to perform online searches based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term, to determine a first preset number of positive samples and a second preset number of negative samples, so as to obtain a corresponding target product sample pool; the search term routing module is used to obtain the target search term currently input by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool; the search data uploading module is used to asynchronously store the detailed data of each search stage into a preset persistent database through a message queue based on preset tracking points when performing the search operation; the search quality evaluation module is used to determine the corresponding quality evaluation result based on the detailed data already stored in the preset persistent database and the calculation formula of each preset quality indicator. This application, by evaluating the detailed data of each search stage based on the target product sample pool when inputting preset search terms to determine the search quality, can realize the quality evaluation of each search stage and effectively ensure the real-time and accuracy of the evaluation, thereby improving search quality, user experience, and the probability of transaction completion. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 A schematic diagram of a real-time search quality assessment device provided in this application;

[0039] Figure 2 This application provides a schematic diagram of a commodity sample pool construction process;

[0040] Figure 3 A flowchart illustrating a specific real-time search quality assessment method provided in this application;

[0041] Figure 4 A schematic diagram of a search accuracy trend report provided in this application;

[0042] Figure 5 A schematic diagram of a search accuracy trend report provided in this application;

[0043] Figure 6 A schematic diagram of a search diversity trend report provided in this application;

[0044] Figure 7 A schematic diagram of a search diversity trend report provided in this application;

[0045] Figure 8 A flowchart of a real-time search quality assessment method provided in this application;

[0046] Figure 9 This application provides a structural diagram of an electronic device. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Existing technical solutions only provide evaluation results after the fact, meaning that the evaluation can only be performed on the user's completed search operation. This not only fails to provide a detailed interpretation of each stage of the search system (recall, ranking, etc.), but also makes the search quality susceptible to deviations due to external factors, potentially resulting in different search quality results from the same search system. Therefore, this application provides a real-time search quality evaluation scheme that enables quality evaluation at each stage of the search process, effectively ensuring the real-time nature and accuracy of the evaluation.

[0049] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a real-time search quality evaluation device, which includes: a product sample pool construction module 11, a search term routing module 12, a search data upload module 13, and a search quality evaluation module 14.

[0050] The product sample pool construction module 11 is used to perform online searches based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term, to determine a first preset number of positive samples and a second preset number of negative samples, so as to obtain the corresponding target product sample pool.

[0051] In this embodiment, combined with Figure 2 As shown, before constructing the target product sample pool, it is necessary to determine each preset search term and the corresponding predicted category and entity recognition result. Therefore, the product sample pool construction module may further include: a preset search term determination unit, used to determine several terms that meet preset search frequencies on the current platform as preset search terms; a term category prediction unit, used to obtain the predicted category corresponding to each preset search term by performing category prediction on each preset search term; and a first entity recognition unit, used to obtain the entity recognition result corresponding to each preset search term by performing entity recognition on each preset search term. The number of preset search terms can be pre-configured by relevant personnel based on actual needs. It should be understood that entity recognition is performed on all products in the online database, i.e., the normal product pool, to obtain the corresponding entity recognition results and persist them as database fields, so that corresponding positive and negative samples can be obtained through subsequent filtering. For example, when the preset search term is "Apple computer with dedicated graphics card 16G", after performing Name Entity Recognition (NER), the corresponding entity recognition result is "computer", and after performing category prediction, the predicted category is "3C digital".

[0052] Specifically, in combination Figure 2 As shown in this embodiment, in the process of constructing the target product sample pool, two different online queries are required. Based on the different search conditions corresponding to the two queries, a corresponding number of products are obtained as positive and negative samples. Then, the target product sample pool is obtained through the products used as positive and negative samples. Therefore, the product sample pool construction module may specifically include: a first online search unit, used to perform online searches by using each preset search term and the predicted category and entity recognition result corresponding to each preset search term as search conditions to obtain a first preset number of positive samples; a second online search unit, used to perform online searches by using each preset search term and the entity recognition result corresponding to each preset search term as search conditions, and using the corresponding predicted category as an exclusion condition to obtain a second preset number of negative samples; and a sample labeling submodule, used to perform corresponding labeling operations on each positive sample and each negative sample based on preset labeling rules. The first preset number and the second preset number can be configured in advance by relevant personnel based on actual needs.

[0053] It's important to understand that NER technology has an error rate, so manual secondary labeling is generally required before the corresponding samples are marked as labeled. Since the first search for each preset search term included a corresponding preset category as a limiting search condition, the NER results for the searched products do not contradict the predicted category. This indicates that the NER results for these products are relatively reliable, and manual labeling can be skipped to directly mark them as labeled, thus identifying them as positive samples. However, since the second search for each preset search term used the corresponding preset category as an exclusion condition, the NER results for these products contradict the predicted category. This indicates that the NER results for these products are relatively unreliable, requiring manual secondary labeling before marking them as labeled, thus identifying them as negative samples.

[0054] Furthermore, in this embodiment, the sample annotation submodule may specifically include: a first annotation unit, used to annotate the corresponding positive samples based on the entity recognition result, and determine the corresponding sample annotation status as annotated; and a second annotation unit, used to annotate the corresponding negative samples based on the entity recognition result, and determine the corresponding sample annotation status as pending manual secondary annotation, so that subsequent manual review can determine whether annotation correction is required.

[0055] The search term routing module 12 is used to obtain the target search term currently input by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool.

[0056] In this embodiment, combined with Figure 3As shown, after obtaining the target product sample pool, when acquiring the target search term input by the user, it is necessary to first determine whether the target search term belongs to the preset search term, and determine whether to trigger the search quality assessment operation based on the determination result. Therefore, the search term routing module may further include: a normal product pool routing module, used to route the target search term to the normal online product pool to perform the corresponding search operation when the target search term does not belong to the preset search term. It can be understood that when the target search term does not belong to the preset search term, normal online search business is performed based on the normal online product pool. When performing the online search business, product recall is performed according to the corresponding search rules, and after sorting the products recalled through multiple channels, the corresponding product list is returned, thereby ending the process. It should be understood that when it is determined that the target search term belongs to the preset search term and the search quality assessment operation is triggered, the corresponding search operation is performed based on the target product sample pool. During this operation, all related caches are closed to ensure the effectiveness of the search process. Specifically, when it is determined that the target search term belongs to the preset search term, a preset online search interface is called based on the target search term, with the parameter qualityMode=true indicating that the search triggers a search quality assessment operation.

[0057] See Figure 3 As shown in the figure, it is important to understand that reindex is an operation API (Application Programming Interface) of the storage middleware Elasticsearch (search server). Reindex with query means reindexing using a query, which refers to filtering data from one data source and writing it to another data source through a query. In this embodiment, it is specifically used to filter products from the normal online product pool and write them to the target product sample pool through a query.

[0058] The search data upload module 13 is used to asynchronously store detailed data of each search stage into a preset persistent database through a message queue when performing the search operation, based on preset tracking points.

[0059] In this embodiment, during the search operation, the step of asynchronously storing the detailed data of the multi-path recall stage and the sorting stage into a preset persistent database via a message queue based on preset tracking points is understood to be normal and does not affect the normal operation of the search and will not adversely affect the search results. Therefore, the search data upload module may specifically include: a first data upload unit, used to asynchronously store the first search detail data of the current stage into the preset persistent database via a message queue during the multi-path recall stage based on a first preset tracking point; wherein the first search detail data includes the first true positive data, the first false negative data, the first false positive data, and the first true negative data corresponding to the multi-path recall stage; and a second data upload unit, used to asynchronously store the second search detail data of the current stage into the preset persistent database via a message queue during the sorting stage based on a second preset tracking point. It is understood that the data types included in the first search detail data are consistent with those included in the second search detail data. These include not only the corresponding True Positive (TP), False Negative (FN), False Positive (FP), and True Negative (TN) data, but also product recall path information, product store information, product supplier information, product health information, product popularity information, product sales information, and product quality information. This data is unaffected by external factors. When operations adjust search parameters or R&D releases new versions, this embodiment can also evaluate the effects of the current changes in real time.

[0060] Search quality assessment module 14 is used to determine the corresponding quality assessment results based on the detailed data stored in the preset persistent database and the calculation formulas of each preset quality indicator.

[0061] In this embodiment, search quality indicators corresponding to each search stage are calculated based on the calculation formulas of each preset quality indicator and the detailed data, thereby determining the corresponding quality assessment results. In this way, by analyzing the quality of each search stage, it is possible to identify which stage is the weak link. The preset search indicators corresponding to the calculation formulas of each preset quality indicator include, but are not limited to, the proportion of recalled products from various sources, the average number of stores recalling products, the average number of suppliers recalling products, the average product health score of recalled products, search accuracy P1, search precision P2, search recall R, and search F1 (considering both precision and recall), to determine whether the search is accurate, comprehensive, and whether the search results are rich and diverse. The product health score is existing data in the product database, and is a type of data that mainly measures product quality based on dimensions such as product details page, main image, product name, and product clicks, views, and add-to-cart.

[0062] Furthermore, in this embodiment, the preset search metrics need to be explained as follows. It should be understood that during the multi-path recall stage, a product set b is obtained that is far greater than the number of requested products *a*. After the sorting stage, the top *a* products of product set b are taken as the final output, resulting in dataset c. At this point, the data in dataset c comes from each path in the multi-path recall stage. Therefore, the product count of each path in dataset c divided by the total number of products in dataset c is the preset quality metric calculation formula corresponding to the proportion of each source of the recalled products. Simultaneously, since dataset c includes all search results corresponding to the preset search terms, the number of stores from which dataset c originates, divided by the number of preset search terms, is the preset quality metric calculation formula corresponding to the average number of stores for the recalled products. Similarly, the number of suppliers from which dataset c originates, divided by the number of preset search terms, is the preset quality metric calculation formula corresponding to the average number of suppliers for the recalled products. Finally, the total health score of products in dataset c divided by the number of products in dataset c is the preset quality metric calculation formula corresponding to the average health score of the recalled products. Furthermore, the preset quality index calculation formulas corresponding to the search accuracy P1, the search precision P2, the search recall R, and the search F1 are shown below.

[0063]

[0064]

[0065]

[0066]

[0067] In this embodiment, to more clearly and specifically display the quality assessment results, it may further include: an assessment result display module, used to generate corresponding search quality assessment reports based on the detailed data stored in the preset persistent database; wherein, the search quality assessment reports include a search accuracy trend report, a search performance trend report, and a search diversity trend report. See specific charts. Figure 4 , Figure 5 , Figure 6 as well as Figure 7 As shown, wherein, Figure 4 The search accuracy trend report is used to reflect the trends of P1, P2, R, and F1 at a certain stage for each of the preset search terms. Figure 5 The search accuracy trend report is used to reflect the trends of P1, P2, R, and F1 at each stage for a preset search term. Figure 6 For the search diversity trend report used to reflect the distribution trend of the number of suppliers in the final search results, the Figure 7 This is a search diversity trend report used to reflect the trend of the proportion of the final search results originating from each recall loop. It should be understood that... Figure 4 and the Figure 6 The terms "printer," "computer," "computer repair," and "mask" in this context can be understood as preset search terms in a specific implementation method.

[0068] Therefore, in this application, the product sample pool construction module is used to perform online searches based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term, to determine a first preset number of positive samples and a second preset number of negative samples, so as to obtain the corresponding target product sample pool; the search term routing module is used to obtain the target search term currently input by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool; the search data uploading module is used to asynchronously store the detailed data of each search stage into a preset persistent database through a message queue based on preset tracking points when performing the search operation; the search quality evaluation module is used to determine the corresponding quality evaluation result based on the detailed data already stored in the preset persistent database and the calculation formula of each preset quality indicator. This application, by evaluating the detailed data of each search stage based on the target product sample pool when inputting preset search terms to determine the search quality, can realize the quality evaluation of each search stage and effectively ensure the real-time and accuracy of the evaluation, thereby improving search quality, user experience, and the probability of transaction completion.

[0069] See Figure 8As shown in the embodiments of this application, a real-time search quality evaluation method is also disclosed, including:

[0070] Step S11: Based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term, perform online search to determine a first preset number of positive samples and a second preset number of negative samples to obtain the corresponding target product sample pool.

[0071] Step S12: Obtain the target search term currently input by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool.

[0072] Step S13: When performing the search operation, the detailed data of each search stage is asynchronously stored into the preset persistent database through a message queue based on the preset tracking points.

[0073] Step S14: Determine the corresponding quality assessment result based on the detailed data already stored in the preset persistent database and the calculation formulas for each preset quality indicator.

[0074] For more detailed information on the working process of each of the above steps, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0075] Therefore, in this application, online searches are performed based on preset search terms and the predicted categories and entity recognition results corresponding to each preset search term to determine a first preset number of positive samples and a second preset number of negative samples, thereby obtaining a corresponding target product sample pool. The user's currently input target search term is obtained, and it is determined whether the target search term belongs to the preset search term. If so, the target search term is routed to the target product sample pool to perform a corresponding search operation within the target product sample pool. During the search operation, a step is taken to asynchronously store the detailed data of each search stage into a preset persistent database via a message queue based on preset tracking points. The corresponding quality assessment result is determined based on the detailed data already stored in the preset persistent database and the calculation formulas for each preset quality indicator. This application, by evaluating the detailed data of each search stage based on the target product sample pool when preset search terms are input to determine search quality, can achieve quality assessment of each search stage and effectively ensure the real-time and accuracy of the assessment, thereby improving search quality, user experience, and the probability of transaction completion.

[0076] Furthermore, embodiments of this application also disclose an electronic device, Figure 9This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0077] Figure 9 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the real-time search quality evaluation method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0078] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0079] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0080] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the real-time search quality evaluation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0081] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned real-time search quality evaluation method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0082] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0083] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0085] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A real-time search quality assessment device, characterized in that, include: The product sample pool construction module is used to perform online searches based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term, and determine a first preset number of positive samples and a second preset number of negative samples to obtain the corresponding target product sample pool. The search term routing module is used to obtain the target search term currently input by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool. The search data upload module is used to asynchronously store detailed data of each search stage into a preset persistent database through a message queue based on preset tracking points when performing the search operation; wherein, during the execution of the search operation, all related caches are closed; The search quality assessment module is used to determine the corresponding quality assessment results based on the detailed data stored in the preset persistent database and the calculation formulas of each preset quality indicator. The commodity sample pool construction module includes: The first online search unit is used to perform online search by using each preset search term and the predicted category and entity recognition result corresponding to each preset search term as search conditions to obtain a first preset number of positive samples. The second online search unit is used to perform an online search by using each preset search term and the entity recognition result corresponding to each preset search term as search conditions, and using the corresponding prediction category as an exclusion condition, to obtain a second preset number of negative samples. The sample annotation submodule is used to perform corresponding annotation operations on each of the positive samples and each of the negative samples based on preset annotation rules; The search data upload module includes: The first data uploading unit is used to asynchronously store the first search detail data of the current stage into a preset persistent database through a message queue based on the first preset tracking point during the multi-channel recall stage; wherein, the first search detail data includes the first true positive data, the first false negative data, the first false positive data, and the first true negative data corresponding to the multi-channel recall stage; The second data upload unit is used to asynchronously store the second search detail data of the current stage into the preset persistent database through a message queue based on the second preset tracking point during the sorting stage. The sample annotation submodule includes: The first annotation unit is used to annotate the corresponding positive samples based on the entity recognition results, and to determine the annotation status of the corresponding samples as annotated. The second annotation unit is used to annotate the corresponding negative samples based on the entity recognition results, and to determine the annotation status of the corresponding samples as needing to be manually annotated again, so that subsequent manual review can determine whether to correct the annotation.

2. The real-time search quality evaluation device according to claim 1, characterized in that, Also includes: The evaluation result display module is used to generate corresponding search quality evaluation reports based on the detailed data stored in the preset persistent database; wherein, the search quality evaluation reports include search accuracy trend reports, search performance trend reports, and search diversity trend reports.

3. The real-time search quality evaluation device according to claim 1, characterized in that, The commodity sample pool construction module also includes: The preset search term determination unit is used to determine a number of terms on the current platform that meet the preset search frequency as preset search terms; The vocabulary category prediction unit is used to predict the category of each preset search term and obtain the predicted category corresponding to each preset search term. The first entity recognition unit is used to perform entity recognition on each of the preset search terms to obtain entity recognition results corresponding to each of the preset search terms.

4. The real-time search quality evaluation device according to claim 1, characterized in that, The search term routing module also includes: The normal product pool routing module is used to route the target search term to the normal online product pool to perform the corresponding search operation when the target search term does not belong to the preset search term.

5. A method for real-time evaluation of search quality, characterized in that, include: Online search is performed based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term to determine a first preset number of positive samples and a second preset number of negative samples, so as to obtain the corresponding target product sample pool. Obtain the target search term currently entered by the user, determine whether the target search term belongs to the preset search term, and if so, route the target search term to the target product sample pool to perform the corresponding search operation in the target product sample pool; When performing the search operation, the detailed data of each search stage is asynchronously stored into a preset persistent database through a message queue based on preset tracking points; wherein, during the execution of the search operation, all related caches are closed; The corresponding quality assessment results are determined based on the detailed data already stored in the preset persistent database and the calculation formulas for each preset quality indicator. The step of performing online searches based on each preset search term and the predicted category and entity recognition results corresponding to each preset search term to determine a first preset number of positive samples and a second preset number of negative samples includes: By using the preset search terms and the predicted categories and entity recognition results corresponding to each preset search term as search conditions, an online search is performed to obtain a first preset number of positive samples. By using each preset search term and the entity recognition result corresponding to each preset search term as search conditions, and using the corresponding prediction category as exclusion conditions, an online search is performed to obtain a second preset number of negative samples. Based on preset annotation rules, perform corresponding annotation operations on each positive sample and each negative sample; The step of asynchronously storing detailed data from each search stage into a preset persistent database via a message queue based on preset tracking points includes: In the multi-path recall phase, a step is taken to asynchronously store the first search detail data of the current phase into a preset persistent database through a message queue based on the first preset tracking point; wherein, the first search detail data includes the first true positive data, the first false negative data, the first false positive data, and the first true negative data corresponding to the multi-path recall phase; During the sorting phase, the second search detail data of the current phase is asynchronously stored into the preset persistent database via a message queue based on the second preset tracking point. The step of performing corresponding annotation operations on each of the positive samples and each of the negative samples based on preset annotation rules includes: Based on the entity recognition results, the corresponding positive samples are labeled, and the labeling status of the corresponding samples is determined to be labeled. Based on the entity recognition results, the corresponding negative samples are labeled, and the labeling status of the corresponding samples is determined to be pending manual secondary labeling, so that subsequent manual review can determine whether labeling correction is necessary.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the real-time search quality evaluation method as described in claim 5.

7. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the real-time search quality evaluation method as described in claim 5.

Images