Evaluation method and device

Abstract

The invention provides a method and device for evaluating a search system, electronic equipment and a non-instantaneous computer readable storage medium, and the method comprises the steps: obtaining an evaluation task corresponding to an evaluation instruction in response to the evaluation instruction; determining a sample set and an evaluation index according to the evaluation task; by utilizing the sample set, calling a search system to obtain a search result set corresponding to the sample set; and evaluating the search system by utilizing the evaluation index, the sample set and the search result set. According to the embodiment of the invention, multiple data sets are selected in batches, so that the confidence coefficient of overall effect evaluation is improved; the execution efficiency of the evaluation method is improved by using a task distribution scheduling technology.

Description

Technical Field

[0001] The present application relates to the field of search, and more particularly, to a method and apparatus for evaluating a search system, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Image search is proposed to meet the needs of users to find the same products and purchase from the source. With the innovation of technology, it is necessary to iterate the version of the image search system. Due to the frequent iteration process, the existing method of manually evaluating the image search effect is time-consuming and inefficient, and the number of test images is small, the coverage rate is low, and the confidence level of the overall effect evaluation is low. Therefore, the current method of manually evaluating the image search effect can no longer meet the iteration requirements of image search. Summary of the Invention

[0003] The present application aims to propose a method and apparatus for evaluating a search system, an electronic device, and a non-transitory computer-readable storage medium to solve the problems of low efficiency and low sample confidence level existing in the existing method of manually evaluating the image search effect.

[0004] According to one aspect of the present application, a method for evaluating a search system is provided, including: in response to an evaluation instruction, obtaining an evaluation task corresponding to the evaluation instruction; determining a sample set and an evaluation index according to the evaluation task; using the sample set to obtain a search result set corresponding to the sample set by calling the search system; and evaluating the search system by using the evaluation index, the sample set, and the search result set.

[0005] According to some embodiments, the evaluation task includes an image search effect evaluation task.

[0006] According to some embodiments, the sample set includes manually evaluated samples and / or users' image search requests.

[0007] According to some embodiments, the evaluation index includes the similarity between the search image and the result image, the fluctuation of the average similarity score, the empty result rate, the similarity of the main image of the first product, and / or the category inconsistency rate.

[0008] According to some embodiments, the method further includes updating the evaluation index.

[0009] According to some embodiments, the evaluation task includes a recommendation effect evaluation task.

[0010] According to some embodiments, the evaluation method further includes statistically analyzing the evaluation result of the search system and visually displaying the statistical content.

[0011] According to one aspect of the present application, there is provided an apparatus for evaluating a search system, including an evaluation task acquisition unit configured to acquire an evaluation task corresponding to the evaluation instruction in response to the evaluation instruction; a sample set and evaluation index determination unit configured to determine a sample set and evaluation indexes according to the evaluation task; a search result set acquisition unit configured to obtain a search result set corresponding to the sample set by invoking the search system by using the sample set; and an evaluation unit configured to evaluate the search system by using the evaluation indexes, the sample set, and the search result set.

[0012] According to one aspect of the present application, there is provided an electronic device, including a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the method as described in any of the previous embodiments.

[0013] According to one aspect of the present application, there is provided a non-transitory computer-readable storage medium having stored thereon computer-readable instructions, which, when executed by a processor, cause the processor to execute the method as described in any of the previous embodiments.

[0014] According to the embodiments of the present application, by batch-selecting multiple data sets, the confidence level of the overall effect evaluation is improved; by using the task distribution and scheduling technology, the execution efficiency of the evaluation method is improved.

[0015] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. By referring to the drawings and describing its exemplary embodiments in detail, the above and other objects, features, and advantages of the present application will become more obvious.

[0017] Figure 1 FIG. shows a flowchart of a method for evaluating a search system according to an exemplary embodiment of the present application.

[0018] Figure 2 FIG. shows another flowchart of a method for evaluating a search system according to an exemplary embodiment of the present application.

[0019] Figure 3 FIG. shows a block diagram of an apparatus for evaluating a search system according to an exemplary embodiment of the present application.

[0020] Figure 4 FIG. shows an electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Identical reference numerals in the figures denote identical or similar parts, and thus their repetitive description will be omitted.

[0022] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0023] The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of these specific details, or other means, components, materials, devices, or operations, etc. may be adopted. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0024] The flowcharts shown in the accompanying drawings are merely illustrative and not necessarily include all the contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0025] The terms "first", "second", etc. in the specification, claims, and above-mentioned accompanying drawings of this application are used to distinguish different objects and not to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0026] Image search is proposed to meet the needs of users to find the same products and purchase from the source. With the innovation of technology, it is necessary to iterate the image search version. Since each version iteration requires an evaluation of the updated version.

[0027] The existing evaluation of image search effects mainly includes two methods. One is to evaluate the search algorithm. This evaluation only assesses the performance of the algorithm and does not involve evaluating indicators such as the matching similarity of search results. Another commonly used method for evaluating image search effects is through manual evaluation. This evaluation method conducts search tests by randomly selecting several pictures and manually evaluating the quality of search results. Therefore, due to the low sample coverage rate, the confidence level of the overall effect evaluation is low; and there are problems such as high cost and long time consumption.

[0028] According to the embodiments of the present application, by batch-selecting multiple data sets, the confidence level of the overall effect evaluation is improved; by using task distribution and scheduling technology, the execution efficiency of the evaluation method is improved.

[0029] The following will describe in detail the specific embodiments according to the present application with reference to the accompanying drawings.

[0030] Figure 1 FIG. shows a flowchart of a method for evaluating a search system according to an exemplary embodiment of the present application. The following will Figure 1 take... as an example to describe in detail a method for evaluating a search system according to an exemplary embodiment of the present application. As Figure 1 shown, Figure 1 the method shown includes steps S101, S103, S105, and S107.

[0031] In step S101, in response to an evaluation instruction, an evaluation task corresponding to the evaluation instruction is obtained.

[0032] According to the embodiments of the present application, the evaluation tasks include image search (abbreviated as "graph search") tasks and recommendation tasks.

[0033] In step S103, a sample set and evaluation indicators are determined according to the evaluation task.

[0034] Since the existing method for evaluating image search effects conducts search tests by randomly selecting several pictures, the confidence level of the overall effect evaluation is relatively low. In the embodiments of the present application, by batch-selecting multiple data sets, the problem of low confidence level in the prior art is solved.

[0035] For example, the sample set participating in the evaluation task includes not only manually evaluated samples but also request samples of real online users.

[0036] In some embodiments, the manually evaluated samples include manually evaluated feedback data and the main product pictures associated with the user's hot search pictures. Each sample contains information such as pictures, corresponding product IDs, and the categories to which they belong.

[0037] In some other embodiments, the request samples of online real users include the image searches that users truly request on the image search platform according to their needs.

[0038] According to the embodiments of the present application, in step S103, by batch-selecting multiple data sets, the confidence level of the overall effect evaluation is improved.

[0039] In step S105, using the sample set, a search result set corresponding to the sample set is obtained by calling the search system.

[0040] As mentioned above, the evaluation tasks include the image search (abbreviated as image search) task and the recommendation task. Correspondingly, the search system in step S105 includes an image search system and a recommendation system. Among them, the image search system is used to provide the same, relevant or similar products for users according to the pictures provided by users. The recommendation system is used to provide the same, relevant or similar products for users according to the keywords provided by users.

[0041] As mentioned above, in step S103, batch-selecting multiple data sets increases the number of the sample set. Therefore, in step S105, using the task distribution and scheduling technology, multiple samples are processed synchronously to achieve the output of the evaluation results corresponding to multiple samples, thereby improving the execution efficiency of this evaluation method.

[0042] In step S107, using the evaluation metrics, the sample set and the search result set, the search system is evaluated.

[0043] According to the embodiments of the present application, the evaluation metrics include the similarity between the search image and the result image, the fluctuation of the average similarity score, the empty result rate, the similarity of the main image of the first product, and / or the category inconsistency rate. Among them, the similarity between the search image and the result image is to calculate the similarity degree between the search image and the result image; the fluctuation of the average similarity score is to calculate the average value or variance of the similarity degree between the search image and the result image of the entire or part of the sample set; the category inconsistency rate is to calculate the ratio of the categories that are different between the search image and the result image. The empty result rate is the ratio of the evaluation results corresponding to the samples being empty.

[0044] For example, the category inconsistency rate is the proportion of the category inconsistencies between the search image and the first 10 product categories in the search results.

[0045] For a sample whose corresponding search result set includes multiple search results, the evaluation metric also includes the similarity of the first product.

[0046] In some embodiments, the search result set is sorted in descending order according to the similarity. The similarity of the first product refers to the similarity between the sample and the first search result in the search result set.

[0047] According to some embodiments of the present application, the evaluation metrics are not fixed and can be added, deleted, or modified according to requirements.

[0048] For example, delete the evaluation metric.

[0049] For another example, dynamically access new evaluation metrics.

[0050] For another example, edit the existing metrics and re-define the existing metrics.

[0051] According to Figure 1 the embodiments shown, by batch-selecting multiple data sets, the confidence of the overall effectiveness evaluation is improved; by using the task distribution and scheduling technology, the execution efficiency of the evaluation method is improved. At the same time, it overcomes the problem in the prior art that only the performance of the search algorithm is considered and the evaluation of metrics such as the matching similarity of search results is not involved.

[0052] Figure 2 shows another method flowchart for evaluating a search system according to an exemplary embodiment of the present application. As Figure 2 shown, in addition to before steps S101 to S107, Figure 1 the method shown further includes step S109. For the sake of brevity, only the differences between the Figure 2 embodiments shown and Figure 1 will be described below, and the detailed description of their similarities will be omitted.

[0053] In step S109, the evaluation results of the search system are statistically analyzed, and the statistical content is visually displayed.

[0054] In some embodiments, in step S109, the evaluation results obtained in step S107 are directly displayed.

[0055] In other embodiments, in step S109, the evaluation results are also statistically analyzed.

[0056] For example, according to the weights of a preset plurality of evaluation metrics, the evaluation statistical value of the search system is calculated, so as to evaluate the search system according to the evaluation statistical value and give evaluation opinions. For example, if the search system evaluation passes, it can be put on the line; or if the search system evaluation fails, it needs to be further optimized.

[0057] It should be noted here that the evaluation opinions of the search system can be evaluated according to the evaluation statistical values of multiple evaluation metrics, or according to any one evaluation metric. For example, when the similarity of the first product main image is less than 0.3, it is considered that the similarity of the first product is too poor, and the search system is determined to have failed the evaluation and needs to be further optimized.

[0058] The above mainly introduced the embodiments of the present application from the perspective of methods. Those skilled in the art should easily realize that, in combination with the operations or steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Those skilled in the art can use different methods to implement the described functions for each specific operation or method, and such implementation should not be considered to exceed the scope of the present application.

[0059] The following describes the device embodiments of the present application. For details not described in the device embodiments of the present application, reference can be made to the method embodiments of the present application.

[0060] Figure 3 The block diagram of a device for evaluating a search system according to an exemplary embodiment of the present application is shown, as Figure 3 The shown device includes an evaluation task acquisition unit 301, a sample set and evaluation index determination unit 303, a search result set acquisition unit 305, and an evaluation unit 307. Among them, the evaluation task acquisition unit 301 is configured to acquire an evaluation task corresponding to the evaluation instruction in response to the evaluation instruction; the sample set and evaluation index determination unit 303 is configured to determine a sample set and evaluation indexes according to the evaluation task; the search result set acquisition unit 305 is configured to use the sample set to obtain a search result set corresponding to the sample set by invoking the search system; the evaluation unit 307 is configured to evaluate the search system by using the evaluation indexes, the sample set, and the search result set.

[0061] Figure 4 An electronic device according to an exemplary embodiment of the present application is shown. The following refers to Figure 4 to describe the electronic device 200 according to this embodiment of the present application. Figure 4 The shown electronic device 200 is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present application.

[0062] As Figure 4 shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0063] Among them, the storage unit stores program codes, and the program codes can be executed by the processing unit 210, so that the processing unit 210 executes the methods according to various exemplary embodiments of the present application described in this specification. For example, the processing unit 210 can execute the method as Figure 1 shown.

[0064] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203.

[0065] The storage unit 220 may also include a program / utilities 2204 having a set (at least one) of program modules 2205. Such program modules 2205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0066] The bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0067] The electronic device 200 may also communicate with one or more external devices 300 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 200, and / or may communicate with any device that enables the electronic device 200 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be through an input / output (I / O) interface 250. Moreover, the electronic device 200 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 through the bus 230. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0068] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. The technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present application.

[0069] A software product may employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0070] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which case the data signal carries the readable program code. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0071] The program code for performing the operations of this application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partially on the user's device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or alternatively, may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0072] The foregoing computer-readable medium bears one or more programs, which when executed by a device, cause the computer-readable medium to implement the foregoing functions.

[0073] Those skilled in the art can understand that the above-mentioned modules can be distributed in the device according to the description of the embodiments, or can be correspondingly changed and distributed in one or more devices that are only different from this embodiment. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0074] According to an embodiment of the present application, a computer program is provided, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the above-described method can be executed.

[0075] The above has introduced the embodiments of the present application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. At the same time, those skilled in the art, based on the idea of the present application, the changes or deformations made in the specific implementation manner and application scope of the present application all belong to the protection scope of the present application. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for evaluating a search system, characterized in that, Comprising: In response to an evaluation instruction, obtain an evaluation task corresponding to the evaluation instruction; Determine a sample set and evaluation metrics according to the evaluation task; Utilize the sample set to obtain a search result set corresponding to the sample set by invoking a search system; Evaluate the search system by using the evaluation metrics, the sample set, and the search result set.

2. The method according to claim 1, characterized in that, The evaluation task includes an evaluation task for the picture search effect.

3. The method according to claim 2, wherein The sample set includes manually evaluated samples and / or the search requests of users.

4. The method according to claim 2, wherein The evaluation metrics include the similarity between the searched picture and the result picture, the fluctuation of the average similarity score, the null result rate, the similarity of the main picture of the first product, and / or the category inconsistency rate.

5. The method according to claim 4, wherein The method further includes: updating the evaluation metrics.

6. The method according to claim 1, wherein The evaluation task includes an evaluation task for the recommendation effect.

7. The method according to claim 1, characterized in that, The evaluation method further includes: Statistically analyze the evaluation result of the search system and visually display the statistical content.

8. A device for evaluating a search system, characterized in that, Comprising: An evaluation task acquisition unit, configured to obtain an evaluation task corresponding to the evaluation instruction in response to the evaluation instruction; A sample set and evaluation metric determination unit, configured to determine a sample set and evaluation metrics according to the evaluation task; A search result set acquisition unit, configured to utilize the sample set to obtain a search result set corresponding to the sample set by invoking a search system; An evaluation unit, configured to evaluate the search system by using the evaluation metrics, the sample set, and the search result set.

9. An electronic device, comprising: A processor; And A memory storing a computer program, which when executed by the processor causes the processor to execute the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium, having stored thereon computer-readable instructions, which when executed by a processor cause the processor to execute the method according to any one of claims 1-7.

Images