Retrieval effect evaluation method and system, electronic equipment and storage medium
By performing update training and sample playback evaluation of the map retrieval model based on user selected points, the problem of inability to confirm the true value and low coverage in map retrieval tests is solved, and effective evaluation and optimization of the search effect is achieved.
Patent Information
- Application Number
- CN202411617458.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-02-18
AI Technical Summary
In the prior art, map search tests cannot confirm the true value, have low coverage, and cannot effectively verify the search effect, which is more suitable for user groups.
By obtaining query parameters and user selected points, model training of the search model based on these data, obtaining the updated model, and sample replaying of the query parameters in the original model and the update model, and calculating the first indicator and the first index of each user selected points Two indicators, conduct differential monitoring to evaluate the model effect.
Quantitative evaluation of the retrieval model and update model effects is realized, and the badcase that leads to degradation of user experience can be quickly positioned and analyzed, the model effect is optimized, and the coverage and accuracy of map retrieval are improved.
Smart Images
Figure CN119474781A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a retrieval effect evaluation method, system, electronic equipment and storage medium. Background Art
[0002] Retrieval is a service that helps users quickly find the address information they need based on their query keywords and returns a list of search results that meet the query requirements. The main goal of the retrieval side is to improve the recall rate and accuracy rate. It is hoped that the address clicked by the user's search will be included in the returned search list, and the higher the ranking, the better.
[0003] Since the existing map retrieval function test can only verify the established rule strategies, but cannot evaluate the quality of strategies and models, and cannot quantify the evaluation indicators and evaluate the quality of the retrieval effect, it is impossible to confirm the true value during the map retrieval test, the coverage rate is low, and it is impossible to verify that the retrieval effect is more suitable for the user group. Summary of the invention
[0004] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a retrieval effect evaluation method, system, electronic device and storage medium, which are used to solve the problems in the prior art that the true value cannot be confirmed during map retrieval testing, the coverage rate is low, and the retrieval effect cannot be verified to be more suitable for the user group.
[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a retrieval effect evaluation method, comprising the following steps: obtaining query parameters and user-selected points; based on the query parameters and the user-selected points, performing model training on the retrieval model to obtain an updated model; performing sample playback of all query parameters in the retrieval model to obtain a first indicator corresponding to each user-selected point; performing sample playback of all query parameters in the updated model to obtain a second indicator corresponding to each user-selected point; performing difference monitoring on the first indicator and the second indicator to obtain a difference evaluation result between the two.
[0006] In one embodiment of the present invention, obtaining query parameters and user-selected locations includes: obtaining query parameters and user-selected locations corresponding to user search behaviors in real time through buried point reporting.
[0007] In one embodiment of the present invention, based on the query parameters and the points selected by the user, the retrieval model is trained to obtain an updated model, including: selecting part of the data in the query parameters and the points selected by the user as training samples; and training the retrieval model through the training samples to obtain an updated model.
[0008] In one embodiment of the present invention, all query parameters are sampled and replayed in the retrieval model to obtain a first indicator corresponding to each user-selected point, including: obtaining a result list returned by the retrieval model based on a playback request for the retrieval model according to the query parameters; monitoring the point information in the result list corresponding to the retrieval model; when the user-selected point exists in the result list corresponding to the retrieval model, obtaining the first result ranking of the user-selected point in the result list corresponding to the retrieval model and the first playback request quantity before the first result ranking in the result list corresponding to the retrieval model; obtaining a first indicator corresponding to the first result ranking based on the total playback request quantity and the first playback request quantity for playback requests in the retrieval model according to the query parameters.
[0009] In one embodiment of the present invention, monitoring of point information in a result list corresponding to a retrieval model includes: determining whether each point in the result list corresponding to the retrieval model has an identity identifier consistent with the point selected by the user; if so, determining that the point selected by the user exists in the result list corresponding to the retrieval model; if not, determining whether each point in the result list corresponding to the retrieval model has a normalized name and address consistent with the point selected by the user; if not, determining that the user selected point does not exist in the result list corresponding to the retrieval model; if so, further monitoring the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the retrieval model and the user selected point; when the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the retrieval model and the user selected point is less than a set value, determining that the user selected point exists in the result list corresponding to the retrieval model.
[0010] In one embodiment of the present invention, all query parameters are sampled and replayed in the update model to obtain a second indicator corresponding to each user-selected point, including: obtaining a result list returned by the update model retrieval based on the query parameters for the update model; monitoring the point information in the result list corresponding to the update model; when the user-selected point exists in the result list corresponding to the update model, obtaining the second result ranking of the user-selected point in the result list corresponding to the update model and the second playback request quantity before the second result ranking in the result list corresponding to the update model; obtaining a second indicator corresponding to the second result ranking based on the total playback request quantity and the second playback request quantity for playback requests in the update model according to the query parameters.
[0011] In one embodiment of the present invention, monitoring of point information in a result list corresponding to an updated model includes: determining whether each point in the result list corresponding to the updated model has an identity identifier consistent with the point selected by the user; if so, determining that the point selected by the user exists in the result list corresponding to the updated model; if not, determining whether each point in the result list corresponding to the updated model has a normalized name and address consistent with the point selected by the user; if not, determining that the point selected by the user does not exist in the result list corresponding to the updated model; if so, further monitoring the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the updated model and the point selected by the user; when the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the updated model and the point selected by the user is less than a set value, determining that the point selected by the user exists in the result list corresponding to the updated model.
[0012] The present invention also provides a retrieval effect evaluation system, including: a data acquisition unit, used to obtain query parameters and user-selected points; a training and updating unit, used to perform model training on the retrieval model based on the query parameters and the user-selected points to obtain an updated model; a first indicator summary unit, used to perform sample playback of all query parameters in the retrieval model to obtain a first indicator corresponding to each user-selected point; a second indicator summary unit, used to perform sample playback of all query parameters in the updated model to obtain a second indicator corresponding to each user-selected point; and a monitoring and analysis unit, used to perform difference monitoring of the first indicator and the second indicator to obtain a difference evaluation result between the two.
[0013] The present invention further provides an electronic device, which includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by one or more processors, the electronic device implements the aforementioned retrieval effect evaluation method.
[0014] The present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the aforementioned retrieval effect evaluation method.
[0015] As described above, a retrieval effect evaluation method, system, electronic device and storage medium of the present invention have the following beneficial effects: by using the query parameters obtained from the buried point report and the user selected point when searching in the retrieval model through the query parameters, the retrieval model can be updated and trained according to the query parameters and the user selected point to obtain an updated model, and then the first result ranking of each user selected point in the result list corresponding to the retrieval model and the second result ranking in the result list corresponding to the updated model can be determined, and the first index is obtained by summarizing the first result ranking, and the second index is obtained by summarizing the second result ranking, so as to compare the model effects of the retrieval model and the updated model according to the difference information between the first index and the second index, and obtain the specific detailed difference information of the different rankings corresponding to the model effects of the retrieval model and the updated model. When judging the pros and cons of the new and old models according to the indicators, bad cases that are very harmful to the user experience can also be quickly excavated, and the problem can be realized in the process of model effect quality assurance, which can quickly help algorithm personnel locate problems, analyze causes, and clarify the direction of subsequent iterations, and then can help map testers quickly test data and interface function iterations. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of a search effect evaluation method provided in an embodiment of the present invention.
[0017] Figure 2 Shown is a flowchart of a retrieval effect evaluation process provided by an embodiment of the present invention.
[0018] Figure 3 A flowchart showing a specific implementation process of the retrieval effect evaluation provided by an embodiment of the present invention is shown.
[0019] Figure 4 Shown is a schematic diagram of a display interface for effect verification provided by an embodiment of the present invention.
[0020] Figure 5 Shown is a structural block diagram of a retrieval effect evaluation system provided by an embodiment of the present invention.
[0021] Figure 6 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention.
[0022] Component number description
[0023] Electronic device 1; evaluation system 11; memory 12; processor 13; data acquisition unit 111, training update unit 112, first indicator summary unit 113, second indicator summary unit 114, monitoring and analysis unit 115. DETAILED DESCRIPTION
[0024] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0025] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0026] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.
[0027] See also Figure 1 In one embodiment of the present invention, the present invention provides a retrieval effect evaluation method, comprising:
[0028] Step S10: Obtain query parameters and the point selected by the user.
[0029] Step S20: Based on the query parameters and the point selected by the user, the retrieval model is trained to obtain an updated model.
[0030] Step S30: replay all query parameters in the retrieval model to obtain the first indicator corresponding to the point selected by each user.
[0031] Step S40: replay all query parameters in the updated model to obtain the second index corresponding to the point selected by each user.
[0032] Step S50: monitor the difference between the first indicator and the second indicator to obtain the difference evaluation result between the two.
[0033] In this embodiment, through step S10: obtaining query parameters and user-selected points, it can be seen that the query parameters may be search keywords, etc. The user-selected point may be the point result finally selected by the user by clicking on the map when the user enters the search keyword to search the map. After obtaining the query parameters and the user-selected point, it is possible to perform sample playback and demand analysis on the retrieval model, retrain and update the model, and perform sample playback and demand analysis on the updated model obtained by training and updating, etc. according to the query parameters and the user-selected point.
[0034] Through step S20: based on the query parameters and the user-selected points, the retrieval model is trained to obtain an updated model. It can be seen that after obtaining the query parameters and the user-selected points, the original retrieval model can be further trained to obtain an updated model after training. After obtaining the updated model, a comparative analysis of the retrieval effect with the original retrieval model can be performed to determine the model effect between the retrieval model and the updated model and the detailed difference information between the first indicator and the second indicator, thereby quickly helping algorithm personnel locate problems, analyze causes, and clarify the direction of subsequent iterative improvements.
[0035] Through step S30: replaying all query parameters in the retrieval model to obtain the first index corresponding to each user selected point, it can be achieved that after obtaining the query parameters and the user selected point, all query parameters are replayed in the retrieval model, so that the first index of each user selected point can be determined after the sample playback. After obtaining the first index, the difference of the index can be compared with the update model during the sample playback to determine the difference evaluation results between the retrieval model and the update model, such as the pros and cons of the model effects between the retrieval model and the update model and the detailed difference information between the first index and the second index.
[0036] Through step S40: replaying all query parameters in the updated model to obtain the second index corresponding to each user's selected point, it can be seen that after replaying all query parameters through the retrieval model to obtain the first index corresponding to each user's selected point, when performing a comparative analysis of the retrieval effect with the corresponding index of the updated model, all query parameters that were replayed in the retrieval model are also replayed in the updated model to obtain the second index corresponding to each user's selected point. After obtaining the first index and the second index, the retrieval effect between the two models and the detailed difference information between the first index and the second index can be determined according to the index size, thereby quickly helping algorithm personnel locate problems, analyze causes, and clarify the direction of subsequent iterations.
[0037] Through step S50: monitoring the difference between the first indicator and the second indicator, and obtaining the difference evaluation results between the two, it can be seen that after obtaining the first indicator and the second indicator, the difference evaluation results between the retrieval model and the update model can be determined by comparing the first indicator and the second indicator, such as the pros and cons of the corresponding model effects between the retrieval model and the update model and the detailed difference information between the first indicator and the second indicator, etc., and then according to the pros and cons of the retrieval effect of the model and the detailed difference information, it can quickly help algorithm personnel locate problems, analyze causes, and clarify the direction of subsequent iterative improvements. Specifically, when the first indicator is greater than the second indicator, it means that the model effect of the retrieval model is better than the update model. Conversely, when the second indicator is greater than the first indicator, it means that the model effect of the update model is better than the retrieval model.
[0038] In one embodiment of the present invention, query parameters and user-selected locations are obtained, including: obtaining query parameters and user-selected locations corresponding to user search behaviors in real time through buried point reporting. In this embodiment, when obtaining query parameters and user-selected locations, query parameters may be search keywords, etc. Before obtaining, buried points may be performed at operation codes such as site selection clicks and order placement on the terminal. When operations such as site selection clicks and order placement are obtained, buried point data for operations such as site selection clicks and order placement are reported, and the buried point data includes query parameters and user-selected locations. Through buried point reporting, samples can be extracted for sample playback and some samples can be selected as training models for update training of the retrieval model to obtain an updated model.
[0039] See also Figure 2 In a preferred embodiment of the present invention, when conducting a retrieval effect test, user behaviors are reported through buried points, such as retrieval keywords and user-selected locations during site selection clicks, order placement, and other operations. Through buried point reporting, samples can be extracted for sample playback and some samples can be selected as training models for updating and training the retrieval model to obtain an updated model. When playing back the samples, the query (query parameter) is extracted from the buried point reporting data, demand analysis is performed, and samples are selected. Specifically, by replaying the queries in all samples in the retrieval model and the update model respectively, the indicators corresponding to the retrieval model and the update model are calculated respectively, and the difference in indicators between the two is compared to determine the difference information and model pros and cons between the retrieval model and the update model. Subsequently, the difference information can be used to quickly mine bad cases that are very harmful to the user experience, so as to expose existing problems and point out the direction for the improvement of subsequent indicators.
[0040] In one embodiment of the present invention, based on the query parameters and the user-selected points, the retrieval model is trained to obtain an updated model, including: selecting the query parameters and part of the data in the user-selected points as training samples; training the retrieval model through the training samples to obtain an updated model. In this embodiment, when performing model update training, the query parameters and the user-selected points can be used as training samples to further train the retrieval model to adjust the updated model.
[0041] In one embodiment of the present invention, all query parameters are sampled and replayed in the retrieval model to obtain a first indicator corresponding to each user-selected point, including: obtaining a result list returned by the retrieval model based on a playback request for the retrieval model according to the query parameters; monitoring the point information in the result list corresponding to the retrieval model; when the user-selected point exists in the result list corresponding to the retrieval model, obtaining the first result ranking of the user-selected point in the result list corresponding to the retrieval model and the first playback request quantity before the first result ranking in the result list corresponding to the retrieval model; obtaining a first indicator corresponding to the first result ranking based on the total playback request quantity and the first playback request quantity for playback requests in the retrieval model according to the query parameters.
[0042] In this embodiment, when all query parameters are sampled for playback in the retrieval model, sample playback is performed on the playback request of the retrieval model according to all query parameters. And after each query parameter is sampled for playback in the retrieval model, the corresponding map retrieval result will be returned to the result list corresponding to the retrieval model. When the point information of the result list corresponding to the retrieval model is monitored, when it is detected that there is a user-selected point in the result list corresponding to the retrieval model, the first result ranking of the corresponding user-selected point in the result list corresponding to the retrieval model will be further calculated. And after obtaining the first result ranking, the first index corresponding to the first result ranking can be further determined based on the first playback request volume for the retrieval model before the first result ranking in the result list, and the total playback request volume for playback requests of all query parameters in the retrieval model. Specifically, the calculation formula of the first index is: Among them, P1 is the first indicator, Q1 is the number of first playback requests for the retrieval model before the first result in the result list, and Q 总1 The total number of playback requests made in the retrieval model for all query parameters.
[0043] In one embodiment of the present invention, monitoring of point information in a result list corresponding to a retrieval model includes: determining whether each point in the result list corresponding to the retrieval model has an identity identifier consistent with the point selected by the user; if so, determining that the point selected by the user exists in the result list corresponding to the retrieval model; if not, determining whether each point in the result list corresponding to the retrieval model has a normalized name and address consistent with the point selected by the user; if not, determining that the point selected by the user does not exist in the result list corresponding to the retrieval model; if so, further monitoring the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the retrieval model and the point selected by the user; when the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the retrieval model and the point selected by the user is less than a set value, determining that the point selected by the user exists in the result list corresponding to the retrieval model.
[0044] In this embodiment, when determining whether there is the same point corresponding to the point selected by the user in the result list corresponding to the retrieval model, it can be determined first whether there is a point in the result list corresponding to the retrieval model whose identity is consistent with the identity of the point selected by the user. If there is a consistent identity, it is determined that the point corresponding to the consistent identity is the same point as the point selected by the user. If they are inconsistent, it is further determined whether the name and address of the point in the result list and the point selected by the user after normalization are consistent. If they are consistent, then when it is determined that the distance between the longitude and latitude coordinates of the corresponding point in the result list and the point selected by the user after normalization is less than a set value, it can also be considered that the corresponding point in the result list and the point selected by the user after normalization are the same as the point selected by the user.
[0045] In one embodiment of the present invention, all query parameters are sampled and replayed in the update model to obtain a second indicator corresponding to each user-selected point, including: obtaining a result list returned by the update model retrieval based on the query parameters for the update model; monitoring the point information in the result list corresponding to the update model; when the user-selected point exists in the result list corresponding to the update model, obtaining the second result ranking of the user-selected point in the result list corresponding to the update model and the second playback request quantity before the second result ranking in the result list corresponding to the update model; obtaining a second indicator corresponding to the second result ranking based on the total playback request quantity and the second playback request quantity for playback requests in the update model according to the query parameters.
[0046] In this embodiment, when all query parameters are sampled for playback in the update model, the playback requests for the update model are sampled for playback in sequence according to all query parameters. And after each query parameter is sampled for playback in the update model, the corresponding map retrieval result will be returned to the result list corresponding to the update model. When the point information of the result list corresponding to the update model is monitored, when it is detected that there is a user-selected point in the result list corresponding to the update model, the second result ranking of the corresponding user-selected point in the result list corresponding to the update model will be further calculated. And after obtaining the second result ranking, the second index corresponding to the second result ranking can be further determined based on the second playback request volume for the update model before the second result ranking in the result list, and the total playback request volume for all query parameters in the update model. Specifically, the calculation formula for the second index is: Among them, P2 is the second indicator, Q2 is the second playback request quantity for the updated model before the second result in the result list, and Q 总2 The total number of playback requests made in the update model for all query parameters.
[0047] In one embodiment of the present invention, monitoring of point information in a result list corresponding to an updated model includes: determining whether each point in the result list corresponding to the updated model has an identity identifier consistent with the point selected by the user; if so, determining that the user selected point exists in the result list corresponding to the updated model; if not, determining whether each point in the result list corresponding to the updated model has a normalized name and address consistent with the user selected point; if not, determining that the user selected point does not exist in the result list corresponding to the updated model; if so, further monitoring the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the updated model and the user selected point; when the longitude and latitude coordinate distance between the corresponding point in the result list corresponding to the updated model and the user selected point is less than a set value, determining that the user selected point exists in the result list corresponding to the updated model.
[0048] In this embodiment, when determining whether there is the same point corresponding to the point selected by the user in the result list corresponding to the updated model, it can be determined whether there is a point in the result list corresponding to the updated model whose identity is consistent with the identity of the point selected by the user. If there is a consistent identity, it is determined that the point corresponding to the consistent identity is the same point as the point selected by the user. If they are inconsistent, it is further determined whether the name and address of the point in the result list and the point selected by the user after normalization are consistent. If they are consistent, it is further determined that when the distance between the longitude and latitude coordinates of the corresponding point in the result list and the point selected by the user after normalization is less than a set value, it can also be considered that the corresponding point in the result list and the point selected by the user after normalization are the same as the point selected by the user.
[0049] See also Figure 3 In a preferred embodiment, when evaluating the retrieval effect, the control group is set to call the query (query parameter) in batches to perform sample playback on the retrieval model, and the experimental group is set to call the query in batches to perform sample playback on the updated model after update training. During sample playback, each query obtains the map retrieval return result, and when there is a return result, it is returned to the result list of the control group and the experimental group. By traversing each (poi) point information in the returned result poi-list (point list), it is determined whether it is equal to the click data. When making a specific judgment, first determine whether the id is connected, that is, determine whether the identity of the point in the result list is consistent with the identity of the point selected by the user. If they are consistent, determine that the point corresponding to the consistent identity is the same point as the point selected by the user; if they are inconsistent, further determine whether the normalized addr (address) / name (name) of the point in the result list is equal to the normalized addr (address) / name (name) of the point selected by the user. If they are equal, further determine whether the distance between the points is ≤100m. If so, determine that the corresponding point in the result list with the name and address of the point selected by the user after normalization is the same as the point selected by the user. After determining that there is a point in the result list that is the same as the point selected by the user, the ranking of the selected point in the result list is determined, and the first index of the control group and the second index of the experimental group are further summarized. Compare the first index of the control group and the second index of the experimental group, further output the conclusion of the model effect, and obtain the specific detailed difference information of the different rankings corresponding to the control group and the experimental group.
[0050] In a preferred embodiment of the present invention, after obtaining the first result ranking of each point in the control group and the second result ranking of the experimental group, the first index corresponding to the control group and the second index of the experimental group can be further calculated by summarizing the result rankings. By comparing the first index and the second index, the advantages and disadvantages between the retrieval model and the update model are determined, and detailed difference information corresponding to different rankings is further obtained, so as to facilitate the subsequent analysis of the existing problems exposed by the bad cases corresponding to the detailed difference information, and point out the direction for the improvement of subsequent indicators.
[0051] See also Figure 4 , the current interface shows the search effect evaluation. The left side includes the experimental group ID input area, the control group ID input area, the experimental group abParam input area, the control group abParam input area, the data level input area, the data source dt input area, and the operating environment selection area; and by selecting "Run", you can get the detailed difference result information list on the right. In the detailed difference result information list on the right, by selecting the specified sequence number, you can get the hit rate (indicator) of the experimental group and the control group under the "Run" option in the right column. Among them, the total ranking in the result list can be set to 10, corresponding to top1 to top10. The currently displayed list is: when the user selects the point ranking as top1, the hit rate of the experimental group is 61.1%, and the hit rate of the control group is 53.94%; when the user selects the point ranking as top3, the hit rate of the experimental group is 78.3%, and the hit rate of the control group is 69.38%; when the user selects the point ranking as top10, the hit rate of the experimental group is 85.08%, and the hit rate of the control group is 75.94%; when the user selects the point ranking as HLLtop1, the hit rate of the experimental group is 8.2%, and the hit rate of the control group is 50.28%.
[0052] like Figure 5 As shown, in one embodiment of the present invention, the present invention also provides a retrieval effect evaluation system 11, including: a data acquisition unit 111, used to obtain query parameters and user-selected points; a training update unit 112, used to perform model training on the retrieval model based on the query parameters and the user-selected points, and obtain an updated model; a first indicator aggregation unit 113, used to perform sample playback of all query parameters in the retrieval model, and obtain the first indicator corresponding to each user-selected point; a second indicator aggregation unit 114, used to perform sample playback of all query parameters in the updated model, and obtain the second indicator corresponding to each user-selected point; and a monitoring and analysis unit 115, used to perform difference monitoring of the first indicator and the second indicator, and obtain the difference evaluation results between the two.
[0053] In this embodiment, in the retrieval effect evaluation system of the present invention, the data acquisition unit 111 can obtain the query parameters and the user-selected points through the point reporting method of the point data. Through the training update unit 112, it can be realized that according to the query parameters and the user-selected points obtained, the retrieval model is further updated and trained to obtain an updated model. And the query parameters in the sample are respectively played back in the retrieval model and the update model through the first indicator summary unit 113 and the second indicator summary unit 114, so as to obtain the first indicator and the second indicator, and the difference between the first indicator and the second indicator is evaluated by the monitoring and analysis unit 115, and the difference evaluation result of the two is obtained based on the comparison. Therefore, the quality of the model effect is determined according to the difference evaluation result, and based on the quality of the model effect, the detailed information of the badcase is output for subsequent iterative optimization.
[0054] It should be noted that the retrieval effect evaluation system 11 provided in the above embodiment and the retrieval effect evaluation method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the retrieval effect evaluation system 11 provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0055] See also Figure 6 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a retrieval effect evaluation program.
[0056] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as codes for retrieval effect evaluation, etc., but also can be used to temporarily store data that has been output or is to be output.
[0057] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, and executes or executes programs or modules (such as retrieval effect evaluation programs, etc.) stored in the memory 12, and calls the data stored in the memory 12 to execute various functions of the electronic device 1 and process data.
[0058] The processor 13 executes the operating system and various installed application programs of the electronic device 1. The processor 13 executes the application programs to implement the steps in the above-mentioned retrieval effect evaluation method.
[0059] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition unit 111, a training update unit 112, a first indicator summary unit 113, a second indicator summary unit 114, and a monitoring and analysis unit 115.
[0060] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the retrieval effect evaluation method described in each embodiment of the present application.
[0061] In summary, the present invention discloses a method, system, electronic device and storage medium for evaluating retrieval effect. By using the query parameters obtained from the buried point report and the user selected point when searching in the retrieval model through the query parameters, the retrieval model can be updated and trained according to the query parameters and the user selected point to obtain the updated model, and then the first result ranking of each user selected point in the result list corresponding to the retrieval model and the second result ranking in the result list corresponding to the updated model can be determined, and the first index is obtained by summarizing the first result ranking, and the second index is obtained by summarizing the second result ranking, so as to compare the model effects of the retrieval model and the updated model according to the difference information between the first index and the second index, and obtain the specific detailed difference information of the different rankings corresponding to the model effects of the retrieval model and the updated model. Thus, the advantages and disadvantages of the new and old models can be determined according to the index, and the bad cases that are very harmful to the user experience can be quickly excavated. In the process of ensuring the quality of the model effect, the problem can be realized first, and the algorithm personnel can be quickly helped to locate the problem, analyze the cause, and clarify the direction of subsequent iteration improvement, so as to help the map test personnel to quickly test the data and interface function iteration. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.
[0062] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A search effect evaluation method, characterized in that: The steps include: Get query parameters and user-selected points; Based on the query parameters and the point selected by the user, the retrieval model is trained to obtain an updated model; Perform sample playback of all the query parameters in the retrieval model to obtain a first indicator corresponding to each point selected by the user; Perform sample playback of all the query parameters in the update model to obtain a second indicator corresponding to each point selected by the user; Perform difference monitoring between the first indicator and the second indicator to obtain a difference evaluation result between the two.
2. The search effect evaluation method according to claim 1, characterized in that: Get query parameters and user-selected points, including: The query parameters corresponding to the user's search behavior and the user's selected point are obtained in real time through point reporting.
3. The search effect evaluation method according to claim 1, characterized in that: Based on the query parameters and the point selected by the user, the retrieval model is trained to obtain an updated model, including: Selecting the query parameter and part of the data in the point selected by the user as training samples; The retrieval model is trained using the training samples to obtain the updated model.
4. The search effect evaluation method according to claim 1, characterized in that: All the query parameters are sampled and replayed in the retrieval model to obtain the first index corresponding to each point selected by the user, including: Requesting a playback of the retrieval model according to the query parameters to obtain a result list returned by searching the retrieval model; Monitoring the point information in the result list corresponding to the retrieval model; When the user-selected point exists in the result list corresponding to the retrieval model, obtaining the first result ranking of the user-selected point in the result list corresponding to the retrieval model and the first playback request quantity before the first result ranking in the result list corresponding to the retrieval model; The first indicator corresponding to the first result ranking is obtained according to the total playback request quantity and the first playback request quantity for playback requests in the retrieval model using the query parameter.
5. The search effect evaluation method according to claim 4, characterized in that: Monitoring the point information in the result list corresponding to the retrieval model includes: Determine whether there is an identity identifier consistent with the point selected by the user for each point in the result list corresponding to the search model; If so, determining that the point selected by the user exists in the result list corresponding to the search model; If not, determining whether each point in the result list corresponding to the search model has a normalized name and address that is consistent with the point selected by the user; If not, determining that the point selected by the user does not exist in the result list corresponding to the retrieval model; If yes, further monitoring the latitude and longitude coordinate distance between the corresponding point in the result list corresponding to the retrieval model and the point selected by the user; When the latitude and longitude coordinate distance between the corresponding point in the result list corresponding to the retrieval model and the point selected by the user is less than a set value, it is determined that the point selected by the user exists in the result list corresponding to the retrieval model.
6. The search effect evaluation method according to claim 1, characterized in that: All the query parameters are sampled and replayed in the update model to obtain the second index corresponding to each point selected by the user, including: Requesting playback of the update model according to the query parameter, obtaining a result list returned by searching the update model; Monitoring the point information in the result list corresponding to the updated model; When the user-selected point exists in the result list corresponding to the updated model, obtaining a second result ranking of the user-selected point in the result list corresponding to the updated model and a second playback request quantity before the second result ranking in the result list corresponding to the updated model; The second indicator corresponding to the second result ranking is obtained according to the total playback request quantity and the second playback request quantity for playback requests in the update model according to the query parameter.
7. The search effect evaluation method according to claim 6, characterized in that: Monitoring the point information in the result list corresponding to the updated model includes: Determine whether each point in the result list corresponding to the updated model has an identity identifier that is consistent with the point selected by the user; If yes, determining that the point selected by the user exists in the result list corresponding to the updated model; If not, determining whether each point in the result list corresponding to the updated model has a normalized name and address that is consistent with the point selected by the user; If not, determining that the point selected by the user does not exist in the result list corresponding to the updated model; If yes, further monitoring the latitude and longitude coordinate distance between the corresponding point in the result list corresponding to the updated model and the point selected by the user; When the latitude and longitude coordinate distance between the corresponding point in the result list corresponding to the updated model and the point selected by the user is less than a set value, it is determined that the point selected by the user exists in the result list corresponding to the updated model.
8. A search effect evaluation system, characterized in that: include: A data acquisition unit, used to obtain query parameters and user-selected points; A training and updating unit, configured to perform model training on the retrieval model based on the query parameters and the point selected by the user to obtain an updated model; A first indicator aggregation unit is used to perform sample playback of all the query parameters in the retrieval model to obtain a first indicator corresponding to each point selected by the user; A second indicator aggregation unit is used to perform sample playback of all the query parameters in the update model to obtain a second indicator corresponding to each point selected by the user; as well as The monitoring and analysis unit is used to monitor the difference between the first indicator and the second indicator and obtain the difference evaluation results between the two.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the retrieval effect evaluation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the retrieval effect evaluation method described in any one of claims 1 to 7.