Medical search model performance evaluation method, device, equipment and medium

Abstract

This application provides a method, apparatus, device, and medium for evaluating the performance of a medical search model. The method first obtains multiple types of data samples based on medical search user behavior parameters; then, based on predetermined indicator judgment logic, determines at least one evaluation indicator; and finally, selects data samples of the type corresponding to the user behavior parameters associated with the determined evaluation indicator to evaluate the medical search model. Because the data samples are divided into different types based on user behavior parameters, different types of samples can be used to reflect different user search characteristics. Furthermore, for different evaluation indicators, data samples associated with the evaluation indicators are used as evaluation samples. This improves the user behavior support of the samples, effectively improving the accuracy of the indicator evaluation and providing a reliable basis for optimizing and improving model performance.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a method, device, equipment and medium for evaluating the performance of a medical search model. Background Art

[0002] Search is a user-initiated information interaction, and the relevance of the results further impacts the product's user experience. For search functionality in the medical field, in addition to considering the user experience, due to the rigor of medical research, the accuracy of results is more important for specific user searches than the availability of search results. For example, when searching for certain medications, accuracy is prioritized over the number of search results to avoid medical risks. For medical search solutions based on medical search models, how to evaluate the performance of these models is a technical challenge that needs to be addressed. Summary of the Invention

[0003] In view of this, the present application provides a medical search model performance evaluation method, device, equipment and medium, the main purpose of which is to evaluate the performance of the medical search model and provide a basis for improving the performance of the medical search model.

[0004] According to one aspect of the present application, a medical search model performance evaluation method is provided, including: obtaining historical search data as an initial data sample set, and / or constructing an initial data sample set based on medical keywords and medical terms, dividing the initial data sample set according to medical search user behavior parameters to obtain multiple types of data samples; according to the type parameters and / or focus parameters of the medical search model, according to a predetermined indicator judgment logic, determining at least one evaluation indicator corresponding to the type parameters and / or focus parameters of the medical search model; according to the determined evaluation indicators, analyzing the relationship between the evaluation indicators and user behavior parameters, selecting data samples of the type corresponding to the user behavior parameters associated with the determined evaluation indicators, and inputting the selected data samples into the medical search model; according to the indicator judgment logic, analyzing the output results of the medical search model, thereby evaluating the performance of the medical search model.

[0005] According to one aspect of the present application, a medical search model performance evaluation device is provided, including: a data sample acquisition unit, used to acquire historical search data as an initial data sample set, and / or, construct an initial data sample set based on medical keywords and medical terms, and divide the initial data sample set according to medical search user behavior parameters to obtain multiple types of data samples; an evaluation index determination unit, used to determine the type parameters and / or focus parameters of the medical search model, and determine at least one evaluation index corresponding to the type parameters and / or focus parameters of the medical search model according to a predetermined index judgment logic; an evaluation execution unit, used to analyze the relationship between the evaluation index and the user behavior parameter based on the determined evaluation index, select data samples of the type corresponding to the user behavior parameter associated with the determined evaluation index, and input the selected data samples into the medical search model, and, according to the index judgment logic, analyze the output results of the medical search model, thereby performing performance evaluation on the medical search model.

[0006] According to one aspect of the present application, a computer device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the above-mentioned medical search model performance evaluation method.

[0007] According to one aspect of the present application, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned medical search model performance evaluation method when running.

[0008] By leveraging the above-described technical solution, this application provides a medical search model performance evaluation method, apparatus, device, and medium. The method first divides data samples into multiple types based on medical search user behavior parameters; then determines at least one evaluation metric based on predetermined metric judgment logic; and finally, selects data samples of the type corresponding to the user behavior parameters associated with the determined evaluation metric to evaluate the medical search model. Because data samples of various types are divided based on user behavior parameters, different user search characteristics corresponding to different types of samples can be reflected. Furthermore, for different evaluation metrics, data samples associated with the metric are used as evaluation samples. This improves the user behavior support of the samples, effectively improving the accuracy of metric evaluation and providing a reliable basis for optimizing and improving model performance.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A schematic diagram of an implementation scenario of a medical search model performance evaluation method provided in an embodiment of the present application is shown;

[0012] Figure 2 A flowchart of a medical search model performance evaluation method provided by the first embodiment of the present application is shown;

[0013] Figure 3 A schematic diagram of a medical search model performance evaluation method provided in the second embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram of the structure of a medical search model performance evaluation device provided in an embodiment of the present application is shown;

[0015] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown;

[0016] Figure 6 A schematic diagram of the structure of another computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0018] See also Figure 1, showing a schematic diagram of an application scenario of a medical search model performance evaluation method provided in an embodiment of the present application. In this scenario, the process of the model from start to finish is shown. In the first stage, the model training stage, a preliminary medical search model is constructed through steps such as data preparation and model method determination; in the second stage, the performance of the constructed initial medical search model is evaluated to determine whether it meets the performance indicators; in the third stage, the model is improved according to the performance evaluation results; in the fourth stage, the model is released or put online. The medical search model performance evaluation method provided in an embodiment of the present application is applied in the above-mentioned second stage, by performing performance evaluation on the initial or to-be-improved medical search model, in order to improve the model performance.

[0019] See also Figure 2 , shows a flow chart of a medical search model performance evaluation method provided in the first embodiment of the present application.

[0020] S201: Acquire historical search data as an initial data sample set, and / or construct an initial data sample set based on medical keywords and medical terms, and divide the initial data sample set according to medical search user behavior parameters to obtain multiple types of data samples.

[0021] In order to evaluate the performance of the medical search model, it is necessary to prepare data samples, input the data samples into the medical search model, analyze the model output results, and thus determine the model performance.

[0022] Data sample preparation can be done by acquiring historical search data from real-world scenarios, or by constructing or accumulating medical keywords and medical terminology to obtain an initial data sample set. Acquiring historical search data from real-world scenarios can involve pulling in online user search behavior data, such as actual medical search data on a daily, weekly, monthly, or annual basis, as the initial data sample set. This method of constructing or accumulating medical keywords and medical terminology is particularly suitable for more specialized and less common medical terms. Given the rigorous and specialized nature of medical search models, accumulating medical keywords and medical terminology can supplement and correct simplified or colloquial medical keywords, providing a data foundation for inquiries from professionals such as doctors.

[0023] After obtaining the initial sample set, the initial sample set is divided according to the medical search user behavior parameters to obtain multiple types of data samples.

[0024] Medical search user behavior parameters refer to parameters that reflect the medical search behavior of different users. This can be understood as determining medical search behavior parameters by classifying or statistically analyzing the behavioral habits of medical search users.

[0025] In one implementation, the initial data sample set is divided according to medical search user behavior parameters to obtain multiple types of data samples, including: determining medical search user behavior parameters based on evaluation requirements, and the medical search user behavior parameters reflect the search behavior of different users; from the initial primary sample set, matching data samples corresponding to different medical search user behavior parameters to obtain data samples of various types.

[0026] Evaluation requirements refer to the need to evaluate medical search models. In practice, these requirements may be determined based on the model type and / or the party proposing the evaluation requirements. For example, if the model to be evaluated is a classification algorithm model, the model's AUC (Area Under ROC Curve, the probability that a predicted positive example is ranked ahead of a negative example) metric will be used. If the evaluation requester is a product provider (the platform that launches the model), metrics such as the model's accuracy and / or coverage will be used. This shows that different evaluation requirements correspond to different evaluation metrics, and different evaluation metrics correspond to different types of data samples. Different types of data samples are determined based on the behavioral parameters of medical search users.

[0027] Therefore, in one implementation, determining medical search user behavior parameters based on evaluation requirements includes determining the evaluation requirements based on predetermined indicator judgment logic, wherein the evaluation requirements include at least one of: prioritizing search accuracy and rapidly improving optimization results, objectively measuring user support, ensuring medical search rigor, and measuring model robustness; and determining the corresponding medical search user behavior parameters for each evaluation requirement. For example, prioritizing search accuracy and rapidly improving optimization results, objectively measuring user support, ensuring medical search rigor, and measuring model robustness may correspond to high-frequency user behavior parameters, real user behavior parameters, rigorous user search behavior parameters, and user behavior diversity parameters, respectively. The indicator judgment logic includes the corresponding relationships between indicator types, sample selection tendencies, specific indicators, and evaluation criteria, which will be described in detail below.

[0028] Accordingly, from the initial primary sample set, data samples corresponding to different medical search user behavior parameters are matched to obtain data samples of various types, including high-frequency user behavior parameters, real user behavior parameters, rigorous user search behavior parameters, and user behavior diversity parameters from the initial primary sample set to obtain high-frequency data samples, real data samples, rigorous data samples, and diverse data samples.

[0029] See Table 1 for a table showing the classification of data samples.

[0030] Table 1

[0031]

[0032] Among them, high-frequency data samples, real data samples, rigorous data samples and diverse data samples are explained as follows respectively.

[0033] To prioritize quality and rapidly improve optimization results, frequent user behavior should be considered as much as possible for high-frequency data samples (A). Below, high-frequency data samples will be replaced by Sample A. Sample A can be obtained by pulling historical search data from real-world scenarios, analyzing user behavior, and extracting the medical search behavior data of high-frequency users as Sample A. Alternatively, high-frequency data samples can be obtained through methods such as knowledge accumulation.

[0034] To objectively measure user support, treat user behavior equally, and recreate real users as closely as possible, we must consider data timeliness and maintain online distribution. Below, we replace the real data sample with Sample B. Generally, Sample B is obtained by pulling down historical search data in real-world scenarios. To ensure the goal of treating users equally, Sample B is widely distributed, reflecting more objective and authentic data.

[0035] To improve the characteristics of medical rigor, rigorous data sample (C) was selected based on rigorous user search behavior. Below, rigorous data sample C is replaced with this one. Sample C can be obtained partially from real-world scenarios and partially through knowledge accumulation.

[0036] Diverse data samples (D) cover as much diversity in user behavior as possible to further measure model robustness. Below, diverse data samples are referred to as sample D. Sample D can be obtained partially from real-world scenarios and partially through knowledge accumulation. To reflect sample diversity, sample D generally focuses on data diversity rather than the breadth of data distribution.

[0037] S202: Determine the type parameters and / or focus parameters of the medical search model, and determine at least one evaluation indicator corresponding to the type parameters and / or focus parameters of the medical search model according to a predetermined indicator judgment logic.

[0038] Among them, the type parameter of the medical search model mainly refers to the type of algorithm adopted by the medical search model, such as a classification algorithm or a recommendation algorithm. Therefore, the type parameter of the medical search model can be determined according to the algorithm type of the medical search model; the focus parameter of the medical search model refers to the focus aspects of the performance evaluation of the medical search model, which is generally determined by the focus points proposed by the evaluation demand proposer. The evaluation demand proposer can be the product demander (the platform that launches the medical search model) or the product development team. For example, if the evaluation demander is the product demander, the focus points generally include indicators such as the accuracy and / or coverage of the search results. If the evaluation demander is the product development team, the focus points generally include indicators such as MRR (Mean Reciprocal Rank) and MAP (mean average precision).

[0039] In one implementation, the following steps are also included: determining the indicator judgment logic based on the model type and / or evaluation requirements, wherein the indicator judgment logic includes the correspondence between the indicator type, sample selection tendency, specific indicators and evaluation criteria; the indicator type includes at least one of the classification algorithm indicator, recommendation algorithm indicator, business indicator and statistical dimension indicator, wherein the statistical dimension indicator includes at least one of the user behavior dimension, result dimension and knowledge dimension.

[0040] Accordingly, in one implementation, according to a predetermined indicator judgment logic, at least one evaluation indicator corresponding to the type parameter and / or focus parameter of the medical search model is determined, including:

[0041] If the type parameter of the medical search model is a classification algorithm, the evaluation index is determined to be a classification algorithm index;

[0042] If the type parameter of the medical search model is a recommendation algorithm, the evaluation index is determined to be a recommendation algorithm index;

[0043] If the focus parameters of the medical search model are business-side parameters, the evaluation indicators are determined to be business indicators;

[0044] If the focus parameter of the medical search model is a statistical dimension parameter, the evaluation index is determined to be a statistical dimension index.

[0045] S203: Analyze the relationship between the evaluation index and the user behavior parameter according to the determined evaluation index, select data samples of the type corresponding to the user behavior parameter associated with the determined evaluation index, and input the selected data samples into the medical search model.

[0046] S204: Analyze the output results of the medical search model according to the indicator judgment logic, and thus perform performance evaluation on the medical search model.

[0047] In one implementation, data samples of the type corresponding to user behavior parameters that are closely related to the determined evaluation indicators are selected, including: for classification algorithm indicators, high-frequency data samples are selected; for recommendation algorithm indicators, real data samples are selected; for business indicators, real data samples are selected; for statistical dimension indicators, high-frequency data samples, real data samples, rigorous data samples and / or diverse data samples are selected.

[0048] See Table 2, which shows the indicator calculation logic in a medical search model performance evaluation method provided in the first embodiment of the present application.

[0049] Table 2

[0050]

[0051] Based on the model launch situation, the characteristics of the medical search business, and the requirements of the production and research project members for model measurement, we summarize and analyze the following aspects of model performance evaluation:

[0052] 1. Based on the launch characteristics, the basic requirements for launching a cold start model or a non-competitive model, analyze the basic support requirements of the model for the medical search business.

[0053] 2. Based on multi-model evaluation, which model provides better support for medical search services?

[0054] 3. Based on the product's operational effectiveness, the model's support for the medical search business, such as search coverage, content relevance, and support for rigorous search behavior.

[0055] 4. Analyze the areas where model support issues exist and identify areas for optimization.

[0056] Based on the above concerns, four categories of indicators are abstracted. The specific evaluation and judgment logic is shown in Figure 4 The following are examples of various types of indicators.

[0057] Category 1: Classification algorithm indicators

[0058] The purpose of classification algorithm indicators is to use the model smoke indicator (smoke index) as a marker for continued testing and acceptance. The sample selection preference is: Sample A. The recommended comparison requirements are: better than the historical version comparison, and the cold start benchmark requirement is >0.5. The specific indicator is AUC (Area Under Curve), which is an evaluation metric for measuring the quality of binary classification models and indicates the probability that the predicted positive example will be ranked before the negative example. Evaluation criteria: Correct results: Labeled as good or excellent.

[0059] Category 2: Recommendation algorithm indicators

[0060] The purpose of recommendation algorithm indicators is to objectively evaluate the theoretical support of the model. The sample selection tendency is: Sample B. The comparison requirement recommendation is: better than the historical version comparison. Specific indicators include: MRR (Mean Reciprocal Rank), MAP (mean average precision), NDCG (Normalized Discounted Cumulative Gain). The evaluation criteria are: a single result score of 7-9 is excellent, 4-6 is good, and 1-3 is poor.

[0061] Category 3: Business indicators

[0062] The purpose of business indicators is to provide indicators to business parties, so that the indicator results are highly consistent with the business and more intuitive. The sample selection tendency is: Sample B. The recommended comparison requirement is: due to historical version comparison. Specific indicators include: accuracy rate, coverage rate, TOP accuracy rate. The evaluation criteria are: results marked as good or excellent are correct.

[0063] Category 4: Statistical latitude indicators (user behavior latitude, result latitude, knowledge latitude)

[0064] For the user behavior dimension, its purpose is to facilitate the development of optimization directions for gradually analyzing behavior and recommending results from the user behavior path. The sample selection tendency is: samples A, B, C, D; the comparison requirement recommendation is: samples A and B are better than historical version comparisons, and for the comparison of different samples of the same version, the support priority is A>B>C>D; specific indicators include: MRR, MAP; evaluation criteria are: user behavior dimension standard: the number of correct results>=50% is considered correct.

[0065] For the result dimension, its purpose is to facilitate the development of optimization directions for gradually analyzing knowledge and behavior from the knowledge recommendation path to recommendation results; its sample selection tends to be: samples A, B, C, D; the comparison requirement recommendation is: samples A and B are better than historical version comparisons, and for the comparison of different samples of the same version, the support priority is A>B>C>D; specific indicators include: NDCG; the evaluation criteria are: marked as good or excellent for correct results, marked as poor for incorrect results.

[0066] For the knowledge dimension, its purpose is to facilitate the development of optimization directions for knowledge learning by gradually analyzing the knowledge path; its sample selection tendency is: samples A, B, C, and D; the comparison requirement recommendation is: better than the historical version comparison; specific indicators include: knowledge response rate, high-frequency knowledge accuracy rate; the evaluation criteria are: marked as good or excellent for the correct result.

[0067] In one implementation, the evaluation indicator(s) for evaluating the model may be determined based on the evaluation indicator priority or the party proposing the evaluation requirement.

[0068] For example, by setting the priority of evaluation indicator types, the medical search model is evaluated according to each type of evaluation indicator in descending order of priority. Assuming that the priorities of classification algorithm indicators, recommendation algorithm indicators, business indicators, and statistical dimension indicators are set from high to low, the model is evaluated according to classification algorithm indicators, recommendation algorithm indicators, business indicators, and statistical dimension indicators in descending order.

[0069] For example, based on the evaluation requirements, at least one evaluation metric of interest to the requesting party can be selected to evaluate the medical search model. For example, if the business side is evaluating the model, the model can be evaluated solely based on business metrics; if the development and optimization R&D perspective is concerned, the model can be evaluated based on statistical dimension metrics, and so on.

[0070] In one implementation, the specific process of S204 may include: obtaining the output result of the medical search model for each data sample, judging the level of the output result based on the specific indicators and evaluation criteria corresponding to the indicator type of the evaluation indicator in the indicator judgment logic; and counting and analyzing the levels of the output results corresponding to all data samples, thereby performing performance evaluation on the medical search model.

[0071] As can be seen, the medical search model performance evaluation method provided in the first embodiment of this application first obtains multiple types of data samples based on the medical search user behavior parameters; then determines at least one evaluation indicator based on a predetermined indicator judgment logic; and finally, selects data samples of the type corresponding to the user behavior parameters that are closely related to the determined evaluation indicator to evaluate the medical search model. Because the various types of data samples are divided according to the user behavior parameters, it can be shown that different types of samples correspond to different user search characteristics. Moreover, for different evaluation indicators, data samples that are correlated with the evaluation indicators are used as evaluation samples. Therefore, the user behavior support of the samples can be improved, the accuracy of the indicator evaluation can be effectively improved, and a reliable basis for optimizing and improving model performance can be provided.

[0072] The second embodiment of this application is introduced below.

[0073] See also Figure 3 , showing a schematic diagram of a medical search model performance evaluation method provided in the second embodiment of the present application.

[0074] First, data samples are prepared and labeled.

[0075] In one implementation, a specific method for preparing a data sample may include the following steps:

[0076] 1. First, check whether the evaluation indicators of data sample A meet the benchmark requirements. Once the requirements are met, prepare samples B, C, and D.

[0077] 2. Evaluate the amount of data that can be reverse-annotated based on manpower and release schedules;

[0078] 3. Pull the number of online user search behaviors per day -> per month -> per year;

[0079] 4. Based on the data volume and number of user behaviors, comprehensively determine the daily / monthly / yearly ratio;

[0080] 5. Determine the logic for the time window for selecting data and the proportion of random data selection;

[0081] 6. Based on manpower, release schedule, and other conditions, decide whether to prepare samples B, C, and D in order of priority from front to back.

[0082] In practical applications, recommendations based on a relatively large knowledge base can be made using pre-labeling or back-labeling methods. The model is required to provide at least twice the number of recommendation results provided to the test as the business results, so that there are more candidate pre-labeling results and the labeling results can cover more knowledge bases.

[0083] For example, the annotation content includes the following fields:

[0084] Annotation field 1: whether there should be a result;

[0085] Mark field 2: whether the returned result is correct;

[0086] Annotation field 3: Add annotation results based on business characteristics, such as the score returned for each result (excellent, good, or poor).

[0087] See Table 3, which shows a schematic diagram of data annotation in a medical search model performance evaluation method provided in the second embodiment of the present application.

[0088] Table 3

[0089]

[0090] In this example, the search keyword is "abc", and the returned results are: "abcdef Compound ABC Syrup (Sugar-free) 20ml*48 sticks", "abcdef", "abc brand cigarettes", and the corresponding positions are: TOP1, TOP2, TOP3, and the relevance is: excellent, good, and poor.

[0091] Then, according to the indicator judgment logic, the sample data corresponding to the determined evaluation indicators are input into the medical search model to obtain the results.

[0092] Finally, determine whether the model needs optimization. If so, further determine the optimization direction. If not, the evaluation is qualified, and model acceptance or model release can be initiated.

[0093] The medical search model performance evaluation method provided in the second embodiment of this application measures model performance from multiple levels based on business and algorithm indicators. The indicators of focus can guide the optimization direction of development, ensure the performance quality of the model online and the continuous and effective optimization of the model.

[0094] See also Figure 4 , shows a schematic diagram of the structure of a medical search model performance evaluation device provided in an embodiment of the present application. The medical search model performance evaluation device includes:

[0095] A data sample acquisition unit 401 is configured to acquire historical search data as an initial data sample set, and / or construct an initial data sample set based on medical keywords and medical terms, and divide the initial data sample set based on medical search user behavior parameters to obtain multiple types of data samples;

[0096] An evaluation index determination unit 402 is configured to determine the type parameter and / or focus parameter of the medical search model, and determine at least one evaluation index corresponding to the type parameter and / or focus parameter of the medical search model according to a predetermined index judgment logic;

[0097] The evaluation execution unit 403 is used to analyze the relationship between the evaluation indicators and the user behavior parameters based on the determined evaluation indicators, select data samples of the corresponding type of user behavior parameters associated with the determined evaluation indicators, and input the selected data samples into the medical search model, and analyze the output results of the medical search model according to the evaluation judgment logic, so as to perform performance evaluation on the medical search model.

[0098] In one implementation, the data sample acquisition unit 401 is specifically used to: determine the medical search user behavior parameters based on evaluation requirements, and the medical search user behavior parameters reflect the search behavior of different users; match the data samples corresponding to different medical search user behavior parameters from the initial primary sample set to obtain data samples of various types.

[0099] In one implementation, the data sample acquisition unit 401 is specifically used to: determine evaluation requirements based on a predetermined indicator judgment logic, the evaluation requirements include: prioritizing search accuracy and rapidly improving optimization effects, objectively measuring user support, ensuring medical search rigor, and measuring model robustness. At least one of determining medical search user behavior parameters includes: determining that prioritizing search accuracy and rapidly improving optimization effects, objectively measuring user support, ensuring medical search rigor, and measuring model robustness correspond to high-frequency user behavior parameters, real user behavior parameters, rigorous user search behavior parameters, and user behavior diversity parameters, respectively.

[0100] In one implementation, the data sample acquisition unit 401 is specifically used to: match data samples corresponding to high-frequency user behavior parameters, real user behavior parameters, rigorous user search behavior parameters, and user behavior diversity parameters from the initial primary sample set, to obtain high-frequency data samples, real data samples, rigorous data samples, and diverse data samples.

[0101] In one implementation, the method further includes:

[0102] The indicator logic determination unit 404 is used to analyze the model type and / or the party proposing the evaluation requirements and determine the indicator judgment logic, wherein the indicator judgment logic includes the correspondence between the indicator type, sample selection tendency, specific indicators and evaluation criteria, and the indicator type includes at least one of the classification algorithm indicator, recommendation algorithm indicator, business indicator and statistical dimension indicator, wherein the statistical dimension indicator includes at least one of the user behavior dimension, result dimension and knowledge dimension.

[0103] In one implementation, the evaluation execution unit 403 is specifically used to: obtain the output result of the medical search model for each data sample, and judge the level of the output result based on the specific indicators and evaluation criteria corresponding to the indicator type of the evaluation indicator in the indicator judgment logic; count and analyze the levels of the output results corresponding to all data samples, so as to perform performance evaluation on the medical search model.

[0104] In one implementation, the evaluation index determination unit 402 is specifically used to: if the type parameter of the medical search model is a classification algorithm, then determine the evaluation index as a classification algorithm index; if the type parameter of the medical search model is a recommendation algorithm, then determine the evaluation index as a recommendation algorithm index; if the focus parameter of the medical search model is a business side parameter, then determine the evaluation index as a business indicator; if the focus parameter of the medical search model is a statistical dimension parameter, then determine the evaluation index as a statistical dimension indicator.

[0105] In one implementation, the evaluation execution unit 403 is specifically used to select high-frequency data samples for classification algorithm indicators; select real data samples for recommendation algorithm indicators; select real data samples for business indicators; and select high-frequency data samples, real data samples, rigorous data samples and / or diverse data samples for statistical dimension indicators.

[0106] In one implementation, it also includes: a sample annotation unit 405; the sample annotation unit 405 is used to annotate the multiple types of data samples; including: determining that the first annotation field is whether there should be a result, the second annotation field is whether the returned result is correct, and the third annotation field is a grade score of the returned result; based on at least one of the first annotation field, the second annotation field, and the third annotation field, the data sample is pre-annotated and / or back-annotated.

[0107] In one implementation, the evaluation execution unit 403 is also used to: set the priority of the evaluation indicator type, and evaluate the medical search model according to each type of evaluation indicator in turn according to the priority of the evaluation indicator type from high to low; or, according to the party proposing the evaluation requirement, select at least one evaluation indicator that the party proposing the evaluation requirement is concerned about, and evaluate the medical search model.

[0108] For the specific definition of the medical search model performance evaluation device, please refer to the definition of the medical search model performance evaluation method above, which will not be repeated here. The various modules in the above-mentioned medical search model performance evaluation device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a medical search model performance evaluation method.

[0110] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a medical search model performance evaluation method.

[0111] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0112] Acquire historical search data as an initial data sample set, and / or construct an initial data sample set based on medical keywords and medical terminology precipitation, and divide the initial data sample set according to medical search user behavior parameters to obtain multiple types of data samples;

[0113] Analyze the type parameters and / or focus parameters of the medical search model, and determine at least one evaluation indicator corresponding to the type parameters and / or focus parameters of the medical search model according to a predetermined indicator judgment logic;

[0114] According to the determined evaluation indicators, the relationship between the evaluation indicators and user behavior parameters is analyzed, and data samples of the type corresponding to the user behavior parameters that are closely related to the determined evaluation indicators are selected. The selected data samples are input into the medical search model, and the results output by the target medical search model are analyzed to obtain performance evaluation results.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Acquire historical search data as an initial data sample set, and / or construct an initial data sample set based on medical keywords and medical terminology precipitation, and divide the initial data sample set according to medical search user behavior parameters to obtain multiple types of data samples;

[0117] Analyze the type parameters and / or focus parameters of the medical search model, and determine at least one evaluation indicator corresponding to the type parameters and / or focus parameters of the medical search model according to a predetermined indicator judgment logic;

[0118] According to the determined evaluation indicators, the relationship between the evaluation indicators and user behavior parameters is analyzed, and data samples of the type corresponding to the user behavior parameters that are closely related to the determined evaluation indicators are selected. The selected data samples are input into the medical search model, and the results output by the target medical search model are analyzed to obtain performance evaluation results.

[0119] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0120] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0121] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0122] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for evaluating the performance of a medical search model, characterized in that: include: Acquiring historical search data as an initial data sample set, and / or constructing an initial data sample set based on medical keywords and medical terminology, determining medical search user behavior parameters based on evaluation requirements, wherein the medical search user behavior parameters reflect the search behaviors of different users, and matching data samples corresponding to different medical search user behavior parameters from the initial data sample set to obtain data samples of various types; Determining the type parameters and focus parameters of the medical search model, and determining at least one evaluation indicator corresponding to the type parameters and focus parameters of the medical search model according to a predetermined indicator judgment logic; According to the determined evaluation indicators, the relationship between the evaluation indicators and the user behavior parameters is analyzed, and data samples of the type corresponding to the user behavior parameters associated with the determined evaluation indicators are selected, wherein the data samples of the type corresponding to the user behavior parameters closely related to the determined evaluation indicators are selected, including: for classification algorithm indicators, selecting high-frequency data samples corresponding to high-frequency user behavior parameters; for recommendation algorithm indicators or business indicators, selecting real data samples corresponding to real user behavior parameters; for statistical dimension indicators, selecting corresponding high-frequency data samples, real data samples, rigorous data samples and / or diverse data samples, and then inputting the selected data samples into the medical search model; According to the indicator judgment logic, the output results of the medical search model are analyzed to perform performance evaluation on the medical search model.

2. The method according to claim 1, characterized in that Determining medical search user behavior parameters based on evaluation requirements includes: Determine evaluation requirements based on pre-determined indicator judgment logic, including at least one of: prioritizing search accuracy and rapidly improving optimization results, objectively measuring user support, ensuring medical search rigor, and measuring model robustness; Determine the various evaluation requirements corresponding to various medical search user behavior parameters. Among them, the medical search user behavior parameters that prioritize ensuring search accuracy and quickly improving optimization effects, objectively measuring user support, ensuring medical search rigor, and measuring model robustness are determined to be high-frequency user behavior parameters, real user behavior parameters, rigorous user search behavior parameters, and user behavior diversity behavior parameters.

3. The method according to claim 2, characterized in that The data samples corresponding to different medical search user behavior parameters are matched from the initial data sample set to obtain data samples of various types, including: From the initial data sample set, data samples corresponding to high-frequency user behavior parameters, real user behavior parameters, rigorous user search behavior parameters, and user behavior diversity parameters are matched to obtain high-frequency data samples, real data samples, rigorous data samples, and diverse data samples.

4. The method according to claim 1, wherein Also includes: Analyze the model type and / or evaluation requirements to determine the indicator judgment logic, wherein the indicator judgment logic includes the correspondence between indicator type, sample selection tendency, specific indicators and evaluation criteria, and the indicator type includes at least one of classification algorithm indicators, recommendation algorithm indicators, business indicators and statistical dimension indicators, wherein the statistical dimension indicators include at least one of user behavior dimension, result dimension and knowledge dimension.

5. The method according to claim 4, characterized in that Analyzing the output results of the medical search model according to the indicator judgment logic, thereby performing a performance evaluation on the medical search model, includes: For each data sample, obtaining an output result of the medical search model, and judging the level of the output result according to the specific indicator and evaluation criteria corresponding to the indicator type of the evaluation indicator in the indicator judgment logic; The levels of the output results corresponding to all data samples are counted and analyzed to evaluate the performance of the medical search model.

6. A medical search model performance evaluation device, characterized in that: include: a data sample acquisition unit configured to acquire historical search data as an initial data sample set, and / or construct an initial data sample set based on medical keywords and medical terminology, determine medical search user behavior parameters based on evaluation requirements, wherein the medical search user behavior parameters reflect the search behavior of different users, and match data samples corresponding to different medical search user behavior parameters from the initial data sample set to obtain data samples of various types; An evaluation index determination unit, configured to determine a type parameter and a focus parameter of the medical search model, and determine at least one evaluation index corresponding to the type parameter and the focus parameter of the medical search model according to a predetermined index judgment logic; An evaluation execution unit is used to analyze the relationship between the evaluation indicators and user behavior parameters based on the determined evaluation indicators, and select data samples of the type corresponding to the user behavior parameters associated with the determined evaluation indicators, wherein the data samples of the type corresponding to the user behavior parameters that are closely related to the determined evaluation indicators are selected, including: for classification algorithm indicators, selecting high-frequency data samples corresponding to high-frequency user behavior parameters; for recommendation algorithm indicators or business indicators, selecting real data samples corresponding to real user behavior parameters; for statistical dimension indicators, selecting corresponding high-frequency data samples, real data samples, rigorous data samples and / or diverse data samples, and then inputting the selected data samples into the medical search model, analyzing the output results of the medical search model, and thus performing performance evaluation on the medical search model.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images