Model evaluation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model evaluation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a plurality of first insurance policies; predicting each first insurance policy through a pre-trained target prediction model to obtain a complaint probability value of each first insurance policy; screening the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain a plurality of second insurance policies; grouping the plurality of second insurance policies to obtain an experimental group and a control group; screening the insurance policies in the experimental group to obtain a first target insurance policy, and screening the insurance policies in the control group to obtain a second target insurance policy; actual complaint results of the first target insurance policy and the second target insurance policy in the first time period are obtained respectively, and a first complaint rate and a second complaint rate are obtained; and determining an evaluation result of the target prediction model according to the first complaint rate and the second complaint rate. According to the embodiment of the invention, the accuracy of model effect evaluation can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model evaluation method, device, electronic device and storage medium. Background Art

[0002] In order to evaluate the actual effect of a model or solution, a comparative experiment between an experimental group and a control group (AB test) is usually used. However, due to the differences in objective conditions in different application scenarios, such as geography, economic level, company policy, and personnel personality, traditional AB tests often cannot guarantee the accuracy of the results, and thus cannot effectively evaluate the model effect.

[0003] Therefore, how to improve the accuracy of model evaluation has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a model evaluation method, device, electronic device and storage medium, aiming to improve the accuracy of model effect evaluation.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model evaluation method, the method comprising:

[0006] Get multiple first policies;

[0007] Predict each of the first insurance policies using the pre-trained target prediction model to obtain a complaint probability value for each first insurance policy;

[0008] Screening the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain a plurality of second insurance policies;

[0009] Grouping the plurality of second insurance policies to obtain an experimental group and a control group;

[0010] Screening the insurance policies in the experimental group to obtain first target insurance policies, and screening the insurance policies in the control group to obtain second target insurance policies, wherein the first target insurance policy and the second target insurance policy each include the same number of insurance policies;

[0011] Respectively obtain actual complaint results of the first target insurance policy and the second target insurance policy in the first time period to obtain a first complaint rate and a second complaint rate;

[0012] An evaluation result of the target prediction model is determined according to the first complaint rate and the second complaint rate.

[0013] In some embodiments, before predicting the insurance policy using the pre-trained target prediction model to obtain the complaint probability value of each insurance policy, the method further includes:

[0014] Acquire multiple training samples, the training samples including sample insurance policies and labels for identifying whether the sample insurance policies have been complained about in the second time period, the sample insurance policies including multiple attributes of the insurance policies and attribute values ​​corresponding to each attribute;

[0015] Using the multiple training samples to train at least two basic models to obtain a prediction model corresponding to each of the basic models;

[0016] The prediction models corresponding to at least two of the basic models are screened to obtain the target prediction model.

[0017] In some embodiments, the step of training at least two basic models using the multiple training samples to obtain a prediction model corresponding to each of the basic models includes:

[0018] Dividing the plurality of training samples into a training set and a validation set;

[0019] Using the training samples in the training set to train at least two basic models to obtain a prediction model corresponding to each of the basic models;

[0020] The step of screening the prediction models corresponding to at least two of the basic models to obtain the target prediction model includes:

[0021] The prediction models corresponding to at least two of the basic models are screened using the training samples in the validation set to obtain the target prediction model.

[0022] In some embodiments, the step of using the training samples in the validation set to screen prediction models corresponding to at least two of the base models to obtain the target prediction model includes:

[0023] Traversing each prediction model, inputting the training samples in the verification set into the prediction model, and obtaining a first complaint probability value for each training sample in the verification set;

[0024] Arrange the multiple first complaint probability values ​​in descending order, and divide the training samples corresponding to the first complaint probability values ​​according to the first rule to obtain multiple first quantile intervals, each of which includes at least one training sample in the validation set;

[0025] Determine the first actual complaint rate for each first quantile interval;

[0026] Inputting the training samples in the training set into each of the prediction models to obtain a second complaint probability value for each training sample in the training set;

[0027] Arrange the multiple second complaint probability values ​​in descending order, and divide the training samples corresponding to the second complaint probability values ​​according to the first rule to obtain multiple second quantile intervals, each of which includes at least one training sample in the training set, and the multiple first quantile intervals are the same as the multiple second quantile intervals;

[0028] Determine the second actual complaint rate for each second quantile interval;

[0029] According to the first actual complaint rate and the second actual complaint rate corresponding to the same first quantile interval and the second quantile interval respectively, the stability index of the corresponding quantile interval is obtained;

[0030] The stability index corresponding to each quantile interval is averaged to obtain the target stability index;

[0031] The prediction model whose target stability index is less than the first preset threshold is determined as the target prediction model.

[0032] In some embodiments, determining the prediction model whose target stability index is less than a first preset threshold as the target prediction model includes:

[0033] Determine the prediction model whose target stability index is less than the first preset threshold as an alternative prediction model;

[0034] Arrange the multiple second complaint probability values ​​in descending order, and divide the training samples corresponding to the second complaint probability values ​​according to the second rule to obtain multiple third quantile intervals, each of the third quantile intervals including at least one training sample in the training set;

[0035] Determine a target evaluation parameter for each third quantile interval, wherein the target evaluation parameter includes at least one of a third actual complaint rate, a recall rate, and a lift, wherein the third actual complaint rate is the actual complaint amount of the training samples in the third quantile interval divided by the total number of training samples in the third quantile interval, the recall rate is the actual complaint amount of the training samples in the third quantile interval divided by the total number of complaints of the training samples in the training set, and the lift is the third actual complaint rate of the third quantile interval divided by the total number of complaints;

[0036] According to the target evaluation parameters, the alternative prediction models are evaluated to determine the target prediction model.

[0037] In some embodiments, the step of screening the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain the plurality of second insurance policies includes:

[0038] Screening the complaint probability value of each insurance policy according to a second preset threshold value to obtain the plurality of second insurance policies;

[0039] The plurality of second insurance policies are grouped to obtain an experimental group and a control group, including:

[0040] The multiple second insurance policies are grouped according to the parity of the last digits of the policy numbers recorded in the insurance policies to obtain the experimental group and the control group.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a model evaluation device, the device comprising: an acquisition module, configured to acquire a plurality of first insurance policies;

[0042] A prediction module, used to predict each of the first insurance policies using a pre-trained target prediction model to obtain a complaint probability value for each first insurance policy;

[0043] A first screening module, configured to screen the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain a plurality of second insurance policies;

[0044] A grouping module, used for grouping the plurality of second insurance policies to obtain an experimental group and a control group;

[0045] A second screening module is used to screen the insurance policies in the experimental group to obtain a first target insurance policy, and to screen the insurance policies in the control group to obtain a second target insurance policy, wherein the first target insurance policy and the second target insurance policy each include the same number of insurance policies;

[0046] A calculation module, used to respectively obtain actual complaint results of the first target insurance policy and the second target insurance policy within a first time period to obtain a first complaint rate and a second complaint rate;

[0047] An evaluation module is used to determine an evaluation result of the target prediction model according to the first complaint rate and the second complaint rate.

[0048] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0049] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0050] The model evaluation method, device, electronic device and storage medium proposed in the present application predict the complaint rate of insurance policies through a training model, and screen out insurance policies with higher complaint rates based on the complaint rate; divide the insurance policies into two groups based on the parity of the last digits of the customer policy numbers in the screened insurance policies, to obtain an experimental group and a control group; screen the insurance policies in the experimental group through the training model to obtain the high-complaint insurance policies in the experimental group, screen the insurance policies in the control group through the original rules to obtain the high-complaint insurance policies in the control group, and finally compare the actual complaint rates of the high-complaint insurance policies in the experimental group and the high-complaint insurance policies in the control group over a period of time, and determine the evaluation results of the training model based on the actual complaint rates; through the model prediction and grouping methods, the defects of the existing AB testing method are effectively solved, and the accuracy and reliability of the model effect evaluation are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the model evaluation method provided in the embodiment of the present application;

[0052] Figure 2 yes Figure 1 A schematic diagram of the process of step S200 in FIG.

[0053] Figure 3 yes Figure 1 A schematic diagram of the process of steps S300 and S400 in FIG.

[0054] Figure 4 yes Figure 1 A schematic diagram of the process of step S500 in FIG.

[0055] Figure 5 is a schematic diagram of the structure of a model evaluation device provided in an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] AB Test: AB test is a commonly used experimental design method to compare the effects of two or more different treatments or strategies to determine which treatment or strategy performs better on a certain indicator. In AB testing, a portion of users or samples will be randomly selected and divided into two or more groups, each group will be applied with a different treatment or strategy, and then their results will be compared.

[0063] Supervised Learning: A supervised model is a machine learning method that uses labeled training data to train a function so that it can predict the output variable Y from the input variable X. In this process, each input sample is associated with a corresponding output label, and the machine learns the mapping relationship between input and output through these associated samples and labels. Supervised learning models use labeled training data annotated by experts, input labeled data, and the model predicts the output of new data by learning these labeled data. The goal of supervised learning is to enable the model to learn the mapping relationship from known input-output pairs, so that it can make accurate predictions on unseen data.

[0064] In insurance risk control scenarios, it is usually necessary to identify risky policies through models. First, identify risky policies through models, and then have dedicated personnel (customer specialists, sales management personnel, compliance personnel, etc.) screen and confirm high-risk policies for processing, thereby reducing the risk incidence of these policies. This method often requires verification of the difference between the effects of risky policies identified by the model and those not identified by the model, and the comparison method usually uses AB Test. However, due to objective differences in different application scenarios, such as factors such as region, economic level, company policy, and personnel personality, traditional AB Tests often find it difficult to guarantee the accuracy of the results, and thus cannot effectively evaluate the model effect.

[0065] Based on this, the embodiments of the present application provide a model evaluation method, device, electronic device and storage medium, aiming to improve the accuracy of model effect evaluation.

[0066] The model evaluation method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the recommended method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The model evaluation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The model evaluation method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms and other basic cloud computing services. Cloud servers; software can be applications that implement model evaluation methods, etc., but are not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] Figure 1 is an optional flowchart of the model evaluation method provided in the embodiment of the present application. Figure 1The method may include but is not limited to steps S100 to S700.

[0073] Step S100, obtaining multiple first insurance policies.

[0074] The multiple first insurance policies may be all the insurance policies in the database, or they may be insurance policies within a certain range selected according to specific business needs. Specifically, the first insurance policies may be insurance policies within a preset time range, or they may be insurance policies within a certain institution or a certain region, or they may be insurance policies of one or more preset types. In this embodiment, the multiple first insurance policies may be all insurance policies for a specified time period divided by institution. Using institutions as units to divide the range can avoid the impact of objective conditions such as differences in economic levels in different regions and differences in policies of different companies on the results of the AB test.

[0075] Step S200, predict each of the first insurance policies using the pre-trained target prediction model to obtain a complaint probability value for each first insurance policy.

[0076] The trained target prediction model is used to predict the complaint probability values ​​of all first insurance policies. The complaint probability values ​​are determined based on the scores of several features in the first insurance policies by the target prediction model. Assigning a risk score or complaint probability value to each customer based on the target prediction model can facilitate risk sorting and hierarchical management of insurance policies, and provide a basis for further data analysis and decision-making.

[0077] Step S300, screening the multiple first insurance policies according to the complaint probability value of each first insurance policy to obtain multiple second insurance policies.

[0078] After predicting the complaint probability value of each first policy according to the trained target prediction model, the policies can be screened according to these complaint probability values ​​to identify policies with higher complaint risks. The multiple second policies are policies with higher complaint probability values ​​screened out according to a preset threshold. The preset threshold can be set according to the target value corresponding to the business needs, or it can be set according to the number of complaint policies required for the experiment. Exemplarily, the preset threshold can be set to 1% of the complaint probability value, or it can be set to the first 100 policies sorted in descending order according to the complaint probability value. By screening out multiple second policies according to the preset threshold, high-risk policies in the first policies can be screened out, and then AB testing can be performed on the second policies with high risks, which can reduce the interference of irrelevant factors in the low-risk policies on the test, while reducing the number of test samples, improving resource utilization, and shortening the test cycle, improving AB testing efficiency.

[0079] Step S400, grouping the multiple second insurance policies to obtain an experimental group and a control group.

[0080] The selected multiple second insurance policies are divided into an experimental group and a control group, ensuring that the two groups are as similar as possible in terms of other features except whether the model is used. In this embodiment, with the help of the law of large numbers, the groups are grouped according to the parity of the last digit of the customer's insurance policy number, so that the experimental group and the control group are basically close in data volume, and there are no other factors that affect the grouping results, and they remain similar overall, thereby making the subsequent AB test results more reliable.

[0081] Step S500, screening the policies in the experimental group to obtain a first target policy, and screening the policies in the control group to obtain a second target policy, wherein the first target policy and the second target policy each include the same number of policies.

[0082] Different methods are used to screen the policies in the experimental group and the control group, and the target number of policies screened by the two methods is the same. The same target number of policies helps ensure that the comparison of the results between the two groups is statistically valid. If the sample sizes differ greatly, the results of the statistical test may be affected by the sample size, leading to misleading conclusions. In addition, by keeping the number the same, selection bias and sampling bias can be reduced, making the experimental results more reliable.

[0083] Step S600, respectively obtain the actual complaint results of the first target insurance policy and the second target insurance policy within the first time period to obtain a first complaint rate and a second complaint rate.

[0084] Determine the start and end time of the first time period based on business needs and data conditions. Count the actual number of complaints about the first target policy and the second target policy in the first time period, and calculate the first complaint rate and the second complaint rate. The first complaint rate is the ratio of the number of target policies complained about in the first time period in the experimental group to the total number of target policies in the experimental group; the second complaint rate is the ratio of the number of target policies complained about in the first time period in the control group to the total number of target policies in the control group.

[0085] By obtaining the actual complaint results of the experimental group and the control group in the first time period and calculating the first complaint rate and the second complaint rate, data support can be provided for subsequent model evaluation and business decision-making.

[0086] Step S700: determining an evaluation result of the target prediction model according to the first complaint rate and the second complaint rate.

[0087] Compare the first complaint rate and the second complaint rate, and analyze the difference in complaint rate between the experimental group and the control group. If the complaint rate of the experimental group is significantly higher than that of the control group, or the complaint rate of the experimental group is higher than the preset multiple of the complaint rate of the control group, it means that the model can effectively identify high-risk policies and the model has a high prediction accuracy; if the complaint rate of the experimental group is close to that of the control group or the difference is not significant, it means that the model is not ideal in predicting high-risk policies, and it is necessary to further improve the model or adjust the threshold. Specifically, the multiple of the effect of the experimental group compared to the control group is determined according to the project requirements, such as 3 times or 5 times.

[0088] Evaluate the model effect based on the difference in complaint rates and business goals. If the model is effective, it can be applied to actual business; if the model effect is not ideal, the model needs to be optimized, such as optimizing model parameters, adjusting screening criteria, etc.

[0089] By obtaining the actual complaint results of the experimental group and the control group in the first time period and calculating the first complaint rate and the second complaint rate, the evaluation results of the target prediction model can be determined, and a decision can be made on whether to adopt or improve the model based on this. It can also provide strong data support for subsequent risk management and business decisions.

[0090] See also Figure 2 In some embodiments, before step S200, the following steps may also be included but not limited to steps S210 to S230:

[0091] Step S210, obtaining multiple training samples, the training samples include sample insurance policies and labels for identifying whether the sample insurance policies have been complained about within the second time period, the sample insurance policies include multiple attributes of the insurance policies and attribute values ​​corresponding to each attribute.

[0092] Obtain multiple training samples, the training samples include sample insurance policies and labels for identifying whether the sample insurance policies have been complained about within the second time period. Specifically, the sample insurance policies include multiple attributes of the insurance policies and attribute values ​​corresponding to each attribute, wherein the attributes of the insurance policies can be set according to dimensions such as customers, insurance policies, and agents, for example: static information such as customer age, gender, marital status, education, occupation, income, etc., customer's recent trace behavior data on the company's various apps (number of active days on the app in the past month, etc.); policy underwriting time, insurance type, premium, etc.; agent age, rank, number of complaints, quality deductions, etc. The second time period is a time period preset according to business needs. In one implementation of this embodiment, the second time period may be 30 days.

[0093] The label can be used to identify whether the sample policy has been complained about within the preset second time period. The complained policies and non-complained policies can be distinguished based on the label. Based on the corresponding attributes and attribute values ​​of the complained policies and non-complained policies, the model can be trained on the characteristics of the complained policies, thereby identifying policies with higher complaint risks.

[0094] In one implementation of this embodiment, the sample insurance policies may be insurance policies corresponding to multiple time points. For example, insurance policies at the time points of 20230601, 20230901, and 20231201 may be selected as training samples, and 20230601 represents all insurance policies with an underwriting time before June 1, 2023.

[0095] The multiple attributes of the sample insurance policy and the attribute value corresponding to each attribute are integrated with the label to form a training sample. In one implementation of this embodiment, the training sample can be presented in a wide table.

[0096] Step S220: Use the multiple training samples to train at least two basic models to obtain a prediction model corresponding to each of the basic models.

[0097] Select multiple basic models for training, where the basic models can be supervised models, such as lightgbm, xgboost, catboost, etc. Use multiple algorithms to train the models separately, and evaluate the effect of the models trained by each algorithm. From the multiple algorithms, select the model with the best effect as the final prediction model.

[0098] Specifically, selecting multiple algorithms for training can prevent the model from overfitting to a specific algorithm and improve the generalization ability of the model. Different algorithms are suitable for different data features and scenarios. By trying multiple algorithms, you can find the model that best suits the current problem.

[0099] Step S230, screening the prediction models corresponding to at least two of the basic models to obtain the target prediction model.

[0100] According to the training effect of the model, the prediction models corresponding to the various basic models are screened, and the prediction model with the best effect is obtained as the target model.

[0101] This embodiment trains multiple basic models and screens them, effectively trains multiple models based on labels, and selects the best model as the target prediction model, thereby improving the accuracy of the model and more accurately predicting the complaint probability of the insurance policy.

[0102] In some embodiments, step S220 may include, but is not limited to, steps S221 to S222:

[0103] Step S221, dividing the plurality of training samples into a training set and a validation set;

[0104] Step S222: Use the training samples in the training set to train at least two basic models to obtain a prediction model corresponding to each of the basic models.

[0105] In this embodiment, the training samples include a training set and a validation set, and the training set and the validation set are determined according to the multiple training samples obtained. In one implementation of this embodiment, in order to make the training data close to the subsequent application, it is possible to consider taking the latest partition 20231201 as the training set, and the other two 20230601 and 20230901 as the validation set, and then set the corresponding parameters for the selected algorithm for training.

[0106] In order to optimize the performance of the selected algorithm model, it is usually necessary to adjust the parameters of the model. In one implementation of this embodiment, the parameters can be adjusted according to past experience, or the GridSearchCV grid parameter adjustment method can be adopted. Among them, the GridSearchCV grid parameter adjustment refers to setting the key parameters in the form of a grid, arranging and combining them, and selecting the best performing parameter combination as the final parameter based on the model effect after training.

[0107] In this embodiment, multiple training samples are divided into a training set and a validation set, and the basic model is trained through the training set to obtain a training model; the effect of the model is then verified through the validation set, and the optimal training model is selected as the target training model based on the verification effect. Dividing the data into a training set and a validation set can prevent the model from overfitting the training set and improve the generalization ability of the model. If the model performs well on the training set but performs poorly on the validation set, it means that the model has learned the training data too well and cannot be generalized well to new data; at the same time, the validation set can also be used to evaluate the performance of different models, and select the best performing model as the target training model, thereby improving the accuracy and generalization ability of the model.

[0108] In some embodiments, step S230 may also include, but is not limited to, step S231:

[0109] Step S231, using the training samples in the validation set to screen the prediction models corresponding to at least two of the basic models to obtain the target prediction model.

[0110] In this embodiment, the prediction models corresponding to the multiple basic models are screened using the training samples in the validation set, and the optimal model is selected based on the performance of the multiple different basic models on the validation set. Specifically, the model effect can be evaluated based on indicators such as model stability, accuracy, F1 score, AUC, and recall rate.

[0111] In some embodiments, step S231 may include, but is not limited to, steps S2311 to S2319:

[0112] Step S2311, traverse each prediction model, input the training samples in the verification set into the prediction model, and obtain the first complaint probability value of each training sample in the verification set;

[0113] Step S2312, arranging the multiple first complaint probability values ​​in descending order, dividing the training samples corresponding to the first complaint probability values ​​according to the first rule, and obtaining multiple first quantile intervals, each of which includes at least one training sample in the validation set;

[0114] Step S2313, determining the first actual complaint rate of each first quantile interval;

[0115] Step S2314, inputting the training samples in the training set into each of the prediction models to obtain a second complaint probability value for each training sample in the training set;

[0116] Step S2315, arranging the multiple second complaint probability values ​​in descending order, dividing the training samples corresponding to the second complaint probability values ​​according to the first rule, and obtaining multiple second quantile intervals, each of which includes at least one training sample in the training set, and the multiple first quantile intervals are the same as the multiple second quantile intervals;

[0117] Step S2316, determining a second actual complaint rate for each second quantile interval;

[0118] Step S2317, obtaining a stability index of the corresponding quantile interval according to the first actual complaint rate and the second actual complaint rate corresponding to the same first quantile interval and the second quantile interval respectively;

[0119] Step S2318, averaging the stability index corresponding to each quantile interval to obtain a target stability index;

[0120] Step S2319: determine the prediction model whose target stability index is less than the first preset threshold as the target prediction model.

[0121] In this embodiment, the stability of the model can be verified by the Population Stability Index (PSI). PSI is an indicator for measuring the stability of a model, which determines the stability of the model by comparing the predicted probability distribution of the model on the training set and the validation set.

[0122] First, the training samples in the validation set are input into each prediction model to obtain the complaint probability prediction value of each sample, that is, the first complaint probability value; the first complaint probability values ​​of all samples are arranged in descending order, and they are divided into multiple intervals (first quantile intervals) according to the set rules (first rules); the number of samples that actually have complaints in each interval is counted, and the actual complaint rate of the interval (first actual complaint rate) is calculated.

[0123] Secondly, the training samples in the training set are input into each prediction model to obtain the complaint probability prediction value of each sample, that is, the second complaint probability value; the second complaint probability values ​​of all samples are arranged in descending order, and the second complaint probability values ​​are divided using the same rule as the first quantile interval (first rule) to obtain multiple second quantile intervals, and multiple first quantile intervals are the same as multiple second quantile intervals; the number of samples that actually suffered complaints in each interval is counted, and the actual complaint rate of the interval (the second actual complaint rate) is calculated.

[0124] Finally, the difference between the first actual complaint rate and the second actual complaint rate corresponding to each interval is calculated to obtain the stability index of the interval; the stability indexes of all intervals are averaged to obtain the target stability index of each prediction model; the target stability index of each prediction model is compared with the first preset threshold. If the target stability index is less than the first preset threshold, it means that the model stability is relatively good and the model can be used. The PSI index reflects the difference in the distribution of prediction results between the training set and the validation set. The smaller the difference, the better the model stability. By calculating the PSI index, the generalization ability of the model can be evaluated, and models with higher stability can be screened for application.

[0125] In one implementation of this embodiment, 20231201 is used as the training set, and 20230601 and 20230901 are combined as the validation set. The validation set is used to predict the model to obtain the first complaint probability value of each sample; the training set is used to predict the model to obtain the second complaint probability value of each sample. The first complaint probability value and the second complaint probability value are arranged in descending order, and the quantile intervals are divided according to the preset first rule. For example, the first rule can take every 10% of the samples as a segment. If the total number of samples in the training set is 1 million, the top 10% of the samples are the first 100,000 samples after the probability values ​​are arranged in descending order; the real complaint samples in these 100,000 are divided into 100,000 to obtain the accuracy of the top 10% of the training set. The other quantiles are calculated according to the same rules to obtain the PSI index table corresponding to each quantile interval, as shown in Table 1:

[0126]

[0127]

[0128] Table 1

[0129] The PSI value of each quantile interval is calculated based on the complaint accuracy of the training set and the validation set in Table 1, and the stability index of all intervals is averaged to obtain the PSI value of the model. The calculation formula of PSI is as follows:

[0130]

[0131] The theoretical criteria for PSI and model stability are shown in Table 2:

[0132]

[0133] Table 2

[0134] It can be seen that the smaller the PSI value, the better the model stability. Specifically, the first preset threshold can be set according to the theoretical standards of PSI and model stability. In this embodiment, the first preset threshold is set to 0.1, and a prediction model with good stability can be screened out. The PSI value of the model calculated according to the data in Table 1 is 0.00018, which is less than the first preset threshold of 0.1, indicating that the model stability is relatively good and the model can be used. The PSI index can be used to evaluate the stability of the model when the data distribution changes, and the target prediction model can be selected through the stability index, which helps to select a more reliable model and improve the prediction accuracy of the model.

[0135] In some embodiments, step S2319 may also include, but is not limited to, steps S23191 to S23194:

[0136] Step S23191, determining the prediction model whose target stability index is less than the first preset threshold as the candidate prediction model;

[0137] Step S23192, arranging the multiple second complaint probability values ​​in descending order, dividing the training samples corresponding to the second complaint probability values ​​according to the second rule, and obtaining multiple third quantile intervals, each of which includes at least one training sample in the training set;

[0138] Step S23193, determining the target evaluation parameters for each third quantile interval, wherein the target evaluation parameters include at least one of a third actual complaint rate, a recall rate, and a lift, wherein the third actual complaint rate is the actual complaint amount of the training samples in the third quantile interval divided by the total number of training samples in the third quantile interval, the recall rate is the actual complaint amount of the training samples in the third quantile interval divided by the total number of complaints of the training samples in the training set, and the lift is the third actual complaint rate of the third quantile interval divided by the total number of complaints;

[0139] Step S23194, evaluating the alternative prediction models according to the target evaluation parameters, and determining the target prediction model.

[0140] After evaluating the stability of the model, the effectiveness of the model can be further evaluated by evaluating the model's precision (the third actual complaint rate), recall rate, and lift. First, based on the condition that the target stability index is less than the first preset threshold, the alternative prediction model is screened out; then the second complaint probability values ​​are sorted in descending order, and the quantile intervals are divided according to the preset second rule; the target evaluation parameters of each quantile interval are calculated, where the target evaluation parameters include the third actual complaint rate, recall rate, and lift; finally, the alternative prediction models are evaluated according to the target evaluation parameters to determine the target prediction model.

[0141] Specifically, the target evaluation parameters include: precision (third actual complaint rate), recall rate and lift. Among them, the precision rate (third actual complaint rate) is calculated by dividing the cumulative number of complaints of training samples in the third quantile interval by the total number of training samples in the third quantile interval; the recall rate is calculated by dividing the actual number of complaints of training samples in the third quantile interval by the total number of complaints of training samples in the training set; the lift is calculated by dividing the third actual complaint rate in the third quantile interval by the total number of complaints of training samples in the training set.

[0142] If the accuracy (the third actual complaint rate), recall rate and lift of the prediction model all reach the expected goals preset according to business requirements, the prediction model meets the standards and can be applied; if multiple prediction models all meet the standards, the best one can be selected from the multiple prediction models.

[0143] In one implementation of this embodiment, the second rule can be set to divide the quantiles according to 0.1. If the total sample is 1 million, the 0.1 quantile corresponds to the first 0.1%, that is, the first 1000 policies, and the 0.2 quantile corresponds to the first 2000 policies. After the probability values ​​of the training set are arranged in reverse order, they are segmented by quantiles to obtain the target evaluation parameters corresponding to each quantile, as shown in Table 3:

[0144] Quantile Cumulative number of policies Cumulative number of complaints Cumulative complaint rate Cumulative recall rate Cumulative improvement 0.1 xx1 xx2 1.529％ 10％ 364.02 0.2 xx xx 0.743％ xx xx 0.3 xx xx 0.517％ xx xx ... ... ... ... ... ... 0.9 xx xx xx xx xx 1 xx xx xx xx xx ... ... ... ... ... ... 90 xx xx 0.005％ xx xx 100 xx 1091 0.004％ 100％ 100％

[0145] Table 3

[0146] For example, the top 0.1% precision, recall, and lift are calculated as follows:

[0147] Accuracy: xx2 / xx1=1.53%;

[0148] Recall rate: xx2 / 1091=10%;

[0149] Improvement degree: 1.53% / 0.004%≈364.

[0150] If the precision, recall and lift of the prediction model all reach the expected targets, for example, the precision of the top 0.1% is greater than 1.5%, the cumulative recall is greater than 10%, and the cumulative lift is greater than 300 times, then the prediction model is considered to have met expectations and can be applied in actual scenarios.

[0151] This embodiment can quantify the performance of the prediction model at different quantiles through target evaluation parameters, and judge whether the prediction model achieves the expected effect based on the evaluation results. According to the performance of the prediction model on the training set, the target prediction model that best suits the business needs is selected.

[0152] See also Figure 3 In some embodiments, step S300 may include but is not limited to step S310:

[0153] Step S310, screening the complaint probability value of each insurance policy according to the second preset threshold value to obtain the multiple second insurance policies.

[0154] In this embodiment, a second preset threshold may be set according to business requirements, and the first insurance policy may be screened according to the second preset threshold to obtain the second insurance policy.

[0155] In one implementation of this embodiment, the second preset threshold can be set according to the project goal. For example, the prediction scores are arranged in descending order, and the samples are divided into multiple intervals. The complaint rate corresponding to each quantile is obtained through quantile statistics, as shown in Table 4:

[0156] Quantile Cumulative number of policies Cumulative number of complaints Cumulative complaint rate 0.1 xx xx 1.5289％ 0.2 xx xx 0.7431％ 0.3 xx xx 0.5165％ ... ... ... ... 0.9 xx xx xx 1 xx xx xx ... ... ... ... 90 xx xx 0.0045％ 100 xx xx 0.0042％

[0157] Table 4

[0158] According to the business objectives, determine the acceptable complaint rate and find the quantile that meets the business requirements. For example, if the business target complaint rate is less than 0.3%, then find the lowest quantile with a cumulative complaint rate less than 0.3%, and the proportion of the cumulative number of policies corresponding to this quantile to the total number of samples is the threshold. If the identified complaint rate at the 0.3 (0.3%) quantile can meet the business application requirements, then take the first 0.3% of the customers' policies predicted and sorted in descending order as the second policy.

[0159] In this embodiment, the second preset threshold is determined according to business needs, and the second preset threshold is used to screen out customer groups that need to be focused on and checked. By analyzing the prediction score, customers with higher risks can be effectively identified and targeted risk management can be performed.

[0160] See also Figure 3Accordingly, in some embodiments, step S400 may include but is not limited to step S410:

[0161] Step S410, grouping the multiple second insurance policies according to the parity of the last digits of the policy numbers recorded in the insurance policies to obtain the experimental group and the control group.

[0162] In this embodiment, the second policy is divided into two groups according to the parity of the last digit of the customer's policy number, with customers with odd-numbered last digits forming one group and customers with even-numbered last digits forming another group. Since the last digits of the policy number are randomly assigned, the parity grouping can be approximately considered as a random grouping process, and according to the law of large numbers, the odd and even numbers of the last digits of the customer's policy number are basically close in terms of data volume. One of the groups is selected as the experimental group and the other as the control group. The experimental group policies will use the model prediction results for risk management, and the control group customers will be risk managed according to the original rules or in a random manner. Through this grouping method, the differences between the experimental group and the control group can be effectively controlled, and scientific risk management experiments can be conducted.

[0163] In addition, since the policies involved in the evaluation may be full policies corresponding to different time points (i.e., all policies before that time point), it means that the same policy will appear repeatedly in the full policies corresponding to different time points. If the traditional random method is used for grouping, the same policy may be assigned to the experimental group and the control group at the same time. This grouping method will cause data mixing and it is impossible to distinguish the effects of the experimental group and the control group. For example, if the obtained policies are full policies at two time points, 20240901 and 20241001, then according to the usual random method, 20240901 may be regarded as the control group, and 20241001 may be regarded as the experimental group, resulting in a large number of repeated policies in the experimental group and the control group, thereby affecting the experimental results. In this embodiment, the grouping is performed by the parity of the tail number, and the tail number of the same policy will not change, so the same policy will always be assigned to the same group, avoiding the problem of data mixing, thereby effectively controlling the grouping stability of the experimental group and the control group, and more accurately evaluating the experimental effect.

[0164] See also Figure 4 In some embodiments, step S500 may include but is not limited to step S510:

[0165] Step S510, determining the insurance policies in the experimental group whose complaint probability values ​​are greater than a third preset threshold as the first target insurance policies;

[0166] Step S520, screening the insurance policies in the control group according to preset rules to obtain the second target insurance policy.

[0167] The complaint probability values ​​of the second insurance policies in the experimental group are screened according to the third preset threshold value to obtain the first target insurance policy with a higher complaint probability value. The third preset threshold value is set according to business needs and can be adjusted according to actual conditions.

[0168] The second insurance policies in the control group are screened according to the preset rules, and the insurance policies with a higher probability of complaints are selected as the second target insurance policies. In one implementation of this embodiment, the preset rules may be the original rules for screening insurance policies with high complaint rates or a portion of insurance policies randomly selected from the insurance policies in the control group as insurance policies with high complaint rates.

[0169] In this embodiment, the first target policy quantity selected by the experimental group and the second target policy quantity selected by the control group must be consistent. The consistency of the number of policies can ensure that the sample sizes of the experimental group and the control group are the same, avoid the impact of sample size differences on the results, and improve the accuracy of the results.

[0170] By determining the first and second target policies and comparing the actual complaint rates of the first and second target policies, we can effectively evaluate the model prediction effect and determine whether the model can effectively identify high-risk policies.

[0171] The embodiment of the present application predicts the complaint rate of insurance policies through a training model, and screens out insurance policies with higher complaint rates based on the complaint rates; divides the insurance policies into two groups based on the parity of the last digits of the customer policy numbers in the screened out insurance policies, to obtain an experimental group and a control group; screens the insurance policies in the experimental group through the training model to obtain high-complaint insurance policies in the experimental group, screens the insurance policies in the control group through the original rules to obtain high-complaint insurance policies in the control group, and finally compares the actual complaint rates of the high-complaint insurance policies in the experimental group and the high-complaint insurance policies in the control group over a period of time, and determines the evaluation result of the training model based on the actual complaint rates; through the model prediction and grouping method, the defects of the existing AB testing method are effectively solved, and the accuracy and reliability of the model effect evaluation are improved.

[0172] See also Figure 5 The embodiment of the present application also provides a model evaluation device, which can implement the above-mentioned model evaluation method, and the device includes:

[0173] An acquisition module 10, used for acquiring a plurality of first insurance policies;

[0174] A prediction module 20, configured to predict each of the first insurance policies using a pre-trained target prediction model to obtain a complaint probability value for each first insurance policy;

[0175] A first screening module 30, configured to screen the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain a plurality of second insurance policies;

[0176] A grouping module 40, used for grouping the plurality of second insurance policies to obtain an experimental group and a control group;

[0177] A second screening module 50 is used to screen the insurance policies in the experimental group to obtain a first target insurance policy, and to screen the insurance policies in the control group to obtain a second target insurance policy, wherein the first target insurance policy and the second target insurance policy each include the same number of insurance policies;

[0178] A calculation module 60 is used to obtain actual complaint results of the first target insurance policy and the second target insurance policy in a first time period, respectively, to obtain a first complaint rate and a second complaint rate;

[0179] The evaluation module 70 is used to determine the evaluation result of the target prediction model according to the first complaint rate and the second complaint rate.

[0180] In some embodiments, the prediction module 20 may include:

[0181] A sample acquisition submodule, used to acquire a plurality of training samples, wherein the training samples include a sample insurance policy and a label for identifying whether the sample insurance policy has been complained about within a second time period, and the sample insurance policy includes a plurality of attributes of the insurance policy and an attribute value corresponding to each attribute;

[0182] A pre-training submodule, used to train at least two basic models using the multiple training samples to obtain a prediction model corresponding to each of the basic models;

[0183] The model screening submodule is used to screen the prediction models corresponding to at least two of the basic models to obtain the target prediction model.

[0184] In some embodiments, the pre-training submodule may include:

[0185] A sample division unit, used for dividing the plurality of training samples into a training set and a validation set;

[0186] The model training unit is used to train at least two basic models using the training samples in the training set to obtain a prediction model corresponding to each of the basic models.

[0187] In some embodiments, the model screening submodule may include:

[0188] A model screening unit is used to screen the prediction models corresponding to at least two of the basic models using the training samples in the verification set to obtain the target prediction model.

[0189] In some embodiments, the model screening unit may include:

[0190] A first input subunit is used to traverse each prediction model, input the training samples in the verification set into the prediction model, and obtain a first complaint probability value of each training sample in the verification set;

[0191] A first quantile subunit is used to arrange the multiple first complaint probability values ​​in descending order, and divide the training samples corresponding to the first complaint probability values ​​according to a first rule to obtain multiple first quantile intervals, each of which includes at least one training sample in the validation set;

[0192] A first calculation subunit, used for determining a first actual complaint rate for each first quantile interval;

[0193] A second input subunit, used for inputting the training samples in the training set into each of the prediction models to obtain a second complaint probability value for each training sample in the training set;

[0194] A second quantile subunit is used to arrange the multiple second complaint probability values ​​in descending order, divide the training samples corresponding to the second complaint probability values ​​according to the first rule, and obtain multiple second quantile intervals, each of which includes at least one training sample in the training set, and the multiple first quantile intervals are the same as the multiple second quantile intervals;

[0195] A second calculation subunit is used to determine a second actual complaint rate for each second quantile interval;

[0196] A stability index subunit is used to obtain the stability index of the corresponding quantile interval according to the first actual complaint rate and the second actual complaint rate corresponding to the same first quantile interval and the second quantile interval respectively;

[0197] The average operation subunit is used to perform an average operation on the stability index corresponding to each quantile interval to obtain a target stability index;

[0198] The first screening subunit is used to determine the prediction model whose target stability index is less than a first preset threshold as the target prediction model.

[0199] In some embodiments, the first screening subunit is also used to determine the prediction model whose target stability index is less than the first preset threshold as an alternative prediction model; arrange the multiple second complaint probability values ​​in descending order, and divide the training samples corresponding to the second complaint probability values ​​according to the second rule to obtain multiple third quantile intervals, each of the third quantile intervals including at least one training sample in the training set; determine the target evaluation parameters of each third quantile interval, the target evaluation parameters including at least one of the third actual complaint rate, recall rate and lift, the third actual complaint rate being the actual complaint amount of the training samples in the third quantile interval divided by the total number of training samples in the third quantile interval, the recall rate being the actual complaint amount of the training samples in the third quantile interval divided by the total number of complaints of the training samples in the training set, and the lift being the third actual complaint rate of the third quantile interval divided by the total number of complaints; based on the target evaluation parameters, evaluate the alternative prediction model to determine the target prediction model.

[0200] In some embodiments, the first screening module 30 may include:

[0201] The first screening submodule is used to screen the complaint probability value of each insurance policy according to a second preset threshold value to obtain the multiple second insurance policies.

[0202] In some embodiments, the grouping module 40 may include:

[0203] The grouping submodule is used to group the multiple second insurance policies according to the parity of the last digits of the insurance policy numbers recorded in the insurance policies to obtain the experimental group and the control group.

[0204] In some embodiments, the second screening module 50 may include:

[0205] An experimental group screening submodule, used to determine the insurance policies in the experimental group whose complaint probability values ​​are greater than a third preset threshold as the first target insurance policies;

[0206] The control group screening submodule is used to screen the insurance policies in the control group according to preset rules to obtain the second target insurance policy.

[0207] The specific implementation of the model evaluation device is basically the same as the specific implementation of the above-mentioned model evaluation method, and will not be repeated here.

[0208] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above model evaluation method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0209] See also Figure 6, Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0210] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0211] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the model evaluation method of the embodiment of this application;

[0212] Input / output interface 603, used to implement information input and output;

[0213] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0214] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0215] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0216] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned model evaluation method is implemented.

[0217] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0218] The model evaluation method, model evaluation device, electronic device and storage medium provided in the embodiment of the present application obtain multiple first insurance policies; predict each first insurance policy through the target prediction model obtained by pre-training to obtain the complaint probability value of each first insurance policy; screen multiple first insurance policies according to the complaint probability value of each first insurance policy to obtain multiple second insurance policies; group multiple second insurance policies to obtain an experimental group and a control group; screen the insurance policies in the experimental group to obtain the first target insurance policy, and screen the insurance policies in the control group to obtain the second target insurance policy; respectively obtain the actual complaint results of the first target insurance policy and the second target insurance policy in the first time period to obtain the first complaint rate and the second complaint rate; determine the evaluation result of the target prediction model according to the first complaint rate and the second complaint rate. The embodiment of the present application can improve the accuracy of the model effect evaluation.

[0219] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0220] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0221] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0223] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0224] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0225] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0226] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0229] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model evaluation method, characterized in that: The method comprises: Get multiple first policies; Predict each of the first insurance policies using the pre-trained target prediction model to obtain a complaint probability value for each first insurance policy; Screening the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain a plurality of second insurance policies; Grouping the plurality of second insurance policies to obtain an experimental group and a control group; Screening the policies in the experimental group to obtain first target policies, and screening the policies in the control group to obtain second target policies, wherein the first target policies and the second target policies each include the same number of policies; Respectively obtain actual complaint results of the first target insurance policy and the second target insurance policy in the first time period to obtain a first complaint rate and a second complaint rate; An evaluation result of the target prediction model is determined according to the first complaint rate and the second complaint rate.

2. The method according to claim 1, characterized in that Before predicting the insurance policy using the pre-trained target prediction model to obtain the complaint probability value of each insurance policy, the method further includes: Acquire multiple training samples, the training samples including sample insurance policies and labels for identifying whether the sample insurance policies have been complained about within a second time period, the sample insurance policies including multiple attributes of the insurance policies and attribute values ​​corresponding to each attribute; Using the multiple training samples to train at least two basic models to obtain a prediction model corresponding to each of the basic models; The prediction models corresponding to at least two of the basic models are screened to obtain the target prediction model.

3. The method according to claim 2, characterized in that The step of training at least two basic models using the multiple training samples to obtain a prediction model corresponding to each of the basic models includes: Dividing the plurality of training samples into a training set and a validation set; Using the training samples in the training set to train at least two basic models to obtain a prediction model corresponding to each of the basic models; The step of screening the prediction models corresponding to at least two of the basic models to obtain the target prediction model includes: The prediction models corresponding to at least two of the basic models are screened using the training samples in the validation set to obtain the target prediction model.

4. The method according to claim 3, characterized in that The step of using the training samples in the validation set to screen prediction models corresponding to at least two of the basic models to obtain the target prediction model includes: Traversing each prediction model, inputting the training samples in the verification set into the prediction model, and obtaining a first complaint probability value for each training sample in the verification set; Arrange the multiple first complaint probability values ​​in descending order, and divide the training samples corresponding to the first complaint probability values ​​according to the first rule to obtain multiple first quantile intervals, each of which includes at least one training sample in the validation set; Determine the first actual complaint rate for each first quantile interval; Inputting the training samples in the training set into each of the prediction models to obtain a second complaint probability value for each training sample in the training set; Arrange the multiple second complaint probability values ​​in descending order, and divide the training samples corresponding to the second complaint probability values ​​according to the first rule to obtain multiple second quantile intervals, each of which includes at least one training sample in the training set, and the multiple first quantile intervals are the same as the multiple second quantile intervals; Determine the second actual complaint rate for each second quantile interval; According to the first actual complaint rate and the second actual complaint rate corresponding to the same first quantile interval and the second quantile interval respectively, the stability index of the corresponding quantile interval is obtained; The stability index corresponding to each quantile interval is averaged to obtain the target stability index; The prediction model whose target stability index is less than the first preset threshold is determined as the target prediction model.

5. The method according to claim 4, characterized in that The step of determining the prediction model whose target stability index is less than a first preset threshold as the target prediction model comprises: Determine the prediction model whose target stability index is less than the first preset threshold as an alternative prediction model; Arrange the multiple second complaint probability values ​​in descending order, and divide the training samples corresponding to the second complaint probability values ​​according to the second rule to obtain multiple third quantile intervals, each of the third quantile intervals including at least one training sample in the training set; Determine a target evaluation parameter for each third quantile interval, wherein the target evaluation parameter includes at least one of a third actual complaint rate, a recall rate, and a lift, wherein the third actual complaint rate is the actual complaint amount of the training samples in the third quantile interval divided by the total number of training samples in the third quantile interval, the recall rate is the actual complaint amount of the training samples in the third quantile interval divided by the total number of complaints of the training samples in the training set, and the lift is the third actual complaint rate of the third quantile interval divided by the total number of complaints; According to the target evaluation parameters, the alternative prediction models are evaluated to determine the target prediction model.

6. The method according to claim 1, characterized in that The step of screening the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain the plurality of second insurance policies comprises: Screening the complaint probability value of each insurance policy according to a second preset threshold value to obtain the plurality of second insurance policies; The plurality of second insurance policies are grouped to obtain an experimental group and a control group, including: The multiple second insurance policies are grouped according to the parity of the last digits of the policy numbers recorded in the insurance policies to obtain the experimental group and the control group.

7. The method according to claim 1, characterized in that The screening of the insurance policies in the experimental group to obtain the first target insurance policy, and the screening of the insurance policies in the control group to obtain the second target insurance policy, includes: Determine the insurance policies in the experimental group whose complaint probability values ​​are greater than a third preset threshold as the first target insurance policies; The insurance policies in the control group are screened according to preset rules to obtain the second target insurance policy.

8. A model evaluation device, characterized in that: The device comprises: An acquisition module, used for acquiring multiple first insurance policies; A prediction module, used to predict each of the first insurance policies using a pre-trained target prediction model to obtain a complaint probability value for each first insurance policy; A first screening module, configured to screen the plurality of first insurance policies according to the complaint probability value of each first insurance policy to obtain a plurality of second insurance policies; A grouping module, used for grouping the plurality of second insurance policies to obtain an experimental group and a control group; A second screening module is used to screen the insurance policies in the experimental group to obtain a first target insurance policy, and to screen the insurance policies in the control group to obtain a second target insurance policy, wherein the first target insurance policy and the second target insurance policy each include the same number of insurance policies; A calculation module, used to respectively obtain actual complaint results of the first target insurance policy and the second target insurance policy within a first time period to obtain a first complaint rate and a second complaint rate; An evaluation module is used to determine an evaluation result of the target prediction model according to the first complaint rate and the second complaint rate.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model evaluation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model evaluation method according to any one of claims 1 to 7 is implemented.

Images