Method and device for determining evaluation index, storage medium, and electronic device

Abstract

The present invention provides a method and device for determining an evaluation index, a storage medium, and an electronic device, wherein the method comprises: obtaining a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure comprises at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, and the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object; inputting the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data comprises: a target data structure, and a target evaluation report corresponding to the target data structure; judging whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a method and device for determining an evaluation index, a storage medium, and an electronic device. Background Art

[0002] In recent years, the intelligent risk control system built on the basis of artificial intelligence and big data technology has developed rapidly. Under the premise of effectively reducing the probability of risk events and losses, the intelligent risk control system has expanded the business coverage of the population, improved the business process, reduced the risk control cost, and realized the automation of the entire chain of pre-loan, mid-loan and post-loan for users in the consumption amount scenario, while also promoting the differentiation of risk control management. By building an intelligent risk control system, breaking through the limitations of traditional risk control that uses manual experience control, its core ability is to help financial institutions gradually transform from rule-driven processes to data-driven and intelligent decision-making, and reshape and transform the previous complex risk control business processes. However, with the development of the business, the external environment, regulatory policies, internal management, and capital are constantly changing. The Internet financial market is changing, and various internal and external factors such as changes in the economic environment, changes in customer groups, and data source acquisition will cause model score deviations and even errors.

[0003] With regard to the related technologies, the problem of reduced accuracy of the strategy analysis module due to various internal and external factors such as the economic environment, changes in customer groups, and data source acquisition has been solved, and no effective technical solution has yet been proposed. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for determining an evaluation index, a storage medium, and an electronic device, so as to at least solve the problem in the related art that the accuracy of the strategy analysis module is reduced due to various internal and external factors such as the economic environment, changes in customer groups, and data source acquisition.

[0005] According to one embodiment of the present invention, a method for determining an evaluation index is provided, comprising: obtaining a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, and the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object; inputting the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure; judging whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions.

[0006] Optionally, obtaining the target data structure corresponding to the target object in the policy analysis module includes: obtaining result information of the target object in the policy analysis module after executing a target event, wherein the target event is used to indicate a record of the target object successfully acquiring resources; and obtaining the target data structure corresponding to the target object from the result information.

[0007] Optionally, the strategy monitoring model includes: multiple evaluation indicators, and the target data structure is input into the strategy monitoring model to obtain a target evaluation report of the target data structure, including: determining an evaluation strategy corresponding to the target data structure based on the input target data structure, and determining a target evaluation indicator corresponding to the evaluation strategy from multiple target evaluation indicators; analyzing the target data structure through the target evaluation indicator corresponding to the evaluation strategy; and determining a target evaluation report of the target data structure based on the analysis results.

[0008] Optionally, the evaluation index includes at least one of the following: a scorecard characteristic IV value, a scorecard contribution index value for stability, a scorecard KS statistical value, and a scorecard Gini coefficient value.

[0009] Optionally, judging whether the target evaluation indicators in the target evaluation report reach the expected indicators to determine whether the decision meets the preset conditions, including: when there are multiple target evaluation indicators in the target evaluation report, obtaining the threshold range of expected indicators corresponding to the multiple target evaluation indicators; when the multiple target evaluation indicators meet the threshold range of the multiple expected indicators, determining that the strategy analysis module corresponding to the target evaluation report meets the preset conditions, and the decision generated by the strategy analysis module meets the requirements; when the multiple target evaluation indicators do not meet the threshold range of expected indicators, determining that the strategy analysis module corresponding to the target evaluation report does not meet the preset conditions, and the decision generated by the strategy analysis module does not meet the requirements, and the strategy analysis module needs to be reset.

[0010] Optionally, after judging whether the target evaluation indicators in the target evaluation report reach the expected indicators to determine whether the strategy analysis module meets the preset conditions, the method also includes: when confirming that the strategy analysis module meets the preset conditions, executing the decision generated by the strategy analysis module, and saving the decision data information in the target evaluation report corresponding to the strategy analysis module.

[0011] According to one embodiment of the present invention, there is provided a device for determining an evaluation index, comprising: an acquisition module, used to acquire a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, and the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object; an evaluation module, used to input the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure; a determination module, used to determine whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions.

[0012] Optionally, the acquisition module is also used for: obtaining result information of the target object in the policy analysis module after executing a target event, wherein the target event is used to indicate a record of the target object successfully acquiring resources; obtaining a target data structure corresponding to the target object from the result information.

[0013] Optionally, the above-mentioned evaluation module also includes: multiple evaluation indicators, which are also used to determine the evaluation strategy corresponding to the target data structure based on the input target data structure, and determine the target evaluation indicator corresponding to the evaluation strategy from multiple target evaluation indicators; analyze the target data structure through the target evaluation indicator corresponding to the evaluation strategy; and determine the target evaluation report of the target data structure based on the analysis results.

[0014] Optionally, the above-mentioned evaluation module includes at least one of the following: a score card feature value IV value, a score card contribution index value for stability, a score card KS statistical value, and a score card Gini coefficient value.

[0015] Optionally, the above-mentioned determination module is also used to obtain the threshold range of expected indicators corresponding to the multiple target evaluation indicators in the target evaluation report when there are multiple target evaluation indicators; when the multiple target evaluation indicators meet the threshold range of the multiple expected indicators, determine that the strategy analysis module corresponding to the target evaluation report meets the preset conditions, and the decision generated by the strategy analysis module meets the requirements; when the multiple target evaluation indicators do not meet the threshold range of the expected indicators, determine that the strategy analysis module corresponding to the target evaluation report does not meet the preset conditions, the decision generated by the strategy analysis module does not meet the requirements, and the strategy analysis module needs to be reset.

[0016] Optionally, the above-mentioned device also includes: a saving module, which is used to execute the decision generated by the strategy analysis module when confirming that the strategy analysis module meets preset conditions, and save the decision data information in the target evaluation report corresponding to the strategy analysis module.

[0017] According to another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0018] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0019] Through the present invention, a target data structure corresponding to a target object in a strategy analysis module is obtained, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, and the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object; the target data structure is input into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure; it is determined whether the target evaluation index in the target evaluation report reaches the expected index to determine whether the decision meets the preset conditions. The above technical solution is adopted to solve the problems in the related art that the accuracy of the strategy analysis module is reduced due to various internal and external factors such as the economic environment, changes in customer groups, and acquisition of data sources. The effectiveness of the strategy analysis module is analyzed by the strategy monitoring model to determine whether the decision made by the strategy analysis module deviates from expectations, and then the analysis strategy of the decision analysis module is adjusted, thereby improving the accuracy of the strategy analysis module under changes in external conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal of a method for determining an evaluation index according to an embodiment of the present invention;

[0022] Figure 2is a flow chart of a method for determining an evaluation index according to an embodiment of the present invention;

[0023] Figure 3 A risk control strategy monitoring system for consumer finance according to an optional embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of a policy monitoring data structure according to an optional embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of comprehensive analysis of loan information according to an optional embodiment of the present invention;

[0026] Figure 6 is a curve diagram of the Gini coefficient according to an optional embodiment of the present invention;

[0027] Figure 7 4 is a structural block diagram of a device for determining evaluation indicators according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The method embodiment provided in the first embodiment of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of a method for determining an evaluation index according to an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more (only one is shown in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the evaluation index in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The embodiment of the present invention provides a method for determining an evaluation index, which is applied to the above-mentioned computer terminal. Figure 2 is a flow chart of a method for determining an evaluation index according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0034] Step S202, obtaining a target data structure corresponding to the target object in the strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, wherein the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object, wherein the decision data information includes scorecard return information and trigger rule information;

[0035] Step S204, inputting the target data structure into the strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure, the target evaluation report corresponds to an analysis report of the information returned by the scorecard, and the analysis report includes: an analysis report of the IV value of the scorecard feature quantity, an analysis report of the stability of the scorecard, a KS statistical analysis report of the scorecard, and a statistical analysis report of the Gini coefficient of the scorecard;

[0036] Step S206, judging whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions.

[0037] Through the above steps, a target data structure corresponding to the target object in the strategy analysis module is obtained, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, and the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object; the target data structure is input into the strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure; it is judged whether the target evaluation index in the target evaluation report reaches the expected index to determine whether the decision meets the preset conditions. The above technical solution is adopted to solve the problem in the related art that the accuracy of the strategy analysis module is reduced due to various internal and external factors such as economic environment, changes in customer groups and data source acquisition. The effectiveness of the strategy analysis module is analyzed by the strategy monitoring model to judge whether the decision made by the strategy analysis module deviates from expectations, and then the analysis strategy of the decision analysis module is adjusted, thereby improving the accuracy of the strategy analysis module under the condition of changes in external conditions.

[0038] Optionally, obtaining the target data structure corresponding to the target object in the policy analysis module includes: obtaining result information of the target object in the policy analysis module after executing a target event, wherein the target event is used to indicate a record of the target object successfully acquiring resources; and obtaining the target data structure corresponding to the target object from the result information.

[0039] In order to ensure that the target data structure of the acquired target object is more consistent with the actual financial situation, when acquiring the target data structure, the result information of the target event of the target object successfully acquiring the resource can also be acquired, thereby ensuring the accuracy of the target data structure.

[0040] For example, determine the user's settlement status after a successful loan and other information. According to the user's repayment behavior, the user's historical and current overdue situations can be evaluated. At this time, the application number in the application information table and the application number in the loan information table can be associated to realize the linkage analysis of decision data and credit repayment performance data, and then realize real-time monitoring, such as the number of applications, the number of successful credit approvals, the credit approval rate, the anti-fraud rejection rate, the average credit limit, and the pass rate of each strategy's hits, approvals, and rejections. Combined with credit performance data, flexible and diversified analysis and credit repayment performance data tracking can be carried out to monitor the stability and accuracy of the scoring model.

[0041] Optionally, the strategy monitoring model includes: multiple evaluation indicators, and the target data structure is input into the strategy monitoring model to obtain a target evaluation report of the target data structure, including: determining an evaluation strategy corresponding to the target data structure based on the input target data structure, and determining a target evaluation indicator corresponding to the evaluation strategy from multiple target evaluation indicators; analyzing the target data structure through the target evaluation indicator corresponding to the evaluation strategy; and determining a target evaluation report of the target data structure based on the analysis results.

[0042] In short, in order to ensure the pertinence of the strategy monitoring model, the evaluation strategy of the target data structure can be confirmed in advance by judging the data information in the target data structure, and then the target evaluation indicators corresponding to the evaluation strategy are determined from a variety of target evaluation indicators to analyze the data information, and the target evaluation report of the target data structure is determined according to the analysis results.

[0043] Optionally, the evaluation index includes at least one of the following: a scorecard characteristic IV value, a scorecard contribution index value for stability, a scorecard KS statistical value, and a scorecard Gini coefficient value.

[0044] Optionally, judging whether the target evaluation indicators in the target evaluation report reach the expected indicators to determine whether the decision meets the preset conditions, including: when there are multiple target evaluation indicators in the target evaluation report, obtaining the threshold range of expected indicators corresponding to the multiple target evaluation indicators; when the multiple target evaluation indicators meet the threshold range of the multiple expected indicators, determining that the strategy analysis module corresponding to the target evaluation report meets the preset conditions, and the decision generated by the strategy analysis module meets the requirements; when the multiple target evaluation indicators do not meet the threshold range of expected indicators, determining that the strategy analysis module corresponding to the target evaluation report does not meet the preset conditions, and the decision generated by the strategy analysis module does not meet the requirements, and the strategy analysis module needs to be reset.

[0045] That is to say, since there may be multiple target evaluation indicators in the target data structure, but different target objects have different requirements for the resources they provide, after obtaining the threshold ranges of multiple expected indicators corresponding to the multiple target evaluation indicators corresponding to the target data structure, the multiple target evaluation indicators are further judged. After determining that all target evaluation indicators meet the threshold range of the expected indicators, it is confirmed that the decision generated by the strategy analysis module meets the requirements and can be smoothly executed without adjustment. When there are target evaluation indicators that do not meet the threshold range of the expected indicators, it means that the decision generated by the strategy analysis module does not meet the requirements. According to the actual situation, the strategy analysis module needs to be reset. The effectiveness of the strategy analysis module is analyzed through the strategy monitoring model to determine whether the decision made by the strategy analysis module deviates from the expectation, and then adjust the analysis strategy of the decision analysis module, which improves the accuracy of the strategy analysis module under the change of external conditions.

[0046] Optionally, after judging whether the target evaluation indicators in the target evaluation report reach the expected indicators to determine whether the strategy analysis module meets the preset conditions, the method also includes: when confirming that the strategy analysis module meets the preset conditions, executing the decision generated by the strategy analysis module, and saving the decision data information in the target evaluation report corresponding to the strategy analysis module.

[0047] In short, after determining that the policy analysis module meets the preset conditions, that is, the accuracy of the policy analysis module meets the application threshold, and the decision-making process does not deviate from expectations, further, based on the judgment result of the policy of the policy analysis module, it is determined whether to provide resources to the target object or not, and the decision data information corresponding to the target object is determined based on the judgment result. When providing resources to the target object, it is used as a reference to speed up the time interval for resource release, and multiple successful records of the target object can also be saved, which can be processed first when executed again, and the records of the number of times the policy analysis module is used effectively and the target evaluation report corresponding to the policy analysis module are saved in the database one by one.

[0048] In order to better understand the determination process of the above evaluation indicators, the following is described in conjunction with optional embodiments, but is not intended to limit the technical solutions of the embodiments of the present invention.

[0049] An optional embodiment of the present invention provides a risk control strategy monitoring system for consumer finance, including: a strategy monitoring data structure module 32, and a strategy model monitoring and analysis module 34. Among them, the strategy monitoring data structure module 32 contains the information data (equivalent to the target object in the embodiment of the present invention) that needs to be collected when the user (equivalent to the target object in the embodiment of the present invention) applies for consumer finance (equivalent to the target data structure in the embodiment of the present invention); the strategy model monitoring and analysis module 34 is used to analyze the user information data stored in the strategy monitoring data structure module 32 through a monitoring scoring model (equivalent to the strategy monitoring model in the embodiment of the present invention) to realize the monitoring of the credit risk control strategy.

[0050] It should be noted that the detailed data structure in the strategy monitoring data structure module 32 is as follows: Figure 4 As shown, the application information stored in the strategy monitoring data structure module 32 includes user information, collateral information, and decision result information; the decision result information includes scorecard return information and trigger rule information, and the scorecard return information can be converted accordingly according to the corresponding characteristic variable information; in addition, since the decision result will be affected by the derivative variable, the decision result information will also include the derivative variable information. User information also includes user credit information, contact information, and fraud information, wherein the user credit information includes user loan information, user credit card credit information, and user query information.

[0051] Optionally, according to the characteristics of consumer finance business, by determining the input variables and output variables of the decision engine in the consumer finance risk control decision-making process, as well as the log data generated by the decision engine in the process of making risk decisions, the corresponding data architecture of the decision data is designed to facilitate the storage and processing of data for subsequent risk strategy monitoring, and also lay the foundation for changes in user portraits during loans, user asset quality analysis, and user behavior analysis.

[0052] For example, in the process of risk decision-making, because one application corresponds to one user, and one user corresponds to multiple credit records, such as the application for multiple credit cards, multiple loans, and credit inquiry records of the People's Bank of China, or multiple loan products processed internally, the borrower has multiple contacts. When the decision engine outputs, the loan application may hit multiple rules and other similar one-to-many situations. At this time, it is necessary to establish a one-to-many table-level association relationship to support the input and output of risk decisions, as well as the derived variable data generated in the decision-making process. When the user finally makes a loan through risk decision and signs the contract, the consumer financial service institution generates a repayment plan for each period of each contract, including the due date, due principal, due interest, and due penalty interest, and synchronizes the actual repayment date, actual principal, actual interest, actual penalty interest, and exemption information in the repayment details every day, and determines the settlement status and other information. According to the user's repayment behavior, the user's historical overdue and current overdue situation can be evaluated. At this time, the application number in the application information table and the application number in the loan information table can be associated to realize the linkage analysis of decision data and credit repayment performance data, and then realize real-time monitoring, such as the number of applications, the number of successful credit orders, the credit approval rate, the anti-fraud rejection rate, the average credit limit, and the pass rate of the hit number, pass number, and rejection number of each strategy. Combined with the credit performance data, flexible and diversified analysis and credit repayment performance data tracking are carried out to monitor the stability of the scoring model.

[0053] like Figure 5 As shown, the user's application information in the application information table is further compared with the corresponding customer information recorded in the system, and a comprehensive analysis is performed in combination with the loan information in the system to ensure the stability of the monitoring scoring model.

[0054] Optionally, the system can monitor the daily incoming application volume, approval volume, approved volume, loan volume, approval rate, approved amount, average number of approved applications, loan amount, and average number of loan applications in real time. The specific table structure is shown in Table 1:

[0055] Table 1

[0056]

[0057]

[0058] Optionally, the analysis of rule violations included in the decision result information can be based on rules, counting the number of times a single rule is triggered, and the final review result statistics corresponding to a single rule. The specific table structure can be shown in Table 2:

[0059] Table 2

[0060]

[0061] It should be noted that when the approval rate suddenly drops, business personnel can use the above monitoring report to analyze the reasons for the drop in the approval rate, whether it is caused by the rejection rules, and which rule causes the drop in the approval rate. Through the strategy analysis logic, the distribution ratio of rejection data of each rule will be compared and analyzed to find out the rules with higher ratios, and then analyze the variables corresponding to the rules (equivalent to the target characteristic values ​​in the embodiment of the present invention), and finally formulate a risk control strategy.

[0062] Optionally, the monitoring scoring model corresponds to a scorecard feature quantity IV (information value, referred to as IV) value analysis report. The feature quantity is the variable corresponding to each scorecard. The goal of the scoring model is to distinguish between good and bad customers. Bad customers are non-target customers identified by the company, such as those with a historical overdue number of times exceeding the set number of overdue payments. The scorecard feature quantity IV value analysis report is further represented by WOE (Weight Of Evidence, referred to as WOE) coding. When calculating the WOE coding, it is necessary to group the variables according to the differences. After grouping, for the i-th group, the calculation formula of WOE is as follows: py i is the proportion of defaulting customers in this group to all defaulting customers in the sample, pn i It is the proportion of non-defaulting customers in this group to non-defaulting customers in all samples. It can also be understood as the difference between the ratio of defaulting customers to non-defaulting customers in the current group and the ratio in all samples. The larger the WOE, the greater the difference, and the greater the possibility of default in the sample in the group. The formula for determining the IV value based on the WOE value is as follows: Then calculate the IV value of the entire variable, the formula is as follows: It is the sum of all IV values. In addition, since the IV value takes into account the impact of the number of variables in each grouping, it can comprehensively reflect the impact of each grouping level.

[0063] Optionally, the scorecard feature IV value analysis report may be as shown in Table 3.

[0064] Table 3

[0065]

[0066] Optionally, the monitoring scoring model corresponds to a scorecard as a stability analysis report, in which PS (Population Stability Index, PSI for short) reflects the stability of the distribution of validation samples in each score segment and the distribution of modeling samples. In modeling, we often use it to screen feature variables and evaluate model stability. Stability has a reference, so there are two distributions-actual distribution (actual) and expected distribution (expected). Among them, the training sample (In the Sample, INS) is usually used as the expected distribution when modeling, and the validation sample is usually used as the actual distribution. Validation samples generally include out-of-sample (Out of Sample, referred to as OOS) and cross-time samples (Out of Time, referred to as OOT). The calculation formula of PSI is as follows: Among them: A stands for actual, E stands for expected, the smaller the PSI value, the smaller the difference between the two distributions, and the more stable it is. The PSI value range and recommendations are shown in Table 4 below:

[0067] Table 4

[0068]

[0069] Optionally, the monitoring scoring model corresponds to a scoring card KS statistical analysis report. KS (Kolmogorov-smirnov, abbreviated as KS) is often used to evaluate the model's discrimination. The greater the discrimination, the stronger the risk ranking ability of the model. The definition between good accounts (good_rate) and bad accounts (bad_rate) is often vague and continuous, depending on actual business needs. The KS indicator tends to measure the difference between the distribution of positive and negative samples from a probability perspective. Generally speaking, the larger the KS, the better the degree of discrimination between positive and negative samples, which is expressed by the following formula:

[0070] KS=max{|cum(bad_rate)-cum(good_rate)|}

[0071] Optionally, the scorecard KS statistical analysis report may be as shown in Table 5.

[0072] Table 5

[0073]

[0074] Optionally, the monitoring scoring model corresponds to a scoring Gini coefficient statistical analysis report. GINI coefficient: It is also used to evaluate the risk differentiation ability of the model. The GINI statistic measures the area between the cumulative distribution of the number of bad accounts on the number of good accounts and the random distribution curve. The greater the difference between the distribution of good accounts and bad accounts, the higher the GINI index, indicating that the model has a stronger risk differentiation ability.

[0075] The calculation steps of GINI coefficient are as follows: Calculate the number of good and bad accounts in each scoring interval. Calculate the ratio of the cumulative number of good accounts to the total number of good accounts (cumulative good%) and the cumulative number of bad accounts to the total number of bad accounts (cumulative bad%) in each scoring interval. According to the cumulative good account ratio and the cumulative bad account ratio, the following is obtained: Figure 6 The curve ADC is shown. Calculate the area of ​​the shaded part. The percentage of the shaded area to the area of ​​the right triangle ABC is the GINI coefficient.

[0076] Optionally, the scorecard KS statistical analysis report may be as shown in Table 6.

[0077] Table 6

[0078]

[0079] An optional embodiment of the present invention requires continuous optimization, adjustment, iteration and updating of intelligent risk control strategies, while a complete consumer finance risk control strategy and model system can dynamically monitor in real time the accuracy of user credit risk assessment of strategies that have been put into operation. By analyzing the monitoring of operating strategies, data-driven business can be achieved, risk control strategies are continuously improved, and the purpose of promoting differentiated risk control management and humanized credit business can be achieved.

[0080] In summary, the optional embodiments of the present invention store the input and output variables of the decision engine and the intermediate data generated in the decision-making process from the perspective of risk business by establishing tables of multi-level parent-child relationships such as application information, applicant information, credit information, decision results, scoring return results, trigger rule information, etc. for unstructured log data in the risk process, so as to monitor the number of applications, the number of successful credit approvals, the credit approval rate, the anti-fraud rejection rate and other pass rates, the average credit limit, and the pass rate of the number of hits, the number of approvals and the number of rejections of each strategy in real time for risk control, and combine credit performance data to conduct flexible and diversified analysis and track credit repayment performance data, provide technical data and technical support for monitoring the stability and effectiveness of the scoring model, and provide approval monitoring and rejection reason analysis for risk control strategies and models. The analysis, overdue distribution report of scoring level, IV value analysis report of scoring card characteristics, stability analysis report of scoring card, KS statistical analysis of scoring card, and Gini coefficient statistical analysis report of scoring card provide an optimization basis for strategy optimization and scoring model. Through the analysis of the monitoring of the running strategy model, it can achieve the purpose of data-driven business, continuous improvement of risk control strategy, promotion of differentiated risk control management and humanization of credit business, so as to make it closer to the consumer finance business scenario. At the same time, in the actual risk strategy monitoring process, it can achieve the purpose of real-time monitoring of strategies and models, which can be directly implemented and automatically generate monitoring reports. At the same time, by storing all the data of the decision engine, it can realize the linkage analysis with the repayment performance data, making the customer portrait analysis more complete and the risk assessment of credit customers more accurate.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0082] In the present embodiment, a device for determining an evaluation index is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the description has been made no further. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0083] Figure 7is a structural block diagram of a device for determining an evaluation index according to an embodiment of the present invention, such as Figure 7 As shown, the device comprises:

[0084] (1) an acquisition module 72, configured to acquire a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information about the target object, application information about the target object, collateral information about the target object, and decision data information about the target object, wherein the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object, wherein the decision data information includes scorecard return information and trigger rule information;

[0085] (2) an evaluation module 74, which is used to input the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure, the target evaluation report corresponds to an analysis report of the information returned by the scorecard, and the analysis report includes: an analysis report on the IV value of the scorecard feature quantity, an analysis report on the stability of the scorecard, a KS statistical analysis report on the scorecard, and a statistical analysis report on the Gini coefficient of the scorecard;

[0086] (3) A determination module 76 is used to determine whether the target evaluation indicators in the target evaluation report meet the expected indicators, so as to determine whether the decision meets the preset conditions.

[0087] Through the above-mentioned device, a target data structure corresponding to a target object in a strategy analysis module is obtained, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, and the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object; the target data structure is input into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure; it is determined whether the target evaluation index in the target evaluation report reaches the expected index to determine whether the decision meets the preset conditions. The above-mentioned technical solution is adopted to solve the problem that the accuracy of the strategy analysis module is reduced due to various internal and external factors such as economic environment, changes in customer groups, and acquisition of data sources in the related technology. The effectiveness of the strategy analysis module is analyzed by the strategy monitoring model to determine whether the decision made by the strategy analysis module deviates from expectations, and then the analysis strategy of the decision analysis module is adjusted, thereby improving the accuracy of the strategy analysis module under the condition of changes in external conditions.

[0088] Optionally, the acquisition module is also used for: obtaining result information of the target object in the policy analysis module after executing a target event, wherein the target event is used to indicate a record of the target object successfully acquiring resources; obtaining a target data structure corresponding to the target object from the result information.

[0089] In order to ensure that the target data structure of the acquired target object is more consistent with the actual financial situation, when acquiring the target data structure, the result information of the target event of the target object successfully acquiring the resource can also be acquired, thereby ensuring the accuracy of the target data structure.

[0090] For example, determine the user's settlement status after a successful loan and other information. According to the user's repayment behavior, the user's historical and current overdue situations can be evaluated. At this time, the application number in the application information table and the application number in the loan information table can be associated to realize the linkage analysis of decision data and credit repayment performance data, and then realize real-time monitoring, such as the number of applications, the number of successful credit approvals, the credit approval rate, the anti-fraud rejection rate, the average credit limit, and the pass rate of each strategy's hits, approvals, and rejections. Combined with credit performance data, flexible and diversified analysis and credit repayment performance data tracking can be carried out to monitor the stability and accuracy of the scoring model.

[0091] Optionally, the above-mentioned evaluation module also includes: multiple evaluation indicators, which are also used to determine the evaluation strategy corresponding to the target data structure based on the input target data structure, and determine the target evaluation indicator corresponding to the evaluation strategy from multiple target evaluation indicators; analyze the target data structure through the target evaluation indicator corresponding to the evaluation strategy; and determine the target evaluation report of the target data structure based on the analysis results.

[0092] In short, in order to ensure the pertinence of the strategy monitoring model, the evaluation strategy of the target data structure can be confirmed in advance by judging the data information in the target data structure, and then the target evaluation indicators corresponding to the evaluation strategy are determined from a variety of target evaluation indicators to analyze the data information, and the target evaluation report of the target data structure is determined according to the analysis results.

[0093] Optionally, the above-mentioned evaluation module includes at least one of the following: a score card feature value IV value, a score card contribution index value for stability, a score card KS statistical value, and a score card Gini coefficient value.

[0094] Optionally, the above-mentioned determination module is also used to obtain the threshold range of expected indicators corresponding to the multiple target evaluation indicators in the target evaluation report when there are multiple target evaluation indicators; when the multiple target evaluation indicators meet the threshold range of the multiple expected indicators, determine that the strategy analysis module corresponding to the target evaluation report meets the preset conditions, and the decision generated by the strategy analysis module meets the requirements; when the multiple target evaluation indicators do not meet the threshold range of the expected indicators, determine that the strategy analysis module corresponding to the target evaluation report does not meet the preset conditions, the decision generated by the strategy analysis module does not meet the requirements, and the strategy analysis module needs to be reset.

[0095] That is to say, since there may be multiple target evaluation indicators in the target data structure, but different target objects have different requirements for the resources they provide, after obtaining the threshold ranges of multiple expected indicators corresponding to the multiple target evaluation indicators corresponding to the target data structure, the multiple target evaluation indicators are further judged. After determining that all target evaluation indicators meet the threshold range of the expected indicators, it is confirmed that the decision generated by the strategy analysis module meets the requirements and can be smoothly executed without adjustment. When there are target evaluation indicators that do not meet the threshold range of the expected indicators, it means that the decision generated by the strategy analysis module does not meet the requirements. According to the actual situation, the strategy analysis module needs to be reset. The effectiveness of the strategy analysis module is analyzed through the strategy monitoring model to determine whether the decision made by the strategy analysis module deviates from the expectation, and then adjust the analysis strategy of the decision analysis module, which improves the accuracy of the strategy analysis module under the change of external conditions.

[0096] Optionally, the above-mentioned device also includes: a saving module, which is used to execute the decision generated by the strategy analysis module when confirming that the strategy analysis module meets preset conditions, and save the decision data information in the target evaluation report corresponding to the strategy analysis module.

[0097] In short, after determining that the policy analysis module meets the preset conditions, that is, the accuracy of the policy analysis module meets the application threshold, and the decision-making process does not deviate from expectations, further, based on the judgment result of the policy of the policy analysis module, it is determined whether to provide resources to the target object or not, and the decision data information corresponding to the target object is determined based on the judgment result. When providing resources to the target object, it is used as a reference to speed up the time interval for resource release, and multiple successful records of the target object can also be saved, which can be processed first when executed again, and the records of the number of times the policy analysis module is used effectively and the target evaluation report corresponding to the policy analysis module are saved in the database one by one.

[0098] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0099] An embodiment of the present invention further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0100] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0101] S1, obtaining a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, wherein the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object;

[0102] S2, inputting the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure;

[0103] S3, judging whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions.

[0104] An embodiment of the present invention further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0105] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0106] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0107] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0108] S1, obtaining a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, wherein the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object;

[0109] S2, inputting the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure;

[0110] S3, judging whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions.

[0111] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0112] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0113] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining an evaluation index, characterized in that: include: Obtaining a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, wherein the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object, wherein the decision data information includes scorecard return information and trigger rule information; Input the target data structure into the strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure, the target evaluation report corresponds to an analysis report of the information returned by the scorecard, and the analysis report includes: an analysis report on the IV value of the scorecard feature quantity, an analysis report on the stability of the scorecard, a KS statistical analysis report on the scorecard, and a statistical analysis report on the Gini coefficient of the scorecard; Determine whether the target evaluation indicators in the target evaluation report meet the expected indicators, so as to determine whether the decision meets the preset conditions; The step of judging whether the target evaluation index in the target evaluation report reaches the expected index to determine whether the decision meets the preset conditions includes: In the case where there are multiple target evaluation indicators in the target evaluation report, obtaining threshold ranges of expected indicators corresponding to the multiple target evaluation indicators; When the plurality of target evaluation indicators meet the threshold range of the plurality of expected indicators, it is determined that the strategy analysis module corresponding to the target evaluation report meets the preset conditions and the decision generated by the strategy analysis module meets the requirements; When a plurality of the target evaluation indicators do not meet the threshold range of the expected indicators, it is determined that the strategy analysis module corresponding to the target evaluation report does not meet the preset conditions, the decision generated by the strategy analysis module does not meet the requirements, and the strategy analysis module needs to be reset.

2. The method according to claim 1, characterized in that Get the target data structure corresponding to the target object in the strategy analysis module, including: Obtaining result information of the target object in the policy analysis module after executing the target event, wherein the target event is used to indicate a record of the target object successfully acquiring resources; A target data structure corresponding to the target object is obtained from the result information.

3. The method according to claim 1, characterized in that After determining whether the target evaluation index in the target evaluation report reaches the expected index to determine whether the strategy analysis module meets the preset conditions, the method further includes: When it is confirmed that the strategy analysis module meets the preset conditions, the decision generated by the strategy analysis module is executed, and the decision data information is saved in the target evaluation report corresponding to the strategy analysis module.

4. A device for determining an evaluation index, characterized in that: include: An acquisition module, used to acquire a target data structure corresponding to a target object in a strategy analysis module, wherein the target data structure includes at least one of the following: information of the target object, application information of the target object, collateral information of the target object, and decision data information for the target object, wherein the decision generated by the strategy analysis module is used to indicate whether to provide resources for the target object, wherein the decision data information includes scorecard return information and trigger rule information; An evaluation module is used to input the target data structure into a strategy monitoring model to obtain a target evaluation report of the target data structure, wherein the strategy monitoring model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: a target data structure, and a target evaluation report corresponding to the target data structure, the target evaluation report corresponds to an analysis report of the information returned by the scorecard, and the analysis report includes: an analysis report of the IV value of the scorecard feature quantity, an analysis report of the stability of the scorecard, a KS statistical analysis report of the scorecard, and a statistical analysis report of the Gini coefficient of the scorecard; A determination module is used to determine whether the target evaluation index in the target evaluation report reaches the expected index, so as to determine whether the decision meets the preset conditions; The determination module is also used to obtain the threshold range of expected indicators corresponding to multiple target evaluation indicators in the target evaluation report when there are multiple target evaluation indicators; when multiple target evaluation indicators meet the threshold range of multiple expected indicators, determine that the strategy analysis module corresponding to the target evaluation report meets the preset conditions and the decision generated by the strategy analysis module meets the requirements; when multiple target evaluation indicators do not meet the threshold range of expected indicators, determine that the strategy analysis module corresponding to the target evaluation report does not meet the preset conditions, the decision generated by the strategy analysis module does not meet the requirements, and the strategy analysis module needs to be reset.

5. The device according to claim 4, characterized in that The acquisition module is also used to: obtain result information of the target object in the policy analysis module after executing a target event, wherein the target event is used to indicate a record of the target object successfully acquiring resources; and obtain a target data structure corresponding to the target object from the result information.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 3 when executed.

7. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 3.

Images