User risk assessment model output data verification method and device, equipment and medium

Abstract

The invention discloses a user risk assessment model output data verification method and device, equipment and a medium, and the method comprises the steps: obtaining a target user data set corresponding to a user data assessment instruction, and obtaining an assessment result distribution matrix corresponding to the target user data set based on a classification model; determining a corresponding risk identification parameter output result based on the risk identification parameter acquisition strategy; determining a corresponding user structure parameter output result based on the user structure parameter acquisition strategy; and fusing the risk identification parameter output result and the user structure parameter output result to obtain a fused output result. According to the embodiment of the invention, after the user data set is input into the classification model to obtain the evaluation result and form the evaluation result distribution matrix, quantitative evaluation is carried out from two dimensions of the risk identification parameter and the user structure parameter to obtain the fusion output result which can be visually displayed, so that the efficiency of obtaining the judgment result is improved; and the reusability of an output result verification mode is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data modeling for automated modeling, and in particular to a method, device, equipment and medium for verifying output data of a user risk assessment model. Background Art

[0002] User risk stratification rating is an indispensable part of the risk control strategy system of financial technology companies, and it is at the end of data processing. When conducting user risk stratification rating, the user data to be evaluated is input into the risk assessment model as input data to obtain an output result of user risk stratification rating. However, in the field of financial technology, whether some specific application ratings meet the target risk requirements, etc., the output result of user risk stratification rating is often judged by manual experience, which not only has low efficiency in obtaining judgment results, but also has low reusability of judgment methods. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, equipment and medium for verifying the output data of a user risk assessment model, aiming to solve the problem in the prior art that when determining whether a specific application of user risk stratification rating meets the target risk requirement, the output result of the user risk stratification rating is often judged by manual experience, which not only results in low efficiency in obtaining the judgment result, but also has low reusability of the judgment method.

[0004] In a first aspect, an embodiment of the present invention provides a method for verifying output data of a user risk assessment model, which includes: In response to a user data evaluation instruction, acquiring a target user data set corresponding to the user data evaluation instruction; Obtaining an evaluation result corresponding to each target user data in the target user data set based on a pre-trained classification model; wherein the classification model is obtained by training the classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an evaluation result having a value of one of a preset value set; Determine an evaluation result distribution matrix corresponding to the target user data set based on a preset evaluation result distribution matrix acquisition strategy and the evaluation result corresponding to each target user data in the target user data set; Based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix, determining a risk identification parameter output result corresponding to the assessment result distribution matrix; Based on a preset user structure parameter acquisition strategy and the evaluation result distribution matrix, determining a user structure parameter output result corresponding to the evaluation result distribution matrix; and The risk identification parameter output result and the user structure parameter output result are fused according to a preset output result fusion strategy to obtain a fused output result, and the fused output result is sent to a receiving terminal.

[0005] In a second aspect, an embodiment of the present invention further provides a user risk assessment model output data verification device, which includes: a target user data set acquisition unit, configured to acquire, in response to a user data evaluation instruction, a target user data set corresponding to the user data evaluation instruction; An evaluation result acquisition unit is used to acquire an evaluation result corresponding to each target user data in the target user data set based on a pre-trained classification model; wherein the classification model is obtained by training a classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an evaluation result whose value is one of a preset value set; An evaluation result distribution matrix acquisition unit, configured to determine an evaluation result distribution matrix corresponding to the target user data set based on a preset evaluation result distribution matrix acquisition strategy and an evaluation result corresponding to each target user data in the target user data set; A first output result acquisition unit, configured to determine a risk identification parameter output result corresponding to the assessment result distribution matrix based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix; A second output result acquisition unit, configured to determine a user structure parameter output result corresponding to the evaluation result distribution matrix based on a preset user structure parameter acquisition strategy and the evaluation result distribution matrix; and The verification result acquisition unit is used to fuse the risk identification parameter output result and the user structure parameter output result according to a preset output result fusion strategy to obtain a fused output result, and send the fused output result to a receiving terminal.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0008] The embodiment of the present invention provides a method, device, equipment and medium for verifying the output data of a user risk assessment model, the method comprising: in response to a user data assessment instruction, obtaining a target user data set corresponding to the user data assessment instruction; obtaining an assessment result corresponding to each target user data in the target user data set based on a pre-trained classification model; wherein the classification model is obtained by training a classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an assessment result whose value is one of a preset value set; based on a preset assessment result distribution matrix acquisition strategy and the assessment result corresponding to each target user data in the target user data set, determining an assessment result distribution matrix corresponding to the target user data set; based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix, determining a risk identification parameter output result corresponding to the assessment result distribution matrix; based on a preset user structure parameter acquisition strategy and the assessment result distribution matrix, determining a user structure parameter output result corresponding to the assessment result distribution matrix; according to a preset output result fusion strategy, the risk identification parameter output result and the user structure parameter output result are output-fused to obtain a fused output result, and the fused output result is sent to a receiving terminal. In the embodiment of the present invention, after the user data set is input into the classification model to obtain the evaluation results and form the evaluation result distribution matrix, a quantitative evaluation is performed from the two dimensions of risk identification parameters and user structure parameters to obtain a fusion output result that can be visualized, which not only improves the efficiency of obtaining judgment results, but also ensures the reusability of the output result verification method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of an application scenario of the method for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a method for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 3 A schematic diagram of a sub-process of a method for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 4 A schematic diagram of a sub-process of a method for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 5A schematic diagram of a sub-process of a method for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 6 A schematic diagram of a sub-process of a method for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 7 A schematic block diagram of a device for verifying output data of a user risk assessment model provided by an embodiment of the present invention; Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0013] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0014] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] Please also see Figure 1 and Figure 2 , Figure 1 Schematic diagram of a scenario of a method for verifying output data of a user risk assessment model according to an embodiment of the present invention. Figure 2 1 is a flow chart of a method for verifying output data of a user risk assessment model provided by an embodiment of the present invention. The method is applied to a server 10, and the server 10 can be connected to a receiving terminal 20 for communication. Figure 2 As shown, the method includes the following steps S110-S160.

[0016] S110. In response to a user data evaluation instruction, obtain a target user data set corresponding to the user data evaluation instruction.

[0017] In this embodiment, the technical solution is described with the server as the execution subject. A user risk assessment model output data verification platform (also understood as a user risk assessment model output data verification system) is deployed in the server. When the user uses a receiving terminal (also understood as a user terminal, such as a smart phone, tablet computer, etc.) to log in to the user risk assessment model output data verification platform in the server and completes the communication connection with the server, the target user data set corresponding to the user data evaluation instruction can be determined first, and then the target user data set is used as the data to be analyzed for subsequent data processing.

[0018] In one embodiment, if Figure 3 As shown, step S110 includes: S111, obtaining the instruction generation time of the user data evaluation instruction, and determining the target time interval according to the current month and the number of preset months corresponding to the instruction generation time; S112: Acquire the target user data set in a local user database based on the target time interval.

[0019] The starting month corresponding to the target time interval is the current month minus the number of preset months, and the ending month corresponding to the target time interval is the current month minus 1.

[0020] In this embodiment, as a preferred embodiment for determining the target time interval, the instruction generation time of the user data evaluation instruction can be obtained first, such as 12:00 on December 1, 20XX, and the corresponding current month is December 20XX; then, the preset number of months, such as 5, is obtained, and the current month is December 20XX and 5 months are calculated forward as the starting month corresponding to the target time interval, such as July 20XX, and the current month is December 20XX and 1 month is calculated forward as the ending month corresponding to the target time interval, such as November 20XX; finally, the target time interval is determined by taking the first day of the starting month corresponding to the target time interval as the starting date, and the last day of the ending month corresponding to the target time interval as the ending date. After determining the target time interval, it can be used as a screening condition to obtain the target user data set in the local user database. Through the above method, rapid screening and acquisition of the target user data set is achieved.

[0021] S120 . Obtaining an evaluation result corresponding to each target user data in the target user data set based on a pre-trained classification model.

[0022] The classification model is obtained by training the classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an evaluation result whose value is one of the values ​​in the preset value set.

[0023] In this embodiment, the preset scenario may be a financial technology scenario, a smart medical scenario, etc. In the server, the classification model to be trained may be obtained based on the user data training set of the preset scenario. More specifically, the classification model is a risk assessment model (such as a decision tree model, etc.). For example, 8 values ​​are set in the preset value set, and the target user data is input into the classification model to obtain a value that is one of the above 8 values.

[0024] S130 , determining an evaluation result distribution matrix corresponding to the target user data set based on a preset evaluation result distribution matrix acquisition strategy and the evaluation result corresponding to each target user data in the target user data set.

[0025] In this embodiment, after obtaining the evaluation results corresponding to each target user data in the target user data set, the distribution of the evaluation results can be statistically analyzed. For example, still referring to the above example, if July 20XX corresponds to month interval 1, August 20XX corresponds to month interval 2, September 20XX corresponds to month interval 3, October 20XX corresponds to month interval 4, and November 20XX corresponds to month interval 5, there is a target user data corresponding to a user data generation time that belongs to the month interval 1 corresponding to July 20XX, and its corresponding evaluation result is value 1 of the 8 values ​​in the preset value set, then the target user data can be counted once by the cumulative number of matrix elements in the 1st row and 1st column of the initial evaluation result distribution matrix (i.e., +1 processing is performed). Afterwards, when the user data generation time corresponding to other target user data belongs to the month interval ii (the value range of ii is 1 to 5) and the evaluation result is equal to the jjth value in the preset value set (the value range of jj is 1 to 8), it is counted once by the cumulative number of matrix elements in the iith row and jjth column in the initial evaluation result distribution matrix. In the above manner, the evaluation result distribution matrix can be quickly constructed by combining the evaluation results corresponding to each target user data in the target user data set.

[0026] In one embodiment, if Figure 4 As shown, step S130 includes: S131. Construct an initial evaluation result distribution matrix; wherein the total number of columns of the initial evaluation result distribution matrix is ​​equal to the total number of intervals divided by the preset data generation time plus one, and the total number of rows of the initial evaluation result distribution matrix is ​​equal to the total number of values ​​included in the preset value set; the column vector of the last column in the initial evaluation result distribution matrix is ​​a statistical column vector stored locally, and the total number of vector values ​​included in the statistical column vector is equal to the total number of values ​​included in the preset value set; S132. Based on the evaluation result distribution matrix acquisition strategy, each target user data in the target user data set is divided into a value interval based on the user data generation time and the evaluation result, and the corresponding target user data is counted by the corresponding matrix elements in the initial evaluation result distribution matrix, and the percentage ratio is calculated with the total number of target user data in the target user data set to obtain the evaluation result distribution matrix.

[0027] In this embodiment, when constructing the initial evaluation result distribution matrix, if the total number of preset data generation time division intervals is N1 (N1 is a positive integer, such as N1=5 in the above example) and the total number of values ​​included in the preset value set is N2 (N2 is a positive integer, such as N2=8 in the above example), then the initial evaluation result distribution matrix is ​​constructed as a matrix with N2 rows and N1+1 columns. It should be noted that the statistical column vectors stored locally on the server are obtained by first inputting all current user data in the server into the classification model to obtain the corresponding evaluation results, and then for each evaluation result, the corresponding target value in the preset value set is determined, so as to obtain the total number of evaluation results corresponding to each value in the preset value set, and then divide it by the total number of all current user data, so as to obtain the proportion of the number of data corresponding to each value in the preset value set, thereby forming a statistical column vector.

[0028] Afterwards, for each matrix element of the first N1 columns and N2 rows of the initial evaluation result distribution matrix, the corresponding target user data in the target user data set is counted based on the user data generation time and the value interval to which the evaluation result belongs by the corresponding matrix element in the initial evaluation result distribution matrix, and then divided by the total number of target user data in the target user data set to obtain the corresponding percentage ratio, thereby obtaining the evaluation result distribution matrix. It can be seen that through the above method, the evaluation result distribution matrix can be quickly constructed by combining the evaluation result corresponding to each target user data in the target user data set, the statistical column vector and the evaluation result distribution matrix acquisition strategy.

[0029] S140. Based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix, determine a risk identification parameter output result corresponding to the assessment result distribution matrix.

[0030] In this embodiment, after determining the evaluation result distribution matrix, the matrix elements included therein can be analyzed and evaluated in various dimensions from both rows and columns to obtain the verification result corresponding to the risk identification dimension of the evaluation result distribution matrix, that is, the risk identification parameter output result.

[0031] In one embodiment, if Figure 5 As shown, step S140 includes: S141, obtaining a ranking degree acquisition sub-strategy, a precision acquisition sub-strategy, a volatility determination sub-strategy, and a discrimination degree acquisition sub-strategy in the risk identification parameter acquisition strategy; S142, determining a ranking degree parameter corresponding to the evaluation result distribution matrix based on the ranking degree acquisition sub-strategy and the evaluation result distribution matrix; S143, determining an accuracy parameter corresponding to the evaluation result distribution matrix based on the accuracy acquisition sub-strategy and the evaluation result distribution matrix; S144, determining a volatility parameter corresponding to the evaluation result distribution matrix based on the volatility determination sub-strategy and the evaluation result distribution matrix; S145, determining an interval degree parameter corresponding to the evaluation result distribution matrix based on the discrimination acquisition sub-strategy and the evaluation result distribution matrix; S146. The risk identification parameter output result is composed of the ranking parameter, the accuracy parameter, the volatility parameter and the interval parameter.

[0032] In this embodiment, the risk identification parameter acquisition strategy includes at least a ranking degree acquisition sub-strategy, a precision acquisition sub-strategy, a volatility determination sub-strategy, and a discrimination degree acquisition sub-strategy, so as to analyze and verify the evaluation result distribution matrix from the four dimensions of ranking degree, precision, volatility, and interval degree, thereby obtaining the risk identification parameter output result. Through the above verification method, a more quantitative and intuitively displayable risk identification parameter output result is obtained.

[0033] In one embodiment, step S142 includes: For each first column vector in the evaluation result distribution matrix, based on the sorting degree acquisition sub-strategy, the difference between multiple groups of two upper and lower adjacent vector elements in the column vector is acquired in order from bottom to top, and the first total number of positive differences is counted; Obtain a first sum value obtained by summing the first total number corresponding to each first column vector in the evaluation result distribution matrix, and input the first sum value into the formula corresponding to the ranking degree acquisition sub-strategy to obtain the ranking degree parameter; wherein the formula corresponding to the ranking degree acquisition sub-strategy is ranking degree parameter = first sum value / [(the total number of rows in the evaluation result distribution matrix - 1) * the total number of columns in the evaluation result distribution matrix] * 100%; Step S143 includes: Obtaining the magnitude of each matrix element in the evaluation result distribution matrix exceeding a preset threshold and calculating the average value to obtain the accuracy parameter; Step S144 includes: For each first row vector in the evaluation result distribution matrix, obtain a first standard deviation and a first average value corresponding to all vector values ​​in the first row vector, and determine a first fluctuation degree value corresponding to the first row vector by a ratio between the first standard deviation and the first average value of the first row vector; Obtaining an average value corresponding to the first fluctuation degree value of each first row vector in the evaluation result distribution matrix as the volatility parameter; Step S145 includes: Based on the non-parametric statistical test model corresponding to the discrimination acquisition sub-strategy and the reference sample data, determine the matching result between the distribution result of the evaluation result distribution matrix and the distribution result of the reference sample data, and use it as the first interval degree sub-parameter value; wherein, when the distribution result of the evaluation result distribution matrix is ​​the same as the distribution result of the reference sample data, the first interval degree sub-parameter value is 1; when the distribution result of the evaluation result distribution matrix is ​​different from the distribution result of the reference sample data, the first interval degree sub-parameter value is less than 1; Based on the ROC curve area acquisition model corresponding to the discrimination acquisition sub-strategy, obtaining the second interval degree sub-parameter value corresponding to the result distribution matrix; The maximum value of the first interval degree sub-parameter value and the second interval degree sub-parameter value is obtained and used as the interval degree parameter.

[0034] In this embodiment, when determining the ranking degree parameter corresponding to the evaluation result distribution matrix based on the ranking degree acquisition sub-strategy, taking the first column vector of the first column in the evaluation result distribution matrix as an example, the first column vector includes N2 vector values, first obtain multiple groups of two adjacent vector elements in the first column vector of the first column in order from bottom to top, and subtract the vector element located above from the vector element located below the two adjacent vector elements to obtain N2-1 differences. Afterwards, the number of positive values ​​in the N2-1 differences is counted to obtain the first total number corresponding to the first column vector of the first column. Then, the first total number corresponding to the first column vector of the other columns is also obtained by referring to the above method. Finally, the first total number corresponding to each first column vector in the evaluation result distribution matrix is ​​summed to obtain the first sum value, and the first sum value is substituted into the ranking degree parameter = first sum value / [(the total number of rows of the evaluation result distribution matrix - 1) * the total number of columns of the evaluation result distribution matrix] * 100%, and the ranking degree parameter is calculated.

[0035] Based on the precision acquisition sub-strategy, the amplitude of each matrix element in the evaluation result distribution matrix exceeding the preset threshold can be obtained, that is, the amplitude of each matrix element in the evaluation result distribution matrix exceeding the preset threshold is determined by (matrix element-preset threshold) / preset threshold*100%, and then the average value corresponding to each amplitude is calculated to obtain the precision parameter.

[0036] When determining the volatility parameter corresponding to the evaluation result distribution matrix based on the volatility determination sub-strategy, take the first row vector of the first row in the evaluation result distribution matrix as an example, the first row vector includes N1+1 vector values, first determine the first standard deviation and the first average value based on the above N1+1 vector values, and determine the first volatility value corresponding to the first row vector of the first row by the ratio between the first standard deviation and the first average value of the first row vector of the first row. Afterwards, the process of obtaining the first volatility values ​​of other first row vectors also refers to the above process. Finally, after obtaining the first volatility values ​​of each first row vector in the evaluation result distribution matrix, the average value is calculated, which can be used as the volatility parameter.

[0037] When determining the discrimination parameter corresponding to the evaluation result distribution matrix based on the discrimination acquisition sub-strategy, the non-parametric statistical test model (such as the KS test model, i.e., the Kolmogorov-Smirnov test model) and the reference sample data corresponding to the discrimination acquisition sub-strategy can be first obtained, and then the matching result between the distribution result of the evaluation result distribution matrix and the distribution result of the reference sample data is determined based on the non-parametric statistical test model, and used as the first interval degree sub-parameter value; wherein, when the distribution result of the evaluation result distribution matrix is ​​different from the distribution result of the reference sample data, the greater the difference between the distribution results of the two, the smaller the corresponding first interval degree sub-parameter value is, but the value is always greater than 0. Afterwards, the second interval degree sub-parameter value corresponding to the result distribution matrix can also be obtained based on the ROC curve area acquisition model corresponding to the discrimination acquisition sub-strategy (the full name of the ROC curve is the Receiver Operating Characteristic Curve, which is a receiver operating characteristic curve); wherein the value range of the second interval degree sub-parameter value is 0-1, and the closer the second interval degree sub-parameter value is to 1, the better the classification result of the classification model is.

[0038] When the ranking parameter, the accuracy parameter, the volatility parameter and the interval parameter are respectively obtained in the above-mentioned specific manner, the risk identification parameter output result including the verification results of four dimensions can be formed.

[0039] S150: Based on a preset user structure parameter acquisition strategy and the evaluation result distribution matrix, determine a user structure parameter output result corresponding to the evaluation result distribution matrix.

[0040] In this embodiment, after determining the evaluation result distribution matrix, the matrix elements included therein can also be analyzed and evaluated in various dimensions from the perspective of data richness, data stability and data practicality, so as to obtain the verification result corresponding to the evaluation result distribution matrix in the dimension of user data structure, that is, the user structure parameter output result.

[0041] In one embodiment, if Figure 6 As shown, step S150 includes: S151, obtaining a richness acquisition sub-strategy, a stability acquisition sub-strategy, and a practicality determination sub-strategy in the user structure parameter acquisition strategy; S152, based on the richness acquisition sub-strategy and the evaluation result distribution matrix, determining a richness parameter corresponding to the evaluation result distribution matrix; S153, determining a stability parameter corresponding to the evaluation result distribution matrix based on the stability acquisition sub-strategy and the evaluation result distribution matrix; S154, determining a utility parameter corresponding to the evaluation result distribution matrix based on the utility determination sub-strategy and the evaluation result distribution matrix; S155. The user structure parameter output result is composed of the richness parameter, the stability parameter and the practicality parameter.

[0042] In this embodiment, the user structure parameter acquisition strategy includes at least a richness acquisition sub-strategy, a stability acquisition sub-strategy, and a practicality determination sub-strategy, so as to perform data analysis and verification on the evaluation result distribution matrix from the three dimensions of richness, stability, and practicality, thereby obtaining the user structure parameter output result. Through the above verification method, a more quantitative and intuitively displayed user structure parameter output result is obtained.

[0043] In one embodiment, step S152 includes: For each row vector in the evaluation result distribution matrix, obtain the target row vectors whose all vector values ​​in the row vector are 0 and the total number of the target row vectors; Obtaining the absolute value of the ratio of the total number of target row vectors to the total number of rows of the evaluation result distribution matrix minus one as the richness parameter; Step S153 includes: For each column vector in the evaluation result distribution matrix, the comparison result of the corresponding PSI deviation degree value and the preset cutoff value is obtained and the average value is calculated as the stability parameter; wherein the PSI deviation degree value is the population deviation degree index value; Step S154 includes: Determine a target weight vector based on the utility determination sub-strategy and the total number of columns of the evaluation result distribution matrix; The scalar product of the target weight vector and each column vector of the evaluation result distribution matrix is ​​obtained, and then the average value is calculated as the practicality parameter.

[0044] In this embodiment, when determining the richness parameter corresponding to the evaluation result distribution matrix based on the richness acquisition sub-strategy, each row vector in the evaluation result distribution matrix is ​​traversed, and the target row vector in which all vector values ​​in the row vector are 0 and the total number of target row vectors are first obtained. Then, the ratio of 1-the total number of target row vectors to the total number of rows of the evaluation result distribution matrix (that is, the absolute value of the ratio of the total number of target row vectors to the total number of rows of the evaluation result distribution matrix minus one) is calculated as the richness parameter.

[0045] When determining the stability parameter corresponding to the evaluation result distribution matrix based on the stability acquisition sub-strategy, taking the calculation of the PSI deviation value of a column vector in the evaluation result distribution matrix as an example, the PSI deviation value of a column vector is expressed as psij Indicates (the value range of j is 1 to N1+1, and j is a positive integer), where in psi j The corresponding comparison result sp when it is greater than the preset cutoff value (such as 0.1) j = 0, in psi j The corresponding comparison result sp when it is less than or equal to the preset cutoff value j =1-psi j / Preset cutoff value, after determining the comparison result corresponding to each row vector in the evaluation result distribution matrix, calculate the average value, and the obtained result can be used as the stability parameter.

[0046] When determining the practicality parameters corresponding to the evaluation result distribution matrix based on the practicality determination sub-strategy, first determine the target weight vector based on the total number of columns of the evaluation result distribution matrix (such as N1+1 in the above example), specifically, first determine whether N1+1 is an odd number or an even number, and when determining that N1+1 is an odd number, first determine the current rounding result of (N1+1) / 2 rounded down, the target weight vector = [current rounding result,... ,2,1,0,1,2, ..., current rounding result], wherein the number of vector values ​​included in the target weight vector is equal to the total number of columns of the evaluation result distribution matrix, the middle vector value is 0, and the two vector values ​​on the left and right of the middle vector value are both 1, and except for the first two vector values ​​which are both current rounding results, the remaining vector values ​​are all equal to 2. When determining that N1+1 is an even number, first determine the current calculation result of (N1+1) / 2, the target weight vector = [current calculation result, ..., 2,1,1,2, ..., current calculation result], where the number of vector values ​​included in the target weight vector is equal to the total number of columns of the evaluation result distribution matrix, the two middle vector values ​​are 1 (that is, the vector values ​​arranged at the (N1+1) / 2th and (N1+1) / 2+1th positions in the target weight vector are 1), and except for the first two vector values, which are both current calculation results, the remaining vector values ​​are all equal to 2. After determining to obtain the target weight vector, the dot product is calculated between it and each column vector of the evaluation result distribution matrix, and then the average value is calculated as the practicality parameter.

[0047] When the richness parameter, the stability parameter and the practicality parameter are respectively obtained through the above-mentioned specific method, the user structure parameter output result including the three-dimensional verification results can be formed.

[0048] S160. According to a preset output result fusion strategy, the risk identification parameter output result and the user structure parameter output result are fused to obtain a fused output result, and the fused output result is sent to a receiving terminal.

[0049] In this embodiment, the output result fusion strategy can be a weighted summation strategy, which can perform weighted summation on all parameters included in the risk identification parameter output result and the user structure parameter output result to obtain a fused output result, and send the fused output result to a receiving terminal. Of course, in specific implementation, the output result fusion strategy can also be to concentrate the risk identification parameter output result and the user structure parameter output result into the same output table or the same output area to obtain a fused output result, and the obtained fused output result can display the verification result of the evaluation result from more dimensions.

[0050] It can be seen that the implementation example of this method can input the user data set into the classification model to obtain the evaluation results and form an evaluation result distribution matrix, and then perform quantitative evaluation from the two dimensions of risk identification parameters and user structure parameters to obtain a fusion output result that can be visualized. This not only improves the efficiency of obtaining judgment results, but also ensures the reusability of the output result verification method.

[0051] Figure 7 is a schematic block diagram of a user risk assessment model output data verification device provided by an embodiment of the present invention. Figure 7 As shown, corresponding to the above user risk assessment model output data verification method, the present invention also provides a user risk assessment model output data verification device 100. The user risk assessment model output data verification device includes: a target user data set acquisition unit 110, an assessment result acquisition unit 120, an assessment result distribution matrix acquisition unit 130, a first output result acquisition unit 140, a second output result acquisition unit 150 and a verification result acquisition unit 160.

[0052] The target user data set acquisition unit 110 is configured to acquire, in response to a user data evaluation instruction, a target user data set corresponding to the user data evaluation instruction.

[0053] In this embodiment, the technical solution is described with the server as the execution subject. A user risk assessment model output data verification platform (also understood as a user risk assessment model output data verification system) is deployed in the server. When the user uses a receiving terminal (also understood as a user terminal, such as a smart phone, tablet computer, etc.) to log in to the user risk assessment model output data verification platform in the server and completes the communication connection with the server, the target user data set corresponding to the user data evaluation instruction can be determined first, and then the target user data set is used as the data to be analyzed for subsequent data processing.

[0054] In one embodiment, the target user data set acquisition unit 110 is used to: Obtaining the instruction generation time of the user data evaluation instruction, and determining the target time interval according to the current month and the number of preset months corresponding to the instruction generation time; The target user data set is acquired in a local user database based on the target time interval.

[0055] The starting month corresponding to the target time interval is the current month minus the number of preset months, and the ending month corresponding to the target time interval is the current month minus 1.

[0056] In this embodiment, as a preferred embodiment for determining the target time interval, the instruction generation time of the user data evaluation instruction can be obtained first, such as 12:00 on December 1, 20XX, and the corresponding current month is December 20XX; then, the preset number of months, such as 5, is obtained, and the current month is December 20XX and 5 months are calculated forward as the starting month corresponding to the target time interval, such as July 20XX, and the current month is December 20XX and 1 month is calculated forward as the ending month corresponding to the target time interval, such as November 20XX; finally, the target time interval is determined by taking the first day of the starting month corresponding to the target time interval as the starting date, and the last day of the ending month corresponding to the target time interval as the ending date. After determining the target time interval, it can be used as a screening condition to obtain the target user data set in the local user database. Through the above method, rapid screening and acquisition of the target user data set is achieved.

[0057] The evaluation result acquisition unit 120 is used to acquire the evaluation result corresponding to each target user data in the target user data set based on the pre-trained classification model.

[0058] The classification model is obtained by training the classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an evaluation result whose value is one of the values ​​in the preset value set.

[0059] In this embodiment, the preset scenario may be a financial technology scenario, a smart medical scenario, etc. In the server, the classification model to be trained may be obtained based on the user data training set of the preset scenario. More specifically, the classification model is a risk assessment model (such as a decision tree model, etc.). For example, 8 values ​​are set in the preset value set, and the target user data is input into the classification model to obtain a value that is one of the above 8 values.

[0060] The evaluation result distribution matrix acquisition unit 130 is used to determine the evaluation result distribution matrix corresponding to the target user data set based on a preset evaluation result distribution matrix acquisition strategy and the evaluation result corresponding to each target user data in the target user data set.

[0061] In this embodiment, after obtaining the evaluation results corresponding to each target user data in the target user data set, the distribution of the evaluation results can be statistically analyzed. For example, still referring to the above example, if July 20XX corresponds to month interval 1, August 20XX corresponds to month interval 2, September 20XX corresponds to month interval 3, October 20XX corresponds to month interval 4, and November 20XX corresponds to month interval 5, there is a target user data corresponding to a user data generation time that belongs to the month interval 1 corresponding to July 20XX, and its corresponding evaluation result is value 1 of the 8 values ​​in the preset value set, then the target user data can be counted once by the cumulative number of matrix elements in the 1st row and 1st column of the initial evaluation result distribution matrix (i.e., +1 processing is performed). Afterwards, when the user data generation time corresponding to other target user data belongs to the month interval ii (the value range of ii is 1 to 5) and the evaluation result is equal to the jjth value in the preset value set (the value range of jj is 1 to 8), it is counted once by the cumulative number of matrix elements in the iith row and jjth column in the initial evaluation result distribution matrix. In the above manner, the evaluation result distribution matrix can be quickly constructed by combining the evaluation results corresponding to each target user data in the target user data set.

[0062] In one embodiment, the evaluation result distribution matrix acquisition unit 130 is used to: Construct an initial evaluation result distribution matrix; wherein the total number of columns of the initial evaluation result distribution matrix is ​​equal to the total number of preset data generation time division intervals plus one, and the total number of rows of the initial evaluation result distribution matrix is ​​equal to the total number of values ​​included in the preset value set; the column vector of the last column in the initial evaluation result distribution matrix is ​​a statistical column vector stored locally, and the total number of vector values ​​included in the statistical column vector is equal to the total number of values ​​included in the preset value set; Based on the evaluation result distribution matrix acquisition strategy, each target user data in the target user data set is divided into a value interval based on the user data generation time and the evaluation result, and the corresponding target user data is counted by the corresponding matrix elements in the initial evaluation result distribution matrix, and the percentage ratio is calculated with the total number of target user data in the target user data set to obtain the evaluation result distribution matrix.

[0063] In this embodiment, when constructing the initial evaluation result distribution matrix, if the total number of preset data generation time division intervals is N1 (N1 is a positive integer, such as N1=5 in the above example) and the total number of values ​​included in the preset value set is N2 (N2 is a positive integer, such as N2=8 in the above example), then the initial evaluation result distribution matrix is ​​constructed as a matrix with N2 rows and N1+1 columns. It should be noted that the statistical column vectors stored locally on the server are obtained by first inputting all current user data in the server into the classification model to obtain the corresponding evaluation results, and then for each evaluation result, the corresponding target value in the preset value set is determined, so as to obtain the total number of evaluation results corresponding to each value in the preset value set, and then divide it by the total number of all current user data, so as to obtain the proportion of the number of data corresponding to each value in the preset value set, thereby forming a statistical column vector.

[0064] Afterwards, for each matrix element of the first N1 columns and N2 rows of the initial evaluation result distribution matrix, the corresponding target user data in the target user data set is counted based on the user data generation time and the value interval to which the evaluation result belongs by the corresponding matrix element in the initial evaluation result distribution matrix, and then divided by the total number of target user data in the target user data set to obtain the corresponding percentage ratio, thereby obtaining the evaluation result distribution matrix. It can be seen that through the above method, the evaluation result distribution matrix can be quickly constructed by combining the evaluation result corresponding to each target user data in the target user data set, the statistical column vector and the evaluation result distribution matrix acquisition strategy.

[0065] The first output result acquisition unit 140 is used to determine the risk identification parameter output result corresponding to the assessment result distribution matrix based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix.

[0066] In this embodiment, after determining the evaluation result distribution matrix, the matrix elements included therein can be analyzed and evaluated in various dimensions from both rows and columns to obtain the verification result corresponding to the risk identification dimension of the evaluation result distribution matrix, that is, the risk identification parameter output result.

[0067] In one embodiment, the first output result obtaining unit 140 is used to: Obtaining the ranking degree acquisition sub-strategy, the accuracy acquisition sub-strategy, the volatility determination sub-strategy and the discrimination degree acquisition sub-strategy in the risk identification parameter acquisition strategy; Based on the ranking degree acquisition sub-strategy and the evaluation result distribution matrix, determining a ranking degree parameter corresponding to the evaluation result distribution matrix; Based on the accuracy acquisition sub-strategy and the evaluation result distribution matrix, determining an accuracy parameter corresponding to the evaluation result distribution matrix; Based on the volatility determination sub-strategy and the evaluation result distribution matrix, determining a volatility parameter corresponding to the evaluation result distribution matrix; Based on the discrimination acquisition sub-strategy and the evaluation result distribution matrix, determining an interval degree parameter corresponding to the evaluation result distribution matrix; The risk identification parameter output result is composed of the ranking parameter, the accuracy parameter, the volatility parameter and the interval parameter.

[0068] In this embodiment, the risk identification parameter acquisition strategy includes at least a ranking degree acquisition sub-strategy, a precision acquisition sub-strategy, a volatility determination sub-strategy, and a discrimination degree acquisition sub-strategy, so as to analyze and verify the evaluation result distribution matrix from the four dimensions of ranking degree, precision, volatility, and interval degree, thereby obtaining the risk identification parameter output result. Through the above verification method, a more quantitative and intuitively displayable risk identification parameter output result is obtained.

[0069] In one embodiment, the obtaining sub-strategy based on the ranking degree and the evaluation result distribution matrix, determining the ranking degree parameter corresponding to the evaluation result distribution matrix, includes: For each first column vector in the evaluation result distribution matrix, based on the sorting degree acquisition sub-strategy, the difference between multiple groups of two upper and lower adjacent vector elements in the column vector is acquired in order from bottom to top, and the first total number of positive differences is counted; Obtain a first sum value obtained by summing the first total number corresponding to each first column vector in the evaluation result distribution matrix, and input the first sum value into the formula corresponding to the ranking degree acquisition sub-strategy to obtain the ranking degree parameter; wherein the formula corresponding to the ranking degree acquisition sub-strategy is ranking degree parameter = first sum value / [(the total number of rows in the evaluation result distribution matrix - 1) * the total number of columns in the evaluation result distribution matrix] * 100%; The step of determining the accuracy parameter corresponding to the evaluation result distribution matrix based on the accuracy acquisition sub-strategy and the evaluation result distribution matrix includes: Obtaining the magnitude of each matrix element in the evaluation result distribution matrix exceeding a preset threshold and calculating the average value to obtain the accuracy parameter; The step of determining the volatility parameter corresponding to the evaluation result distribution matrix based on the volatility determination sub-strategy and the evaluation result distribution matrix includes: For each first row vector in the evaluation result distribution matrix, obtain a first standard deviation and a first average value corresponding to all vector values ​​in the first row vector, and determine a first fluctuation degree value corresponding to the first row vector by a ratio between the first standard deviation and the first average value of the first row vector; Obtaining an average value corresponding to the first fluctuation degree value of each first row vector in the evaluation result distribution matrix as the volatility parameter; The obtaining sub-strategy based on the discrimination and the evaluation result distribution matrix, determining the interval degree parameter corresponding to the evaluation result distribution matrix, comprises: Based on the non-parametric statistical test model corresponding to the discrimination acquisition sub-strategy and the reference sample data, determine the matching result between the distribution result of the evaluation result distribution matrix and the distribution result of the reference sample data, and use it as the first interval degree sub-parameter value; wherein, when the distribution result of the evaluation result distribution matrix is ​​the same as the distribution result of the reference sample data, the first interval degree sub-parameter value is 1; when the distribution result of the evaluation result distribution matrix is ​​different from the distribution result of the reference sample data, the first interval degree sub-parameter value is less than 1; Based on the ROC curve area acquisition model corresponding to the discrimination acquisition sub-strategy, obtaining the second interval degree sub-parameter value corresponding to the result distribution matrix; The maximum value of the first interval degree sub-parameter value and the second interval degree sub-parameter value is obtained and used as the interval degree parameter.

[0070] In this embodiment, when determining the ranking degree parameter corresponding to the evaluation result distribution matrix based on the ranking degree acquisition sub-strategy, taking the first column vector of the first column in the evaluation result distribution matrix as an example, the first column vector includes N2 vector values, first obtain multiple groups of two adjacent vector elements in the first column vector of the first column in order from bottom to top, and subtract the vector element located above from the vector element located below the two adjacent vector elements to obtain N2-1 differences. Afterwards, the number of positive values ​​in the N2-1 differences is counted to obtain the first total number corresponding to the first column vector of the first column. Then, the first total number corresponding to the first column vector of the other columns is also obtained by referring to the above method. Finally, the first total number corresponding to each first column vector in the evaluation result distribution matrix is ​​summed to obtain the first sum value, and the first sum value is substituted into the ranking degree parameter = first sum value / [(the total number of rows of the evaluation result distribution matrix - 1) * the total number of columns of the evaluation result distribution matrix] * 100%, and the ranking degree parameter is calculated.

[0071] Based on the precision acquisition sub-strategy, the amplitude by which each matrix element in the evaluation result distribution matrix exceeds the preset threshold can be obtained, that is, the amplitude by which each matrix element in the above evaluation result distribution matrix exceeds the preset threshold is determined by (matrix element-preset threshold) / preset threshold*100%, and then the average value corresponding to each amplitude is calculated to obtain the precision parameter.

[0072] When determining the volatility parameter corresponding to the evaluation result distribution matrix based on the volatility determination sub-strategy, take the first row vector of the first row in the evaluation result distribution matrix as an example, the first row vector includes N1+1 vector values, first determine the first standard deviation and the first average value based on the above N1+1 vector values, and determine the first volatility value corresponding to the first row vector of the first row by the ratio between the first standard deviation and the first average value of the first row vector of the first row. Afterwards, the process of obtaining the first volatility values ​​of other first row vectors also refers to the above process. Finally, after obtaining the first volatility values ​​of each first row vector in the evaluation result distribution matrix, the average value is calculated, which can be used as the volatility parameter.

[0073] When determining the discrimination parameter corresponding to the evaluation result distribution matrix based on the discrimination acquisition sub-strategy, the non-parametric statistical test model (such as the KS test model, i.e., the Kolmogorov-Smirnov test model) and the reference sample data corresponding to the discrimination acquisition sub-strategy can be first obtained, and then the matching result between the distribution result of the evaluation result distribution matrix and the distribution result of the reference sample data is determined based on the non-parametric statistical test model, and used as the first interval degree sub-parameter value; wherein, when the distribution result of the evaluation result distribution matrix is ​​different from the distribution result of the reference sample data, the greater the difference between the distribution results of the two, the smaller the corresponding first interval degree sub-parameter value is, but the value is always greater than 0. Afterwards, the second interval degree sub-parameter value corresponding to the result distribution matrix can also be obtained based on the ROC curve area acquisition model corresponding to the discrimination acquisition sub-strategy (the full name of the ROC curve is the Receiver Operating Characteristic Curve, which is a receiver operating characteristic curve); wherein the value range of the second interval degree sub-parameter value is 0-1, and the closer the second interval degree sub-parameter value is to 1, the better the classification result of the classification model is.

[0074] When the ranking parameter, the accuracy parameter, the volatility parameter and the interval parameter are respectively obtained in the above-mentioned specific manner, the risk identification parameter output result including the verification results of four dimensions can be formed.

[0075] The second output result acquisition unit 150 is used to determine the user structure parameter output result corresponding to the evaluation result distribution matrix based on a preset user structure parameter acquisition strategy and the evaluation result distribution matrix.

[0076] In this embodiment, after determining the evaluation result distribution matrix, the matrix elements included therein can also be analyzed and evaluated in various dimensions from the perspective of data richness, data stability and data practicality, so as to obtain the verification result corresponding to the evaluation result distribution matrix in the dimension of user data structure, that is, the user structure parameter output result.

[0077] In one embodiment, the second output result obtaining unit 150 is used to: Obtaining a richness acquisition sub-strategy, a stability acquisition sub-strategy, and a practicality determination sub-strategy in the user structure parameter acquisition strategy; Based on the richness acquisition sub-strategy and the evaluation result distribution matrix, determining a richness parameter corresponding to the evaluation result distribution matrix; Based on the stability acquisition sub-strategy and the evaluation result distribution matrix, determining a stability parameter corresponding to the evaluation result distribution matrix; Based on the utility determination sub-strategy and the evaluation result distribution matrix, determining a utility parameter corresponding to the evaluation result distribution matrix; The user structure parameter output result is composed of the richness parameter, the stability parameter and the practicality parameter.

[0078] In this embodiment, the user structure parameter acquisition strategy includes at least a richness acquisition sub-strategy, a stability acquisition sub-strategy, and a practicality determination sub-strategy, so as to perform data analysis and verification on the evaluation result distribution matrix from the three dimensions of richness, stability, and practicality, thereby obtaining the user structure parameter output result. Through the above verification method, a more quantitative and intuitively displayed user structure parameter output result is obtained.

[0079] In one embodiment, the determining, based on the richness acquisition sub-strategy and the evaluation result distribution matrix, a richness parameter corresponding to the evaluation result distribution matrix includes: For each row vector in the evaluation result distribution matrix, obtain the target row vectors whose all vector values ​​in the row vector are 0 and the total number of the target row vectors; Obtaining the absolute value of the ratio of the total number of target row vectors to the total number of rows of the evaluation result distribution matrix minus one as the richness parameter; The determining, based on the stability acquisition sub-strategy and the evaluation result distribution matrix, a stability parameter corresponding to the evaluation result distribution matrix comprises: For each column vector in the evaluation result distribution matrix, the comparison result of the corresponding PSI deviation degree value and the preset cutoff value is obtained and the average value is calculated as the stability parameter; wherein the PSI deviation degree value is the population deviation degree index value; The determining of the utility parameter corresponding to the evaluation result distribution matrix based on the utility determination sub-strategy and the evaluation result distribution matrix includes: Determine a target weight vector based on the utility determination sub-strategy and the total number of columns of the evaluation result distribution matrix; The scalar product of the target weight vector and each column vector of the evaluation result distribution matrix is ​​obtained, and then the average value is calculated as the practicality parameter.

[0080] In this embodiment, when determining the richness parameter corresponding to the evaluation result distribution matrix based on the richness acquisition sub-strategy, each row vector in the evaluation result distribution matrix is ​​traversed, and the target row vector in which all vector values ​​in the row vector are 0 and the total number of target row vectors are first obtained. Then, the ratio of 1-the total number of target row vectors to the total number of rows of the evaluation result distribution matrix (that is, the absolute value of the ratio of the total number of target row vectors to the total number of rows of the evaluation result distribution matrix minus one) is calculated as the richness parameter.

[0081] When determining the stability parameter corresponding to the evaluation result distribution matrix based on the stability acquisition sub-strategy, taking the calculation of the PSI deviation value of a column vector in the evaluation result distribution matrix as an example, the PSI deviation value of a column vector is expressed as psi j Indicates (the value range of j is 1 to N1+1, and j is a positive integer), where in psi j The corresponding comparison result sp when it is greater than the preset cutoff value (such as 0.1) j = 0, in psi j The corresponding comparison result sp when it is less than or equal to the preset cutoff value j =1-psi j / Preset cutoff value, after determining the comparison result corresponding to each row vector in the evaluation result distribution matrix, calculate the average value, and the obtained result can be used as the stability parameter.

[0082] When determining the practicality parameters corresponding to the evaluation result distribution matrix based on the practicality determination sub-strategy, first determine the target weight vector based on the total number of columns of the evaluation result distribution matrix (such as N1+1 in the above example), specifically, first determine whether N1+1 is an odd number or an even number, and when determining that N1+1 is an odd number, first determine the current rounding result of (N1+1) / 2 rounded down, the target weight vector = [current rounding result,... ,2,1,0,1,2, ..., current rounding result], wherein the number of vector values ​​included in the target weight vector is equal to the total number of columns of the evaluation result distribution matrix, the middle vector value is 0, and the two vector values ​​on the left and right of the middle vector value are both 1, and except for the first two vector values ​​which are both current rounding results, the remaining vector values ​​are all equal to 2. When it is determined that N1+1 is an even number, the current calculation result of (N1+1) / 2 is first determined, and the target weight vector = [current calculation result, ..., 2, 1, 1, 2, ..., current calculation result], where the number of vector values ​​included in the target weight vector is equal to the total number of columns of the evaluation result distribution matrix, and the two middle vector values ​​are 1 (that is, the vector values ​​arranged at the (N1+1) / 2th and (N1+1) / 2+1th positions in the target weight vector are 1), and except for the first two vector values, which are both current calculation results, the remaining vector values ​​are all equal to 2. After determining to obtain the target weight vector, the dot product is calculated between it and each column vector of the evaluation result distribution matrix, and then the average value is calculated as the practicality parameter.

[0083] When the richness parameter, the stability parameter and the practicality parameter are respectively obtained through the above-mentioned specific method, the user structure parameter output result including the three-dimensional verification results can be formed.

[0084] The verification result acquisition unit 160 is used to fuse the risk identification parameter output result and the user structure parameter output result according to a preset output result fusion strategy to obtain a fused output result, and send the fused output result to a receiving terminal.

[0085] In this embodiment, the output result fusion strategy can be a weighted summation strategy, which can perform weighted summation on all parameters included in the risk identification parameter output result and the user structure parameter output result to obtain a fused output result, and send the fused output result to a receiving terminal. Of course, in specific implementation, the output result fusion strategy can also be to concentrate the risk identification parameter output result and the user structure parameter output result into the same output table or the same output area to obtain a fused output result, and the obtained fused output result can display the verification result of the evaluation result from more dimensions.

[0086] It can be seen that the implementation example of the device can input the user data set into the classification model to obtain the evaluation results and form an evaluation result distribution matrix, and then perform quantitative evaluation from the two dimensions of risk identification parameters and user structure parameters to obtain a fusion output result that can be visualized, which not only improves the efficiency of obtaining judgment results, but also ensures the reusability of the output result verification method.

[0087] The above-mentioned user risk assessment model output data verification device can be implemented in the form of a computer program. The computer program can be used in Figure 8 Runs on the computer device shown.

[0088] See also Figure 8 , Figure 8 The schematic block diagram of a computer device provided by an embodiment of the present invention is shown in FIG. The computer device integrates any user risk assessment model output data verification device provided by an embodiment of the present invention.

[0089] See also Figure 8 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .

[0090] The storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute the above-mentioned user risk assessment model output data verification method.

[0091] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0092] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned user risk assessment model output data verification method.

[0093] The network interface 405 is used to communicate with other devices over the network. Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] The processor 402 is used to run the computer program 4032 stored in the memory to implement the above-mentioned method for verifying the output data of the user risk assessment model.

[0095] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0096] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0097] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the above-mentioned user risk assessment model output data verification method.

[0098] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0100] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0101] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0103] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for verifying output data of a user risk assessment model, characterized in that: include: In response to a user data evaluation instruction, acquiring a target user data set corresponding to the user data evaluation instruction; Obtaining an evaluation result corresponding to each target user data in the target user data set based on a pre-trained classification model; wherein the classification model is obtained by training the classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an evaluation result having a value of one of a preset value set; Determine an evaluation result distribution matrix corresponding to the target user data set based on a preset evaluation result distribution matrix acquisition strategy and the evaluation result corresponding to each target user data in the target user data set; Based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix, determining a risk identification parameter output result corresponding to the assessment result distribution matrix; Based on a preset user structure parameter acquisition strategy and the evaluation result distribution matrix, determining a user structure parameter output result corresponding to the evaluation result distribution matrix; and The risk identification parameter output result and the user structure parameter output result are fused according to a preset output result fusion strategy to obtain a fused output result, and the fused output result is sent to a receiving terminal.

2. The method according to claim 1, characterized in that The step of determining the evaluation result distribution matrix corresponding to the target user data set based on the preset evaluation result distribution matrix acquisition strategy and the evaluation result corresponding to each target user data in the target user data set includes: Construct an initial evaluation result distribution matrix; wherein the total number of columns of the initial evaluation result distribution matrix is ​​equal to the total number of preset data generation time division intervals plus one, and the total number of rows of the initial evaluation result distribution matrix is ​​equal to the total number of values ​​included in the preset value set; the column vector of the last column in the initial evaluation result distribution matrix is ​​a statistical column vector stored locally, and the total number of vector values ​​included in the statistical column vector is equal to the total number of values ​​included in the preset value set; Based on the evaluation result distribution matrix acquisition strategy, each target user data in the target user data set is divided into a value interval based on the user data generation time and the evaluation result, and the corresponding target user data is counted by the corresponding matrix elements in the initial evaluation result distribution matrix, and the percentage ratio is calculated with the total number of target user data in the target user data set to obtain the evaluation result distribution matrix.

3. The method according to claim 1, characterized in that The step of determining the risk identification parameter output result corresponding to the assessment result distribution matrix based on the preset risk identification parameter acquisition strategy and the assessment result distribution matrix includes: Obtaining the ranking degree acquisition sub-strategy, the accuracy acquisition sub-strategy, the volatility determination sub-strategy and the discrimination degree acquisition sub-strategy in the risk identification parameter acquisition strategy; Based on the ranking degree acquisition sub-strategy and the evaluation result distribution matrix, determining a ranking degree parameter corresponding to the evaluation result distribution matrix; Based on the accuracy acquisition sub-strategy and the evaluation result distribution matrix, determining an accuracy parameter corresponding to the evaluation result distribution matrix; Based on the volatility determination sub-strategy and the evaluation result distribution matrix, determining a volatility parameter corresponding to the evaluation result distribution matrix; Based on the discrimination acquisition sub-strategy and the evaluation result distribution matrix, determining an interval degree parameter corresponding to the evaluation result distribution matrix; The risk identification parameter output result is composed of the ranking parameter, the accuracy parameter, the volatility parameter and the interval parameter.

4. The method according to claim 3, characterized in that The obtaining sub-strategy based on the ranking degree and the evaluation result distribution matrix, determining the ranking degree parameter corresponding to the evaluation result distribution matrix, comprises: For each first column vector in the evaluation result distribution matrix, based on the sorting degree acquisition sub-strategy, the difference between multiple groups of two upper and lower adjacent vector elements in the column vector is acquired in order from bottom to top, and the first total number of positive differences is counted; Obtain a first sum value obtained by summing the first total number corresponding to each first column vector in the evaluation result distribution matrix, and input the first sum value into the formula corresponding to the ranking degree acquisition sub-strategy to obtain the ranking degree parameter; wherein the formula corresponding to the ranking degree acquisition sub-strategy is ranking degree parameter = first sum value / [(the total number of rows in the evaluation result distribution matrix - 1) * the total number of columns in the evaluation result distribution matrix] * 100%; The step of determining the accuracy parameter corresponding to the evaluation result distribution matrix based on the accuracy acquisition sub-strategy and the evaluation result distribution matrix includes: Obtaining the magnitude of each matrix element in the evaluation result distribution matrix exceeding a preset threshold and calculating the average value to obtain the accuracy parameter; The step of determining the volatility parameter corresponding to the evaluation result distribution matrix based on the volatility determination sub-strategy and the evaluation result distribution matrix includes: For each first row vector in the evaluation result distribution matrix, obtain a first standard deviation and a first average value corresponding to all vector values ​​in the first row vector, and determine a first fluctuation degree value corresponding to the first row vector by a ratio between the first standard deviation and the first average value of the first row vector; Obtaining an average value corresponding to the first fluctuation degree value of each first row vector in the evaluation result distribution matrix as the volatility parameter; The obtaining sub-strategy based on the discrimination and the evaluation result distribution matrix, determining the interval degree parameter corresponding to the evaluation result distribution matrix, comprises: Based on the non-parametric statistical test model corresponding to the discrimination acquisition sub-strategy and the reference sample data, determine the matching result between the distribution result of the evaluation result distribution matrix and the distribution result of the reference sample data, and use it as the first interval degree sub-parameter value; wherein, when the distribution result of the evaluation result distribution matrix is ​​the same as the distribution result of the reference sample data, the first interval degree sub-parameter value is 1; when the distribution result of the evaluation result distribution matrix is ​​different from the distribution result of the reference sample data, the first interval degree sub-parameter value is less than 1; Based on the ROC curve area acquisition model corresponding to the discrimination acquisition sub-strategy, obtaining the second interval degree sub-parameter value corresponding to the result distribution matrix; The maximum value of the first interval degree sub-parameter value and the second interval degree sub-parameter value is obtained and used as the interval degree parameter.

5. The method according to claim 1, characterized in that The step of determining the user structure parameter output result corresponding to the evaluation result distribution matrix based on the preset user structure parameter acquisition strategy and the evaluation result distribution matrix includes: Obtaining a richness acquisition sub-strategy, a stability acquisition sub-strategy, and a practicality determination sub-strategy in the user structure parameter acquisition strategy; Based on the richness acquisition sub-strategy and the evaluation result distribution matrix, determining a richness parameter corresponding to the evaluation result distribution matrix; Based on the stability acquisition sub-strategy and the evaluation result distribution matrix, determining a stability parameter corresponding to the evaluation result distribution matrix; Based on the utility determination sub-strategy and the evaluation result distribution matrix, determining a utility parameter corresponding to the evaluation result distribution matrix; The user structure parameter output result is composed of the richness parameter, the stability parameter and the practicality parameter.

6. The method according to claim 5, characterized in that The determining, based on the richness acquisition sub-strategy and the evaluation result distribution matrix, a richness parameter corresponding to the evaluation result distribution matrix comprises: For each row vector in the evaluation result distribution matrix, obtain the target row vectors whose all vector values ​​in the row vector are 0 and the total number of the target row vectors; Obtaining the absolute value of the ratio of the total number of target row vectors to the total number of rows of the evaluation result distribution matrix minus one as the richness parameter; The determining, based on the stability acquisition sub-strategy and the evaluation result distribution matrix, a stability parameter corresponding to the evaluation result distribution matrix comprises: For each column vector in the evaluation result distribution matrix, the comparison result of the corresponding PSI deviation degree value and the preset cutoff value is obtained and the average value is calculated as the stability parameter; wherein the PSI deviation degree value is the population deviation degree index value; The determining of the utility parameter corresponding to the evaluation result distribution matrix based on the utility determination sub-strategy and the evaluation result distribution matrix includes: Determine a target weight vector based on the utility determination sub-strategy and the total number of columns of the evaluation result distribution matrix; The scalar product of the target weight vector and each column vector of the evaluation result distribution matrix is ​​obtained, and then the average value is calculated as the practicality parameter.

7. The method according to claim 1, characterized in that The acquiring of a target user data set corresponding to the user data evaluation instruction includes: Obtaining the instruction generation time of the user data evaluation instruction, and determining the target time interval according to the current month and the preset number of months corresponding to the instruction generation time; wherein the starting month corresponding to the target time interval is the current month minus the preset number of months, and the ending month corresponding to the target time interval is the current month minus 1; The target user data set is acquired in a local user database based on the target time interval.

8. A user risk assessment model output data verification device, characterized in that: include: a target user data set acquisition unit, configured to acquire, in response to a user data evaluation instruction, a target user data set corresponding to the user data evaluation instruction; An evaluation result acquisition unit is used to acquire an evaluation result corresponding to each target user data in the target user data set based on a pre-trained classification model; wherein the classification model is obtained by training a classification model to be trained based on a user data training set of a preset scenario, and the target user data is input into the classification model to obtain an evaluation result whose value is one of a preset value set; An evaluation result distribution matrix acquisition unit, configured to determine an evaluation result distribution matrix corresponding to the target user data set based on a preset evaluation result distribution matrix acquisition strategy and an evaluation result corresponding to each target user data in the target user data set; A first output result acquisition unit, configured to determine a risk identification parameter output result corresponding to the assessment result distribution matrix based on a preset risk identification parameter acquisition strategy and the assessment result distribution matrix; A second output result acquisition unit, configured to determine a user structure parameter output result corresponding to the evaluation result distribution matrix based on a preset user structure parameter acquisition strategy and the evaluation result distribution matrix; and The verification result acquisition unit is used to fuse the risk identification parameter output result and the user structure parameter output result according to a preset output result fusion strategy to obtain a fused output result, and send the fused output result to a receiving terminal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the user risk assessment model output data verification method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the method for verifying the output data of a user risk assessment model as described in any one of claims 1 to 7 can be implemented.

Images