Evaluation information determination method, device, apparatus, and computer storage medium

Abstract

The application discloses an evaluation information determination method and device, equipment and a computer storage medium, which can train a preset neural network through user historical service data, obtain a first risk identification model and first quality evaluation information of the model. Then, service feature information is generated by using a target model to be evaluated, and the neural network is trained again through the service feature information and the user historical service data. Thus, after retraining by introducing new feature information (i.e., service feature information), a second risk identification model and corresponding second quality evaluation information can be obtained. Then, according to the comparison result of the two quality evaluation information, it can be reflected whether the training degree of the risk identification model obtained by training twice is improved, so as to quickly verify and evaluate whether the target model (i.e., the matter identification model) is effective, and the reliability of the matter identification model in the application in the risk matter identification scene can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of finance, and particularly relates to an evaluation information determination method and device, equipment and a computer storage medium. BACKGROUND

[0002] Generally, financial institutions identify various risk events in the public credit scenario through a credit management business system. However, the causes of credit risks are complex, which leads to a large number of credit risk investigation items generated by the credit management business system, and all of them need to be manually identified, judged and finally audited and confirmed by credit practitioners and credit managers for the authenticity and effectiveness of the risk items. If the business system has insufficient identification capability, a large number of invalid risk items will be identified, and a large amount of manpower will be consumed for review, which is high in labor cost, low in efficiency and low in reliability. SUMMARY

[0003] The embodiments of the present application provide an evaluation information determination method, device, equipment and computer storage medium, which can improve the reliability of risk item identification in the credit management scenario.

[0004] The embodiments of the first aspect of the present application provide an evaluation information determination method, which comprises:

[0005] Obtaining user historical business data;

[0006] Training a preset neural network through the user historical business data to obtain a first risk identification model and first quality evaluation information of the first risk identification model;

[0007] Generating business feature information using a target model to be evaluated;

[0008] Training the neural network through the business feature information and the user historical business data to obtain a second risk identification model and second quality evaluation information of the second risk identification model;

[0009] Determining target evaluation information of the target model according to a comparison result of the first quality evaluation information and the second quality evaluation information.

[0010] In the embodiments of the first aspect of the present application, the user historical business data corresponds to multiple feature dimensions, and the first quality evaluation information includes a first accuracy rate and / or a first recall rate.

[0011] Training the preset neural network through the user historical business data to obtain the first risk identification model and the first quality evaluation information of the first risk identification model comprises:

[0012] Inputting the user historical business data into the neural network;

[0013] The neural network is trained according to the plurality of feature dimensions, to obtain the first risk identification model and first output information of the first risk identification model;

[0014] The first accuracy and / or the first recall rate are obtained according to the first output information.

[0015] In the embodiment of the first aspect of the present application, the target model is a transaction identification model, and the business feature information is generated by using the target model to be evaluated, including:

[0016] The business feature information is generated according to at least one transaction feature of the transaction identification model,

[0017] The feature dimension corresponding to the business feature information is different from the plurality of feature dimensions of the user historical business data.

[0018] In the embodiment of the first aspect of the present application, the second quality evaluation information includes a second accuracy and / or a second recall rate,

[0019] The neural network is trained by using the business feature information and the user historical business data, to obtain the second risk identification model and the second quality evaluation information of the second risk identification model, including:

[0020] The user historical business data is input into the neural network;

[0021] The neural network is trained according to the feature dimension corresponding to the business feature information and the plurality of feature dimensions corresponding to the user historical business data, to obtain the second risk identification model and second output information of the second risk identification model;

[0022] The second accuracy and / or the second recall rate are obtained according to the second output information.

[0023] In the embodiment of the first aspect of the present application, the target model is a transaction identification model, and after the target evaluation information of the target model, the method further includes:

[0024] In the case where the target evaluation information meets a preset condition, the user business data to be identified is obtained;

[0025] The transaction identification information is obtained by performing feature extraction on the user business data to be identified by using the transaction identification model.

[0026] The embodiment of the second aspect of the present application provides an evaluation information determination device, and the device includes:

[0027] The first acquisition module is configured to acquire user historical business data;

[0028] The first training module is configured to train a preset neural network by using the user historical service data to obtain a first risk identification model and first quality evaluation information of the first risk identification model.

[0029] The generation module is configured to generate service feature information by using the target model to be evaluated.

[0030] The second training module is configured to train the neural network by using the service feature information and the user historical service data to obtain a second risk identification model and second quality evaluation information of the second risk identification model.

[0031] The determination module is configured to determine target evaluation information of the target model according to the first quality evaluation information and the second quality evaluation information.

[0032] In the embodiment of the second aspect of the application, the user historical service data corresponds to a plurality of feature dimensions, and the first quality evaluation information includes a first accuracy rate and / or a first recall rate.

[0033] The first training module includes:

[0034] The first input submodule is configured to input the user historical service data into the neural network.

[0035] The first training submodule is configured to train the neural network according to the plurality of feature dimensions to obtain the first risk identification model and first output information of the first risk identification model.

[0036] The first calculation submodule is configured to calculate according to the first output information to obtain the first accuracy rate and / or the first recall rate.

[0037] In the embodiment of the second aspect of the application, the target model is a transaction identification model, and the generation module is specifically configured to:

[0038] generate the service feature information according to at least one transaction feature of the transaction identification model,

[0039] wherein the feature dimension corresponding to the service feature information is different from the plurality of feature dimensions of the user historical service data.

[0040] In the embodiment of the second aspect of the application, the second quality evaluation information includes a second accuracy rate and / or a second recall rate,

[0041] The second training module includes:

[0042] The second input submodule is configured to input the user historical service data into the neural network.

[0043] The second training submodule is configured to train the neural network according to the feature dimension corresponding to the service feature information and the plurality of feature dimensions corresponding to the user historical service data, to obtain the second risk identification model and second output information of the second risk identification model.

[0044] The second calculation submodule is configured to calculate according to the second output information to obtain the second accuracy rate and / or the second recall rate.

[0045] In the embodiments of the second aspect of the present application, the target model is a transaction identification model, and the device further comprises:

[0046] The second acquisition module is configured to acquire the user service data to be identified in a case where the target evaluation information meets a preset condition.

[0047] The extraction module is configured to perform feature extraction on the user service data to be identified by the transaction identification model to obtain transaction identification information.

[0048] The third aspect of the present application provides a computer device, the device comprising: a processor and a memory storing computer program instructions;

[0049] The processor executes the computer program instructions to realize the evaluation information determination method of any one of the embodiments of the first aspect.

[0050] The fourth aspect of the present application provides a computer storage medium, the computer storage medium storing computer program instructions, and the computer program instructions are executed by the processor to realize the evaluation information determination method of any one of the embodiments of the first aspect.

[0051] The fifth aspect of the present application provides a computer program product, and instructions in the computer program product are executed by the processor of the computer device to enable the computer device to execute the evaluation information determination method of any one of the embodiments of the first aspect.

[0052] The evaluation information determination method, device, equipment and computer storage medium provided by the embodiments of the present application can train a preset neural network through user historical service data, obtain a first risk identification model and first quality evaluation information of the model. Then, service feature information is generated by using a target model to be evaluated, and the neural network is trained again through the service feature information and the user historical service data. In this way, after retraining by introducing new feature information (i.e., service feature information), a second risk identification model and corresponding second quality evaluation information can be obtained. Then, according to the comparison result of the two quality evaluation information, it can be reflected whether the training degree of the risk identification model obtained by twice training is improved. In this way, it can be quickly verified and evaluated whether the target model (i.e., the transaction identification model) is effective, and the reliability of the transaction identification model in the risk transaction identification scene can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced. For those skilled in the art, other drawings can also be obtained without creative labor on the premise of not paying creative labor.

[0054] Figure 1 is a flowchart of the evaluation information determination method provided by an embodiment of the present application;

[0055] Figure 2 is a flowchart of the evaluation information determination method in a specific embodiment of the present application

[0056] Figure 3 is a structural diagram of the evaluation information determination device provided by another embodiment of the present application;

[0057] Figure 4 is a hardware structural diagram of the computer equipment provided by another embodiment of the present application. DETAILED DESCRIPTION

[0058] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0059] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0060] Financial institutions' credit management systems typically need to identify various risk events at each stage of corporate lending—pre-loan, during-loan, and post-loan—using credit management models. These risk events are represented as credit risk information in the business system. Due to the numerous and complex causes of risk events, credit management models differ across credit sectors and stages. Each credit management model identifies risk events for users, and the results are aggregated into the credit management business system. This generates a large number of credit risk screening items, requiring manual identification, judgment, and final review by relevant personnel. This wastes time and energy, increases labor costs, and leads to low credit management efficiency. Since the risk violations identified by the credit management model require manual verification to confirm their authenticity, if the credit management model configured in the business system has low effectiveness, it will identify a large number of invalid risk events, consuming significant manpower for review.

[0061] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, device, and computer storage medium for determining evaluation information. The method for determining evaluation information provided in this application will be described first. It should be noted that the acquisition, storage, use, and processing of asset, financial, and other related data in this application's technical solution all comply with relevant national laws and regulations.

[0062] Figure 1 A flowchart illustrating an embodiment of the evaluation information determination method provided in this application is shown. Figure 1 As shown, the method includes steps S101 to S105:

[0063] S101 retrieves historical business data of users;

[0064] S102 trains a pre-set neural network using historical user business data to obtain a first risk identification model and the first quality evaluation information of the first risk identification model.

[0065] S103 generating service feature information by using the target model to be evaluated;

[0066] S104 training the neural network by using the service feature information and the user historical service data, to obtain a second risk identification model and second quality evaluation information of the second risk identification model;

[0067] S105 determining target evaluation information of the target model according to the first quality evaluation information and the second quality evaluation information.

[0068] According to the embodiments of the present application, the preset neural network can be trained by using the user historical service data to obtain a first risk identification model and first quality evaluation information of the model. Then, service feature information is generated by using the target model to be evaluated, and the neural network is trained again by using the service feature information and the user historical service data. In this way, the second risk identification model and corresponding second quality evaluation information can be obtained by training again by introducing new feature information (i.e., service feature information). Then, according to the comparison result of the two quality evaluation information, it can be reflected whether the training degree of the risk identification model obtained by training twice is improved. In this way, it can be quickly verified and evaluated whether the target model (i.e., the transaction identification model) is effective, which is beneficial to improving the reliability of the transaction identification model in the risk transaction identification scenario.

[0069] In some embodiments, the user historical service data obtained in step S101 can be user service data in a historical period (e.g., one year or 6 months).

[0070] For example, the user historical service data can include any two or more feature information in the following: borrower (i.e., user) credit information, guarantee mitigation measure information, user warning information, major credit risk transaction information, borrower rating information, etc. The user service data can also include other service data that can reflect risk events, which is not limited in the present embodiment.

[0071] For example, the feature information included in the user historical service data can be determined according to expert experience.

[0072] In the present embodiment, each item of information included in the user historical service data corresponds to a feature dimension. For example, the borrower credit information can correspond to credit situation features, the guarantee mitigation measure information can correspond to risk mitigation ability features, the user warning information can correspond to warning situation features, the major credit risk transaction information can correspond to risk situation features, and the borrower rating information can correspond to credit rating features.

[0073] Each feature is a dimension, in order to adapt to the risk cause complication existing in the credit risk identification scene, the user historical business data can correspond to multiple feature dimensions. In the embodiment, the user historical business data can be input into the neural network based on the feature dimensions to train and establish a benchmark model, that is, the first risk identification model.

[0074] For example, the preset neural network can be a convolutional neural network model built by using the Tensorflow framework, the user credit and risk related data are extracted as features (that is, the above multiple feature dimensions), the input data corresponding to the features (such as historical business data) are obtained, and the identification result of whether the user will actually default is obtained, which is represented by a probability value.

[0075] In the embodiment, after obtaining the user historical business data, the user historical business data is taken as a sample set, step S102 is performed, and the preset neural network is trained by using the user historical business data to obtain the first risk identification model and the first quality evaluation information of the first risk identification model.

[0076] The first quality evaluation information is information for evaluating the degree of goodness and badness of the first risk identification model. For example, the first quality evaluation information can include a first accuracy rate and / or a first recall rate. The accuracy rate is the accuracy degree of the first risk identification model, and the recall rate is the proportion of samples predicted as positive in the real positive samples.

[0077] The first accuracy rate and the first recall rate are both indexes for evaluating the first risk identification model trained by using the user historical business data, so as to evaluate the quality of the model, such as the prediction effect and performance. Specifically, the indexes can be calculated by using the output probability of the first risk identification model. Therefore, in the embodiment, step S102 can specifically include S1021-S1023:

[0078] S1021 inputs the user historical business data into the preset neural network;

[0079] S1022 trains the neural network according to the multiple feature dimensions to obtain the first risk identification model and the first output information of the first risk identification model;

[0080] S1023 calculates the first accuracy rate and / or the first recall rate according to the first output information.

[0081] In the embodiment, the user historical business data is a large amount of user credit data in a period of time, and each data includes the information of the above multiple feature dimensions and the information of whether to default. The user historical business data forms a sample set containing positive and negative samples, and is input into the risk identification model through step S1021.

[0082] The preset neural network is a convolutional neural network model. In the process of training the neural network through step S1022, the neural network can be trained based on user historical service data according to multiple feature dimensions, and a first risk identification model and first output information of the model are obtained. The first output information is the result of this training.

[0083] Based on the training result, step S1023 can be performed to calculate the first accuracy and / or the first recall. According to the training result and the real sample set, a confusion matrix can be formed, that is:

[0084] TP: the sample is positive, and the output result is positive;

[0085] FP: the sample is negative, and the output result is positive;

[0086] TN: the sample is negative, and the output result is negative;

[0087] FN: the sample is positive, and the output result is negative.

[0088] Therefore, the first accuracy is the ratio of the correct number to the total number of samples, which can be expressed by formula (1):

[0089] The first accuracy = (TP+TN) / (TP+FP+TN+FN) (1)

[0090] The first recall is the proportion of the sample with a positive output result in the sample with a real positive result, which can be expressed by formula (2):

[0091] The first recall = TP / (TP+FN) (2)

[0092] In this embodiment, the first accuracy and the first recall can be used to evaluate the result of this round of model training, that is, to evaluate the prediction effect and performance of the first risk identification model. For example, the higher the value of the first accuracy, the higher the prediction accuracy of the first risk identification model, and the higher the value of the first recall, the smaller the misjudgment rate of the first risk identification model.

[0093] In the credit management scenario, the main function of the credit management model is risk event identification, that is, the credit management model is an event identification model, and various risks identified by the credit management model ultimately reflect credit default risk. Therefore, in this embodiment, the preset event identification model is used as a target model to be evaluated, and the effectiveness (such as available or unavailable) of the target model is verified by the first risk identification model established as a benchmark model. Specifically, after training the neural network using user historical service data, the method can perform step S103 to generate a new feature (i.e., a feature dimension corresponding to service feature information) using the target model to be evaluated, for retraining the above preset neural network.

[0094] In some embodiments, the step S103, in the embodiments of the first aspect of the present application, the target model is a transaction identification model, and the business feature information is generated by using the target model to be evaluated, which can specifically include:

[0095] generating the business feature information according to at least one transaction feature of the transaction identification model,

[0096] wherein the feature dimension corresponding to the business feature information is different from the plurality of feature dimensions corresponding to the user historical business data.

[0097] In the present embodiment, in order to evaluate the transaction identification model, the target model to be evaluated, feature extraction is performed on the transaction identification model, and the corresponding business feature information can be generated. Wherein the transaction identification model is a credit management model, and is usually configured with one or more transactions to perform user credit management (such as credit approval) according to the identification result of the transaction, for example, the transaction includes “whether to admit the enterprise with risk behavior” and “the branch bank approval is passed for the business of the head office approval for veto or negotiation”. In order to verify the effectiveness of the transaction identification model, at least one transaction feature of the transaction identification model, such as the transaction feature of “whether to admit the enterprise with risk behavior”, is used as a new feature of the preset neural network, and the preset neural network is retrained. That is, the transaction identification model is equivalent to being abstracted as a new feature dimension, which is used as business feature information to participate in neural network training, so the feature dimension corresponding to the business feature information is different from the plurality of feature dimensions corresponding to the user historical business data. In this way, if the introduced business feature information can improve the quality of the second risk identification model obtained by training, it proves that the transaction identification model is effective and can be put into actual use.

[0098] Exemplarily, in the process of retraining the preset neural network through step S104, a second risk identification model and second quality evaluation information of the model can be obtained, wherein the second quality evaluation information is information for evaluating the degree of goodness of the second risk identification model. Exemplarily, the second quality evaluation information can include a second accuracy and / or a second recall rate. Specifically, the step S104 trains the neural network by using the business feature information and the user historical business data to obtain the second risk identification model and the second quality evaluation information of the second risk identification model, which can specifically include S1041-S1043:

[0099] S1041 inputs the user historical business data into the neural network;

[0100] S1042 trains the neural network according to the feature dimension corresponding to the business feature information and the plurality of feature dimensions corresponding to the user historical business data, to obtain the second risk identification model and the second output information of the second risk identification model.

[0101] S1043 calculates a second accuracy rate and / or a second recall rate according to the second output information.

[0102] In this embodiment, the training sample set used can be the same sample set as in step S102, i.e., the user historical service data described above, which is input into the preset neural network in step S1041.

[0103] Based on the input user historical service data, step S1042 is performed to train the neural network according to the feature dimensions corresponding to the service feature information and the multiple feature dimensions corresponding to the user historical service data, so that the second risk identification model and the second output information of the second risk identification model can be obtained. The second output information is the result of this training.

[0104] Based on the training result, step S1043 can be performed to calculate the second accuracy rate and / or the second recall rate. The calculation method of the second accuracy rate is the same as that of the first accuracy rate, and the calculation method of the second recall rate is the same as that of the first recall rate, which will not be described here.

[0105] In this embodiment, the second accuracy rate and the second recall rate can both be used to evaluate the result of this round of model training, i.e., to evaluate the prediction effect and performance of the second risk identification model.

[0106] After obtaining the second quality evaluation information for evaluating the prediction effect and performance of the second risk identification model, step S105 can be performed to determine the target evaluation information of the target model according to the first quality evaluation information and the second quality evaluation information. In step S105, by comparing the first quality evaluation information and the second quality evaluation information, if the accuracy rate is improved and / or the recall rate is improved, it can be proved that the added service feature information is effective. For example, the added service feature information is "the branch approval of the service whose total line approval is veto or continuation is passed", the obtained second accuracy rate is higher than the first accuracy rate, and the second recall rate is higher than the first recall rate, it can be judged that "the branch approval of the service whose total line approval is veto or continuation is passed" has a high correlation with the actual default, so it is judged that the corresponding matter identification model is effective, and evaluation information describing that the corresponding matter identification model is effective is generated. On the contrary, if the second information is compared with the first information, the accuracy rate and / or the recall rate is not improved, it can be judged that the matter identification model is ineffective, and evaluation information describing that the model is ineffective is generated.

[0107] According to the embodiments of the present application, the risk identification model (including the first risk identification model and the second risk identification model) trained can be used as a criterion, and the transaction identification model (i.e., the credit management model) in each credit field is formed into a new feature to participate in the retraining of the risk identification model, so as to improve the effectiveness of the risk identification model and judge the effectiveness of the transaction identification model. Compared with the traditional manual review method for confirming the authenticity of the transaction generated by the credit management model, the embodiments of the present application are more efficient and can greatly reduce the labor cost. Moreover, compared with the manual review method, the embodiments of the present application can avoid being limited by the professional ability of the relevant personnel and are beneficial to obtaining more objective effectiveness evaluation information.

[0108] When the transaction identification model is confirmed to be effective according to the target evaluation information, the model can be applied to risk transaction identification. Therefore, optionally, in the embodiments of the present application, after the target evaluation information of the target model is determined, as shown in Figure 2 the method can further include:

[0109] S106 obtaining user business data to be identified in a case where the target evaluation information meets a preset condition;

[0110] S107 performing feature extraction on the user business data to be identified by using the transaction identification model to obtain transaction identification information.

[0111] The user business data to be identified can include business data required by the transaction identification model applied to the field. For example, in the pre-loan link of the public credit, user rating and credit identification need to be performed according to the user business data, therefore, the transaction identification model applied to this link collects user business data to be identified, which can include data information of corresponding feature dimensions, such as borrower (i.e., user) credit information, guarantee release measure information, borrower rating information, and the like. Correspondingly, the transaction identification model needs to identify user rating transactions and whether to grant credit and the like.

[0112] It should be understood that the feature dimensions of the user business data to be identified required by the corresponding transaction identification model are different in different fields and different links, and the transactions to be identified are different.

[0113] Therefore, in the embodiments, the user business data to be identified is obtained, input into the corresponding transaction identification model, feature extraction is performed, and transaction identification information is obtained. For example, the transaction identification model can be a logistic regression (LR) model, a support vector machine (SVM) model, a random forest (RF) model, or a decision tree model, and the like, which is not limited in the embodiments.

[0114] According to the embodiments of this application, the reliability of user-related risk identification can be achieved after verifying the effectiveness of the event identification model (i.e., the credit management model) based on a benchmark-based risk identification model. This avoids the use of a large number of ineffective credit management models for actual risk identification, thereby improving the efficiency of credit management.

[0115] Combining the text Figure 1 and Figure 2 The method for determining evaluation information according to embodiments of this application is described in detail below. Figure 3 The apparatus of the embodiments of this application is described in detail below.

[0116] Figure 3 A schematic diagram of the evaluation information determination device provided in an embodiment of this application is shown. Figure 3 As shown, the device includes:

[0117] The first acquisition module 301 is used to acquire user historical business data;

[0118] The first training module 302 is used to train a neural network using historical business data of users to obtain a first risk identification model and the first quality evaluation information of the first risk identification model.

[0119] The generation module 303 is used to generate business feature information using the target model to be evaluated;

[0120] The second training module 304 is used to train the neural network using business feature information and user historical business data to obtain the second risk identification model and the second quality evaluation information of the second risk identification model.

[0121] The determination module 305 is used to determine the target evaluation information of the target model based on the comparison result of the first quality evaluation information and the second quality evaluation information.

[0122] According to embodiments of this application, a pre-defined neural network can be trained using historical user business data to obtain a first risk identification model and its first quality evaluation information. Then, business feature information is generated using the target model to be evaluated. This business feature information, along with historical user business data, is used to retrain the neural network. By introducing new feature information (i.e., business feature information) and retraining, a second risk identification model and corresponding second quality evaluation information can be obtained. The comparison results of the two quality evaluation information can then reflect whether the training level of the risk identification model obtained from the two training sessions has improved. This allows for rapid verification and evaluation of the effectiveness of the target model (i.e., the event identification model), thereby improving the reliability of the event identification model in risk event identification scenarios.

[0123] In some embodiments, the user historical service data acquired by the first acquisition module 301 can be user service data in a historical period (e.g., one year or six months).

[0124] For example, the user historical service data can include any two or more characteristic information in the borrower (i.e., user) credit information, guarantee release measure information, user warning information, major credit risk event information, and borrower rating information. The user service data can also include other service data that can reflect risk events, which are not limited in the present embodiment. In the present embodiment, each item of information included in the user historical service data corresponds to a characteristic dimension.

[0125] Each characteristic is a dimension. In order to adapt to the complex risk causes in the credit risk identification scenario, the user historical service data can correspond to multiple characteristic dimensions. In the present embodiment, the user historical service data can be input into a neural network for training based on these characteristic dimensions to establish a first risk identification model as a benchmark.

[0126] For example, the preset neural network can be a convolutional neural network model built using the Tensorflow framework. The user credit and risk-related data are extracted as characteristics (i.e., the above-mentioned multiple characteristic dimensions), the input data corresponding to the characteristics are acquired, and the identification result of whether the user will actually default is obtained, which is represented by a probability value.

[0127] Optionally, in some embodiments, the user historical service data corresponds to multiple characteristic dimensions, and the first quality evaluation information includes a first accuracy rate and / or a first recall rate. The first training module 302 can include:

[0128] A first input submodule for inputting the user historical service data into the neural network;

[0129] A first training submodule for training the neural network according to the multiple characteristic dimensions to obtain the first risk identification model and first output information of the first risk identification model;

[0130] A first calculation submodule for calculating according to the first output information to obtain the first accuracy rate and / or the first recall rate.

[0131] The accuracy rate is the degree of accuracy, and the recall rate is the proportion of samples predicted as positive in samples that are actually positive.

[0132] In the present embodiment, the user historical service data is a large amount of user credit data in a past period of time. Each data includes information of the above-mentioned multiple characteristic dimensions and information of whether to default. These user historical service data form a sample set containing positive and negative samples, which are input into the risk identification model through the first input submodule.

[0133] The preset neural network, as a convolutional neural network model, can be trained based on the user historical business data according to multiple feature dimensions, and a first risk identification model and first output information of the model are obtained. The first output information is the result of this training.

[0134] Based on the training result, the first calculation sub-module calculates the first accuracy rate and / or the first recall rate. In this embodiment, the first accuracy rate and the first recall rate can be used to evaluate the result of this round of model training, that is, to evaluate the prediction effect and performance of the first risk identification model. For example, the higher the value of the first accuracy rate, the higher the prediction accuracy of the first risk identification model, and the higher the value of the first recall rate, the smaller the misjudgment rate of the first risk identification model.

[0135] In the credit management scenario, the main function of the credit management model is risk event identification, that is, the credit management model is an event identification model, and various risks identified by the credit management model ultimately manifest as credit default risk. Therefore, in this embodiment, the preset event identification model is taken as a target model to be evaluated, and the effectiveness (such as usability or non-usability) of the target model is verified by the first risk identification model established as a benchmark model. Specifically, the generation module 303 can be configured to:

[0136] generate business feature information according to at least one event feature of the event identification model,

[0137] wherein the feature dimension corresponding to the business feature information is different from the multiple feature dimensions corresponding to the user historical business data.

[0138] In order to evaluate the event identification model, the event identification model to be evaluated, that is, the target model, is subjected to feature extraction, and corresponding business feature information can be generated. The event identification model is usually configured with one or more events to perform user credit management (such as credit approval) according to the identification result of the event, for example, the events include “whether to admit a company with risk behavior” and “branch approval for a business approved by the head office for veto or further negotiation”. In order to verify the effectiveness of the event identification model, at least one event feature of the event identification model, such as the event feature “whether to admit a company with risk behavior”, is taken as a new feature of the preset neural network, and the preset neural network is retrained. That is, the event identification model is abstracted as a new feature dimension, which is taken as business feature information and participates in neural network training, so the feature dimension corresponding to the business feature information is different from the multiple feature dimensions corresponding to the user historical business data. If the introduced business feature information can improve the quality of the second risk identification model obtained by training, it proves that the event identification model is effective and can be put into actual use.

[0139] In the process of retraining the preset neural network by the second training module 304, a second risk identification model and second quality evaluation information of the model can be obtained, wherein the second quality evaluation information is information for evaluating the degree of advantages and disadvantages of the second risk identification model. For example, the second quality evaluation information can include a second accuracy rate and / or a second recall rate. The second training module 304 can specifically include:

[0140] a second input sub-module configured to input the user historical service data into the preset neural network;

[0141] a second training sub-module configured to train the neural network according to the feature dimensions corresponding to the service feature information and the plurality of feature dimensions corresponding to the user historical service data, to obtain the second risk identification model and second output information of the second risk identification model;

[0142] a second calculation sub-module configured to calculate the second accuracy rate and / or the second recall rate according to the second output information.

[0143] In this embodiment, the training sample set used can be the same sample set as that used by the first training module 302, i.e., the user historical service data described above, which is input into the preset neural network by the second input sub-module.

[0144] Based on the input user historical service data, the second training sub-module trains the neural network according to the feature dimensions corresponding to the service feature information and the plurality of feature dimensions corresponding to the user historical service data, to obtain the second risk identification model and second output information of the second risk identification model. The second output information is the result of this training.

[0145] Based on the training result, the second calculation sub-module calculates the second accuracy rate and / or the second recall rate. The calculation method of the second accuracy rate is the same as that of the first accuracy rate, and the calculation method of the second recall rate is the same as that of the first recall rate, which will not be described here.

[0146] In this embodiment, the second accuracy rate and the second recall rate can both be used to evaluate the result of this round of model training, i.e., to evaluate the prediction effect and performance of the second risk identification model.

[0147] After obtaining the second quality evaluation information evaluating the prediction effect and performance of the second risk identification model, the target evaluation information can be determined by the determination module 305 according to the first quality evaluation information and the second quality evaluation information. In this process, by comparing the first quality evaluation information and the second quality evaluation information, if the accuracy rate is improved and / or the recall rate is improved, it can be proved that the added business feature information is effective, that is, the information has a high correlation with the actual default, so as to judge that the corresponding matter identification model is effective, and generate evaluation information describing that the corresponding matter identification model is effective. On the contrary, if the second information does not improve the accuracy rate and / or the recall rate compared with the first information, it can be judged that the matter identification model is ineffective, and evaluation information describing that the model is ineffective is generated.

[0148] According to the embodiments of the present application, the risk identification model (including the first risk identification model and the second risk identification model) trained can be used as a standard to measure the matter identification model (that is, the credit management model) in each credit field to form a new feature, participate in the retraining of the risk identification model, and judge the effectiveness of the matter identification model by improving the effectiveness of the risk identification model. Compared with the traditional manual review method to confirm the authenticity of the matter generated by the credit management model, the embodiments of the present application are more efficient and can greatly reduce the labor cost. And compared with the manual review method, the embodiments of the present application can avoid being limited by the professional ability of the relevant personnel, and are conducive to obtaining more objective effectiveness target evaluation information.

[0149] When the matter identification model is confirmed to be effective according to the target evaluation information, the model can be applied to risk matter identification. Therefore, optionally, in the embodiments of the present application, the device 300 can further include:

[0150] The second acquisition module is configured to acquire user business data to be identified in a case where the target evaluation information meets a preset condition.

[0151] The extraction module is configured to perform feature extraction on the user business data to be identified by the matter identification model to obtain matter identification information.

[0152] The user business data to be identified can include business data needed in the field to which the corresponding matter identification model is applied. For example, in the pre-loan link of public credit, user rating and credit identification need to be performed according to user business data, so the matter identification model applied in this link collects user business data to be identified including data information of corresponding feature dimensions, such as borrower (that is, user) credit information, guarantee release measure information, borrower rating information, etc. Correspondingly, the matter identification model needs to identify user rating matters and whether to grant credit and other matters.

[0153] It should be understood that the feature dimensions of the user business data to be identified required by the corresponding matter identification model are different in different fields and different links, and the matters to be identified are different.

[0154] Therefore, in this embodiment, the user business data to be identified is obtained, input into the corresponding matter identification model, feature extraction is performed, and matter identification information is obtained. For example, the matter identification model can be a logistic regression (LR) model, a support vector machine (SVM) model, a random forest (RF) model, or a decision tree model, and the like, and the embodiment is not limited to be unique.

[0155] According to the embodiments of the present application, the effectiveness of the matter identification model (i.e., the credit management model) can be verified after the risk identification model based on the benchmark, and the reliability of the user matter risk can be identified. In this way, a large number of invalid credit management models can be avoided for actual risk identification, thereby improving the efficiency of credit management.

[0156] It should be noted that all related contents of each step involved in the above method embodiments can be cited to the function description of the corresponding function module, and the corresponding technical effects can be achieved. For brief description, it will not be repeated here.

[0157] Figure 4 A hardware structure schematic diagram of a computer device provided by the embodiments of the present application is shown.

[0158] The computer device 400 can include a processor 401 and a memory 402 storing computer program instructions.

[0159] Specifically, the processor 401 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0160] The memory 402 can include a mass storage for data or instructions. For example, but not limited to, the memory 402 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of the above. In appropriate cases, the memory 402 can include removable or non-removable (or fixed) media. In appropriate cases, the memory 402 can be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 402 is a non-volatile solid state memory.

[0161] The memory can include read-only memory (ROM), random access memory (RAM), magnetic disk storage mediums devices, optical storage mediums devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage mediums (e.g., memory devices) encoded with software including computer-executable instructions that, when executed (e.g., by one or more processors), are operable to perform the operations described with reference to the methods according to an aspect of the present application.

[0162] The processor 401 implements any one of the evaluation information determination methods in the above-described embodiments by reading and executing computer program instructions stored in the memory 402.

[0163] In one example, the computer device 400 can further include a communication interface 403 and a bus 410. As shown, the processor 401, the memory 402, and the communication interface 403 are connected through the bus 410 and complete communication therebetween. Figure 4

[0164] The communication interface 403 is mainly used to realize the communication between the modules, devices, units, and / or equipment in the embodiments of the present application.

[0165] The bus 410 includes hardware, software, or both, which couples the components of the computer device 400 to each other. By way of example, and not limitation, the bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, the bus 410 can include one or more buses. Although the computer device 400 is described and shown with respect to a particular bus, the present application contemplates any suitable bus or interconnect.

[0166] In addition, in combination with the evaluation information determination method in the above-described embodiments, the embodiments of the present application can provide a computer storage medium to implement. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any one of the evaluation information determination methods in the above-described embodiments.

[0167] ​Moreover, in combination with the evaluation information determination method in the above-mentioned embodiments, the embodiments of the present application can provide a computer program product to implement. The instructions in the computer program product are executed by the processor of the computer device, so that the computer device executes any one of the evaluation information determination methods in the above-mentioned embodiments.

[0168] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted. In the above-mentioned embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0169] The functional modules shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0170] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be executed simultaneously.

[0171] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0172] The above describes only specific implementation of the present application. For the convenience and brevity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described herein. It should be understood that the protection scope of the present application is not limited in this way. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. An evaluation information determination method characterized by comprising: The method comprises: obtaining user historical service data corresponding to a plurality of feature dimensions; inputting the user historical service data into a preset neural network; training the neural network according to the plurality of feature dimensions to obtain a first risk identification model and first output information of the first risk identification model, the output information of the first risk identification model being a default probability of a user; calculating according to the first output information to obtain a first accuracy rate and / or a first recall rate; generating service feature information according to at least one transaction feature of a transaction identification model to be evaluated, wherein the feature dimension corresponding to the service feature information is different from the plurality of feature dimensions of the user historical service data; inputting the user historical service data into the neural network; training the neural network according to the feature dimension corresponding to the service feature information and the plurality of feature dimensions corresponding to the user historical service data to obtain a second risk identification model and second output information of the second risk identification model, the output information of the second risk identification model being a default probability of a user; calculating according to the second output information to obtain a second accuracy rate and / or a second recall rate; in a case where the second accuracy rate is higher than the first accuracy rate and the second recall rate is higher than the first recall rate, determining that the transaction identification model is effective and generating evaluation information describing that the transaction identification model is effective; in a case where the second accuracy rate is lower than the first accuracy rate and / or the second recall rate is lower than the first recall rate, determining that the transaction identification model is ineffective and generating evaluation information describing that the transaction identification model is ineffective; in a case where it is determined that the transaction identification model is effective, obtaining user service data to be identified; extracting features from the user service data to be identified through the transaction identification model to obtain transaction identification information, the transaction identification information including user rating transactions and whether to grant credit transactions.

2. An evaluation information determining apparatus characterized by comprising: The device comprises: a first obtaining module configured to obtain user historical service data corresponding to a plurality of feature dimensions; a first input submodule configured to input the user historical service data into a preset neural network; a first training submodule configured to train the neural network according to the plurality of feature dimensions to obtain a first risk identification model and first output information of the first risk identification model, the output information of the first risk identification model being a default probability of a user; a first calculation submodule configured to calculate according to the first output information to obtain a first accuracy rate and / or a first recall rate; a generation module configured to generate service feature information according to at least one transaction feature of a transaction identification model to be evaluated, wherein the feature dimension corresponding to the service feature information is different from the plurality of feature dimensions of the user historical service data; a second input submodule configured to input the user historical service data into the neural network; The second training submodule is configured to train the neural network according to the feature dimension corresponding to the service feature information and the plurality of feature dimensions corresponding to the user historical service data, to obtain a second risk identification model and second output information of the second risk identification model, wherein the output information of the second risk identification model is a default probability of a user. The second calculation submodule is configured to calculate according to the second output information to obtain a second accuracy rate and / or a second recall rate. The determination module is configured to determine that the transaction identification model is effective and generate evaluation information describing that the transaction identification model is effective when the second accuracy rate is higher than the first accuracy rate and the second recall rate is higher than the first recall rate, and determine that the transaction identification model is ineffective and generate evaluation information describing that the transaction identification model is ineffective when the second accuracy rate is lower than the first accuracy rate and / or the second recall rate is lower than the first recall rate. The second acquisition module is configured to acquire user service data to be identified when it is determined that the transaction identification model is effective. The extraction module is configured to perform feature extraction on the user service data to be identified by using the transaction identification model to obtain transaction identification information, wherein the transaction identification information includes a user rating transaction and a credit granting transaction.

3. A computer device, comprising: The device comprises a processor and a memory storing computer program instructions. The processor executes the computer program instructions to implement the evaluation information determination method of claim 1.

4. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the evaluation information determination method of claim 1.

5. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the computer device to enable the computer device to perform the evaluation information determination method of claim 1.

Images