Data risk level determination method and device and storage medium

Abstract

The embodiment of the invention provides a data risk level determination method and device and a storage medium, and the method comprises the steps: inputting a feature combination of to-be-recognized data into a target model, so as to determine a target risk level of the to-be-recognized data through the target model; the target model is trained by the following steps: acquiring a sample data set; determining the first feature of the positive sample data, the second feature of the negative sample data and the third feature of the diffusion sample data as a first feature set; combining features included in the first feature set to obtain a plurality of first feature combinations; training the initial model by using the first feature combination to obtain a training model; and when the first score of the training model is smaller than a first preset threshold value, training the training model by using the second feature combination, and determining the trained training model as a target model. Through the method and the device, the technical problem of low spoofing behavior recognition accuracy is solved, and the effect of improving the spoofing behavior recognition accuracy is achieved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of information security, and in particular, to a method, device and storage medium for determining a data risk level. Background Art

[0002] In related technologies, the implementation of fraudulent behavior usually requires a long incubation period. The links that can be identified and controlled by important anti-fraud entities such as operators, the Internet, and financial institutions are relatively lagging, resulting in low efficiency in identifying fraudulent behavior.

[0003] From this, we can see that there is a technical problem in the relevant technology that the accuracy of identifying deceptive behavior is low.

[0004] With respect to the above-mentioned problems existing in the related technologies, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and storage medium for determining data risk level, so as to at least solve the technical problem of low accuracy in identifying fraudulent behavior in the related art.

[0006] According to an embodiment of the present invention, a method for determining a data risk level is provided, comprising: inputting a feature combination of data to be identified into a target model to determine a target risk level of the data to be identified through the target model; wherein the target model is trained in the following manner: obtaining a sample data set, wherein the sample data set includes positive sample data whose event type is marked as the abnormal event type, negative sample data whose event type is marked as the non-abnormal event type, and diffusion sample data whose event type is not marked as the abnormal event type, wherein the data type of the diffusion sample data is the abnormal event type; inputting a first feature of the positive sample data, a first feature of the negative sample data, a second feature of the positive sample data, a second feature of the negative sample data, a second feature of the positive sample data, a second feature of the negative sample data The second feature of the present data and the third feature of the diffusion sample data are determined as a first feature set; the features included in the first feature set are combined to obtain multiple first feature combinations; the initial model is trained using the first feature combination to obtain a training model; when the first score of the training model is less than a first preset threshold, the training model is trained using the second feature combination, and the trained training model is determined as the target model, wherein the second feature combination is determined in the following manner: the risk level of the samples included in the sample data set is determined by the training model; the features of the samples with the target risk level are combined to obtain the second feature combination.

[0007] In an exemplary embodiment, before determining the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffused positive sample data as a first feature set, the method further includes: determining the first feature of the positive sample data, wherein determining the first feature of the positive sample data includes at least one of the following: performing statistical calculations on the positive sample data to obtain a first statistical feature, determining a first time series feature of sample data included in the positive sample data within a predetermined time period, combining features of data of different dimensions included in the positive sample data to obtain a first cross-combination feature; determining the second feature of the negative sample data, wherein determining the second feature of the negative sample data The method comprises at least one of the following: performing statistical calculations on the negative sample data to obtain a second statistical feature, determining a second time series feature of the sample data included in the negative sample data and within a predetermined time period, and combining features of data of different dimensions included in the negative sample data to obtain a second cross-combination feature; determining the third feature of the diffusion sample data, wherein determining the third feature of the diffusion sample data comprises at least one of the following: performing statistical calculations on the diffusion sample data to obtain a third statistical feature, determining a third time series feature of the sample data included in the diffusion sample data and within a predetermined time period, and combining features of data of different dimensions included in the diffusion sample data to obtain a third cross-combination feature.

[0008] In an exemplary embodiment, combining the features included in the first feature set to obtain multiple first feature combinations includes: determining a fourth feature included in the first feature set whose feature missing rate is less than or equal to a preset missing rate; determining a fifth feature included in the first feature set whose feature value similarity rate is less than or equal to a preset similarity rate; determining a sixth feature included in the first feature set whose information value index is greater than a preset index; determining the fourth feature, the fifth feature and the sixth feature as a first screening feature set; and obtaining multiple first feature combinations based on combining the features included in the first screening feature set.

[0009] In an exemplary embodiment, combining features included in the first screening feature set to obtain multiple first feature combinations includes: sorting features included in the first screening feature set whose feature scores are greater than a second preset threshold to obtain a second screening feature set; training an initial model using the second screening feature set to determine an evaluation index of the initial model; when the evaluation index is greater than a third preset threshold, eliminating N elimination features included in the second screening feature set to obtain a third screening feature set, wherein the feature scores of the elimination features are greater than the feature scores of features included in the second screening feature set other than the elimination features; combining the features included in the third screening feature set to obtain multiple groups of third feature combinations; training the initial model separately through each group of feature combinations included in the multiple groups of the third feature combinations to obtain a second score of the initial model; and determining the first feature combination based on the second score and the third feature combination.

[0010] In an exemplary embodiment, determining the first feature combination based on the second score and the third feature combination includes one of the following: determining the feature combination corresponding to the third score whose second score is greater than a fourth threshold as the first feature combination; determining the third feature combination corresponding to the third score whose second score is greater than a fourth threshold, and determining the combination included in the third feature combination whose number of features is less than a preset number as the first feature combination.

[0011] In an exemplary embodiment, the features included in the third screening feature are combined to obtain multiple groups of third feature combinations including at least one of the following: the features in the third screening feature are combined according to preset feature types to obtain the third feature combination; the feature group included in the third screening feature is determined as the third feature combination, wherein the sum of the feature scores of all the features included in the feature group is greater than a preset feature score accumulation threshold; the features included in the third screening feature are combined according to a preset algorithm to obtain the third feature combination.

[0012] In an exemplary embodiment, the features of the samples with the target risk level are combined to obtain the second feature combination, including: determining a seventh feature whose feature missing rate included in the features of the samples with the target risk level is less than or equal to a preset missing rate; determining an eighth feature whose feature value similarity rate included in the features of the samples with the target risk level is less than or equal to a preset similarity rate; determining a ninth feature whose information value index included in the features of the samples with the target risk level is greater than a preset index; determining the seventh feature, the eighth feature and the ninth feature as a second feature set; and combining the features included in the second feature set to obtain the second feature combination.

[0013] In an exemplary embodiment, combining the features included in the second feature set to obtain the second feature combination includes: determining a fourth feature combination included in the second feature set, wherein, after the training model is trained by the fourth feature combination, the fourth score of the training model is greater than the first preset threshold; setting a score threshold for the training model; combining the features included in the fourth feature combination to obtain a fifth feature combination, wherein, after the training model is trained by the fifth feature combination, the trained model score is greater than the score threshold; and determining the fifth feature combination as the second feature combination.

[0014] According to another embodiment of the present invention, there is provided a device for determining a data risk level, comprising: a determination module, for inputting a feature combination of data to be identified into a target model, so as to determine a target risk level of the data to be identified through the target model; wherein the target model is trained in the following manner: obtaining a sample data set, wherein the sample data set includes positive sample data whose event type is marked as the abnormal event type, negative sample data whose event type is marked as a non-abnormal event type, and diffusion sample data whose event type is not marked as the abnormal event type, wherein the data type of the diffusion sample data is the abnormal event type; inputting a first feature of the positive sample data, The second feature of the negative sample data and the third feature of the diffusion sample data are determined as a first feature set; the features included in the first feature set are combined to obtain multiple first feature combinations; the initial model is trained using the first feature combination to obtain a training model; when the first score of the training model is less than a first preset threshold, the training model is trained using the second feature combination, and the trained training model is determined as the target model, wherein the second feature combination is determined in the following manner: the risk level of the samples included in the sample data set is determined by the training model; the features of the samples with a risk level of the target level are combined to obtain the second feature combination.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0016] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to yet another embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0018] Through the present invention, the positive sample data, negative sample data and diffusion sample data are obtained, and all the features in the first feature of the positive sample data, the second feature of the negative sample data and the third feature of the diffusion sample data are combined. The initial model is trained by the obtained multiple first feature combinations to obtain a training model. When the first score of the training model is less than the first preset threshold, it means that the training model effect does not meet the standard. The training model can be trained using the second feature combination with the target risk level to obtain the final target model. Since the unbalanced samples containing the diffusion sample data are trained until the training model reaches the standard, the training is stopped. Therefore, the technical problem of low accuracy in identifying deceptive behavior in the related technology can be solved, and the accuracy of identifying deceptive behavior can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a hardware structure block diagram of a mobile terminal according to a method for determining a data risk level in an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for determining a data risk level according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a target model building process according to an embodiment of the present invention;

[0022] Figure 4 is a flow chart of a method for data risk level according to a specific embodiment of the present invention;

[0023] Figure 5 4 is a structural block diagram of a device for determining a data risk level according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for determining a data risk level according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the data risk level in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a method for determining the data risk level is provided. Figure 2 is a flow chart of a method for determining a data risk level according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0030] Step S202, inputting the feature combination of the data to be identified into the target model, so as to determine the target risk level of the data to be identified through the target model;

[0031] The target model is trained in the following way:

[0032] Acquire a sample data set, wherein the sample data set includes positive sample data whose event type is marked as the abnormal event type, negative sample data whose event type is marked as a non-abnormal event type, and diffusion sample data whose event type is not marked as the abnormal event type, wherein the data type of the diffusion sample data is the abnormal event type;

[0033] Combining the features included in the first feature set to obtain a plurality of first feature combinations;

[0034] Determine the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffusion sample data as a first feature set;

[0035] Using the first feature combination to train an initial model to obtain a training model;

[0036] When the first score of the training model is less than a first preset threshold, the training model is trained using a second feature combination, and the trained training model is determined as the target model, wherein the second feature combination is determined in the following manner: the risk level of the samples included in the sample data set is determined by the training model; and the features of the samples whose risk level is the target level are combined to obtain the second feature combination.

[0037] In the above embodiment, the abnormal event may be that the user is deceived, etc. The target risk level is the result obtained by the target model judging the abnormal event of the user. The classification of the target risk level can refer to the following: Figure 3 , Figure 3 Schematic diagram of the target model building process according to an embodiment of the present invention. Figure 3 As shown, the target risk level can include high risk level, medium risk level and low risk level. The feature combination of the data to be identified is input into the target model for detection, and the target risk level corresponding to the data to be identified can be obtained. Due to the long-term latent and sudden outbreak characteristics of fraud incidents, the control of financial management institutions among the important anti-fraud participants is relatively lagging. Therefore, it is necessary to widely accumulate external effective data and fully explore the effective features of its own channel embedding points, transaction flow, credit reporting and other dimensions (that is, the above-mentioned sample data set).

[0038] In the above embodiment, when mining valid sample data sets inside and outside the application, it is possible to consider whether the required data sources meet the regulations and whether the customer has authorized them. For external data, from the perspective of time series, data crossing should be strictly prevented in advance and during the process, and the focus can also be on identifying the pre-timeliness; from the perspective of dimensions, data such as operators, the Internet, hardware equipment, bank-to-bank transactions, credit reporting, and notifications from competent authorities can be covered as much as possible; from the perspective of processes, it can be defined from the legality of data acquisition, procurement compliance, sample transmission compliance, and whether the test results after restoration sampling meet business objectives. For internal data, from the perspective of dimensions, it is possible to fully mine loan channel equipment information, buried point behavior information, account opening information, transaction information, adjustment and release control and other information. From the perspective of matters, it is possible to focus on mining characteristics such as consistency, volatility, and abnormality. From the perspective of processes, if the data of the parent and subsidiary companies of the group is involved, the data interaction mode and deployment mode of the parent and subsidiary companies can be fully considered.

[0039] In the above embodiment, positive sample data is an important element of supervised and semi-supervised modeling, that is, the number and accuracy of the accumulated positive sample data determine the detection effect of the target model. Therefore, samples identified by the competent authority or personally admitted by the customer before, during, or after the event can be used as the standard. In view of the problem that some victims are unwilling to admit and feedback that they have been deceived, resulting in inaccurate sample accumulation, the data of behaviors such as account freezing within a period after the loan can be combined with the characteristics of deceptive behavior and business experience, and the data of behaviors such as account freezing within a period after the loan can be used as diffusion sample data. Among them, positive sample data and negative sample data are used to train and evaluate the initial model, and diffusion sample data is used to train the initial model. Due to the variety of deception methods and frequent changes, the newly accumulated positive sample data and the negative sample data of the same period can be used for modeling; but considering that some victims are unwilling or unable to feedback in time that they have been deceived, the acquisition and confirmation of positive samples have different degrees of lag, therefore, it is possible to choose to use the positive sample data that occurred and accumulated 2 months ago for modeling, for example: a user was deceived when applying for a loan in March, but did not feedback abnormalities at the time, and only reported that he was deceived in March in May. In addition, through verification, customer service, collection and other sources, before, during and after the event, the data acknowledged by the customer can be used as positive sample data, and other customers in the same period can be sampled as negative sample data that have not been deceived. For abnormal events that occurred within two months, their positive sample data and negative sample data can also be used for judgment.

[0040] In the above embodiment, the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffusion sample data are extracted, and the features in the first feature, the second feature, and the third feature are combined to obtain a plurality of different feature combinations (i.e., the above first feature combinations). The initial model is trained respectively using different first feature combinations to obtain a trained training model. In the above embodiment, after determining the first feature combination, the open source hyperopt algorithm can be used to set different objective functions, evaluation function combinations, and other parameter ranges of the lgb model, and output the training model parameters. For the case where the predicted probability is negative based on the training model after parameter adjustment, a scaling conversion can be supplemented to make the predicted probability range satisfy [0-1]. Among them, the initial model can be an lgb model, and the first preset threshold can be understood as a score set based on business objectives, which may include: AUC (Area Under ROC Curve, model evaluation index), KS (Kolmogorov-Smirnov, Kolmogorov-Smirnov test), trigger rate, precision rate, recall rate, F1 (F1Score), Fbeta (Fbeta Score), LIFT (LIFT Chart, improvement index) and topN effect (topN score). The predicted probability can be converted into a score value to form a cutoff analysis and strategy application recommendations. Among them, the business goal can be understood as combining business management goals, formulating key prevention and control types, balancing business development and risk prevention and control, input and output as the main indicators, which may include: low cost, low trigger, high precision, high recall and other goals. When the first score of the training model is less than the first preset threshold, that is, the training model does not meet the standard, the samples of high risk level and low risk level are eliminated to increase the combination rule mining, and the data features of the medium risk level (that is, the above target level) in the target risk level are combined to obtain the second feature combination. The training model is then trained using the second feature combination to obtain the target model.

[0041] Through the present invention, the positive sample data, negative sample data and diffusion sample data are obtained, and all the features in the first feature of the positive sample data, the second feature of the negative sample data and the third feature of the diffusion sample data are combined. The initial model is trained by the obtained multiple first feature combinations to obtain a training model. When the first score of the training model is less than the first preset threshold, it means that the training model effect does not meet the standard. The training model can be trained using the second feature combination with the target risk level to obtain the final target model. Since the unbalanced samples containing the diffusion sample data are trained until the training model reaches the standard, the training is stopped. Therefore, the technical problem of low accuracy in identifying deceptive behavior in the related technology can be solved, and the accuracy of identifying deceptive behavior can be improved.

[0042] Optionally, the execution subject of the above steps may be a background processor, but is not limited thereto.

[0043] In an exemplary embodiment, before determining the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffused positive sample data as a first feature set, the method further includes: determining the first feature of the positive sample data, wherein determining the first feature of the positive sample data includes at least one of the following: performing statistical calculations on the positive sample data to obtain a first statistical feature, determining a first time series feature of sample data included in the positive sample data within a predetermined time period, combining features of data of different dimensions included in the positive sample data to obtain a first cross-combination feature; determining the second feature of the negative sample data, wherein determining the second feature of the negative sample data The method comprises at least one of the following: performing statistical calculations on the negative sample data to obtain a second statistical feature, determining a second time series feature of the sample data included in the negative sample data and within a predetermined time period, and combining features of data of different dimensions included in the negative sample data to obtain a second cross-combination feature; determining the third feature of the diffusion sample data, wherein determining the third feature of the diffusion sample data comprises at least one of the following: performing statistical calculations on the diffusion sample data to obtain a third statistical feature, determining a third time series feature of the sample data included in the diffusion sample data and within a predetermined time period, and combining features of data of different dimensions included in the diffusion sample data to obtain a third cross-combination feature.

[0044] In the above embodiment, when processing the features of positive sample data, negative sample data, and diffusion sample data, from the perspective of time series, the features within a period of time before the intended prevention and control link can be selected to prevent the existence of post-event features from causing the pre-event and in-event prevention and control timeliness after production. From the perspective of dimensions, data from different sources such as operators, the Internet, hardware equipment, UnionPay and NetsUnion transactions, credit reporting, and notifications from competent authorities can be used to process the relevant features of suspected deceived customers, such as: browsing and clicking records of bad and suspected fraudulent websites on the Internet, attack records on the equipment, and transaction records of unfamiliar accounts in UnionPay, NetsUnion, and account transaction flows. From the perspective of credit reporting, data with short-term long positions or suspected fraud by relevant authorities before the loan can be processed.

[0045] In the above embodiments, the data in the sample data set can be processed for statistical characteristics (i.e., the above-mentioned statistical calculation) to obtain a first statistical feature, the sample data set within a predetermined time period can be processed to obtain a first time series feature, the data features of different dimensions included in the sample data set can be combined to obtain a first cross-combination feature, and specific business meaning processing can also be performed, wherein the statistical characteristic processing can be mean calculation, maximum value calculation, variance calculation, etc. of the features; the specific business feature can be the processing of data features according to different business goals.

[0046] In an exemplary embodiment, combining the features included in the first feature set to obtain multiple first feature combinations includes: determining a fourth feature included in the first feature set whose feature missing rate is less than or equal to a preset missing rate; determining a fifth feature included in the first feature set whose feature value similarity rate is less than or equal to a preset similarity rate; determining a sixth feature included in the first feature set whose information value index is greater than a preset index; determining the fourth feature, the fifth feature and the sixth feature as a first screening feature set; and obtaining multiple first feature combinations based on combining the features included in the first screening feature set.

[0047] In the above embodiment, the features in the first feature set can be combined in combination with business objectives (such as: accurate overall sorting, accurate and comprehensive positive sample data identification), the fourth feature in the first feature set whose feature missing rate is less than or equal to the preset missing rate can be retained, the fifth feature in the first feature set whose feature value identity rate is less than or equal to the preset same value rate can be retained, and the sixth feature in the first feature set whose information value index is greater than the preset index can be retained. The retained fourth feature, fifth feature and sixth feature can form the first screening feature set, and the features in the first screening feature set can be combined to obtain multiple first feature combinations. Among them, the preset missing rate can be a preset feature missing rate of 50%, or a feature missing rate of 80%, but is not limited to this. The preset same value rate can be a preset degree of similarity between features, which can be 40% or 50%, but is not limited to this. The information value index can be understood as an IV (Information Value) value, and the preset index can be a preset feature information value index, which can be 0.6 or 0.9, but is not limited to this.

[0048] In an exemplary embodiment, combining features included in the first screening feature set to obtain multiple first feature combinations includes: sorting features included in the first screening feature set whose feature scores are greater than a second preset threshold to obtain a second screening feature set; training an initial model using the second screening feature set to determine an evaluation index of the initial model; when the evaluation index is greater than a third preset threshold, eliminating N elimination features included in the second screening feature set to obtain a third screening feature set, wherein the feature scores of the elimination features are greater than the feature scores of features included in the second screening feature set other than the elimination features; combining the features included in the third screening feature set to obtain multiple groups of third feature combinations; training the initial model separately through each group of feature combinations included in the multiple groups of the third feature combinations to obtain a second score of the initial model; and determining the first feature combination based on the second score and the third feature combination.

[0049] In the above embodiment, the second preset threshold value can be a value greater than 0 in feature importance. Specific parameters and differentiated loss functions can be set for the initial model according to different business objectives. The features whose feature importance (i.e., the above feature score) is greater than 0 in the first screening feature set are retained and sorted, and the second screening feature set can be obtained. The initial model is trained using the feature combination in the second screening feature set and the adversarial verification algorithm. When the AUC value (i.e., the above evaluation index) is greater than the third preset threshold value, that is, there is a large difference in the distribution of the training set and the test set, the N features with the highest feature importance ranking can be eliminated to obtain the third screening feature set, which can reduce the overfitting in the subsequent target modeling process. Wherein, N can be 1, 2, 3...N, and the AUC value can be understood as a model evaluation index, usually between 0-1. By combining the features in the third screening feature set, multiple groups of third feature combinations can be obtained. By using multiple groups of third feature combinations to train the initial model, the second score of the initial model (i.e., the training model score result determined based on the business objective) can be obtained, and the second score and the third feature combination are determined as the first feature combination.

[0050] In an exemplary embodiment, determining the first feature combination based on the second score and the third feature combination includes one of the following: determining the feature combination corresponding to the third score whose second score is greater than a fourth threshold as the first feature combination; determining the third feature combination corresponding to the third score whose second score is greater than a fourth threshold, and determining the combination included in the third feature combination whose number of features is less than a preset number as the first feature combination.

[0051] In the above embodiment, the fourth threshold may be a pre-set score determined based on the business goal, and the feature combination corresponding to the third score whose second score is greater than the fourth threshold may be determined as the first feature combination, and the combination whose number of features in the third feature combination corresponding to the third score is less than the preset number may be determined as the first feature combination. The preset number may be 2, 3, etc., but is not limited thereto. Example: Feature combination 1 includes features a and b, and feature combination 2 includes features d, f, and g. The scores of feature combination 1 and feature combination 2 are both greater than the fourth threshold. Since the number of features in feature combination 1 is less than the number of features in feature combination 2, feature combination 2 may be determined as the first feature combination.

[0052] In an exemplary embodiment, the features included in the third screening feature are combined to obtain multiple groups of third feature combinations including at least one of the following: the features in the third screening feature are combined according to preset feature types to obtain the third feature combination; the feature group included in the third screening feature is determined as the third feature combination, wherein the sum of the feature scores of all the features included in the feature group is greater than a preset feature score accumulation threshold; the features included in the third screening feature are combined according to a preset algorithm to obtain the third feature combination.

[0053] In the above embodiment, the features included in the third screening feature can be combined according to the pre-set feature types, the features included in the third screening feature whose importance and reach the preset feature score accumulation threshold can be combined, the features included in the third screening feature can be combined according to the preset algorithm, and the features included in the third screening feature can be combined based on the evaluation effect by automatically adding features one by one. Among them, the pre-set feature type can be the pre-set feature importance topM. The preset algorithm can be the RFECV (Recursive feature elimination, Cross Validation) automatic step-by-step elimination algorithm.

[0054] In an exemplary embodiment, the features of the samples with the target risk level are combined to obtain the second feature combination, including: determining a seventh feature whose feature missing rate included in the features of the samples with the target risk level is less than or equal to a preset missing rate; determining an eighth feature whose feature value similarity rate included in the features of the samples with the target risk level is less than or equal to a preset similarity rate; determining a ninth feature whose information value index included in the features of the samples with the target risk level is greater than a preset index; determining the seventh feature, the eighth feature and the ninth feature as a second feature set; and combining the features included in the second feature set to obtain the second feature combination.

[0055] In the above embodiment, for unbalanced samples (and the above sample data sets), the model has a good prediction effect at the head, but the prediction effect below the head may not be satisfactory. Directly applying the model results may have problems such as accurate but incomplete, complete but inaccurate. Therefore, in the case of limited model optimization space, for the part where the model prediction is not very accurate, the training model can be used again to quickly develop a small number of combination rules, so as to better take into account the business goals of low triggering, high precision, and high completeness. When applying, a pyramid decision-making method can be adopted: the accurate part of the model head + the inaccurate and incomplete part of the model except the head, which can be retrieved using the combination rule method, can improve the overall application effect.

[0056] In the above embodiment, the features with a target risk level of medium risk level can be combined in combination with business objectives (such as: accurate overall sorting, accurate and comprehensive identification of positive sample data), the seventh feature with a feature missing rate less than or equal to a preset missing rate in the medium risk level can be retained, the eighth feature with a feature value identity rate less than or equal to a preset identity rate in the medium risk level can be retained, and the ninth feature with an information value index greater than a preset index in the medium risk level can also be retained. The retained seventh feature, eighth feature, and ninth feature can form a second feature set, and the features in the second feature set can be combined to obtain a second feature combination.

[0057] In an exemplary embodiment, combining the features included in the second feature set to obtain the second feature combination includes: determining a fourth feature combination included in the second feature set, wherein, after the training model is trained by the fourth feature combination, the fourth score of the training model is greater than the first preset threshold; setting a score threshold for the training model; combining the features included in the fourth feature combination to obtain a fifth feature combination, wherein, after the training model is trained by the fifth feature combination, the trained model score is greater than the score threshold; and determining the fifth feature combination as the second feature combination.

[0058] In the above embodiment, each path of the GBDT (Gradient Boosting Decision Tree) model can be used as each single combination rule to implement combination rule mining. In the training set and the test set, a single combination rule with low triggering, high precision and high recall can be automatically selected by setting a threshold. The training model is trained using each feature combination, and in combination with business goals (trigger rate, precision rate, recall rate, number of rule sets, etc.), a fourth feature combination with a fourth score greater than the first preset threshold can be determined. The feature combinations in the fourth feature combination are trained, sorted by Fbeta based on the scoring threshold of the training model (i.e., the business goal), and features are added one by one for training. When the model score of the fifth feature combination after the addition of the features meets the scoring threshold, it is determined to be the second feature combination.

[0059] In the above embodiment, manual verification can also be performed. For the model prediction results, the shap value is used to display the risk factors of individual customers to the verification post as auxiliary verification information. The target model needs to undergo a period of trial operation. During the trial operation, the actual effect expected after the model or rule is put into production can be evaluated in combination with the newly confirmed potential victims of fraud. For the trial operation rules whose effects meet the business objectives, the best ones can be transferred to formal operation in a timely manner. For the target model that has been put into formal operation, the problem that the fixed threshold is easy to be cracked can be solved by using a floating threshold, which can be determined in advance in the modeling and analysis stage.

[0060] The following is an explanation of the method for determining the data risk level in conjunction with a specific implementation method:

[0061] Figure 4 is a flow chart of a method for determining data risk levels according to a specific embodiment of the present invention. Figure 4 As shown, the process includes:

[0062] Step S402, data import preparation;

[0063] Step S404, business scenario and goal definition;

[0064] Step S406, risk control engine strategy operation;

[0065] Step S408, manual verification;

[0066] Step S410, sample accumulation;

[0067] Step S412, sample selection and key definition;

[0068] Step S414, feature processing;

[0069] Step S416, feature screening;

[0070] Step S418, model establishment;

[0071] Step S420, result prediction;

[0072] Step S422, score conversion;

[0073] Step S424, effect evaluation.

[0074] In the above embodiments, the internal and external characteristics of the Internet, operators, credit, accounts, transactions, channels and other dimensions are fully introduced during the data introduction process, and an anti-fraud data ecosystem is constructed, which can comprehensively collect risk signals of various dimensions of customers during the latent period of being deceived and the implementation period of being deceived, and can protect potential victims of deception, identify them in advance and accurately. In terms of strategic operation management, the experience of flexible risk control of Internet institutions is learned, and experts with both model development and strategic operation experience take the lead in modeling, which reduces communication costs, ensures the quantity and quality of sample accumulation, and realizes closed-loop management of data collection, data processing, tool development, strategy development, strategy operation monitoring, and continuous iteration. In terms of modeling, by using stream computing to process real-time features, for unbalanced samples containing diffusion data, combined with business goals, a variety of solutions are provided in terms of feature screening, model parameter tuning, combination rule mining, model + combination rule strategy combination, etc., which can realize automatic selection from many solutions and finally form the optimal real-time machine learning model that meets business goals.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0076] In this embodiment, a device for determining a data risk level is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0077] Figure 5 is a structural block diagram of a device for determining a data risk level according to an embodiment of the present invention. Figure 5 As shown, the device comprises

[0078] A determination module 502 is used to input the feature combination of the data to be identified into the target model to determine the target risk level of the data to be identified through the target model;

[0079] The target model is trained in the following way:

[0080] Acquire a sample data set, wherein the sample data set includes positive sample data whose event type is marked as the abnormal event type, negative sample data whose event type is marked as a non-abnormal event type, and diffusion sample data whose event type is not marked as the abnormal event type, wherein the data type of the diffusion sample data is the abnormal event type;

[0081] Determine the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffusion sample data as a first feature set;

[0082] Combining the features included in the first feature set to obtain a plurality of first feature combinations;

[0083] Using the first feature combination to train an initial model to obtain a training model;

[0084] When the first score of the training model is less than a first preset threshold, the training model is trained using a second feature combination, and the trained training model is determined as the target model, wherein the second feature combination is determined in the following manner: the risk level of the samples included in the sample data set is determined by the training model; and the features of the samples whose risk level is the target level are combined to obtain the second feature combination.

[0085] In an exemplary embodiment, the device can be used to determine the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffused positive sample data as a first feature set before: determining the first feature of the positive sample data, wherein determining the first feature of the positive sample data includes at least one of the following: performing statistical calculations on the positive sample data to obtain a first statistical feature, determining a first time series feature of sample data included in the positive sample data within a predetermined time period, combining features of data of different dimensions included in the positive sample data to obtain a first cross-combination feature; determining the second feature of the negative sample data, wherein determining the second feature of the negative sample data The method comprises at least one of the following: performing statistical calculations on the negative sample data to obtain a second statistical feature, determining a second time series feature of the sample data included in the negative sample data and within a predetermined time period, and combining features of data of different dimensions included in the negative sample data to obtain a second cross-combination feature; determining the third feature of the diffusion sample data, wherein determining the third feature of the diffusion sample data comprises at least one of the following: performing statistical calculations on the diffusion sample data to obtain a third statistical feature, determining a third time series feature of the sample data included in the diffusion sample data and within a predetermined time period, and combining features of data of different dimensions included in the diffusion sample data to obtain a third cross-combination feature.

[0086] In an exemplary embodiment, the device can implement the combination of features included in the first feature set to obtain multiple first feature combinations in the following manner: determine a fourth feature included in the first feature set whose feature missing rate is less than or equal to a preset missing rate; determine a fifth feature included in the first feature set whose feature value similarity rate is less than or equal to a preset similarity rate; determine a sixth feature included in the first feature set whose information value index is greater than a preset index; determine the fourth feature, the fifth feature and the sixth feature as a first screening feature set; and obtain multiple first feature combinations by combining the features included in the first screening feature set.

[0087] In an exemplary embodiment, the device can achieve combination based on the features included in the first screening feature set to obtain multiple first feature combinations in the following manner: sorting the features included in the first screening feature set whose feature scores are greater than a second preset threshold to obtain a second screening feature set; using the second screening feature set to train the initial model to determine the evaluation index of the initial model; when the evaluation index is greater than a third preset threshold, eliminating N elimination features included in the second screening feature set to obtain a third screening feature set, wherein the feature score of the elimination feature is greater than the feature score of the features included in the second screening feature set other than the elimination feature; combining the features included in the third screening feature set to obtain multiple groups of third feature combinations; training the initial model separately through each group of feature combinations included in the multiple groups of the third feature combinations to obtain a second score of the initial model; determining the first feature combination based on the second score and the third feature combination.

[0088] In an exemplary embodiment, the device can be used to determine one of the following for the first feature combination based on the second score and the third feature combination: determine the feature combination corresponding to the third score whose second score is greater than a fourth threshold as the first feature combination; determine the third feature combination corresponding to the third score whose second score is greater than a fourth threshold, and determine the combination included in the third feature combination that includes a number of features less than a preset number as the first feature combination.

[0089] In an exemplary embodiment, the device can also be used to combine the features included in the third screening feature to obtain multiple groups of third feature combinations, including at least one of the following: combining the features in the third screening feature according to preset feature types to obtain the third feature combination; determining the feature group included in the third screening feature as the third feature combination, wherein the sum of the feature scores of all features included in the feature group is greater than a preset feature score accumulation threshold; combining the features included in the third screening feature according to a preset algorithm to obtain the third feature combination.

[0090] In an exemplary embodiment, the device can also be used to combine the features of the samples with the target risk level to obtain the second feature combination: determine the seventh feature included in the features of the samples with the target risk level, whose feature missing rate is less than or equal to the preset missing rate; determine the eighth feature included in the features of the samples with the target risk level, whose feature value similarity rate is less than or equal to the preset similarity rate; determine the ninth feature included in the features of the samples with the target risk level, whose information value index is greater than the preset index; determine the seventh feature, the eighth feature and the ninth feature as the second feature set; and combine the features included in the second feature set to obtain the second feature combination.

[0091] In an exemplary embodiment, the device can also be used to combine the features included in the second feature set to obtain the second feature combination: determine a fourth feature combination included in the second feature set, wherein, after the training model is trained by the fourth feature combination, the fourth score of the training model is greater than the first preset threshold; set the score threshold of the training model; combine the features included in the fourth feature combination to obtain a fifth feature combination, wherein, after the training model is trained by the fifth feature combination, the trained model score is greater than the score threshold; determine the fifth feature combination as the second feature combination.

[0092] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0093] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0094] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0095] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0096] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0097] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the method in each embodiment of the present application when the computer program is executed by a processor.

[0098] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0099] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining data risk level, characterized in that: include: Inputting the feature combination of the data to be identified into the target model to determine the target risk level of the data to be identified through the target model; The target model is trained in the following way: Acquire a sample data set, wherein the sample data set includes positive sample data whose event type is marked as the abnormal event type, negative sample data whose event type is marked as a non-abnormal event type, and diffusion sample data whose event type is not marked as the abnormal event type, wherein the data type of the diffusion sample data is the abnormal event type; Determine the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffusion sample data as a first feature set; Combining the features included in the first feature set to obtain a plurality of first feature combinations; Using the first feature combination to train an initial model to obtain a training model; When the first score of the training model is less than a first preset threshold, the training model is trained using a second feature combination, and the trained training model is determined as the target model, wherein the second feature combination is determined in the following manner: the risk level of the samples included in the sample data set is determined by the training model; and the features of the samples whose risk level is the target level are combined to obtain the second feature combination.

2. The method according to claim 1, characterized in that Before determining the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffused positive sample data as a first feature set, the method further includes: Determine the first feature of the positive sample data, wherein determining the first feature of the positive sample data comprises at least one of the following: performing statistical calculations on the positive sample data to obtain a first statistical feature, determining a first time series feature of sample data included in the positive sample data and within a predetermined time period, and combining features of data of different dimensions included in the positive sample data to obtain a first cross-combination feature; Determine the second feature of the negative sample data, wherein determining the second feature of the negative sample data comprises at least one of the following: performing statistical calculations on the negative sample data to obtain a second statistical feature, determining a second time series feature of sample data included in the negative sample data and within a predetermined time period, and combining features of data of different dimensions included in the negative sample data to obtain a second cross-combination feature; Determine the third feature of the diffusion sample data, wherein determining the third feature of the diffusion sample data comprises at least one of the following: performing statistical calculations on the diffusion sample data to obtain a third statistical feature, determining a third time series feature of sample data included in the diffusion sample data and within a predetermined time period, and combining features of data of different dimensions included in the diffusion sample data to obtain a third cross-combination feature.

3. The method according to claim 1, characterized in that: The combining of the features included in the first feature set to obtain a plurality of first feature combinations includes: Determine a fourth feature included in the first feature set whose feature missing rate is less than or equal to a preset missing rate; Determine a fifth feature whose identical rate of feature values ​​included in the first feature set is less than or equal to a preset identical rate; Determine a sixth feature whose information value index included in the first feature set is greater than a preset index; Determine the fourth feature, the fifth feature, and the sixth feature as a first screening feature set; Based on the features included in the first screening feature set, a plurality of first feature combinations are obtained.

4. The method according to claim 3, characterized in that Based on the combination of features included in the first screening feature set, a plurality of first feature combinations are obtained, including: Sort the features in the first screening feature set whose feature scores are greater than a second preset threshold to obtain a second screening feature set; Using the second screening feature set to train an initial model to determine an evaluation index of the initial model; When the evaluation index is greater than a third preset threshold, N elimination features included in the second screening feature set are eliminated to obtain a third screening feature set, wherein the feature scores of the elimination features are greater than the feature scores of the features other than the elimination features included in the second screening feature set; Combining the features included in the third screening feature set to obtain multiple groups of third feature combinations; The initial model is trained respectively by each feature combination included in the plurality of third feature combinations to obtain a second score of the initial model; The first feature combination is determined based on the second score and the third feature combination.

5. The method according to claim 4, characterized in that Determining the first feature combination based on the second score and the third feature combination includes one of the following: Determine the feature combination corresponding to the third score whose second score is greater than a fourth threshold as the first feature combination; A third feature combination corresponding to a third score whose second score is greater than a fourth threshold is determined, and a combination whose number of features is less than a preset number and is included in the third feature combination is determined as the first feature combination.

6. The method according to claim 4, characterized in that The features included in the third screening feature are combined to obtain multiple groups of third feature combinations including at least one of the following: Combining the features in the third screening feature according to the preset feature types to obtain the third feature combination; Determine a feature group included in the third screening feature as the third feature combination, wherein the sum of feature scores of all features included in the feature group is greater than a preset feature score accumulation threshold; The features included in the third screening feature are combined according to a preset algorithm to obtain the third feature combination.

7. The method according to claim 1, characterized in that Combining the features of the samples whose risk level is the target level to obtain the second feature combination includes: Determine that the features of the sample with the risk level of the target level include a seventh feature whose feature missing rate is less than or equal to a preset missing rate; Determine an eighth feature in which the same rate of feature values ​​included in the features of the samples whose risk level is the target level is less than or equal to a preset same value rate; Determine a ninth feature whose information value index included in the features of the sample whose risk level is the target level is greater than a preset index; Determine the seventh feature, the eighth feature, and the ninth feature as a second feature set; The features included in the second feature set are combined to obtain the second feature combination.

8. The method according to claim 7, characterized in that The features included in the second feature set are combined to obtain the second feature combination including: Determining a fourth feature combination included in the second feature set, wherein after the training model is trained by the fourth feature combination, a fourth score of the training model is greater than the first preset threshold; Setting a scoring threshold for the training model; Combining the features included in the fourth feature combination to obtain a fifth feature combination, wherein after training the training model by using the fifth feature combination, a trained model score obtained is greater than the score threshold; The fifth feature combination is determined as the second feature combination.

9. A device for determining data risk level, characterized in that: include: A determination module, used for inputting the feature combination of the data to be identified into the target model, so as to determine the target risk level of the data to be identified through the target model; The target model is trained in the following way: Acquire a sample data set, wherein the sample data set includes positive sample data whose event type is marked as the abnormal event type, negative sample data whose event type is marked as a non-abnormal event type, and diffusion sample data whose event type is not marked as the abnormal event type, wherein the data type of the diffusion sample data is the abnormal event type; Determine the first feature of the positive sample data, the second feature of the negative sample data, and the third feature of the diffusion sample data as a first feature set; Combining the features included in the first feature set to obtain a plurality of first feature combinations; Using the first feature combination to train an initial model to obtain a training model; When the first score of the training model is less than a first preset threshold, the training model is trained using a second feature combination, and the trained training model is determined as the target model, wherein the second feature combination is determined in the following manner: the risk level of the samples included in the sample data set is determined by the training model; and the features of the samples whose risk level is the target level are combined to obtain the second feature combination.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images