Individual identification level evaluation-based fraud-related number detection method and device, and medium

Through the combination of individual identification evaluation indicators and confusion matrix, the problem that traditional evaluation indicators cannot accurately reflect model performance in fraudulent phone number identification is solved, and more accurate performance evaluation and more realistic performance measurement are achieved, which is suitable for extreme imbalances. dataset.

CN118940158APending Publication Date: 2024-11-12广州市申迪计算机系统有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410964562.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional machine learning evaluation indicators cannot accurately reflect model performance in fraudulent phone number identification tasks, especially in the case of extreme uneven positive and negative samples, resulting in misjudgment and increased training costs.

Method used

Using an individual identification-level evaluation method, the LightGBM model is trained and evaluated through indicators such as individual identification-level accuracy, accuracy, recall, F1 score and early recall. The confusion matrix is ​​used to perform the classification detection of telephone numbers, considering The repetition of numbers and the impact of misclassification.

Benefits of technology

More accurately evaluate the overall classification accuracy of the model, provide more realistic performance measurement, and are not affected by the misleading high scores of a large number of non-fraud-related numbers. It is suitable for extremely unbalanced data sets, and comprehensively and objectively measure model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940158A_ABST
    Figure CN118940158A_ABST
Patent Text Reader

Abstract

The invention discloses a fraud-related number detection method and device based on individual identification level evaluation and a medium. The method comprises the following steps: acquiring a preset number set, and constructing a sample data set according to the preset number set; establishing individual identification level evaluation indexes; training the LightGBM model by taking the sample data set as input to obtain a trained LightGBM model; and using the trained LightGBM model to carry out classification detection evaluation on the phone numbers in the real scene, and judging whether the phone numbers belong to fraud-related numbers or not according to the individual identification level evaluation indexes of the phone numbers. According to the method, the overall classification accuracy of the individual identifiers can be evaluated more accurately by considering the repeatability of the numbers and the influence of misclassification. The method is particularly suitable for processing extremely unbalanced data sets such as fraud-related phone numbers and the like, more real performance measurement can be provided, misleading high scores cannot be displayed due to a large number of non-fraud-related numbers, and the model performance can be measured more comprehensively and objectively.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning and data analysis, and in particular to a method, device and medium for detecting fraudulent numbers based on individual identification level evaluation. Background Art

[0002] Machine learning models can automatically learn and extract effective features from large amounts of data. In various target classification and detection tasks, compared with traditional manual feature extraction methods, machine learning can automatically learn and adapt to various complex scenarios and target forms, and gradually adapt and improve the accuracy of classification and detection by learning a large amount of training data. After the machine learning model completes the classification and detection of the target, the classification and detection effect of the target is evaluated through indicators such as accuracy and recall rate, and the performance of the model is measured accordingly.

[0003] However, in specific tasks such as fraudulent phone number recognition, traditional evaluation indicators may not be able to fully and accurately reflect model performance due to the extreme imbalance of positive and negative samples. For example, fraudulent numbers (positive sample numbers) are extremely rare, and only when fraudulent activities actually occur or are reported, the relevant numbers are confirmed to be fraudulent; while non-fraudulent numbers (negative sample numbers) are extremely common, but suspected fraudulent numbers may also be mixed in when selecting samples. Therefore, in the phone number fraud detection scenario, if the model is evaluated according to the traditional machine learning evaluation method, it may not accurately reflect the performance of the model, resulting in misjudgment of the model's fraudulent number detection effect and increased training costs. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device and medium for detecting fraudulent numbers based on individual identification level evaluation.

[0005] The first aspect of the present invention provides a method for detecting fraudulent numbers based on individual identification level evaluation, comprising the following steps:

[0006] Obtain a preset number set, and construct a sample data set according to the preset number set; the preset number set includes at least one normal number and at least one fraudulent number;

[0007] Establishing individual identification level evaluation indicators; the individual identification level evaluation indicators specifically include individual identification level accuracy, individual identification level precision, individual identification level recall rate, individual identification level F1 score and individual identification level early recall rate;

[0008] The LightGBM model is trained using the sample data set as input to obtain a trained LightGBM model;

[0009] The trained LightGBM model is used to perform classification detection and evaluation on telephone numbers in real scenarios, and the effectiveness of the classification results of the trained LightGBM model is judged based on the individual identification-level evaluation indicators of the telephone numbers.

[0010] Furthermore, the normal number refers to a telephone number that is not suspected of fraud risk; the fraud-related number refers to a telephone number that is suspected of fraud risk.

[0011] Furthermore, constructing a sample data set according to a preset number set specifically includes the following steps:

[0012] Performing data cleaning on the telephone numbers in the preset number set; the data cleaning includes removing abnormal values ​​and missing values;

[0013] Constructing feature vectors for the telephone numbers in the preset number set based on the statistical features and time window features of the telephone numbers to form sample data;

[0014] The sample data is divided into a training set and a data set in a preset ratio, and the training set and the data set are used as sample data sets.

[0015] Furthermore, the individual identification level accuracy is used to characterize the accuracy of the model in classifying and detecting each telephone number;

[0016] The individual identification level accuracy is used to characterize the degree of error in the classification detection of each telephone number by the model;

[0017] The individual identifier level recall rate is used to characterize the effectiveness of the model's classification detection results for each phone number;

[0018] The individual identifier level F1 score is used to characterize the ability of the model to balance the individual identifier level precision and the individual identifier level recall when classifying and detecting telephone numbers;

[0019] The individual identifier-level early recall rate is used to characterize the ability of the model to classify and detect telephone numbers before a specified event time.

[0020] Furthermore, the LightGBM model uses a confusion matrix to classify and detect telephone numbers; wherein fraudulent numbers are treated as positive samples, and normal numbers are treated as negative samples; the confusion matrix includes a first area, a second area, a third area, and a fourth area; each telephone number is placed in one of the first area, the second area, the third area, and the fourth area after classification and detection by the LightGBM model;

[0021] The first area is an individual true positive area, which refers to an individual delivery area of ​​a telephone number that is actually a positive sample and is correctly predicted as a positive sample at least once in the classification test;

[0022] The second area is an individual false positive area, which refers to an individual delivery area of ​​a telephone number that is actually a negative sample but is mistakenly predicted as a positive sample at least once in the classification test;

[0023] The third area is an individual false negative area, which refers to an individual delivery area of ​​a telephone number that is actually a negative sample but is mistakenly predicted as a negative sample in each classification test;

[0024] The fourth area is an individual true negative area, which refers to an individual delivery area of ​​telephone numbers that are actually negative samples and are correctly predicted as negative samples in each classification test.

[0025] Further, the first region includes a first sub-region and a second sub-region;

[0026] The first sub-region is an individual advance true positive region, which refers to an individual delivery region of a telephone number that is actually a positive sample and has been correctly predicted as a positive sample in classification detection at least once before a specified event time;

[0027] The second sub-area is an individual non-advance true positive area, which refers to an individual delivery area of ​​a telephone number that is actually a positive sample and has not been correctly predicted as a positive sample in the classification test before the specified event time, but has been correctly predicted as a positive sample in the classification test at least once at the specified event time and after the specified event time.

[0028] Furthermore, the individual identification level evaluation index is calculated by the following steps:

[0029] Individual identification level accuracy: obtained by dividing the sum of the number of telephone numbers in the first area and the fourth area by the sum of the number of telephone numbers in the first area, the second area, the third area, and the fourth area;

[0030] Individual identification level accuracy: obtained by dividing the number of individual phone numbers in the first area by the sum of the number of individual phone numbers in the first area and the second area;

[0031] Individual identifier level recall rate: obtained by dividing the number of telephone number individuals in the first area by the sum of the number of telephone number individuals in the first area and the third area;

[0032] Individual identifier level F1 score: obtained by weighted average of individual identifier level precision and individual identifier level recall;

[0033] Individual identification level early recall rate: obtained by dividing the number of telephone number individuals in the first sub-region by the sum of the number of telephone number individuals in the first region and the third region.

[0034] Furthermore, the LightGBM model is trained using the sample data set as input to obtain a trained LightGBM model, specifically comprising the following steps:

[0035] Determine the key parameters of the LightGBM model, tune the key parameters of the model, and obtain multiple sets of parameter combinations;

[0036] Use the sample data set to train the LightGBM model under each parameter combination to obtain multiple sets of model prediction results;

[0037] Using the individual identification level evaluation index to evaluate the model prediction results, to obtain multiple groups of evaluation results;

[0038] The LightGBM model with the best parameter combination is selected as the LightGBM model to complete the training.

[0039] A second aspect of the present invention provides an electronic device, characterized in that it includes a processor and a memory;

[0040] The memory is used to store programs;

[0041] The processor executes the program to implement the method for detecting fraudulent numbers based on individual identification level evaluation.

[0042] A third aspect of the present invention provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method for detecting fraudulent numbers based on individual identification level evaluation.

[0043] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0044] The embodiments of the present invention have the following beneficial effects: the method, device and medium for detecting fraudulent numbers based on individual identification level evaluation provided by the present invention use confusion matrix to perform classification detection of fraudulent numbers, and use individual identification level evaluation indicators to evaluate model performance. By considering the impact of number repeatability and misclassification, the present invention can more accurately evaluate the overall classification accuracy of individual identifications. This is especially important for the detection of fraudulent phone numbers. The present invention is particularly suitable for processing extremely unbalanced data sets such as fraudulent phone numbers, and can provide a more realistic performance measurement, will not show misleading high scores due to a large number of non-fraudulent numbers, and can measure model performance more comprehensively and objectively.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 This is a diagram of the traditional confusion matrix used in traditional machine learning model classification detection.

[0048] Figure 2 It is a basic flow chart of a fraudulent number detection method based on individual identification level evaluation of the present invention.

[0049] Figure 3 It is a schematic diagram of a confusion matrix used in a fraudulent number detection method based on individual identification level evaluation of the present invention.

[0050] Figure 4 It is a schematic diagram of an electronic device of the present invention.

[0051] Figure 5 It is a schematic diagram of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Typical machine learning models rely on a series of traditional evaluation indicators to measure their performance, including accuracy, precision, recall, and F1 score. Figure 1 As shown in the figure, these indicators are usually clearly displayed through the confusion matrix. In the confusion matrix, TP (True Positive) represents the true positive example, which refers to the number of samples that are actually positive and correctly predicted as positive; TN (True Negative) represents the true negative example, which refers to the number of samples that are actually negative and correctly predicted as negative; FP (False Positive) represents the false positive example, which refers to the number of samples that are actually negative but wrongly predicted as positive; FN (False Negative) represents the false negative example, which refers to the number of samples that are actually positive but wrongly predicted as negative.

[0054] In the phone number fraud detection scenario, there are some obvious deficiencies when evaluating the classification detection effect of machine learning models:

[0055] (1) Poor evaluation effect on imbalanced data

[0056] Traditional evaluation metrics, such as accuracy, often show misleadingly high scores when faced with extremely unbalanced datasets. For example, in the task of identifying fraudulent numbers, the number of positive samples (fraudulent numbers) is very small, while the number of negative samples (non-fraudulent numbers) is huge. Even if the model fails to accurately identify most of the positive samples, it can still achieve a high accuracy by identifying all the negative samples. In this scenario, accuracy is not a suitable performance metric.

[0057] (2) Not considering the time when the fraudulent number was involved

[0058] Generally speaking, traditional evaluation indicators do not consider the time series characteristics of data, but are evaluated based on static samples. In the identification of fraudulent numbers, the pattern of fraudulent behavior may change over time, and the behavior of the same number at different times may have different results. For example, there are multiple sample records for the same number in the sample. This is because some users may conduct fraudulent activities on different users at different times, and the fraud is successful at some time points, but not at other time points. Sometimes, the fraud is successful on different users at multiple time points. In addition, there is a lag in the notification of fraudulent numbers, and sometimes fraudulent activities are still being carried out after the time of the case. In short, fraudulent users often have both successful fraud records and records of failed or continuous attempts. Traditional evaluation indicators cannot correctly reflect the exact time point of identification, so it is impossible to evaluate the recall rate of the model before and after the fraud and other time-related indicators. If the fraudulent number can be identified before the fraud, even if the behavior is not identified after the fraud, this also proves the effectiveness of the model, because once the fraudulent number is detected, it will be immediately shut down to prevent it from continuing to commit fraud.

[0059] (3) The mutual influence of multiple sample records of the same user is not considered

[0060] In phone number fraud detection, recall is particularly important because it measures the proportion of real fraudulent numbers that the model can identify. In traditional evaluation indicators, a seemingly good recall rate may only be improved by identifying a large number of sample records of the same number, while the number of fraudulent numbers actually identified may not be high. Every real fraudulent number that is missed can have serious consequences. In addition, although a low precision rate may be shown in traditional indicators, most of the fraudulent users may have been identified. Because only a small number of sample records of fraudulent users may be accurately identified, while most of the sample records may not be fully identified, but in practice, once the identification is passed, it will be shut down. If it is not restored, no new call records will be generated. Therefore, traditional methods may underestimate the actual performance of the precision rate.

[0061] In view of the above shortcomings, the present invention aims to provide a fraudulent number detection method, device and medium based on individual identification level evaluation, which can more accurately reflect the actual effect of the telephone number recognition task by combining with traditional evaluation indicators.

[0062] like Figure 2 As shown, the first embodiment of the present invention provides a method for detecting fraudulent numbers based on individual identification level evaluation, comprising the following steps:

[0063] S1. Obtain a preset number set and construct a sample data set based on the preset number set;

[0064] S2. Establish individual identification level evaluation indicators; individual identification level evaluation indicators, specifically including individual identification level accuracy, individual identification level precision, individual identification level recall rate, individual identification level F1 score and individual identification level early recall rate;

[0065] S3. Train the LightGBM model with the sample data set as input to obtain a trained LightGBM model;

[0066] S4. Use the trained LightGBM model to classify, detect and evaluate phone numbers in real scenarios, and judge the effectiveness of the classification results of the trained LightGBM model based on the individual identification-level evaluation indicators of the phone numbers.

[0067] S1. Obtain a preset number set and construct a sample data set based on the preset number set.

[0068] In an embodiment of the present invention, the preset number set used as basic data should include at least one normal number and at least one fraudulent number; wherein a normal number refers to a phone number that is not suspected of fraud risk; a fraudulent number refers to a phone number that is suspected of fraud risk. In real scenarios, the number of normal numbers accounts for the majority, while the fraudulent numbers generally only include a few scattered numbers. In this scenario of extreme imbalance of positive and negative samples, the model performance cannot be effectively evaluated using conventional model performance evaluation indicators. At the same time, the multidimensional features of each phone number evolve over time to form time series data, which means that a fraudulent user may exhibit fraudulent features in different time periods, which means that a single number may correspond to multiple sample records when sampling, and each record maps the potential fraud behavior status at different time points.

[0069] Therefore, the sample records in the phone number fraud detection scenario are different from those in other scenarios. Normally, in other scenarios, there is only one sample record for each individual identifier, which may be a behavior vector of the user, representing a behavior pattern of the user. However, fraud detection involves time series data. For example, the call characteristics of a fraudulent user may be suspected of fraud for several days or even weeks, but the fraud has not been successful or has not been reported or has not been identified and shut down by the model. It is not until it is reported that it is confirmed to be a fraudulent number. Therefore, when sampling, the feature records of the fraudulent number within the suspicious time will be selected as samples, and there will be multiple sample records for the same user, making the evaluation of the model more complicated.

[0070] The embodiment of the present invention uses the LightGBM model to classify, detect and evaluate fraudulent numbers. The LightGBM model is a decision tree-based gradient boosting framework that aims to improve the accuracy and efficiency of machine learning algorithms. It uses a variety of optimization techniques, such as histogram algorithms, unilateral gradient sampling algorithms, and leaf-wise algorithms, to increase training speed and reduce memory consumption.

[0071] In the embodiment of the present invention, the preset number set cannot be directly used as the input data of the model, because the model cannot identify the feature vector of the preset number machine. Therefore, the embodiment of the present invention constructs a sample data set after performing feature engineering and other operations according to the preset number set. Constructing a sample data set according to the preset number set specifically includes the following steps:

[0072] S1-1. Perform data cleaning on the telephone numbers in the preset number set; data cleaning includes removing abnormal values ​​and missing values.

[0073] S1-2. Based on the statistical characteristics of the phone number (such as average call duration, call volume, etc.) and time window characteristics (such as call characteristics in the past week, month, etc.), construct feature vectors for the phone numbers in the preset number set to form sample data.

[0074] S1-3. Divide the sample data into a training set and a data set in a preset ratio, and use the training set and the data set as the sample data set. For example, the data set can be divided into a training set (80%) and a test set (20%), and the feature records of the same user number can only be in the same set.

[0075] S2. Establish individual identification level evaluation indicators.

[0076] Traditional classification detection model evaluation indicators include:

[0077] Accuracy: Accuracy measures the proportion of samples that are correctly predicted by the model to the total samples. In the traditional confusion matrix, its calculation formula is: Accuracy = (TP + TN) / (TP + TN + FP + FN).

[0078] Precision: Precision refers to the proportion of positive class prediction results that are actually positive, and is often used to evaluate the accuracy of the model's prediction of the positive class. In the traditional confusion matrix, its calculation formula is: Precision = (TP) / (TP + FP).

[0079] Recall: Recall measures the proportion of positive samples correctly identified by the model to all actual positive samples, and is used to evaluate the model's coverage of positive samples. In the traditional confusion matrix, its calculation formula is: Recall = (TP) / (TP + FN).

[0080] F1 score: The F1 score is the harmonic mean of precision and recall. In the traditional confusion matrix, the calculation formula is 2*(Precision*Recall) / (Precision+Recall). The F1 score uses the harmonic mean to force both precision and recall to be relatively high in order to get a high F1 score. The harmonic mean is different from the arithmetic mean or geometric mean, and it is more sensitive to lower values. Even if one of the indicators (precision or recall) is very low, the low value will significantly lower the overall F1 score.

[0081] In addition to traditional evaluation indicators, individual identification level evaluation indicators are further used to evaluate model performance. The individual identification level evaluation indicators specifically include individual identification level accuracy, individual identification level precision, individual identification level recall, individual identification level F1 score and individual identification level early recall.

[0082] Object-level accuracy: Object-level accuracy is used to characterize the accuracy of the model in classifying and detecting each phone number.

[0083] Object-level Precision: Object-level Precision is used to characterize the degree of error in the model's classification of each phone number.

[0084] Object-level recall: Object-level recall is used to characterize the effectiveness of the model’s classification and detection results for each phone number.

[0085] Object-level F1 Score: The object-level F1 score is used to characterize the model's ability to balance the precision and recall of the object-level when classifying and detecting phone numbers.

[0086] Early Object-level Recall: Early Object-level Recall is used to characterize the model’s ability to classify and detect phone numbers before a specified event time.

[0087] Through the improvement of the above evaluation indicators, the present invention can provide a more accurate and practical evaluation system to meet the specific needs and challenges of modern phone number fraud detection tasks. These improved indicators are also helpful for developers to better understand and optimize their phone number recognition systems.

[0088] S3. Train the LightGBM model with the sample data set as input to obtain a trained LightGBM model. The embodiment of the present invention uses the LightGBM model to identify telephone fraud numbers, applies the above improved evaluation indicators to evaluate the model, and analyzes and compares. By using the modified evaluation indicators, the recognition effect of fraudulent numbers can be more accurately evaluated, while the problem of misclassification of numbers is more comprehensively considered. Specifically, the LightGBM model of the present invention uses a confusion matrix to perform classification detection of telephone numbers; wherein, fraudulent numbers are treated as positive samples, and normal numbers are treated as negative samples. Figure 3 As shown, the evaluation index of the embodiment of the present invention is calculated based on each fraudulent number individual. As long as the positive sample individual (fraudulent number) in the sample has been correctly classified, it is considered to be correctly identified, and as long as the negative sample individual (normal number) has been incorrectly classified, it is considered to be incorrectly identified. The confusion matrix includes the first area, the second area, the third area and the fourth area; each phone number is placed in one of the first area, the second area, the third area and the fourth area after classification and detection by the LightGBM model.

[0089] The first area is the individual true positive area Object-level True Positives (OL-TP), which refers to the individual delivery area of ​​telephone numbers that are actually positive samples (Positive) and have been correctly predicted as positive samples at least once in the classification test;

[0090] The second area is the Object-level False Positives (OL-FP) area, which refers to the individual phone number placement area that is actually a negative sample but was mistakenly predicted as a positive sample at least once in the classification test;

[0091] The third area is the Object-level False Negatives (OL-FN) area, which refers to the individual phone number placement area that is actually a negative sample but is mistakenly predicted as a negative sample in each classification test;

[0092] The fourth area is the individual true negative area Object-level True Negatives (OL-TN), which refers to the individual telephone number delivery area that is actually a negative sample (Negative) and is correctly predicted as a negative sample in each classification test.

[0093] Further, the first region includes a first sub-region and a second sub-region;

[0094] The first sub-region is the Object-level Early True Positives (OLE-TP) region, which refers to the individual delivery region of telephone numbers that are actually positive samples and have been correctly predicted as positive samples in classification detection at least once before the specified event time;

[0095] The second sub-region is the individual non-early true positive region Object-level Late True Positives (OLL-TP), which refers to the individual delivery region of telephone numbers that are actually positive samples and have not been correctly predicted as positive samples in the classification test before the specified event time, but have been correctly predicted as positive samples in the classification test at least once at the specified event time and after the specified event time.

[0096] The individual identification level evaluation index provided by the embodiment of the present invention is Figure 3 The confusion matrix shown is calculated by the following steps:

[0097] Individual identification level accuracy: Taking into account the impact of number duplication and misclassification, this indicator aims to evaluate the overall classification accuracy of individual identification. Its calculation formula is:

[0098]

[0099] In the phone number fraud detection scenario, this is equivalent to:

[0100]

[0101] In the improved confusion matrix, this is equivalent to:

[0102]

[0103] Individual identification level accuracy: Considering the importance of correct classification and the necessity of avoiding incorrect classification, its calculation formula is:

[0104]

[0105] In the phone number fraud detection scenario, this is equivalent to:

[0106]

[0107] In the improved confusion matrix, this is equivalent to:

[0108]

[0109] Individual identification level recall rate: Considering the need to correctly identify all positive sample individuals as much as possible, the calculation formula is:

[0110]

[0111] In the phone number fraud detection scenario, this is equivalent to:

[0112]

[0113] In the improved confusion matrix, this is equivalent to:

[0114]

[0115] Individual identification level F1 score: It is the harmonic mean of individual identification level precision and individual identification level recall, and its calculation formula is:

[0116]

[0117] Individual identification level early recall rate: Considering the importance of timely identifying fraudulent numbers before fraud occurs, the calculation formula is:

[0118]

[0119] In the phone number fraud detection scenario, this is equivalent to:

[0120]

[0121] In the improved confusion matrix, this is equivalent to:

[0122]

[0123] In step S3, the LightGBM model is trained with the sample data set as input to obtain a trained LightGBM model, which specifically includes the following steps:

[0124] S3-1. Determine the key parameters of the LightGBM model, tune the key parameters of the model, and obtain multiple sets of parameter combinations.

[0125] S3-2. Use the sample data set to train the LightGBM model under each set of parameter combinations to obtain multiple sets of model prediction results.

[0126] S3-3. Use individual identification level evaluation indicators to evaluate the model prediction results and obtain multiple groups of evaluation results.

[0127] S3-4. Select the LightGBM model with the best parameter combination as the LightGBM model to complete the training.

[0128] The present invention mainly focuses on the following key parameters for tuning:

[0129] n_estimators: The number of decision trees, the number of weak learners used in ensemble learning.

[0130] num_leaves: The maximum number of leaf nodes, which directly affects the complexity and training speed of the model.

[0131] max_depth: Maximum depth of the tree.

[0132] learning_rate: The step size for updating weights at each iteration.

[0133] min_child_samples: The sum of the minimum sample weights on the leaf nodes.

[0134] feature_fraction: The feature ratio randomly sampled by each tree to prevent overfitting.

[0135] bagging_fraction: The sampling ratio of each tree training subsample.

[0136] class_weight: Class weight, used to deal with class imbalance, and can automatically or manually adjust class importance.

[0137] In the tuning method, the embodiment of the present invention adopts a combination of grid search and cross validation to traverse the pre-set parameter interval to determine the optimal parameter combination. After tuning the parameters, the model evaluation index results of several groups of parameter values ​​are finally selected for comparison.

[0138] S4. Use the trained LightGBM model to classify, detect and evaluate phone numbers in real scenarios, and judge the effectiveness of the classification results of the trained LightGBM model based on the individual identification-level evaluation indicators of the phone numbers.

[0139] Traditional evaluation indicators focus more on the performance of the overall classification, while the evaluation indicators established in the embodiments of the present invention focus more on the identification of each individual, especially the identification of key individuals (such as fraudulent numbers). The individual identification level accuracy and precision provide in-depth insights into the accuracy and missed detection rate of the model in identifying specific categories. The recall rate and early recall rate focus more on timeliness and effectiveness. The early recall rate measures the proportion of correct judgments that the model can make before the fraudulent behavior occurs. The F1 score can also reflect the overall performance of the model at the individual level.

[0140] The traditional evaluation indicators and the evaluation indicators established in the embodiments of the present invention have their own focuses, and their combined use can more comprehensively evaluate the model performance. The traditional evaluation indicators can help us understand the overall performance of the model, while the improved indicators can reveal the model's recognition ability on key groups or events. For example, if the traditional accuracy is high, but the individual identification level recall is low, this may mean that the model has problems dealing with a few important categories. Conversely, if both the individual identification level accuracy and recall are high, but the individual identification level precision is low, it means that the model may be too inclined to predict positive examples, resulting in more misjudgments.

[0141] The following is a specific implementation example of phone number fraud detection. The preset number set contains 36,437 sample records, including 601 fraudulent numbers (positive sample numbers), 3,211 positive sample records, 5,859 complaint numbers (negative sample numbers), and 33,226 negative sample records, which fully reflects the extreme imbalance of the data. Based on the above data set, the performance of the model under multiple sets of key parameters is evaluated, and the following results are obtained:

[0142]

[0143]

[0144] From the overall confusion matrix and the individual identification-level confusion matrix, we can see that there are a total of 188+66+34+1507=1795 sample records and 77+19+22+619=737 user numbers in the test set, of which there are 254 positive sample records and 96 positive sample numbers.

[0145] Take the fourth group of experiments as an example for detailed analysis:

[0146] Accuracy: 94.43% means that the model correctly predicted 94.43% of the overall samples, including both positive and negative samples. However, since the individual identification level accuracy is 93.89%, this indicates that despite the overall good performance, the model may not be accurate enough in identifying specific user numbers.

[0147] Precision: 82.63% means that 82.63% of the sample records predicted by the model are indeed feature records of fraudulent numbers, but 17.37% of the feature records are misjudged as fraudulent feature records. The individual identification level precision is even lower at 72.97%, indicating that the model misjudged more negative sample numbers as fraudulent in some cases.

[0148] Recall rate: 76.77% means that the model correctly identified 76.77% of the feature records of actual fraudulent numbers, but the individual identification-level recall rate was as high as 84.38%, indicating that 84.38% of the fraudulent numbers were correctly identified. In particular, the early recall rate (81.25%) means that the model can handle the model identification before it is involved in a case to reduce subsequent losses. However, only looking at the recall rate tends to underestimate the model's ability to identify fraud.

[0149] F1 score: This is a weighted average of precision and recall, and the F1 score of 79.59% reflects the model's performance in balancing precision and recall. The individual identifier-level F1 score of 78.26% is slightly lower than the overall F1 score, but the individual-level evaluation better reflects the model's ability to distinguish individual user numbers.

[0150] Through the first, second, third, fifth and sixth groups of experiments, the individual identification level accuracy, individual identification level recall and individual identification level F1 score are mostly higher than the traditional accuracy, recall and F1 score, and the individual identification level precision is lower than the traditional precision. This is mainly because as long as there is one negative sample number that has been misclassified in the individual, it is considered to be misclassified. Compared with the indicators in the overall confusion matrix, the indicators at the individual identification level are more stringent for negative sample numbers, because it requires that the prediction of the feature record of each negative sample number must be accurate to be considered correctly classified, reflecting the strict standards of the recognition system; while it is more relaxed for positive sample numbers, because as long as it is correctly predicted once, it is considered to be correctly identified as the fraudulent number, and at the same time, the early recall rate is increased to examine the possibility that the system can effectively identify before the actual fraudulent behavior occurs. This high proportion of early identification is crucial, because timely identification of fraudulent numbers can quickly take measures, such as shutting down and disposing, thereby effectively preventing potential economic losses and fraudulent activities, and generally reflects the sensitivity of the system to potential fraudulent behavior.

[0151] From the case analysis, we can see that traditional indicators may sometimes not fully reflect the performance of the model in a specific dimension. For example, the individual identification level recall rate is higher than the traditional recall rate, indicating that the model's ability to identify potential fraudulent numbers is underestimated by traditional indicators. Conversely, the individual identification level precision rate is lower than the traditional precision rate, suggesting that the model may have false positives on some individuals, which is a problem that traditional indicators may cover up. In summary, the combination of traditional indicators and improved individual identification level indicators can provide a more three-dimensional and comprehensive model evaluation perspective, which not only helps to understand the model's macro classification performance, but also deeply reveals the model's performance in key individual identification, thereby guiding the further optimization of the model and risk control strategies in practical applications.

[0152] Figure 4 : is a schematic diagram of the structure of the electronic device proposed in the second embodiment of the present invention. In the memory of this embodiment, program instructions for implementing the fraudulent number detection method based on individual identification level evaluation of any of the above embodiments are stored. The processor is used to execute the program instructions stored in the memory to perform fraudulent number detection based on individual identification level evaluation. Among them, the processor can also be called CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0153] The contents of the method in the first embodiment of the present invention are all applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0154] Figure 5It is a schematic diagram of the structure of the computer-readable storage medium of the third embodiment of the present invention. The computer-readable storage medium of the fourth embodiment of the present invention stores program instructions that can implement the above-mentioned fraudulent number detection method based on individual identification level evaluation, wherein the program instructions can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned computer-readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0155] The contents of the method in the first embodiment of the present invention are all applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0156] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the fraudulent number detection method based on individual identification level evaluation provided in the above embodiment.

[0157] Those skilled in the art will appreciate that the modules in the devices in the embodiments of the present invention may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments of the present invention may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including corresponding claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0158] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0159] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0160] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to method embodiments, the relevant parts can refer to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed in multiple places, such as nodes of a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying creative labor.

[0161] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0163] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0164] In the embodiments of the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present invention may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context in the specific embodiment.

[0165] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations of the present invention, and those of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention. Those skilled in the art will readily come to think of other embodiments of the present invention after considering the specification and practicing the present invention. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

Claims

1. A fraudulent number detection method based on individual identification level evaluation, It is characterized in that The following steps are involved: Obtain a preset number set, and construct a sample data set according to the preset number set; the preset number set includes at least one normal number and at least one fraudulent number; Establishing individual identification level evaluation indicators; the individual identification level evaluation indicators specifically include individual identification level accuracy, individual identification level precision, individual identification level recall rate, individual identification level F1 score and individual identification level early recall rate; The LightGBM model is trained using the sample data set as input to obtain a trained LightGBM model; The trained LightGBM model is used to perform classification detection and evaluation on telephone numbers in real scenarios, and the effectiveness of the classification results of the trained LightGBM model is judged based on the individual identification-level evaluation indicators of the telephone numbers.

2. According to claim 1, a method for detecting fraudulent numbers based on individual identification level evaluation, It is characterized in that The normal number refers to a telephone number that is not suspected of fraud risk; the fraud-related number refers to a telephone number that is suspected of fraud risk.

3. According to claim 1, a method for detecting fraudulent numbers based on individual identification level evaluation, It is characterized in that The step of constructing a sample data set according to a preset number set specifically includes the following steps: Performing data cleaning on the telephone numbers in the preset number set; the data cleaning includes removing abnormal values ​​and missing values; Constructing feature vectors for the telephone numbers in the preset number set based on the statistical features and time window features of the telephone numbers to form sample data; The sample data is divided into a training set and a data set in a preset ratio, and the training set and the data set are used as sample data sets.

4. According to claim 1, a method for detecting fraudulent numbers based on individual identification level evaluation, It is characterized in that The individual identification level accuracy is used to characterize the accuracy of the model in classifying and detecting each telephone number; The individual identification level accuracy is used to characterize the degree of error in the classification detection of each telephone number by the model; The individual identifier level recall rate is used to characterize the effectiveness of the model's classification detection results for each phone number; The individual identifier level F1 score is used to characterize the ability of the model to balance the individual identifier level precision and the individual identifier level recall when classifying and detecting telephone numbers; The individual identifier-level early recall rate is used to characterize the ability of the model to classify and detect telephone numbers before a specified event time.

5. According to claim 1, a method for detecting fraudulent numbers based on individual identification level evaluation, It is characterized in that The LightGBM model uses a confusion matrix to classify and detect phone numbers; wherein fraudulent numbers are treated as positive samples, and normal numbers are treated as negative samples; the confusion matrix includes a first area, a second area, a third area, and a fourth area; each phone number is placed in one of the first area, the second area, the third area, and the fourth area after classification and detection by the LightGBM model; The first area is an individual true positive area, which refers to an individual delivery area of ​​a telephone number that is actually a positive sample and is correctly predicted as a positive sample at least once in the classification test; The second area is an individual false positive area, which refers to an individual delivery area of ​​a telephone number that is actually a negative sample but is mistakenly predicted as a positive sample at least once in the classification test; The third area is an individual false negative area, which refers to an individual delivery area of ​​a telephone number that is actually a negative sample but is mistakenly predicted as a negative sample in each classification test; The fourth area is an individual true negative area, which refers to an individual delivery area of ​​telephone numbers that are actually negative samples and are correctly predicted as negative samples in each classification test.

6. A method for detecting fraudulent numbers based on individual identification level evaluation according to claim 5, It is characterized in that The first region includes a first sub-region and a second sub-region; The first sub-region is an individual advance true positive region, which refers to an individual delivery region of a telephone number that is actually a positive sample and has been correctly predicted as a positive sample in classification detection at least once before a specified event time; The second sub-area is an individual non-advance true positive area, which refers to an individual delivery area of ​​a telephone number that is actually a positive sample and has not been correctly predicted as a positive sample in the classification test before the specified event time, but has been correctly predicted as a positive sample in the classification test at least once at the specified event time and after the specified event time.

7. A method for detecting fraudulent numbers based on individual identification level evaluation according to claim 6, It is characterized in that The individual identification level evaluation index is calculated by the following steps: Individual identification level accuracy: obtained by dividing the sum of the number of telephone numbers in the first area and the fourth area by the sum of the number of telephone numbers in the first area, the second area, the third area, and the fourth area; Individual identification level accuracy: obtained by dividing the number of individual phone numbers in the first area by the sum of the number of individual phone numbers in the first area and the second area; Individual identifier level recall rate: obtained by dividing the number of telephone number individuals in the first area by the sum of the number of telephone number individuals in the first area and the third area; Individual identifier level F1 score: obtained by weighted average of individual identifier level precision and individual identifier level recall; Individual identification level early recall rate: obtained by dividing the number of telephone number individuals in the first sub-region by the sum of the number of telephone number individuals in the first region and the third region.

8. According to claim 1, a method for detecting fraudulent numbers based on individual identification level evaluation, It is characterized in that The LightGBM model is trained using the sample data set as input to obtain a trained LightGBM model, specifically comprising the following steps: Determine the key parameters of the LightGBM model, tune the key parameters of the model, and obtain multiple sets of parameter combinations; Use the sample data set to train the LightGBM model under each parameter combination to obtain multiple sets of model prediction results; Using the individual identification level evaluation index to evaluate the model prediction results, to obtain multiple groups of evaluation results; The LightGBM model with the best parameter combination is selected as the LightGBM model to complete the training.

9. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store programs; The processor executes the program to implement a fraudulent number detection method based on individual identification level evaluation as described in any one of claims 1-8.

10. A computer-readable storage medium, It is characterized in that The storage medium stores a program, and the program is executed by a processor to implement a fraudulent number detection method based on individual identification level evaluation as described in any one of claims 1 to 8.