Dynamic confrontation evaluation method and device for electric fraud model and readable storage medium

Abstract

The invention provides an electric fraud model dynamic confrontation evaluation method and device and a readable storage medium. The method comprises the steps of obtaining source data related to an electric fraud; based on the source data, multi-dimensional dynamic index calculation is carried out on the electric fraud model, dynamic indexes and weighted comprehensive scores of the electric fraud model are obtained, and the dynamic indexes comprise a confrontation sensitivity index, a dynamic adaptability index and a user experience influence index; generating an adversarial sample based on the source data, and performing a robustness test on the electric fraud model through the adversarial sample to obtain a robustness score of the electric fraud model; and establishing a loss function in which the false alarm rate of the electric fraud model is associated with the actual service loss, and performing optimal parameter configuration adjustment on the electric fraud model through the loss function. According to the method, the limitation of an existing telecommunication operator electric fraud model evaluation method is solved, the performance of the electric fraud model in a real scene can be comprehensively evaluated, a scientific model optimization basis is provided for an operator, and the telecommunication network fraud risk is remarkably reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dynamic countermeasure evaluation method, device and readable storage medium for telecommunications fraud models. Background Technology

[0002] Telecommunications fraud (referred to as "telecom fraud") refers to criminal acts that use telephone, internet and text messages to fabricate false information, set up scams, and remotely and non-contactly defraud victims, inducing them to make payments or transfers.

[0003] Telecom fraud is characterized by three main features: strong adversarial nature (rapidly iterating fraud methods), high concealment (blurred boundaries between fraudulent behavior and normal transactions), and dynamism (criminal characteristics change with time, region, and target groups). This places specific demands on the evaluation system for telecom fraud models: a balance must be struck between sensitivity to new types of fraud (avoiding missed detections that lead to user losses); adaptation to the dynamic evolution of fraud patterns (avoiding model failure due to fixed characteristics); and a comprehensive assessment of the ability to protect user privacy (avoiding misjudging normal transactions and impacting user experience).

[0004] However, existing methods for evaluating the effectiveness of telecom fraud models cannot meet the aforementioned specific needs. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a dynamic countermeasure evaluation method, device and readable storage medium for telecommunications fraud models, which addresses the above-mentioned shortcomings of the prior art and solves the problems existing in the prior art.

[0006] Firstly, this application provides a dynamic countermeasure evaluation method for a telecom fraud model, the method comprising:

[0007] S1. Obtain source data related to telecommunications fraud;

[0008] S2. Based on the source data, perform multi-dimensional dynamic index calculation on the telecom fraud model to obtain the dynamic index and weighted comprehensive score of the telecom fraud model. The dynamic index includes the confrontation sensitivity index, the dynamic adaptability index, and the user experience impact index.

[0009] S3. Generate adversarial examples based on the source data, and conduct robustness tests on the telecom fraud model using the adversarial examples to obtain a robustness score for the telecom fraud model.

[0010] S4. Establish a loss function that correlates the false alarm rate of the telecom fraud model with the actual business loss, with minimizing business loss as the optimization objective, and adjust the optimal parameter configuration of the telecom fraud model through the loss function.

[0011] In some embodiments, the source data includes at least one of call logs, SMS content, a fraud case database, and user behavior logs;

[0012] The call records include at least one of the following: caller ID, called number, call duration, roaming route, and call frequency.

[0013] The SMS content includes at least one of the following: SMS text, sending time, sending frequency, and keywords;

[0014] The fraud case database includes information on fraud types and timestamps.

[0015] The user behavior log includes at least one of the following: user application (APP) usage behavior, traffic distribution, and device information.

[0016] In some embodiments, S2, the calculation process of the adversarial sensitivity index includes:

[0017] Based on the source data, normal samples and adversarial samples are obtained;

[0018] The normal samples and adversarial samples are input into the fraud model to obtain the normal sample accuracy of the fraud model in identifying fraud based on the normal samples, and the adversarial sample accuracy in identifying fraud based on the adversarial samples.

[0019] Based on the accuracy of normal samples and the accuracy of adversarial samples, the adversarial sensitivity index is determined using the following formula:

[0020] .

[0021] In some embodiments, in S2, the calculation process of the dynamic adaptability index includes:

[0022] The fraud model is trained based on historical fraud data in the source data to obtain the historical fraud accuracy of the fraud model in identifying fraud based on the historical fraud data.

[0023] The trained fraud model is used to identify new types of fraud data, and the accuracy of the trained fraud model in identifying new types of fraud data is obtained. The fraud type of the new types of fraud data is different from the fraud type of the historical fraud data.

[0024] Based on the historical fraud accuracy rate and the new fraud accuracy rate, the dynamic adaptability index is determined using the following formula:

[0025] .

[0026] In some embodiments, in S2, the calculation process of the user experience impact index includes:

[0027] The false alarm rate, user complaint rate, and manual review cost of the fraud model are obtained based on historical data statistics.

[0028] Based on the false positive rate, user complaint rate, and manual review cost, the user experience impact index is determined using the following formula:

[0029] ;

[0030] in, Indicates the false alarm rate. Indicates the user complaint rate. This indicates the cost of manual review.

[0031] In some embodiments, S3, the robustness test of the telecom fraud model is performed using adversarial examples to obtain a robustness score for the telecom fraud model, including:

[0032] Normal samples and adversarial samples are input into the fraud model to obtain the normal sample misclassification rate of the fraud model for identifying fraud in the normal samples, and the adversarial sample misclassification rate for identifying fraud in the adversarial samples.

[0033] The robustness score is determined using the following formula based on the normal sample misclassification rate and the adversarial sample misclassification rate:

[0034] ;

[0035] in, This represents the false positive rate for normal samples. This represents the adversarial sample misclassification rate.

[0036] In some embodiments, in S4, the loss function is:

[0037] ;

[0038] in, Indicates the false alarm rate. Indicates the user complaint rate. This indicates the cost of manual review. This represents business loss, and α represents the penalty coefficient for underreporting.

[0039] Secondly, this application provides a dynamic anti-fraud evaluation device for a telecom fraud model, the device comprising:

[0040] The data acquisition module is configured to acquire source data related to telecommunications fraud.

[0041] The dynamic indicator calculation module is configured to perform multi-dimensional dynamic indicator calculations on the telecom fraud model based on the source data, and obtain the dynamic indicators and weighted comprehensive score of the telecom fraud model. The dynamic indicators include the confrontation sensitivity indicator, the dynamic adaptability indicator, and the user experience impact indicator.

[0042] The adversarial testing module is configured to generate adversarial samples based on the source data, and to perform robustness testing on the telecom fraud model using the adversarial samples to obtain a robustness score for the telecom fraud model.

[0043] The false alarm cost quantification module is configured to establish a loss function that correlates the false alarm rate of the telecom fraud model with the actual business loss, with the optimization objective of minimizing business loss, and adjusts the optimal parameter configuration of the telecom fraud model through the loss function.

[0044] Thirdly, this application provides a dynamic countermeasure evaluation device for a telecom fraud model, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to implement the dynamic countermeasure evaluation method for a telecom fraud model described in the first aspect.

[0045] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the dynamic countermeasure evaluation method for telecom fraud models described in the first aspect.

[0046] This application provides a dynamic adversarial evaluation method, apparatus, and readable storage medium for telecom fraud models. The method includes: acquiring source data related to telecom fraud; calculating multi-dimensional dynamic indicators for the telecom fraud model based on the source data to obtain dynamic indicators and a weighted comprehensive score for the telecom fraud model, wherein the dynamic indicators include adversarial sensitivity indicators, dynamic adaptability indicators, and user experience impact indicators; generating adversarial samples based on the source data, and conducting robustness tests on the telecom fraud model using the adversarial samples to obtain a robustness score for the telecom fraud model; establishing a loss function that correlates the false alarm rate of the telecom fraud model with actual business losses, with minimizing business losses as the optimization objective, and adjusting the optimal parameter configuration of the telecom fraud model through the loss function. This application provides a dynamic adversarial evaluation method for telecom fraud models, which, by constructing a multi-dimensional dynamic indicator system, designing an adversarial testing mechanism, and quantifying false alarm costs, covers three core indicators: adversarial sensitivity, dynamic adaptability, and user experience impact, thus overcoming the limitations of existing telecom operator telecom fraud model evaluation methods. This application can comprehensively evaluate the performance of telecom fraud models in real-world scenarios, providing operators with a scientific basis for model optimization and significantly reducing the risk of telecom network fraud. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 A flowchart of a dynamic countermeasure evaluation method for a telecom fraud model provided in this application embodiment;

[0049] Figure 2 This is a schematic diagram of the structure of a dynamic countermeasure evaluation device for a telecommunications fraud model provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of another dynamic anti-fraud evaluation device for a telecom fraud model provided in an embodiment of this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solution of this application, the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0053] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining this application and are not intended to limit this application.

[0054] It is understood that, without conflict, the various embodiments and features in the embodiments of this application can be combined with each other.

[0055] It is understood that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, while parts unrelated to this application are not shown in the drawings.

[0056] It is understood that each unit or module involved in the embodiments of this application may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0057] It is understood that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0058] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this application may occur in a different order than those marked in the accompanying drawings.

[0059] It is understood that the flowcharts and block diagrams of this application illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this application. Each block in a flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagrams and flowcharts may be implemented using a hardware-based system to implement the specified function, or using a combination of hardware and computer instructions.

[0060] It is understood that the units and modules involved in the embodiments of this application can be implemented by software or by hardware. For example, the units and modules can be located in the processor.

[0061] It is understood that the specific values ​​of each parameter in this application are merely illustrative examples, and in practical applications, the parameters can be optimized and adjusted based on specific requirements.

[0062] Existing technologies primarily focus on model building and detection (such as neural networks, deep learning classifiers, and behavioral analysis), but research on methods for evaluating model performance remains significantly insufficient. Although some solutions propose multi-dimensional risk index assessment systems, their core objective is to construct a framework for preventing and controlling telecom fraud, rather than to systematically evaluate the model's performance, robustness, or generalization ability.

[0063] Specifically, existing telecom fraud models mainly rely on general machine learning evaluation metrics (such as accuracy, recall, and F1 score), but these metrics have significant shortcomings in telecom fraud scenarios:

[0064] 1. Accuracy failure: The sample size of telecom fraud is naturally sparse (fraudulent transactions usually account for less than 1%). The model can obtain extremely high accuracy (such as 99%) by simply judging "all transactions are normal", but it completely loses the ability to identify fraud.

[0065] 2. One-sidedness of recall rate: Simply pursuing a high recall rate (such as detecting 90% of fraudulent transactions) may lead to a large number of normal transactions being misjudged as fraudulent (high false alarm rate), resulting in user complaints, a surge in manual review costs, and even triggering the "boy who cried wolf" effect in the risk control system (users becoming desensitized to warnings).

[0066] 3. F1 score cannot reflect dynamic adversarial response: The F1 score is the harmonic mean of precision and recall, but its calculation is based on static data distribution and cannot measure the model's adaptability to new fraud methods (for example, when fraudsters switch to "cryptocurrency money laundering" mode, the model performance may drop sharply, but the F1 score cannot capture this change in time).

[0067] Besides the shortcomings of the indicators, traditional evaluation methods also have the following problems:

[0068] (1) Lack of adversarial testing: The counter-investigation capabilities of fraudsters (such as evading detection by forging transaction links and confusing behavioral characteristics) were not considered, resulting in the model's distorted performance in real adversarial scenarios;

[0069] (2) Ignoring the impact on user experience: Only focusing on the model's ability to identify fraud, without quantifying the negative impact of false alarms on user trust and platform reputation (for example, frequent false alarms may lead to user churn).

[0070] (3) Static evaluation is out of touch with dynamic environment: It relies on historical labeled data for training and testing, but the characteristics of telecom fraud evolve rapidly over time (such as from "fake customer service" to "brushing orders and rebates"), and static indicators cannot reflect the continuous effectiveness of the model in real dynamic environment.

[0071] To address the aforementioned issues, this application proposes a dynamic adversarial evaluation method for telecom fraud models. By constructing a multi-dimensional dynamic indicator system, designing an adversarial testing mechanism, and quantifying the marginal cost of false positives, the method comprehensively evaluates the performance of the model in real-world scenarios.

[0072] The main concepts of this application include:

[0073] Introducing a multi-dimensional dynamic indicator system: Combining factors such as the evolution trend of fraud methods, user behavior drift, and false alarm costs, a comprehensive evaluation framework covering "anti-fraud sensitivity," "dynamic adaptability," and "user experience impact" is constructed.

[0074] Design an adversarial testing mechanism: simulate the counter-reconnaissance capabilities of fraudsters (such as feature obfuscation and link forgery) and evaluate the robustness of the model in real attack and defense scenarios;

[0075] Quantifying the marginal cost of false positives: By using data such as user complaint rates and manual review load, a correlation model between false positive rates and actual business losses is established to avoid the negative effects caused by simply pursuing high recall rates.

[0076] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0077] This application provides a dynamic countermeasure evaluation method for a telecommunications fraud model. The workflow of this method can be implemented by electronic devices, such as computers and handheld smart terminals. For ease of explanation, the embodiments of this application are described with the computer as the subject of the method execution.

[0078] Figure 1This is a schematic diagram of the dynamic countermeasure evaluation method for the telecom fraud model provided in the embodiments of this application, as shown below. Figure 1 As shown, this application provides a dynamic countermeasure evaluation method for a telecom fraud model, the method comprising S1-S4, as follows:

[0079] S1. Obtain source data related to telecommunications fraud;

[0080] In some embodiments, the source data includes at least one of call logs, SMS content, a fraud case database, and user behavior logs;

[0081] The call records include at least one of the following: caller ID, called number, call duration, roaming route, and call frequency.

[0082] The SMS content includes at least one of the following: SMS text, sending time, sending frequency, and keywords;

[0083] The fraud case database includes information on fraud types and timestamps.

[0084] The user behavior log includes at least one of the following: user application (APP) usage behavior, traffic distribution, and device information.

[0085] Specifically, when collecting data related to telecommunications fraud, this application includes the following source data:

[0086] Call logs include caller ID, called number, call duration, roaming history, and call frequency.

[0087] SMS content: SMS text, sending time, sending frequency, keywords (such as "secure account" or "winning");

[0088] Fraud Case Database: Marks fraud types (such as impersonating law enforcement officials, fake online shopping scams, and fraudulent investments) and timestamps;

[0089] User behavior logs: User app usage behavior, traffic distribution, device information, etc.

[0090] Optionally, after obtaining the source data related to telecom fraud, this application also includes preprocessing the source data, including: cleaning the data (such as handling duplicate data, handling missing values, and handling invalid / outlier values); standardizing the timestamp format; and dividing the data into training set, validation set, and test set.

[0091] The goal of data cleaning is to remove noise, errors, and inconsistencies from the source data to ensure data quality and reliability. A specific example is shown below:

[0092] Example 1: Handling Duplicate Data

[0093] The same phone call or text message may occur multiple times due to system resending, duplicate log recording, etc. By processing duplicate data, we can avoid the duplicate data interfering with model training and causing the model to overfit to these duplicate patterns.

[0094] Operations for handling duplicate data include:

[0095] (1) Precise deduplication: For records with all fields being exactly the same, duplicates are directly deleted;

[0096] (2) Deduplication of key business information: Sometimes the data collection time is different, but the core business information (such as caller ID, called ID, and call time) is the same. In this case, it is necessary to retain the latest or most complete record and delete other duplicates according to the business logic.

[0097] Example 2: Handling Missing Values

[0098] Some record fields may be empty (such as empty SMS content or missing call duration), so missing values ​​need to be handled.

[0099] Operations for handling missing values ​​include:

[0100] (1) Analyze the cause of the missing data: determine whether it is a random missing data or a systematic missing data (e.g., the base station data in some areas is not uploaded completely).

[0101] (2) Deletion: If a record has missing key fields (such as caller ID, timestamp) or a high missing rate, the record will be deleted directly.

[0102] (3) Fill / Mark:

[0103] Numerical fields (such as call duration): can be filled with the mean, median, or predicted values ​​based on other features.

[0104] Category fields (such as call type, location): can be filled with "unknown" or the most common category.

[0105] Text fields (such as SMS content): These are usually difficult to fill. If they are missing, they can be treated as a special category (such as "no content").

[0106] Example 3: Handling invalid / outlier values

[0107] The data contains records that clearly do not conform to logic or business rules, so invalid / outlier values ​​need to be handled.

[0108] Operations for handling invalid / outlier values ​​include:

[0109] (1) Logical verification: Filter out call records where the calling number and the called number are the same; identify and process records with abnormally long call durations (such as more than 24 hours) or negative values.

[0110] (2) Format verification: Check whether the mobile phone number format is correct (e.g., whether it is 11 digits); check whether the URL link format is standardized.

[0111] (3) Statistical outlier detection: For numerical fields (such as the number of calls per day, the number of SMS messages), use methods such as box plots or Z-Scores to identify and handle extreme outliers. These may be data entry errors or genuine fraudulent activities (requires judgment based on business needs).

[0112] After data cleaning, data transformation and standardization are performed. This step aims to convert the data into a format that is more suitable for machine learning models to process.

[0113] Specifically, the timestamps in the source data may come from different systems and have different formats (such as 2023-10-01 14:30:00, 01 / 10 / 2023 2:30PM, or Unix timestamps). Therefore, all timestamps are parsed into a unified format, such as YYYY-MM-DDHH:MM:SS or a datetime object in Python.

[0114] After standardizing the timestamp format, the dataset is partitioned to ensure the fairness and generalization ability of the model evaluation.

[0115] The partitioning strategy can be conventional, which involves randomly dividing the data into training, validation, and test sets. The training set is used for model training, allowing the model to learn patterns in the data. The validation set is used to adjust hyperparameters, select models, and stop early during training to prevent overfitting. The test set is used to evaluate the model's generalization performance and is completely invisible throughout the training and hyperparameter tuning process, simulating performance in real-world environments.

[0116] Datasets can also be partitioned proportionally, for example: training set: validation set: test set = 60%:20%:20% or 70%:15%:15%. When the amount of data is extremely large, the ratio of the test set to the validation set can be even smaller.

[0117] S2. Based on the source data, perform multi-dimensional dynamic index calculation on the telecom fraud model to obtain the dynamic index and weighted comprehensive score of the telecom fraud model. The dynamic index includes the confrontation sensitivity index, the dynamic adaptability index, and the user experience impact index.

[0118] Among them, the telecom fraud model refers to a model for identifying and classifying telecom fraud behavior.

[0119] In some embodiments, S2, the calculation process of the adversarial sensitivity index includes:

[0120] Based on the source data, normal samples and adversarial samples are obtained;

[0121] The normal samples and adversarial samples are input into the fraud model to obtain the normal sample accuracy of the fraud model in identifying fraud based on the normal samples, and the adversarial sample accuracy in identifying fraud based on the adversarial samples.

[0122] Based on the accuracy of normal samples and the accuracy of adversarial samples, the adversarial sensitivity index is determined using the following formula:

[0123] .

[0124] Among them, adversarial examples are generated in the following ways:

[0125] (1) Feature obfuscation: Add noise to normal call records (such as randomly modifying call duration or forging IP addresses) to generate adversarial samples.

[0126] (2) Link forgery: Simulate fraudsters evading detection by collaborating with multiple accounts (such as the call link of ABC).

[0127] This application calculates the adversarial sensitivity index to determine the rate of accuracy reduction of the telecom fraud model under adversarial samples.

[0128] In some embodiments, in S2, the calculation process of the dynamic adaptability index includes:

[0129] The fraud model is trained based on historical fraud data in the source data to obtain the historical fraud accuracy of the fraud model in identifying fraud based on the historical fraud data.

[0130] The trained fraud model is used to identify new types of fraud data, and the accuracy of the trained fraud model in identifying new types of fraud data is obtained. The fraud type of the new types of fraud data is different from the fraud type of the historical fraud data.

[0131] Based on the historical fraud accuracy rate and the new fraud accuracy rate, the dynamic adaptability index is determined using the following formula:

[0132] .

[0133] Specifically, historical telecom fraud data can be related data on historical fraud types (such as "impersonating customer service" or "brushing orders for rebates"), while new telecom fraud data can be related data on new types of fraud (such as "cryptocurrency money laundering").

[0134] This application can perform fraud type transfer tests on the telecom fraud model by calculating dynamic adaptability indicators, and test the telecom fraud model's ability to identify new types of fraud.

[0135] In some embodiments, in S2, the calculation process of the user experience impact index includes:

[0136] The false alarm rate, user complaint rate, and manual review cost of the fraud model are obtained based on historical data statistics.

[0137] Based on the false positive rate, user complaint rate, and manual review cost, the user experience impact index is determined using the following formula:

[0138] ;

[0139] in, Indicates the false alarm rate. Indicates the user complaint rate. This indicates the cost of manual review.

[0140] In some embodiments, after obtaining the adversarial sensitivity index, dynamic adaptability index, and user experience impact index, a further enhanced summation is performed to obtain a comprehensive score.

[0141] Specifically, the weighted scoring formula is as follows:

[0142] Overall score = w1 * adversarial sensitivity + w2 * dynamic adaptability + w3 * user experience impact;

[0143] The weights can be adjusted according to business needs (e.g., w1=0.4, w2=0.3, w3=0.3).

[0144] This application can intuitively reflect the overall performance of the telecom fraud model in terms of adversarial nature, adaptability, and user experience by outputting a comprehensive score of the model.

[0145] S3. Generate adversarial examples based on the source data, and conduct robustness tests on the telecom fraud model using the adversarial examples to obtain a robustness score for the telecom fraud model.

[0146] In some embodiments, S3, the robustness test of the telecom fraud model is performed using adversarial examples to obtain a robustness score for the telecom fraud model, including:

[0147] Normal samples and adversarial samples are input into the fraud model to obtain the normal sample misclassification rate of the fraud model for identifying fraud in the normal samples, and the adversarial sample misclassification rate for identifying fraud in the adversarial samples.

[0148] The robustness score is determined using the following formula based on the normal sample misclassification rate and the adversarial sample misclassification rate:

[0149] ;

[0150] in, This represents the false positive rate for normal samples. This represents the adversarial sample misclassification rate.

[0151] Specifically, this application conducts adversarial tests on the telecom fraud model based on adversarial examples, and evaluates the robustness of the model by simulating the counter-investigation capabilities of fraudsters.

[0152] Adversarial examples include methods such as forging call links and obfuscating SMS features.

[0153] In this application, the model robustness testing process includes: injecting adversarial samples (accounting for 5%-10%) into the validation set; calculating the model's misclassification rate for adversarial samples; comparing the difference in misclassification rates between normal samples and adversarial samples to determine the robustness score.

[0154] This application quantifies the performance of telecom fraud models in real-world adversarial scenarios by determining a robustness score for the model.

[0155] S4. Establish a loss function that correlates the false alarm rate of the telecom fraud model with the actual business loss, with minimizing business loss as the optimization objective, and adjust the optimal parameter configuration of the telecom fraud model through the loss function.

[0156] Specifically, this application aims to establish a correlation model between false positive rate and actual business losses by quantifying the cost of false positives, thereby avoiding user churn caused by high false positive rates.

[0157] In some embodiments, in S4, the loss function is:

[0158] ;

[0159] in, Indicates the false alarm rate. Indicates the user complaint rate. This indicates the cost of manual review. This represents business loss, and α represents the penalty coefficient for underreporting.

[0160] The penalty coefficient for underreporting can be adjusted based on business risks.

[0161] When quantifying the cost of false positives, this application aims to minimize business losses. It seeks a balance point that minimizes business losses by adjusting the parameter thresholds of the telecom fraud model (such as the classification confidence threshold). Ultimately, it outputs the optimal threshold configuration, balancing the false positive rate and the false negative rate to reduce business losses.

[0162] This application provides a dynamic evaluation method for telecom fraud models. By constructing a multi-dimensional dynamic indicator system, designing an adversarial testing mechanism, and quantifying false alarm costs, it covers three core indicators: adversarial sensitivity, dynamic adaptability, and user experience impact. This method overcomes the limitations of existing telecom operator telecom fraud model evaluation methods. This application can comprehensively evaluate the performance of telecom fraud models in real-world scenarios, providing operators with a scientific basis for model optimization and significantly reducing the risk of telecom network fraud.

[0163] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0164] Figure 2 This is a schematic diagram of the dynamic anti-fraud evaluation device for the telecommunications fraud model provided in the embodiments of this application, as shown below. Figure 2 As shown, this application provides a dynamic anti-fraud evaluation device for a telecom fraud model, the device comprising:

[0165] Data acquisition module 11 is configured to acquire source data related to telecommunications fraud;

[0166] The dynamic indicator calculation module 12 is configured to perform multi-dimensional dynamic indicator calculation on the telecom fraud model based on the source data, and obtain the dynamic indicators and weighted comprehensive score of the telecom fraud model. The dynamic indicators include the confrontation sensitivity indicator, the dynamic adaptability indicator, and the user experience impact indicator.

[0167] The adversarial testing module 13 is configured to generate adversarial samples based on the source data, and to perform robustness testing on the telecom fraud model through the adversarial samples to obtain a robustness score of the telecom fraud model.

[0168] The false alarm cost quantification module 14 is configured to establish a loss function that correlates the false alarm rate of the telecom fraud model with the actual business loss, with the goal of minimizing business loss, and to adjust the optimal parameter configuration of the telecom fraud model through the loss function.

[0169] Regarding the limitations on the dynamic countermeasure evaluation device for telecom fraud models, please refer to the limitations on the dynamic countermeasure evaluation method for telecom fraud models in the above embodiments of this application, which will not be repeated here.

[0170] Figure 3 Another schematic diagram of the dynamic countermeasure evaluation device for the telecom fraud model provided in the embodiments of this application is shown below. Figure 3 As shown, the device includes a memory 22 and a processor 21. The memory stores a computer program, and the processor is configured to run the computer program to perform the methods described in the above embodiments of this application.

[0171] The memory is connected to the processor. The memory can be flash memory, read-only memory or other types of memory. The processor can be a central processing unit or a microcontroller.

[0172] In some embodiments, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described in the above embodiments of this application.

[0173] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules or other data. Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0174] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of this application, and this application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this application, and these modifications and improvements are also considered to be within the scope of protection of this application.

Claims

1. A dynamic countermeasure evaluation method for a telecom fraud model, characterized in that, The method includes: S1. Obtain source data related to telecommunications fraud; S2. Based on the source data, perform multi-dimensional dynamic index calculation on the telecom fraud model to obtain the dynamic index and weighted comprehensive score of the telecom fraud model. The dynamic index includes the confrontation sensitivity index, the dynamic adaptability index, and the user experience impact index. S3. Generate adversarial examples based on the source data, and conduct robustness tests on the telecom fraud model using the adversarial examples to obtain a robustness score for the telecom fraud model. S4. Establish a loss function that correlates the false alarm rate of the telecom fraud model with the actual business loss, with minimizing business loss as the optimization objective, and adjust the optimal parameter configuration of the telecom fraud model through the loss function.

2. The dynamic countermeasure evaluation method for telecom fraud models according to claim 1, characterized in that, The source data includes at least one of the following: call logs, SMS content, fraud case database, and user behavior logs. The call records include at least one of the following: caller ID, called number, call duration, roaming route, and call frequency. The SMS content includes at least one of the following: SMS text, sending time, sending frequency, and keywords; The fraud case database includes information on fraud types and timestamps. The user behavior log includes at least one of the following: user application (APP) usage behavior, traffic distribution, and device information.

3. The dynamic countermeasure evaluation method for telecom fraud models according to claim 1, characterized in that, In S2, the calculation process of the adversarial sensitivity index includes: Based on the source data, normal samples and adversarial samples are obtained; The normal samples and adversarial samples are input into the fraud model to obtain the normal sample accuracy of the fraud model in identifying fraud based on the normal samples, and the adversarial sample accuracy in identifying fraud based on the adversarial samples. Based on the accuracy of normal samples and the accuracy of adversarial samples, the adversarial sensitivity index is determined using the following formula: 。 4. The dynamic countermeasure evaluation method for telecom fraud models according to claim 1, characterized in that, In S2, the calculation process of the dynamic adaptability index includes: The fraud model is trained based on historical fraud data in the source data to obtain the historical fraud accuracy of the fraud model in identifying fraud based on the historical fraud data. The trained fraud model is used to identify new types of fraud data, and the accuracy of the trained fraud model in identifying new types of fraud data is obtained. The fraud type of the new types of fraud data is different from the fraud type of the historical fraud data. Based on the historical fraud accuracy rate and the new fraud accuracy rate, the dynamic adaptability index is determined using the following formula: 。 5. The dynamic countermeasure evaluation method for telecom fraud models according to claim 1, characterized in that, In S2, the calculation process of the user experience impact index includes: The false alarm rate, user complaint rate, and manual review cost of the fraud model are obtained based on historical data statistics. Based on the false positive rate, user complaint rate, and manual review cost, the user experience impact index is determined using the following formula: ； in, Indicates the false alarm rate. Indicates the user complaint rate. This indicates the cost of manual review.

6. The dynamic countermeasure evaluation method for telecom fraud models according to claim 1, characterized in that, In S3, the robustness of the telecom fraud model is tested using adversarial examples to obtain a robustness score for the model, including: Normal samples and adversarial samples are input into the fraud model to obtain the normal sample misclassification rate of the fraud model for identifying fraud in the normal samples, and the adversarial sample misclassification rate for identifying fraud in the adversarial samples. The robustness score is determined using the following formula based on the normal sample misclassification rate and the adversarial sample misclassification rate: ； in, This represents the false positive rate for normal samples. This represents the adversarial sample misclassification rate.

7. The dynamic countermeasure evaluation method for telecom fraud models according to claim 1, characterized in that, In S4, the loss function is: ； in, Indicates the false alarm rate. Indicates the user complaint rate. This indicates the cost of manual review. This represents business loss, and α represents the penalty coefficient for underreporting.

8. A dynamic countermeasure evaluation device for a telecom fraud model, characterized in that, The device includes: The data acquisition module is configured to acquire source data related to telecommunications fraud. The dynamic indicator calculation module is configured to perform multi-dimensional dynamic indicator calculations on the telecom fraud model based on the source data, and obtain the dynamic indicators and weighted comprehensive score of the telecom fraud model. The dynamic indicators include the confrontation sensitivity indicator, the dynamic adaptability indicator, and the user experience impact indicator. The adversarial testing module is configured to generate adversarial samples based on the source data, and to perform robustness testing on the telecom fraud model using the adversarial samples to obtain a robustness score for the telecom fraud model. The false alarm cost quantification module is configured to establish a loss function that correlates the false alarm rate of the telecom fraud model with the actual business loss, with the optimization objective of minimizing business loss, and adjusts the optimal parameter configuration of the telecom fraud model through the loss function.

9. A dynamic countermeasure evaluation device for a telecommunications fraud model, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the dynamic countermeasure evaluation method for telecom fraud models as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the dynamic countermeasure evaluation method for telecom fraud models as described in any one of claims 1-7.

Images