Graph neural network-based payment supervision model credibility evaluation method

Abstract

The invention belongs to the technical field of credible artificial intelligence, and particularly discloses a payment supervision model credibility evaluation method based on a graph neural network, which comprises the steps of training a payment supervision model used for classifying nodes in the graph neural network; boundary nodes and corresponding labels of the graph neural network are used as a clean fingerprint sequence; taking boundary nodes of the graph neural network obtained through the suspicious tampering model and corresponding labels as a suspicious fingerprint sequence; and determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence. According to the method, five-dimension assessment definitions, including privacy, interpretability, security, robustness and fairness, in a payment supervision model scene are proposed for the first time. The five dimensions comprehensively cover the core evaluation field of the payment supervision model, and the blank of the industry in the field is filled. The payment supervision model credibility evaluation method provided by the invention has important significance for promoting the development and improvement of the payment supervision model.

Description

Technical Field

[0001] The invention discloses a credible assessment method for a payment supervision model based on a graph neural network, and belongs to the technical field of credible artificial intelligence. Background Art

[0002] Currently, artificial intelligence technology is increasingly being used in various industries, including the financial trade sector. However, as the application deepens, the trust characteristics of artificial intelligence models, including privacy, explainability, security, robustness, and fairness, are receiving more and more attention. The industry has proposed some methods for evaluating these trust characteristics, but these methods are mainly aimed at general AI models and cannot be directly applied to the definition and evaluation of trust characteristics in financial trade scenarios.

[0003] In terms of privacy, existing evaluation methods mainly evaluate the risk of leakage of the model's training data and the model itself by means of model inversion attacks and knowledge distillation. However, in the financial trade scenario, these evaluation methods may not fully consider the particularity and sensitivity of financial data, and therefore cannot produce accurate evaluation results. In terms of interpretability, existing evaluation methods mainly evaluate the interpretability of the model by comparing the output of the model with the expected results of humans. However, in the financial trade scenario, due to the complex transaction logic and rules involved, these evaluation methods may not accurately evaluate the interpretability of the model. In terms of security, existing evaluation methods mainly evaluate the security of the model by means of model behavior analysis. However, in the financial trade scenario, due to the large amount of financial transaction data and complex security requirements involved, these evaluation methods may not be able to fully evaluate the security of the model. In terms of robustness, existing evaluation methods mainly evaluate the performance of the model in different situations by creating malicious adversarial samples. However, due to the complex transaction environment involved, these evaluation methods ignore their feasibility in the financial trade scenario, so the present invention aims to realize potential attacks in real trade scenarios. In terms of fairness, existing evaluation methods mainly focus on resampling technology and virtual sample generation technology, which balances the training data by reducing the number of majority class samples or increasing the number of minority class samples, thereby improving fairness. However, in financial trading scenarios, these evaluation methods may not be able to accurately evaluate the fairness of the model due to the involvement of various complex trading institutions and trading rules.

[0004] Therefore, the existing trust feature evaluation methods of general AI models cannot meet the special needs in financial trade scenarios, and it is necessary to redefine and design trust feature evaluation methods in financial trade scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a trustworthy assessment method for a payment supervision model based on a graph neural network, so as to solve the technical problem that the trustworthy characteristic assessment method of a general AI model in the prior art is difficult to meet the trustworthy assessment of the payment supervision model in the financial trade scenario.

[0006] The present invention provides a payment supervision model credibility assessment method based on graph neural network, comprising:

[0007] Training a payment supervision model, wherein the payment supervision model is used to classify nodes in a graph neural network;

[0008] Taking the boundary nodes and corresponding labels of the graph neural network as a clean fingerprint sequence;

[0009] The boundary nodes and corresponding labels of the graph neural network obtained through the suspected tampering model are used as a suspicious fingerprint sequence;

[0010] The security of the payment supervision model is determined according to the clean fingerprint sequence and the suspicious fingerprint sequence.

[0011] Preferably, the boundary nodes and corresponding labels of the graph neural network are used as clean fingerprint sequences, specifically including:

[0012] Obtain a label corresponding to each node in the graph neural network;

[0013] Determining the boundary nodes in the graph neural network according to a preset attribution threshold;

[0014] The boundary nodes and corresponding labels are used as clean fingerprint sequences.

[0015] Preferably, determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence specifically includes:

[0016] Obtaining the number of matches between multiple suspicious fingerprint sequences and labels corresponding to the same boundary node in the clean fingerprint sequence;

[0017] The security of the payment supervision model is determined according to the number of matches and the length of the clean fingerprint sequence.

[0018] Preferably, after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes:

[0019] Using random training sets to train multiple negative models, and using attack methods to attack the payment supervision model to obtain multiple positive models;

[0020] Inputting the adversarial sample into the negative model and the positive model respectively;

[0021] The copyright security of the payment supervision model is determined according to the classification result of the negative model and the classification result of the positive model.

[0022] Preferably, after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes:

[0023] Generate adversarial samples using a fast gradient sign method, and use the adversarial samples to attack the payment supervision model;

[0024] The robustness of the payment supervision model is determined according to the number of risky users detected by the payment supervision model and the number of all risky users.

[0025] Preferably, after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes:

[0026] Extracting local information of the graph neural network and training an attacker model that classifies nodes in the local information;

[0027] The privacy of the payment supervision model is determined according to the classification results of the payment supervision model and the attacker model.

[0028] Preferably, after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes:

[0029] Obtaining the weight of a high-value group and the weight of a low-value group in the graph neural network, wherein the high-value group is a group consisting of nodes whose values ​​are greater than an average value;

[0030] adjusting a loss function within the payment regulation model according to a weight of a high-value group and a weight of a low-value group;

[0031] The fairness of the payment supervision model is determined based on the classification results of the high-value group and the low-value group by the adjusted payment supervision model.

[0032] Preferably, after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes:

[0033] The original transaction graph data set is repeatedly input into the risk subgraph generation algorithm to obtain multiple original risk subgraphs;

[0034] Add perturbations to the original transaction graph dataset and obtain the perturbation risk subgraph through the risk subgraph generation algorithm;

[0035] The interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbed risk subgraph.

[0036] Preferably, the evaluation index of the interpretability is stability and fidelity, and the interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbation risk subgraph, specifically:

[0037] The stability and fidelity of the original risk subgraph are determined according to the Jaccard similarity between the original risk subgraph and the perturbed risk subgraph.

[0038] Preferably, the evaluation index of the interpretability is repeatability, and the interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbation risk subgraph, specifically as follows:

[0039] The repeatability of the original risk subgraphs is determined according to the Jaccard similarities of the multiple original risk subgraphs.

[0040] Compared with the prior art, the payment supervision model trustworthiness assessment method based on graph neural network of the present invention has the following beneficial effects:

[0041] The present invention proposes for the first time the five-dimensional evaluation definitions in the payment regulatory model scenario, including privacy, explainability, security, robustness and fairness. These five dimensions comprehensively cover the core evaluation areas of the payment regulatory model and fill the gap in this field in the industry. Secondly, the present invention designs the evaluation technology and quantitative criteria of the trade payment regulatory model corresponding to the above five dimensions. Among them, the privacy evaluation technology uses the means of knowledge distillation to measure the leakage risks of training data and the model itself, which can detect the privacy of the model, thereby discovering possible model privacy risks, and then help financial institutions improve the privacy of the payment regulatory model; the interpretability evaluation measures the accuracy of the model decision explanation, whether it conforms to expert knowledge and industry consensus, which can help people better understand the working principle and decision-making process of the model, thereby enhancing the explainability of the model and improving the trust of financial institutions and regulatory authorities in the model; the security evaluation technology uses model fingerprint matching and model behavior analysis to detect whether the model has been tampered with and whether the copyright has been stolen. It can detect the security of the model and discover possible security risks. For example, it can detect malicious tampering of the distribution model (including fine-tuning, pruning, etc.) or copyright theft of the distribution model, thereby helping financial institutions improve the security of the payment supervision model; the robustness assessment technology utilizes the adversarial attack technology of trusted artificial intelligence to measure the performance of the model in responding to malicious attacks. It can detect the performance of the model when facing malicious attacks, thereby helping financial institutions enhance the robustness of the payment supervision model and improve the stability of the model in complex environments; fairness assessment is to detect whether the model is biased towards a certain attribute when conducting cross-border trade supervision. It can discover the performance differences of the model when processing different types of data, thereby helping financial institutions ensure the fairness of the payment supervision model and avoid incorrect predictions.

[0042] The present invention provides a comprehensive, scientific and objective payment supervision model evaluation method, which is of great significance for promoting the development and improvement of the payment supervision model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flow chart of security fingerprint evaluation in an embodiment of the present invention.

[0044] Figure 2 The present invention is a flowchart of an embodiment of the present invention using the fast gradient sign method to generate adversarial samples. DETAILED DESCRIPTION

[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0046] The embodiment of the present invention provides a payment supervision model credibility assessment method based on graph neural network, including:

[0047] Step 1: Train a payment supervision model, where the payment supervision model is used to classify nodes in the graph neural network. The payment supervision model is a deep neural network model.

[0048] The nodes in the embodiment of the present invention correspond to merchants, and the payment supervision model is used to classify merchants into normal merchants and risky merchants.

[0049] Step 2: Take the boundary nodes and corresponding labels of the graph neural network as the clean fingerprint sequence.

[0050] The security of the payment supervision model refers to the effective protection of intellectual property rights in the process of design, training, deployment and use of the DNN model through the technology of deep intellectual property protection, to prevent unauthorized copying, distribution, abuse or tampering. Deep intellectual property protection is not only the key to protecting the model, but also the core link in building a trusted artificial intelligence (TAI) system. It extends to multiple fields such as model anti-tampering, user authentication and management, and tracking of the model distribution chain. Among them, the security fingerprint assessment technology is used to evaluate the tamper-proof aspects of the model. In view of the possibility of various tampering of the distribution model by attackers (fine-tuning, pruning, etc.), the security fingerprint assessment technology can effectively identify whether the model has been tampered with. For a neural network model, its decision boundary can be used as a unique identity to identify the neural network. Therefore, the present invention chooses to extract and verify the model decision boundary data points and their corresponding labels as the fingerprint of the model for verifying the security of the model.

[0051] The goal of determining the security of the payment supervision model in the embodiment of the present invention is to provide anti-tampering detection for the payment supervision model. In the application scenario, the model supplier (i.e., the party that provides the payment supervision model, such as the central bank, UnionPay headquarters, etc.) will provide the payment supervision model to local institutions for use and transaction supervision. The security evaluation of the embodiment of the present invention is to detect whether the model distributed by the supplier to the local institution (i.e., the distribution model mentioned above) has been tampered with.

[0052] The theft situation considered by the present invention in security fingerprint assessment is tampering with a known distribution model, so the present invention will use the distribution model, payment supervision model and verification set for security fingerprint assessment.

[0053] Step 2 of the embodiment of the present invention is specifically:

[0054] Step 2.1: Get the label corresponding to each node in the graph neural network. Specifically, input the verification set into the distribution model to get the label corresponding to each node.

[0055] Step 2.2: Determine the boundary nodes in the graph neural network according to the preset attribution threshold. Specifically, given an attribution threshold for a boundary point, input the validation set into the distribution model, and use the decision boundary search method to find the graph network on the decision boundary.

[0056] The above decision boundary search method is an optimization method, which can be exemplarily an ant colony algorithm, a particle swarm algorithm, etc.

[0057] Step 2.3: Take the boundary nodes and corresponding labels as the clean fingerprint sequence, specifically: extract the graph boundary points within the attribution threshold and their corresponding labels as the clean fingerprint sequence in the known distribution model.

[0058] The order of the above steps 2.1 and 2.2 may be changed.

[0059] Step 3: The boundary nodes and corresponding labels of the graph neural network obtained through the suspicious tampering model are used as the suspicious fingerprint sequence.

[0060] The suspicious tampering model in the embodiment of the present invention is obtained by tampering the above distribution model using the tampering parameters set by the hyperparameters, and there are multiple suspicious tampering models.

[0061] For the suspected tampering model, the embodiment of the present invention also inputs the verification set into the suspected tampering model, and uses the label of the clean fingerprint to extract the label corresponding to the boundary node given by the suspected tampering model for the distribution model, thereby obtaining the suspicious fingerprint sequence extracted from the suspected tampering model.

[0062] Step 4: Determine the security of the payment supervision model based on the clean fingerprint sequence and the suspicious fingerprint sequence, including:

[0063] Step 4.1, obtain the matching number of labels corresponding to the same boundary node in multiple suspicious fingerprint sequences and clean fingerprint sequences.

[0064] Step 4.2 determines the security of the payment supervision model according to the number of matches and the length of the clean fingerprint sequence. For example, the security of the payment supervision model is determined according to the following formula (1):

[0065]

[0066] Where MR is the matching rate; n is the number of matches; and m is the length of the clean fingerprint sequence.

[0067] In the embodiment of the present invention, a threshold α may be set to determine whether the suspicious distribution model has been tampered with. If MR is greater than the threshold α, it is considered that the distribution model has not been tampered with.

[0068] Furthermore, after obtaining the matching rate of each suspected tampering model, the embodiment of the present invention also uses the ratio of the number of tampering situations identified in multiple tampering models as the detection success rate. The detection success rate ranges from 0 to 1, where 1 represents a highly secure classifier that identifies all tampered models. At the same time, the embodiment of the present invention can also construct a sensitivity index (sensitivity), which is defined as the detection recognition rate for each fingerprint point in all tampering models, which can reflect the sensitivity of a particular fingerprint point. The higher the value, the more sensitive it is, that is, the label of the fingerprint point has changed in more tampering models. With the help of the sensitivity index, the present invention can screen out sensitive points for subsequent improvement of security fingerprint assessment technology.

[0069] The process of the above-mentioned security fingerprint assessment technology in the embodiment of the present invention is as follows: Figure 1shown.

[0070] In addition, the embodiments of the present invention need to ensure that the intellectual property rights of the payment supervision model are fully protected, for example, by embedding a watermark in the payment supervision model or extracting information from the payment supervision model, so that the source of the model can be traced when it is used without authorization.

[0071] In the node classification task of the payment supervision model, an attacker may steal the original payment supervision model, thereby obtaining the model copyright to gain benefits. The embodiment of the present invention verifies the copyright protection capability of the payment supervision model by simulation, specifically: after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, it also includes:

[0072] Step A1: Use random training sets to train multiple negative models, and use attack methods to attack the payment supervision model to obtain multiple positive models.

[0073] The embodiment of the present invention defines negative samples as models of other institutions according to the standard of hypothesis testing in statistics, and its function is to test the payment supervision model's ability to identify negative models that may be mistaken for current models; positive samples are stolen models that may have been modified.

[0074] The method for generating the above negative model is: using a random training set to train multiple negative models to obtain a group of negative models.

[0075] The above positive models are generated by combining attack methods, such as fine-tuning, pruning and other technical means to attack the payment supervision model, simulating the situation where the stolen model is obtained and illegally used by the thief. These models obtained by attack methods are classified as positive models.

[0076] Step A2: Input the adversarial samples into the negative model and the positive model respectively. This step is essentially to construct an intellectual property identifier for the payment supervision model.

[0077] In order to evaluate the copyright security of the payment supervision model, the embodiment of the present invention needs to extract model information to test its robustness and representativeness. An effective method is to use adversarial samples D ad Test model. Adversarial samples are related to the decision boundary of the neural network and can deceive the neural network and make it make wrong predictions or classifications. ad Input into the negative model and the positive model respectively. At this time, the loss function in the negative model and the positive model is shown in formula (2):

[0078]

[0079] Where, L ceis the loss function, x is the input of the model, y is the output of the model, ∈ is the added disturbance, and θ is the parameter in the model.

[0080] Step A3: Determine the copyright security of the payment supervision model based on the classification results of the negative model and the classification results of the positive model. This step is essentially to verify the intellectual property ownership of all suspicious models (negative models and positive models).

[0081] Exemplarily, the copyright security of the payment supervision model is determined using the following formula (3).

[0082]

[0083] In the formula, pre represents the suspicious model (including positive model and negative model) and the payment supervision model for abnormal samples D ad The probability of consistent performance, when pre exceeds a certain threshold, the present invention considers that the corresponding suspicious model is stolen (developed) from the payment supervision model of the present invention; f θ (x) is the payment supervision model, Suspicious model.

[0084] The embodiment of the present invention queries samples with special triggers in a total of K positive models and negative models to obtain the prediction results of each positive model and negative model. pre_t and pre_f respectively represent the prediction accuracy of the positive model and the negative model for samples with special triggers as the standard for judging whether the model is positive. The model is divided according to a specific threshold, and the part exceeding the threshold is judged as a positive model.

[0085] In order to intuitively characterize the performance of copyright inspection, the embodiment of the present invention uses the ROC curve to display the performance of copyright inspection by plotting the True Positive Rate (TPR) on the vertical axis and the False Positive Rate (FPR) on the horizontal axis under different threshold settings. The ROC curve is a coordinate diagram analysis tool in signal detection theory.

[0086] Exemplarily, a threshold set is first determined: the embodiment of the present invention determines a series of thresholds {i}. These thresholds are used to convert the predicted probability of the model into a binary classification.

[0087] Secondly, calculate TPR and FPR: For each threshold i, the embodiment of the present invention calculates TPR_t and FPR_f. True positive (TPR) is the number of correctly identified positive models divided by the total number of actual positive models, that is, TPR = TP / (TP+FN). False positive (FPR) is the number of negative models incorrectly identified as positive models divided by the total number of actual negative models, that is, FPR = FP / (FP+TN).

[0088] Draw the ROC curve again: For each threshold, use the FPR value as the horizontal axis and the TPR value as the vertical axis to draw the ROC curve.

[0089] Finally, the AUC value is calculated: The embodiment of the present invention calculates the area under the ROC curve, that is, the AUC value, according to the trapezoidal method, as shown in formula (4):

[0090]

[0091] The above AUC values ​​range from 0 to 1. An AUC of 1 indicates a perfect classifier; a larger AUC value indicates a better detection effect and a better copyright protection capability of the payment supervision model.

[0092] Payment supervision models in cross-border trade supervision may be affected by adversarial attacks. In this case, potential risk users (i.e. attackers) attempt to attack the original transaction graph by registering new merchants (i.e. adding nodes) or conducting new transactions (i.e. adding edges). The purpose of this attack is to make the payment supervision model predict it as a benign user, thereby evading bank scrutiny. Therefore, it is necessary to evaluate the ability of the payment supervision model to maintain its detection performance (i.e. robustness) under potential adversarial attacks.

[0093] In the node classification task of the payment supervision model, the attacker implements the attack by modifying the existing graph neural network, that is, adding nodes or edges, and further modifying the node features. The robustness evaluation scenario proposed in the present invention is a black box scenario. In this scenario, the attack method is first applied to an alternative model (trained by the attacker himself) to generate adversarial samples, and then applied to the real payment supervision model. The present invention proposes an attack scenario with practical significance, that is, attacking the supervision model while limiting the attack cost and the attacker's transaction rights. In this scenario, the attacker injects new registered nodes and the connection edges between the new nodes and the risky users in the original graph, which corresponds to the real scenario that the attacker registers a fixed number of new merchants and generates transactions with the risky users in the original transaction graph, thereby changing the detection results of the supervision model for risky users.

[0094] The embodiment of the present invention defines the payment supervision model as a function that maps the transaction graph to a probability vector with multiple categories. The objective function of the adversarial attack can be expressed as maximizing the number of incorrect predictions of the graph risk detection model under the constraints of the number of modified edges and the range of modified features. in represents the adjacency matrix of N nodes, Represents a set of nodes with D-dimensional features. The risk detection model is defined as in This model will trade graph Mapped to a probability vector with L categories. The goal of the adversarial attack on the payment supervision model can be stated as:

[0095] in This is the picture of being attacked. and It is a metric space and The attacker attempts to and Maximize the number of incorrect predictions of the graph risk detection model under the constraint of

[0096] In order to evaluate the ability of the payment supervision model to maintain its detection performance (i.e., robustness) under potential adversarial attacks, after determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the embodiment of the present invention further includes:

[0097] Step B1: Generate adversarial samples using the fast gradient sign method, and use the adversarial samples to attack the payment supervision model.

[0098] The present invention uses an iterative version of the fast gradient sign method to perform graph injection attacks. The iterative fast gradient sign method linearizes the cost function around the current parameter value to obtain an optimal maximum norm constrained perturbation. The specific steps are as follows: Figure 2 As shown:

[0099] Step 1 (Initialize adversarial sample): First, initialize the adversarial sample, which can be a normal transaction graph data.

[0100] Step 2 (Define loss function): Define the loss function, which represents the error between the risk user prediction result and the true label of the model on the adversarial sample.

[0101] Step 3 (Calculate perturbations using the fast gradient symbolic method) Use the fast gradient symbolic method to calculate the optimal perturbation of the transaction graph at each iteration. This perturbation is obtained by linearizing the loss function around the current parameter value. The calculated perturbation will be added to the current adversarial example.

[0102] Step 4 (update adversarial sample): add the calculated perturbation to the current adversarial sample to obtain a new adversarial sample. The new adversarial sample should satisfy specific constraints, such as the number of modified edges and the range of modified features.

[0103] Step 5 (iterative update): Repeat steps 3 and 4 until the stopping condition is met, such as reaching the preset maximum number of iterations, or the prediction error of the model on the adversarial sample has reached the maximum. The adversarial sample generated at the end will be used to evaluate the robustness of the payment supervision model.

[0104] Step B2: Determine the robustness of the payment supervision model based on the number of risky users detected by the payment supervision model and the number of all risky users.

[0105] Under this fast gradient sign attack method, the robustness of the graph risk detection model is quantified as the detection success rate on adversarial examples generated by the attacker:

[0106]

[0107] The higher the above robustness score, the stronger the ability of the payment supervision model to maintain its detection performance under potential adversarial attacks.

[0108] In cross-border trade supervision, payment supervision models face the risk of information theft, which may lead to privacy leakage. Specifically, attackers may obtain part of the model information through illegal means and use this information to retrain a replica model with similar performance to the original model. The purpose of the privacy assessment algorithm is to evaluate the degree of model performance that the attacker can restore under the premise that the attacker obtains part of the model data. This assessment process is crucial to ensure model privacy.

[0109] The present invention constructs an attacker model by selecting part of the training data, and evaluates the privacy of the original model by comparing its performance gap with the original model. Specifically, the attacker privately collects part of the real transaction data and its risk information, and uses part of the node attributes and structure of the target graph to construct a shadow data set to train the attacker model. Finally, the privacy of the original model is evaluated by comparing the performance of the two models, that is, measuring the degree to which the attacker can restore the rules of the original model. Therefore, after determining the security of the payment supervision model based on the clean fingerprint sequence and the suspicious fingerprint sequence, the present invention also includes:

[0110] Step C1: extract local information of the graph neural network and train an attacker model that classifies nodes in the local information.

[0111] Exemplarily, the attacker first obtains the attributes and connection information of some nodes in the graph neural network, and constructs a comprehensive graph containing all known node attributes, graph structure, and some node labels.

[0112] Using the above comprehensive graph, the attacker trains a new payment supervision model through supervised learning method, which is denoted as the attacker model.

[0113] Step C2: Determine the privacy of the payment supervision model according to the classification results of the payment supervision model and the attacker model.

[0114] In order to evaluate the extent to which the attacker model steals the privacy of the payment supervision model, the privacy loss can be quantified in the following ways:

[0115] 1. Privacy metric based on accuracy: The degree of privacy leakage is evaluated by calculating the difference between the accuracy of the payment supervision model and the attacker model classification using formula (6), where acc total represents the classification accuracy of the payment supervision model, acc part Represents the classification accuracy of the attacker model:

[0116] Δacc=|acc total -acc part | (6)

[0117] 2. Privacy measurement based on F1 score: The difference between the F1 scores of the payment supervision model and the attacker model classification is calculated by formula (7) to evaluate the privacy leakage: where F1 total represents the F1 score of the payment supervision model, F1 part Represents the F1 score of the attacker model.

[0118] ΔF1=|F1 total -F1 part | (7)

[0119] 3. Privacy metric based on Cohen's Kappa coefficient: Cohen's Kappa coefficient is used to measure the consistency of two classifiers in classification tasks. It evaluates the consistency of classification by comparing the actual consistency and the expected consistency:

[0120]

[0121] In the formula, p o is the observed consistency ratio, that is, the ratio of samples of the same category that are consistently classified by two classifiers; p e is the expected agreement ratio, that is, the expected probability that two classifiers will classify the same category consistently under random circumstances.

[0122] The greater the difference in accuracy and F1 score between the payment supervision model and the attacker model, the stronger the privacy of the model; similarly, the lower the Cohen's Kappa coefficient between the two, the stronger the privacy of the model.

[0123] Fairness is about whether the model provides unbiased risk assessments for subgroups of different attributes. In graph risk detection algorithms, this means ensuring that different user groups are not unfairly exposed to risk due to algorithmic bias. If there is bias in the training process of graph neural networks, it may lead to biased transactions for certain specific attributes or features.

[0124] In the node classification task of the payment supervision model, the data set may have the problem of uneven data distribution within the class, that is, within the same class, the number of different data samples varies significantly. This imbalance will cause the model to be biased towards the majority class and ignore the minority class during training, thus affecting the overall performance of the model. Therefore, the present invention provides an evaluation algorithm for fairness, which evaluates the performance differences of different attribute groups based on the graph neural network model.

[0125] After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the embodiment of the present invention further includes:

[0126] Step D1, obtain the weights of the high-value group and the low-value group in the graph neural network, the high-value group is a group composed of nodes with values ​​greater than the average value; correspondingly, the low-value group is a group composed of nodes with values ​​less than or equal to the average value.

[0127] The above step D1 specifically includes:

[0128] Step D11: classify the attributes of the nodes in the graph neural network and obtain the average value of each attribute.

[0129] Assume that the total number of groups in attribute j is N (j) , the number of low value groups is The number of high value groups is The present invention determines the threshold T by calculating the average value of all user values ​​on attribute j:

[0130]

[0131] In the formula, Represents the value of attribute j for the i-th user.

[0132] Step D12: Determine the number of high-value groups and the number of low-value groups under the corresponding attribute according to the average value.

[0133] For example, the number of high-value groups is shown in formula (10):

[0134]

[0135] Where 1 is an indicator function, which takes the value 1 when the condition in the brackets is true, and takes the value 0 otherwise.

[0136] The number of low-value groups is shown in formula (11):

[0137]

[0138] Step D13: Determine the weight of the high-value group and the weight of the low-value group under the corresponding attribute according to the number of the high-value group and the number of the low-value group.

[0139] The present invention normalizes the inverse of the number of samples in each category to determine the category weight. Specifically, the weight of the high-value group is calculated by formula (12):

[0140]

[0141] The weight of the low-value group is calculated by formula (13)

[0142]

[0143] Step D2: Adjust the loss function in the payment supervision model according to the weight of the high-value group and the weight of the low-value group. Specifically: During the training process, the above-mentioned category weights calculated are applied to the loss function. In the cross entropy loss function, the weighted loss function can be expressed as formula (14):

[0144]

[0145] Step D3: Determine the fairness of the payment supervision model based on the classification results of the high-value group and the low-value group by the adjusted payment supervision model.

[0146] The embodiment of the present invention realizes fairness quantification of the node classification task of graph risk detection through bias metrics based on accuracy and bias metrics based on F1 score:

[0147] 1. Bias measure based on accuracy: Accuracy measures the proportion of correct predictions made by the model in all samples. Calculate the difference between the accuracy of the model in the high-value group and the low-value group, that is:

[0148] Δ acc =|acc high -acc low | (15)

[0149] In the formula, acc high is the accuracy of the adjusted payment supervision model in classifying high-value groups; low The accuracy of the adjusted payment supervision model in classifying the low-value group.

[0150] 2. Bias metric based on F1 score: The F1 score takes into account the indicators of recall and precision. It is the harmonic mean of recall and precision and reflects the overall performance of the model. The difference between the F1 scores of the model on the high-value group and the low-value group is calculated, that is:

[0151] Δ F1score =|F1 high -F1 low | (16)

[0152] Where F1 high is the F1 score of the adjusted payment supervision model for high-value group classification; F1 low F1 score of the adjusted payment supervision model for classification of low-value groups.

[0153] The larger the difference, the more unfair the model's predictions are for different groups. These two bias metrics can help evaluate the fairness performance of the model between different groups, which is important for ensuring model fairness and reducing bias.

[0154] The background of interpretability research stems from the need for model transparency, the challenge of understanding complex models, and the requirements of regulatory ethics. In this context, researchers are committed to finding a balance that ensures the high performance of the model while enabling users and decision makers to clearly understand the decision-making process of the model.

[0155] Explainability in risk detection algorithms refers to the ability of the algorithm to provide transparency in its decision-making process, allowing human users to understand why a transaction is marked as high risk. Explainability is necessary in cross-border trade payment supervision because it helps audits while also enhancing user trust in the system.

[0156] The embodiment of the present invention mainly evaluates the interpretability of the model through the following two aspects:

[0157] 1. Long text report generated by manual evaluation: After the classification results output by the payment supervision model are input into the explainability model to obtain the explanation text, the explanation text generated by the explainability model is manually evaluated, mainly including the clarity and accuracy of the generated text.

[0158] 2. Consistency of interpretation of risk subgraph patterns.

[0159] After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the embodiment of the present invention further includes:

[0160] Step E1: repeatedly input the original transaction graph data set into the risk subgraph generation algorithm to obtain multiple original risk subgraphs.

[0161] Step E2: Add disturbance to the original transaction graph dataset and obtain the disturbance risk subgraph through the risk subgraph generation algorithm.

[0162] Step E3: Determine the interpretability of the original risk subgraph according to the original risk subgraph and the perturbed risk subgraph.

[0163] The interpretation consistency evaluation of the risk subgraph model in the embodiment of the present invention includes the following three points:

[0164] Stability evaluation: When a small perturbation is made to non-critical nodes or edges, if the consistency of the interpreter is good, it means that it is stable.

[0165] Repeatability evaluation: Run the explanation program repeatedly on the same graph structure. If the consistency is good, it means that the program is repeatable.

[0166] Fidelity evaluation: If the consistency of the interpreter is poor after perturbing the key nodes or edges, it means that the modified key points or edges have a great impact on the results, thus proving the fidelity.

[0167] Exemplarily, when the evaluation index of interpretability is stability and fidelity, the interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbed risk subgraph, specifically:

[0168] The stability and fidelity of the original risk subgraph are determined based on the Jaccard similarity between the original risk subgraph and the perturbed risk subgraph.

[0169] When the evaluation index of interpretability is repeatability, the interpretability of the original risk subgraph is determined based on the original risk subgraph and the perturbed risk subgraph, specifically:

[0170] The reproducibility of the original risk subgraphs was determined based on the Jaccard similarity of multiple original risk subgraphs.

[0171] The Jaccard Similarity is an indicator used to measure the similarity between two sets. The calculation formula is as follows:

[0172]

[0173] Jaccard similarity is used to calculate the intersection-and-union ratio of nodes in graph A and graph B, with a value range of 0 to 1, where 1 means that the two sets are exactly the same and 0 means that there are no common elements. This quantitative standard is used in the experiment. In the stability and repeatability evaluation, the higher the Jaccard similarity, the better the interpretability of the subgraph explanation; while in the fidelity evaluation, the lower the Jaccard similarity, the worse the interpretability.

[0174] The present invention proposes for the first time the five-dimensional evaluation definitions in the payment regulatory model scenario, including privacy, explainability, security, robustness and fairness. These five dimensions comprehensively cover the core evaluation areas of the payment regulatory model and fill the gap in this field in the industry. Secondly, the present invention designs the evaluation technology and quantitative criteria of the trade payment regulatory model corresponding to the above five dimensions. Among them, the privacy evaluation technology uses the means of knowledge distillation to measure the leakage risks of training data and the model itself, which can detect the privacy of the model, thereby discovering possible model privacy risks, and then help financial institutions improve the privacy of the payment regulatory model; the interpretability evaluation measures the accuracy of the model decision explanation, whether it conforms to expert knowledge and industry consensus, which can help people better understand the working principle and decision-making process of the model, thereby enhancing the explainability of the model and improving the trust of financial institutions and regulatory authorities in the model; the security evaluation technology uses model fingerprint matching and model behavior analysis to detect whether the model has been tampered with and whether the copyright has been stolen. It can detect the security of the model and discover possible security risks. For example, it can detect malicious tampering of the distribution model (including fine-tuning, pruning, etc.) or copyright theft of the distribution model, thereby helping financial institutions improve the security of the payment supervision model; the robustness assessment technology utilizes the adversarial attack technology of trusted artificial intelligence to measure the performance of the model in responding to malicious attacks. It can detect the performance of the model when facing malicious attacks, thereby helping financial institutions enhance the robustness of the payment supervision model and improve the stability of the model in complex environments; fairness assessment is to detect whether the model is biased towards a certain attribute when conducting cross-border trade supervision. It can discover the performance differences of the model when processing different types of data, thereby helping financial institutions ensure the fairness of the payment supervision model and avoid incorrect predictions.

[0175] The present invention provides a comprehensive, scientific and objective payment supervision model evaluation method, which is of great significance for promoting the development and improvement of the payment supervision model.

[0176] The above are only several embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, using the above disclosed technical content to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A credible assessment method for payment supervision model based on graph neural network, characterized in that: include: Training a payment supervision model, wherein the payment supervision model is used to classify nodes in a graph neural network; Taking the boundary nodes and corresponding labels of the graph neural network as a clean fingerprint sequence; The boundary nodes and corresponding labels of the graph neural network obtained through the suspected tampering model are used as a suspicious fingerprint sequence; The security of the payment supervision model is determined according to the clean fingerprint sequence and the suspicious fingerprint sequence.

2. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: The boundary nodes and corresponding labels of the graph neural network are used as a clean fingerprint sequence, specifically including: Obtain a label corresponding to each node in the graph neural network; Determining the boundary nodes in the graph neural network according to a preset attribution threshold; The boundary nodes and corresponding labels are used as clean fingerprint sequences.

3. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: Determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence specifically includes: Obtaining the number of matches between multiple suspicious fingerprint sequences and labels corresponding to the same boundary node in the clean fingerprint sequence; The security of the payment supervision model is determined according to the number of matches and the length of the clean fingerprint sequence.

4. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes: Using random training sets to train multiple negative models, and using attack methods to attack the payment supervision model to obtain multiple positive models; Inputting the adversarial sample into the negative model and the positive model respectively; The copyright security of the payment supervision model is determined according to the classification result of the negative model and the classification result of the positive model.

5. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes: Generate adversarial samples using a fast gradient sign method, and use the adversarial samples to attack the payment supervision model; The robustness of the payment supervision model is determined according to the number of risky users detected by the payment supervision model and the number of all risky users.

6. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes: Extracting local information of the graph neural network and training an attacker model that classifies nodes in the local information; The privacy of the payment supervision model is determined according to the classification results of the payment supervision model and the attacker model.

7. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes: Obtaining the weight of a high-value group and the weight of a low-value group in the graph neural network, wherein the high-value group is a group consisting of nodes whose values ​​are greater than an average value; adjusting a loss function within the payment regulation model according to a weight of a high-value group and a weight of a low-value group; The fairness of the payment supervision model is determined based on the classification results of the high-value group and the low-value group by the adjusted payment supervision model.

8. The method for credible evaluation of payment supervision model based on graph neural network according to claim 1 is characterized in that: After determining the security of the payment supervision model according to the clean fingerprint sequence and the suspicious fingerprint sequence, the method further includes: The original transaction graph data set is repeatedly input into the risk subgraph generation algorithm to obtain multiple original risk subgraphs; Add perturbations to the original transaction graph dataset and obtain the perturbation risk subgraph through the risk subgraph generation algorithm; The interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbed risk subgraph.

9. The method for credible evaluation of payment supervision model based on graph neural network according to claim 8 is characterized in that: The evaluation index of the interpretability is stability and fidelity, and the interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbation risk subgraph, specifically: The stability and fidelity of the original risk subgraph are determined according to the Jaccard similarity between the original risk subgraph and the perturbed risk subgraph.

10. The method for credible evaluation of payment supervision model based on graph neural network according to claim 8 is characterized in that: The evaluation index of the interpretability is repeatability, and the interpretability of the original risk subgraph is determined according to the original risk subgraph and the perturbation risk subgraph, specifically: The repeatability of the original risk subgraphs is determined according to the Jaccard similarities of the multiple original risk subgraphs.

Images