Method and apparatus for grading training samples in transaction risk control

Abstract

This specification provides a method and apparatus for classifying training samples in transaction risk control. The method includes: acquiring a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information; training a first large model using the training sample set for N iterations, obtaining the sample gradient of each training sample participating in each iteration in each iteration; for any first training sample in the training sample set, calculating the ratio of the sample gradient norms of each adjacent iteration in several iterations in which the first training sample participates, and determining the contribution evaluation of the first training sample to the model training based on the ratio; determining the classification level information of each training sample for the target model based on the contribution evaluation corresponding to each training sample in the training sample set; the target model is used to predict transaction risk.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computers, and more particularly to a method and apparatus for grading training samples in transaction risk control. Background Technology

[0002] With the increasing application of neural network models in the field of trading risk control, they have demonstrated significant potential in tasks such as risk identification, risk event analysis, and report generation. However, the training effectiveness of these models is highly dependent on the effective selection of training samples to scientifically construct the training sample set. Therefore, it is necessary to classify the training samples used in trading risk control to facilitate the effective selection of training samples based on the classification results.

[0003] In existing technologies, the grading of training samples in transaction risk control often relies on human experience or a single-dimensional evaluation of the sample's conventional attributes, which makes it difficult to improve the model's training effect and results in low model training efficiency.

[0004] Therefore, an improved scheme is needed to make the classification of training samples more reasonable, which will help improve the efficiency of model training. Summary of the Invention

[0005] This specification describes one or more embodiments of a method and apparatus for classifying training samples in transaction risk control, which makes the classification of training samples more reasonable and helps to improve model training efficiency.

[0006] Firstly, a hierarchical method for training samples in transaction risk control is provided, the method including:

[0007] Obtain the training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information;

[0008] The first large model is trained through N iterations using the training sample set. In each iteration, the sample gradient of each training sample participating in that iteration is obtained.

[0009] For any first training sample in the training sample set, calculate the ratio of the sample gradient norms of each adjacent iteration in several iterations in which the first training sample participates, and determine the contribution of the first training sample to the model training based on the ratio.

[0010] Based on the contribution evaluation of each training sample in the training sample set, the classification level information of each training sample for the training target model is determined; the target model is used to predict transaction risk.

[0011] In one possible implementation, the model training is a single training process, and the contribution evaluation is a contribution score or contribution level calculated based on the single training process.

[0012] In one possible implementation, the model training includes multiple training processes; the training sample set is divided into multiple sub-training sample sets, and one training process in the multiple training processes includes multiple rounds of iteration using one sub-training sample set from the multiple sub-training sample sets;

[0013] Determining the contribution of the first training sample to the model training includes:

[0014] Based on the several iterations in which the first training sample participates in each training process, the contribution score of the first training sample in each training process is calculated respectively.

[0015] Based on the contribution score of the first training sample in each training process, the comprehensive contribution evaluation of the first training sample is determined.

[0016] Furthermore, the comprehensive contribution evaluation is a contribution level; determining the comprehensive contribution evaluation of the first training sample includes:

[0017] Based on the contribution scores of the first training sample in each training process, a comprehensive contribution score for the first training sample is determined, and a hierarchical classification is performed based on this comprehensive contribution score to determine the contribution level of the first training sample; or,

[0018] Based on the contribution scores of the first training sample in each training process, the levels are divided to obtain the contribution level of the first training sample in each training process. Based on the contribution level of the first training sample in each training process, its final contribution level is determined.

[0019] In one possible implementation, the method further includes at least one of the following:

[0020] Based on the user attribute data and transaction-related data implemented by the user, as well as transaction risk indication information included in the sample data, multiple training samples in the training sample set are classified, and a classification label is generated for each training sample.

[0021] Based on the performance metrics obtained after processing the training samples by calling at least one second-largest model, the difficulty of multiple training samples in the training sample set is classified, and a difficulty level is generated for each training sample.

[0022] Based on the scoring results of the output content of the training samples on multiple preset quality dimensions by calling the quality inspection big model, the quality of multiple training samples in the training sample set is graded, and a quality level is generated for each training sample.

[0023] In this context, at least one of the classification label, difficulty level, and quality level, together with the contribution evaluation, constitutes the classification level information of the training samples, and the classification level information serves as the basis for selecting the target training sample set for training the target model.

[0024] Furthermore, the classification of multiple training samples in the training sample set includes:

[0025] Based on the user attribute data and transaction-related data implemented by the user, transaction risk indication information is matched with various tags in the preset tag system to assign multiple sample tags to any training sample. The tag system covers user attribute tags, transaction mode tags, and transaction risk tags.

[0026] Based on the sample labels of each training sample, the training sample set is clustered, and a cluster group label is generated for each training sample as the classification label.

[0027] Furthermore, the step of classifying the difficulty of multiple training samples in the training sample set includes:

[0028] The difficulty level of the training sample is determined by applying a single second-largest model to the same training sample multiple times and then analyzing the performance metrics of the multiple processing results; or,

[0029] Multiple different second-largest models are invoked to process the same training sample in parallel, and the difficulty level of the training sample is determined based on the combined performance index of the multiple processing results.

[0030] Furthermore, the performance metrics are determined based on the type of training samples:

[0031] When the training samples are of a type with clear reference results, the performance metric is the output accuracy.

[0032] When the training samples are of a type without a clear reference result, the performance metric is perplexity.

[0033] Furthermore, the multiple preset quality dimensions include at least two of the following: hallucination detection, output format standardization, output structure integrity, subject profile description accuracy, analysis dimension richness, argument sufficiency, external evidence citation rationality, risk type orientation, related subject analysis depth, risk confrontation logic, and evidence chain integrity; the quality level is determined based on the weighted sum or arithmetic sum of the scores of each dimension.

[0034] Furthermore, the method also includes:

[0035] Based on the classification labels, the training sample set is stratified and sampled to construct the target training sample set with a preset classification ratio; or,

[0036] The proportion of training samples at different difficulty levels in the target training sample set is dynamically adjusted according to the aforementioned difficulty level; or,

[0037] Training samples with contribution evaluations higher than a first preset threshold and quality levels higher than a second preset threshold are selected to construct the target training sample set.

[0038] Furthermore, the target model is used for risk discrimination tasks and is configured to output risk discrimination results or risk event reports; the target training sample set is used for the supervised fine-tuning SFT stage or reinforcement learning stage of the target model to improve the accuracy, interpretability of conclusions, and quality of risk event report generation in risk discrimination tasks.

[0039] Furthermore, the target model may be the same model or a different model from the quality inspection large model, the first large model, or the second large model.

[0040] Secondly, a hierarchical device for training samples in transaction risk control is provided, the device comprising:

[0041] The acquisition unit is used to acquire a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information.

[0042] The training unit is used to train the first large model through N rounds of iterations using the training sample set obtained by the acquisition unit. In each round of iteration, the sample gradient of each training sample participating in that round of iteration is obtained.

[0043] An evaluation unit is used to calculate, for any first training sample in the training sample set, the ratio of the sample gradient norm of each adjacent iteration in several iterations in which the first training sample participates, and determine the contribution of the first training sample to the model training performed by the training unit based on the ratio.

[0044] The determining unit is used to evaluate the contribution of each training sample in the training sample set obtained by the evaluation unit, and to determine the classification level information of each training sample for the training target model; the target model is used to predict transaction risk.

[0045] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0046] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0047] The method and apparatus provided in the embodiments of this specification first obtain a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the tag data includes transaction risk indication information; then, the first large model is trained using the training sample set for N rounds of iteration, and in each round of iteration, the sample gradient of each training sample participating in that round of iteration is obtained; next, for any first training sample in the training sample set, the ratio of the sample gradient norm of each adjacent iteration in several rounds of iteration in which the first training sample participates is calculated, and the contribution evaluation of the first training sample to the model training is determined based on the ratio; finally, based on the contribution evaluation corresponding to each training sample in the training sample set, the classification level information of each training sample for training the target model is determined; the target model is used to predict transaction risk. As can be seen from the above, the embodiments in this specification do not simply evaluate the training samples based on a single dimension of the samples' conventional attributes, but rather classify them based on the quantitative contribution of each individual training sample to the model training. Since different samples have different degrees of influence on the model learning process, samples with high contribution have a greater impact on the model training effect. Therefore, classifying based on the model learning contribution is more conducive to selecting efficient samples, ensuring that the input samples can help the model's knowledge and learning, thereby improving the efficiency of training samples. In short, this makes the classification of training samples more reasonable and helps to improve the efficiency of model training. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0050] Figure 2 A flowchart illustrating a method for grading training samples in transaction risk control according to one embodiment is shown.

[0051] Figure 3A comparative diagram illustrating sample contribution scores according to one embodiment is shown.

[0052] Figure 4 This diagram illustrates the composition of classification level information according to one embodiment.

[0053] Figure 5 A schematic block diagram of a grading device for training samples in transaction risk control according to one embodiment is shown. Detailed Implementation

[0054] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0055] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves the grading of training samples in transaction risk control. It is understood that in transaction risk control, the target model is a neural network model used to predict transaction risks. A large number of training samples are needed to train the target model to improve its prediction accuracy. The selection of training samples affects the training effect of the target model, and the grading of training samples helps to select suitable training samples. The grading of training samples belongs to data classification and grading. Data classification and grading divides data into categories and levels based on factors such as data attributes, characteristics, and importance, so as to manage, protect, and use the data in a targeted manner. Reasonable data classification and grading can improve data efficiency. Data efficiency refers to the efficiency of various processes applicable to data (such as storage, access, filtering, sharing, etc.). In the embodiments of this specification, data efficiency may specifically include the efficiency of sample selection.

[0056] Reference Figure 1 Taking a training sample set containing 1000 training samples as an example, these training samples are denoted as sample A1, sample A2... sample A1000. By classifying these training samples, they can be divided into several levels, such as level one and level two. Samples A1, A5, A11... belong to level one, while samples A2, A3, A4... belong to level two. Training samples of level one contribute more to model training, while training samples of level two contribute less. When selecting samples later, level one training samples can be prioritized for training the target model, which can accelerate the model iteration efficiency.

[0057] Transaction risk control refers to the use of legal, institutional, and technological means to identify and prevent the legitimization of illicit funds through the financial system. Its core is to track and block the flow of illicit funds into the normal economic system, thereby maintaining financial security, economic order, and social stability.

[0058] The target model can be a large model, which is an artificial intelligence model with a huge number of parameters and a complex structure, usually built based on deep learning techniques. Further, it can be a large language model (LLM), which is a natural language processing model based on deep learning, capable of generating text or performing reasoning through training on massive amounts of data.

[0059] Figure 2 This diagram illustrates a tiered method for training samples in transaction risk control according to one embodiment. The method can be based on... Figure 1 The implementation scenario is shown. For example... Figure 2 As shown, the method for classifying training samples in transaction risk control in this embodiment includes the following steps: Step 21, obtaining a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information; Step 22, using the training sample set to train the first large model for N rounds of iterations, and obtaining the sample gradient of each training sample participating in each round of iteration in each round of iteration; Step 23, for any first training sample in the training sample set, calculating the ratio of the sample gradient norm of each adjacent iteration in several rounds of iteration in which the first training sample participates, and determining the contribution evaluation of the first training sample to the model training based on the ratio; Step 24, based on the contribution evaluation corresponding to each training sample in the training sample set, determining the classification level information of each training sample for the training target model; the target model is used to predict transaction risk. The specific execution method of each of the above steps is described below.

[0060] First, in step 21, a training sample set is obtained. The sample data for each training sample includes user attribute data and transaction-related data for that user. The tag data includes transaction risk indication information. It is understood that user attribute data may include multiple sub-data items, such as individual / enterprise, business / consumption, city level, asset level, occupation, industry type, and business scale. Transaction-related data may also include multiple sub-data items, such as transaction amount range, transaction number range, transaction price range, and transaction time period. Transaction risk indication information may include at least one of the following, such as risk type, risk method tag, and percentage of suspicious funds.

[0061] In the embodiments of this specification, in the field of transaction risk control, the construction of the training sample set may, but is not limited to, follow the standardized process below. First, data sources are collected, extracting raw data from business systems such as transaction systems, user behavior logs, and risk handling work orders, covering user attribute data, transaction-related data, and tag data; then, the extracted raw data is preprocessed and de-identified, including cleaning, feature engineering, and compliance de-identification of the raw data, and preliminary screening of suspected risk samples according to business rules; finally, annotation and verification are performed, with domain experts manually annotating or reviewing the samples to form a labeled sample library.

[0062] The training sample sets constructed using the above methods are unfiltered and unoptimized, and generally suffer from problems such as annotation noise, inconsistent quality, imbalanced distribution of difficulty levels, and sparsity of high-value samples. Directly using them for model training can easily lead to slow convergence, overfitting noise, or insufficient learning of key risk patterns. Therefore, a scientific classification and grading mechanism is needed to evaluate and filter these training sample sets, extracting high-quality subsets to provide accurate data support for model training, and improving data utilization efficiency through intelligent filtering.

[0063] Then, in step 22, the first large model is trained using the training sample set for N iterations. In each iteration, the gradient of each training sample participating in that iteration is obtained. It can be understood that the model processes a batch of samples in each iteration. Unlike the usual processing method, in forward propagation, the loss is calculated separately for each sample in the current batch, and the gradient vector is calculated separately for each sample.

[0064] It should be noted that in conventional model training, only the average gradient of each sample within a batch is calculated. This average gradient is used to update the model parameters, and there is no sample-level gradient storage. In the embodiments of this specification, the individual gradient of each sample is additionally recorded. The gradient is used both for updating the model parameters to ensure normal training and for subsequent sample classification processing. The sample gradient is persistently stored for subsequent contribution evaluation through calculation.

[0065] Next, in step 23, for any first training sample in the training sample set, the ratio of the sample gradient norms of each adjacent iteration in several iterations in which the first training sample participated is calculated. The contribution of the first training sample to the model training is then evaluated based on this ratio. It is understood that the above ratio reflects the change in the sample gradient norms of adjacent iterations. For example, as the number of iterations increases, if the sample gradient norm of a sample decreases significantly, it indicates that the sample contributes significantly to the model training.

[0066] In one example, the model training is a single training process, and the contribution evaluation is a contribution score or contribution level calculated based on this single training process.

[0067] In this example, during a single training run, the number of contribution scores or contribution levels obtained for a single sample is one. For instance, if the training sample set contains 100,000 samples, and these are used for training all at once, each sample will receive a corresponding contribution score or contribution level.

[0068] In another example, the model training includes multiple training processes; the training sample set is divided into multiple sub-training sample sets, and one training process in the multiple training processes includes multiple rounds of iteration using one sub-training sample set from the multiple sub-training sample sets;

[0069] Determining the contribution of the first training sample to the model training includes:

[0070] Based on the several iterations in which the first training sample participates in each training process, the contribution score of the first training sample in each training process is calculated respectively.

[0071] Based on the contribution score of the first training sample in each training process, the comprehensive contribution evaluation of the first training sample is determined.

[0072] In this example, the multiple sub-training sample sets mentioned above can share common training samples, and the first training sample can exist in more than one sub-training sample set. During multiple training processes, a single sample can obtain more than one contribution score. For example, if the training sample set contains 100,000 samples, and during multiple training processes, 30,000 samples are randomly selected each time, resulting in a contribution score for each of these 30,000 samples. After repeated sampling and multiple training processes, some samples may eventually have multiple contribution scores. The average of these multiple contribution scores is taken as the overall contribution evaluation for that sample.

[0073] Furthermore, the comprehensive contribution evaluation is a contribution level; determining the comprehensive contribution evaluation of the first training sample includes:

[0074] Based on the contribution scores of the first training sample in each training process, a comprehensive contribution score for the first training sample is determined, and a hierarchical classification is performed based on this comprehensive contribution score to determine the contribution level of the first training sample; or,

[0075] Based on the contribution scores of the first training sample in each training process, the levels are divided to obtain the contribution level of the first training sample in each training process. Based on the contribution level of the first training sample in each training process, its final contribution level is determined.

[0076] In the embodiments of this specification, if a sample has one contribution score, then based on the contribution scores of all samples, analysis can be performed to stratify the samples into three levels (e.g., high / medium / low), five levels, or more, facilitating selection during model training. If a sample has multiple contribution scores, it can be handled in two ways: first, by statistically analyzing the contribution scores of each sample (e.g., removing the average / maximum / minimum values) to obtain a comprehensive contribution score, and then stratifying the samples to obtain their contribution levels; second, by stratifying multiple times to obtain multiple contribution levels for a single sample, and then performing a majority vote to obtain the final contribution level for the sample.

[0077] Below is a specific example of calculating the contribution score of a sample. The contribution score of a sample reflects its contribution to the model's learning during model training (e.g., the amount of loss reduction, the amount of gradient update).

[0078] sample Contribution score It can be calculated in the following way:

[0079]

[0080] From training step 1 to T, calculate The ratio of the gradient norm before and after each iteration is summed. For the sample The gradient norm at the t-th training step, For the sample The gradient norm at the (t+1)th training step, The model parameters represent the training steps in round t. The model parameters represent the training steps in round t+1. This represents the loss function; ideally, the gradient norm of a sample decreases significantly from the initial training stage to each later round, indicating that the model has learned a lot from that sample, and this is the sample's contribution score. The value of will be larger. To address the different number of iteration rounds each sample participates in, the above contribution score can be normalized, for example, by dividing it by the number of iteration rounds the sample participated in, thus facilitating the comparison of contribution scores of different samples.

[0081] Figure 3 This diagram illustrates a comparison of sample contribution scores according to one embodiment. (Refer to...) Figure 3 The samples are shown respectively. , , The gradient norm at the t-th training step, and the sample , , The gradient norm corresponding to the (t+1)th training step. For the sample The gradient at the t-th training step, For the sample The gradient norm at the t-th training step, For the sample The gradient at the t-th training step, For the sample The gradient norm at the t-th training step, For the sample The gradient at the t-th training step, For the sample The gradient norm at the t-th training step, For the (t+1)th training step, the average gradient vector of all training data, if the gradient of a single sample is... The smaller the angle, the better it is for model training; For the sample The gradient at the (t+1)th training step, For the sample The gradient norm at the (t+1)th training step, For the sample The gradient at the (t+1)th training step, For the sample The gradient norm at the (t+1)th training step, For the sample The gradient at the (t+1)th training step, For the sample The gradient norm at the (t+1)th training step, Let be the average gradient vector of all training data at the (t+2)th training step. By comparing the changes in the gradient norm of each sample in the two rounds in the graph, we can obtain the sample... The change in gradient norm > sample The change in gradient norm > sample The change in the gradient norm, therefore, if we only look at the gradient norm of these two rounds, the sample Contribution score > Sample Contribution score > Sample Contribution score.

[0082] Finally, in step 24, based on the contribution evaluation of each training sample in the training sample set, the classification level information of each training sample for the target model is determined; the target model is used to predict transaction risk. It is understood that the classification level information of the target model may include only one item, which is determined based on the sample's contribution evaluation; the classification level information of the target model may also include multiple items, one of which is determined based on the sample's contribution evaluation.

[0083] In one example, the method further includes at least one of the following:

[0084] Based on the user attribute data and transaction-related data implemented by the user, as well as transaction risk indication information included in the sample data, multiple training samples in the training sample set are classified, and a classification label is generated for each training sample.

[0085] Based on the performance metrics obtained after processing the training samples by calling at least one second-largest model, the difficulty of multiple training samples in the training sample set is classified, and a difficulty level is generated for each training sample.

[0086] Based on the scoring results of the output content of the training samples on multiple preset quality dimensions by calling the quality inspection big model, the quality of multiple training samples in the training sample set is graded, and a quality level is generated for each training sample.

[0087] In this context, at least one of the classification label, difficulty level, and quality level, together with the contribution evaluation, constitutes the classification level information of the training samples, and the classification level information serves as the basis for selecting the target training sample set for training the target model.

[0088] In this example, the classification level information includes multiple components, one of which is a contribution evaluation. It also includes at least one of the classification label, difficulty level, and quality level. Combining these multiple components of the classification level information helps to filter samples from multiple perspectives, which can further accelerate model iteration efficiency.

[0089] Figure 4 A schematic diagram illustrating the composition of classification level information according to one embodiment is shown. (Refer to...) Figure 4 The classification level information includes four items: contribution evaluation, classification label, difficulty level, and quality level. Contribution evaluation helps select high-contribution samples, ensuring that input samples contribute to the model's knowledge and learning, thus improving training efficiency. Classification label enhances sample diversity, enabling rapid stratified sampling of diverse sample sets. Difficulty level ensures a reasonable ratio of easy to difficult samples across different training stages. Quality level helps select high-quality samples, ensuring all input samples are of high quality and preventing low-quality samples from affecting model accuracy and causing output illusions. The combination of these multiple pieces of information effectively avoids negative impacts caused by issues such as sample data ratio or sample quality, accelerating model iteration efficiency.

[0090] Furthermore, the classification of multiple training samples in the training sample set includes:

[0091] Based on the user attribute data and transaction-related data implemented by the user, transaction risk indication information is matched with various tags in the preset tag system to assign multiple sample tags to any training sample. The tag system covers user attribute tags, transaction mode tags, and transaction risk tags.

[0092] Based on the sample labels of each training sample, the training sample set is clustered, and a cluster group label is generated for each training sample as the classification label.

[0093] In this example, given the large amount of content in the sample data, the labeling system and clustering grouping can satisfy various classifications from coarse to fine, better helping modelers construct diverse samples.

[0094] The embodiments in this specification allow for the direct processing of existing data in the database to obtain sample labels, with each sample being labeled based on the user identifier. Since each sample has numerous labels, increasing the difficulty of stratified sampling, clustering can be performed using traditional clustering algorithms, conventional deep learning algorithms, or large-scale models, assigning a cluster group label to each sample.

[0095] For example, traditional clustering algorithms, based on sample labels and statistical features, use conventional clustering algorithms, including but not limited to K-Means, DBSCAN, spectral clustering, and hierarchical clustering, to cluster and group each sample.

[0096] Conventional deep learning algorithms can be categorized into two approaches: a two-step approach, where the first step involves representation learning based on sample labels and statistical features, using a deep learning model to extract embedded representation vectors, and the second step uses a traditional clustering algorithm to group the samples; and an end-to-end approach, which involves comparative learning based on sample labels and statistical features, and then performs end-to-end output tasks for the clustering layer.

[0097] Based on the large model, the sample labels and statistical features are reasonably textualized. The large model is called to summarize and label the users, and then the large model is used to cluster the samples into multiple classes to complete the grouping and labeling based on the large model annotation.

[0098] Furthermore, the step of classifying the difficulty of multiple training samples in the training sample set includes:

[0099] The difficulty level of the training sample is determined by applying a single second-largest model to the same training sample multiple times and then analyzing the performance metrics of the multiple processing results; or,

[0100] Multiple different second-largest models are invoked to process the same training sample in parallel, and the difficulty level of the training sample is determined based on the combined performance index of the multiple processing results.

[0101] In this example, due to the problem of high randomness in the difficulty of making decisions on large models with a single call, a comprehensive difficulty assessment is carried out by calling a single model multiple times or calling multiple models. This effectively avoids the bias caused by the randomness of a single model and makes the difficulty classification more accurate.

[0102] Furthermore, the performance metrics are determined based on the type of training samples:

[0103] When the training samples are of a type with clear reference results, the performance metric is the output accuracy.

[0104] When the training samples are of a type without a clear reference result, the performance metric is perplexity.

[0105] In this example, difficulty levels can be effectively graded for different types of training samples.

[0106] For example, a difficulty score is assigned to each training sample, and then the difficulty level is determined based on the score. For types with clear reference results, such as the model output indicating whether there is risk, the accuracy of the output can be calculated by calling the large model multiple times as the difficulty score. For types without clear reference results, such as the model output indicating the writing of a report, the perplexity of the model output can be used as the difficulty score. Finally, the difficulty is stratified based on the score, resulting in three levels (e.g., hard / medium / easy), five levels, or more, as needed, which facilitates selection during model training.

[0107] Specifically, there are multiple ways to implement difficulty grading, such as single-model difficulty grading, multi-model difficulty grading, and finally, combining the results of multiple difficulty grading to obtain the final difficulty level of the sample.

[0108] This approach, based on single-model difficulty grading, involves using a single model to run the same sample N times, statistically analyzing the output accuracy or average / maximum / minimum perplexity scores. The single model can be a well-performing open-source model or a finely tuned large model for the given scenario. The same model is run N times, and the output accuracy (or perplexity score) for that sample is statistically analyzed. Finally, the entire dataset is analyzed, and stratification is performed based on the output accuracy (or perplexity) to obtain multi-level difficulty grading results as needed. Fine-tuning the large model, i.e., the target model, is more conducive to the next iteration of training the target model.

[0109] This approach utilizes a multi-model difficulty grading system, where multiple large models are simultaneously invoked on the same sample, and the output accuracy (or average / maximum / minimum perplexity score, etc.) is statistically analyzed. The system selects the necessary open-source large models for comparison, as well as several existing fine-tuned large models. It then selects N models to simultaneously invoke the same sample, statistically analyzing the output accuracy (or perplexity statistics) for that sample. Finally, it performs a comprehensive analysis of the entire sample set, stratifying the data based on the output accuracy (or perplexity) to obtain multi-level difficulty grading results as needed.

[0110] There are various methods for assigning difficulty scores to individual samples, based on single-model or multi-model difficulty grading, and for determining the overall difficulty grading results. For ease of application, multiple difficulty grading results can be combined to obtain a comprehensive difficulty grading. One approach is to vote on multiple difficulty grading results, with the grading receiving the most votes becoming the comprehensive grading result. Another approach is to statistically analyze multiple difficulty grading scores and then stratify them, for example, by statistically analyzing the comprehensive difficulty scores of various models (e.g., average / maximum / minimum output accuracy). It is expected that the comprehensive difficulty scores will be stratified as needed to complete the difficulty grading.

[0111] Furthermore, the multiple preset quality dimensions include at least two of the following: hallucination detection, output format standardization, output structure integrity, subject profile description accuracy, analysis dimension richness, argument sufficiency, external evidence citation rationality, risk type orientation, related subject analysis depth, risk confrontation logic, and evidence chain integrity; the quality level is determined based on the weighted sum or arithmetic sum of the scores of each dimension.

[0112] In this example, the preset quality dimension considers not only the model's output but also the relevant intermediate thought processes, facilitating the selection of high-quality samples. A quality score is provided based on a large model, improving scoring efficiency and eliminating the subjectivity of manual scoring.

[0113] The embodiments in this specification can complete sample quality grading based on sample quality scores. After analysis and stratification according to all sample quality scores, three-level (e.g., high / medium / low), five-level, or more-level quality gradings can be obtained as needed, facilitating selection during model training.

[0114] Furthermore, the method also includes:

[0115] Based on the classification labels, the training sample set is stratified and sampled to construct the target training sample set with a preset classification ratio; or,

[0116] The proportion of training samples at different difficulty levels in the target training sample set is dynamically adjusted according to the aforementioned difficulty level; or,

[0117] Training samples with contribution evaluations higher than a first preset threshold and quality levels higher than a second preset threshold are selected to construct the target training sample set.

[0118] In this example, training samples can be selected by combining one or more of the classification level information. Stratified sampling of the training sample set based on the classification labels ensures the diversity of the target training sample set, preventing the model from learning only discrimination patterns for specific groups or risk types, and enabling the model to achieve comprehensive coverage. Dynamically adjusting the proportion of training samples of different difficulty levels in the target training sample set according to the difficulty level ensures that the samples in the target training sample set are of moderate difficulty. A dataset consisting entirely of simple samples may cause the model to learn only simple patterns and fail to identify difficult samples, while a dataset consisting entirely of difficult samples may lead to high learning difficulty and convergence difficulties. Selecting samples with high contribution ratings improves model training performance. Each training sample involves different industries and risk discrimination knowledge related to its transaction patterns, and different samples have different impacts on the model learning process. Research has shown that samples with high knowledge content and high contribution have a significant impact on the final model performance. Selecting high-quality samples leads to better model training results. The quality of the target training sample set is more important than the quantity of samples. In the post-training stage of large models, a large amount of practical experience and related research have shown that the quality of training samples is more important than the quantity of samples.

[0119] Furthermore, the target model is used for risk discrimination tasks and is configured to output risk discrimination results or risk event reports; the target training sample set is used for the supervised fine-tuning SFT stage or reinforcement learning stage of the target model to improve the accuracy, interpretability of conclusions, and quality of risk event report generation in risk discrimination tasks.

[0120] Furthermore, the target model may be the same model or a different model from the quality inspection large model, the first large model, or the second large model.

[0121] The method provided in the embodiments of this specification first obtains a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information; then, the first large model is trained using the training sample set for N rounds of iteration, and in each round of iteration, the sample gradient of each training sample participating in that round of iteration is obtained; next, for any first training sample in the training sample set, the ratio of the sample gradient norm of each adjacent iteration in several rounds of iteration in which the first training sample participates is calculated, and the contribution evaluation of the first training sample to the model training is determined based on the ratio; finally, based on the contribution evaluation corresponding to each training sample in the training sample set, the classification level information of each training sample for training the target model is determined; the target model is used to predict transaction risk. As can be seen from the above, the embodiments in this specification do not simply evaluate the training samples based on a single dimension of the samples' conventional attributes, but rather classify them based on the quantitative contribution of each individual training sample to the model training. Since different samples have different degrees of influence on the model learning process, samples with high contribution have a greater impact on the model training effect. Therefore, classifying based on the model learning contribution is more conducive to selecting efficient samples, ensuring that the input samples can help the model's knowledge and learning, thereby improving the efficiency of training samples. In short, this makes the classification of training samples more reasonable and helps to improve the efficiency of model training.

[0122] According to another embodiment, a grading device for training samples in transaction risk control is also provided, which is used to perform the method provided in the embodiments of this specification. Figure 5 A schematic block diagram of a grading device for training samples in transaction risk control according to one embodiment is shown. Figure 5 As shown, the device 500 includes:

[0123] The acquisition unit 51 is used to acquire a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information.

[0124] The training unit 52 is used to train the first large model through N rounds of iterations using the training sample set obtained by the acquisition unit 51. In each round of iteration, the sample gradient of each training sample participating in that round of iteration is obtained.

[0125] Evaluation unit 53 is used to calculate the ratio of the sample gradient norm of each adjacent iteration in several iterations in which the first training sample participates in several rounds of iteration for any first training sample in the training sample set, and determine the contribution evaluation of the first training sample to the model training performed by the training unit 52 based on the ratio.

[0126] The determining unit 54 is used to evaluate the contribution of each training sample in the training sample set obtained by the evaluation unit 53, and to determine the classification level information of each training sample for the training target model; the target model is used to predict transaction risk.

[0127] Optionally, as an embodiment, the model training is a single training process, and the contribution evaluation is a contribution score or contribution level calculated based on the single training process.

[0128] Optionally, as an embodiment, the model training includes multiple training processes; the training sample set is divided into multiple sub-training sample sets, and one training process in the multiple training processes includes multiple rounds of iteration using one sub-training sample set from the multiple sub-training sample sets;

[0129] Evaluation unit 53 includes:

[0130] The evaluation subunit is used to calculate the contribution score of the first training sample in each training process based on the several iterations in which the first training sample participates in each training process.

[0131] A sub-unit is determined, which is used to determine the comprehensive contribution evaluation of the first training sample based on the contribution score of the first training sample obtained from the evaluation sub-unit in each training process.

[0132] Further, the comprehensive contribution evaluation is a contribution level; the determination sub-unit is specifically used for: determining the comprehensive contribution score of the first training sample based on the contribution scores corresponding to the first training sample in each training process, and performing hierarchical division based on the comprehensive contribution score to determine the contribution level of the first training sample; or, performing hierarchical division based on the contribution scores corresponding to the first training sample in each training process to obtain the contribution level of the first training sample in each training process, and determining its final contribution level based on the contribution level of the first training sample in each training process.

[0133] Optionally, as an embodiment, the device further includes at least one of the following units:

[0134] A classification unit is used to classify multiple training samples in the training sample set based on the user attribute data and transaction-related data implemented by the user, as well as transaction risk indication information, and to generate a classification label for each training sample.

[0135] The difficulty grading unit is used to grade the difficulty of multiple training samples in the training sample set based on the performance index obtained after processing the training samples by calling at least one second-largest model, and to generate a difficulty level for each training sample.

[0136] The quality grading unit is used to grade the quality of multiple training samples in the training sample set based on the scoring results of the output content of the training samples on multiple preset quality dimensions by calling the quality inspection big model, and to generate a quality level for each training sample.

[0137] In this context, at least one of the classification label, difficulty level, and quality level, together with the contribution evaluation, constitutes the classification level information of the training samples, and the classification level information serves as the basis for selecting the target training sample set for training the target model.

[0138] Furthermore, the classification unit includes:

[0139] The tag allocation subunit is used to match the user attribute data and transaction-related data implemented by the user, transaction risk indication information, and various tags in the preset tag system to assign multiple sample tags to any training sample. The tag system covers user attribute tags, transaction mode tags, and transaction risk tags.

[0140] The clustering subunit is used to perform clustering processing on the training sample set based on the sample labels obtained from the label allocation subunit, and generate a clustering group label for each training sample as the classification label.

[0141] Furthermore, the difficulty grading unit is specifically used to: call a single second-largest model to process the same training sample multiple times, and determine the difficulty level of the training sample based on the statistical value of the performance index of the multiple processing results; or, call multiple different second-largest models to process the same training sample in parallel, and determine the difficulty level of the training sample based on the comprehensive value of the performance index of the multiple processing results.

[0142] Furthermore, the performance metrics are determined based on the type of training samples:

[0143] When the training samples are of a type with clear reference results, the performance metric is the output accuracy.

[0144] When the training samples are of a type without a clear reference result, the performance metric is perplexity.

[0145] Furthermore, the multiple preset quality dimensions include at least two of the following: hallucination detection, output format standardization, output structure integrity, subject profile description accuracy, analysis dimension richness, argument sufficiency, external evidence citation rationality, risk type orientation, related subject analysis depth, risk confrontation logic, and evidence chain integrity; the quality level is determined based on the weighted sum or arithmetic sum of the scores of each dimension.

[0146] Furthermore, the device also includes:

[0147] The construction unit is used to perform stratified sampling of the training sample set based on the classification label to construct the target training sample set with a preset classification ratio; or, dynamically adjust the ratio of training samples of different difficulty levels in the target training sample set according to the difficulty level; or, screen training samples with contribution evaluation higher than a first preset threshold and quality level higher than a second preset threshold to construct the target training sample set.

[0148] Furthermore, the target model is used for risk discrimination tasks and is configured to output risk discrimination results or risk event reports; the target training sample set is used for the supervised fine-tuning SFT stage or reinforcement learning stage of the target model to improve the accuracy, interpretability of conclusions, and quality of risk event report generation in risk discrimination tasks.

[0149] Furthermore, the target model may be the same model or a different model from the quality inspection large model, the first large model, or the second large model.

[0150] Using the apparatus provided in the embodiments of this specification, firstly, the acquisition unit 51 acquires a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the tag data includes transaction risk indication information; then, the training unit 52 uses the training sample set to perform N rounds of model training on the first large model, and in each round of iteration, obtains the sample gradient of each training sample participating in that round of iteration; next, the evaluation unit 53 calculates the ratio of the sample gradient norm of each adjacent iteration in several rounds of iteration for any first training sample in the training sample set, and determines the contribution evaluation of the first training sample to the model training based on the ratio; finally, the determination unit 54 determines the classification level information of each training sample for the training target model based on the contribution evaluation corresponding to each training sample in the training sample set; the target model is used to predict transaction risk. As can be seen from the above, the embodiments in this specification do not simply evaluate the training samples based on a single dimension of the samples' conventional attributes, but rather classify them based on the quantitative contribution of each individual training sample to the model training. Since different samples have different degrees of influence on the model learning process, samples with high contribution have a greater impact on the model training effect. Therefore, classifying based on the model learning contribution is more conducive to selecting efficient samples, ensuring that the input samples can help the model's knowledge and learning, thereby improving the efficiency of training samples. In short, this makes the classification of training samples more reasonable and helps to improve the efficiency of model training.

[0151] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0152] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.

[0153] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0154] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for grading training samples in transaction risk control, the method comprising: Obtain the training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information; The first large model is trained through N iterations using the training sample set. In each iteration, the sample gradient of each training sample participating in that iteration is obtained. For any first training sample in the training sample set, calculate the ratio of the sample gradient norms of each adjacent iteration in several iterations in which the first training sample participates, and determine the contribution of the first training sample to the model training based on the ratio. Based on the contribution evaluation of each training sample in the training sample set, the classification level information of each training sample for the training target model is determined. The target model is used to predict transaction risk.

2. The method of claim 1, wherein, The model training is a single training process, and the contribution evaluation is a contribution score or contribution level calculated based on this single training process.

3. The method as described in claim 1, wherein, The model training includes multiple training processes; the training sample set is divided into multiple sub-training sample sets, and one training process in the multiple training processes includes multiple rounds of iteration using one sub-training sample set from the multiple sub-training sample sets; Determining the contribution of the first training sample to the model training includes: Based on the several iterations in which the first training sample participates in each training process, the contribution score of the first training sample in each training process is calculated respectively. Based on the contribution score of the first training sample in each training process, the comprehensive contribution evaluation of the first training sample is determined.

4. The method of claim 3, wherein, The comprehensive contribution evaluation is a contribution level; Determine the overall contribution evaluation of the first training sample, including: Based on the contribution scores of the first training sample in each training process, the comprehensive contribution score of the first training sample is determined, and the contribution level of the first training sample is determined by hierarchical division based on the comprehensive contribution score. or, Based on the contribution scores of the first training sample in each training process, the levels are divided to obtain the contribution level of the first training sample in each training process. Based on the contribution level of the first training sample in each training process, its final contribution level is determined.

5. The method of claim 1, wherein, The method further includes at least one of the following: Based on the user attribute data and transaction-related data implemented by the user, as well as transaction risk indication information included in the sample data, multiple training samples in the training sample set are classified, and a classification label is generated for each training sample. Based on the performance metrics obtained after processing the training samples by calling at least one second-largest model, the difficulty of multiple training samples in the training sample set is classified, and a difficulty level is generated for each training sample. Based on the scoring results of the output content of the training samples on multiple preset quality dimensions by calling the quality inspection big model, the quality of multiple training samples in the training sample set is graded, and a quality level is generated for each training sample. In this context, at least one of the classification label, difficulty level, and quality level, together with the contribution evaluation, constitutes the classification level information of the training samples, and the classification level information serves as the basis for selecting the target training sample set for training the target model.

6. The method of claim 5, wherein, The classification of multiple training samples in the training sample set includes: Based on the user attribute data and transaction-related data implemented by the user, transaction risk indication information is matched with various tags in the preset tag system to assign multiple sample tags to any training sample. The tag system covers user attribute tags, transaction mode tags, and transaction risk tags. Based on the sample labels of each training sample, the training sample set is clustered, and a cluster group label is generated for each training sample as the classification label.

7. The method of claim 5, wherein, The step of classifying the difficulty of multiple training samples in the training sample set includes: The difficulty level of the training sample is determined by applying a single second-largest model to the same training sample multiple times and then analyzing the performance metrics of the multiple processing results; or, Multiple different second-largest models are invoked to process the same training sample in parallel, and the difficulty level of the training sample is determined based on the combined performance index of the multiple processing results.

8. The method of claim 7, wherein, The performance metrics are determined based on the type of training samples: When the training samples are of a type with clear reference results, the performance metric is the output accuracy. When the training samples are of a type without a clear reference result, the performance metric is perplexity.

9. The method of claim 5, wherein, The multiple preset quality dimensions include at least two of the following: hallucination detection, output format standardization, output structure integrity, subject profile description accuracy, analysis dimension richness, argument sufficiency, external evidence citation rationality, risk type orientation, related subject analysis depth, risk confrontation logic, and evidence chain integrity; the quality level is determined based on the weighted sum or arithmetic sum of the scores of each dimension.

10. The method of claim 5, wherein, The method further includes: Based on the classification labels, the training sample set is stratified and sampled to construct the target training sample set with a preset classification ratio; or, The proportion of training samples at different difficulty levels in the target training sample set is dynamically adjusted according to the aforementioned difficulty level; or, Training samples with contribution evaluations higher than a first preset threshold and quality levels higher than a second preset threshold are selected to construct the target training sample set.

11. The method of claim 10, wherein, The target model is used for risk discrimination tasks and is configured to output risk discrimination results or risk event reports. The target training sample set is used for the supervised fine-tuning SFT stage or reinforcement learning stage of the target model to improve the accuracy, interpretability of conclusions, and quality of risk event report generation in risk discrimination tasks.

12. The method of claim 5, wherein, The target model is the same model as the quality inspection large model, the first large model, or the second large model, or they may be different models.

13. A grading device for training samples in transaction risk control, the device comprising: The acquisition unit is used to acquire a training sample set, wherein the sample data of each training sample includes user attribute data and transaction-related data implemented by the user, and the label data includes transaction risk indication information. The training unit is used to train the first large model through N rounds of iterations using the training sample set obtained by the acquisition unit. In each round of iteration, the sample gradient of each training sample participating in that round of iteration is obtained. An evaluation unit is used to calculate, for any first training sample in the training sample set, the ratio of the sample gradient norm of each adjacent iteration in several iterations in which the first training sample participates, and determine the contribution of the first training sample to the model training performed by the training unit based on the ratio. The determining unit is used to evaluate the contribution of each training sample in the training sample set obtained by the evaluation unit and determine the classification level information of each training sample for the training target model. The target model is used to predict transaction risks.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-12.

15. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-12.

Images