Method and apparatus for training a risk detection model
By selecting risk samples from transaction data and combining them with business scenario characteristics, and by using boundary queries and incremental learning to train the model, the problems of low accuracy of risk detection models and high cost of manual labeling were solved, achieving high-accuracy risk detection under limited budget.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
- Filing Date
- 2023-03-10
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, the accuracy of risk samples used to train risk detection models is low, and manual labeling is costly and makes it difficult to obtain high-accuracy risk samples.
By screening candidate samples from transaction data, combining business scenario dialogue features and preset features, risk samples are determined, and a risk detection model is trained using boundary query strategy and incremental learning method.
Accurately identifying risk samples within a limited budget improves the accuracy of the risk detection model, compensates for omissions in sample identification, and maintains the model's performance stability on historical samples.
Smart Images
Figure CN116306975B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the field of information security technology, and in particular to a method and apparatus for training risk detection models. Background Technology
[0002] Currently, risk samples used to train risk detection models are typically mined through anomaly detection models. Therefore, the accuracy of these risk samples is relatively low. To improve the accuracy, samples need to be manually labeled to obtain high-accuracy risk samples. However, manual labeling relies heavily on expert experience and is extremely time-consuming. Therefore, how to obtain high-accuracy risk samples and thus develop a precise risk detection model is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0003] This specification describes one or more embodiments of a risk detection model training method and apparatus that can obtain risk samples with high accuracy, thereby obtaining an accurate risk detection model.
[0004] According to the first aspect, this specification provides a method for training a risk detection model, the method comprising:
[0005] Candidate samples are determined from a massive sample of various transaction data; wherein the number of candidate samples is less than a first threshold, and the feature values of the candidate samples satisfy a second threshold.
[0006] Based on the corresponding dialogue scripts for business scenarios and the preset dialogue features, additional features of candidate samples are determined. These additional features are used to characterize the credibility of the samples.
[0007] Risk samples are determined based on the candidate samples and their additional characteristics;
[0008] Risk detection models are trained based on risk samples.
[0009] Optionally, the second threshold value ranges from 0.6 to 1.
[0010] Optionally, the dialogue features include at least one of the following: the progress of the dialogue, the mood of the dialogue, and features that deviate from reality.
[0011] Optionally, risk samples are determined based on candidate samples and their additional characteristics, including:
[0012] Among the candidate samples, the sample whose credibility meets the first preset condition is determined as the first target sample;
[0013] Based on the boundary query strategy, risk samples in the first target sample are identified. The boundary query strategy is to select samples whose confidence level is within the boundary value range.
[0014] Optionally, based on the boundary query strategy, risk samples in the second target sample are identified, including:
[0015] Based on the boundary query strategy, the second target sample is selected from the first target sample;
[0016] When the error between the credibility of the second target sample and the preset standard value meets the second preset condition, the second target sample is identified as a risk sample.
[0017] Optionally, a risk detection model is trained based on risk samples, including:
[0018] A risk detection model is trained based on risk samples and the risk values of each sample within those risk samples. The risk values of the samples are determined by the historical risk detection model based on the risk samples.
[0019] Optionally, a risk detection model is created based on risk samples, including:
[0020] When the number of risk samples meets the preset value, the risk detection model is trained based on the risk samples and historical samples in the sample library.
[0021] According to the second aspect, this specification provides a training device for a risk detection model, the device comprising: a first determining module, a second determining module, and a training module, wherein,
[0022] The first determination module is configured to: determine candidate samples from a massive sample of various transaction data; wherein the number of candidate samples is less than a first threshold, and the feature values of the candidate samples satisfy a second threshold;
[0023] The second determination module is configured as follows: based on the corresponding dialogue scripts for the business scenario and the preset dialogue features, determine the additional features of the candidate samples. The additional features are used to characterize the credibility of the samples.
[0024] The third determination module is configured as follows: based on the candidate samples and their additional features, determine the risk samples;
[0025] The training module is configured to train a risk detection model based on risk samples.
[0026] Optionally, the second threshold value ranges from 0.6 to 1.
[0027] Optionally, the dialogue features include at least one of the following: the progress of the dialogue, the mood of the dialogue, and features that deviate from reality.
[0028] Optionally, the third determining module is configured as follows:
[0029] Among the candidate samples, the sample whose credibility meets the first preset condition is determined as the first target sample;
[0030] Based on the boundary query strategy, risk samples in the first target sample are identified. The boundary query strategy is to select samples whose confidence level is within the boundary value range.
[0031] Optionally, the third determining module is configured as follows:
[0032] Based on the boundary query strategy, the second target sample is selected from the first target sample;
[0033] When the error between the credibility of the second target sample and the preset standard value meets the second preset condition, the second target sample is identified as a risk sample.
[0034] Optionally, the training module is configured as follows:
[0035] A risk detection model is trained based on risk samples and the risk values of each sample within those risk samples. The risk values of the samples are determined by the historical risk detection model based on the risk samples.
[0036] Optionally, the training module is configured as follows:
[0037] When the number of risk samples meets the preset value, the risk detection model is trained based on the risk samples and historical samples in the sample library.
[0038] According to a third aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in any embodiment of this specification.
[0039] The risk detection model training method and apparatus provided in the embodiments of this specification determine candidate samples from a massive sample of various transaction data, and then determine risk samples by combining them with the corresponding dialogue based on the business scenario and preset dialogue features. This can compensate for the problem of missed risk samples during the risk sample identification process, and obtain more and more accurate risk samples within a limited budget. Subsequently, the risk detection model is trained based on the risk samples, thereby obtaining an accurate risk detection model.
[0040] A sample model is obtained by training samples with configured labels, and this model is used to filter risk samples. This allows for accurate identification of risk samples during the sample screening process. Furthermore, a sample model is designed for specific application scenarios. After the dialogue ends, it comprehensively considers dialogue information, emotional information, and specific scene features to make a judgment, thus overcoming the problem of missing too many samples when relying solely on text information for recognition. With a limited outbound call budget, more risk samples can be identified.
[0041] When the number of risk samples is similar to the number of historical samples in the sample library, the risk detection model can be trained using both risk samples and historical samples from the sample library. In this way, when the number of risk samples meets a preset value, it becomes similar in magnitude to the number of historical samples in the sample library. Therefore, directly adding risk samples to historical samples for joint retraining avoids the problem of the risk sample size being too small to be effective.
[0042] When the number of risk samples differs significantly from the number of historical samples in the sample library, an incremental learning method is employed. The risk sample is input into the historical risk detection model, which outputs the risk value for each sample in the risk sample. Then, using the risk sample and the risk values output by the historical risk detection model, a new risk detection model is trained. This constrains the parameters of the risk detection model, ensuring that the prediction results of the new risk detection model on risk samples are similar to those of the historical risk detection model. This prevents the performance of the new risk detection model from degrading on historical samples while maintaining good recognition results on risk samples. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is a flowchart of a training method for a risk detection model provided in one embodiment of this specification;
[0045] Figure 2 This is a flowchart of a training method for a risk detection model provided in one embodiment of this specification;
[0046] Figure 3 This is a schematic diagram of a training device for a risk detection model in one embodiment of this specification. Detailed Implementation
[0047] The solution provided in this specification will now be described with reference to the accompanying drawings.
[0048] Figure 1 A flowchart illustrating a training method for a risk detection model according to one embodiment is provided. It will be understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. See also... Figure 1 The training method for the risk detection model provided in the embodiments of this specification includes the following steps:
[0049] Step 101: Determine candidate samples from a massive sample of various transaction data.
[0050] The number of candidate samples is less than a first threshold, and the feature values of the candidate samples satisfy a second threshold. Optionally, the value range of the second threshold can be 0.6-1. That is, the second threshold can be any value between 0.6 and 1. The feature values of the candidate samples are any values between the second threshold and 0.6-1.
[0051] The first and second thresholds are arbitrarily defined values, and are not specifically limited in this embodiment of the invention. In actual implementation, they are set according to actual needs.
[0052] This step can be specifically implemented as follows: An unsupervised anomaly detection algorithm is used to determine candidate samples from a massive sample of various transaction data. For example, the unsupervised anomaly detection algorithm can be the Isolation Forest algorithm. Of course, other algorithms can also be used for unsupervised anomaly detection, and this embodiment of the invention does not impose specific limitations. Specifically, based on the Isolation Forest algorithm, samples with a number less than a first threshold are selected from the massive sample of various transaction data, and then samples whose feature values satisfy a second threshold are selected from these samples; these samples are the candidate samples.
[0053] Step 103: Based on the corresponding dialogue scripts for the business scenario and the preset dialogue features, determine the additional features of the candidate samples. The additional features are used to characterize the credibility of the samples.
[0054] The additional feature can be used to characterize the credibility of the sample. For example, when the sample is credible, the additional feature can be 1; when the sample is uncredible, the additional feature can be 0. Of course, the additional feature can also be expressed as a percentage. For example, when the sample is credible, the additional feature can be 100%; when the sample is both credible and uncredible, the additional feature can be 50%; when the sample is uncredible, the additional feature can be 0.
[0055] Optionally, the dialogue features include at least one of the following: the progress of the dialogue, the mood of the dialogue, and features that deviate from reality.
[0056] The progress of the conversation can be understood as the progress within the set script. For example, suppose the script is: Are you buying a product? Have you paid? Is the payment amount the specified amount? Is there a refund? Correspondingly, the progress of the conversation can be understood as which question has been asked according to the script.
[0057] In this context, the emotion in the dialogue can refer to the respondent's emotion. This emotion can be determined based on emotional text and / or interjections in the dialogue, such as "I've said it several times already," "Ugh," "I'm very busy," etc., which would indicate that the emotion in the dialogue is irritability. The emotion can also be determined based on the tone of voice; for example, by acquiring the audio of the dialogue and determining the tone of voice based on the amplitude of the audio, the emotion in the dialogue can be determined.
[0058] Among these, the deviation from reality can be understood as whether it conforms to the facts. For example, if the actual transaction involves equipment, but the transaction takes place in a supermarket, then it deviates from reality, and the characteristic of this deviation is "equipment". Another example is purchasing a product; if the actual transaction amount is 100,000, but the transaction takes place in a supermarket where the largest transaction amount is 10,000, then it deviates from reality, and the characteristic of this deviation is "100,000".
[0059] This step can be specifically implemented as follows: Based on the candidate samples obtained in step 101, the number of outbound call samples can be selected according to the outbound call budget. For example, if the outbound call budget is 10,000 calls per day, then 10,000 samples are randomly selected from the candidate samples obtained in step 101. Labels are assigned to the 10,000 samples based on the corresponding dialogue for the business scenario. For example, in a cash-out scenario, the corresponding dialogue could be whether you are buying goods, whether you have paid, whether the payment amount is the specified amount, whether there is a refund, etc. Similarly, in a fraud scenario, the corresponding dialogue could be whether someone claims to be from the public security, procuratorate, or court, whether there is a promise of investment returns, etc. At the end of the conversation, it is determined whether the user is lying based on the content of the conversation. If the user is lying, a reliable label is assigned to the sample; if the user is not lying, an unreliable label is assigned to the sample. Because users are often wary in many risky scenarios and usually do not answer questions directly, the proportion of risky samples collected is relatively low and may not meet the training requirements for training a risk detection model. To obtain accurate risk samples, the aforementioned business scenarios correspond to pre-defined dialogue features, which include at least one of the following: the progress of the dialogue, the emotional tone of the dialogue, and deviations from reality. Based on the dialogue script corresponding to the business scenarios and the pre-defined dialogue features, labels are then assigned to 10,000 samples. For example, a reliable label can be assigned a value of 1, and an unreliable label can be assigned a value of 0.
[0060] Of course, in this embodiment of the invention, the samples with the configured labels described above can be used to train a sample model, which is then used to screen for risky samples. This allows for accurate identification of risky samples during the sample screening process. Furthermore, a sample model is designed for specific application scenarios. After the dialogue ends, it comprehensively considers dialogue information, emotional information, and specific scene features to make a judgment, thus overcoming the problem of missing too many samples when relying solely on textual information for recognition. With a limited outbound call budget, more risky samples can be identified.
[0061] Step 105: Based on the candidate samples and their additional characteristics, determine the risk samples.
[0062] In a specific feasible method, such as Figure 2 As shown, step 105 may specifically include: Step 1051: In the candidate samples, determine the samples whose credibility meets the first preset condition as the first target samples; wherein, the first preset condition can be a condition set according to the scenario, for example, the first preset condition may include a label value of 1. That is, step 1051 specifically means: in the candidate samples, determine the samples whose label value is 1 as the first target samples. Alternatively, the first preset condition may include a label value of 45%-100%. That is, step 1051 specifically means: in the candidate samples, determine the samples whose label value is 45%-100% as the first target samples. Step 1053: Based on the boundary query strategy, determine the risk samples in the first target samples, where the boundary query strategy is to select samples whose credibility is within the boundary value range. Or, the boundary query strategy is: the error between the credibility and the preset standard value meets the second preset condition.
[0063] Optionally, step 1053 may specifically involve: based on a boundary query strategy, selecting a second target sample from the first target sample; when the error between the credibility of the second target sample and a preset standard value meets a second preset condition, the second target sample is identified as a risk sample. The second preset condition can be 5%, meaning that samples with a label assignment value of 45%-55% are selected as the second target sample, which is a sample that is relatively difficult to judge.
[0064] Step 107: Train the risk detection model based on risk samples.
[0065] In one specific implementation, step 107 can be: when the number of risk samples meets a preset value, a risk detection model is trained based on the risk samples and historical samples in the sample library. That is, when the number of risk samples is similar to the number of historical samples in the sample library, the risk detection model can be trained using the risk samples and historical samples in the sample library.
[0066] In this way, when the number of risk samples meets the preset value, it can be close to the order of magnitude of the historical samples in the sample library. This allows risk samples to be directly put into historical samples for joint retraining, effectively avoiding the phenomenon that the number of risk samples is too small to have an obvious effect.
[0067] In another specific implementation, step 107 can be: training a risk detection model based on risk samples and the risk values of each sample in the risk samples, wherein the risk values of the samples are determined by the historical risk detection model based on the risk samples.
[0068] The historical risk detection model can be understood as an existing risk detection model, which is trained based on historical samples, rather than on the risk samples obtained in the embodiments of this application.
[0069] In this step, after obtaining the risk samples from steps 101-105 above, an incremental learning method is used to input these risk samples into the historical risk detection model, outputting the risk value of each sample in the risk sample. Then, the risk samples and their respective risk values are used to train the risk detection model. This constrains the parameters of the risk detection model, ensuring that its prediction results on the risk samples are similar to those of the historical risk detection model, thus preventing performance degradation on historical samples while maintaining good recognition results on the risk samples.
[0070] This specification also provides a training device for the risk detection model, which can be any device, equipment, platform, or cluster of devices with computing and processing capabilities. See also Figure 3 The risk detection model training device 300 provided in this embodiment includes: a first determining module 301, a second determining module 303, a third determining module 305, and a training module 307, wherein...
[0071] The first determining module 301 is configured to: determine candidate samples from a massive sample of various transaction data; wherein the number of candidate samples is less than a first threshold, and the feature values of the candidate samples satisfy a second threshold;
[0072] The second determining module 303 is configured to: determine additional features of candidate samples based on the corresponding dialogue scripts for the business scenario and preset dialogue features. The additional features are used to characterize the credibility of the samples.
[0073] The third determination module 305 is configured to: determine risk samples based on candidate samples and their additional characteristics;
[0074] Training module 307 is configured to train a risk detection model based on risk samples.
[0075] Optionally, the second threshold value ranges from 0.6 to 1.
[0076] Optionally, the dialogue features include at least one of the following: the progress of the dialogue, the mood of the dialogue, and features that deviate from reality.
[0077] Optionally, the third determining module 305 is configured as follows:
[0078] Among the candidate samples, the sample whose credibility meets the first preset condition is determined as the first target sample;
[0079] Based on the boundary query strategy, risk samples in the first target sample are identified. The boundary query strategy is to select samples whose confidence level is within the boundary value range.
[0080] Optionally, the third determining module 305 is configured as follows:
[0081] Based on the boundary query strategy, the second target sample is selected from the first target sample;
[0082] When the error between the credibility of the second target sample and the preset standard value meets the second preset condition, the second target sample is identified as a risk sample.
[0083] Optionally, training module 307 is configured as follows:
[0084] A risk detection model is trained based on risk samples and the risk values of each sample within those risk samples. The risk values of the samples are determined by the historical risk detection model based on the risk samples.
[0085] Optionally, training module 307 is configured as follows:
[0086] When the number of risk samples meets the preset value, the risk detection model is trained based on the risk samples and historical samples in the sample library.
[0087] The training apparatus for the risk detection model provided in the embodiments of this specification determines candidate samples from a massive sample of various transaction data, and then determines risk samples by combining them with the corresponding dialogue based on the business scenario and preset dialogue features. This can compensate for the problem of missed risk samples during the risk sample identification process, and obtain more and more accurate risk samples within a limited budget. Subsequently, the risk detection model is trained based on the risk samples, thereby obtaining an accurate risk detection model.
[0088] This specification provides, in one embodiment, a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods of any embodiment in the specification.
[0089] This specification provides a computing device according to one embodiment, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to perform the method of any embodiment of the specification.
[0090] It is understood that the structures illustrated in the embodiments of this specification do not constitute a specific limitation on the training device for the risk detection model. In other embodiments of this specification, the training device for the risk detection model may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0091] The information interaction and execution process between the modules in the above-mentioned device and system are based on the same concept as the method embodiments in this specification, and the specific details can be found in the descriptions in the method embodiments in this specification, so they will not be repeated here.
[0092] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0093] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0094] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for training a risk detection model, the method comprising: Candidate samples are determined from a massive sample of various transaction data; wherein the number of candidate samples is less than a first threshold, and the feature values of the candidate samples satisfy a second threshold; Based on the corresponding dialogue scripts for the business scenario and the preset dialogue features, additional features of the candidate samples are determined, and the additional features are used to characterize the credibility of the samples. Based on the candidate samples and their additional characteristics, risk samples are determined; A risk detection model is trained based on the aforementioned risk samples; The dialogue features include at least one of the following: the progress of the dialogue, the mood of the dialogue, and features that deviate from reality; The process of determining the risk sample based on the candidate samples and their additional characteristics includes: Among the candidate samples, the samples whose credibility meets the first preset condition are determined as the first target samples; wherein, the first preset condition is a condition set according to the scenario; Based on the boundary query strategy, the risk samples in the first target sample are determined, and the boundary query strategy is to select samples whose confidence level is within the boundary value range.
2. The method according to claim 1, wherein the numerical range of the second threshold is 0.6-1.
3. The method according to claim 1, wherein determining the risk sample in the first target sample based on the boundary query strategy comprises: Based on the boundary query strategy, a second target sample is selected from the first target sample; When the error between the credibility of the second target sample and the preset standard value meets the second preset condition, the second target sample is determined as the risk sample.
4. The method according to any one of claims 1-3, wherein training the risk detection model based on the risk samples comprises: The risk detection model is trained based on the risk samples and the risk values of each sample in the risk samples. The risk values of the samples are determined by the historical risk detection model based on the risk samples.
5. The method according to any one of claims 1-3, wherein creating a risk detection model based on the risk sample comprises: When the number of risk samples meets the preset value, the risk detection model is trained based on the risk samples and historical samples in the sample library.
6. A training device for a risk detection model, the device comprising: The system comprises a first determination module, a second determination module, a third determination module, and a training module, wherein... The first determining module is configured to: determine candidate samples from a massive sample of various transaction data; wherein the number of candidate samples is less than a first threshold, and the feature values of the candidate samples satisfy a second threshold; The second determining module is configured to: determine additional features of the candidate samples based on the corresponding dialogue scripts for the business scenario and preset dialogue features, wherein the additional features are used to characterize the credibility of the samples; The third determining module is configured to: determine risk samples based on the candidate samples and their additional features; The training module is configured to train a risk detection model based on the risk samples. The dialogue features include at least one of the following: the progress of the dialogue, the mood of the dialogue, and features that deviate from reality; The third determining module is configured as follows: Among the candidate samples, the samples whose credibility meets the first preset condition are determined as the first target samples; wherein, the first preset condition is a condition set according to the scenario. Based on the boundary query strategy, risk samples in the first target sample are identified. The boundary query strategy is to select samples whose confidence level is within the boundary value range.
7. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-5.
8. A computer-readable storage medium having a computer program stored thereon, which, when the computer... When the program is executed in a computer, it causes the computer to perform the method according to any one of claims 1-5.