Method and device for evaluating the effectiveness of list screening system

Abstract

The embodiments of this specification provide a method and device for evaluating the effectiveness of a list screening system. The method includes: inputting at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain the adversarial business information corresponding to the original business information; inputting at least one adversarial business information output by the test case generation model into the list screening system; obtaining the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system. The embodiments of this specification can more accurately evaluate the effectiveness of a list screening system.

Description

Technical Field

[0001] One or more embodiments of this specification relate to network communication technology, and more particularly, to a method and apparatus for evaluating the effectiveness of a list screening system. Background Art

[0002] With the continuous development of network technology, various illegal online activities have emerged. To promptly identify illegal activities, blacklists are often established, adding relevant business information that has been historically identified as illegal, such as user names, account information, or place names, to the blacklist. In subsequent business activities, such as bank remittances, a list screening system is used to determine whether relevant business information in the current business activity appears on the blacklist. If it does, for example, if the account information in the current business activity is on the blacklist, then the current business activity can be determined to be illegal.

[0003] Clearly, a list screening system is crucial for accurately identifying illegal businesses. The stronger the system's screening capabilities, the more accurate its detection rate. Therefore, it's necessary to evaluate the effectiveness of the list screening systems used by each business department. For example, this involves determining whether a business department's list screening system meets business needs or meets predetermined screening accuracy requirements, thereby confirming the reliability of the list screening system. Summary of the Invention

[0004] One or more embodiments of this specification describe a method and apparatus for evaluating the effectiveness of a list screening system, which can more accurately evaluate the effectiveness of a list screening system.

[0005] According to a first aspect, a method for evaluating the effectiveness of a list screening system is provided, wherein the method comprises:

[0006] Inputting at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain adversarial business information corresponding to the original business information;

[0007] Inputting at least one adversarial business information output by the test case generation model into the list screening system, so that the list screening system performs variation identification on the adversarial business information;

[0008] The effectiveness of the list screening system is obtained based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system.

[0009] Wherein, the method further comprises:

[0010] Mutating normal business information based on human expert experience and / or pre-set general rules to generate at least one test case; and

[0011] inputting at least one test case into a list screening system so that the list screening system performs variant identification on the test case;

[0012] The obtaining of the validity of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system includes:

[0013] The effectiveness of the list screening system is obtained based on at least one original business information, at least one adversarial business information output by the test case generation model, the adversarial business information identified by the list screening system, and the test case identified by the list screening system.

[0014] The general rules include: variation rules for different types of business information in different languages.

[0015] The training method of the test case generation model includes:

[0016] Obtain at least one real business information sample from a real business scenario;

[0017] Obtain adversarial business information samples corresponding to each real business information sample from real business scenarios; the adversarial business information samples are obtained by violators mutating the real business information samples;

[0018] The test case generation model is trained using the real business information samples and their corresponding adversarial business information samples.

[0019] The step of training the test case generation model using the real business information sample and its corresponding adversarial business information sample includes:

[0020] Convert a real business information sample and its corresponding adversarial business information sample into feature vectors respectively;

[0021] Calculate the distance between the two transformed eigenvectors;

[0022] Determining an initial value of a distance parameter in a loss function used by the test case generation model based on the calculated distance;

[0023] Each pair of real business information samples and their corresponding adversarial business information samples are input into the test case generation model for training. The training objectives include: the gap between the adversarial business information samples and the real business information samples is as large as possible under the constraint of a preset distance parameter value.

[0024] According to a second aspect, a device for evaluating the effectiveness of a list screening system is provided, wherein the device comprises:

[0025] A test case acquisition module is configured to input at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain the adversarial business information corresponding to the original business information;

[0026] a test case input module configured to input at least one adversarial business information output by the test case generation model into a list screening system, so that the list screening system can perform variation identification on the adversarial business information;

[0027] The evaluation module is configured to obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system.

[0028] The test case acquisition module is further configured to perform: mutating normal business information according to human expert experience and / or pre-set general rules, thereby generating at least one test case;

[0029] The test case input module is further configured to: input at least one test case into a list screening system so that the list screening system performs variation identification on the test case;

[0030] The evaluation module is configured to obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, the adversarial business information identified by the list screening system, and the test case identified by the list screening system.

[0031] The apparatus further includes a model training module;

[0032] The model training module is configured to perform:

[0033] Obtain at least one real business information sample from a real business scenario;

[0034] Obtain adversarial business information samples corresponding to each real business information sample from real business scenarios; the adversarial business information samples are obtained by violators mutating the real business information samples;

[0035] The test case generation model is trained using the real business information samples and their corresponding adversarial business information samples.

[0036] When the model training module trains the test case generation model using the real business information sample and its corresponding adversarial business information sample, the model training module is configured to execute:

[0037] Convert a real business information sample and its corresponding adversarial business information sample into feature vectors respectively;

[0038] Calculate the distance between the two transformed eigenvectors;

[0039] Determining an initial value of a distance parameter in a loss function used by the test case generation model based on the calculated distance;

[0040] Each pair of real business information samples and their corresponding adversarial business information samples are input into the test case generation model for training. The training objectives include: the gap between the adversarial business information samples and the real business information samples is as large as possible under the constraint of a preset distance parameter value.

[0041] According to a third aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any embodiment of this specification is implemented.

[0042] The methods and devices for evaluating the effectiveness of the list screening system provided in various embodiments of this specification, when used alone or in combination, have at least the following beneficial effects:

[0043] 1. First, the pre-trained test case generation model will be used to mutate the original business information, because the test case generation model is trained using real business information samples obtained from real business scenarios and real adversarial business information samples obtained from real business scenarios. Among them, the adversarial business information samples are obtained by mutating the real business information samples by violators rather than managers. In other words, the adversarial business information samples actually exist in the actual business scenario, rather than being defined by rules. Therefore, the test case generation model learns the mutation method from the data in the actual scenario. This ensures that the adversarial business information mutated by the test case generation model during the effectiveness evaluation can better reflect the real mutation situation in the actual scenario. Using the adversarial business information mutated by the test case generation model to test the recognition rate of the list screening system can more accurately reflect the effectiveness of the list screening system.

[0044] 2. We can use not only the adversarial business information generated by the test case generation model as test cases, but also test cases generated based on manual definitions and / or pre-defined general rules to test the recognition rate of the list screening system and evaluate its effectiveness. As can be seen, the rich variety of test case types, that is, the diverse mutation methods, can more accurately reflect the effectiveness of the list screening system.

[0045] 3. The general rules in the embodiments of this specification are variation rules for different types of business information in different languages. The variation methods are more diverse and can better reflect the actual business situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flowchart of a method for evaluating the effectiveness of a list screening system in one embodiment of this specification.

[0048] Figure 2 This is a flowchart of a method for training a test case generation model in one embodiment of this specification.

[0049] Figure 3 It is a flowchart of a method for evaluating the effectiveness of a list screening system in another embodiment of this specification.

[0050] Figure 4It is a schematic structural diagram of an effectiveness evaluation device for a list screening system in an embodiment of this specification. Detailed implementation manners

[0051] In real life, in order to evade the screening of the list screening system and avoid the interception of services, violators usually mutate business information and conduct illegal businesses using the mutated business information. For example, when using Address A as the shipping address for trading illegal goods, in order to evade screening, violators usually mutate Address A. For example, if the address includes: XXX, Guangzhou City, the violator may use the homograph 'factory' to replace 'Guang' and use homophonic words to replace one word in the address, etc. for various mutations. Therefore, the list screening system also needs to be able to accurately identify such mutated business information.

[0052] In the prior art, when evaluating the effectiveness of a list screening system, mutated business information is used to test the list screening system. In the prior art, mutations are usually based on pre-set general rules, such as using 'factory' to replace 'Guang'. However, when mutating according to the pre-set general rules, because the general rules are pre-set artificially based on experience, they are limited by human experience and other reasons and cannot better reflect the mutation methods that will be encountered in actual business. Additionally, in reality, a large number of mutation methods used by violators may be self-created and not universal, and cannot be defined by general rules. Therefore, if only pre-set general rules are used to mutate normal business information and the mutated business information is used to evaluate the list screening system when evaluating the effectiveness of the list screening system, the evaluation results often tend to be inaccurate. For example, because the general rules cannot reflect the mutations in real life, the evaluation result for the list screening system is a screening accuracy rate of 95%. However, once such a list screening system is applied to real life, the screening accuracy rate may be less than 50%. Therefore, the methods in the prior art cannot accurately evaluate the effectiveness of the list screening system.

[0053] The solution provided in this specification will be described below with reference to the accompanying drawings.

[0054] First, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms 'a', 'the', and 'that' used in the embodiments of the present invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0055] Figure 1This is a flow chart of a method for evaluating the effectiveness of a list screening system in one embodiment of this specification. The method is performed by an evaluation device for the list screening system. It is understood that the method can also be performed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 1 , the method comprising:

[0056] Step 101: Input at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein, the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain the adversarial business information corresponding to the original business information.

[0057] Step 103: Input the at least one adversarial business information output by the test case generation model into the list screening system, so that the list screening system can perform variation identification on the adversarial business information.

[0058] Step 105: Obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system.

[0059] According to the above Figure 1 As can be seen from the process shown, in the embodiments of this specification, the original business information is not mutated solely by using pre-set general rules. Instead, a pre-trained test case generation model is first used to mutate the original business information, because the test case generation model is trained using real business information samples obtained from actual business scenarios and real adversarial business information samples obtained from actual business scenarios, wherein the adversarial business information samples are obtained by violators rather than managers after mutating the real business information samples. In other words, the adversarial business information samples actually exist in actual business scenarios, rather than being defined by rules. Therefore, the test case generation model learns the mutation method from data in actual scenarios, which ensures that the adversarial business information mutated by the test case generation model during effectiveness evaluation can better reflect the actual mutation situation in the actual scenario. Using the adversarial business information mutated by the test case generation model to test the recognition rate of the list screening system can more accurately reflect the effectiveness of the list screening system.

[0060] Of course, in order to further improve the accuracy of the effectiveness evaluation of the list screening system, not only the adversarial business information obtained by the test case generation model can be used as a test case, but also test cases generated based on manual definitions and / or test cases generated based on pre-set general rules can be further used to test the recognition rate of the list screening system together, thereby evaluating its effectiveness. That is to say, in one embodiment of the present specification, the test cases may include: test cases generated based on the test case generation model (i.e., adversarial business information), test cases generated based on manual definitions, and test cases generated based on pre-set general rules. It can be seen that the types of test cases are rich, that is, the mutation methods are diverse, and therefore the effectiveness of the list screening system can be more accurately reflected.

[0061] In the embodiment of this specification, the type and number of test cases to be selected may be determined based on the evaluation depth.

[0062] The effectiveness evaluation method of the list screening system can actually specifically include: pre-training the test case generation model, generating test cases based on the test depth requirements, using the list screening system to identify the test cases, and evaluating the effectiveness of the list screening system based on the recognition rate of the list screening system.

[0063] First, as mentioned above, Figure 1 Before the process shown, it is necessary to pre-train the test case generation model using real business information. Figure 2 ,The training methods of the test case generation model include:

[0064] Step 201: Obtain at least one real business information sample from a real business scenario.

[0065] Step 203: Obtain an adversarial business information sample corresponding to each real business information sample from a real business scenario; wherein the adversarial business information sample is obtained by the violator mutating the real business information sample.

[0066] Step 205: Use real business information samples and their corresponding adversarial business information samples to train the test case generation model.

[0067] According to the above Figure 2As shown in the process, when training the test case generation model, the real business information samples and their corresponding adversarial business information samples are all obtained from real business scenarios. In other words, the mutation method from the real business information samples to the adversarial business information samples exists and has appeared in real business scenarios. Therefore, it can better reflect the characteristics of the actual business scenarios. The test case generation model is not based on artificially set rules to learn, but is based on the mutation methods that exist and have appeared in real business scenarios. Therefore, it can learn more real and practical mutation methods. For example, in reality, some mutations may not be defined by general rules. For example, each violator’s mutation habits or mutation methods are different and have no commonalities. In this case, general rules cannot be used. However, in the embodiments of this specification, the test case generation model can learn this mutation method.

[0068] In the above Figure 2 In the process shown, for steps 201 to 203: obtain at least one real business information sample from a real business scenario; obtain an adversarial business information sample corresponding to each real business information sample from the real business scenario; wherein, the adversarial business information sample is obtained by the violator mutating the real business information sample.

[0069] In real-world business scenarios, it's possible to obtain numerous samples of real business information and their corresponding adversarial business information. For example, through manual identification or risk control system recognition, a pair of real business information and adversarial business information, formed by mutating the real business information, can be identified in various actual transaction behaviors. For example, the adversarial business information might replace numbers in the real business information with agreed-upon animal names, or add special symbols to the real business information. Once identified, the real business information can be used as the real business information sample, and the adversarial business information formed by mutating the real business information as the adversarial business information sample.

[0070] Next, for step 205 : use real business information samples and their corresponding adversarial business information samples to train the test case generation model.

[0071] The specific implementation process of this step 205 includes:

[0072] Step 2051: Convert a real business information sample and its corresponding adversarial business information sample into feature vectors respectively.

[0073] Step 2053: Calculate the distance between the two transformed feature vectors.

[0074] Step 2055: Determine the initial value of the distance parameter in the loss function used by the test case generation model based on the calculated distance.

[0075] Step 2057: Each pair of real business information samples and their corresponding adversarial business information samples are input into the test case generation model for training. The training objectives include: the gap between the adversarial business information samples and the real business information samples is as large as possible under the constraint of the preset distance parameter value.

[0076] In the embodiments of this specification, real business information samples and their corresponding adversarial business information samples are closer to the risks of real scenarios. The test case generation model can adopt the SeqGAN model, for example, including: a discriminator D, a generator G, and the scoring result of D as the long-term reward of the generator. The generator is updated by the reinforcement learning policy gradient method, reword is the overall benefit, and the benefit of each word is sampled and averaged by MC (Monte Carlo method). The loss function for the test case generated by the model as an adversarial sample is adjusted to:

[0077] min[distance(x adv , x ori )-α*metric(model(x adv ), model(x ori )]

[0078] The training goal is to maximize the difference between the generated adversarial business information samples and the real business information samples under the test case generation model, subject to a certain distance parameter value (distance) constraint. (For black-box adversarial samples, this is measured by the change in model scores; for white-box adversarial samples, this is measured by the gradient change in the model loss function.) In other words, a good adversarial sample is one that perturbs the original sample slightly but produces a large difference in model discrimination. As shown in the loss function formula, the training objective of the test case generation model includes a distance metric constraint loss and a score comparison loss for the real and adversarial business information samples. a, as a constant, influences the overall convergence of the model training process. An optimal parameter value can be obtained by optimizing the search based on the distance parameter (distance) and the score distribution of the test case generation model.

[0079] At this point, the test case generation model has been pre-trained.

[0080] Next, we need to generate test cases based on the test depth requirements, use the list screening system to identify the test cases, and evaluate the effectiveness of the list screening system based on the recognition rate of the list screening system. Figure 1 The steps shown are explained in Figure 3 , the process includes:

[0081] Step 301: Input at least one original business information into a pre-trained test case generation model, and obtain at least one adversarial business information output by the test case generation model. Among them, the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain the adversarial business information corresponding to the original business information.

[0082] This step 301 corresponds to step 101 above. As mentioned before, since the test case generation model is trained using real business information, the test case generation model does not learn based on artificially set rules, but based on the mutation methods that exist and have occurred in the real business scenario. Therefore, it can learn more realistic and practical mutation methods. For example, in reality, some mutations may not be defined by general rules. The mutation habits or methods of each violator are different and do not have commonalities. In such cases, the test case generation model can learn such mutation methods.

[0083] Therefore, in step 101 / step 301, when it is necessary to evaluate the effectiveness of the list screening system, the test case generation model can mutate the original business information in line with the actual business characteristics. That is to say, the obtained adversarial business information is more authentic and can better reflect the mutation methods of violators in reality.

[0084] It can be understood that for each original business information, the test case generation model will output an adversarial business information corresponding to the original business information, forming a test case pair composed of the original business information and its corresponding adversarial business information. Usually, a large number of original business information will be used in step 101 / step 301 to obtain a large number of test case pairs, and a large number of test case pairs will be used to test the list screening system.

[0085] The above step 301 generates test cases based on the test case generation model.

[0086] Step 303: Generate test cases based on artificial definition.

[0087] Generally, test cases are designed under the guidance of the experience of compliance experts. For example, for the OCR text scanning scenario, design "广" -> "厂" homograph replacement, etc., and use this to define a homograph library to batch replace the names or addresses in the list to generate test cases.

[0088] Generating test cases based on artificial definition is a basic rule transformation.

[0089] Step 305: Generate test cases based on pre-set general rules.

[0090] The general rules set in the prior art have the following disadvantages:

[0091] 1. The rules for deformation generation are relatively basic. For example, replacing any letter i->l makes it difficult to balance semantics and adversarial effectiveness.

[0092] 2. They are all generated by general rules, which are very different from the risk types faced in actual scenarios and are not universal;

[0093] 3. Does not support multiple languages, only supports English deformation generation;

[0094] 4. The supported types are limited, only personal names and company names are supported.

[0095] The embodiments of this specification also pre-set general rules. However, unlike the prior art, the general rules in this specification are specific to different types of business information in different languages. For example, different language variations of different types of business information, such as names of people, companies, addresses, ships, and ports, are defined. These rules can be used interchangeably, as shown in Table 1, which adds more rules for different types and languages.

[0096]

[0097] Table 1

[0098] In the embodiments of this specification, the type and number of test cases to be selected can be determined based on the evaluation depth. That is, if the evaluation depth is not high, only the model-based test case generation in step 301 can be performed, and the number of generated test cases can also be relatively small. If the evaluation depth is further increased, the manual definition-based test case generation in step 303 can be further performed. If the evaluation depth is further increased, the general rule-based test case generation in step 305 can be performed.

[0099] Step 307: Input the at least one adversarial business information output by the test case generation model into the list screening system, so that the list screening system can perform anomaly identification on the adversarial business information.

[0100] The process of step 307 corresponds to the above step 103.

[0101] Step 309: Input the test cases generated in steps 303 and 305 into the list screening system so that the list screening system can perform anomaly identification on the adversarial business information.

[0102] If the above steps 303 and 305 are not performed based on requirements such as the evaluation depth, there is no need to perform this step 309.

[0103] Step 311: Obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model (i.e., a test case generated based on the test case generation model), the test cases generated in steps 303 and 305, and the adversarial business information identified by the list screening system.

[0104] Here, the effectiveness of the list screening system is evaluated based on the recognition accuracy of the mutated test cases identified by the list screening system.

[0105] In one embodiment of the present specification, the effectiveness evaluation index includes at least one of the following:

[0106] Comprehensive risk coverage, which includes: the number of test cases generated by mutations identified by the list screening system / the total number of test cases generated by mutations;

[0107] Risk coverage at different depths, including: the number of true black sample hits in test cases at different deformation depths / the number of true black sample hits in test cases of this type;

[0108] The risk coverage under different mutation rules is used to evaluate the effectiveness of the list screening system in identifying this type of mutation.

[0109] In one embodiment of this specification, a device for evaluating the effectiveness of a list screening system is proposed. Figure 4 , the device comprises:

[0110] The test case acquisition module 401 is configured to input at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain the adversarial business information corresponding to the original business information;

[0111] A test case input module 402 is configured to input at least one adversarial business information output by the test case generation model into a list screening system so that the list screening system can perform variation identification on the adversarial business information;

[0112] The evaluation module 403 is configured to obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system.

[0113] In one embodiment of the apparatus of this specification, the test case acquisition module 401 is further configured to perform: mutating normal business information according to human expert experience and / or pre-set general rules, thereby generating at least one test case;

[0114] The test case input module 402 is further configured to: input at least one test case into the list screening system so that the list screening system performs variation identification on the test case;

[0115] The evaluation module 403 is configured to obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, the adversarial business information identified by the list screening system, and the test case identified by the list screening system.

[0116] In the embodiment of the device of this specification, the general rules include: variation rules for different types of business information in different languages.

[0117] In one embodiment of the apparatus of this specification, the apparatus further comprises a model training module configured to execute:

[0118] Obtain at least one real business information sample from a real business scenario;

[0119] Obtain adversarial business information samples corresponding to each real business information sample from real business scenarios; the adversarial business information samples are obtained by violators mutating the real business information samples;

[0120] The test case generation model is trained using the real business information samples and their corresponding adversarial business information samples.

[0121] In one embodiment of the apparatus of this specification, when the model training module trains the test case generation model using the real business information sample and its corresponding adversarial business information sample, it is configured to execute:

[0122] Convert a real business information sample and its corresponding adversarial business information sample into feature vectors respectively;

[0123] Calculate the distance between the two transformed eigenvectors;

[0124] Determining an initial value of a distance parameter in a loss function used by the test case generation model based on the calculated distance;

[0125] Each pair of real business information samples and their corresponding adversarial business information samples are input into the test case generation model for training. The training objectives include: the gap between the adversarial business information samples and the real business information samples is as large as possible under the constraint of a preset distance parameter value.

[0126] It should be noted that the aforementioned devices are typically implemented on a server side and can be installed on separate servers, or some or all of them can be installed on the same server. The server can be a single server or a server cluster consisting of multiple servers. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the management difficulties and weak service scalability of traditional physical hosts and virtual private servers (VPs). The aforementioned devices can also be implemented on computer terminals with strong computing capabilities.

[0127] One embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method in any one of the embodiments of the present specification.

[0128] An embodiment of the present specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of any embodiment of the present specification is implemented.

[0129] It should be understood that the structures illustrated in the embodiments of this specification do not constitute specific limitations on the devices of the embodiments of this specification. In other embodiments of the specification, the above-mentioned devices may include more or fewer components than shown in the figure, or combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0130] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0131] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented using hardware, software, widgets, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0132] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. Methods for evaluating the effectiveness of list screening systems, including: The method includes: Inputting at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain adversarial business information corresponding to the original business information; Inputting at least one adversarial business information output by the test case generation model into the list screening system, so that the list screening system performs variation identification on the adversarial business information; Obtaining effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system; The training method of the test case generation model includes: Obtain at least one real business information sample from a real business scenario; Obtain adversarial business information samples corresponding to each real business information sample from real business scenarios; the adversarial business information samples are obtained by violators mutating the real business information samples; Using the real business information samples and their corresponding adversarial business information samples to train the test case generation model; The step of training the test case generation model using the real business information sample and its corresponding adversarial business information sample includes: Convert a real business information sample and its corresponding adversarial business information sample into feature vectors respectively; Calculate the distance between the two transformed eigenvectors; Determining an initial value of a distance parameter in a loss function used by the test case generation model based on the calculated distance; Each pair of real business information samples and their corresponding adversarial business information samples are input into the test case generation model for training. The training objectives include: the gap between the adversarial business information samples and the real business information samples is as large as possible under the constraint of a preset distance parameter value.

2. The method according to claim 1, wherein The method further comprises: Mutating normal business information based on human expert experience and / or pre-set general rules to generate at least one test case; and inputting at least one test case into a list screening system so that the list screening system performs variant identification on the test case; The obtaining of the validity of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system includes: The effectiveness of the list screening system is obtained based on at least one original business information, at least one adversarial business information output by the test case generation model, the adversarial business information identified by the list screening system, and the test case identified by the list screening system.

3. The method according to claim 2, wherein: The general rules include: variation rules for different types of business information in different languages.

4. A device for evaluating the effectiveness of a list screening system, wherein: The device includes: A test case acquisition module is configured to input at least one original business information into a pre-trained test case generation model to obtain at least one adversarial business information output by the test case generation model; wherein the test case generation model is trained using real business information, and the test case generation model mutates each original business information to obtain the adversarial business information corresponding to the original business information; a test case input module configured to input at least one adversarial business information output by the test case generation model into a list screening system, so that the list screening system can perform variation identification on the adversarial business information; an evaluation module configured to obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, and the adversarial business information identified by the list screening system; The apparatus further includes a model training module; The model training module is configured to perform: Obtain at least one real business information sample from a real business scenario; Obtain adversarial business information samples corresponding to each real business information sample from real business scenarios; the adversarial business information samples are obtained by violators mutating the real business information samples; Using the real business information samples and their corresponding adversarial business information samples to train the test case generation model; When the model training module trains the test case generation model using the real business information sample and its corresponding adversarial business information sample, the model training module is configured to execute: Convert a real business information sample and its corresponding adversarial business information sample into feature vectors respectively; Calculate the distance between the two transformed eigenvectors; Determining an initial value of a distance parameter in a loss function used by the test case generation model based on the calculated distance; Each pair of real business information samples and their corresponding adversarial business information samples are input into the test case generation model for training. The training objectives include: the gap between the adversarial business information samples and the real business information samples is as large as possible under the constraint of a preset distance parameter value.

5. The device according to claim 4, wherein The test case acquisition module is further configured to perform: mutating normal business information according to human expert experience and / or pre-set general rules, thereby generating at least one test case; The test case input module is further configured to: input at least one test case into a list screening system so that the list screening system performs variation identification on the test case; The evaluation module is configured to obtain the effectiveness of the list screening system based on at least one original business information, at least one adversarial business information output by the test case generation model, the adversarial business information identified by the list screening system, and the test case identified by the list screening system.

6. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 3 is implemented.

Images