Data processing method, device, electronic device and storage medium

Abstract

The present application relates to the field of data verification technology, and in particular to a data processing method, device, electronic device and storage medium. It is used to solve the problem of low screening efficiency and screening accuracy in blacklist-based screening under existing technologies. The method is as follows: when it is determined that the attribute identification information of the event to be processed is included in the blacklist set, the attribute identification information is mapped to the data bits of the bit array of the Bloom filter, and based on the value of the data bit, it is judged whether the rule associated with the attribute identification information of the event to be processed is included in the rule set, wherein the value of each data bit of the bit array is set based on the rule set of the Bloom filter, and the rule set contains at least one rule that matches each attribute identification information; if it is determined that the rule associated with the attribute identification information of the event to be processed is not included in the rule set, the event to be processed is filtered and released; in this way, the screening accuracy and screening efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data verification technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] Under existing technologies, in the financial field, screening of certain information is usually achieved by configuring a blacklist, that is, the attribute identification information of the event to be processed is screened against the blacklist, and then manual judgment is made as to whether the submitted information and the hit list point to the same item to determine whether it is a true hit.

[0003] However, since the same item usually contains multiple description forms, the blacklist cannot be configured with all the information, that is, the blacklist has a small fault tolerance space, which leads to a large number of missed detections and reduces the screening accuracy; correspondingly, due to the small fault tolerance space of the blacklist, it also causes a high false alarm rate, and the high false alarm rate is bound to cause the existing screening system to frequently issue alarms, which further leads to the jamming of the screening system process, thereby affecting the screening efficiency.

[0004] In summary, a new method needs to be designed to solve the above problems. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, device, electronic device and storage medium to solve the problems of low screening efficiency and screening accuracy in blacklist-based screening under existing technologies.

[0006] The specific technical solutions provided in the embodiments of this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a data processing method, comprising:

[0008] When it is determined that the attribute identification information of the event to be processed is included in the blacklist set, mapping the attribute identification information to data bits of a bit array of a Bloom filter, wherein the value of each data bit of the bit array is set based on a rule set of the Bloom filter, the rule set including at least one rule matching each attribute identification information;

[0009] determining, based on the value of the data bit, whether the rule associated with the attribute identification information of the event to be processed is included in the rule set;

[0010] If it is determined that the rule associated with the attribute identification information of the event to be processed is not included in the rule set, the event to be processed is filtered and released.

[0011] The above method maps the attribute identification information of the event to be processed to the data bit of the bit array of the corresponding Bloom filter, and then compares the value of the data bit with the various values ​​contained in the preset rule set of the Bloom filter to determine whether the attribute identification information of the event to be processed is included in the rule set. Therefore, based on the mathematical principle that the probability of conflict between two completely random numbers of the Bloom filter is extremely small, with the help of simple and easy mathematical tools, it is possible to accurately determine whether the attribute identification information of the event to be processed is in the rule set of the corresponding Bloom filter, thereby improving the screening accuracy and efficiency.

[0012] In some embodiments, whether the attribute identification information of the event to be processed is included in the blacklist set is determined by performing the following operations:

[0013] Acquire the event to be processed, and preprocess the event to be processed to obtain an attribute identification information set of the event to be processed, wherein the preprocessing includes any one or any combination of word segmentation processing, filtering processing, and character mapping processing, and the attribute identification information set includes attribute identification information of different dimensions of the event to be processed;

[0014] Calling a preset text search engine to perform a text search on each attribute identification information included in the attribute identification information set of the event to be processed based on the blacklist set to obtain search results;

[0015] If the attribute identification information representing the search result matches any grouping category included in the blacklist set, it is determined that the attribute identification information of the event to be processed is included in the blacklist set, wherein the attribute identification information is any attribute identification information in the attribute identification information set.

[0016] The above method, based on the constructed blacklist set, performs text search on the events to be processed through a preset text search engine. First, it filters out some events whose attribute identification information sets of the events to be processed are not included in the blacklist set, that is, it filters out some events to be processed that do not have certain characteristic rules, thereby reducing the screening load of the screening system. At the same time, it also paves the way for the subsequent screening by the Bloom filter subsystem. Through text search, the rule identification information corresponding to the events to be processed is determined, thereby reducing the false alarm rate of the screening system to a greater extent, so as to improve the screening accuracy.

[0017] In some embodiments, the text search algorithm configured by the text search engine includes any one or any combination of a fuzzy query rule set, a vector space model, a word frequency inverse text frequency index algorithm, a word frequency saturation algorithm, a length normalization algorithm, and a relevance weight algorithm.

[0018] The above method uses multiple text search algorithms to make the search results of text search more accurate, thereby improving the screening accuracy and screening efficiency.

[0019] In some embodiments, after determining that the attribute identification information of the event to be processed is included in the blacklist set and before mapping the attribute identification information to the data bits of the bit array of the Bloom filter, the method further includes:

[0020] Acquire other attribute identification information associated with the attribute identification information from the attribute identification information set of the event to be processed;

[0021] Determining a service type of the event to be processed based on the other attribute identification information, and determining a corresponding hash function set based on the service type, wherein the number of hash functions included in the hash function set is predetermined based on the service type of the message;

[0022] Mapping the attribute identification information to data bits of a bit array of a Bloom filter includes:

[0023] Based on each hash function included in the hash function set and a preset mapping rule, the attribute identification information and the other attribute identification information are respectively mapped to data bits of a bit array of a corresponding Bloom filter.

[0024] The above method configures the corresponding Bloom filter based on the business type, wherein the business type represents the minimum granularity business type corresponding to the event to be processed. Then, based on different business types, a corresponding set of hash functions is obtained, and a preset mapping rule is used to map the attribute identification information and other attribute identification information of the event to be processed to the corresponding Bloom filter. In this way, the mathematical principle of the Bloom filter that the probability of collision between two completely random numbers is extremely small is adopted to minimize the false alarm rate of the Bloom filter and improve the accuracy and efficiency of screening.

[0025] In some embodiments, after determining that the attribute identification information of the event to be processed is included in the blacklist set and before mapping the attribute identification information to the data bits of the bit array of the Bloom filter, the method further includes:

[0026] Based on the rule identification information associated with the attribute identification information, obtaining the rule content of the rule identification information, and judging whether the rule identification information includes a license exemption exception rule based on the rule content;

[0027] Mapping the attribute identification information to data bits of a bit array of a Bloom filter includes:

[0028] If it is determined that the rule identification information does not include a license exemption exception rule, mapping the attribute identification information to a data bit of the bit array that does not include a Bloom filter corresponding to the license exemption exception rule;

[0029] or,

[0030] If it is determined that the rule identification information includes a license exemption exception rule, the attribute identification information is mapped to a data bit of a bit array of a Bloom filter corresponding to the license exemption exception rule.

[0031] In the above method, since different rule identification information corresponds to different rule contents, customizing corresponding Bloom filters for rules that include license exemption exception rules and / or do not include license exemption exception rules can greatly improve the screening accuracy and reduce the screening load caused by false alarms to subsequent further screening.

[0032] In some embodiments, the rule identification information associated with the attribute identification information is determined by performing the following operations:

[0033] Based on the search results, rule identification information of the matched grouping category is acquired, and the rule identification information of the grouping category is used as rule identification information associated with the attribute identification information.

[0034] The above method configures corresponding rule identification information for each grouping category, and can perform text search on the attribute identification information set of the event to be processed. When it is determined that the attribute identification information of the event to be processed is included in the blacklist set, the approximate rules that match the event to be processed are determined, thereby paving the way for the subsequent Bloom filter screening process.

[0035] In some embodiments, the determining, based on the value of the data bit, whether the rule associated with the attribute identification information of the event to be processed is included in the rule set includes:

[0036] Comparing each value of the data bit of the attribute identification information with each value included in a preset rule set of a Bloom filter corresponding to the attribute identification information to obtain a first comparison result, and determining whether the rule associated with the attribute identification information of the event to be processed is included in the corresponding rule set based on the first comparison result;

[0037] In addition, the various numerical values ​​of the data bits of the other attribute identification information are compared with the various numerical values ​​contained in the preset data set of the Bloom filter corresponding to the other attribute identification information to obtain a second comparison result, and based on the second comparison result, it is determined whether the rules associated with the other attribute identification information of the event to be processed are included in the corresponding rule set.

[0038] The above method is based on the low false alarm rate characteristic of the Bloom filter and uses simple and easy mathematical tools to achieve the purpose of reducing the false alarm rate, thereby improving the screening accuracy and screening efficiency.

[0039] In some embodiments, the method further comprises:

[0040] If it is determined that the attribute identification information of the event to be processed and the rules associated with the other attribute identification information are all included in the corresponding rule set, the event to be processed will be transferred to the next data processing device for data-oriented processing, or the event to be processed will be pushed to the display device so that relevant personnel can manually review the event to be processed through the data processing device.

[0041] The above method pushes the event to be processed to the next link when it is determined that the rule associated with the event to be processed is included in the rule set, thereby reducing the screening load of the next link and improving the screening efficiency.

[0042] In some embodiments, after pushing the pending event to the display device, the method further includes:

[0043] When it is determined that the business type of the event to be processed meets the optimization condition, the following operations are performed to obtain an updated rule set of the Bloom filter corresponding to the business type of the event to be processed:

[0044] Obtaining optimization configuration information corresponding to the optimization condition, and determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information;

[0045] According to a preset splicing rule, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed to obtain a new bit array, and based on the new bit array, the value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated;

[0046] and / or,

[0047] According to the preset splicing rules, the target feature value array and the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed are spliced ​​to obtain a new bit array, and based on the new bit array, the numerical value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated.

[0048] The above method configures corresponding optimization conditions. When it is determined that the screening system meets the optimization conditions, the screening features are increased by means of a more targeted feature value array, thereby further improving the screening accuracy of the screening system, thereby improving the screening efficiency, and further reducing the screening load of the next link.

[0049] In some embodiments, the optimized conditions include some or all of the following:

[0050] Receiving a manually triggered optimization instruction, wherein the optimization instruction includes the business type of the event to be processed, and the optimization instruction is issued when it is determined that the misjudgment rate of processed events corresponding to the business type of the event to be processed is greater than a misjudgment rate threshold;

[0051] The rule for determining the association between the attribute identification information of the event to be processed and any one of the other attribute identification information is included in the rule set of the corresponding Bloom filter.

[0052] The above method is an optimized way to configure and update the rule set of the Bloom filter for the screening system. By adding screening features, the optimized operation of the screening system is achieved, thereby using simple and easy mathematical tools to reduce the screening false alarm rate, thereby improving the screening accuracy and efficiency.

[0053] In some embodiments, determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information includes:

[0054] Acquire a candidate event set, wherein the candidate event set includes multiple processed events, and attribute identification information of each processed event and rules associated with other attribute identification information are included in a corresponding rule set;

[0055] For each processed event in the candidate event set, perform the following operations:

[0056] Obtaining a preset first number of attribute identification information from the attribute identification information set of the processed event, and obtaining a matching degree between each attribute identification information and a rule corresponding to the processed event according to a preset second number of dimension information included in the optimization configuration information, and determining a sub-feature value corresponding to each obtained attribute identification information based on each obtained matching degree, wherein each dimension information represents feature information of a dimension corresponding to the processed event;

[0057] Based on the sub-eigenvalue corresponding to each attribute identification information, the eigenvalue corresponding to each dimension information is determined, and the determined eigenvalues ​​are sorted according to a preset sorting rule to obtain a sorted candidate eigenvalue array.

[0058] Deduplication processing is performed on each obtained candidate eigenvalue array, and each eigenvalue contained in each candidate eigenvalue array after deduplication processing is calculated according to a preset calculation method to obtain a target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed.

[0059] The above method updates the rule set of the Bloom filter with the assistance of the information fingerprint subsystem. Since the multi-dimensional feature values ​​determined by the information fingerprint subsystem are common features extracted for the business type, the target feature value array is spliced ​​with the pre-built bit array of the Bloom filter to reconstruct the bit value of the Bloom filter, thereby optimizing the screening system and further improving the screening accuracy of the screening system.

[0060] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0061] a determination module, configured to, when determining that attribute identification information of the event to be processed is included in the blacklist set, map the attribute identification information to data bits of a bit array of a Bloom filter, wherein a value of each data bit of the bit array is set based on a rule set of the Bloom filter, the rule set including at least one rule matching each attribute identification information;

[0062] a judgment module, configured to judge, based on the value of the data bit, whether the rule associated with the attribute identification information of the event to be processed is included in the rule set;

[0063] The push module is configured to filter and release the event to be processed if it is determined that the rule associated with the attribute identification information of the event to be processed is not included in the rule set.

[0064] In some embodiments, whether the attribute identification information of the event to be processed is included in the blacklist set is determined by performing the following operations:

[0065] Acquire the event to be processed, and preprocess the event to be processed to obtain an attribute identification information set of the event to be processed, wherein the preprocessing includes any one or any combination of word segmentation processing, filtering processing, and character mapping processing, and the attribute identification information set includes attribute identification information of different dimensions of the event to be processed;

[0066] Calling a preset text search engine to perform a text search on each attribute identification information included in the attribute identification information set of the event to be processed based on the blacklist set to obtain search results;

[0067] If the attribute identification information representing the search result matches any grouping category included in the blacklist set, it is determined that the attribute identification information of the event to be processed is included in the blacklist set, wherein the attribute identification information is any attribute identification information in the attribute identification information set.

[0068] In some embodiments, the text search algorithm configured by the text search engine includes any one or any combination of a fuzzy query rule set, a vector space model, a word frequency inverse text frequency index algorithm, a word frequency saturation algorithm, a length normalization algorithm, and a relevance weight algorithm.

[0069] In some embodiments, after determining that the attribute identification information of the event to be processed is included in the blacklist set and before mapping the attribute identification information to the data bits of the bit array of the Bloom filter, the determining module is further configured to:

[0070] Acquire other attribute identification information associated with the attribute identification information from the attribute identification information set of the event to be processed;

[0071] Determining a service type of the event to be processed based on the other attribute identification information, and determining a corresponding hash function set based on the service type, wherein the number of hash functions included in the hash function set is predetermined based on the service type of the message;

[0072] Mapping the attribute identification information to data bits of a bit array of a Bloom filter includes:

[0073] Based on each hash function included in the hash function set and a preset mapping rule, the attribute identification information and the other attribute identification information are respectively mapped to data bits of a bit array of a corresponding Bloom filter.

[0074] In some embodiments, after determining that the attribute identification information of the event to be processed is included in the blacklist set and before mapping the attribute identification information to the data bits of the bit array of the Bloom filter, the determining module is further configured to:

[0075] Based on the rule identification information associated with the attribute identification information, obtaining the rule content of the rule identification information, and judging whether the rule identification information includes a license exemption exception rule based on the rule content;

[0076] Mapping the attribute identification information to data bits of a bit array of a Bloom filter includes:

[0077] If it is determined that the rule identification information does not include a license exemption exception rule, mapping the attribute identification information to a data bit of the bit array that does not include a Bloom filter corresponding to the license exemption exception rule;

[0078] or,

[0079] If it is determined that the rule identification information includes a license exemption exception rule, the attribute identification information is mapped to a data bit of a bit array of a Bloom filter corresponding to the license exemption exception rule.

[0080] In some embodiments, the rule identification information associated with the attribute identification information is determined by performing the following operations:

[0081] Based on the search results, rule identification information of the matched grouping category is acquired, and the rule identification information of the grouping category is used as rule identification information associated with the attribute identification information.

[0082] In some embodiments, the determining module determines whether the rule associated with the attribute identification information of the event to be processed is included in the rule set based on the value of the data bit, and is configured to:

[0083] Comparing each value of the data bit of the attribute identification information with each value included in a preset rule set of a Bloom filter corresponding to the attribute identification information to obtain a first comparison result, and determining whether the rule associated with the attribute identification information of the event to be processed is included in the corresponding rule set based on the first comparison result;

[0084] In addition, the various numerical values ​​of the data bits of the other attribute identification information are compared with the various numerical values ​​contained in the preset data set of the Bloom filter corresponding to the other attribute identification information to obtain a second comparison result, and based on the second comparison result, it is determined whether the rules associated with the other attribute identification information of the event to be processed are included in the corresponding rule set.

[0085] In some embodiments, the push module is further configured to:

[0086] If it is determined that the attribute identification information of the event to be processed and the rules associated with the other attribute identification information are all included in the corresponding rule set, the event to be processed will be transferred to the next data processing device for data-oriented processing, or the event to be processed will be pushed to the display device so that relevant personnel can manually review the event to be processed through the data processing device.

[0087] In some embodiments, after pushing the pending event to the display device, the pushing module is further configured to:

[0088] When it is determined that the business type of the event to be processed meets the optimization condition, the following operations are performed to obtain an updated rule set of the Bloom filter corresponding to the business type of the event to be processed:

[0089] Obtaining optimization configuration information corresponding to the optimization condition, and determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information;

[0090] According to a preset splicing rule, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed to obtain a new bit array, and based on the new bit array, the value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated;

[0091] and / or,

[0092] According to the preset splicing rules, the target feature value array and the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed are spliced ​​to obtain a new bit array, and based on the new bit array, the numerical value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated.

[0093] In some embodiments, the optimized conditions include some or all of the following:

[0094] Receiving a manually triggered optimization instruction, wherein the optimization instruction includes the business type of the event to be processed, and the optimization instruction is issued when it is determined that the misjudgment rate of processed events corresponding to the business type of the event to be processed is greater than a misjudgment rate threshold;

[0095] The rule for determining the association between the attribute identification information of the event to be processed and any one of the other attribute identification information is included in the rule set of the corresponding Bloom filter.

[0096] In some embodiments, based on the optimization configuration information, a target feature value array of a Bloom filter corresponding to the business type of the event to be processed is determined, and the push module is configured to:

[0097] Acquire a candidate event set, wherein the candidate event set includes multiple processed events, and attribute identification information of each processed event and rules associated with other attribute identification information are included in a corresponding rule set;

[0098] For each processed event in the candidate event set, perform the following operations:

[0099] Obtaining a preset first number of attribute identification information from the attribute identification information set of the processed event, and obtaining a matching degree between each attribute identification information and a rule corresponding to the processed event according to a preset second number of dimension information included in the optimization configuration information, and determining a sub-feature value corresponding to each obtained attribute identification information based on each obtained matching degree, wherein each dimension information represents feature information of a dimension corresponding to the processed event;

[0100] Based on the sub-eigenvalue corresponding to each attribute identification information, the eigenvalue corresponding to each dimension information is determined, and the determined eigenvalues ​​are sorted according to a preset sorting rule to obtain a sorted candidate eigenvalue array.

[0101] Deduplication processing is performed on each obtained candidate eigenvalue array, and each eigenvalue contained in each candidate eigenvalue array after deduplication processing is calculated according to a preset calculation method to obtain a target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed.

[0102] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0103] a memory for storing program instructions;

[0104] The processor is configured to call the program instructions stored in the memory and execute the method as described in any one of the first aspects above according to the obtained program instructions.

[0105] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method as described in any one of the above-mentioned first aspects.

[0106] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute the method as described in any one of the above-mentioned first aspects.

[0107] In addition, the technical effects brought about by any implementation method in the second to fifth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 This is a schematic diagram of the system architecture of a screening system in an embodiment of the present application;

[0109] Figure 2 This is a partial flow chart of a data processing method in an embodiment of the present application;

[0110] Figure 3 This is a partial flow chart of a data processing method in an embodiment of the present application;

[0111] Figure 4 This is a schematic diagram of obtaining rule identification information of an event to be processed in an embodiment of the present application;

[0112] Figure 5 This is a schematic diagram of the principle of a Bloom filter in an embodiment of the present application;

[0113] Figure 6 A flowchart of determining whether a license exemption exception rule is included in an embodiment of the present application;

[0114] Figure 7A 、 Figure 7B 、 Figure 7C A schematic diagram of an application scenario in an embodiment of the present application;

[0115] Figure 8 This is a flow chart of an optimization method in an embodiment of the present application;

[0116] Figure 9 This is a schematic diagram of a process for obtaining a target eigenvalue array in an embodiment of the present application;

[0117] Figure 10 This is a flow chart of a data processing method in an embodiment of the present application;

[0118] Figure 11 This is a schematic diagram of the logical architecture of a data processing device according to an embodiment of the present application;

[0119] Figure 12 This is a schematic diagram of the physical structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0120] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0121] For ease of understanding, the terms involved in the embodiments of this application are explained below:

[0122] Any number of elements in the drawings is for illustration and not limitation, and any naming is for distinction only and does not have any limiting meaning.

[0123] It should be noted that the terms "first," "second," "third," etc. in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0124] In order to solve the problems of low screening efficiency and accuracy in blacklist-based screening under the existing technology, in an embodiment of the present application, when it is determined that the attribute identification information of the event to be processed is included in the blacklist set, the attribute identification information is mapped to the data bit of the bit array of the Bloom filter, and based on the value of the data bit, it is judged whether the rule associated with the attribute identification information of the event to be processed is included in the rule set, wherein the value of each data bit of the bit array is set based on the rule set of the Bloom filter, and the rule set contains at least one rule that matches each attribute identification information; if it is determined that the rule associated with the attribute identification information of the event to be processed is included in the rule set, If it is not included in the rule set, the event to be processed is filtered and released; in this way, the attribute identification information of the event to be processed is mapped to the data bit of the bit array of the corresponding Bloom filter, and then the value of the data bit is compared with the various values ​​contained in the preset rule set of the Bloom filter to determine whether the attribute identification information of the event to be processed is included in the rule set. Therefore, based on the mathematical principle that the probability of conflict between two completely random numbers of the Bloom filter is extremely small, it is possible to accurately determine whether the attribute identification information of the event to be processed is in the rule set of the corresponding Bloom filter, thereby improving the screening accuracy and efficiency.

[0125] The preferred implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments can be combined with each other if there is no conflict.

[0126] Figure 1 A schematic diagram of the system architecture of a screening system is shown. Figure 1 As shown, the screening system 100 includes a text search engine 110, a Bloom filter subsystem 120, and an information fingerprint subsystem 130;

[0127] The text search engine 110 is used to perform a text search on the event to be processed, determine whether the attribute identification information of the event to be processed is included in the blacklist set, and push the event to be processed to the Bloom filter subsystem 120 when it is determined that the attribute identification information of the event to be processed is included in the blacklist set;

[0128] The Bloom filter subsystem 120 is used to further screen the event to be processed to determine whether the attribute identification information of the event to be processed is included in the rule set of the corresponding Bloom filter;

[0129] The information fingerprint subsystem 130 is used to assist in optimizing the Bloom filter subsystem 120 .

[0130] In an embodiment of the present application, the Bloom filter subsystem 120 includes a first subsystem and a second subsystem, wherein the first subsystem is used to screen whether the attribute identification information of the event to be processed is included in the rule set of the Bloom filter, which rule set corresponds to rules for a specific industry; the second subsystem is used to screen whether there is an exemption license rule associated with the attribute identification information of the event to be processed, and in which exemption license rule the attribute identification information of the event to be processed is included.

[0131] It should be noted that in the above-mentioned screening system 100, the purpose of the text search engine 110 performing text search on the event to be processed is to determine whether the event to be processed matches a rule with certain characteristics. When it is determined that the event to be processed matches a rule with certain characteristics, the rule identification information of the rule with certain characteristics can be obtained. Then, based on the rule identification information, the further rule content in the Bloom filter subsystem 120 is determined, and by mapping the attribute identification information of the event to be processed to the corresponding Bloom filter (i.e., the first subsystem and the second subsystem), the event to be processed is further screened to see whether it matches a rule in the rule set of the corresponding Bloom filter, thereby determining the final screening result.

[0132] It should be noted that the information fingerprint subsystem 130 in the screening system 100 is used to assist the Bloom filter subsystem 120 in performing optimization operations when the Bloom filter subsystem 120 determines that the event to be processed is included in the rule set of the Bloom filter.

[0133] In the embodiment of the present application, after introducing the above-mentioned screening system, before carrying out the data processing process, it is necessary to first collect the blacklist set.

[0134] In an embodiment of the present application, the specific process of constructing a blacklist set is as follows: based on each preset acquisition path, a subset corresponding to each acquisition path is obtained, wherein each subset contains lists and / or blacklists issued by different institutions or organizations for different fields, as well as rules for associating lists and / or blacklists; then, descriptive information of each item contained in each subset is obtained, and each obtained descriptive information is associated with the corresponding item.

[0135] In the embodiment of the present application, based on each subset, the associated association information is matched, and the matched association information is combined into a word material set, wherein the association information includes but is not limited to the following information:

[0136] 1. Country code, which includes but is not limited to the full name in the native language, abbreviation, name in the mainstream language, ISO country code, etc.

[0137] 2. Synonyms, including but not limited to relevant professional terms for each item, and different phonetic spellings compatible with specific nouns, such as Postal Pinyin, Wade-Giles romanization, Cantonese Pinyin, etc.

[0138] 3. Stop words, including but not limited to key words, abbreviations of specific names, etc. for each of the above items in the specified language system.

[0139] 4. Noise words, including but not limited to "Co., Ltd.", "Limited Liability Company", "Joint Stock Limited Company", etc.

[0140] It should be noted that, in the embodiment of the present application, when the above-mentioned related information is collected, it needs to be segmented, filtered and subjected to natural language processing, etc., to ensure accurate matching during text search.

[0141] Finally, the above-mentioned sub-lists, the description information of each item contained in each sub-list, and the above-mentioned vocabulary collection are grouped according to country, and corresponding rule identification information is configured for each group, and the groups configured with rule identification information are formed into a blacklist collection.

[0142] For example, take the "XX Catalog" whose sub-collection is published by XX announcement as an example.

[0143] Assume that the Chinese definition of the chemical sodium cyanide in the "XX Catalogue" published in XX Announcement is as shown in Table 1:

[0144] Serial number Product Name describe XX product number unit 52 … … … … 53 Sodium cyanide Category 3: Chemicals that can be used to produce XX 2837111000 kilogram 54 … … … …

[0145] In Country M's "YY List", sodium cyanide is assigned a specific Export Control Classification Number (ECCN) code, such as Sodium cyanide - 1C350.d.44.

[0146] Then, when constructing the above blacklist set, first, according to the preset acquisition path, obtain the names of each product in the above two sub-sets (i.e., the "XX Catalog" and the CCL of the US EAR), and then obtain the description information of each item contained in the above two sub-sets.

[0147] That is, the collection of descriptive information of sodium cyanide includes but is not limited to collecting the scientific properties of sodium cyanide (such as the chemical CAS registration number and other scientific characteristic properties), the special definition properties of sodium cyanide (such as the Chinese definition 2837111000, ECCN code), the control category code and the license category code, etc.

[0148] Then, the obtained description information is associated with sodium cyanide.

[0149] In an embodiment of the present application, after collecting the descriptive information of sodium cyanide, a vocabulary set matching the "XX Catalog" is set. Taking sodium cyanide as an example, the vocabulary set includes the country code of the country to which the "XX Catalog" belongs, synonyms and stop words of sodium cyanide associated with the item sodium cyanide, as well as noise words associated with sodium cyanide, such as "Limited Liability Company" in the company entity name.

[0150] Then, the above subsets and other acquired subsets and vocabulary sets are grouped according to country, and corresponding rule identification information is configured for each group, and each group configured with rule identification information forms a blacklist set.

[0151] In the embodiment of the present application, the data files collected for the same item are in XML data format and stored using Schema Group elements.

[0152] See Figure 2 As shown, in an embodiment of the present application, the specific process of the text search engine part in a data processing method provided is as follows:

[0153] S100. Obtain an event to be processed, and preprocess the event to be processed to obtain an attribute identification information set of the event to be processed, wherein the preprocessing includes any one or any combination of word segmentation processing, filtering processing, and character mapping processing, and the attribute identification information set includes attribute identification information of different dimensions of the event to be processed.

[0154] In the embodiment of the present application, after the blacklist set is collected, a preliminary screening of the events to be processed can be performed based on the blacklist set.

[0155] In a specific implementation, when S100 is executed, an event to be processed is obtained, and any one or any combination of pre-processing processes of analysis processing, filtering processing and character mapping processing is performed on the event to be processed to obtain an attribute identification information set of the event to be processed.

[0156] In the embodiment of the present application, it is preferred to perform the above-mentioned analysis processing, filtering processing and character mapping processing on the event to be processed, so as to obtain more accurate and precise attribute identification information, thereby reducing the misjudgment rate when executing S101.

[0157] S101 : Calling a preset text search engine, and performing a text search on each attribute identification information included in the attribute identification information set of the event to be processed based on a blacklist set, to obtain search results.

[0158] In the embodiment of the present application, a preset text search engine is called, and a text search is performed on the event to be processed using a text search algorithm configured by the text search engine, thereby obtaining corresponding search results. The text search algorithm configured by the text search engine includes any one or any combination of a fuzzy query rule set, a vector space model, a word frequency inverse text frequency index algorithm, a word frequency saturation algorithm, a length normalization algorithm, and a relevance weighting algorithm.

[0159] S102 . Based on the search results, determine whether the attribute identification information included in the attribute identification information set of the event to be processed is included in the blacklist set. If so, execute S103 ; otherwise, execute S104 .

[0160] S103. If the attribute identification information represented by the search result matches any grouping category contained in the blacklist set, it is determined that the attribute identification information of the event to be processed is included in the blacklist set, and the event to be processed is pushed to the Bloom filter subsystem; wherein the attribute identification information is any attribute identification information in the attribute identification information set.

[0161] In an embodiment of the present application, if the search result obtained by executing S102 represents any attribute identification information in the attribute identification information set of the event to be processed, and matches any grouping category contained in the blacklist set, then it is determined that the attribute identification information of the event to be processed is included in the blacklist set, that is, it represents that the event to be processed matches a rule with certain characteristics, then, the event to be processed is pushed to the Bloom filter subsystem for further screening.

[0162] S104. If the attribute identification information represented by the search result does not match any grouping category included in the blacklist set, it is determined that the attribute identification information of the event to be processed is not included in the blacklist set, and the event to be processed is filtered and released, wherein the attribute identification information is any attribute identification information in the attribute identification information set.

[0163] In an embodiment of the present application, if the search result obtained by executing S102 indicates that none of the attribute identification information in the attribute identification information set of the event to be processed matches any grouping category contained in the blacklist set, it is determined that the attribute identification information of the event to be processed is not included in the blacklist set, that is, it indicates that the event to be processed does not match a rule with certain characteristics, then the event to be processed is filtered and released.

[0164] See Figure 3 As shown, in an embodiment of the present application, the specific process of the screening method of the Bloom filter subsystem in a data processing method provided is as follows:

[0165] S105 . When it is determined that the attribute identification information of the event to be processed is included in the blacklist set, based on the search results, obtain the rule identification information of the matched grouping category, and use the rule identification information of the grouping category as the rule identification information associated with the attribute identification information.

[0166] In an embodiment of the present application, when executing S103, it has been determined that any attribute identification information contained in the attribute identification information set of the event to be processed is contained in the blacklist set. Then, when executing S105, first, based on the search results, the rule identification information of the grouping category contained in the matched blacklist set is obtained, and then the rule identification information of the grouping category is used as the rule identification information associated with the attribute identification information of the event to be processed.

[0167] For example, see Figure 4 As shown, the pending event is the export of CC by Company A in Country X and Company B in Country Y, the export volume of CC is DD kilograms, and the business number of the pending event is 20202627893 as an example.

[0168] Assume that the attribute identification information set of the event to be processed is as follows:

[0169] Business number 20202627893: {XY, Company A, Company B, Export, CC, Export volume DD kg}.

[0170] The attribute identification information set of the event to be processed obtained by the screening system is business number 20202627893: {XY, Company A, Company B, export, CC, export volume DD kilograms}.

[0171] Then, a text search engine is called to perform a text search on the above-mentioned event to be processed, and a search result of the above-mentioned event to be processed is obtained.

[0172] Assume that the attribute identification information {XY, CC} of the attribute identification information set representing the event to be processed in the search result matches {ZZ transaction rules between country X and country Y}, and the rule identification information of the grouping category of {ZZ transaction rules between country X and country Y} is 07167x.

[0173] The rule identification information of the grouping category is used as the rule identification information associated with the attribute identification information of the event to be processed, that is, the rule identification information associated with the attribute identification information of the event to be processed is 07167x.

[0174] S106: Acquire other attribute identification information associated with the attribute identification information from the attribute identification information set of the event to be processed.

[0175] In an embodiment of the present application, a text search is performed on the attribute identification information set of the event to be processed to determine the blacklist set contained in the attribute identification information of the event to be processed, thereby determining the rule identification information associated with the attribute identification information of the event to be processed, that is, the rule identification information associated with the event to be processed.

[0176] Then, when S106 is executed, other attribute identification information associated with the attribute identification information is obtained from the attribute identification information set of the event to be processed.

[0177] S107. Determine the service type of the event to be processed based on other attribute identification information, and determine a corresponding hash function set based on the service type, wherein the number of hash functions included in the hash function set is predetermined based on the service type of the message.

[0178] In the embodiment of the present application, since the subsequent screening process of the Bloom filter is configured based on each business type, after determining the rule identification information associated with the attribute identification information of the event to be processed, it is necessary to first determine the business type of the event to be processed.

[0179] However, since the text search engine is only used to search whether there are rules with certain characteristics associated with the event to be processed, and which rule the event to be processed is specifically associated with, it is necessary to determine the business type of the event to be processed through other attribute identification information, so as to determine the specific rule associated with the event to be processed. Then, based on the business type, the hash function set of the Bloom filter is determined, so that the attribute identification information and other attribute identification information of the event to be processed can be further screened based on the determined hash function of the Bloom filter, thereby reducing the misjudgment rate of simple text search and improving the system screening accuracy.

[0180] S108 . Map the attribute identification information and other attribute identification information to data bits of the bit array of the corresponding Bloom filter based on the hash functions included in the hash function set and the preset mapping rule.

[0181] In an embodiment of the present application, multiple Bloom filters are pre-initialized, and each Bloom filter is a random data structure with high space efficiency. The basic idea is that when an attribute identification information (or other attribute identification information) is added to a rule set, the attribute identification information (or other attribute identification information) is mapped into multiple data bits in a bit array through multiple hash functions (hash functions), and then the multiple data bits are set to 1. Later, during screening, it is only necessary to detect the attribute identification information (or other attribute identification information) of the event to be processed and map it based on the multiple hash functions (hash functions). Whether the values ​​of the multiple data bits are all 1 can determine whether the attribute identification information (or other attribute identification information) is included in the rule set. If one or more of the values ​​of the multiple data bits is 0, it means that the attribute identification information (or other attribute identification information) is definitely not included in the rule set; if the values ​​of the multiple data bits are all 1, it means that the attribute identification information (or other attribute identification information) is likely to be included in the rule set.

[0182] See Figure 5 The following is a schematic diagram of the principle of Bloom filter. Figure 5 In the example, the rule set S consists of three elements: a, b, and c. The bit array below is a sequence of data bits, all initialized to 0. Each element is mapped to three data bits in the bit array using three hash functions, and the values ​​of these bits are set to 1. To determine whether element a is in the rule set S, simply check whether the values ​​of the three data bits mapped to element a are all 1.

[0183] In the embodiment of the present application, attribute identification information is used as an element in the rule set. When the Bloom filter is initialized, the values ​​of all data bits in the bit array are set to 0, and the attribute identification information of each rule included in the screening system is written into the rule set of the Bloom filter. This attribute identification information is mapped to each data bit in the bit array of the Bloom filter, and the value of each data bit is set to 1. In other words, the value of each data bit in the bit array is set according to the rule set of the Bloom filter, and the rule set contains every existing rule.

[0184] Therefore, in an embodiment of the present application, when executing S108, based on the various hash functions included in the hash function set, the attribute identification information and other attribute identification information are mapped to the data bits of the bit array of the corresponding Bloom filter in accordance with the preset mapping rules, thereby completing the Bloom filter screening of the event to be processed, wherein the preset mapping rules include but are not limited to or hash functions, so that it can be accurately determined whether the rules associated with the attribute identification information of the event to be processed are included in the rule set.

[0185] For example, let's take the case where Company A in Country X and Company B in Country Y export CC, the export quantity of CC is DD kilograms, and the business number of the case is 20202627893.

[0186] It is still assumed that the rule identification information associated with the to-be-processed event is determined to be 07167x through the above text search engine.

[0187] The Bloom filter subsystem can then learn, based on the rule identification information, that the type of the to-be-processed event is included in the rule set of the Bloom filter.

[0188] Then, based on other attribute identification information obtained from the attribute identification information set, the business type of the event to be processed is determined.

[0189] Assume that other attribute identification information is {Company A, Company B, Export}, and Export is the inquiry link.

[0190] Then, based on other attribute identification information {Company A, Company B, Export}, it is determined that the business type of the event to be processed is the inquiry link in the import and export business.

[0191] Based on the above business types, a set of hash functions corresponding to the inquiry process in the import and export business is determined.

[0192] Then, based on the determined hash function set and in accordance with preset mapping rules, the attribute identification information of the event to be processed and other attribute identification information are mapped to data bits of the bit array of the corresponding Bloom filter.

[0193] S109. Based on the values ​​of the data bits of the attribute identification information, determine whether the rules associated with the attribute identification information of the event to be processed are included in the corresponding rule set; and based on the values ​​of the data bits of other attribute identification information, determine whether the rules associated with other attribute identification information of the event to be processed are included in the corresponding rule set. If both are yes, execute S110; otherwise, execute S111.

[0194] In a specific embodiment, each numerical value of the data bit of the attribute identification information is compared with each numerical value contained in a preset rule set of a Bloom filter corresponding to the attribute identification information to obtain a first comparison result, and based on the first comparison result, it is judged whether the rules associated with the attribute identification information of the event to be processed are included in the corresponding rule set; and each numerical value of the data bit of other attribute identification information is compared with each numerical value contained in a preset data set of a Bloom filter corresponding to other attribute identification information to obtain a second comparison result, and based on the second comparison result, it is judged whether the rules associated with other attribute identification information of the event to be processed are included in the corresponding rule set.

[0195] S110. If it is determined that the attribute identification information of the event to be processed and the rules associated with other attribute identification information are included in the corresponding rule set, the event to be processed is transferred to the next data processing device for data-oriented processing, or the event to be processed is pushed to the display device so that relevant personnel can manually review the event to be processed through the data processing device.

[0196] S111 : If it is determined that at least one rule associated with the attribute identification information of the event to be processed and other attribute identification information is not included in the rule set, the event to be processed is filtered and released.

[0197] In an embodiment of the present application, in order to improve the screening accuracy and reduce the false positive rate, a first subsystem and a second subsystem are configured in the Bloom filter subsystem, which are respectively used to perform a second re-examination of each attribute identification information of the above-mentioned events to be processed, wherein the first subsystem is used to screen whether the attribute identification information of the events to be processed is included in the rule set of the Bloom filter, which corresponds to the rules for a specific industry; the second subsystem is used to screen whether there is an exemption license rule associated with the attribute identification information of the events to be processed, and which exemption license rule the attribute identification information of the events to be processed is included in.

[0198] It should be noted that, in the embodiments of this application, refer to Figure 6 As shown, in the second subsystem of the Bloom filter subsystem, after executing S107 and before executing S108, the following steps need to be executed:

[0199] S107 - 1 . Based on the rule identification information associated with the attribute identification information, obtain the rule content of the rule identification information.

[0200] S107-2. Based on the rule content, determine whether the rule identification information includes a license exemption exception rule. If so, execute S108-1; otherwise, execute S108-2.

[0201] In the embodiment of the present application, after executing S107-2, when executing S108, the above functions are implemented by executing S108-1 or S108-2:

[0202] S108 - 1 . If it is determined that the rule identification information includes a permission exemption exception rule, the attribute identification information is mapped to a data bit of a bit array of a Bloom filter corresponding to the permission exemption exception rule.

[0203] In a specific implementation, if the determination rule identification information includes a license exemption exception rule, the attribute identification information is mapped to the data bits of the bit array of the Bloom filter corresponding to the license exemption exception rule based on the hash functions included in the hash function set and the preset mapping rules.

[0204] S108-2: If it is determined that the rule identification information does not include the permission exemption exception rule, the attribute identification information is mapped to a data bit of a bit array that does not include a Bloom filter corresponding to the permission exemption exception rule.

[0205] In a specific implementation, if it is determined that the rule identification information does not include the license exemption exception rule, based on the hash functions and preset mapping rules contained in the hash function set, the attribute identification information is mapped to the data bits of the bit array that does not include the Bloom filter corresponding to the license exemption exception rule.

[0206] For example, let's take the case where Company A in Country X and Company B in Country Y export CC, the export quantity of CC is DD kilograms, and the business number of the case is 20202627893.

[0207] Assume that based on the determined rule identification information, the obtained rule content contains a license exemption exception rule.

[0208] Since the content of various license application regulations in the rules usually do not overlap, it is assumed that the obtained rule content contains the following information:

[0209] 1. Items that are eligible for license application as described in Chapters AB and C;

[0210] 2. Items exempted from D default license application (“EXCEPT” description);

[0211] 3. Items on Country M’s YY List that are high-risk and cannot apply for a license (“EXCEPT” description);

[0212] 4. The EEF chapter describes items that need to be released to standard organizations and are exempted from license applications (the "EXCEPT" description part).

[0213] The sorted licensing exemption exception rules are as follows:

[0214] 1) Items in Chapters AB (licenses available)

[0215] 2) Items in Chapter C (licenses available)

[0216] 3) D Default license application exemption items (no license application required)

[0217] 4) High-risk items included in Country M’s “YY List” (licenses cannot be applied for)

[0218] 5) EEF Chapter Items (No license required)

[0219] Then, in the embodiment of the present application, the above five types are integrated into two major categories and three subcategories, namely, without license exemption exception rules (i.e., not including license exemption exception rules) - a license can be applied for; with license exemption exception rules (i.e., including license exemption exception rules) - no need to apply for a license and no license can be applied for.

[0220] In an embodiment of the present application, the attribute identification information of the event to be processed is mapped to the three subclass license application situations mentioned above through an AND or hash function, and the numerical value of the data bit of the corresponding Bloom filter bit array is obtained, and compared with the various numerical values ​​contained in the preset Bloom filter rule set. When it is determined that the comparison result indicates that the rule associated with the attribute identification information of the event to be processed is included in the corresponding rule set, it is determined that the event to be processed meets the license exemption exception rule, or does not meet the license exemption exception rule.

[0221] See Figure 7A As shown, it is assumed that the event to be processed involves the application scenario of "1) AB chapter items (license can be applied for)".

[0222] Because the existing license exemption exception rules are "1. Items described in Chapters AB and C that can apply for a license."

[0223] The pending incident involves "1) AB Chapter Items (for which a license can be applied)", but does not involve "2) C Chapter Items (for which a license can be applied)" and "4) High-risk items in Country M's 'YY List' (for which a license cannot be applied)".

[0224] Then, if mapping is performed through a simple hash function, only the first value of the data bit can be obtained as 1, and the subsequent two data bits will not obtain corresponding values. Therefore, in an embodiment of the present application, in order to avoid the above situation, for this scenario, a preset hash function set and a preset mapping rule (such as AND or Hash function) are configured, and the values ​​of each data bit of the bit group contained in the rule set of the Bloom filter in this scenario are configured to be 1, 1, 1.

[0225] In this way, since the event to be processed does not meet the license exemption exception rules (i.e., does not include the license exemption exception rules) - a license can be applied for, the attribute identification information of the event to be processed can be mapped through the above-mentioned hash function set and preset mapping rules, and the values ​​of each data bit of the Bloom filter bit group obtained are 1, 1, 1.

[0226] The obtained rule set of the Bloom filter corresponding to the attribute identification information contains values ​​of 1, 1, 1.

[0227] Therefore, the attribute identification information of the event to be processed, through the determined set of hash functions and preset mapping rules, the value of each data bit obtained is the same as the value of the data bit contained in the rule set of the corresponding Bloom filter, then it is determined that the rule associated with the event to be processed is contained in the above-mentioned rule set of the Bloom filter corresponding to the attribute identification information, that is, it is accurately determined that the event to be processed meets the scenario of "1) AB Chapter Items (License Can Be Applied For)", then, the event to be processed is pushed to the display device so that humans can manually review the event to be processed through the display device, or, the event to be processed is pushed to the next data processing device for data-directed processing.

[0228] See Figure 7B As shown, it is assumed that the event to be processed involves the application scenario of "3)D default license application exemption items (no license application required)".

[0229] Since the existing license exemption exception rule is "2. Items exempted from D default license application ("EXCEPT" description part)", it corresponds to the pending event involving "3) D default license application exemption items (no need to apply for a license)".

[0230] Then, the attribute identification information of the event to be processed is mapped using a preset set of hash functions, and the value of each data bit of the bit group of the Bloom filter corresponding to the event to be processed is 1. At this time, since "3)D default license application exemption items (no need to apply for a license)" corresponds to a scenario where no license application is required, that is, no subsequent manual review is required, or the next data processing device performs directional data processing. Therefore, in an embodiment of the present application, the value of each data bit of the bit group of the Bloom filter in this scenario is configured to 0, 0, 0.

[0231] In this way, the attribute identification information of the event to be processed passes through the preset hash function set and the preset mapping rule, and the values ​​of the respective data bits of the bit group of the Bloom filter obtained are 1, 1, 1.

[0232] The obtained rule set of the Bloom filter corresponding to the attribute identification information contains values ​​of 0, 0, 0.

[0233] Therefore, if the attribute identification information of the event to be processed, through the determined hash function set and preset mapping rules, the value of each data bit obtained is different from the value of the data bit contained in the corresponding Bloom filter rule set, then it is determined that the rule associated with the event to be processed is not included in the above-mentioned Bloom filter rule set corresponding to the attribute identification information, that is, it is determined that the rule associated with the event to be processed is not included in the Bloom filter rule set, then the event to be processed is directly filtered and released.

[0234] In this way, since there is no need for further processing of the pending event in this scenario, that is, there is no need to push the pending event to a display device for manual review of the pending event through the display device, and there is no need to push the pending event to the next data processing device for data-directed processing, through the above-mentioned screening of the Bloom filter, it can be accurately determined that the pending event meets the scenario of "no need to apply for a license", thereby improving the screening accuracy and avoiding the increased time and labor costs of pushing the pending event to a display device or the next data processing device for further processing when it is determined in the traditional screening method that the pending event hits "2. Belongs to D default license application exemption items ("EXCEPT" description part)".

[0235] See Figure 7C As shown, it is assumed that the event to be processed involves the application scenario of "4) high-risk items in the "YY List" of Country M (licenses cannot be applied for)".

[0236] Since the existing license exemption exception rule is "3. Items that are high-risk and cannot apply for a license in the "YY List" of Country M (the "EXCEPT" description part)", it corresponds to the pending incident involving "4) High-risk items in the "YY List" of Country M (cannot apply for a license)".

[0237] Then, in the embodiment of the present application, the values ​​of the data bits of the bit group of the Bloom filter in this scenario are configured to be 1, 1, 1.

[0238] That is, the attribute identification information of the event to be processed is mapped using a preset hash function set, and the value of each data bit of the bit group of the Bloom filter corresponding to the event to be processed is obtained to be 1.

[0239] The obtained rule set of the Bloom filter corresponding to the attribute identification information contains values ​​of 1, 1, 1.

[0240] Therefore, if the attribute identification information of the event to be processed, through the determined hash function set and the preset mapping rules, obtains the same numerical value of each data bit as the numerical value of the data bit contained in the rule set of the corresponding Bloom filter, then it is determined that the rule associated with the event to be processed is contained in the rule set of the Bloom filter corresponding to the attribute identification information, that is, it is determined that the rule associated with the event to be processed is contained in the rule set of the Bloom filter, then the event to be processed is pushed to the display device so that the event to be processed can be manually reviewed by humans through the display device, or the event to be processed is pushed to the next data processing device for data-directed processing.

[0241] It should be noted that, in the embodiments of the present application, the attribute identification information and other attribute identification information input into the Bloom filter mentioned above can be pre-customized according to the needs of the application scenario, and is not limited to the input of the attribute identification information of the hit blacklist set into the Bloom filter corresponding to the license exemption exception rule mentioned in the embodiments of the present application. The attribute identification information mentioned in the embodiments of the present application are all taken from the attribute identification information set of the event to be processed. Which specific attribute identification information is input into the corresponding Bloom filter can be determined based on the specific application scenario and is not specifically limited here.

[0242] Further, in the embodiments of this application, see Figure 8 As shown, a method for optimizing the Bloom filter subsystem included in the screening system is also provided. The specific process of the above optimization method is as follows:

[0243] S800: When it is determined that the service type of the event to be processed meets the optimization condition, perform the following operations to obtain an updated rule set of the Bloom filter corresponding to the service type of the event to be processed:

[0244] S800 - 1. Obtain optimization configuration information corresponding to the optimization condition, and determine, based on the optimization configuration information, a target feature value array of a Bloom filter corresponding to a service type of an event to be processed.

[0245] S800-2. Concatenate the target feature value array with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed according to a preset concatenation rule to obtain a new bit array. Based on the new bit array, update the value corresponding to the attribute identification information in the rule set of the Bloom filter.

[0246] and / or,

[0247] According to the preset splicing rules, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed to obtain a new bit array, and based on the above new bit array, the numerical values ​​corresponding to the above other attribute identification information in the rule set of the Bloom filter are updated.

[0248] In the embodiment of the present application, the above-mentioned optimization conditions include some or all of the following conditions:

[0249] Condition 1: receiving a manually triggered optimization instruction, wherein the optimization instruction includes the business type of the event to be processed, and the optimization instruction is issued when it is manually determined that the misjudgment rate of the processed event corresponding to the business type of the event to be processed is greater than the misjudgment rate threshold.

[0250] Condition 2: It is determined that the rule associated with the attribute identification information of the event to be processed and any one of the other attribute identification information is included in the rule set of the corresponding Bloom filter.

[0251] In the embodiments of this application, see Figure 9 As shown, when executing S800-1, the above functions can be realized by executing the following steps:

[0252] S900: Acquire a candidate event set, wherein the candidate event set includes multiple processed events, and attribute identification information of each processed event and rules associated with other attribute identification information are included in a corresponding rule set.

[0253] S910: Perform the following operations for each processed event in the candidate event set:

[0254] A preset first number of attribute identification information is obtained from the attribute identification information set of the processed event, and according to the preset second number of dimension information contained in the optimization configuration information, the matching degree between each attribute identification information and the rule corresponding to the processed event is obtained, and based on the calculated matching degrees, the sub-feature values ​​corresponding to each of the obtained attribute identification information are determined, wherein each dimension information represents the feature information of the dimension corresponding to the processed event; based on the sub-feature value corresponding to each attribute identification information, the feature value corresponding to each dimension information is determined, and the determined feature values ​​are sorted according to the preset sorting rules to obtain a sorted candidate feature value array.

[0255] S920. De-duplicate the obtained candidate eigenvalue arrays, and perform operations on the eigenvalues ​​contained in the de-duplicated candidate eigenvalue arrays according to a preset operation method to obtain a target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed, wherein the preset operation method is a logical OR operation method.

[0256] In this way, in the embodiment of the present application, the information fingerprint subsystem can be used to assist in updating the rule set of the Bloom filter. Since the multi-dimensional feature values ​​determined by the information fingerprint subsystem are common features extracted for the business type, the target feature value array is spliced ​​with the pre-constructed bit array of the Bloom filter, and the bit value of the Bloom filter is reconstructed, thereby optimizing the screening system and further improving the screening accuracy of the screening system.

[0257] For example, let's take the case where Company A in Country X and Company B in Country Y export CC, the export quantity of CC is DD kilograms, and the business number of the case is 20202627893.

[0258] Assume that the screening system determines that the business type of the event to be processed meets the optimization conditions.

[0259] Then, the optimization configuration information corresponding to the optimization condition and the candidate event set are obtained, wherein the candidate event set includes multiple processed events, and the attribute identification information of each processed event and the rules associated with other attribute identification information are included in the corresponding rule set.

[0260] Assume that the candidate event set contains 5 processed events, which are recorded as processed event 1, processed event 2, processed event 3, processed event 4, and processed event 5 respectively.

[0261] It is also assumed that two attribute identification information are obtained from the attribute identification information set of each processed event, and the above-mentioned optimization configuration information includes dimension information of eight dimensions, such as r1, r2, r3, r4, r5, r6, r7, and r8.

[0262] Then, the candidate feature value arrays of each processed event in the candidate event set are obtained respectively. In the specific implementation:

[0263] 1. Handled event 1:

[0264] Two attribute identification information are obtained from the attribute identification information set of the processed event 1, and the matching degree between each attribute identification information and the rule corresponding to the processed event is obtained according to the eight dimension information included in the optimization configuration information.

[0265] Then, based on the obtained matching degrees, the sub-feature values ​​1 corresponding to the above eight dimensions are determined. If the first attribute identification information is determined to be associated with the rule corresponding to the processed event, the corresponding sub-feature value 1 is w1; otherwise, the corresponding sub-feature value 1 is -w1; if the second attribute identification information is associated with the rule corresponding to the processed time, the corresponding sub-feature value 1' is w2; otherwise, the corresponding sub-feature value 1' is -w2.

[0266] Assume that the matching results are as follows:

[0267] Table 2

[0268]

[0269] Then, the sub-eigenvalues ​​1 and 1' of the same dimension are accumulated to obtain the final eigenvalue 1, as shown in the following table:

[0270] Table 3

[0271]

[0272] Then, the determined eigenvalues ​​1 are sorted according to a preset sorting rule to obtain a sorted candidate eigenvalue array 1. For example, the obtained candidate eigenvalue array 1 is 11000101.

[0273] 2. Handled event 2:

[0274] Two attribute identification information are obtained from the attribute identification information set of the processed event 2, and the matching degree between each attribute identification information and the rule corresponding to the processed event is obtained according to the eight dimension information included in the optimization configuration information.

[0275] Then, based on the obtained matching degrees, the sub-eigenvalues ​​2 and sub-eigenvalues ​​2' corresponding to the above 8 dimensions are determined, and the sub-eigenvalues ​​2 and sub-eigenvalues ​​2' of the same dimension are accumulated to obtain the final eigenvalue 2. Then, according to the preset sorting rules, the determined eigenvalues ​​2 are sorted to obtain the sorted candidate eigenvalue array 2. For example, the obtained candidate eigenvalue array 2 is 11000101.

[0276] 3. Processed event 3, processed event 4, and processed event 5:

[0277] The same operation is performed on processed events 3, 4, and 5. Assume that the obtained candidate eigenvalue array 3 is 01101100, the obtained candidate eigenvalue array 4 is 10101100, and the obtained candidate eigenvalue array 5 is 01101100.

[0278] Then, the obtained 5 candidate feature value arrays are deduplicated, and the obtained candidate feature group arrays after deduplication are shown in Table 4:

[0279] Table 4

[0280] category Numerical Array of candidate eigenvalues ​​after deduplication Candidate eigenvalue array 1 11000101 11000101 Candidate eigenvalue array 2 11000101 / Candidate eigenvalue array 3 01101100 01101100 Candidate eigenvalue array 4 10101100 10101100 Candidate eigenvalue array 5 01101100 /

[0281] Then, according to the preset calculation method, each eigenvalue contained in each candidate eigenvalue array after deduplication is calculated, and the target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed is obtained as shown in Table 5:

[0282] Table 5

[0283]

[0284] Finally, according to the preset splicing rules, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed to obtain a new bit array, such as 11101101+bit array 1 (the original constructed bit value); and based on the new bit array (i.e., 11101101+bit array), the value corresponding to the attribute identification information in the rule set of the Bloom filter is updated;

[0285] and / or,

[0286] Then, according to the preset splicing rules, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed to obtain a new bit array, such as 11101101+bit array 2 (the original constructed bit value); and based on the new bit array (i.e., 11101101+bit array), the values ​​corresponding to other attribute identification information in the rule set of the Bloom filter are updated.

[0287] It should be noted that in an embodiment of the present application, the above-mentioned S800, S800-1, and S800-2 can be used to perform multiple optimization operations on the Bloom filter included in the screening system, wherein, in one optimization operation, the attribute identification information selected from different processed events is the same; in this way, through the above-mentioned optimization operations, the screening accuracy of the screening system can be improved, and the screening efficiency of the screening system can also be improved.

[0288] It should be noted that in the embodiment of the present application, after the above-mentioned optimization method is used to update the rule set of the Bloom filter corresponding to the business type of the event to be processed, the subsequent events to be processed screened by the screening system are all screened using the updated rule set of the Bloom filter. Accordingly, after the optimization operation, the corresponding hash function set needs to be adjusted accordingly so that subsequent screening can proceed smoothly. In the embodiment of the present application, since only the value of one business type is updated in the rule set of the Bloom filter, other business types can still be compared with the values ​​in the rule set contained in the pre-built corresponding Bloom filter during the screening process.

[0289] In the embodiments of this application, see Figure 10 As shown, a specific process of a data processing method is provided as follows:

[0290] S1000. When it is determined that the attribute identification information of the event to be processed is included in the blacklist set, the attribute identification information is mapped to the data bits of the bit array of the Bloom filter, wherein the values ​​of the respective data bits of the bit array are set based on the rule set of the Bloom filter, and the rule set includes at least one rule that matches each attribute identification information.

[0291] S1010: Determine whether the rule associated with the attribute identification information of the event to be processed is included in the rule set based on the value of the data bit.

[0292] S1020: If it is determined that the rule associated with the attribute identification information of the event to be processed is not included in the rule set, filter and release the event to be processed.

[0293] In an embodiment of the present application, the above-mentioned data processing method adopts the mathematical principle that the probability of conflict between two completely random numbers of the Bloom filter is extremely small. The attribute identification information of the event to be processed is mapped to the data bit of the bit array of the corresponding Bloom filter, and then the value of the data bit is compared with the various values ​​contained in the preset rule set of the Bloom filter to determine whether the attribute identification information of the event to be processed is included in the rule set. In this way, it is possible to accurately determine whether the attribute identification information of the event to be processed is in the rule set of the corresponding Bloom filter, thereby improving the screening accuracy and improving the screening efficiency.

[0294] In the embodiments of the present application, the text search, hash function mapping, etc. performed on the above-mentioned events to be processed are all performed based on the acquired data associated with the events to be processed. It should be emphasized that the acquisition, storage, use, and processing of the data associated with the above-mentioned events to be processed are in compliance with the relevant provisions of national laws and regulations.

[0295] Based on the same inventive concept, see Figure 11 As shown, an embodiment of the present application provides a data processing device, including:

[0296] a determination module 111 configured to, when determining that the attribute identification information of the event to be processed is included in the blacklist set, map the attribute identification information to data bits of a bit array of a Bloom filter, wherein the value of each data bit of the bit array is set based on a rule set of the Bloom filter, the rule set including at least one rule matching each attribute identification information;

[0297] A judgment module 112, configured to judge whether the rule associated with the attribute identification information of the event to be processed is included in the rule set based on the value of the data bit;

[0298] The push module 113 is configured to filter and release the event to be processed if it is determined that the rule associated with the attribute identification information of the event to be processed is not included in the rule set.

[0299] In some embodiments, whether the attribute identification information of the event to be processed is included in the blacklist set is determined by performing the following operations:

[0300] Acquire the event to be processed, and preprocess the event to be processed to obtain an attribute identification information set of the event to be processed, wherein the preprocessing includes any one or any combination of word segmentation processing, filtering processing, and character mapping processing, and the attribute identification information set includes attribute identification information of different dimensions of the event to be processed;

[0301] Calling a preset text search engine to perform a text search on each attribute identification information included in the attribute identification information set of the event to be processed based on the blacklist set to obtain search results;

[0302] If the attribute identification information representing the search result matches any grouping category included in the blacklist set, it is determined that the attribute identification information of the event to be processed is included in the blacklist set, wherein the attribute identification information is any attribute identification information in the attribute identification information set.

[0303] In some embodiments, the text search algorithm configured by the text search engine includes any one or any combination of a fuzzy query rule set, a vector space model, a word frequency inverse text frequency index algorithm, a word frequency saturation algorithm, a length normalization algorithm, and a relevance weight algorithm.

[0304] In some embodiments, after determining that the attribute identification information of the event to be processed is included in the blacklist set and before mapping the attribute identification information to the data bits of the bit array of the Bloom filter, the determining module 111 is further configured to:

[0305] Acquire other attribute identification information associated with the attribute identification information from the attribute identification information set of the event to be processed;

[0306] Determining a service type of the event to be processed based on the other attribute identification information, and determining a corresponding hash function set based on the service type, wherein the number of hash functions included in the hash function set is predetermined based on the service type of the message;

[0307] Mapping the attribute identification information to data bits of a bit array of a Bloom filter includes:

[0308] Based on each hash function included in the hash function set and a preset mapping rule, the attribute identification information and the other attribute identification information are respectively mapped to data bits of a bit array of a corresponding Bloom filter.

[0309] In some embodiments, after determining that the attribute identification information of the event to be processed is included in the blacklist set and before mapping the attribute identification information to the data bits of the bit array of the Bloom filter, the determining module 111 is further configured to:

[0310] Based on the rule identification information associated with the attribute identification information, obtaining the rule content of the rule identification information, and judging whether the rule identification information includes a license exemption exception rule based on the rule content;

[0311] Mapping the attribute identification information to data bits of a bit array of a Bloom filter includes:

[0312] If it is determined that the rule identification information does not include a license exemption exception rule, mapping the attribute identification information to a data bit of the bit array that does not include a Bloom filter corresponding to the license exemption exception rule;

[0313] or,

[0314] If it is determined that the rule identification information includes a license exemption exception rule, the attribute identification information is mapped to a data bit of a bit array of a Bloom filter corresponding to the license exemption exception rule.

[0315] In some embodiments, the rule identification information associated with the attribute identification information is determined by performing the following operations:

[0316] Based on the search results, rule identification information of the matched grouping category is acquired, and the rule identification information of the grouping category is used as rule identification information associated with the attribute identification information.

[0317] In some embodiments, the determining module 112 is configured to determine whether the rule associated with the attribute identification information of the event to be processed is included in the rule set based on the value of the data bit:

[0318] Comparing each value of the data bit of the attribute identification information with each value included in a preset rule set of a Bloom filter corresponding to the attribute identification information to obtain a first comparison result, and determining whether the rule associated with the attribute identification information of the event to be processed is included in the corresponding rule set based on the first comparison result;

[0319] In addition, the various numerical values ​​of the data bits of the other attribute identification information are compared with the various numerical values ​​contained in the preset data set of the Bloom filter corresponding to the other attribute identification information to obtain a second comparison result, and based on the second comparison result, it is determined whether the rules associated with the other attribute identification information of the event to be processed are included in the corresponding rule set.

[0320] In some embodiments, the push module 113 is further configured to:

[0321] If it is determined that the attribute identification information of the event to be processed and the rules associated with the other attribute identification information are all included in the corresponding rule set, the event to be processed will be transferred to the next data processing device for data-oriented processing, or the event to be processed will be pushed to the display device so that relevant personnel can manually review the event to be processed through the data processing device.

[0322] In some embodiments, after pushing the pending event to the display device, the pushing module 113 is further configured to:

[0323] When it is determined that the business type of the event to be processed meets the optimization condition, the following operations are performed to obtain an updated rule set of the Bloom filter corresponding to the business type of the event to be processed:

[0324] Obtaining optimization configuration information corresponding to the optimization condition, and determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information;

[0325] According to a preset splicing rule, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed to obtain a new bit array, and based on the new bit array, the value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated;

[0326] and / or,

[0327] According to the preset splicing rules, the target feature value array and the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed are spliced ​​to obtain a new bit array, and based on the new bit array, the numerical value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated.

[0328] In some embodiments, the optimized conditions include some or all of the following:

[0329] Receiving a manually triggered optimization instruction, wherein the optimization instruction includes the business type of the event to be processed, and the optimization instruction is issued when it is determined that the misjudgment rate of processed events corresponding to the business type of the event to be processed is greater than a misjudgment rate threshold;

[0330] The rule for determining the association between the attribute identification information of the event to be processed and any one of the other attribute identification information is included in the rule set of the corresponding Bloom filter.

[0331] In some embodiments, based on the optimization configuration information, the target feature value array of the Bloom filter corresponding to the business type of the event to be processed is determined, and the push module 113 is used to:

[0332] Acquire a candidate event set, wherein the candidate event set includes multiple processed events, and attribute identification information of each processed event and rules associated with other attribute identification information are included in a corresponding rule set;

[0333] For each processed event in the candidate event set, perform the following operations:

[0334] Obtaining a preset first number of attribute identification information from the attribute identification information set of the processed event, and obtaining a matching degree between each attribute identification information and a rule corresponding to the processed event according to a preset second number of dimension information included in the optimization configuration information, and determining a sub-feature value corresponding to each obtained attribute identification information based on each obtained matching degree, wherein each dimension information represents feature information of a dimension corresponding to the processed event;

[0335] Based on the sub-eigenvalue corresponding to each attribute identification information, the eigenvalue corresponding to each dimension information is determined, and the determined eigenvalues ​​are sorted according to a preset sorting rule to obtain a sorted candidate eigenvalue array.

[0336] Deduplication processing is performed on each obtained candidate eigenvalue array, and each eigenvalue contained in each candidate eigenvalue array after deduplication processing is calculated according to a preset calculation method to obtain a target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed.

[0337] Based on the same inventive concept as the above-mentioned data processing method, the embodiment of the present application also provides an electronic device, which can be a control device or control system inside a smart device, or an external device that communicates with the smart device, such as a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), a server, etc. Figure 12 As shown, the electronic device may include a processor 1201 and a memory 1202 .

[0338] The processor 1201 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0339] Memory 1202 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory can include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disc, etc. Memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1202 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0340] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any one of the methods in the various exemplary embodiments described above.

[0341] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0342] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one flow chart or multiple flows and / or one box or multiple boxes in the block diagram.

[0343] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0344] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0345] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: When it is determined that the attribute identification information of the event to be processed is included in the blacklist set, obtaining other attribute identification information associated with the attribute identification information from the attribute identification information set of the event to be processed, and determining the business type of the event to be processed based on the other attribute identification information; Based on each hash function included in the hash function set corresponding to the business type and a preset mapping rule, respectively mapping the attribute identification information and the other attribute identification information to data bits of a bit array of a corresponding Bloom filter, wherein the value of each data bit of any bit array is set based on a rule set of the corresponding Bloom filter, and the rule set includes at least one rule that matches each attribute identification information or each other attribute identification information; Comparing the value of the data bit of the attribute identification information with the value included in the preset rule set of the Bloom filter corresponding to the attribute identification information, and comparing the value of the data bit of the other attribute identification information with the value included in the preset data set of the Bloom filter corresponding to the other attribute identification information, to determine whether the rules associated with the attribute identification information and the other attribute identification information are included in the respective corresponding rule sets; If it is determined that a rule for associating at least one of the attribute identification information and the other attribute identification information is not included in the corresponding rule set, the event to be processed is filtered and released.

2. The method according to claim 1, wherein Determine whether the attribute identification information of the event to be processed is included in the blacklist set by performing the following operations: Acquire the event to be processed, and preprocess the event to be processed to obtain an attribute identification information set of the event to be processed, wherein the preprocessing includes any one or any combination of word segmentation processing, filtering processing, and character mapping processing, and the attribute identification information set includes attribute identification information of different dimensions of the event to be processed; Calling a preset text search engine to perform a text search on each attribute identification information included in the attribute identification information set of the event to be processed based on the blacklist set to obtain search results; If the attribute identification information representing the search result matches any grouping category included in the blacklist set, it is determined that the attribute identification information of the event to be processed is included in the blacklist set, wherein the attribute identification information is any attribute identification information in the attribute identification information set.

3. The method according to claim 2, wherein The text search algorithm configured by the text search engine includes any one or any combination of fuzzy query rule set, vector space model, word frequency inverse text frequency index algorithm, word frequency saturation algorithm, length normalization algorithm and relevance weight algorithm.

4. The method according to any one of claims 1 to 3, characterized in that: The hash functions included in the hash function set corresponding to the business type and the preset mapping rules respectively map the attribute identification information and the other attribute identification information to data bits of the bit array of the corresponding Bloom filter, including: Determining, based on the service type, a set of hash functions corresponding to the service type, wherein the number of hash functions included in the set of hash functions is predetermined based on the service type of the message; Based on each hash function included in the hash function set and the preset mapping rule, the attribute identification information and the other attribute identification information are respectively mapped to data bits of the bit array of the corresponding Bloom filter.

5. The method according to claim 4, wherein After determining that the attribute identification information of the event to be processed is included in the blacklist set, and before mapping the attribute identification information to the data bits of the bit array of the corresponding Bloom filter, the method further includes: Based on the rule identification information associated with the attribute identification information, obtaining the rule content of the rule identification information, and judging whether the rule identification information includes a license exemption exception rule based on the rule content; Mapping the attribute identification information to data bits of a bit array of a corresponding Bloom filter includes: If it is determined that the rule identification information does not include a license exemption exception rule, mapping the attribute identification information to a data bit of the bit array that does not include a Bloom filter corresponding to the license exemption exception rule; or, If it is determined that the rule identification information includes a license exemption exception rule, the attribute identification information is mapped to a data bit of a bit array of a Bloom filter corresponding to the license exemption exception rule.

6. The method according to claim 5, wherein Determine the rule identification information associated with the attribute identification information by performing the following operations: Based on the search results, rule identification information of the matched grouping category is acquired, and the rule identification information of the grouping category is used as rule identification information associated with the attribute identification information.

7. The method according to claim 4, wherein The method further comprises: If it is determined that the attribute identification information of the event to be processed and the rules associated with the other attribute identification information are all included in the corresponding rule set, the event to be processed will be transferred to the next data processing device for data-oriented processing, or the event to be processed will be pushed to the display device so that relevant personnel can manually review the event to be processed through the data processing device.

8. The method according to claim 7, wherein After pushing the event to be processed to the display device, the method further includes: When it is determined that the business type of the event to be processed meets the optimization condition, the following operations are performed to obtain an updated rule set of the Bloom filter corresponding to the business type of the event to be processed: Obtaining optimization configuration information corresponding to the optimization condition, and determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information; According to a preset splicing rule, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed to obtain a new bit array, and based on the new bit array, the value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated; and / or, According to the preset splicing rules, the target feature value array and the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed are spliced ​​to obtain a new bit array, and based on the new bit array, the numerical value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated.

9. The method according to claim 8, wherein The optimization conditions include some or all of the following: Receiving a manually triggered optimization instruction, wherein the optimization instruction includes the business type of the event to be processed, and the optimization instruction is issued when it is determined that the misjudgment rate of processed events corresponding to the business type of the event to be processed is greater than a misjudgment rate threshold; The rule for determining the association between the attribute identification information of the event to be processed and any one of the other attribute identification information is included in the rule set of the corresponding Bloom filter.

10. The method according to claim 8, wherein Determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information includes: Acquire a candidate event set, wherein the candidate event set includes multiple processed events, and attribute identification information of each processed event and rules associated with other attribute identification information are included in a corresponding rule set; For each processed event in the candidate event set, perform the following operations: Obtaining a preset first number of attribute identification information from the attribute identification information set of the processed event, and obtaining a matching degree between each attribute identification information and a rule corresponding to the processed event according to a preset second number of dimension information included in the optimization configuration information, and determining a sub-feature value corresponding to each obtained attribute identification information based on each obtained matching degree, wherein each dimension information represents feature information of a dimension corresponding to the processed event; Based on the sub-feature value corresponding to each attribute identification information, determine the feature value corresponding to each dimension information, and sort the determined feature values ​​according to the preset sorting rules to obtain a sorted candidate feature value array; Deduplication processing is performed on each obtained candidate eigenvalue array, and each eigenvalue contained in each candidate eigenvalue array after deduplication processing is calculated according to a preset calculation method to obtain a target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed.

11. A data processing device, characterized in that: include: A determination module, configured to, when determining that the attribute identification information of the event to be processed is included in the blacklist set, obtain other attribute identification information associated with the attribute identification information from the attribute identification information set of the event to be processed, and determine the business type of the event to be processed based on the other attribute identification information; based on each hash function included in the hash function set corresponding to the business type and a preset mapping rule, map the attribute identification information and the other attribute identification information to data bits of a bit array of a corresponding Bloom filter, wherein the value of each data bit of any bit array is set based on a rule set of the corresponding Bloom filter, and the rule set includes at least one rule that matches each attribute identification information or each other attribute identification information; a judgment module, configured to compare the value of the data bit of the attribute identification information with the value contained in the preset rule set of the Bloom filter corresponding to the attribute identification information, and to compare the value of the data bit of the other attribute identification information with the value contained in the preset data set of the Bloom filter corresponding to the other attribute identification information, so as to determine whether the rules associated with the attribute identification information and the other attribute identification information are included in the respective corresponding rule sets; The push module is configured to filter and release the event to be processed if it is determined that a rule for associating at least one of the attribute identification information and the other attribute identification information is not included in the corresponding rule set.

12. The device according to claim 11, wherein Determine whether the attribute identification information of the event to be processed is included in the blacklist set by performing the following operations: Acquire the event to be processed, and preprocess the event to be processed to obtain an attribute identification information set of the event to be processed, wherein the preprocessing includes any one or any combination of word segmentation processing, filtering processing, and character mapping processing, and the attribute identification information set includes attribute identification information of different dimensions of the event to be processed; Calling a preset text search engine to perform a text search on each attribute identification information included in the attribute identification information set of the event to be processed based on the blacklist set to obtain search results; If the attribute identification information representing the search result matches any grouping category included in the blacklist set, it is determined that the attribute identification information of the event to be processed is included in the blacklist set, wherein the attribute identification information is any attribute identification information in the attribute identification information set.

13. The device according to claim 12, wherein The text search algorithm configured by the text search engine includes any one or any combination of fuzzy query rule set, vector space model, word frequency inverse text frequency index algorithm, word frequency saturation algorithm, length normalization algorithm and relevance weight algorithm.

14. The device according to any one of claims 11 to 13, characterized in that: The determining module is specifically configured to: Determining, based on the service type, a hash function set corresponding to the service type, wherein the number of hash functions included in the hash function set is predetermined based on the service type of the message; Based on each hash function included in the hash function set and the preset mapping rule, the attribute identification information and the other attribute identification information are respectively mapped to data bits of the bit array of the corresponding Bloom filter.

15. The device according to claim 14, wherein After determining that the attribute identification information of the event to be processed is included in the blacklist set, and before mapping the attribute identification information to the data bits of the bit array of the corresponding Bloom filter, the determining module is further configured to: Based on the rule identification information associated with the attribute identification information, obtaining the rule content of the rule identification information, and judging whether the rule identification information includes a license exemption exception rule based on the rule content; The determining module is specifically configured to: If it is determined that the rule identification information does not include a license exemption exception rule, mapping the attribute identification information to a data bit of the bit array that does not include a Bloom filter corresponding to the license exemption exception rule; or, If it is determined that the rule identification information includes a license exemption exception rule, the attribute identification information is mapped to a data bit of a bit array of a Bloom filter corresponding to the license exemption exception rule.

16. The device according to claim 15, characterized in that Determine the rule identification information associated with the attribute identification information by performing the following operations: Based on the search results, rule identification information of the matched grouping category is acquired, and the rule identification information of the grouping category is used as rule identification information associated with the attribute identification information.

17. The device according to claim 14, wherein The push module is also used for: If it is determined that the attribute identification information of the event to be processed and the rules associated with the other attribute identification information are all included in the corresponding rule set, the event to be processed will be transferred to the next data processing device for data-oriented processing, or the event to be processed will be pushed to the display device so that relevant personnel can manually review the event to be processed through the data processing device.

18. The device according to claim 17, wherein After pushing the to-be-processed event to the display device, the pushing module is further configured to: When it is determined that the business type of the event to be processed meets the optimization condition, the following operations are performed to obtain an updated rule set of the Bloom filter corresponding to the business type of the event to be processed: Obtaining optimization configuration information corresponding to the optimization condition, and determining a target feature value array of a Bloom filter corresponding to the business type of the event to be processed based on the optimization configuration information; According to a preset splicing rule, the target feature value array is spliced ​​with the bit array of the Bloom filter corresponding to the attribute identification information of the event to be processed to obtain a new bit array, and based on the new bit array, the value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated; and / or, According to the preset splicing rules, the target feature value array and the bit array of the Bloom filter corresponding to other attribute identification information of the event to be processed are spliced ​​to obtain a new bit array, and based on the new bit array, the numerical value corresponding to the other attribute identification information in the rule set of the Bloom filter is updated.

19. The device according to claim 18, wherein The optimization conditions include some or all of the following: Receiving a manually triggered optimization instruction, wherein the optimization instruction includes the business type of the event to be processed, and the optimization instruction is issued when it is determined that the misjudgment rate of processed events corresponding to the business type of the event to be processed is greater than a misjudgment rate threshold; The rule for determining the association between the attribute identification information of the event to be processed and any one of the other attribute identification information is included in the rule set of the corresponding Bloom filter.

20. The device according to claim 18, wherein The push module is specifically used for: Acquire a candidate event set, wherein the candidate event set includes multiple processed events, and attribute identification information of each processed event and rules associated with other attribute identification information are included in a corresponding rule set; For each processed event in the candidate event set, perform the following operations: Obtaining a preset first number of attribute identification information from the attribute identification information set of the processed event, and obtaining a matching degree between each attribute identification information and a rule corresponding to the processed event according to a preset second number of dimension information included in the optimization configuration information, and determining a sub-feature value corresponding to each obtained attribute identification information based on each obtained matching degree, wherein each dimension information represents feature information of a dimension corresponding to the processed event; Based on the sub-feature value corresponding to each attribute identification information, determine the feature value corresponding to each dimension information, and sort the determined feature values ​​according to the preset sorting rules to obtain a sorted candidate feature value array; Deduplication processing is performed on each obtained candidate eigenvalue array, and each eigenvalue contained in each candidate eigenvalue array after deduplication processing is calculated according to a preset calculation method to obtain a target eigenvalue array of the Bloom filter corresponding to the business type of the event to be processed.

21. An electronic device, characterized in that: include: a memory for storing program instructions; The processor is configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 10 according to the obtained program instructions.

22. A computer-readable storage medium, characterized in that The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The method comprises computer-executable instructions for causing a computer to execute the method according to any one of claims 1 to 10.

Images