Data screening method and device, electronic equipment and storage medium

Abstract

The invention discloses a data screening method and device, electronic equipment and a storage medium. Comprising the steps of obtaining to-be-processed data; performing hash processing on the to-be-processed data based on the first number of different hash function families to obtain a first number of hash data sets of the to-be-processed data, and determining mapping position information of each hash data set on the bit array block to obtain a first number of mapping position information corresponding to the to-be-processed data; judging the first quantity of mapping position information corresponding to the to-be-processed data based on the second quantity of bit arrays corresponding to the verification data of the set type to obtain a judgment result; and determining a screening result corresponding to the to-be-processed data according to the judgment result. According to the scheme, the to-be-processed data is subjected to approximate processing to obtain the plurality of bit arrays, so that the to-be-processed data is verified according to the plurality of bit arrays, a variable-length fuzzy query method is realized, and the verification efficiency, accuracy and applicability of the to-be-processed data are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data screening method, device, electronic equipment and storage medium. Background Art

[0002] Traditional methods such as manual screening or linear matching of data to be screened in banks require huge human, material and time resources. In the data age, if we only rely on linear query or manual screening methods to screen data, it may be difficult to cope with the rapidly growing data traffic, and the large amount of duplicate data that may exist will increase space consumption and slow the screening speed. At the same time, the traditional approximate query Bloom algorithm does not support the adjustment of the approximate degree of fuzzy queries. When a relatively accurate or more fuzzy matching scheme is required, the above algorithms do not support it, and fuzzy queries cannot be performed. There is also a high misjudgment rate problem. It is urgent to find a fast, efficient and low-resource data screening method. Summary of the invention

[0003] The present invention provides a data screening method, device, electronic device and storage medium to solve the problem that Bloom's algorithm does not support fuzzy query, as well as the problems of low efficiency, low accuracy and low applicability of data screening.

[0004] According to one aspect of the present invention, there is provided a data screening method, comprising:

[0005] Get the data to be processed;

[0006] Based on the first number of hash function families, hash processing is performed on the data to be processed respectively to obtain the first number of hash data sets of the data to be processed, and mapping position information of each hash data set on the bit array block is determined to obtain the first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different;

[0007] Determine the mapping position information of the first quantity corresponding to the to-be-processed data based on the second quantity bit array corresponding to the verification data of the set type, and obtain a determination result, wherein the second quantity bit array corresponding to the verification data is obtained by mapping the hash data set obtained by processing the verification data based on the second quantity hash function family, and the first quantity is less than or equal to the second quantity;

[0008] When the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data, the screening result corresponding to the data to be processed is determined to be that the data to be processed belongs to the set type.

[0009] Optionally, the hash function family includes a locality-sensitive hash function family; based on a first number of hash function families, hash processing is performed on the data to be processed respectively to obtain a first number of hash data sets of the data to be processed, including: determining a first number of locality-sensitive hash function families; for each locality-sensitive hash function family, processing the data to be processed respectively based on multiple locality-sensitive hash functions in the locality-sensitive hash function family to obtain multiple hash data of the data to be processed, and the multiple hash data form a hash data set of the data to be processed; based on the hash data set corresponding to each obtained locality-sensitive hash function family, the first number of hash data sets of the data to be processed are formed.

[0010] Optionally, determining a first number of local sensitive hash function families includes: obtaining an initial local sensitive hash function, wherein the initial local sensitive hash function includes adjustable parameters, and the adjustable parameters include a random vector parameter, a random number parameter, and a hash bucket width parameter; constructing a local sensitive hash function that satisfies a third number by performing a plurality of different adjustments to the random vector parameter and / or the random number corresponding to the initial local sensitive hash function, and the third number of local sensitive hash functions forms a first local sensitive hash function family; performing a plurality of different adjustments to the hash bucket width parameters of the local sensitive hash functions in the first local sensitive hash function family to obtain a local sensitive hash function family that satisfies the first number.

[0011] Optionally, determining mapping position information of each hash data set on the bit array block includes: obtaining a fourth number of random hash functions; for each hash data set, processing the hash data set based on the fourth number of random hash functions to determine the mapping position information of the hash data set on the bit array block.

[0012] Optionally, the hash data set is processed based on a fourth number of random hash functions to determine mapping position information of the hash data set on the bit array block, including: hashing the hash data sets based on the fourth number of random hash functions to obtain multiple hash values ​​corresponding to the hash data sets; for each hash value, performing a modulus operation on the length of the bit array block based on the hash value to determine the mapping position information of the hash value.

[0013] Optionally, there is at least one verification data of the set type; based on the second number of bit arrays corresponding to the verification data of the set type, the mapping position information of the first number of data to be processed is judged, including: traversing at least one verification data of the set type, and for the second number of bit arrays corresponding to each verification data, determining the value corresponding to the position information that matches any mapping position information in the first number of mapping position information corresponding to the data to be processed; if the values ​​corresponding to the matched position information are all preset values, then determining that the result of the judgment is that the first number of mapping position information corresponding to the data to be processed hits the second number of bit arrays corresponding to the verification data; if any of the values ​​corresponding to the matched position information is not a preset value, then determining that the result of the judgment is that the first number of mapping position information corresponding to the data to be processed does not hit the second number of bit arrays corresponding to the verification data.

[0014] Optionally, obtaining the data to be processed includes: obtaining information to be processed and identifying the data type of the information to be processed; if the data type is a digital type, using the information to be processed as the data to be processed; if the data type is a non-digital type, performing data conversion on the information to be processed to obtain data of a digital type corresponding to the information to be processed, and using the data of the digital type corresponding to the information to be processed as the data to be processed.

[0015] According to another aspect of the present invention, there is provided a data screening device, comprising:

[0016] A module for acquiring data to be processed, used for acquiring data to be processed;

[0017] A mapping position information determination module, configured to perform hash processing on the data to be processed based on a first number of hash function families, respectively, to obtain a first number of hash data sets of the data to be processed, and to determine mapping position information of each hash data set on a bit array block, respectively, to obtain a first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes a plurality of hash functions, and the plurality of hash functions in each hash function family are different;

[0018] A determination result determination module, configured to determine, based on a second number of bit arrays corresponding to the verification data of a set type, mapping position information of a first number of data to be processed, to obtain a determination result, wherein the second number of bit arrays corresponding to the verification data are obtained by mapping a hash data set obtained by processing the verification data based on a second number of hash function families, and the first number is less than or equal to the second number;

[0019] The screening result determination module is used to determine that the screening result corresponding to the data to be processed is that the data to be processed belongs to a set type when the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the data screening method of any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the data screening method of any embodiment of the present invention when executed by a processor.

[0025] The technical solution of the embodiment of the present invention is to obtain data to be processed; perform hash processing on the data to be processed based on a first number of hash function families to obtain a first number of hash data sets of the data to be processed, and determine the mapping position information of each hash data set on the bit array block to obtain the first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different; based on a second number of bit arrays corresponding to verification data of a set type, the mapping position information of the first number of data to be processed is judged to obtain a judgment result, wherein the second number of bit arrays corresponding to the verification data are mapped to the hash data sets obtained by processing the verification data based on the second number of hash function families, and the first number is less than or equal to the second number; when the judgment result is that the mapping position information of the first number of data to be processed hits in the second number of bit arrays corresponding to the verification data, it is determined that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type. This solution obtains multiple bit arrays corresponding to the data to be processed by performing approximate processing on the data to be processed, and then makes a judgment based on the multiple bit data and the verification data of the preset type to determine the set type to which the data to be processed belongs, and obtains the screening result corresponding to the data to be processed. This overcomes the problem that the traditional approximate query Bloom algorithm does not support the variable approximate range of the query data, implements the variable-length fuzzy query method, obtains accurate screening results, and improves the efficiency, accuracy and applicability of the verification of the data to be processed.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1 is a flow chart of a data screening method provided by Embodiment 1 of the present invention;

[0029] Figure 2 is a flow chart of a data screening method provided by Embodiment 2 of the present invention;

[0030] Figure 3 is a structural schematic diagram of a data screening device provided in Embodiment 3 of the present invention;

[0031] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the data screening method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1This is a flow chart of a data screening method provided by the first embodiment of the present invention. This embodiment is applicable to data screening. The method can be performed by a data screening device. The data screening device can be implemented in the form of hardware and / or software. The data screening device can be configured in electronic devices such as computers, servers and data processors. Figure 1 As shown, the method includes:

[0036] S110: Obtain data to be processed.

[0037] The data to be processed can be specifically understood as data to be screened, which can be data generated during business processing in a business system or data in a data transmission line. Data generated during business processing in a business system can be collected in real time, and the collected business-related data can be used as data to be processed, or data to be screened can be read from a target storage device as data to be processed. For example, the business system can be a banking business system, or other systems with financial business, or a system involving sensitive information during business processing, which is not limited here.

[0038] Specifically, data is collected from the business system in real time or at regular intervals, and the business data generated in the process of business processing by the business system is used as the data to be processed. All the read business data can be used as the data to be processed, or the business data can be matched according to the pre-set keyword information. The business data that successfully matches the keyword is used as the data to be processed, which can effectively reduce the amount of data to be screened and improve the screening efficiency of the data to be processed. The keyword information can be specifically characterized as the field information of the information that needs to be paid attention to. The information that needs to be paid attention to includes but is not limited to customer information, transaction information and message information. The information that needs to be paid attention to can be extracted in advance according to the relevant information regulations and business operation data, and set as keyword information. The keyword information can also be adjusted according to the actual data screening needs.

[0039] Optionally, obtaining the data to be processed includes: obtaining information to be processed and identifying the data type of the information to be processed; if the data type is a digital type, using the information to be processed as the data to be processed; if the data type is a non-digital type, performing data conversion on the information to be processed to obtain data of a digital type corresponding to the information to be processed, and using the data of the digital type corresponding to the information to be processed as the data to be processed.

[0040] It should be noted that the data that needs to be screened often includes data of multiple data types, including but not limited to digital types and character types. For data screening and processing, the screening rate of digital data is the highest. In order to quickly screen data, the data to be processed should be digital data. Specifically, when the information to be processed is obtained, it is necessary to identify the data type of the information to be processed, where the information to be processed is the original data of the business system-related data obtained; if the information to be processed is identified as digital data, the information to be processed can be directly used as the data to be processed. If the data type of the information to be processed is identified as non-digital type, the data conversion algorithm is called to convert the information to be processed to obtain the digital data corresponding to the information to be processed, and the digital data corresponding to the information to be processed is used as the data to be processed.

[0041] S120. Based on the first number of hash function families, hash processes are performed on the data to be processed respectively to obtain a first number of hash data sets of the data to be processed, and mapping position information of each hash data set on the bit array block is determined to obtain a first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different.

[0042] Among them, the first number specifically represents the number of hash function families, which are used to determine the approximation degree of the fuzzy query. The larger the first number is, the larger the approximation range is, and the more fuzzy the approximate query is. It can be set according to actual needs. Preferably, the first number should be less than or equal to the number of hash function families corresponding to the verification data, wherein the verification data is data used to verify the data to be processed, and is pre-determined and stored as different data under different setting types. The hash function family specifically refers to a hash data set used to perform hash processing on the data to be processed to obtain multiple hash data sets with different approximations of the data to be processed. Each hash function family includes multiple hash functions, and the multiple hash functions are different hash functions. It should be noted that although the multiple hash functions are different hash functions, the multiple hash functions should belong to the same type of function. The type of hash function can be selected according to actual needs, including but not limited to locality sensitive hash functions and distance sensitive hash functions. The corresponding hash function family is composed of multiple hash functions. Exemplarily, the corresponding locality sensitive hash function family can be composed of multiple locality sensitive hash functions, and the corresponding distance sensitive hash function family can be composed of multiple distance sensitive hash functions. In this embodiment, the selected hash function is a locality sensitive hash function. A bit array block is a data structure used to efficiently utilize memory space to store a large amount of binary data. The data to be processed can be hashed through a hash function to determine multiple mapping position information corresponding to the data to be processed, wherein the mapping position information can be specifically understood as the position information of each hash data set mapped in the corresponding bit array block. The hash data sets can be processed separately through multiple hash functions to obtain the corresponding mapping position information.

[0043] Specifically, an editing function of the first quantity can be set on the visual interface, and the user can edit the first quantity on the visual interface. If the edited first quantity is greater than or equal to the number of hash function families corresponding to the verification data, the number of hash function families corresponding to the verification data is used as the first quantity. If the edited first quantity is less than the number of hash function families corresponding to the verification data, the number edited by the user is used as the first quantity. After obtaining the data to be processed, the hash function families that meet the first quantity are obtained from the preset storage space, and the data to be processed are hashed by the first number of hash function families to obtain hash data sets corresponding to each hash function family, thereby obtaining the first number of hash data sets of the data to be processed, and each hash data set is taken as a whole, and each hash data set is processed separately by a random number of multiple hash functions to obtain mapping position information of each hash data set on the bit array block, and the mapping position information of the first quantity corresponding to the data to be processed is obtained; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different.

[0044] In this embodiment, multiple hash function families are used to process the data to be processed to obtain corresponding hash data sets, and then the hash data sets are mapped into bit arrays for subsequent fuzzy queries. The hash function families can also be adjusted according to actual conditions to meet the needs of fuzzy queries on different data, thereby improving the flexibility and applicability of data screening.

[0045] Optionally, the hash function family includes a locality-sensitive hash function family; based on a first number of hash function families, hash processing is performed on the data to be processed respectively to obtain a first number of hash data sets of the data to be processed, including: determining a first number of locality-sensitive hash function families; for each locality-sensitive hash function family, processing the data to be processed respectively based on multiple locality-sensitive hash functions in the locality-sensitive hash function family to obtain multiple hash data of the data to be processed, and the multiple hash data form a hash data set of the data to be processed; based on the hash data set corresponding to each obtained locality-sensitive hash function family, the first number of hash data sets of the data to be processed are formed.

[0046] In this embodiment, the hash function family includes a locality-sensitive hash function family, that is, each hash function in the hash function family is a locality-sensitive hash function. In the process of screening the data to be processed, the locality-sensitive hash function family that meets the first quantity can be read from the target storage space. It should be noted that the hash function family used in the data screening process should be the same as the hash function family used when determining the verification data, but the number of hash function families used can be different. The number of hash function families used in the screening process is less than or equal to the number of hash function families used in the process of determining the verification data, wherein the relevant information of the hash function family used in the process of verifying the data can be pre-stored in the target storage space.

[0047] Specifically, a local sensitive hash function family that satisfies a first quantity is obtained from a preset storage space, and the data to be processed is processed by each local sensitive hash function family, that is, the data to be processed is processed by each local sensitive hash function in each local sensitive hash function family, and the hash value corresponding to each local sensitive hash function is determined, thereby obtaining multiple hash data sets corresponding to the data to be processed, and a hash data set of the data to be processed is formed from the multiple hash data, and the hash data sets corresponding to each obtained local sensitive hash function family form the first number of hash data sets of the data to be processed.

[0048] Optionally, determining a first number of local sensitive hash function families includes: obtaining an initial local sensitive hash function, wherein the initial local sensitive hash function includes adjustable parameters, and the adjustable parameters include a random vector parameter, a random number parameter, and a hash bucket width parameter; constructing a local sensitive hash function that satisfies a third number by performing a plurality of different adjustments to the random vector parameter and / or the random number corresponding to the initial local sensitive hash function, and the third number of local sensitive hash functions forms a first local sensitive hash function family; performing a plurality of different adjustments to the hash bucket width parameters of the local sensitive hash functions in the first local sensitive hash function family to obtain a local sensitive hash function family that satisfies the first number.

[0049] Among them, the initial local sensitive hash function is an initial function that does not assign values ​​to the adjustable parameters of the local sensitive hash function. The adjustable parameters in the local sensitive hash function can be adjusted according to a preset parameter adjustment method to obtain multiple different local sensitive hash functions. The adjustable parameters include but are not limited to random vector parameters, random number parameters and hash bucket width parameters.

[0050] Specifically, the initial local sensitivity function obtained is: Wherein, v is the input data, i.e., the data to be processed; c is the random vector parameter, which is a random vector with the same dimension as the input data. Each component in c is obtained from the Gaussian distribution according to the data to be processed; w is the hash bucket width, and d is a random number with a value range of 0≤d≤w; The random vector parameter and / or random number in the adjustable parameter of the local sensitive hash function is adjusted by a preset parameter adjustment method, and the number of adjustments needs to meet the third number of adjustment times, and the third number of local sensitive hash functions is obtained, and the function family formed by the third number of local sensitive hash functions is called the first local sensitive hash function family; further, the hash bucket width parameter of the local hash function in the first local sensitive hash function family is adjusted multiple times, and the number of adjustments needs to meet the first number, and the hash bucket width value obtained after multiple adjustments is set for each local sensitive hash function family respectively, so that the hash bucket width parameter corresponding to each local sensitive hash function family is different from the hash bucket width parameter corresponding to other local sensitive hash function families, and the hash bucket width parameters of each local sensitive hash function in each local sensitive hash function family are the same. Preferably, for the hash bucket width parameter in each local sensitive hash function family, the hash bucket width parameter of the previous local sensitive hash function family is increased in sequence. It should be noted that the increase in the hash bucket width parameter means that the probability of similar data falling into the same bucket is greater, and the approximation degree is looser. By controlling and adjusting the hash bucket width of each hash function family, the approximate range can be enlarged or reduced, so that the approximate range of the query can be variable, thereby improving the flexibility and applicability of data screening.

[0051] S130. Determine the mapping position information of the first quantity of data to be processed corresponding to the second quantity of bit arrays corresponding to the verification data of the set type to obtain a determination result, wherein the second quantity of bit arrays corresponding to the verification data is obtained by mapping the hash data set obtained by processing the verification data based on the second quantity of hash function families, and the first quantity is less than or equal to the second quantity.

[0052] Among them, the set type can be specifically understood as the classification type set for screening data, and the classification type includes but is not limited to sensitive user type and suspicious transaction type, which can be set according to actual conditions. The verification data of the set type can be specifically understood as data used to determine whether the data to be processed belongs to the set type. The verification data of the set type includes one or more data, and the verification data can be processed in advance according to the second number of hash function families to obtain the hash data set corresponding to the verification data, and the second number of bit arrays can be mapped and stored. It should be noted that in the process of determining the bit array of the verification data of the set type, it is necessary to determine as many corresponding bit arrays as possible. In the process of screening applications, it is not necessary to determine too many bit arrays corresponding to the data to be processed. Therefore, the first number is less than or equal to the second number. In the process of data screening, it can be directly matched with the bit array of the verification data. For the storage of the verification data of the set type, the second number of bit arrays corresponding to the verification data of the set type are stored, rather than the storage data itself. The mapping and storage of the verification data can be realized through the hash function and multiple bit array blocks. The required space consumption is only the space required for storing the bit array blocks, which can reduce the consumption of storage space and save storage resources.

[0053] Specifically, a second number of bit arrays corresponding to the set type of verification data is obtained from the target storage space, and the mapping position information of the first number of data to be processed is matched respectively according to the second number of bit arrays corresponding to the set type of verification data, and it is determined whether the value corresponding to the position information matching the first number of mapping position information corresponding to the data to be processed in the second number of bit arrays corresponding to the verification data satisfies the preset value, thereby obtaining a determination result, wherein the second number of bit arrays corresponding to the verification data is obtained by mapping the hash data set obtained by processing the verification data based on a second number of hash function families, and the first number is less than or equal to the second number.

[0054] In this embodiment, the mapping position information of the first number of data to be processed corresponding to the second number of bit arrays corresponding to the verification data of the set type is judged to obtain a judgment result, thereby realizing fuzzy query of the data to be processed in the verification data, and thus efficiently screening the data to be processed.

[0055] Optionally, there is at least one verification data of the set type; based on the second number of bit arrays corresponding to the verification data of the set type, the mapping position information of the first number of data to be processed is judged, including: traversing at least one verification data of the set type, and for the second number of bit arrays corresponding to each verification data, determining the value corresponding to the position information that matches any mapping position information in the first number of mapping position information corresponding to the data to be processed; if the values ​​corresponding to the matched position information are all preset values, then determining that the result of the judgment is that the first number of mapping position information corresponding to the data to be processed hits the second number of bit arrays corresponding to the verification data; if any of the values ​​corresponding to the matched position information is not a preset value, then determining that the result of the judgment is that the first number of mapping position information corresponding to the data to be processed does not hit the second number of bit arrays corresponding to the verification data.

[0056] Specifically, there is at least one verification data of a set type, and the set type used for data screening can be predetermined. At least one verification data of the set type is traversed to obtain a second number of bit arrays corresponding to each verification data, determine the position information that matches any mapping position information in the first number of mapping position information corresponding to the data to be processed, and extract the numerical value corresponding to each group of matching position information. If the numerical values ​​corresponding to any group of matching position information extracted are all preset values, then the judgment result is determined to be that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data; if any numerical value corresponding to any group of matching position information extracted exists that is not a preset value, then the judgment result is determined to be that the first number of mapping position information corresponding to the data to be processed does not hit in the second number of bit arrays corresponding to the verification data, wherein the preset value is specifically a value set for marking the mapping position in the bit array corresponding to the verification data. Exemplarily, the preset value can be set to 1.

[0057] S140. When the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data, determine that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type.

[0058] Specifically, after obtaining the determination result, it is determined whether the data to be processed belongs to the set type according to the determination result. If the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data, then the screening result is determined to be that the data to be processed belongs to the set type. If the determination result is that the first number of mapping position information corresponding to the data to be processed does not hit in the second number of bit arrays corresponding to the verification data, then the screening result is determined to be that the data to be processed does not belong to the set type.

[0059] The technical solution of this embodiment is as follows: obtaining the data to be processed; performing hash processing on the data to be processed based on a first number of hash function families respectively to obtain a first number of hash data sets of the data to be processed, and determining the mapping position information of each hash data set on the bit array block respectively to obtain the first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different; judging the first number of mapping position information corresponding to the data to be processed based on a second number of bit arrays corresponding to verification data of a set type to obtain a judgment result, wherein the second number of bit arrays corresponding to the verification data are obtained by mapping the hash data sets obtained by processing the verification data based on the second number of hash function families, and the first number is less than or equal to the second number; when the judgment result is that the mapping position information of the first number of the data to be processed hits in the second number of bit arrays corresponding to the verification data, it is determined that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type. This solution obtains multiple bit arrays corresponding to the data to be processed by performing approximate processing on the data to be processed, and then makes a judgment based on the multiple bit data and the verification data of the preset type to determine the set type to which the data to be processed belongs, and obtains the screening result corresponding to the data to be processed. This overcomes the problem that the traditional approximate query Bloom algorithm does not support the variable approximate range of the query data, implements the variable-length fuzzy query method, obtains accurate screening results, and improves the efficiency, accuracy and applicability of the verification of the data to be processed.

[0060] Embodiment 2

[0061] Figure 2 It is a flow chart of a data screening method provided in Embodiment 2 of the present invention. The method of this embodiment is a further optimization of the method of the above-mentioned embodiment. Optionally, a fourth number of random hash functions are obtained; for each hash data set, the hash data set is processed based on the fourth number of random hash functions to determine the mapping position information of the hash data set on the bit array block.

[0062] like Figure 2 As shown, the method includes:

[0063] S210: Obtain data to be processed.

[0064] S220 , performing hash processing on the data to be processed based on the first number of hash function families respectively, to obtain a first number of hash data sets of the data to be processed.

[0065] S230: Obtain a fourth number of random hash functions.

[0066] The fourth number represents the number of mapping positions of each hash data set in the corresponding bit array block of the preset length. The calculation formula of the fourth number k and the preset length m is as follows:

[0067]

[0068]

[0069] Wherein, n is the number of elements of the data to be processed, p is the preset false positive rate, and p can be set according to actual needs. It should be noted that the fourth number of random hash functions obtained here should be consistent with the random hash function used when determining the second number of bit arrays corresponding to the verification data.

[0070] Obtain a random hash function that satisfies the fourth quantity, where the random hash function includes but is not limited to MurmurHash ((multiply and rotate)and (multiply and rotate)Hash, a hash algorithm of multiplication and rotation) and MD5 (Message Digest Algorithm 5).

[0071] S240: For each hash data set, process the hash data set based on a fourth number of random hash functions to determine mapping position information of the hash data set on the bit array block.

[0072] Specifically, for each hash data set, hash processing is performed on the hash data set respectively through a fourth number of random functions to determine the position information of each hash data set mapped on the corresponding bit array block.

[0073] Optionally, the hash data set is processed based on a fourth number of random hash functions to determine mapping position information of the hash data set on the bit array block, including: hashing the hash data sets based on the fourth number of random hash functions to obtain multiple hash values ​​corresponding to the hash data sets; for each hash value, performing a modulus operation on the length of the bit array block based on the hash value to determine the mapping position information of the hash value.

[0074] Specifically, for each hash data set, each hash data set is taken as a whole; the hash data sets are hashed respectively through a fourth number of random hash functions to obtain multiple hash values ​​corresponding to the hash data sets, and for each hash value, the result of a modulo operation between each hash value and the length of the bit array block is calculated, and the result is a mapping position of the corresponding hash value, and the mapping position information corresponding to each hash data set is constructed according to the mapping position corresponding to each hash value.

[0075] S250. Determine the mapping position information of the first quantity of data to be processed corresponding to the second quantity of bit arrays corresponding to the verification data of the set type to obtain a determination result, wherein the second quantity of bit arrays corresponding to the verification data is obtained by mapping the hash data set obtained by processing the verification data based on the second quantity of hash function families, and the first quantity is less than or equal to the second quantity.

[0076] S260. When the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data, determine that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type.

[0077] The technical solution of this embodiment is to obtain data to be processed; perform hash processing on the data to be processed based on a first number of hash function families respectively to obtain a first number of hash data sets of the data to be processed; obtain a fourth number of random hash functions; for each hash data set, process the hash data set based on the fourth number of random hash functions to determine the mapping position information of the hash data set on the bit array block; determine the mapping position information of the first number of data to be processed based on a second number of bit arrays corresponding to verification data of a set type to obtain a determination result, wherein the second number of bit arrays corresponding to the verification data are obtained by mapping the hash data sets obtained by processing the verification data based on the second number of hash function families, and the first number is less than or equal to the second number; when the determination result is that the mapping position information of the first number of data to be processed hits in the second number of bit arrays corresponding to the verification data, it is determined that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type. This solution obtains multiple bit arrays corresponding to the data to be processed by performing approximate processing on the data to be processed, and then makes a judgment based on the multiple bit data and the verification data of the preset type to determine the set type to which the data to be processed belongs, and obtains the screening result corresponding to the data to be processed. This overcomes the problem that the traditional approximate query Bloom algorithm does not support the variable approximate range of the query data, implements the variable-length fuzzy query method, obtains accurate screening results, and improves the efficiency, accuracy and applicability of the verification of the data to be processed.

[0078] Embodiment 3

[0079] Figure 3 Schematic diagram of the structure of a data screening device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0080] The to-be-processed data acquisition module 310 is used to acquire the to-be-processed data;

[0081] A mapping position information determination module 320 is used to perform hash processing on the data to be processed based on the first number of hash function families, respectively, to obtain a first number of hash data sets of the data to be processed, and to determine the mapping position information of each hash data set on the bit array block, respectively, to obtain the first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different;

[0082] A determination result determination module 330 is used to determine the mapping position information of the first number of the data to be processed based on the second number of bit arrays corresponding to the verification data of the set type, and obtain a determination result, wherein the second number of bit arrays corresponding to the verification data are obtained by mapping the hash data set obtained by processing the verification data based on the second number of hash function families, and the first number is less than or equal to the second number;

[0083] The screening result determination module 340 is used to determine that the screening result corresponding to the data to be processed is that the data to be processed belongs to a set type when the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data.

[0084] The technical solution of this embodiment is to obtain the data to be processed through the data to be processed acquisition module; the mapping position information determination module performs hash processing on the data to be processed based on the first number of hash function families respectively to obtain the first number of hash data sets of the data to be processed, and determines the mapping position information of each hash data set on the bit array block respectively to obtain the first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different; the judgment result determination module judges the first number of mapping position information corresponding to the data to be processed based on the second number of bit arrays corresponding to the verification data of the set type to obtain a judgment result, wherein the second number of bit arrays corresponding to the verification data are mapped to the hash data sets obtained by processing the verification data based on the second number of hash function families, and the first number is less than or equal to the second number; the screening result determination module determines that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type when the judgment result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data. This solution obtains multiple bit arrays corresponding to the data to be processed by performing approximate processing on the data to be processed, and then makes a judgment based on the multiple bit data and the verification data of the preset type to determine the set type to which the data to be processed belongs, and obtains the screening result corresponding to the data to be processed. This overcomes the problem that the traditional approximate query Bloom algorithm does not support the variable approximate range of the query data, implements the variable-length fuzzy query method, obtains accurate screening results, and improves the efficiency, accuracy and applicability of the verification of the data to be processed.

[0085] On the basis of the above embodiment, the hash function family includes a local sensitive hash function family; optionally, the mapping position information determination module 320 includes a hash data set determination unit, the hash data set determination unit is used to determine a first number of local sensitive hash function families; for each local sensitive hash function family, based on multiple local sensitive hash functions in the local sensitive hash function family, the data to be processed are processed respectively to obtain multiple hash data of the data to be processed, and the multiple hash data form a hash data set of the data to be processed; based on the hash data set corresponding to each local sensitive hash function family obtained, a first number of hash data sets of the data to be processed are formed. The hash data set determination unit is specifically used to obtain an initial local sensitive hash function, wherein the initial local sensitive hash function includes adjustable parameters, and the adjustable parameters include random vector parameters, random number parameters and hash bucket width parameters; by performing multiple different adjustments on the random vector parameters and / or random numbers corresponding to the initial local sensitive hash function, a local sensitive hash function that meets the third number is constructed, and the third number of local sensitive hash functions forms a first local sensitive hash function family; the hash bucket width parameters of the local sensitive hash functions in the first local sensitive hash function family are adjusted in multiple different ways to obtain a local sensitive hash function family that meets the first number.

[0086] Optionally, the mapping position information determination module 320 includes a mapping position information determination unit, which is used to obtain a fourth number of random hash functions; for each hash data set, process the hash data set based on the fourth number of random hash functions to determine the mapping position information of the hash data set on the bit array block. The mapping position information determination unit is specifically used to perform hash processing on the hash data set based on the fourth number of random hash functions to obtain multiple hash values ​​corresponding to the hash data set; for each hash value, perform a modulus operation on the length of the bit array block based on the hash value to determine the mapping position information of the hash value.

[0087] Optionally, there is at least one verification data of a set type; the judgment result determination module 330 is specifically used to traverse at least one verification data of a set type, and for the second number of bit arrays corresponding to each verification data, determine the value corresponding to the position information that matches any mapping position information in the first number of mapping position information corresponding to the data to be processed; if the values ​​corresponding to the matched position information are all preset values, then it is determined that the judgment result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data; if there is any value in the values ​​corresponding to the matched position information that is not a preset value, then it is determined that the judgment result is that the first number of mapping position information corresponding to the data to be processed does not hit in the second number of bit arrays corresponding to the verification data.

[0088] Optionally, the module 310 for acquiring data to be processed is specifically used to acquire information to be processed and identify the data type of the information to be processed; if the data type is a digital type, the information to be processed is used as the data to be processed; if the data type is a non-digital type, data conversion is performed on the information to be processed to obtain data of the digital type corresponding to the information to be processed, and the data of the digital type corresponding to the information to be processed is used as the data to be processed.

[0089] The data screening device provided in the embodiment of the present invention can execute the data screening method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0090] Embodiment 4

[0091] Figure 4 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0092] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0093] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0094] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data screening method.

[0095] In some embodiments, the data screening method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data screening method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data screening method in any other suitable manner (e.g., by means of firmware).

[0096] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] The computer programs for implementing the data screening methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0098] Embodiment 5

[0099] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a data screening method, the method comprising:

[0100] Get the data to be processed;

[0101] Based on the first number of hash function families, hash processing is performed on the data to be processed respectively to obtain the first number of hash data sets of the data to be processed, and mapping position information of each hash data set on the bit array block is determined to obtain the first number of mapping position information corresponding to the data to be processed; wherein each hash function family includes multiple hash functions, and the multiple hash functions in each hash function family are different;

[0102] Determine the mapping position information of the first quantity corresponding to the to-be-processed data based on the second quantity bit array corresponding to the verification data of the set type, and obtain a determination result, wherein the second quantity bit array corresponding to the verification data is obtained by mapping the hash data set obtained by processing the verification data based on the second quantity hash function family, and the first quantity is less than or equal to the second quantity;

[0103] When the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data, the screening result corresponding to the data to be processed is determined to be that the data to be processed belongs to the set type.

[0104] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0106] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0107] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0109] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data screening method, characterized in that: include: Get the data to be processed; Based on the first number of hash function families, the data to be processed are hashed respectively to obtain the first number of hash data sets of the data to be processed, and the mapping position information of each of the hash data sets on the bit array block is determined to obtain the first number of mapping position information corresponding to the data to be processed; wherein each of the hash function families includes a plurality of hash functions, and the plurality of hash functions in each of the hash function families are different; Determine the mapping position information of the first quantity corresponding to the data to be processed based on the second quantity of bit arrays corresponding to the verification data of the set type, and obtain a determination result, wherein the second quantity of bit arrays corresponding to the verification data are obtained by mapping the hash data set obtained by processing the verification data based on the second quantity of hash function families, and the first quantity is less than or equal to the second quantity; When the determination result is that the first number of mapping position information corresponding to the data to be processed hits the second number of bit arrays corresponding to the verification data, the screening result corresponding to the data to be processed is determined to be that the data to be processed belongs to the set type.

2. The method according to claim 1, characterized in that: The hash function family includes a locality-sensitive hash function family; The step of performing hash processing on the data to be processed based on the first number of hash function families to obtain a first number of hash data sets of the data to be processed includes: determining said first number of locality sensitive hash function families; For each of the locality-sensitive hash function families, the data to be processed are processed based on a plurality of locality-sensitive hash functions in the locality-sensitive hash function family to obtain a plurality of hash data of the data to be processed, wherein the plurality of hash data form a hash data set of the data to be processed; A first number of hash data sets of the data to be processed are formed based on the obtained hash data sets corresponding to each of the locality sensitive hash function families.

3. The method according to claim 2, characterized in that The determining the first number of locality sensitive hash function families includes: Obtaining an initial locality sensitive hash function, wherein the initial locality sensitive hash function includes adjustable parameters, and the adjustable parameters include a random vector parameter, a random number parameter, and a hash bucket width parameter; By performing a plurality of different adjustments on the random vector parameter and / or the random number corresponding to the initial local sensitive hash function, a local sensitive hash function satisfying a third number is constructed, wherein the third number of local sensitive hash functions forms a first local sensitive hash function family; A hash bucket width parameter of a local sensitive hash function in the first local sensitive hash function family is adjusted in a variety of different ways to obtain a local sensitive hash function family that satisfies the first quantity.

4. The method according to claim 1, characterized in that: The determining of mapping position information of each of the hash data sets on the bit array blocks includes: obtaining a fourth number of random hash functions; For each of the hash data sets, the hash data set is processed based on the fourth number of random hash functions to determine mapping position information of the hash data set on the bit array block.

5. The method according to claim 4, characterized in that The processing of the hash data set based on the fourth number of random hash functions to determine mapping position information of the hash data set on a bit array block includes: Performing hash processing on the hash data sets respectively based on the fourth number of random hash functions to obtain multiple hash values ​​corresponding to the hash data sets; For each of the hash values, a modulo operation is performed on the length of the bit array block based on the hash value to determine mapping position information of the hash value.

6. The method according to claim 1, characterized in that The verification data of the set type is at least one; The determining of the first amount of mapping position information corresponding to the data to be processed based on the second amount of bit arrays corresponding to the verification data of the set type includes: Traversing at least one of the verification data of the set type, and for each second number of bit arrays corresponding to the verification data, determining a value corresponding to position information that matches any mapping position information in the first number of mapping position information corresponding to the data to be processed; If the values ​​corresponding to the matched position information are all preset values, determining that the determination result is that the first number of mapped position information corresponding to the to-be-processed data hits the second number of bit arrays corresponding to the verification data; If any of the values ​​corresponding to the matched position information is not a preset value, it is determined that the determination result is that the first number of mapping position information corresponding to the data to be processed does not hit the second number of bit arrays corresponding to the verification data.

7. The method according to claim 1, characterized in that The obtaining of the data to be processed comprises: Acquire information to be processed, and identify the data type of the information to be processed; If the data type is a digital type, the information to be processed is used as data to be processed; If the data type is a non-numeric type, data conversion is performed on the information to be processed to obtain data of a numeric type corresponding to the information to be processed, and the data of the numeric type corresponding to the information to be processed is used as the data to be processed.

8. A data screening device, characterized in that: include: A module for acquiring data to be processed, used for acquiring data to be processed; A mapping position information determination module, configured to perform hash processing on the data to be processed based on a first number of hash function families, respectively, to obtain a first number of hash data sets of the data to be processed, and to determine mapping position information of each of the hash data sets on a bit array block, respectively, to obtain a first number of mapping position information corresponding to the data to be processed; wherein each of the hash function families includes a plurality of hash functions, and the plurality of hash functions in each of the hash function families are different; A determination result determination module, configured to determine the mapping position information of the first quantity corresponding to the to-be-processed data based on the second quantity of bit arrays corresponding to the verification data of a set type, to obtain a determination result, wherein the second quantity of bit arrays corresponding to the verification data is obtained by mapping the hash data set obtained by processing the verification data based on the second quantity of hash function families, and the first quantity is less than or equal to the second quantity; A screening result determination module is used to determine that the screening result corresponding to the data to be processed is that the data to be processed belongs to the set type when the determination result is that the first number of mapping position information corresponding to the data to be processed hits in the second number of bit arrays corresponding to the verification data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data screening method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the data screening method described in any one of claims 1 to 7 when executed by a processor.

Images