Blacklist query method, system, electronic device and storage medium

Abstract

The present disclosure provides a blacklist query method, which is applied to the financial technology field or other fields, including: constructing a BitMap array to store blacklist data; calculating a hash value according to the element to be queried, and obtaining a corresponding BitMap array address according to the hash value calculation; obtaining a first result corresponding to the corresponding BitMap array address through Redis, and judging whether the first result matches; if it matches, further comparing the element to be queried with the table item according to the database table of Redis to obtain a second result; if the second result also matches, the element to be queried is in the blacklist. The present disclosure also provides a blacklist query system, an electronic device, a computer-readable storage medium, and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of financial technology, and in particular to a blacklist query method, system, electronic device and storage medium. Background Art

[0002] Blacklist collisions are involved in all areas of the Internet. Through blacklist collisions of different dimensions, relevant permissions can be well controlled and system security can be protected. In the financial industry, blacklist collisions are also widely used. Through financial blacklist collisions of different dimensions, technical support can be provided in terms of protecting accounts and financial security. However, with the development of the financial industry, the dimensions and number of various lists have increased rapidly, and the development of the Internet has also placed higher and higher requirements on the efficiency of blacklist collision processing.

[0003] Existing conventional methods and their shortcomings:

[0004] Method 1: Use a relational database to maintain the list through a database table. Send SQL query statements to the database through the program to check whether the data exists in the database table to achieve the purpose of list collision. Disadvantages: In this way, each list collision requires connecting to the database to execute related queries. In the case of high concurrency, the database response efficiency is low, and the database will become the weak link of the entire system, or even crash. It cannot meet the high-concurrency usage scenarios well.

[0005] Method 2: Use local cache to import the list data maintained by the database into the local cache of the application when the application starts, synchronize the list data of the database to the local cache regularly, and directly perform list collision through the local cache. Disadvantages: In this method, in the same cluster, different nodes maintain a copy of the list data, which will consume a lot of memory space; at the same time, because different nodes need to update the list data regularly with relational data, the timeliness of the list and the consistency of the data are not high.

[0006] Method 3: Use an in-memory database to cache the list in the relational database. Generally, the list data is stored in strings. The full list data is written into the memory, and the list collision is performed through the in-memory database. Disadvantages: Each list is stored in the form of a string. As the number of lists increases, the memory consumption will increase sharply. The list dimensions that can be maintained in a single in-memory database cluster are limited, and the cost of use is high. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the above problems, the present disclosure provides a blacklist query method, system, electronic device and storage medium, which are used to at least partially solve the technical problems of low response efficiency and low timeliness of traditional query methods.

[0009] (II) Technical solution

[0010] On the one hand, the present disclosure provides a blacklist query method, including: constructing a BitMap array to store blacklist data; calculating a hash value of an element to be queried, and obtaining a corresponding BitMap array address according to the hash value calculation; obtaining a first result corresponding to the corresponding BitMap array address through Redis, and determining whether the first result matches; if they match, further comparing the element to be queried with a table item according to a database table of Redis to obtain a second result; if the second result also matches, the element to be queried is in the blacklist.

[0011] Furthermore, constructing a BitMap array to store blacklist data includes: determining the BitMap length and number of hashes required to store the blacklist data based on the amount of blacklist data and the estimated misjudgment rate; establishing an array using the BitMap length as the capacity of the array element; and storing the blacklist data in the array elements of the array.

[0012] Further, according to the amount of blacklist data and the estimated misjudgment rate, determining the BitMap length and the number of hashes required to store the blacklist data includes:

[0013] The BitMap length and the number of hashes are calculated according to the following formula:

[0014]

[0015] k=-log2p

[0016] Among them, m represents the length of the bitmap array, k represents the number of hashes, n represents the estimated number of list elements, and p represents the estimated misjudgment rate.

[0017] Further, calculating a hash value of the element to be queried, and obtaining a corresponding BitMap array address according to the hash value includes: calculating the hash value of the element to be queried according to a configured hash function and the number of hashes; and obtaining a corresponding BitMap array address by taking a modulus of the BitMap length according to the hash value.

[0018] Furthermore, obtaining the first result corresponding to the corresponding BitMap array address through Redis includes: obtaining the bit value corresponding to the corresponding BitMap array address through the array operation method provided by Redis; if the bit value is not all 1, the element to be queried is not in the blacklist; if the bit value is all 1, continue to compare.

[0019] Furthermore, comparing the element to be queried with the table item according to the Redis database table includes: querying in the Redis database table according to the unique identifier of the element to be queried; if the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

[0020] Furthermore, constructing a BitMap array to store blacklist data also includes updating a Redis database table, including: batch reading elements to be updated and creating a Redis database temporary table; bitmap processing the elements to be stored in turn and storing them in array elements of the BitMap array; updating the effective status of the elements to be stored in the Redis database temporary table; and replacing the original Redis database table with the updated Redis database temporary table.

[0021] Furthermore, bitmap processing is performed on the storage elements in sequence, and the storage is stored in the array elements of the BitMap array, including: bitmap processing is performed on the storage elements in sequence to obtain the corresponding BitMap array address; through the array operation method provided by Redis, the value of the bit corresponding to the corresponding BitMap array address is set to 1; and the corresponding effective status in the Redis database table is set to effective.

[0022] Furthermore, setting the corresponding effective state in the Redis database table to effective also includes retrying, including: reading an element in the Redis database table whose effective state is effective and whose corresponding bit value is 0; and re-performing a bitmap processing process on the element.

[0023] Furthermore, constructing a BitMap array to store blacklist data also includes deleting elements, including: setting the effective state corresponding to the to-be-deleted element in the Redis database table to invalid.

[0024] In another aspect, the present disclosure provides a blacklist query system, including: a construction module, used to construct a BitMap array to store blacklist data; a calculation module, used to calculate the hash value of the element to be queried, and obtain the corresponding BitMap array address according to the hash value calculation; a matching module, used to obtain the first result corresponding to the corresponding BitMap array address through Redis, and determine whether the first result matches; a comparison module, used to further compare the element to be queried with the table item according to the database table of Redis to obtain a second result if there is a match; if the second result also matches, the element to be queried is in the blacklist.

[0025] Another aspect of the present disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor executes the blacklist query method as described above.

[0026] Another aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the blacklist query method as described above is implemented.

[0027] Another aspect of the present disclosure provides a computer program product, including a computer program, which implements the aforementioned blacklist query method when executed by a processor.

[0028] (III) Beneficial effects

[0029] The present invention discloses a blacklist query method, system, electronic device and storage medium. The method adopts a Redis memory database to support a high concurrency and a short response time, and can quickly filter out a large number of blacklist elements that cannot be queried. The method adopts a binary BitMap data structure to significantly reduce memory consumption, and can support more blacklists and blacklist elements in a single cluster, saving the hardware cost of the system. The method also introduces a secondary comparison mechanism. After filtering out a large number of missed elements, a secondary comparison is performed through a database table, and blacklist queries can still be accurately performed under the characteristics of Redis's fast response and small BitMap memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram schematically shows an application scenario of the blacklist query method according to an embodiment of the present disclosure;

[0032] Figure 2 A flowchart of a blacklist query method according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 3 The basic principle diagram of Bitmap according to an embodiment of the present disclosure is schematically shown;

[0034] Figure 4 A flowchart of a method for constructing a BitMap array to store blacklist data according to an embodiment of the present disclosure is schematically shown;

[0035] Figure 5 A flowchart of a method for calculating a BitMap array address according to an embodiment of the present disclosure is schematically shown;

[0036] Figure 6 A flowchart of a method for updating a Redis database table in which a BitMap array is stored to store blacklist data according to an embodiment of the present disclosure is schematically shown;

[0037] Figure 7 A flowchart of a method for storing elements into array elements of a BitMap array according to an embodiment of the present disclosure is schematically shown;

[0038] Figure 8 A flowchart of the process of creating and remaking a list according to an embodiment of the present disclosure is schematically shown;

[0039] Fig. 9 A flowchart of processing a list collision according to an embodiment of the present disclosure is schematically shown;

[0040] Fig.10 The following schematically shows a process flow chart of adding list elements according to an embodiment of the present disclosure;

[0041] Fig.11 The process flow chart of deleting a list element according to an embodiment of the present disclosure is schematically shown;

[0042] Fig.12 A block diagram of a blacklist query system according to an embodiment of the present disclosure is schematically shown;

[0043] Fig.13 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0044] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0045] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0046] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0047] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0048] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some boxes or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or used in conjunction with an instruction execution system.

[0049] Based on the shortcomings of traditional blacklist query methods, the embodiments of the present invention provide a blacklist query method, system, electronic device and storage medium, which retain the advantages of different traditional methods and avoid some of their existing shortcomings; can provide higher concurrency, better timeliness, better data consistency, and at the same time can significantly reduce the memory consumption of the in-memory database. The list dimensions maintained in a single in-memory database can be more, reducing the construction cost of the system.

[0050] Figure 1 The exemplary system architecture 100 that can be applied to the blacklist query method according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0051] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0052] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, search applications, instant messaging tools, social platform software, etc. (only examples).

[0053] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0054] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0055] It should be noted that the blacklist query method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the system of the blacklist query method provided in the embodiment of the present disclosure can generally be set in the server 105. The blacklist query method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the system of the blacklist query method provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0056] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0057] Usually, we face a huge number of users and visits, such as millions or tens of millions of users, or tens of millions or even hundreds of millions of visits. When it reaches the hundreds of millions and will continue to grow rapidly in the future, it is difficult to support it with the traditional way of querying databases, and storage capacity and query efficiency are facing challenges. Therefore, we must choose a collection type that can count large amounts of data (such as hundreds of millions) very efficiently.

[0058] The present disclosure uses a BitMap array to store data. The BitMap array uses a bit to mark the value corresponding to an element, and the key is this element. Since data is stored in bits, storage space can be greatly saved. In the BitMap array, each array element is "0" or "1", indicating that the corresponding array element does not exist or exists. For example: on a 32-bit machine, an integer int a occupies 32 bits in the memory, and the corresponding 32 bits can be used to represent decimal numbers 0-31. The representation of the BitMap array in the memory can be a binary table consisting of a column of 0 and 1, such as 0101001, etc. When a bit in the BitMap array is occupied, the value of the bit is set to 1; when a bit is idle, the value of the bit is set to 0. The number of each bit is the offset of the bit in the BitMap array from the starting point.

[0059] Figure 2 The flowchart of the blacklist query method according to the embodiment of the present disclosure is schematically shown.

[0060] like Figure 2 As shown, the blacklist query method 200 may include operations S210 to S240.

[0061] In operation S210, a BitMap array is constructed to store blacklist data.

[0062] The underlying structure of Bitmap is a binary array. Figure 3 This is the basic principle diagram of Bitmap. Each bit stores binary '0' or '1'. When initializing, a fixed-length binary array must be determined and all data items must be assigned '0'. If a string is used to store blacklist data, the unique identifier corresponding to each element must be stored as an int, which occupies 4 bytes or 32 bits. A single identifier only occupies one bit in Bitmap, saving 32 times the memory. That is, the final Bitmap stores the bits corresponding to the element items, saving a lot of storage space.

[0063] In operation S220, a hash value of the element to be queried is calculated, and a corresponding BitMap array address is obtained according to the hash value calculation.

[0064] To determine whether an element is in a set, it is generally necessary to save all the elements and then determine them by comparison. Data structures such as linked lists and trees belong to this method. However, as the number of elements in the set increases, the storage space we need becomes larger and larger, and the retrieval speed becomes slower and slower. However, there is also a data structure called a hash table (also called a hash table, Hashtable), which can map an element to a point in a bit array through a hash function. In this way, as long as you determine whether this point is 1, you can know whether the element is in the set. That is, the hash table accesses records by mapping the key code value to a position in the table to speed up the search.

[0065] Through the specified hash function and the required number of calculations, multiple hash values ​​are calculated for the element, and the single position where the element item is stored in the array is obtained by taking the modulus of the Bitmap array length, and the assignment or matching operation is performed ('1' indicates that the current bit matches). Here, multiple hash calculations are used to determine whether the value exists, which greatly reduces the hash conflict situation.

[0066] In operation S230, a first result corresponding to a corresponding BitMap array address is obtained through Redis, and it is determined whether the first result matches.

[0067] Here, the bitmap record in redis is used to store the blacklist data. The underlying data structure of Bitmap uses the SDS data structure of the String type to save the bit array. Redis uses 8 bits of each byte array, and each bit represents the binary state of an element (either 0 or 1).

[0068] Bitmap provides GETBIT and SETBIT operations, which read and write the bit at the offset position of the bit array through an offset value offset. It should be noted that offset starts from 0. For example, when judging the user's login status, only one key = login_status is needed to store the user's login status collection data. The user ID is used as the offset, which is set to 1 if online and 0 if offline. GETBIT is used to determine whether the corresponding user is online. 500 million users only need 6MB of space. GETBIT obtains the value of the bit at the offset of the key value. When the key does not exist, it returns 0.

[0069] Through the bit array operation method GETBIT provided by Redis, read the binary value corresponding to the corresponding name unit array subscript, and use the binary value.

[0070] In operation S240, if there is a match, the element to be queried is further compared with the table item according to the Redis database table to obtain a second result; if the second result also matches, the element to be queried is in the blacklist.

[0071] Since there are theoretical conflicts between hash functions and hash values ​​and array length (the probability of conflict is calculated by the formula below, which is related to the array length and the number of hashes), accurate collisions need to be avoided through secondary collisions (this method uses a database table). Therefore, when judging matches based on the above binary values, it is not possible to completely determine whether the element to be queried is in the blacklist. For example, in the financial industry, when complete accuracy is required, secondary confirmation can be performed through the database table (the number of hits has been filtered by BitMap, and the remaining amount is small). If the element to be queried also matches the table entry, it is considered that the element to be queried is in the blacklist.

[0072] In this disclosure, Redis itself, as an in-memory database, can provide a high concurrency. At the same time, since the String type key of Redis is implemented by the SDS data structure, it is binary safe and supports the operation of the String type key as a bit array, providing basic support.

[0073] Figure 4 A flowchart of a method for constructing a BitMap array to store blacklist data according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 4 As shown, the method for constructing a BitMap array to store blacklist data includes:

[0075] In operation S211, the BitMap length and the number of hashing times required to store the blacklist data are determined according to the data volume of the blacklist data and the estimated misjudgment rate.

[0076] Although the use of multiple hash calculations to determine whether a value exists greatly reduces the possibility of hash conflicts, there are still certain defects, that is, it is easy to make misjudgments. If a number is judged to be non-existent by the Bloom filter, it means that the number really does not exist and there will be no misjudgment; if a number is judged to exist by the Bloom filter, it is possible to make a misjudgment at this time, and it may not exist. However, as long as the number of hash functions and the ratio between the bitmap size and the number of numbers to be stored are adjusted, the probability of such misjudgment can be reduced to a very low level.

[0077] The BitMap length and the number of hashes are calculated according to the following formula:

[0078]

[0079] k=-log2p

[0080] Among them, m represents the length of the BitMap array, k represents the number of hashes, n represents the estimated number of list elements, and p represents the estimated misjudgment rate.

[0081] In this disclosure, murmur3_128 is used as the basic hash function. The bitmap array is initialized in Redis by determining the length of the BitMap.

[0082] In operation S212, an array is created using the BitMap length as the capacity of the array element.

[0083] When the number of hash functions k is constant: the larger the length m of the BitMap array, the lower the misjudgment rate; the larger the number n of inserted elements, the higher the misjudgment rate. The length of the BitMap can be obtained based on the estimated misjudgment rate and the estimated number of elements.

[0084] In operation S213, the blacklist data is stored in an array element of the array.

[0085] After determining the length of the BitMap array, divide each array element in the BitMap array by its capacity, and set the subscript for the array element according to its offset position in the array. For example, if a bitmap with a length of 10 stores ten integers from 0 to 9, then the corresponding bit of each bitmap is 0. When the stored number is 3, the bitmap is 0. Figure 3 The position will become 1.

[0086] Figure 5 The flowchart schematically shows a method for calculating a hash value of an element to be queried and obtaining a corresponding BitMap array address according to the hash value according to an embodiment of the present disclosure.

[0087] like Figure 5 As shown, the method of calculating the hash value of the element to be queried and obtaining the corresponding BitMap array address according to the hash value includes:

[0088] In operation S221, a hash value of the element to be queried is calculated according to the configured hash function and the number of hashes.

[0089] First, calculate the hash value of the element to be queried. Given a table M, there exists a function f(key). For any given keyword value key, if the address of the record containing the keyword in the table can be obtained after substituting it into the function, then the table M is called a hash table and the function f(key) is called a hash function.

[0090] In an optional embodiment of the present disclosure, the preset hash algorithm is the murmur hash algorithm. Of course, the use of the murmur hash algorithm is only an example description, and those skilled in the art can flexibly adjust which hash algorithm to use according to needs. The use of any low collision rate and / or efficient hash algorithm to implement the calculation of the BitMap array address is within the scope of the protection concept of the present disclosure.

[0091] In operation S222, the BitMap length is modulo the hash value to obtain the corresponding BitMap array address.

[0092] According to the length of the bitmap binary array in the list, use hashcode%(length-1) to calculate the corresponding bitmap binary array index.

[0093] Based on the above embodiment, obtaining the first result corresponding to the corresponding BitMap array address through Redis includes: obtaining the value of the bit corresponding to the corresponding BitMap array address through the array operation method provided by Redis; if the value of the bit is not all 1, the element to be queried is not in the blacklist; if the value of the bit is all 1, continue to compare.

[0094] Through the getbit operation method provided by Redis, read the binary value corresponding to the corresponding name unit array index. If the value of the corresponding position is not all '1', it means that the element is not in the list and the list is not hit. The result is returned directly. If the bit array index is all '1', due to the bitmap structure, it cannot be completely determined that the element is in the list. In the case of financial industry, when complete accuracy is required, secondary confirmation can be performed through the database table.

[0095] Based on the above embodiment, comparing the element to be queried with the table item according to the Redis database table includes: querying in the Redis database table according to the unique identifier of the element to be queried; if the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

[0096] Table 1 is the database table structure used in this method.

[0097] Table 1

[0098]

[0099] Perform secondary confirmation through the database table, query the database table according to the "List Element Name" and "Effective Status" fields, and return the corresponding result to see whether the result of the secondary confirmation hits the list.

[0100] Figure 6 A flowchart of a method for updating a Redis database table in which a BitMap array is constructed to store blacklist data according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 6 As shown, the update method of the Redis database table includes:

[0102] In operation S601, elements to be updated are read in batches, and a temporary table of the Redis database is created.

[0103] In operation S602, BitMap processing is performed on the elements to be stored in sequence and stored in array elements of the BitMap array.

[0104] In operation S603, the validity status of the element to be stored in the temporary table of the Redis database is updated.

[0105] In operation S604, the updated Redis database temporary table is used to replace the original Redis database table.

[0106] In order to remove the invalid elements in the bitmap and keep the bitmap structure clean, a timer (for example, 7 days) is used to redo the list. During the redo process, a temporary bitmap is generated and then replaced with the original bitmap.

[0107] Figure 7 The flowchart schematically shows a method for sequentially performing BitMap processing on elements to be stored and storing them into array elements of a BitMap array according to an embodiment of the present disclosure.

[0108] like Figure 7 As shown, the method of performing BitMap processing on the elements to be stored in turn and storing them in the array elements of the BitMap array includes:

[0109] In operation S701, BitMap processing is performed on the elements to be stored in sequence to obtain corresponding BitMap array addresses.

[0110] In operation S702, the value of the bit corresponding to the corresponding BitMap array address is set to 1 through the array operation method provided by Redis.

[0111] In operation S703, the corresponding validity status in the Redis database table is set to valid.

[0112] First, the element to be stored is BitMapped to obtain the array address, and then the corresponding binary value under the array address is set to '1'; after the operation is successful, the "Bitmap Status" field of the database is updated to '1', indicating that it has been placed in the bitmap structure.

[0113] Based on the above embodiment, setting the corresponding effective state in the Redis database table to effective also includes retrying, including: reading the elements in the Redis database table whose effective state is effective and whose corresponding bit value is 0; and re-performing the BitMap processing process on the elements.

[0114] To prevent errors during the adding process, relevant retries are required. The background uses batch tasks to periodically read elements whose "effective status" field is '1' but whose "Bitmap status" field is '0', and re-execute the bitmap entry process.

[0115] On the basis of the above embodiment, constructing a BitMap array to store blacklist data also includes deleting elements, including: setting the effective state corresponding to the to-be-deleted element in the Redis database table to invalid.

[0116] In the list element deletion operation, the specified element cannot be deleted directly from the bitmap. The "effective status" field corresponding to the updated element is updated to '0' in the database table to change its setting to the invalid state.

[0117] The steps of this method are further described below with reference to a specific embodiment.

[0118] The present disclosure combines the Redis memory database, the bitmap data structure and introduces a secondary collision mechanism, which can support a higher concurrency, reduce memory consumption, and accurately perform list collision. The list collision in this embodiment is the aforementioned blacklist query, which achieves the purpose of list collision by checking whether the data exists in the database table.

[0119] 1. List creation process. Figure 8 The flowchart of list creation and redoing is shown. List creation and redoing step 800 includes: step 801, batch reading of valid list elements in MySQL table. Step 802, Redis creates a new temporary list name. Step 803, loop processing of a single list element, the processing process of a single list element includes steps 804 to 806. Step 804, multiple HashCode calculation of the list element, through the following formula: m represents the length of the bitmap array, k represents the number of hashes, n represents the estimated number of list elements, and p represents the estimated misjudgment rate;

[0120]

[0121] k=-log2p

[0122] The array length of the bitmap and the number of hashes can be determined. Murmur3_128 is used as the basic hash function in this method. Step 805, the element bitmap subscript index is calculated. Step 806, the temporary list bitmap index value is set to binary 1. Step 807, the MySQL table list batch status is updated, and the bitmap array is initialized in Redis by determining the bitmap length. Through the batch timing program, the list elements with "effective status" of '1' in the database table are read, and the bitmap processing of the list elements is completed in batches. Step 808, all the elements of the list are processed, and Redis renames the temporary bitmap list name to the original list name. In order to remove the invalid elements in the bitmap and keep the bitmap structure neat, the timing program (7 days) completes the redo of the list. During the redo process, a temporary bitmap is generated first, and the original bitmap is replaced after completion.

[0123] 2. List collision process. Fig. 9 The flowchart of list collision processing is shown. List collision step 900 includes: step 901, element multiple HashCode calculation; the added element is added according to the hash function configured in the list and the number of element hash calculations to obtain multiple hashcodes of the element. Step 902, element multiple bitmap subscript index calculation; then according to the length of the list bitmap binary array, use hashcode% (length-1) to calculate the bitmap binary array subscript. Step 903, element multiple bitmap subscript value acquisition, through the bit array operation method getbit provided by Redis, read the binary value corresponding to the corresponding name unit array subscript. Step 904, determine whether the binary value is all binary 1. Step 906, if the value of the corresponding position is not all '1', it means that the element is not in the list, the list is not hit, and the result is returned directly. Step 905, if the bit array subscripts are all '1', due to the bitmap structure, it is not possible to completely determine whether the element is in the list. In the case of the financial industry needing to fully guarantee accuracy, a second confirmation can be made through the database table (the number of hits has been bitmap filtered, and the remaining number is small). The database table is queried through the "List Element Name" and "Effective Status" fields to check whether the result of the second confirmation hits the list. Step 907, if the result of the second confirmation is yes, it means that the collision is successful and the element is in the list, and the corresponding result is returned.

[0124] 3. The process of adding list elements. Fig.10The flowchart of adding list elements is shown. Step 1000 of adding list elements includes: Step 1001, writing the list elements into MySQL. Step 1002, calculating the multiple hashcodes of the list elements; obtaining multiple hashcodes of the elements according to the hash function configured in the list and the number of element hash calculations. When an exception occurs in the list, the list bitmap update exception handling step 1003 is performed; Step 1004, MySQL queries the failed list items according to the status bit; Step 1005, looping and processing a single list element, and then entering the HashCode calculation process for adding list elements, and the subsequent steps are the same as the list element addition processing process. Step 1006, calculating the multiple bitmap subscript index of the element; then using the length of the list bitmap binary array length, using hashcode% (length-1) to calculate the multiple bitmap binary array subscripts. Step 1007, setting the list bitmap index value to binary 1; using the bit array operation method setbit provided by Redis, the corresponding binary value of the name unit array subscript is set to '1'. Step 1008, judging whether all are successful. Step 1009, after the operation is successful, the "Bitmap Status" field of the database is updated to '1', indicating that it has been put into the bitmap structure. To ensure that there are no errors in the adding process, relevant retries are required. The background uses batch tasks to periodically read the elements whose "Effective Status" field is '1' but whose "Bitmap Status" field is '0', and re-execute the bitmap entry process.

[0125] 4. List element deletion process. Fig.11 The process flow chart of list element deletion is shown. The list element deletion step 1100 includes: step 1101, updating the element MySQL table status to invalid; due to the data structure of the bitmap, the specified element cannot be directly deleted from the bitmap in the list element deletion operation, and the corresponding "effective status" field of the element is updated in the database table to the invalid status of '0'. When the element is verified twice by the database, the element status is invalid, and the purpose of deleting the list element is achieved.

[0126] The present invention combines multiple data structures and technologies to achieve the effect of combining the advantages of various methods, while avoiding various shortcomings of a single method, and has good practicality. The method has the following advantages:

[0127] Advantage 1: Low response time. Using the Redis memory database, it supports high concurrency, short response time, and can quickly filter out a large number of elements that cannot collide with the hit list, providing a better user experience.

[0128] Advantage 2: Reduce memory resource consumption of in-memory database. The binary bitmap data structure of the disclosed method can significantly reduce memory consumption, support more lists and more list elements in a single cluster, and save hardware costs between systems.

[0129] Advantage 3: Accurate list collision. By introducing a secondary collision mechanism, a large number of missed elements are filtered out and a secondary collision is performed through the database table, which can accurately perform list collision while retaining the characteristics of Redis's fast speed and small bitmap memory usage.

[0130] Fig.12 A block diagram of a blacklist query system according to an embodiment of the present disclosure is schematically shown.

[0131] like Fig.12 As shown, the blacklist query system 1200 includes: a construction module 1210 , a calculation module 1220 , a matching module 1230 , and a comparison module 1240 .

[0132] The construction module 1210 is used to construct a BitMap array to store blacklist data; according to an embodiment of the present disclosure, the construction module 1210 can be used to execute the above reference Figure 2 The described step S210 will not be repeated here.

[0133] The calculation module 1220 is used to calculate the hash value of the element to be queried, and obtain the corresponding BitMap array address according to the hash value. According to the embodiment of the present disclosure, the calculation module 1220 can be used to execute the above reference Figure 2 The described step S220 will not be repeated here.

[0134] The matching module 1230 is used to obtain the first result corresponding to the corresponding BitMap array address through Redis, and determine whether the first result matches; according to an embodiment of the present disclosure, the matching module 1230 can be used to execute the above reference Figure 2 The described step S230 will not be repeated here.

[0135] The comparison module 1240 is used to further compare the element to be queried with the table item according to the Redis database table to obtain a second result if a match is found; if the second result also matches, the element to be queried is in the blacklist; according to an embodiment of the present disclosure, the comparison module 1240 can be used to execute the above reference Figure 2 The described step S240 will not be repeated here.

[0136] It should be noted that any one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure, or at least part of the functions of any one of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0137] For example, any multiple of the building module 1210, the calculation module 1220, the matching module 1230, and the comparison module 1240 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the building module 1210, the calculation module 1220, the matching module 1230, and the comparison module 1240 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a suitable combination of any of them. Alternatively, at least one of the construction module 1210 , the calculation module 1220 , the matching module 1230 , and the comparison module 1240 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0138] The present invention provides a blacklist query system and method, which can be used in the fields of financial technology, and provides a combination of Redis memory database, bitmap data structure and secondary collision mechanism, which can support higher concurrency, reduce memory consumption, and accurately perform list collision.

[0139] Fig.13A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Fig.13 As shown, the electronic device 1300 described in this embodiment includes: a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 to a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include an onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0141] In RAM 1303, various programs and data required for the operation of electronic device 1300 are stored. Processor 1301, ROM 1302 and RAM 1303 are connected to each other via bus 1304. Processor 1301 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1302 and / or RAM 1303. It should be noted that the program can also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0142] According to an embodiment of the present disclosure, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to the bus 1304. The electronic device 1300 may further include one or more of the following components connected to the I / O interface 1305: an input portion 1306 including a keyboard, a mouse, etc.; an output portion 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1308 including a hard disk, etc.; and a communication portion 1309 including a network interface card such as a LAN card, a modem, etc. The communication portion 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that a computer program read therefrom is installed into the storage portion 1308 as needed.

[0143] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0144] The embodiments of the present disclosure also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the blacklist query method according to the embodiments of the present disclosure is implemented.

[0145] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1302 and / or RAM 1303 described above and / or one or more memories other than ROM 1302 and RAM 1303.

[0146] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the blacklist query method provided by the embodiment of the present disclosure.

[0147] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1301. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0148] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1309, and / or installed from the removable medium 1311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0149] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0150] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0151] It should be noted that the functional modules in the various embodiments of the present disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure may essentially or in other words, the part that contributes to the prior art or all or part of the technical solution may be embodied in the form of a software product.

[0152] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0154] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A blacklist query method, characterized in that: include: Construct a BitMap array to store blacklist data; Calculate the hash value of the element to be queried, and obtain the corresponding BitMap array address according to the hash value; Obtaining a first result corresponding to the corresponding BitMap array address through Redis, and determining whether the first result matches; If they match, the element to be queried is further compared with the table item according to the Redis database table to obtain a second result; if the second result also matches, the element to be queried is in the blacklist; The step of comparing the element to be queried with the table item according to the Redis database table includes: Query the Redis database table according to the unique identifier of the element to be queried; If the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

2. The blacklist query method according to claim 1, characterized in that: The construction of the BitMap array to store blacklist data includes: Determine the BitMap length and the number of hashes required to store the blacklist data according to the amount of the blacklist data and the estimated misjudgment rate; Create an array using the length of the BitMap as the capacity of the array element; The blacklist data is stored in an array element of the array.

3. The blacklist query method according to claim 2, characterized in that: Determining the BitMap length and the number of hashes required to store the blacklist data according to the amount of the blacklist data and the estimated misjudgment rate includes: The BitMap length and the number of hashes are calculated according to the following formula: k=-log2p Among them, m represents the length of the BitMap array, k represents the number of hashes, n represents the estimated number of list elements, and p represents the estimated misjudgment rate.

4. The blacklist query method according to claim 2, characterized in that: The step of calculating a hash value of the element to be queried and obtaining a corresponding BitMap array address according to the hash value comprises: Calculate the hash value of the element to be queried according to the configured hash function and the number of hash operations; The BitMap length is modulo the hash value to obtain the corresponding BitMap array address.

5. The blacklist query method according to claim 1, characterized in that: The first result corresponding to the corresponding BitMap array address obtained through Redis includes: Obtain the value of the bit corresponding to the corresponding BitMap array address through the array operation method provided by Redis; If the values ​​of the bits are not all 1, the element to be queried is not in the blacklist; If the values ​​of the bits are all 1, the comparison continues.

6. The blacklist query method according to claim 1, characterized in that: The construction of the BitMap array to store the blacklist data also includes updating the Redis database table, including: Batch read the elements to be updated and create a temporary table in the Redis database; Performing BitMap processing on the elements to be stored in turn, and storing them in the array elements of the BitMap array; Update the effective status of the element to be stored in the temporary table of the Redis database; The updated Redis database temporary table is used to replace the original Redis database table.

7. The blacklist query method according to claim 6, characterized in that: The sequentially performing BitMap processing on the elements to be stored and storing them in array elements of the BitMap array includes: Performing BitMap processing on the elements to be stored in sequence to obtain the corresponding BitMap array address; By using the array operation method provided by Redis, the value of the bit corresponding to the corresponding BitMap array address is set to 1; Set the corresponding validity status in the Redis database table to valid.

8. The blacklist query method according to claim 7, characterized in that: The setting of the corresponding effective status in the Redis database table to effective also includes retrying, including: Read the elements in the Redis database table whose effective status is effective and whose corresponding bit value is 0; The BitMap processing is performed again on the element.

9. The blacklist query method according to claim 7, characterized in that: The constructing of the BitMap array to store the blacklist data also includes deleting the elements, including: Set the effective status corresponding to the element to be deleted in the Redis database table to invalid.

10. A blacklist query system, characterized in that: include: Construction module, used to construct a BitMap array to store blacklist data; A calculation module, used to calculate a hash value of the element to be queried, and obtain a corresponding BitMap array address according to the hash value; A matching module, used to obtain a first result corresponding to the corresponding BitMap array address through Redis, and determine whether the first result matches; A comparison module is used to further compare the element to be queried with the table item according to the Redis database table to obtain a second result if there is a match; if the second result also matches, the element to be queried is in the blacklist; wherein, comparing the element to be queried with the table item according to the Redis database table includes: querying in the Redis database table according to the unique identifier of the element to be queried; if the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

11. An electronic device, comprising: processor; A memory storing a computer executable program, which, when executed by the processor, enables the processor to execute the blacklist query method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the blacklist query method as described in any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the blacklist query method according to any one of claims 1 to 9 is implemented.

Images