Method and device for checking data duplication

Abstract

The present application provides a method and device for checking for duplicate data, which is applied to the field of data processing technology to solve the problem that the space occupied, the efficiency of checking for duplicate data, and the accuracy of checking for duplicate data cannot be achieved simultaneously. The method specifically includes: obtaining the first directory information of the first shard to be stored in the distributed cache system, the first directory information is used to identify the first data to be stored in the database, and the first directory information includes at least one directory item; generating a first compressed bitmap according to the first directory information, the bitmap bits in the first compressed bitmap corresponding one-to-one to the directory items in the first directory information; determining the directory items in the first directory information that are repeated with the directory information already stored in the first shard according to the first compressed bitmap and the second compressed bitmap; wherein the bitmap bits in the second compressed bitmap correspond one-to-one to the directory items in the directory information already stored in the first shard; determining the data that is repeated with the data already stored in the database according to the repeated directory items.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for checking for duplicate data. Background Art

[0002] When maintaining a database, the same data may be entered repeatedly due to practical reasons, such as human error during entry or duplicate data entry due to information asymmetry. This duplicate data can adversely impact the normal operation of the system, including but not limited to system anomalies, statistical errors, and data display errors. To prevent these adverse effects, the data to be entered is typically checked for duplicates against existing data before entry. This prevents duplicate data from appearing in the database and causing the aforementioned adverse effects.

[0003] However, the currently commonly used methods of checking for duplicates have the problem of occupying space, failing to achieve both high efficiency and high accuracy in checking for duplicates. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for data duplication checking, which are used to solve the problem that space occupied, duplication checking efficiency and duplication checking accuracy cannot be achieved simultaneously during data duplication checking.

[0005] In a first aspect, a method for checking for duplicate data is provided, comprising: obtaining first directory information of a first shard to be stored in a distributed cache system, the first directory information being used to identify first data to be stored in a database, the first directory information comprising at least one directory entry, the data volume length of each directory entry in the at least one directory entry being M, where M is a positive integer; the distributed cache system being used to store directory information corresponding to the data stored in the database; generating a first compressed bitmap based on the first directory information, the bitmap bits in the first compressed bitmap corresponding one-to-one to the directory entries in the first directory information; determining, based on the first compressed bitmap and the second compressed bitmap, directory entries in the first directory information that are repeated with directory information already stored in the first shard; wherein the bitmap bits in the second compressed bitmap correspond one-to-one to directory entries in directory information already stored in the first shard; and determining, based on the repeated directory entries, that the first data is duplicated with data already stored in the database.

[0006] In this solution, data duplication checking is performed in a distributed cache system, reducing database access pressure. Furthermore, concurrent duplication checking through the distributed cache system improves duplication checking efficiency. Using compressed bitmap comparison for duplication checking not only saves directory information space in the distributed cache system, but also ensures both efficiency and accuracy. Using compressed bitmaps to record directory information effectively improves storage space utilization, while comparing compressed bitmaps for duplication checking reduces the time complexity of duplication checking through simple binary search and bitwise operations.

[0007] Optionally, the fragment number of the first fragment corresponds to the first S bits of all directory entries in the first directory information.

[0008] Through this method, the directory information of the data to be stored in the database can be stored in the distributed cache system in an orderly and dispersed manner, thereby improving the efficiency of data storage.

[0009] Optionally, each directory entry has a data length of 64 bits.

[0010] Through this method, the calculation result is a 64-bit hash value, which can perform duplicate checking operations on a larger number of first data, thereby improving the applicability of the duplicate checking method provided in this application.

[0011] Optionally, the first directory information includes a directory entry; generating a first compressed bitmap based on the first directory information includes: creating a first target container based on the first N bits of data of the directory entry, storing the last MN bits of the directory entry in the first target container, and using the first N bits of data of the directory entry to identify the first target container; or, the first directory information includes multiple directory entries; generating a first compressed bitmap based on the first directory information includes: creating the second target container based on the first N bits of the first directory entry, storing the last MN bits of other directory entries whose first N bits are the same as the first N bits of the first directory entry in the second target container, wherein the first directory entry is any one of the multiple directory entries.

[0012] In this manner, the first directory information is recorded in a manner of generating a compressed bitmap, which effectively reduces the storage space occupied when storing the directory information and improves the utilization rate of the storage space.

[0013] Optionally, an index is established based on the first N bits of data of the directory entry, wherein the index adopts an adaptive radix tree ART data structure, and the index is used to indicate the correspondence between the at least one directory entry and the container in the compressed bitmap; based on the first N bits of the second directory entry and the index, the target container corresponding to the second directory entry is searched; wherein the second directory entry is any one of the at least one directory entry; and the last MN bits of the second directory entry are stored in the target container.

[0014] In this method, indexing the containers in the compressed bitmap can effectively improve the query efficiency of the containers in the compressed bitmap. At the same time, using the ART tree data structure to establish the index can not only improve the query efficiency but also save the space occupied by the index.

[0015] Optionally, a new indication is received, where the new indication is used to indicate that data in the first data that is non-duplicate with data already stored in the database has been stored in the database; in response to the new indication, the second compression bitmap is updated according to the first directory information and the duplicate directory entries.

[0016] Optionally, a deletion indication is received, where the deletion indication is used to indicate that the second data has been deleted from the database, and the deletion indication includes second directory information corresponding to the second data, where the second directory information is stored on the first shard; in response to the deletion indication, the second compression bitmap is updated according to the second directory information.

[0017] Through the above method, the directory information stored in the distributed cache system can be updated in a timely manner, thereby improving the accuracy of duplicate checking.

[0018] In a second aspect, a device for checking data duplication is provided, comprising: an acquisition module for acquiring first directory information of a first shard to be stored in a distributed cache system, the first directory information being used to identify first data to be stored in a database, the first directory information comprising at least one directory entry, the data volume length of each directory entry in the at least one directory entry being M, where M is a positive integer; the distributed cache system being used to store directory information corresponding to the data stored in the database; a processing module for generating a first compressed bitmap based on the first directory information, the bitmap bits in the first compressed bitmap corresponding one-to-one to the directory entries in the first directory information; determining directory entries in the first directory information that are repeated with directory information already stored in the first shard based on the first compressed bitmap and the second compressed bitmap; wherein the bitmap bits in the second compressed bitmap correspond one-to-one to directory entries in directory information already stored in the first shard; and determining data that is repeated with data already stored in the database based on the repeated directory entries.

[0019] Optionally, the fragment number of the first fragment corresponds to the first S bits of all directory entries in the first directory information.

[0020] Optionally, the data length of each directory entry is 64 bits.

[0021] Optionally, the processing module is further configured to create a target container based on the first N bits of data of the directory entry, store the last MN bits of the directory entry in the target container, and identify the target container using the first N bits of data of the directory entry; or, the first directory information includes multiple directory entries; generating a first compressed bitmap based on the first directory information includes: creating the target container based on the first N bits of the first directory entry, storing the last MN bits of other directory entries whose first N bits are the same as the first N bits of the first directory entry in the target container, wherein the first directory entry is any one of the multiple directory entries.

[0022] Optionally, the processing module is further used to: establish an index based on the first N bits of data of the directory entry, wherein the index adopts an adaptive radix tree ART data structure, and the index is used to indicate the correspondence between the at least one directory entry and the container in the compressed bitmap; search for the target container corresponding to the second directory entry based on the first N bits of the second directory entry and the index; wherein the second directory entry is any one of the at least one directory entry; and store the last MN bits of the second directory entry in the target container.

[0023] Optionally, the data duplication checking device also includes a receiving module for receiving a new indication, wherein the new indication is used to indicate that data in the first data that is non-duplicate with data already stored in the database has been stored in the database; the processing module is also used to update the second compression bitmap according to the first directory information and the duplicate directory entries in response to the new indication.

[0024] Optionally, the receiving module is also used to receive a deletion indication, wherein the deletion indication is used to indicate that the second data has been deleted from the database, and the deletion indication includes second directory information corresponding to the second data, wherein the second directory information is stored on the first shard; the processing module is also used to respond to the deletion indication and update the second compression bitmap according to the second directory information.

[0025] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the instructions stored in the memory, thereby enabling the device to execute the method described in the first aspect or any optional embodiment of the first aspect.

[0026] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store instructions, and when the instructions are executed, the method in the first aspect or any optional embodiment of the first aspect is implemented.

[0027] In a fifth aspect, a computer program product comprising instructions is provided, wherein the computer program product stores instructions that, when executed on a computer, enable the computer to execute the method described in the first aspect or any optional embodiment of the first aspect.

[0028] The technical effects or advantages of one or more technical solutions provided in the second, third, fourth and fifth aspects of the embodiments of this application can be explained by the technical effects or advantages of the corresponding one or more technical solutions provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a possible application scenario of the duplicate checking method provided in an embodiment of the present application;

[0030] Figure 2 A flowchart of a duplicate checking method provided in an embodiment of the present application;

[0031] Figure 3a A flowchart of a method for updating a compressed bitmap provided in an embodiment of the present application;

[0032] Figure 3b A flowchart of another method for updating a compressed bitmap provided in an embodiment of the present application;

[0033] Figure 4 This is a structural diagram of a data duplication checking device provided in an embodiment of the present application;

[0034] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. Unless there is a conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0036] It should be understood that in the description of the embodiments of this application, the terms "first" and "second" are used only for the purpose of distinguishing the description and should not be understood to indicate or imply relative importance or order. In the description of the embodiments of this application, "plurality" means two or more.

[0037] The term "and / or" in the embodiments of this application is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0038] In order to facilitate understanding of the solutions of the embodiments of the present application, a brief introduction to the relevant terms involved in the embodiments of the present application is first given.

[0039] A distributed cache system caches disk data in memory and provides data writing and reading services in a distributed manner. A distributed system consists of multiple servers, each handling different tasks and coordinating and collaborating to complete the overall system's work. A distributed cache system uses different servers to collectively complete the caching work required of the entire system.

[0040] Sharding is a concept in database design that, as the name implies, involves creating smaller parts from a larger one. In a database context, sharding results in the creation of smaller partitions within the ledger, which are called shards. In this application, different shards of a distributed cache system refer to different servers within the distributed cache system.

[0041] A bitmap is a data structure that uses binary bits to identify a corresponding value. Because it can only represent 0 or 1, it is typically used to indicate whether a value exists. For example, if there is a bitmap that represents whether a person has a car, then there is one and only one binary bit in the bitmap corresponding to that person, and the 0 or 1 recorded in that bit is used to indicate whether the person has a car.

[0042] The following describes possible application scenarios of the embodiments of this application. Figure 1 , which is a possible application scenario provided by an embodiment of the present application. In this application scenario, there are a business system, a distributed cache system, and a database. These business systems, distributed cache systems, and databases are all logically functional divisions. In actual applications, these business systems, distributed cache systems, and databases can be hosted by different computers or servers, or by the same computer or server.

[0043] Among them, the business system is a front-end business system that connects and communicates with the database. The business system is responsible for entering data information into the database and can also obtain data information from the database.

[0044] In such Figure 1In the scenario shown, the business system needs to enter the acquired data information into the database. In order to avoid duplication of the data information to be entered with the data information already existing in the database, before the data information is entered into the database, the data information to be entered will be checked for duplicates in the distributed cache system, that is, the entered information will be compared with the database information to determine whether there is duplicate data in the entered information. When duplicate data exists, the business system can skip the duplicate data and enter the non-duplicate data into the database. The embodiment of the present application is used to check the duplicates of the entered data.

[0045] See also Figure 2 , is a duplicate checking method provided in an embodiment of the present application, which is applied to Figure 1 In the distributed cache system shown, the distributed cache system may include one or more servers, and the caches in the one or more servers constitute the distributed cache system in the embodiment of the present application. In one possible implementation, a database is provided in the server, and the database may be a distributed database such as HBase, Mongo Database (MongoDB), Distributed Relational Database Service (DRDS), Volt Database (VoltDB), and ScaleBase.

[0046] The specific implementation steps of the method are as follows:

[0047] Step S201: Obtain the first directory information of the first shard to be stored in the distributed cache system, where the first directory information is used to identify the first data to be stored in the database. The first directory information includes at least one directory item, and the data volume length of each directory item in the at least one directory item is M, where M is a positive integer; the distributed cache system is used to store the directory information corresponding to the data stored in the database.

[0048] The first data is data to be stored in the database. Before storing the first data in the database, it is necessary to check for duplicates of the first data based on the data in the database, so as to avoid storing duplicate data in the database.

[0049] The first directory information is used to identify the first data, and the first directory information includes at least one directory entry, wherein the data length of the directory entry is M bits, and M is a positive integer; optionally, the above M can be 32 bits or 64 bits, or 2 n Bit.

[0050] In a possible implementation, the directory entry may be a hash value with a data length of 64 bits, the hash value is entirely composed of 0s and 1s, and is calculated from the first data.

[0051] Optionally, in the first directory information, the first S bits of all directory entries correspond to the number of the first fragment.

[0052] For example, assume that the first directory entry is a hash value with a data length of 64 bits, and the first S bits of the directory entry are the first 8 bits. Then, when the directory information is stored in the distributed cache system, the distributed cache system will send the directory information to different shards based on the first 8 bits of the directory entry in the directory information, and ensure that the first 8 bits of the directory entry correspond to the number of the shard being sent. For example, after converting the first 8 bits of the 64-bit hash value into a decimal number, its possible value will be in the range of 0-255. Assuming that there are 10 shards in the distributed cache system, and the shard numbers are 0-9, then after converting the first 8 bits of the data into a decimal number, the corresponding directory entries with values ​​in the range of 0-26 are sent to the shard numbered 0.

[0053] Through the above method, the directory information of the data to be stored in the database can be stored in the distributed cache system in an orderly and dispersed manner, thereby improving the efficiency of data storage.

[0054] Optionally, metadata information of the first data is obtained, and the metadata information is calculated using an xxhash algorithm to obtain a hash value, and the hash value is used as the first directory information.

[0055] Specifically, the metadata uniquely represents the first data. For example, if the first data is about an elderly person, it may include Elder A's name, gender, age, and ID number. In this case, the metadata is Elder A's ID number, thus uniquely representing Elder A's data. For another example, if the first data is about a vehicle, it may include Vehicle A's license plate number, model, and body color. In this case, the metadata can be Vehicle A's license plate number, thus uniquely representing Vehicle A's data.

[0056] After obtaining the metadata for the first data, a hash calculation is performed on the metadata to produce a hash value with a fixed length. This hash value is the directory entry in the first directory information. Even if the metadata changes, the same calculation will produce a different hash value. This means that the hash value uniquely corresponds to the metadata, and the first directory information uniquely identifies the metadata. Since the metadata uniquely represents the first data, the first data can also be uniquely identified based on the first directory information.

[0057] Through the above method, the first data is uniquely identified by a hash value with a fixed data length, which reduces the adverse impact of redundant information on data duplication checking and improves the efficiency of duplication checking.

[0058] Optionally, the xxhash algorithm can be used when hashing the metadata. The xxhash algorithm is a type of hash algorithm that splits the input string into four separate streams. Each stream is processed separately, and the results are combined into a final hash value after all streams are processed.

[0059] In the above method, the xxhash algorithm is used to perform hash calculation on the metadata, which not only ensures the success rate of hash value calculation, but also ensures the execution efficiency of hash calculation, and ultimately ensures the efficiency of obtaining the first directory information.

[0060] In one possible implementation, the xxhash64 algorithm can be used for hash calculation, so that the data length of the hash value obtained is 64 bits. In this method, the calculation result is a 64-bit hash value, which can ensure that the duplicate checking operation is performed on a larger amount of first data, thereby improving the applicability of the duplicate checking method provided in this application.

[0061] In an embodiment of the present application, the purpose of the distributed cache system includes caching directory information of all data to be stored in the database, and these directory information are cached in different shards of the distributed cache system, wherein the first shard in the distributed cache system caches the first directory information.

[0062] The first data corresponding to the first directory information is the data to be stored in the database, so it is necessary to perform a duplicate check based on the first directory information in the distributed cache system. In the duplicate checking method provided in the embodiment of the present application, a duplicate check is performed by comparing compressed bitmaps. Therefore, it is necessary to create a compressed bitmap based on the directory information of the data to be stored in the database, and compare it with the compressed bitmap created based on the directory information of the data already stored in the database. Therefore, the following steps need to be performed:

[0063] Step S202: Generate a first compressed bitmap according to the first directory information, where bitmap bits in the first compressed bitmap correspond one-to-one to directory entries in the first directory information.

[0064] Among them, compressed bitmap is a technology that is optimized relative to bitmap. Optionally, in the embodiment of the present application, the RoaringBitMap technology is used to implement compressed bitmap. Specifically, to construct a compressed bitmap, it is necessary to create a container based on the first P bits of the binary unsigned integer data. The number of containers can be up to 2P In short, a compressed bitmap is a collection of containers. When storing data in a compressed bitmap, you first need to find the corresponding container based on the first P bits of the binary unsigned integer (if there is no corresponding container, create a new one), and then store the last Q bits of the binary unsigned integer in the container.

[0065] A container is a data structure used to store the last Q bits of a binary unsigned integer in a compressed bitmap.

[0066] In step S202, the first compressed bitmap is generated based on the first N bits of data of the directory entry in the first directory information. N Containers are used to store the last MN bits of the directory entry.

[0067] Optionally, taking the generation of the first compressed bitmap as an example, how to generate a compressed bitmap based on directory information is described. For ease of understanding, the first directory information includes 10 directory entries as an example, each of which is a different binary 64-bit hash value.

[0068] The following is a specific implementation method for generating the first compressed bitmap:

[0069] First, determining the first 48 bits of data of each directory entry in the at least one directory entry in the first directory information; second, creating a container based on the determined first 48 bits of data, and identifying the container with the first 48 bits of data; finally, storing the last 16 bits of data of all directory entries in the first directory information that have the same first 48 bits of data in the same container.

[0070] In this way, there will be at least one and at most 10 containers storing the last 16 bits of data of the above 10 directory entries, and these containers after storing the data are the first compressed bitmap.

[0071] In the above method, the last 16 bits of directory entries with the same first 48 bits of data are stored in the same container, and the corresponding container is created only when data is needed. In this way, the space required for data storage can be effectively saved.

[0072] Optionally, still taking the directory entry as a 64-bit hash value as an example, when more data needs to be stored in the first compressed bitmap, the following storage method can also be used.

[0073] Specifically, the first 48 bits of the directory entry are obtained, and an index is established based on the first 48 bits of the directory entry. The index adopts an Adaptive Radix Tree (ART) data structure and is used to indicate the correspondence between the directory entry and the container in the first compressed bitmap.

[0074] The ART tree is an indexed data structure in which each radix tree node automatically adjusts as data changes. In an ART tree, the initial node is Node4, which can store four pieces of data. When the fifth piece of data needs to be stored, the node expands to Node16, and so on, eventually expanding to Node256. At this point, if the current layer cannot store a key, it splits into child nodes, which continue to store the remaining portion of the key in the same manner as described above.

[0075] Taking the first 48 bits of data of the directory entry in this application as an example, when establishing the ART tree index, the first layer is used to store the first 8 bits of the 48-bit data, and then the 9th to 16th bits of the 48-bit data are stored in the split child node, and so on, continuously splitting new child nodes until all 48 bits of data are stored.

[0076] After the index is established in this way, the path to the node of the ART tree leaf is the stored data.

[0077] In this application, the container corresponding to the first 48 bits of data of the directory entry and the index can be found.

[0078] In this method, indexing the containers in the compressed bitmap can effectively improve the query efficiency of the containers in the compressed bitmap. At the same time, using the ART tree data structure to establish the index can not only improve the query efficiency but also save the space occupied by the index.

[0079] When searching for a container corresponding to the first 48 bits of data of a directory entry, the following two situations may occur: situation 1, the first container exists; situation 2, the first container does not exist.

[0080] The first method is to determine whether the first container is an array container or a bitmap container when the first container exists. If the first container is an array container, the last 16 bits of the directory entry are converted to decimal and then stored in the first container. If the first container is a bitmap container, the last 16 bits of the directory entry are used to perform an OR operation on the bitmap corresponding to the bitmap container, so that the bits corresponding to the last 16 bits of the data in the bitmap are set to 1, indicating that the last 16 bits of data have been stored in the first compressed bitmap.

[0081] The second method is to create an array type container (ie, the first container) corresponding to the first 48 bits when the first container does not exist, and store the last 16 bits of data in the array type container.

[0082] To save storage space, an array-type container is usually created when a new container is created. When the cardinality of the array-type container is greater than 4096, the array-type container is converted to a bitmap-type container to store more data. This is because, in the first compressed bitmap, the bitmap-type container is used to store the last 16 bits of data of the first directory information, so its bitmap size is fixed at 8k. When the cardinality of the array-type container is less than 4096, the storage space it occupies is less than 8k. When the cardinality of the container is greater than 4096, the size of the array-type container will exceed 8k. Therefore, based on the overall space saving considerations, the above two types of containers will coexist in the compressed bitmap.

[0083] Through the above manner, the distributed cache system records the first directory information in a manner of generating a compressed bitmap, which effectively reduces the storage space occupied when storing the directory information and improves the utilization rate of the storage space.

[0084] Step S203: Determine the directory entries in the first directory information that are repeated in the directory information already stored in the first fragment based on the first compressed bitmap and the second compressed bitmap; wherein the bitmap bits in the second compressed bitmap correspond one-to-one to the directory entries in the directory information already stored in the first fragment.

[0085] The first compressed bitmap records the first directory information, while the second compressed bitmap is used to record the third directory information. Similar to the first directory information, the third directory information is also a hash value with a fixed data length and is used to identify a certain data.

[0086] The difference is that the first directory information is used to identify the first data to be stored in the database, while the third directory information is used to identify the second data that has already been stored in the database and is of the same type as the first data. The third directory information corresponding to the second data is stored in the first shard of the distributed cache system. For example, if the first data is elderly data, then the second data in the database is also elderly data, that is, the first data and the second data are of the same type.

[0087] Therefore, the first data to be stored in the database needs to be checked for duplication based on the second data already stored in the database to ensure that the data finally stored in the database does not duplicate the second data.

[0088] The second compressed bitmap has been pre-stored in the first shard of the distributed cache system to record the third directory information corresponding to the second data. After the first compressed bitmap is generated, it is necessary to compare the first compressed bitmap with the second compressed bitmap to determine the target directory information in the first directory information that is repeated in the third directory information.

[0089] Specifically, an AND operation is performed on the first compression bitmap and the second compression bitmap to obtain a duplicate data compression bitmap, and the duplicate data compression bitmap is used to record target directory information in the first directory information that is duplicated with the third directory information.

[0090] Exemplarily, the above operations are described by using a container in a compressed bitmap.

[0091] Get the index of the second container in the first compressed bitmap, and use the index to find the third container at the same index in the second compressed bitmap. There are two possible cases: 1. The third container exists, 2. The third container does not exist.

[0092] In the first case, the container type of the third container is determined. If the third container is an array container, a binary search is performed in the third container for each data item in the second container, deleting any data from the second container that is not found in the third container. If the third container is a bitmap container, an AND operation is performed on the first bitmap corresponding to the second container and the second bitmap corresponding to the third container. This way, only data that appears in the same position in both bitmaps is retained in the second container. In this way, the second container retains only data that appears in both the second and third containers.

[0093] When the second situation occurs, the second container is deleted from the first compression bitmap.

[0094] The above operation is performed on all containers in the first compression bitmap to obtain a duplicate data compression bitmap. All directory entries recorded in the bitmap are target directory entries.

[0095] Step S204: determining, based on the duplicate directory entries, whether the first data is duplicated with data already stored in the database.

[0096] Since the target directory entry is a part of the first directory information, the target directory entry can uniquely identify the target data in the first data, ie, the data that is repeated with the data already stored in the database.

[0097] In the above solution, data duplication checking is performed in the distributed cache system, reducing database access pressure. Furthermore, concurrent duplication checking through the distributed cache system improves duplication checking efficiency. Using compressed bitmap comparison for duplication checking not only saves directory information space in the distributed cache system, but also ensures both efficiency and accuracy. Using compressed bitmaps to record directory information effectively improves storage space utilization, while comparing compressed bitmaps for duplication checking reduces the time complexity of duplication checking through simple binary search and bitwise operations.

[0098] Optionally, after completing the above duplicate checking operation, Figure 1 The business system shown stores non-duplicate data in a database. At this time, the data in the database has changed, so the corresponding directory information and the compressed bitmap recording the directory information in the distributed cache system need to be updated.

[0099] See also Figure 3a , is a flowchart of a method for updating a compressed bitmap provided in an embodiment of the present application, and its specific implementation steps are as follows:

[0100] Step S301a: receiving a new addition indication, where the new addition indication is used to indicate that data in the first data that is not duplicated with data already stored in the database has been stored in the database.

[0101] Before executing step S301a, the distributed cache system needs to send the found duplicate data to Figure 1 In the illustrated business system, upon obtaining duplicate data, the business system avoids storing the duplicate data in the database when storing the first data in the database. After successfully storing the data in the database, the business system sends a new indication to the distributed cache system, indicating that data in the first data that is not duplicated with data already stored in the database has been stored in the database.

[0102] Step S302a: In response to the new addition indication, update the second compression bitmap according to the first directory information and the repeated directory entries.

[0103] Taking a directory entry as a 64-bit hash value as an example, it is explained how to update the second compression bitmap according to the first directory information and repeated directory entries.

[0104] First, at least one directory entry of non-duplicate data entered into the database is obtained based on the first directory information and the repeated directory entries.

[0105] Secondly, based on the first 48 bits of data of the directory entry, the container corresponding to the first 48 bits of data is searched in the second compressed bitmap.

[0106] At this time, there are two cases according to whether the container exists or not: 1. The container does not exist, 2. The container exists.

[0107] 1. If the container does not exist, create an array-type container corresponding to the first 48 bits of data in the directory entry, and store the last 16 bits of data in the directory entry into the container.

[0108] 2. If the container exists, perform operations in the container according to the different container types:

[0109] 1) If the container is an array type container, the last 16 bits of data in the above directory entry are stored in the container;

[0110] 2) If the container is a bitmap container, perform an OR operation on the last 16 bits of data in the above directory entry and the bitmap corresponding to the container.

[0111] Through the above method, the directory information stored in the distributed cache system can be updated in a timely manner, thereby improving the accuracy of duplicate checking.

[0112] See also Figure 3b , is a flowchart of a method for updating a compressed bitmap provided in an embodiment of the present application, and its specific implementation steps are as follows:

[0113] Step S301b: Receive a deletion instruction, where the deletion instruction is used to indicate that the second data has been deleted from the database, and the deletion instruction includes second directory information corresponding to the second data, where the second directory information is stored on the first shard;

[0114] Step S302b: In response to the deletion instruction, update the second compression bitmap according to the second directory information.

[0115] Also taking the hash value of the directory entry as 64 as an example, it is explained how to update the second compression bitmap according to the first directory information and the repeated directory entries.

[0116] According to the first 48 bits of data of the directory entry in the second directory information in the deletion indication, a container corresponding to the first 48 bits of data of the directory entry in the second compressed bitmap is searched.

[0117] When the container is an array type container, a binary search is performed in the container, and the last 16 bits of the found directory entry are deleted from the container;

[0118] When the container is a bitmap container, the last 16 bits of the directory entry are XORed with the bitmap corresponding to the container, so that the value of the bitmap bit corresponding to the directory entry in the bitmap is 0.

[0119] Through the above method, the directory information stored in the distributed cache system can be updated in a timely manner, thereby improving the accuracy of duplicate checking.

[0120] It should be understood that the various embodiments herein can be combined with each other to achieve different technical effects.

[0121] The above describes the method provided by the embodiment of the present application, and the following describes the device provided by the embodiment of the present application.

[0122] See also Figure 4 An embodiment of the present application provides an image detection device, which may be the back-end server (or first device) mentioned above or a chip or integrated circuit in the server device, etc. The device includes modules / units / technical means for executing the method performed by the server device in the above method embodiment.

[0123] Exemplarily, the apparatus 400 may include:

[0124] An acquisition module 401 is configured to acquire a first image captured by a first type of image acquisition device disposed in a preset scene, wherein the first type of image acquisition device is configured to capture images of first type of objects in the preset scene; and acquire a second image captured by a second type of image acquisition device disposed in the preset scene, wherein the second type of image acquisition device is configured to capture images of second type of objects in the preset scene.

[0125] Processing module 402 is used to determine whether there is a target object in the preset scene based on the first image, wherein the target object is a first type of object that performs a first preset behavior; if the target object exists, determine whether the second type of object performs a second preset behavior based on the second image; if so, determine whether the target object is a misjudgment based on the second image.

[0126] It should be understood that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. As a possible product form of the above device, see Figure 5 , an embodiment of the present application further provides an electronic device 500, including:

[0127] At least one processor 501; and a communication interface 503 communicatively connected to the at least one processor 501; the at least one processor 501 executes instructions stored in the memory 502, so that the electronic device 500 executes the method steps performed by any device in the above method embodiment through the communication interface 503.

[0128] Optionally, the memory 502 is located outside the electronic device 500 .

[0129] Optionally, the electronic device 500 includes the memory 502, the memory 502 is connected to the at least one processor 501, and the memory 502 stores instructions that can be executed by the at least one processor 501. Figure 5 The dashed lines indicate that the memory 502 is optional for the electronic device 500 .

[0130] The processor 501 and the memory 502 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0131] The specific connection medium between the processor 501, the memory 502 and the communication interface 503 is not limited in the embodiment of the present application. Figure 5 In the embodiment, the processor 501, the memory 502 and the communication interface 503 are connected via a bus 504. Figure 5 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0132] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.

[0133] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0134] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0135] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0136] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0137] As another possible product form, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the computer executes the method steps performed by any device in the above method example.

[0138] As another possible product form, an embodiment of the present application also provides a computer program product containing instructions, wherein the computer program product stores instructions that, when executed on a computer, enable the computer to execute the method steps performed by any device in the above method embodiment.

[0139] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0143] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for checking data duplication, characterized in that: include: Obtaining first directory information of a first shard to be stored in a distributed cache system, the first directory information being used to identify first data to be stored in a database, the first directory information including at least one directory entry, the data volume length of each directory entry in the at least one directory entry being M, where M is a positive integer; the distributed cache system being used to store directory information corresponding to the data stored in the database; generating a first compressed bitmap according to the first directory information, wherein bitmap bits in the first compressed bitmap correspond one-to-one to directory entries in the first directory information; Determining, based on the first compressed bitmap and the second compressed bitmap, directory entries in the first directory information that are duplicates of directory information already stored in the first fragment, wherein the bitmap bits in the second compressed bitmap correspond one-to-one to the directory entries in the directory information already stored in the first fragment; Determining, based on the duplicate directory entries, whether the first data is duplicated with data already stored in the database; The step of generating a first compressed bitmap according to the first directory information includes: When the first directory information includes one directory entry, creating a target container according to the first N bits of data of the directory entry, storing the last MN bits of the directory entry in the target container, and using the first N bits of data of the directory entry to identify the target container; or When the first directory information includes multiple directory entries, the target container is created based on the first N bits of the first directory entry, and the last MN bits of other directory entries whose first N bits are the same as the first N bits of the first directory entry are stored in the target container, where the first directory entry is any one of the multiple directory entries.

2. The method according to claim 1, wherein The fragment number of the first fragment corresponds to the first S bits of all directory entries in the first directory information.

3. The method according to claim 1, wherein The data length of each directory entry is 64 bits.

4. The method according to claim 1, wherein The method further comprises: Establishing an index based on first N bits of data of the directory entry, wherein the index adopts an adaptive radix tree ART data structure, and the index is used to indicate a corresponding relationship between the at least one directory entry and the container in the compressed bitmap; searching, based on the first N bits of the second directory entry and the index, for the target container corresponding to the second directory entry; wherein the second directory entry is any one of the at least one directory entry; The last MN bit of the second directory entry is stored in the target container.

5. The method according to claim 1, wherein The method further comprises: receiving a new addition indication, the new addition indication being used to indicate that data in the first data that is not duplicated with data already stored in the database has been stored in the database; In response to the new addition indication, the second compression bitmap is updated according to the first directory information and the repeated directory entries.

6. The method according to claim 5, wherein The method further comprises: receiving a deletion indication, the deletion indication being used to indicate that second data has been deleted from the database, the deletion indication including second directory information corresponding to the second data, wherein the second directory information is stored on the first shard; In response to the deletion indication, the second compression bitmap is updated according to the second directory information.

7. A device for checking data duplication, characterized in that: include: an acquisition module, configured to acquire first directory information of a first shard to be stored in a distributed cache system, the first directory information being used to identify first data to be stored in a database, the first directory information including at least one directory entry, the data volume length of each directory entry in the at least one directory entry being M, where M is a positive integer; the distributed cache system being configured to store directory information corresponding to the data stored in the database; a processing module configured to generate a first compressed bitmap based on the first directory information, wherein bitmap bits in the first compressed bitmap correspond one-to-one to directory entries in the first directory information; determine directory entries in the first directory information that are duplicated with directory information already stored in the first shard based on the first compressed bitmap and a second compressed bitmap; wherein the bitmap bits in the second compressed bitmap correspond one-to-one to directory entries in the directory information already stored in the first shard; and determine that the first data is duplicated with data already stored in the database based on the duplicated directory entries; When the processing module is used to generate a first compressed bitmap based on the first directory information, it is specifically used to: when the first directory information includes one directory entry, create a target container based on the first N bits of data of the directory entry, store the last MN bits of the directory entry in the target container, and identify the target container using the first N bits of data of the directory entry; or, when the first directory information includes multiple directory entries, create the target container based on the first N bits of the first directory entry, store the last MN bits of other directory entries with the same first N bits as the first N bits of the first directory entry in the target container, where the first directory entry is any one of the multiple directory entries.

8. The device according to claim 7, wherein The fragment number of the first fragment corresponds to the first S bits of all directory entries in the first directory information.

9. The device according to claim 7, wherein The data length of each directory entry is 64 bits.

10. The device according to claim 7, wherein The processing module is further configured to: Establishing an index based on first N bits of data of the directory entry, wherein the index adopts an adaptive radix tree ART data structure, and the index is used to indicate a corresponding relationship between the at least one directory entry and the container in the compressed bitmap; searching, based on the first N bits of the second directory entry and the index, for the target container corresponding to the second directory entry; wherein the second directory entry is any one of the at least one directory entry; The last MN bit of the second directory entry is stored in the target container.

11. The device according to claim 7, wherein Also includes: A receiving module, configured to receive a new addition indication, wherein the new addition indication is used to indicate that data in the first data that is not duplicated with data already stored in the database has been stored in the database; The processing module is further configured to, in response to the new addition indication, update the second compression bitmap according to the first directory information and the repeated directory entries.

12. The device according to claim 11, wherein Also includes: The receiving module is further configured to receive a deletion indication, the deletion indication being configured to indicate that the second data has been deleted from the database, the deletion indication including second directory information corresponding to the second data, wherein the second directory information is stored on the first shard; The processing module is further configured to update the second compressed bitmap according to the second directory information in response to the deletion instruction.

13. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method according to any one of claims 1 to 6 by executing the instructions stored in the memory.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store instructions, and when the instructions are executed, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product comprising instructions, characterized in that The computer program product stores instructions, and when the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images