Data maintenance method, device, equipment and readable medium for distributed storage device

Abstract

The present disclosure provides a data maintenance method, apparatus, device and readable medium of a distributed storage device, wherein the data maintenance method of the distributed storage device includes: receiving batch data in a blocking queue; constructing a Bloom filter of a source library and a Bloom filter of a target library according to a key in the batch data, and maintaining an association relationship between the source library and the target library through a logical table; determining data differences according to the Bloom filter of the source library and the Bloom filter of the target library. Through the embodiments of the present disclosure, the complexity of the user's perception of the sharding information of the stored data is reduced, and the security, reliability and verification efficiency of data storage are improved.

Description

Background Art

[0002] Currently, in distributed databases, there are two main levels of data synchronization: one is to achieve data synchronization through background program coding, and the other is to directly act on the database to achieve data synchronization at the database level.

[0003] In related technologies, a distributed database consists of the following parts:

[0004] Source database: currently supports distributed relational databases, distributed file systems, and unstructured databases.

[0005] Target database: currently supports distributed relational databases, distributed file systems, and unstructured databases.

[0006] Management node cluster: used for data verification configuration, pushing data verification configuration to verification nodes. At the same time, receiving data synchronization status, progress and other information fed back by verification nodes.

[0007] Synchronous node cluster: a module that performs specific data verification processes.

[0008] Coordinator cluster: A module used to coordinate data verification.

[0009] However, when scaling existing distributed databases, users need to pay attention to a large amount of complex sharding information.

[0010] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0011] The objective of the present disclosure is to provide a data maintenance method, apparatus, device and readable medium for a distributed storage device, which are used to overcome, at least to a certain extent, the problem of complex database expansion and contraction caused by the limitations and defects of related technologies.

[0012] According to a first aspect of an embodiment of the present disclosure, a data maintenance method for a distributed storage device is provided, comprising: receiving batch data in a blocking queue; constructing a Bloom filter of a source library and a Bloom filter of a target library according to keys in the batch data, wherein an association relationship is maintained between the source library and the target library through a logical table; and determining data differences according to the Bloom filter of the source library and the Bloom filter of the target library.

[0013] In an exemplary embodiment of the present disclosure, constructing a Bloom filter of a source library and a Bloom filter of a target library according to a key in the batch data includes: determining a key corresponding to a specified field in the batch data; constructing a Bloom filter of the source library and a Bloom filter of the target library according to the key, maintaining an association relationship between the source library and the target library through the logical table; and writing the batch data into the source library and the target library.

[0014] In an exemplary embodiment of the present disclosure, writing the batch data into the source library and the target library includes: compressing the batch data; encrypting the compressed batch data; and writing the encrypted batch data into the source library and the target library.

[0015] In an exemplary embodiment of the present disclosure, compressing the batch data includes: merging the batch data using GZIP; and compressing the merged batch data.

[0016] In an exemplary embodiment of the present disclosure, determining data differences based on the Bloom filter of the source library and the Bloom filter of the target library includes: comparing the data extracted from the source library with the Bloom filter of the target library; and generating a missing data report for the target library based on the comparison result of the Bloom filter of the target library.

[0017] In an exemplary embodiment of the present disclosure, determining data differences based on the Bloom filter of the source library and the Bloom filter of the target library also includes: comparing the data extracted from the target library with the Bloom filter of the source library; and generating a missing data report for the source library based on the comparison result of the Bloom filter of the source library.

[0018] In an exemplary embodiment of the present disclosure, determining data differences based on the Bloom filter of the source library and the Bloom filter of the target library also includes: generating a difference data report based on the missing data report of the source library and the missing data report of the target library; and repairing the data in the source library and / or the data in the target library based on the difference data report.

[0019] According to a second aspect of an embodiment of the present disclosure, a data maintenance device of a distributed storage device is provided, comprising: a receiving module, configured to receive batch data in a blocking queue; a building module, configured to build a Bloom filter of a source library and a Bloom filter of a target library according to keys in the batch data; and a determining module, configured to determine data differences according to the Bloom filter of the source library and the Bloom filter of the target library.

[0020] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the data maintenance method of the distributed storage device as described in any one of the above items is implemented.

[0022] The disclosed embodiment improves the efficiency of data extraction and reduces the memory usage of storage procedures by receiving batch data in a blocking queue and constructing a Bloom filter of a source library and a Bloom filter of a target library according to the keys in the batch data. In addition, by determining data differences based on the Bloom filter of the source library and the Bloom filter of the target library, users do not need to perceive complex sharding information, which reduces the difficulty of data maintenance and improves data verification efficiency, security and reliability.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0025] Figure 1 A schematic diagram showing an exemplary system architecture of a data maintenance solution for a distributed storage device to which an embodiment of the present invention can be applied;

[0026] Figure 2 is a flow chart of a data maintenance method for a distributed storage device in an exemplary embodiment of the present disclosure;

[0027] Figure 3 is a flow chart of another distributed storage device data maintenance method in an exemplary embodiment of the present disclosure;

[0028] Figure 4 is a flow chart of another distributed storage device data maintenance method in an exemplary embodiment of the present disclosure;

[0029] Figure 5 is a flow chart of another distributed storage device data maintenance method in an exemplary embodiment of the present disclosure;

[0030] Figure 6 is a flow chart of another distributed storage device data maintenance method in an exemplary embodiment of the present disclosure;

[0031] Figure 7 is a flow chart of another distributed storage device data maintenance method in an exemplary embodiment of the present disclosure;

[0032] Figure 8 is a topological diagram of a data maintenance solution for a distributed storage device in an exemplary embodiment of the present disclosure;

[0033] Fig. 9 It is a schematic diagram of a mapping relationship of a data maintenance solution for a distributed storage device in an exemplary embodiment of the present disclosure;

[0034] Fig.10 It is a data checking schematic diagram of a data maintenance solution for a distributed storage device in an exemplary embodiment of the present disclosure;

[0035] Fig.11 It is a persistence schematic diagram of a data maintenance solution for a distributed storage device in an exemplary embodiment of the present disclosure;

[0036] Fig.12 is a schematic diagram of constructing a Bloom filter in a data maintenance method for a distributed storage device in an exemplary embodiment of the present disclosure;

[0037] Fig.13 It is a schematic diagram of data checking in a data maintenance method of a distributed storage device in an exemplary embodiment of the present disclosure;

[0038] Fig.14 is a block diagram of a data maintenance device of a distributed storage device in an exemplary embodiment of the present disclosure;

[0039] Fig.15 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0041] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0042] Figure 1 A schematic diagram showing an exemplary system architecture of a data maintenance solution for a distributed storage device to which an embodiment of the present invention can be applied.

[0043] like Figure 1 As shown, the system architecture 100 may include a distributed relational database 102, a distributed file system 104, an unstructured database 106, a management node cluster 108, a coordinator cluster 110, and a synchronization node cluster 112. The connections between clusters may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0044] It should be understood that Figure 1 The number of clusters, systems, and databases in the example is only for illustration. Any number of target clusters, systems, and databases may be provided as required. For example, the distributed relational database 102 may be a database cluster composed of multiple databases.

[0045] The user can use the terminal device to interact with the distributed relational database 102 through a network connection to receive or send messages, etc. The terminal device can be various electronic devices with a display screen, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc.

[0046] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0047] Figure 2 is a flow chart of a data maintenance method of a distributed storage device in an exemplary embodiment of the present disclosure.

[0048] refer to Figure 2 , the data maintenance method of the distributed storage device may include:

[0049] Step S202, receiving batch data in the blocking queue.

[0050] Step S204: construct a Bloom filter of the source library and a Bloom filter of the target library according to the key in the batch data, and the association relationship between the source library and the target library is maintained through a logic table.

[0051] Step S206: determining data differences according to the Bloom filter of the source database and the Bloom filter of the target database.

[0052] The disclosed embodiment improves the efficiency of data extraction and reduces the memory usage of storage procedures by receiving batch data in a blocking queue and constructing a Bloom filter of a source library and a Bloom filter of a target library according to the keys in the batch data. In addition, by determining data differences based on the Bloom filter of the source library and the Bloom filter of the target library, users do not need to perceive complex sharding information, which reduces the difficulty of data maintenance and improves data verification efficiency, security and reliability.

[0053] In an exemplary embodiment of the present disclosure, efficient data verification and repair based on double Bloom filters calls the REST API interface of the distributed database to obtain the sub-library and sub-table information of the distributed database, parses it into corresponding logical libraries, physical libraries, logical tables, and physical tables, and forms an association relationship between them, which is stored in the database as shown in Table 1 below:

[0054] Logical library: contains multiple physical libraries and shields the specific physical library information from the user.

[0055] Physical library: corresponds to a specific library on a database node.

[0056] Logical table: contains multiple physical tables and shields the specific physical table information from the user.

[0057] Physical table: corresponds to a specific table on the database node.

[0058] Based on the embodiments of the present disclosure and the construction of Table 1, when configuring the mapping relationship, only the logical table needs to be considered, without having to pay attention to the complex sharding information. At the same time, it supports real-time updating of changes in the sharding data of the distributed database. Therefore, it can effectively avoid the problem of reconfiguring the mapping relationship when adding or deleting sharding tables when the distributed database is expanded or reduced.

[0059] Furthermore, the automatic generation of the mapping relationship is achieved by matching the logical table name and the target table name, which reduces the workload of configuring the mapping relationship and improves the flexibility of the configuration.

[0060] Table 1

[0061] TYPE NAME IS_LOGICAL_MEDIA LOGICAL_MEDIA_ID DATABASE_ID Logic Library equal.* 1 NA NA Physics Library equal.* 0 128 NA Logical table Not equal to.* 1 NA 128 Physical Table Not equal to.* 0 130 129

[0062] The names of the logical library and the physical library are both ".*". For the logical library, its flag IS_LOGICAL_MEDIA is 1, and for the physical library, the flag is 0.

[0063] The names of logical tables and physical tables are not ".*". For logical tables, the flag IS_LOGICAL_MEDIA is 1, and for physical tables, the flag is 0. In addition, logical tables and physical tables have another additional field to associate with the * library they belong to.

[0064] The physical library is associated with its logical library through LOGICAL_MEDIA_ID, the logical table is associated with its logical library through DATABASE_ID, the physical table is associated with its physical library through DATABASE_ID, and the physical table is associated with its logical table through LOGICAL_MEDIA_ID.

[0065] The following describes in detail the steps of the data maintenance method for the distributed storage device.

[0066] In an exemplary embodiment of the present disclosure, Figure 3 As shown, constructing a Bloom filter of a source library and a Bloom filter of a target library according to the keys in the batch data includes:

[0067] Step S302, determining the key corresponding to the specified field in the batch data.

[0068] Step S304: construct a Bloom filter of the source library and a Bloom filter of the target library according to the key, and maintain an association relationship between the source library and the target library through the logic table.

[0069] Step S306: writing the batch data into the source database and the target database.

[0070] In an exemplary embodiment of the present disclosure, Figure 4As shown, writing the batch data into the source database and the target database includes:

[0071] Step S402: compress the batch data.

[0072] Step S404, encrypting the compressed batch data.

[0073] Step S406, writing the encrypted batch data into the source database and the target database.

[0074] In an exemplary embodiment of the present disclosure, compressing the batch data includes:

[0075] The batch data are merged using GZIP.

[0076] The merged batch data is compressed.

[0077] In an exemplary embodiment of the present disclosure, Figure 5 As shown, determining the data difference according to the Bloom filter of the source library and the Bloom filter of the target library includes:

[0078] Step S502: Check the data extracted from the source database with the Bloom filter of the target database.

[0079] Step S504: generating a missing data report of the target library according to the verification result of the Bloom filter of the target library.

[0080] In an exemplary embodiment of the present disclosure, Figure 6 As shown, determining the data difference according to the Bloom filter of the source library and the Bloom filter of the target library also includes:

[0081] Step S602: Check the data extracted from the target database with the Bloom filter of the source database.

[0082] Step S604: Generate a missing data report of the source database according to the verification result of the Bloom filter of the source database.

[0083] In an exemplary embodiment of the present disclosure, Figure 7 As shown, determining the data difference according to the Bloom filter of the source library and the Bloom filter of the target library also includes:

[0084] Step S702: Generate a difference data report according to the missing data report of the source library and the missing data report of the target library.

[0085] Step S704: repair the data in the source database and / or the data in the target database according to the difference data report.

[0086] Combine the following Figures 8 to 13 The data maintenance solution of the distributed storage device disclosed in the present invention is specifically described.

[0087] like Figure 8 As shown, the data maintenance architecture of the distributed storage device disclosed in the present invention includes a source library 802 and a target library 808. A target table 806 is maintained in the target library 808. The data in the source library 802 and the target table 806 are mapped and maintained in a relationship (i.e., an association relationship) through a logic table 804. The source library 802 determines the data batch according to the key identifier of "iot_inst". For example, the batch data is "iot_inst_0001" and "iot_inst_0002", and the keys of the batch data are "prod_inst" and "pod_inst_attr". "pod_inst_attr" is a key determined by "pod_inst_attr_0" and "pod_inst_attr_1".

[0088] like Fig. 9 As shown, the process 904 of generating mapping relationships for data in the distributed relational database 902 includes: pulling sub-library and sub-table information; parsing into logical libraries, physical libraries, logical tables, and physical tables; storing logical libraries, physical libraries, logical tables, and physical tables in the configuration library; and automatically generating mapping relationships based on matching logical table names and target end table names.

[0089] like Fig.10 As shown, the source distributed database 1002 includes shard physical library 1 and shard physical library 2, shard physical library 1 includes shard table 1 and shard table 2, and shard physical library 2 includes shard table 3 and shard table 4. The target distributed database 1006 includes shard physical library 1 and shard physical library 2, shard physical library 1 includes shard table 1 and shard table 2, and shard physical library 2 includes shard table 3 and shard table 4. The logical table 1 blocking queue and the logical table 2 blocking queue generate the logical library 1004 according to the extracted data, and the logical library 1004 includes the source library Bloom filter, the target library Bloom filter, the source database file, and the target database file.

[0090] like Fig.10 As shown, the data verification process of the distributed database is as follows:

[0091] The logical table of the source database is found according to the mapping relationship, and the associated shard table is found through the logical table. The data extraction module extracts the data that needs to be synchronized from each shard database according to the name of the shard table.

[0092] According to the association between the shard table and the logical table, the extracted data is gathered into the corresponding blocking queue. Then, the data extracted from the blocking queue is used to construct two Bloom filters for the source library and the target library respectively, and the data is written to the file at the same time.

[0093] Generate a difference data report based on the Bloom filters of the source and target libraries, including: redundant data in the target library, missing data in the target library, conflicting data between the source and target libraries, and automatically generate difference repair statements.

[0094] like Fig.11 As shown, the source distributed database 1102 includes source table 1 and source table 2, the target distributed database 1106 includes target table 1 and target table 2, and the mapping relationship between the source table and the target table is maintained through the logical table 1104. Specifically, the incremental update Bloom filter and Bloom filter persistence process are as follows:

[0095] After the data verification is completed, the incremental log data of the source database and the target database can be read respectively to continuously update the Bloom filters of the source database and the target database. The Bloom filters can be used to continuously verify and repair the daily added data. Data can also be extracted according to the time range or specific fields for verification and repair.

[0096] The Bloom filter is persisted to disk based on the incremental location + database + table name + timestamp, and supports loading Bloom filter data from disk.

[0097] like Fig.12 As shown, the process of building a double Bloom filter is as follows:

[0098] Step S1202, obtaining batch data from the blocking queue.

[0099] Step S1204, determine whether the batch data is obtained successfully, if so, execute step S1206, if not, execute step S1202.

[0100] Step S1206, determine whether the Bloom filter has been initialized, if so, execute step S1208, if not, execute step S1210.

[0101] Specifically, the Bloom filters of the source and target libraries are initialized. A Bloom filter can check up to 100 billion levels of structured data, and the memory used is only about 500M.

[0102] Step S1208, use the specific content of the fields in the data as the key.

[0103] Specifically, the specific contents of the fields in the data are used as the key to construct two Bloom filters for the source database and the target database respectively, to ensure that partial inconsistencies in the contents of records at both ends can be identified.

[0104] Step S1210, initializing the Bloom filters of the source database and the target database.

[0105] Step S1212: Use the data key to construct two Bloom filters for the source library and the target library respectively, write them into the files of the source library and the target library respectively, and perform decompression and encryption processing.

[0106] Specifically, the data is extracted to construct the Bloom filter and written to the files of the source and target databases respectively, and the files are compressed and encrypted to reduce the data extraction pressure of the database and prevent the problem of inaccurate verification results caused by dynamically updated data. GZIP is used to merge and compress the data files, so that the compressed file size is only 6% of the source file size. AES encryption technology is used to encrypt the compressed files to prevent data from being stolen or tampered with, and protect data security.

[0107] Step S1214, determine whether the Bloom filter is constructed successfully, if so, execute step S1216, if not, execute step S1212.

[0108] Step S1216, completing the construction of the Bloom filter.

[0109] like Fig.13 As shown, the steps in the process of generating a difference data report include:

[0110] Step S1302: extract data from the files in the source library and the files in the target library in parallel.

[0111] Step S1304, determine whether the batch data is extracted successfully, if so, execute step S1306, if not, execute step S1302.

[0112] Step S1306, extracting data from the files in the source library and checking it against the Bloom filter in the target library, and generating a report on the data missing from the target library.

[0113] Step S1308, extracting data from the files in the target database and checking it against the Bloom filter in the source database, and generating a report on the data missing from the source database.

[0114] Step S1310, comparing the data reports of the source database and the target database, and generating a difference data report.

[0115] Step S1312, determine whether the data report is generated successfully, if so, execute step S1314, if not, execute step S1310.

[0116] Step S1314, determining a difference data report, including: redundant data of the target library, missing data of the target library, conflicting data between the source library and the target library, and generating a difference repair statement.

[0117] Specifically, data is extracted from files in the source library in parallel and compared with the Bloom filter of the target library to generate a report on data missing from the target library. Data is extracted from files in the target library in parallel and compared with the Bloom filter of the source library to generate a report on data missing from the source library.

[0118] For example, the source database and the target database business systems are both distributed databases, with 2 billion records and a data size of about 800G. The steps for data verification according to the present disclosure include:

[0119] To perform verification using the embodiments of the present disclosure, it is only necessary to select the mode as full verification.

[0120] By filling in the corresponding source database information and target database information, you can automatically generate the synchronization mapping relationship with one click.

[0121] The embodiments of the present disclosure can quickly extract the full data of the source library and the target library, respectively construct the Bloom filters of the source library and the target library, and write the extracted data into the files of the source library and the target library respectively, and compress and encrypt the files, and use GZIP to merge and compress the data files, so that the compressed file size is only 6% of the source file size, which can greatly reduce the disk usage of massive data.

[0122] In the above embodiment, gzip is the abbreviation of GNUzip, which was first used for file compression in UNIX systems. The gzip encoding on the HTTP protocol is a technology used to improve the performance of web applications. The web server and the client (browser) must support gzip together. Currently, the mainstream browsers, Chrome, Firefox and IE, etc. all support this protocol. The gzip compression ratio is about 3 to 10 times, which can greatly save the network bandwidth of the server. In actual applications, not all files are compressed, but usually only static files are compressed.

[0123] In addition, using AES (Advanced Encryption Standard) encryption technology to encrypt compressed files can prevent data from being stolen or tampered with, protecting data security. Extracting data from files can reduce the pressure of data extraction from the database, and can also prevent the problem of inaccurate verification results caused by dynamically updated data.

[0124] Based on the embodiments of the present disclosure, the verification time is about 2.8 hours, the average verification performance is 200,000 records per second, the verification performance is stable, the verification data size is 800G, and the memory used is only 3G.

[0125] Corresponding to the above method embodiments, the present disclosure also provides a data maintenance device for a distributed storage device, which can be used to execute the above method embodiments.

[0126] Fig.14 It is a block diagram of a data maintenance apparatus of a distributed storage device in an exemplary embodiment of the present disclosure.

[0127] refer to Fig.14 , the data maintenance device 1400 of the distributed storage device may include:

[0128] The receiving module 1402 is configured to receive batch data in the blocking queue.

[0129] The construction module 1404 is configured to construct a Bloom filter of a source database and a Bloom filter of a target database according to the keys in the batch data.

[0130] The determination module 1406 is configured to determine data differences according to the Bloom filter of the source database and the Bloom filter of the target database.

[0131] In an exemplary embodiment of the present disclosure, the construction module 1404 is further configured to: determine the key corresponding to the specified field in the batch data; construct the Bloom filter of the source library and the Bloom filter of the target library according to the key; and write the batch data into the source library and the target library.

[0132] In an exemplary embodiment of the present disclosure, the construction module 1404 is further configured to: compress the batch data; encrypt the compressed batch data; and write the encrypted batch data into the source library and the target library.

[0133] In an exemplary embodiment of the present disclosure, the construction module 1404 is further configured to: use GZIP to merge the batch data; and compress the merged batch data.

[0134] In an exemplary embodiment of the present disclosure, the determination module 1406 is further configured to: check the data extracted from the source library with the Bloom filter of the target library; and generate a missing data report of the target library based on the check result of the Bloom filter of the target library.

[0135] In an exemplary embodiment of the present disclosure, the determination module 1406 is further configured to: check the data extracted from the target library with the Bloom filter of the source library; and generate a missing data report of the source library based on the check result of the Bloom filter of the source library.

[0136] In an exemplary embodiment of the present disclosure, the determination module 1406 is further configured to: generate a difference data report based on the missing data report of the source library and the missing data report of the target library; and repair the data in the source library and / or the data in the target library based on the difference data report.

[0137] Since the functions of the data maintenance device 1400 of the distributed storage device have been described in detail in the corresponding method embodiments, the present disclosure will not repeat them here.

[0138] Compared with the prior art, the technical solution proposed in this disclosure has the following advantages:

[0139] (1) It realizes the automatic configuration of data synchronization of distributed databases: It decouples user configuration from database table structure, which can greatly improve the configuration efficiency and flexibility of distributed database synchronization, especially for distributed databases.

[0140] (2) Efficient extraction of structured data is achieved: multi-threaded concurrent technology is used to divide the data into multiple batches to concurrently pull record data from the source and target data tables, and synchronize them to the blocking queue for caching. Data extraction and construction of double Bloom filters are decoupled through blocking queues, which can greatly improve the efficiency of data verification and repair.

[0141] (3) Achieved efficient parallel verification and repair of data: The use of double Bloom filters to achieve the ability to verify the source and target databases in parallel greatly speeds up the verification efficiency. At the same time, the data in the blocking queue is written into multiple files, and data is extracted from multiple files in parallel for verification, which can greatly reduce the data extraction pressure of the database and prevent the problem of inaccurate verification results caused by dynamically updated data. The actual single-machine verification performance is more than 200,000 items per second.

[0142] (4) Flexible setting of verification granularity is achieved: data can be extracted for verification and repair based on the time range or specific fields of the data, or several fields in the table can be arbitrarily specified for field-level verification and repair.

[0143] It can realize incremental update and timed persistence of Bloom filters. It continuously updates the Bloom filters of the source and target libraries by reading their incremental log data, and regularly persists the Bloom filters to disk based on the incremental location + database + table name + timestamp to prevent Bloom filter data loss due to database crashes. It can realize daily real-time incremental verification and repair of data in active-active disaster recovery scenarios, greatly improving data reliability and consistency in active-active scenarios.

[0144] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0145] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0146] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0147] Refer to the following Fig.15 15 to 16. An electronic device 1500 according to this embodiment of the present invention is described. Fig.15 The electronic device 1500 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0148] like Fig.15 As shown, the electronic device 1500 is in the form of a general computing device. The components of the electronic device 1500 may include but are not limited to: at least one processing unit 1510, at least one storage unit 1520, and a bus 1530 connecting different system components (including the storage unit 1520 and the processing unit 1510).

[0149] The storage unit stores program codes, which can be executed by the processing unit 1510, so that the processing unit 1510 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1510 can perform the method shown in the embodiment of the present disclosure.

[0150] The storage unit 1520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 15201 and / or a cache storage unit 15202 , and may further include a read-only storage unit (ROM) 15203 .

[0151] The storage unit 1520 may also include a program / utility 15204 having a set (at least one) of program modules 15205, such program modules 15205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0152] Bus 1530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0153] The electronic device 1500 may also communicate with one or more external devices 1540 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1500, and / or communicate with any device that enables the electronic device 1500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1550. Furthermore, the electronic device 1500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1560. As shown, the network adapter 1560 communicates with other modules of the electronic device 1500 via a bus 1530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a database, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0155] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0156] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0157] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0158] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0159] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0160] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or database. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0161] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0162] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A data maintenance method for a distributed storage device, characterized in that: include: Receive batch data in the blocking queue; Constructing a Bloom filter of a source database and a Bloom filter of a target database according to the key in the batch data, wherein an association relationship between the source database and the target database is maintained through a logic table; The data difference is determined according to the Bloom filter of the source library and the Bloom filter of the target library.

2. The data maintenance method of the distributed storage device according to claim 1, characterized in that: Constructing the Bloom filter of the source library and the Bloom filter of the target library according to the keys in the batch data includes: Determine a key corresponding to a specified field in the batch data; Constructing a Bloom filter of the source database and a Bloom filter of the target database according to the key, and maintaining an association relationship between the source database and the target database through the logic table; The batch data is written into the source database and the target database.

3. The data maintenance method of the distributed storage device according to claim 2, characterized in that: Writing the batch data into the source database and the target database includes: Performing compression processing on the batch data; Encrypting the compressed batch data; The encrypted batch data is written into the source database and the target database.

4. The data maintenance method of the distributed storage device according to claim 3, characterized in that: The compression processing of the batch data includes: The batch data are merged using GZIP; The merged batch data is compressed.

5. The data maintenance method of a distributed storage device according to any one of claims 1 to 4, characterized in that: Determining data differences according to the Bloom filter of the source database and the Bloom filter of the target database includes: Compare the data extracted from the source database with the Bloom filter of the target database; A missing data report of the target library is generated according to the verification result of the Bloom filter of the target library.

6. The data maintenance method of a distributed storage device according to any one of claims 1 to 4, characterized in that: Determining the data difference according to the Bloom filter of the source database and the Bloom filter of the target database further includes: Compare the data extracted from the target database with the Bloom filter of the source database; A missing data report of the source database is generated according to the verification result of the Bloom filter of the source database.

7. The data maintenance method of a distributed storage device according to any one of claims 1 to 4, characterized in that: Determining the data difference according to the Bloom filter of the source database and the Bloom filter of the target database further includes: Generate a difference data report according to the missing data report of the source library and the missing data report of the target library; The data in the source database and / or the data in the target database are repaired according to the difference data report.

8. A data maintenance device for a distributed storage device, characterized in that: include: A receiving module, configured to receive batch data in a blocking queue; A construction module, configured to construct a Bloom filter of a source library and a Bloom filter of a target library according to a key in the batch data; The determination module is configured to determine data differences according to the Bloom filter of the source library and the Bloom filter of the target library.

9. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the data maintenance method of the distributed storage device according to any one of claims 1 to 7 based on instructions stored in the memory.

10. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the data maintenance method for a distributed storage device according to any one of claims 1 to 7 is implemented.

Images