A database synchronization method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a database synchronization method and device, electronic equipment and storage medium, belonging to the technical field of database. The method comprises: executing a first persistence instruction through a first database node; listening to the first database node through a second database node to obtain the first persistence instruction; calculating a first hash value of the first persistence instruction, and performing first idempotency verification through the first hash value; if the first hash value passes the first idempotency verification, executing the first persistence instruction through the second database node; performing second idempotency verification through the first hash value, and the second idempotency verification is used to prevent the first database node and the second database node from executing in a loop. Embodiments of the present application can make each database node have full data, if a database node is down, only one database node can undertake all business, improve the database fault tolerance, and improve the business processing efficiency and work efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and particularly relates to a database synchronization method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the related art, a database pool usually adopts a master-slave architecture. If a certain database node is down, the data in the down database node may not exist in other databases, and the tasks responsible for by the down database node cannot be continued to be executed, so that the system fault tolerance is low, and the business processing efficiency is affected.

[0003] Therefore, the technical problems in the related art need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a database synchronization method and device, electronic equipment and storage medium, so that each database node has full data, which is beneficial to improve the database fault tolerance.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application provides a database synchronization method, which comprises the following steps:

[0006] executing a first persistence instruction by a first database node;

[0007] listening to the first database node by a second database node to obtain the first persistence instruction;

[0008] calculating a first hash value of the first persistence instruction, and performing a first idempotency verification by the first hash value;

[0009] if the first hash value passes the first idempotency verification, executing the first persistence instruction by the second database node;

[0010] performing a second idempotency verification by the first hash value, and the second idempotency verification is used to prevent the first database node and the second database node from executing in a loop.

[0011] In some embodiments, the executing the first persistence instruction by the first database node comprises:

[0012] listening to the first persistence instruction of the first database node by a second synchronization tool, and calculating a first hash value of the first persistence instruction;

[0013] judging whether a first verification database contains the first hash value, if not, setting a first verification value as negative one, if yes, reducing the first verification value by one, and the first verification value corresponds to the first hash value.

[0014] In some embodiments, the computing the first hash value of the first persistent instruction, the first idempotency verification by the first hash value, comprises:

[0015] computing the first hash value of the first persistent instruction;

[0016] judging whether the second verification database contains the first hash value;

[0017] if the second verification database does not contain the first hash value, setting the second verification value as one;

[0018] if the second verification database contains the first hash value, obtaining the second verification value corresponding to the first hash value, and adding one to the second verification value;

[0019] judging whether the second verification value is greater than zero, if yes, determining that the first hash value passes the first idempotency verification, if not, determining that the first hash value fails the first idempotency verification.

[0020] In some embodiments, the if the first hash value passes the first idempotency verification, executing the first persistent instruction by the second database node, comprises:

[0021] listening to the first persistent instruction of the second database node by a first synchronization tool, and computing the first hash value of the first persistent instruction;

[0022] judging whether the second verification database contains the first hash value, if not, setting the second verification value as negative one, if yes, reducing the second verification value by one, the second verification value corresponding to the first hash value.

[0023] In some embodiments, the second idempotency verification by the first hash value, comprises:

[0024] listening to the second database node by the first database node, obtaining the first persistent instruction, and computing the first hash value;

[0025] judging whether the first verification database contains the first hash value;

[0026] if the first verification database does not contain the first hash value, setting the first verification value as one;

[0027] if the first verification database contains the first hash value, obtaining the first verification value corresponding to the first hash value, and adding one to the first verification value;

[0028] judging whether the first verification value is greater than zero, if yes, determining that the first hash value passes the second idempotency verification, and the first database node executes the first persistent instruction.

[0029] If no, it is determined that the first hash value fails the second idempotency verification, and the first database node does not execute the first persistent instruction.

[0030] In some embodiments, the method further comprises:

[0031] Performing a health check on a synchronization tool configured to listen to the first database node and the second database node.

[0032] If the health check fails, obtaining a historical failure number.

[0033] Determining a current failure number according to the historical failure number.

[0034] If the current failure number is greater than a preset failure threshold, stopping the synchronization tool from listening to the first database node and the second database node.

[0035] In some embodiments, the method further comprises:

[0036] Querying a first offset of the first database node.

[0037] Querying a second offset of the second database node.

[0038] Performing offset verification by the first offset and the second offset to obtain a verification result.

[0039] To achieve the above object, another aspect of the embodiment of the present application proposes a database synchronization device, which comprises:

[0040] A first execution module configured to execute a first persistent instruction by a first database node.

[0041] A listening module configured to listen to the first database node by a second database node to obtain the first persistent instruction.

[0042] A first verification module configured to calculate a first hash value of the first persistent instruction and perform a first idempotency verification by the first hash value.

[0043] A second execution module configured to execute the first persistent instruction by the second database node if the first hash value passes the first idempotency verification.

[0044] A second verification module configured to perform a second idempotency verification by the first hash value, which is configured to prevent the first database node and the second database node from executing in a loop.

[0045] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0046] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the method described above when executed by a processor.

[0047] The embodiment of the present application at least has the following beneficial effects: the present application provides a database synchronization method and device, electronic device and storage medium, the scheme executes the first persistence instruction through the first database node; obtains the first persistence instruction through the second database node listening to the first database node; calculates the first hash value of the first persistence instruction, and performs the first idempotency verification through the first hash value; if the first hash value passes the first idempotency verification, executes the first persistence instruction through the second database node; performs the second idempotency verification through the first hash value, and the second idempotency verification is used to prevent the execution cycle of the first database node and the second database node. The embodiment of the present application can make each database node have full data, and if the database node appears to be down, only one database node can undertake all businesses. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the database synchronization method provided by the embodiment of the present application;

[0049] Figure 2 is a flowchart of the health check step of the database synchronization method provided by the embodiment of the present application;

[0050] Figure 3 is a flowchart of the offset verification step of the database synchronization method provided by the embodiment of the present application;

[0051] Figure 4 is an architecture diagram of the redis memory database distributed data pool provided by the embodiment of the present application;

[0052] Figure 5 is a schematic diagram of the redis memory database distributed data pool provided by the embodiment of the present application applied to a toll station system;

[0053] Figure 6 is a specific implementation flowchart of the database synchronization method provided by the embodiment of the present application applied to the accompanying program of the redis memory database distributed data pool;

[0054] Figure 7is a Redis cluster architecture diagram provided by an embodiment of the present application;

[0055] Figure 8 is a redis master-slave sentinel architecture diagram provided by an embodiment of the present application;

[0056] Figure 9 is a structure schematic diagram of a database synchronization device provided by an embodiment of the present application;

[0057] Figure 10 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0059] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0060] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0063] 1) Redis (Remote Dictionary Server), an open-source, high-performance key-value storage database, is widely used in caching, message queuing, session storage, etc. Redis has very high performance, supports rich data structures and various operations, and can effectively handle real-time storage and access of large amounts of data.

[0064] In related technologies, the database pool usually adopts a master-slave architecture. If a certain database node is down, the data in the down database node may not exist in other databases, and the system cannot continue to execute the tasks that the down database node is responsible for, which reduces the fault tolerance rate and affects the business processing efficiency.

[0065] In summary, the technical problems in related technologies need to be improved.

[0066] Therefore, in the embodiments of the present application, a database synchronization method, device, equipment and medium are provided. The first database node executes a first persistence instruction. The second database node listens to the first database node to obtain the first persistence instruction. The first hash value of the first persistence instruction is calculated, and the first hash value is used for first idempotency verification. If the first hash value passes the first idempotency verification, the second database node executes the first persistence instruction. The first hash value is used for second idempotency verification, which is used to prevent the first database node and the second database node from executing in a loop. The embodiments of the present application can ensure that each database node has full data. If a database node is down, only one database node is needed to handle all business.

[0067] The database synchronization method provided in the embodiments of the present application relates to the technical field of databases. The database synchronization method provided in the embodiments of the present application can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The server can be configured as a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. The server can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the database synchronization method, but is not limited to the above forms.

[0068] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0069] It should be noted that in each specific embodiment of the present application, when it is necessary to perform relevant processing according to user information, user behavior data, user history data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0070] Figure 1 is an optional flowchart of the database synchronization method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to including steps S101 to S105.

[0071] Step S101, executing a first persistence instruction by a first database node.

[0072] Specifically, the first persistence instruction is used to perform a write operation on the database node, and the first persistence instruction is generated by the first database node.

[0073] Optionally, the first persistence instruction is executed by the first database node to generate an rdb file and an aof file.

[0074] Optionally, when the first persistence instruction is generated and executed by the first database node, the hash value of the first persistence instruction is stored by the first verification database.

[0075] In some embodiments, the first persistent instruction of the first database node is listened to by the second synchronization tool, a first hash value of the first persistent instruction is calculated, it is judged whether the first verification database contains the first hash value, if not, the first verification value is set to negative one, if yes, the first verification value is reduced by one. The first verification value corresponds to the first hash value, and the second synchronization tool is used for listening to the write operation of the first database node, but is not used for synchronizing the first persistent instruction to the second database node.

[0076] Optionally, the first verification value and the first hash value adopt a key-value pair structure, the first hash value is key, and the first verification value is value.

[0077] In the embodiment, the first persistent instruction is executed by the first database node, and preparation is made for the subsequent synchronization of the first persistent instruction and the second idempotence verification by the second database node.

[0078] In step S102, the first database node is listened to by the second database node, and the first persistent instruction is obtained.

[0079] Optionally, the first database node is listened to by the second database node by using the first synchronization tool, and the first persistent instruction is obtained. The first synchronization tool is used for listening to the write operation of the first database node and synchronizing the first persistent instruction to the second database node.

[0080] Further, the rdb file and the aof file of the first database node are listened to by the second database node by using the first synchronization tool, and the first persistent instruction is obtained.

[0081] The application does not make specific limitation on the listening mode, and the actual detection needs can be flexibly selected. For example, the synchronization tool can be used for listening, log listening can be performed, message middleware can be used for listening, and the publish-subscribe mode can be used for listening.

[0082] In the embodiment, the first database node is listened to by the second database node, and the first persistent instruction is obtained, so that the second database node can know the persistent operation of the first database node in time, and the system fault tolerance is improved.

[0083] In step S103, a first hash value of the first persistent instruction is calculated, and the first hash value is used for first idempotence verification.

[0084] Specifically, the first idempotence verification is used for verifying whether the first persistent instruction has been executed by the second database node, so as to prevent the second database node from repeatedly executing the same operation.

[0085] In some embodiments, a first hash value of the first persistent instruction is calculated, and the first idempotency verification is performed by using the first hash value.

[0086] Optionally, when the second database node executes the instruction for the first time, a hash value of the instruction is stored in a second verification database corresponding to the second database node. The second verification database is configured to store hash values of executed instructions.

[0087] It can be understood that the hash value already exists in the second verification database before the second database node executes the instruction again, and thus the instruction execution is rejected.

[0088] Further, the hash value is stored as a key of a key-value pair in the verification database, and a verification value is used as a value of the key-value pair. The verification value is configured to verify the idempotency of the instruction.

[0089] In some embodiments, a first hash value of the first persistent instruction is calculated, and the first idempotency verification is performed by using the first hash value. The second verification database is determined whether to contain the first hash value. If the second verification database does not contain the first hash value, the second verification value is set to one. If the second verification database contains the first hash value, a second verification value corresponding to the first hash value is obtained, and the second verification value is incremented by one. It is determined whether the second verification value is greater than zero. If yes, it is determined that the first hash value passes the first idempotency verification. If no, it is determined that the first hash value fails the first idempotency verification.

[0090] In the embodiment, a first hash value of the first persistent instruction is calculated, and the first idempotency verification is performed by using the first hash value. This is beneficial to prevent the second database node from repeatedly executing the same instruction operation, and to realize the idempotency of the operation.

[0091] In step S104, if the first hash value passes the first idempotency verification, the first persistent instruction is executed by the second database node.

[0092] In some embodiments, the first persistent instruction is executed by the second database node. Optionally, the instruction is executed in a local transaction or a distributed transaction.

[0093] Optionally, the first persistent instruction of the second database node is listened to by using the first synchronization tool, a first hash value of the first persistent instruction is calculated, and it is determined whether the second verification database contains the first hash value. If no, the second verification value is set to negative one. If yes, the second verification value is decremented by one. The second verification value corresponds to the first hash value.

[0094] It can be understood that after the first idempotency verification is performed by using the first hash value, the second verification value is one. After the first persistent instruction is executed, the second verification value is decremented by one, and the second verification value is zero. This is beneficial to guarantee the idempotency of the operation.

[0095] In the embodiment, if the first hash value passes the first idempotency verification, the first persistence instruction is executed by the second database node, the synchronization of the database nodes is realized, the cross-node data consistency is realized, the system availability and fault tolerance are improved, and the business processing efficiency and work efficiency are improved.

[0096] In step S105, the second idempotency verification is performed by using the first hash value.

[0097] Specifically, the second idempotency verification is used to prevent the first database node and the second database node from executing in a loop.

[0098] In some embodiments, when the first database node generates and executes the first persistence instruction, the hash value of the first persistence instruction is stored in the first verification database, and then when the first database node listens to the second database node to obtain the first persistence instruction, the second idempotency verification is performed to prevent the execution in a loop.

[0099] Optionally, before the first database node executes the instruction, it is determined whether the first verification database contains the hash value of the instruction. If not, the instruction is executed. If yes, the execution of the instruction is refused.

[0100] It can be understood that the second idempotency verification is beneficial to prevent the infinite loop of “first database node→second database node→first database node→second database node”.

[0101] Optionally, a global transaction ID can be used instead of the hash value to reduce the calculation overhead.

[0102] In some embodiments, the first database node listens to the second database node to obtain the first persistence instruction and calculates the first hash value. It is determined whether the first verification database contains the first hash value. If the first verification database does not contain the first hash value, the first verification value is set to one. If the first verification database contains the first hash value, the first verification value corresponding to the first hash value is obtained, and the first verification value is incremented by one. It is determined whether the first verification value is greater than zero. If yes, it is determined that the first hash value passes the second idempotency verification, and the first database node executes the first persistence instruction. If no, it is determined that the first hash value does not pass the second idempotency verification, and the first database node does not execute the first persistence instruction.

[0103] In the embodiment, the second idempotency verification is performed by using the first hash value, and the bidirectional idempotency verification is beneficial to avoid the infinite loop between the database nodes, thereby improving the stability and consistency of the system.

[0104] The steps S101 to S105 shown in the embodiments of the present application are that the first persistent instruction is executed by the first database node; the first persistent instruction is obtained by the second database node listening to the first database node; the first hash value of the first persistent instruction is calculated, and the first idempotency verification is performed by using the first hash value, which is beneficial to prevent the second database node from repeatedly executing the same instruction operation, and realizes the idempotency of the operation; if the first hash value passes the first idempotency verification, the first persistent instruction is executed by the second database node, the synchronization of the database nodes is realized, the cross-node data consistency is realized, the system availability and fault tolerance are improved, and the business processing efficiency and work efficiency are improved; the second idempotency verification is performed by using the first hash value, and the bidirectional idempotency verification is performed, which is beneficial to avoid the infinite loop between the database nodes, thereby improving the stability and consistency of the system.

[0105] Please refer to Figure 2 In some embodiments, the database synchronization method provided by the embodiments of the present application further includes a health check step, which can include but is not limited to steps S201 to S204:

[0106] Step S201, performing health check on the synchronization tool.

[0107] Specifically, the synchronization tool is used to listen to the first database node and the second database node.

[0108] In step S201 of some embodiments, the first synchronization tool and the second synchronization tool are health checked.

[0109] Optionally, the synchronization tool listens to the aof of the database node, and if the aof listens to a network problem, the health check fails.

[0110] It should be noted that if the execution of the instruction process is problematic, the synchronization tool will retry, and after the retry fails, the synchronization tool will be restarted, the previous hash value in the database is cleared, the synchronization is read from the latest aof again, and then the writing is attempted again.

[0111] Step S202, if the health check fails, obtaining the historical failure number.

[0112] In step S202 of some embodiments, if the health check fails, the historical failure number is obtained, and if there is no historical failure number, the historical failure number is determined as zero.

[0113] Each synchronization tool corresponds to a historical failure number.

[0114] It can be understood that the historical failure number corresponding to the synchronization tool is obtained.

[0115] Step S203, determining the current failure number according to the historical failure number.

[0116] In step S203 of some embodiments, the historical failure number is added by one as the current failure number.

[0117] Step S204, if the current failure number is greater than the preset failure threshold, stopping the monitoring of the first database node and the second database node by the synchronization tool.

[0118] In step S204 of some embodiments, the preset failure threshold corresponding to the synchronization tool is obtained.

[0119] Optionally, each synchronization tool corresponds to a preset failure threshold.

[0120] It can be understood that the health check is performed on the first synchronization tool, the first historical failure number is obtained, the first current failure number is determined, the first preset failure threshold is obtained, it is judged whether the first current failure number is greater than the first preset failure threshold, and if yes, the monitoring of the first database node and the second database node by the first synchronization tool is stopped.

[0121] Please refer to Figure 3 In some embodiments, the database synchronization method provided by the embodiments of the present application further includes an offset verification step, which can include but is not limited to steps S301 to S303:

[0122] Step S301, querying a first offset of a first database node.

[0123] In step S301 of some embodiments, the first offset is obtained by querying the log of the first database node.

[0124] Step S302, querying a second offset of a second database node.

[0125] In step S302 of some embodiments, the second offset is obtained by querying the log of the second database node.

[0126] Step S303, performing offset verification by the first offset and the second offset to obtain a verification result.

[0127] In step S303 of some embodiments, it is judged whether the first offset and the second offset are the same, if yes, the verification is successful, and if not, the verification fails.

[0128] Optionally, if the verification fails, the larger offset and the smaller offset are determined from the first offset and the second offset, and the database node corresponding to the smaller offset is synchronized with the database node corresponding to the larger offset.

[0129] Further, if the check fails, automatically retry the synchronization, restart the synchronization tool of the database node corresponding to the smaller offset, and re-synchronize, and then observe whether the database node catches up with the offset.

[0130] Optionally, if the check fails, manually export the missing log from the database node corresponding to the larger offset and apply it to the database node corresponding to the smaller offset, and perform incremental synchronization.

[0131] It can be understood that if the difference is too large, the database node corresponding to the smaller offset re-performs full synchronization.

[0132] Taking a redis in-memory database distributed data pool as an example, Figure 4 is a structural diagram of the redis in-memory database distributed data pool, Figure 5 is a schematic diagram of the redis in-memory database distributed data pool applied to a toll station system, Figure 6 is a specific implementation flowchart of the database synchronization method provided by the embodiment of the application applied to the companion program of the redis in-memory database distributed data pool, Figure 6 The method in the embodiment can include but is not limited to the following steps.

[0133] Step 1, readerself connects the output redis.

[0134] Specifically, as shown in Figure 4 , the redis lane is a database node, and the companion program redisSync is a synchronization tool, and the companion program redisSync includes readerself, reader, and writer.

[0135] As shown in Figure 5 , wherein node 1, node 2, node 3, and node 4 are redis nodes, and the companion software is the RedisSync companion software corresponding to each node.

[0136] In some embodiments, a single-point redis+redisSync environment is built for each lane environment, and the companion RedisSync listens to the rdb and aof files of other lane redises, and synchronizes the commands to be executed in the current lane redis. Similarly, other lanes also listen to the rdb and aof files of the current lane redis and write. Thus, in all lane redises, there is a full amount of redis data, that is, a new type of redis in-memory database distributed data pool.

[0137] In some embodiments, redisSync accompanies two database nodes. Exemplarily, two redisSyncs accompany two database nodes as follows:

[0138] redis a(input) -> redisSync-1 -> redis b(output)

[0139] redis b(input) -> redisSync-2 -> redis a(output)

[0140] Wherein, the reader of redisSync listens to the aof of input redis, the reader self of redisSync listens to the aof of output redis, the writer of redisSync writes the command into output redis, redisSync-1 is the first synchronization tool, redis a is the first database node, redisSync-2 is the second synchronization tool, and redis b is the second database node.

[0141] It can be understood that the reader of redisSync-1 listens to the aof of redis a to obtain the command, the writer of redisSync-1 writes the command into output redis, and the reader self of redisSync-1 listens to the aof of redis b.

[0142] Step 2, the reader connects the input redis.

[0143] In some embodiments, the reader of redisSync-1 connects redis a, and the reader of redisSync-2 connects redis b.

[0144] Step 3, the writer connects the output redis.

[0145] In some embodiments, the writer of redisSync-1 connects redis a, and the writer of redisSync-2 connects redis b.

[0146] Step 4, it is judged whether the number of health check failures of the associated program redisSync is > 3.

[0147] In some embodiments, in order to ensure the consistency of redis data, the ping command is used for health check between redisSync and redis, and if the health check fails, the synchronization is interrupted.

[0148] If the health check of the associated program redisSync fails, it is determined whether the number of times of health check failure of the associated program redisSync is greater than 3, and if yes, the synchronization is interrupted.

[0149] Step 5: It is determined whether the number of times of health check failure of readerself is greater than 3.

[0150] In some embodiments, if the health check of readerself fails, it is determined whether the number of times of health check failure of readerself is greater than 3, and if yes, the synchronization is interrupted.

[0151] Step 6: It is determined whether the number of times of health check failure of reader is greater than 3.

[0152] In some embodiments, if the health check of reader fails, it is determined whether the number of times of health check failure of reader is greater than 3, and if yes, the synchronization is interrupted.

[0153] Step 7: It is determined whether the number of times of health check failure of writer is greater than 3.

[0154] In some embodiments, if the health check of writer fails, it is determined whether the number of times of health check failure of writer is greater than 3, and if yes, the synchronization is interrupted.

[0155] Step 8: The aof / rdb of readerself is monitored.

[0156] In some embodiments, the aof / rdb of readerself is monitored.

[0157] For example, the first persistence instruction is executed by the first database node (i.e., redis a), the aof / rdb of the first database node is monitored by the readerself of the second synchronization tool (i.e., redisync-2), and the first persistence instruction is obtained.

[0158] Step 9: The command is hash calculated.

[0159] In some embodiments, the first hash value of the first persistence instruction is calculated.

[0160] It should be noted that, in order to solve the problem of mutual monitoring and possible circular writing, the hash calculation is performed on each redis command, and the hash value is written into the redis db15 as a key, and the idempotency is ensured to prevent the occurrence of writing cycle among the redis nodes.

[0161] The synchronization tool (redisSync) listens to the aof of redis. If the aof listens to a network problem, it will not go to the hash addition and subtraction step. If the writer to redis process is out of order, the tool will retry, and after failure, it will trigger a restart, and the previous hash will be cleared. Then it reads from the latest aof and tries to write again.

[0162] Step 10: Determine whether the redis command hash exists in the db 15. If yes, the value of the key is -1, and if not, the value of the key is -1.

[0163] Specifically, the key is the hash value, and the value of the key is the verification value.

[0164] It can be understood that the aof / rdb of the first database node is monitored, and it is determined whether the redis command hash exists in the db 15 corresponding to the first database. The db 15 is a verification database.

[0165] For example, redis a executes set x 1 (i.e., the first persistence instruction), at this time, redisync-2reader listens to the aof of redis a, and writes hash (set x 1) = -1 in the db 15 of redis a. The db 15 of redis a is the first verification database.

[0166] It can be understood that it is determined whether the first verification database contains the first hash value. If not, the first verification value is set to -1, and if yes, the first verification value is reduced by 1.

[0167] Step 11: The aof / rdb is monitored by the reader.

[0168] In some embodiments, the aof / rdb of redis a (the first database node) is monitored by the reader of redisync-1 (i.e., the first synchronization tool), and the first persistence instruction is obtained.

[0169] Step 12: The reader pushes to the memory queue.

[0170] In some embodiments, the reader of redisync-1 (i.e., the first synchronization tool) pushes the instruction to the memory queue.

[0171] Step 13: The writer listens to the memory queue.

[0172] In some embodiments, the writer of redisync-1 writes the hash (set x 1) = 1 to the db 15 of redis b.

[0173] Step 14, hash the command.

[0174] In some embodiments, a first hash value of the first persistent instruction is calculated.

[0175] Step 15, determine whether the redis command hash exists in the db 15. If yes, increase the value of the key by 1. If no, set the value of the key to 1.

[0176] It can be understood that the writer of redisync-1 writes the command to redis b. Therefore, the writer of redisync-1 writes the hash (set x 1) = 1 to the db 15 (i.e., the second verification database) of redis b.

[0177] Exemplarily, the redisSync-1 reader listens to the aof of redis a and writes to the memory queue. The redisSync-1 writer writes the hash (set x 1) = 1 to the db 15 of redis b.

[0178] It can be understood that it is determined whether the second verification database contains the first hash value. If no, the second verification value is set to one. If yes, the second verification value is increased by one.

[0179] In some embodiments, it is determined whether the second verification database contains the first hash value. If the second verification database does not contain the first hash value, the second verification value is set to one. If the second verification database contains the first hash value, the second verification value corresponding to the first hash value is obtained, and the second verification value is increased by one.

[0180] Step 16, determine whether the value corresponding to the key is greater than 0. If yes, execute the command. If no, do not execute.

[0181] In some embodiments, the first idempotency verification is performed.

[0182] In some embodiments, it is determined whether the second verification value is greater than zero. If yes, it is determined that the first hash value passes the first idempotency verification. If no, it is determined that the first hash value fails the first idempotency verification.

[0183] Exemplarily, the redisSync-1writer writes hash(set x 1)=1 to the db15 of the redis b, at this time, hash(set x 1)>0, the redisSync-1 executes the set x 1 command and writes to the redis b, and the aof of the redis b also generates the set x 1.

[0184] Further, the aof of the redis b also generates the set x 1, the reader of the redisSync-1 listens to the aof of the redis b, at this time, hash(set x 1)=-1 in the db15 of the redis b, the value of hash(set x 1) is added by 1, and the result is that hash(set x 1)=0 in the db15 of the redis b, which is idempotent.

[0185] Optionally, the second idempotency verification is performed.

[0186] Further, the aof of the redis b also generates the set x 1, the reader of the redisSync-2 (i.e. the second synchronization tool) listens to the aof of the redis b (i.e. the second database node), at this time, hash(set x 1)=-1 in the db15 (i.e. the first verification database) of the redis a, the writer of the redisSync-2 adds the value of hash(set x 1) by 1, at this time, hash(set x 1)=0, which is idempotent, and the set x 1 command is not executed.

[0187] In the method, the first database node listens to the second database node to obtain a first persistence instruction and calculate a first hash value, and it is judged whether the first verification database contains the first hash value; if the first verification database does not contain the first hash value, the first verification value is set to one; if the first verification database contains the first hash value, the first verification value corresponding to the first hash value is obtained, and the first verification value is added by one; it is judged whether the first verification value is greater than zero, if yes, it is determined that the first hash value passes the second idempotency verification, and the first database node executes the first persistence instruction; if no, it is determined that the first hash value does not pass the second idempotency verification, and the first database node does not execute the first persistence instruction.

[0188] In the embodiment, each redis has full data.

[0189] In step 17, the aof file offset verification is performed.

[0190] In some embodiments, the accompanying procedure RedisSync judges the synchronization state of each redis through health check of the redis ping command and checking the offset of aof monitoring.

[0191] It should be noted that, as Figure 4 shown, when the redis of lane 1 is down, the lane is unavailable in the "highway toll system", and other lanes need to carry his business. Each lane's redis has full data, so other lanes have the data of lane 1. The business of vehicle a can continue to be carried, and the vehicle can be released normally.

[0192] Exemplarily, the Redis cluster architecture diagram is as shown in Figure 7 , wherein the master node refers to redis-master, the slave node refers to redis-slave, the sentinel refers to redis-sentinel, and the hash ring refers to hash ring.

[0193] It can be understood that when the master and slave of lane 1 of the redis cluster are down at the same time, the lane is unavailable in the "highway toll system", and other lanes need to carry his business. Since the redis cluster is based on hash slot, each node collectively persists data, so other lanes may not have the data of lane 1. The vehicle cannot be released normally.

[0194] Exemplarily, the redis master-slave sentinel architecture diagram is as shown in Figure 8 , wherein the master node refers to redis-master, the slave node refers to redis-slave, the sentinel refers to redis-sentinel, and the sentinel cluster refers to Redis sentinel cluster.

[0195] It can be understood that when the master and slave of lane 1 of the redis master-slave sentinel are down at the same time, the lane is unavailable in the "highway toll system", and other lanes need to carry his business. Since the redis master-slave sentinel is responsible for the data respectively, so other lanes do not have the data of lane 1. The vehicle a cannot be released normally.

[0196] Therefore, the embodiment has the characteristic of independent data tags based on real-time business cache data of the highway toll system, and the "new redis memory database distribution" can pass through the redis data of each lane, and each redis has full data. When a single or multiple redis appears to be down, only one redis can handle all lane services. Compared with redis cluster and redis master-slave sentinel cluster, the advantage is obvious.

[0197] In the embodiment, RedisSync includes the following functions:

[0198] 1. Compatible with Redis versions from 2.8 to 7.2, and support various redis deployment methods, including single machine, master-slave, sentinel and cluster.

[0199] 2. RedisSync associated with a single node redis instance startup has monitoring and operation and maintenance monitoring capabilities such as monitoring the running state of redis and application data verification.

[0200] 3. RedisSync associated with two or more node redis instance startups not only provides operation and maintenance monitoring capabilities, but also realizes real-time synchronization of redis instance memory cache data while two or more node redis instances can read and write operations, and at any time, any node can get all node cache full memory data.

[0201] 4. RedisSync provides data migration capability, which can detect and self-heal (or manually maintain) associated redis that fails without stopping business applications.

[0202] Please refer to Figure 9 The embodiment of the application also provides a database synchronization device, which can implement the above-mentioned database synchronization method, and the device comprises:

[0203] The first execution module 901 is configured to execute the first persistence instruction through the first database node;

[0204] The listening module 902 is configured to listen to the first database node through the second database node to obtain the first persistence instruction;

[0205] The first verification module 903 is configured to calculate a first hash value of the first persistence instruction, and perform first idempotency verification through the first hash value;

[0206] The second execution module 904 is configured to execute the first persistence instruction through the second database node if the first hash value passes the first idempotency verification;

[0207] The second verification module 905 is configured to perform second idempotency verification by using the first hash value, and the second idempotency verification is used to prevent the first database node and the second database node from performing a loop.

[0208] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0209] The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0210] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0211] Please refer to Figure 10 , Figure 10 The electronic device of another embodiment is illustrated, and the electronic device includes:

[0212] The processor 1001 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present application.

[0213] The memory 1002 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the database synchronization method in the embodiments of the present application.

[0214] The input / output interface 1003 is configured to implement information input and output.

[0215] The communication interface 1004 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0216] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device.

[0217] The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are connected to each other through the bus 1005 to realize the communication connection between the device.

[0218] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the database synchronization method.

[0219] It can be understood that the contents in the above method embodiments are applicable to the storage medium embodiments, the storage medium embodiments specifically realize the functions of the above method embodiments, and the beneficial effects achieved by the storage medium embodiments are the same as the beneficial effects achieved by the above method embodiments.

[0220] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0221] The database synchronization method, the database synchronization device, the electronic device, and the storage medium provided by the embodiment of the present application can execute a first persistence instruction through a first database node, listen to the first database node through a second database node to obtain the first persistence instruction, calculate a first hash value of the first persistence instruction, perform a first idempotency verification through the first hash value, execute the first persistence instruction through the second database node if the first hash value passes the first idempotency verification, perform a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent the first database node and the second database node from executing in a loop. The embodiment of the present application can enable each database node to have full data, and if a database node is down, only one database node is needed to undertake all services.

[0222] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0223] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0224] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0225] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0226] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and above-described drawings of the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0227] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0228] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0229] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0230] In addition, each functional unit in each embodiment of the application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0231] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0232] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A database synchronization method, characterized by, The method comprises the following steps: executing a first persistent instruction by a first database node; listening to the first database node by a second database node to obtain the first persistent instruction; calculating a first hash value of the first persistent instruction and performing a first idempotency verification by the first hash value; if the first hash value passes the first idempotency verification, executing the first persistent instruction by the second database node; performing a second idempotency verification by the first hash value, which is used to prevent the first database node and the second database node from executing in a loop; the calculating of the first hash value of the first persistent instruction and the performing of the first idempotency verification by the first hash value comprises: calculating the first hash value of the first persistent instruction; judging whether a second verification database contains the first hash value; if the second verification database does not contain the first hash value, setting a second verification value to one; if the second verification database contains the first hash value, obtaining a second verification value corresponding to the first hash value and adding one to the second verification value; judging whether the second verification value is greater than zero, if yes, determining that the first hash value passes the first idempotency verification, if not, determining that the first hash value does not pass the first idempotency verification.

2. The method of claim 1, wherein, the executing of the first persistent instruction by the first database node comprises: listening to the first persistent instruction of the first database node by a second synchronization tool and calculating a first hash value of the first persistent instruction; judging whether a first verification database contains the first hash value, if not, setting a first verification value to negative one, if yes, reducing one from the first verification value, the first verification value corresponding to the first hash value.

3. The method of claim 1, wherein, the executing of the first persistent instruction by the second database node if the first hash value passes the first idempotency verification comprises: listening to the first persistent instruction of the second database node by a first synchronization tool and calculating a first hash value of the first persistent instruction; judging whether a second verification database contains the first hash value, if not, setting a second verification value to negative one, if yes, reducing one from the second verification value, the second verification value corresponding to the first hash value.

4. The method of claim 1, wherein, the performing of the second idempotency verification by the first hash value comprises: listening to the second database node by the first database node to obtain a first persistent instruction and calculating the first hash value; judging whether a first verification database contains the first hash value; if the first verification database does not contain the first hash value, setting a first verification value to one; if the first verification database contains the first hash value, obtaining a first verification value corresponding to the first hash value and adding one to the first verification value; judging whether the first verification value is greater than zero, if yes, determining that the first hash value passes the second idempotency verification and the first database node executes the first persistent instruction; if not, determining that the first hash value does not pass the second idempotency verification and the first database node does not execute the first persistent instruction.

5. The method of claim 1, wherein, the method further comprises: performing a health check on a synchronization tool configured to listen to the first database node and the second database node; if the health check fails, obtaining a historical failure number; determining a current failure number according to the historical failure number; if the current failure number is greater than a preset failure threshold, stopping the synchronization tool from listening to the first database node and the second database node.

6. The method of claim 1, wherein, The method further comprises: querying a first offset of the first database node; querying a second offset of the second database node; performing offset verification on the first offset and the second offset to obtain a verification result.

7. A database synchronization apparatus, characterized by comprising: The apparatus comprises: a first execution module configured to execute a first persistent instruction via a first database node; a listening module configured to listen to the first database node via a second database node to obtain the first persistent instruction; a first verification module configured to calculate a first hash value of the first persistent instruction and perform first idempotency verification on the first hash value; a second execution module configured to execute the first persistent instruction via the second database node if the first hash value passes the first idempotency verification; a second verification module configured to perform second idempotency verification on the first hash value, the second idempotency verification being configured to prevent the first database node and the second database node from executing in a loop; the first verification module is specifically configured to: calculate the first hash value of the first persistent instruction; determine whether a second verification database contains the first hash value; if the second verification database does not contain the first hash value, set a second verification value to one; if the second verification database contains the first hash value, obtain a second verification value corresponding to the first hash value and increase the second verification value by one; determine whether the second verification value is greater than zero, and if yes, determine that the first hash value passes the first idempotency verification, and if no, determine that the first hash value does not pass the first idempotency verification.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method in any one of claims 1 to 6.

Images