Database synchronization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a database synchronization method and device, electronic equipment and a storage medium, and belongs to the technical field of databases. The method comprises the following steps: executing a first persistence instruction through a first database node; monitoring the first database node through the second database node to obtain a first persistence instruction; calculating a first hash value of the first persistence instruction, and performing first idempotent verification through the first hash value; if the first hash value passes the first idempotence verification, executing the first persistence instruction through the second database node; and performing second idempotent verification through the first hash value, wherein the second idempotent verification is used for preventing the first database node and the second database node from generating an execution loop. According to the embodiment of the invention, each database node can have full data, and if the database node is down, only one database node is needed to undertake all services, so that the fault tolerance of the database is improved, and the service processing efficiency and the working efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular, to a database synchronization method, apparatus, electronic device, and storage medium. Background Art

[0002] In related technologies, database pools usually adopt a master-slave architecture. If a certain database node fails, the data in it may not exist in other databases, and the tasks responsible for the failed database node cannot be continued, resulting in a low system fault tolerance and affecting the business processing efficiency.

[0003] In summary, the technical problems existing in related technologies need to be improved. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a database synchronization method, apparatus, electronic device, and storage medium to ensure that each database node has all the data, which is beneficial to improving the fault tolerance of the database.

[0005] To achieve the above object, on the one hand, an embodiment of this application proposes a database synchronization method, which includes the following steps:

[0006] Execute a first persistence instruction through a first database node;

[0007] Listen to the first database node through a second database node to obtain the first persistence instruction;

[0008] Calculate a first hash value of the first persistence instruction, and perform a first idempotency verification through the first hash value;

[0009] If the first hash value passes the first idempotency verification, execute the first persistence instruction through the second database node;

[0010] Perform a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent an execution loop between the first database node and the second database node.

[0011] In some embodiments, the step of executing a first persistence instruction through a first database node includes:

[0012] Listen to the first persistence instruction of the first database node through a second synchronization tool, and calculate a first hash value of the first persistence instruction;

[0013] [[ID=4३]]Judge whether a first verification database contains the first hash value. If not, set a first verification value to -1. If so, subtract 1 from the first verification value, and the first verification value corresponds to the first hash value.

[0014] In some embodiments, calculating a first hash value of the first persistence instruction and performing a first idempotency verification through the first hash value includes:

[0015] Calculate a first hash value of the first persistence instruction;

[0016] Determine whether a second verification database contains the first hash value;

[0017] If the second verification database does not contain the first hash value, set a second verification value to one;

[0018] If the second verification database contains the first hash value, obtain the second verification value corresponding to the first hash value and increment the second verification value by one;

[0019] Determine whether the second verification value is greater than zero. If so, determine that the first hash value passes the first idempotency verification; if not, determine that the first hash value fails the first idempotency verification.

[0020] In some embodiments, if the first hash value passes the first idempotency verification, executing the first persistence instruction through the second database node includes:

[0021] Monitor the first persistence instruction of the second database node through a first synchronization tool and calculate a first hash value of the first persistence instruction;

[0022] Determine whether the second verification database contains the first hash value. If not, set the second verification value to negative one; if so, decrement the second verification value by one. The second verification value corresponds to the first hash value.

[0023] In some embodiments, performing a second idempotency verification through the first hash value includes:

[0024] Monitor the second database node through the first database node to obtain a first persistence instruction and calculate the first hash value;

[0025] Determine whether a first verification database contains the first hash value;

[0026] If the first verification database does not contain the first hash value, set a first verification value to one;

[0027] If the first verification database contains the first hash value, obtain the first verification value corresponding to the first hash value and increment the first verification value by one;

[0028] Determine whether the first verification value is greater than zero. If so, determine that the first hash value passes the second idempotency verification, and the first database node executes the first persistence instruction;

[0029] Otherwise, it is determined that the first hash value fails the second idempotency verification, and the first database node does not execute the first persistence instruction.

[0030] In some embodiments, the method further includes:

[0031] Performing a health check on a synchronization tool, where the synchronization tool is used to monitor the first database node and the second database node;

[0032] If the health check fails, obtaining the historical failure count;

[0033] Determining the current failure count according to the historical failure count;

[0034] If the current failure count is greater than a preset failure threshold, stopping the synchronization tool from monitoring the first database node and the second database node.

[0035] In some embodiments, the method further includes:

[0036] Querying a first offset of the first database node;

[0037] Querying a second offset of the second database node;

[0038] Performing offset verification through the first offset and the second offset to obtain a verification result.

[0039] To achieve the above object, another aspect of the embodiments of the present application provides a database synchronization device, the device includes:

[0040] A first execution module, configured to execute a first persistence instruction through a first database node;

[0041] A monitoring module, configured to monitor the first database node through a second database node to obtain a first persistence instruction;

[0042] A first verification module, configured to calculate a first hash value of the first persistence instruction and perform a first idempotency verification through the first hash value;

[0043] A second execution module, configured to, if the first hash value passes the first idempotency verification, execute the first persistence instruction through the second database node;

[0044] A second verification module, configured to perform a second idempotency verification through the first hash value, where the second idempotency verification is used to prevent execution loops from occurring between the first database node and the second database node.

[0045] To achieve the above object, on the other hand, an embodiment of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the foregoing method is implemented.

[0046] To achieve the above object, on the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the foregoing method is implemented.

[0047] The embodiments of the present application at least include the following beneficial effects: The present application provides a database synchronization method, a device, an electronic device, and a storage medium. In this solution, a first database node executes a first persistence instruction; a second database node listens to the first database node to obtain the first persistence instruction; calculates a first hash value of the first persistence instruction, and performs a first idempotency verification through the first hash value; if the first hash value passes the first idempotency verification, the second database node executes the first persistence instruction; performs a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent an execution loop from occurring between the first database node and the second database node. The embodiments of the present application can ensure that each database node has a full amount of data. If a database node fails, only one database node is required to undertake all services. Description of the Drawings

[0048] Figure 1 is a flowchart of the database synchronization method provided by the embodiment of the present application;

[0049] Figure 2 is a flowchart of the health check step of the database synchronization method provided by the embodiment of the present application;

[0050] Figure 3 is a flowchart of the offset verification step of the database synchronization method provided by the embodiment of the present application;

[0051] Figure 4 is an architecture diagram of a distributed data pool of a redis in-memory database provided by the embodiment of the present application;

[0052] Figure 5 is a schematic diagram of the application of the distributed data pool of the redis in-memory database provided by the embodiment of the present application to a toll station system;

[0053] Figure 6 is a specific implementation flowchart when the database synchronization method provided by the embodiment of the present application is applied to an accompanying program of a distributed data pool of a redis in-memory database;

[0054] Figure 7It is the Redis cluster architecture diagram provided by the embodiments of the present application;

[0055] Figure 8 It is the master-slave sentinel architecture diagram of redis provided by the embodiments of the present application;

[0056] Figure 9 It is the structural schematic diagram of the database synchronization device provided by the embodiments of the present application;

[0057] Figure 10 It is the hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application detailed in the appended claims.

[0059] It can be understood that the terms "first", "second", etc. used in the present application can be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be called the second information, and similarly, the second information can also be called the first information. Depending on the context, the words "if", "when" as used herein can be interpreted as "when...", "when...", or "in response to determining".

[0060] The terms "at least one", "a plurality", "each", "any one", etc. used in the present application, at least one includes one, two or more than two, a plurality includes two or more than two, each refers to each of the corresponding plurality, and any one refers to any one of the plurality.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] Before elaborating on the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0063] 1) Redis (Remote Dictionary Server) is an open-source, high-performance key-value storage database that is widely used in scenarios such as caching, message queues, and session storage. Redis has very high performance, supports rich data structures and various operations, and can effectively handle the real-time storage and access of a large amount of data.

[0064] In related technologies, the database pool usually adopts a master-slave architecture. If a certain database node fails, the data in it may not exist in other databases, and the tasks responsible for the failed database node cannot be continued, resulting in a low system fault tolerance and affecting the business processing efficiency.

[0065] In summary, the technical problems existing in related technologies need to be improved.

[0066] In view of this, the embodiments of the present application provide a database synchronization method, device, device, and medium. This solution executes a first persistence instruction through a first database node; listens to the first database node through a second database node to obtain the first persistence instruction; calculates the first hash value of the first persistence instruction, and performs a first idempotency verification through the first hash value; if the first hash value passes the first idempotency verification, then the second database node executes the first persistence instruction; performs a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent execution loops from occurring between the first database node and the second database node. The embodiments of the present application can enable each database node to have all the data. If a database node fails, only one database node is required to undertake all the services.

[0067] The database synchronization method provided by the embodiments of the present application relates to the field of database technologies. The database synchronization method provided by the embodiments of the present application can be applied to a terminal, a server, or software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the database synchronization method, etc., but is not limited to the above forms.

[0068] This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] It should be noted that in each specific embodiment of this application, when it comes to relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of this application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of this application will be obtained.

[0070] Figure 1 is an optional flowchart of the database synchronization method provided by the embodiments of this application, Figure 1 The method in may include but is not limited to steps S101 to S105.

[0071] Step S101, execute a first persistence instruction through a first database node.

[0072] Specifically, the first persistence instruction is used to perform a write operation on the database node, and the first persistence instruction is generated by the first database node.

[0073] Optionally, by executing the first persistence instruction through the first database node, an rdb file and an aof file are generated.

[0074] Optionally, when the first database node generates and executes the first persistence instruction, the hash value of the first persistence instruction is stored in the first verification database.

[0075] In some embodiments, a second synchronization tool is used to monitor the first persistence instruction of the first database node, calculate the first hash value of the first persistence instruction; determine whether the first verification database contains the first hash value, and if not, set the first verification value to -1, and if so, decrement the first verification value by 1. The first verification value corresponds to the first hash value, and the second synchronization tool is used to monitor the write operations of the first database node, but not to synchronize the first persistence instruction to the second database node.

[0076] Optionally, the first verification value and the first hash value adopt a key-value pair structure, with the first hash value as the key and the first verification value as the value.

[0077] In this embodiment, the first persistence instruction is executed by the first database node, preparing for subsequent synchronization of the first persistence instruction and the second idempotency verification by the second database node.

[0078] Step S102: The second database node monitors the first database node to obtain the first persistence instruction.

[0079] Optionally, the second database node uses a first synchronization tool to monitor the first database node to obtain the first persistence instruction. The first synchronization tool is used to monitor the write operations of the first database node and synchronize the first persistence instruction to the second database node.

[0080] Furthermore, the second database node uses the first synchronization tool to monitor the rdb file and the aof file of the first database node to obtain the first persistence instruction.

[0081] This application does not make specific limitations on the monitoring method, which can be flexibly selected according to actual detection needs. Exemplarily, a synchronization tool can be used for monitoring, log monitoring can be performed, monitoring can also be carried out through a message middleware, or monitoring can be performed through a publish-subscribe mode.

[0082] In this embodiment, the second database node monitors the first database node to obtain the first persistence instruction, enabling the second database node to promptly know the persistence operations of the first database node and improving the fault tolerance of the system.

[0083] Step S103: Calculate the first hash value of the first persistence instruction and perform the first idempotency verification through the first hash value.

[0084] Specifically, the first idempotency verification is used to verify whether the first persistence instruction has been executed by the second database node, preventing the second database node from repeatedly executing the same operation.

[0085] In some embodiments, calculate the first hash value of the first persistent instruction and perform the first idempotency verification through the first hash value.

[0086] Optionally, when the second database node executes the instruction for the first time, store the hash value of the instruction in the second verification database corresponding to the second database node. The second verification database is used to store the hash values of the executed instructions.

[0087] It can be understood that before the second database node executes the instruction again, the hash value already exists in the second verification database, so the instruction execution is rejected.

[0088] Further, store the hash value as the key of the key-value pair in the verification database, and use the verification value for the value of the key-value pair. The verification value is used to verify the idempotency of the instruction.

[0089] In some embodiments, calculate the first hash value of the first persistent instruction; determine whether the second verification database contains the first hash value; if the second verification database does not contain the first hash value, set the second verification value to one; if the second verification database contains the first hash value, obtain the second verification value corresponding to the first hash value and increment the second verification value by one; determine whether the second verification value is greater than zero. If so, determine that the first hash value passes the first idempotency verification, and if not, determine that the first hash value fails the first idempotency verification.

[0090] In this embodiment, calculating the first hash value of the first persistent instruction and performing the first idempotency verification through the first hash value helps prevent the second database node from repeatedly executing the same instruction operation and achieve the idempotency of the operation.

[0091] Step S104, if the first hash value passes the first idempotency verification, execute the first persistent instruction through the second database node.

[0092] In some embodiments, execute the first persistent instruction on the second database node. Optionally, execute the instruction in a local transaction or a distributed transaction.

[0093] Optionally, listen for the first persistent instruction of the second database node through the first synchronization tool, calculate the first hash value of the first persistent instruction; determine whether the second verification database contains the first hash value. If not, set the second verification value to negative one, and if so, decrement the second verification value by one. The second verification value corresponds to the first hash value.

[0094] It can be understood that after performing the first idempotency verification through the first hash value, the second verification value is 1. After executing the first persistent instruction, the second verification value is decremented by one and the second verification value becomes zero, which helps ensure the idempotency of the operation.

[0095] In this embodiment, if the first hash value passes the first idempotency verification, the first persistence instruction is executed through the second database node to synchronize the database nodes, achieve cross-node data consistency, improve system availability and fault tolerance, and improve business processing efficiency and work efficiency.

[0096] Step S105, perform a second idempotency verification through the first hash value.

[0097] Specifically, the second idempotency verification is used to prevent an execution loop from occurring between the first database node and the second database node.

[0098] In some embodiments, when the first database node generates and executes the first persistence instruction, the hash value of the first persistence instruction is stored in the first verification database. Then, when the first database node listens to the second database node and obtains the first persistence instruction, a second idempotency verification is performed to prevent an execution loop.

[0099] Optionally, before the first database node executes an instruction, it is determined whether the first verification database contains the hash value of the instruction. If not, the instruction is executed; if so, the execution of the instruction is rejected.

[0100] It can be understood that the second idempotency verification helps prevent an infinite loop of "first database node → second database node → first database node → second database node".

[0101] Optionally, a global transaction ID can be used to replace the hash value to reduce the computational overhead.

[0102] In some embodiments, the first database node listens to the second database node to obtain the first persistence instruction and calculates the first hash value; it is determined whether the first verification database contains the first hash value; if the first verification database does not contain the first hash value, the first verification value is set to one; if the first verification database contains the first hash value, the first verification value corresponding to the first hash value is obtained and the first verification value is incremented by one; it is determined whether the first verification value is greater than zero. If so, it is determined that the first hash value passes the second idempotency verification, and the first database node executes the first persistence instruction; if not, it is determined that the first hash value does not pass the second idempotency verification, and the first database node does not execute the first persistence instruction.

[0103] In this embodiment, the second idempotency verification is performed through the first hash value. Through two-way idempotency verification, it is beneficial to avoid an infinite loop between database nodes, thereby improving the stability and consistency of the system.

[0104] Steps S101 to S105 shown in the embodiments of the present application execute a first persistence instruction through a first database node; listen to the first database node through a second database node to obtain the first persistence instruction; calculate a first hash value of the first persistence instruction, and perform a first idempotency verification through the first hash value, which is beneficial to preventing the second database node from repeatedly executing the same instruction operation and realizing the idempotency of the operation; if the first hash value passes the first idempotency verification, then execute the first persistence instruction through the second database node to achieve the synchronization of database nodes, realize cross-node data consistency, improve system availability and fault tolerance, and improve business processing efficiency and work efficiency; perform a second idempotency verification through the first hash value, and through two-way idempotency verification, it is beneficial to avoid infinite loops between database nodes, thereby improving the stability and consistency of the system.

[0105] Please refer to Figure 2 , in some embodiments, the database synchronization method provided by the embodiments of the present application further includes a health check step, and the health check step may include but is not limited to steps S201 to S204:

[0106] Step S201, perform a health check on the synchronization tool.

[0107] Specifically, the synchronization tool is used to listen to the first database node and the second database node.

[0108] In step S201 of some embodiments, perform a health check on the first synchronization tool and the second synchronization tool.

[0109] Optionally, the synchronization tool listens to the aof of the database node. If a network problem is detected by the aof, the health check fails.

[0110] It should be noted that if there is a problem during the execution of the instruction, the synchronization tool will retry. After the retry fails, it will trigger the restart of the synchronization tool, clear the previous hash value and verification value in the database, synchronize and read from the latest aof again, and then try to write.

[0111] Step S202, if the health check fails, obtain the historical failure count.

[0112] In step S202 of some embodiments, if the health check fails, obtain the historical failure count. If there is no historical failure count, determine the historical failure count as zero.

[0113] Among them, each synchronization tool corresponds to a historical failure count.

[0114] It can be understood that obtain the historical failure count corresponding to the synchronization tool.

[0115] Step S203: Determine the current failure count according to the historical failure count.

[0116] In step S203 of some embodiments, increment the historical failure count by one as the current failure count.

[0117] Step S204: If the current failure count is greater than the preset failure threshold, stop the synchronization tool from listening to the first database node and the second database node.

[0118] In step S204 of some embodiments, obtain the preset failure threshold corresponding to the synchronization tool.

[0119] Optionally, each synchronization tool corresponds to a preset failure threshold.

[0120] It can be understood that perform a health check on the first synchronization tool, obtain the first historical failure count, determine the first current failure count, obtain the first preset failure threshold, and determine whether the first current failure count is greater than the first preset failure threshold. If so, stop the first synchronization tool from listening to the first database node and the second database node.

[0121] Please refer to Figure 3 , in some embodiments, the database synchronization method provided by the embodiments of the present application further includes an offset verification step, and the offset verification step may include but is not limited to steps S301 to S303:

[0122] Step S301: Query the first offset of the first database node.

[0123] In step S301 of some embodiments, query the log of the first database node to obtain the first offset.

[0124] Step S302: Query the second offset of the second database node.

[0125] In step S302 of some embodiments, query the log of the second database node to obtain the second offset.

[0126] Step S303: Perform offset verification through the first offset and the second offset to obtain a verification result.

[0127] In step S303 of some embodiments, determine whether the first offset is the same as the second offset. If so, the verification is successful; if not, the verification fails.

[0128] Optionally, if the verification fails, determine the larger offset and the smaller offset from the first offset and the second offset, and synchronize the database node corresponding to the larger offset through the database node corresponding to the smaller offset.

[0129] Further, if the verification fails, automatically retry the synchronization, restart the synchronization tool of the database node corresponding to the smaller offset, and perform the synchronization again. Then observe whether the database node catches up with the offset.

[0130] Optionally, if the verification fails, manually export the missing logs from the database node corresponding to the larger offset and apply them to the database node corresponding to the smaller offset for incremental synchronization.

[0131] It can be understood that if the difference is too large, the database node corresponding to the smaller offset performs a full synchronization again.

[0132] Taking the redis in-memory database distributed data pool as an example, Figure 4 is the architecture diagram of the redis in-memory database distributed data pool. Figure 5 is the schematic diagram of the redis in-memory database distributed data pool applied to the toll station system. Figure 6 is a specific implementation flowchart when the database synchronization method provided by the embodiment of the present application is applied to the companion program of the redis in-memory database distributed data pool. Figure 6 The method in

[0133] Step 1, readerself connects to output redis.

[0134] Specifically, as Figure 4 shown, the redis lane is the database node, and the companion program redisSync is the synchronization tool. The companion program redisSync includes readerself, reader, and writer.

[0135] As Figure 5 shown, among them, Node 1, Node 2, Node 3, and Node 4 are all redis nodes, and the companion software is the RedisSync companion software corresponding to each node.

[0136] In some embodiments, a single-point redis + redisSync environment is built for each lane environment. The companion RedisSync listens to the rdb and aof files of the redis in other lanes, and synchronously executes the commands in the redis of the current lane. Similarly, other lanes will also listen to the redis rdb and aof files of the current lane and write them. Thus, in the redis of all lanes, there is a full set of redis data, that is, a new redis in-memory database distributed data pool.

[0137] In some embodiments, redisSync companions two database nodes. Exemplarily, two redisSync companions two database nodes as follows:

[0138] Redis a (input) -> RedisSync-1 -> Redis b (output)

[0139] Redis b (input) -> RedisSync-2 -> Redis a (output)

[0140] Among them, the reader of RedisSync listens to the AOF of input Redis, the readerself of RedisSync listens to the AOF of output Redis, the writer of RedisSync writes commands into output Redis, RedisSync-1 is the first synchronization tool, Redis a is the first database node, RedisSync-2 is the second synchronization tool, and Redis b is the second database node.

[0141] It can be understood that the reader of RedisSync-1 listens to the AOF of Redis a to obtain instructions, the writer of RedisSync-1 writes commands into output Redis, and the readerself of RedisSync-1 listens to the AOF of Redis b.

[0142] Step 2, the reader connects to input Redis.

[0143] In some embodiments, the reader of RedisSync-1 connects to Redis a, and the reader of RedisSync-2 connects to Redis b.

[0144] Step 3, the writer connects to output Redis.

[0145] In some embodiments, the writer of RedisSync-1 connects to Redis a, and the writer of RedisSync-2 connects to Redis b.

[0146] Step 4, determine whether the number of failed health checks of the associated program RedisSync is > 3.

[0147] In some embodiments, to ensure the consistency of Redis data, health checks are performed between RedisSync and Redis through the ping command. If the health check fails, the synchronization will be interrupted.

[0148] Among them, if the associated program redisSync fails the health check, it is judged whether the number of times the associated program redisSync fails the health check is > 3. If so, the synchronization is interrupted.

[0149] Step 5, judge whether the number of times readerself fails the health check is > 3.

[0150] In some embodiments, if readerself fails the health check, it is judged whether the number of times readerself fails the health check is > 3. If so, the synchronization is interrupted.

[0151] Step 6, judge whether the number of times reader fails the health check is > 3.

[0152] In some embodiments, if reader fails the health check, it is judged whether the number of times reader fails the health check is > 3. If so, the synchronization is interrupted.

[0153] Step 7, judge whether the number of times writer fails the health check is > 3.

[0154] In some embodiments, if writer fails the health check, it is judged whether the number of times writer fails the health check is > 3. If so, the synchronization is interrupted.

[0155] Step 8, use readerself to monitor aof / rdb.

[0156] In some embodiments, use readerself to monitor aof / rdb.

[0157] Exemplarily, execute the first persistence instruction through the first database node (i.e., redis a), and use the readerself of the second synchronization tool (i.e., redisync-2) to monitor the aof / rdb of the first database node to obtain the first persistence instruction.

[0158] Step 9, perform a hash calculation on the command.

[0159] In some embodiments, calculate the first hash value of the first persistence instruction.

[0160] It should be noted that to solve the problem of mutual monitoring that may cause circular writing, a hash calculation is performed on each redis command and used as the key to write into redis db15. By ensuring idempotency, the occurrence of circular writing between redis nodes can be prevented.

[0161] Among them, the synchronization tool (redisSync) listens to the AOF of Redis. If there is a network problem detected by the AOF listener, it will not proceed to the hash addition or subtraction steps. If there is a problem during the process of the tool writer sending data to Redis, the tool will retry. After failure, it will trigger a restart, clear the previous hash to 0, read from the latest AOF again, and then attempt to write again.

[0162] Step 10: Check if the Redis command hash exists in database 15. If it does, decrement the value of the key by 1; if not, set the value of the key to -1.

[0163] Specifically, the key is the hash value, and the value of the key is the verification value.

[0164] It can be understood that by listening to the AOF / RDB of the first database node, it is determined whether the Redis command hash exists in database 15 corresponding to the first database. Here, database 15 is the verification database.

[0165] Exemplarily, Redis A executes SET x 1 (i.e., the first persistence instruction). At this time, redisync-2readerself listens to the AOF of Redis A and writes hash(SET x 1)=-1 in database 15 of Redis A. Here, database 15 of Redis A is the first verification database.

[0166] It can be understood that it is determined whether the first verification database contains the first hash value. If not, set the first verification value to -1; if so, decrement the first verification value by 1.

[0167] Step 11: Use the reader to listen to the AOF / RDB.

[0168] In some embodiments, the reader of redisync-1 (i.e., the first synchronization tool) listens to the AOF / RDB of Redis A (the first database node) to obtain the first persistence instruction.

[0169] Step 12: Push it to the memory queue through the reader.

[0170] In some embodiments, the reader of redisync-1 (i.e., the first synchronization tool) pushes the instruction to the memory queue.

[0171] Step 13: Use the writer to listen to the memory queue.

[0172] In some embodiments, the writer of redisync-1 (i.e., the first synchronization tool) listens to the memory queue and writes hash(set x 1)=1 to db15 of redis b.

[0173] Step 14, perform a hash calculation on the command.

[0174] In some embodiments, calculate the first hash value of the first persistence instruction.

[0175] Step 15, determine whether the redis command hash exists in db 15? If so, increment the value of the key by 1; if not, set the value of the key to 1.

[0176] It can be understood that the writer of redisync-1 writes the command to redis b. Therefore, the writer of redisync-1 writes hash(set x 1)=1 to db15 of redis b (i.e., the second verification database).

[0177] Exemplarily, the redisSync-1reader listens to the aof of redis a, writes it to the memory queue, and the redisSync-1writer writes hash(set x 1)=1 to db15 of redis b.

[0178] It can be understood that determine whether the second verification database contains the first hash value. If not, set the second verification value to one; if so, increment the second verification value by one.

[0179] In some embodiments, determine whether the second verification database contains the first hash value; if the second verification database does not contain the first hash value, set the second verification value to one; if the second verification database contains the first hash value, obtain the second verification value corresponding to the first hash value and increment the second verification value by one.

[0180] Step 16, determine whether the value corresponding to the key>0? If so, execute the command; if not, do not execute.

[0181] Among them, perform the first idempotency verification.

[0182] In some embodiments, determine whether the second verification value is greater than zero. If so, determine that the first hash value passes the first idempotency verification; if not, determine that the first hash value fails the first idempotency verification.

[0183] Exemplarily, redisSync-1writer writes hash(set x 1)=1 to db15 of redis b. At this time, hash(set x 1)>0, and redisSync-1 executes the set x 1 command and writes it to redis b. At the same time, the aof of redis b will also generate set x 1.

[0184] Furthermore, the aof of redis b will also generate set x 1. The readerself of redisSync-1 monitors the aof of redis b. At this time, in db 15 of redis b, hash(set x 1)=-1, so the value of hash(set x1) is incremented by 1. As a result, hash(set x 1)=0 in db15 of redis b, which is idempotent.

[0185] Optionally, perform the second idempotency verification.

[0186] Furthermore, the aof of redis b will also generate set x 1. The reader of redisSync-2 (i.e., the second synchronization tool) monitors the aof of redis b (i.e., the second database node). At this time, in db 15 of redis a (i.e., the first verification database), hash(set x 1)=-1. The writer of redisSync-2 increments the value of hash(set x 1) by 1. At this time, hash(set x 1)=0, which is idempotent, and the set x 1 command is not executed.

[0187] Among them, the first database node monitors the second database node to obtain the first persistence instruction, calculates the first hash value, and determines whether the first verification database contains the first hash value; if the first verification database does not contain the first hash value, the first verification value is set to one; if the first verification database contains the first hash value, the first verification value corresponding to the first hash value is obtained, and the first verification value is incremented by one; it is determined whether the first verification value is greater than zero. If so, it is determined that the first hash value passes the second idempotency verification, and the first database node executes the first persistence instruction; if not, it is determined that the first hash value fails the second idempotency verification, and the first database node does not execute the first persistence instruction.

[0188] In this embodiment, each redis has full data.

[0189] Step 17, perform aof file offset verification.

[0190] In some embodiments, the associated program RedisSync determines the synchronization status of each redis by means of the health check of the redis ping command and the verification of the offset of the aof monitoring.

[0191] It should be noted that, as Figure 4 shown, when the redis in Lane 1 fails, in the "Highway Toll System", this lane becomes unavailable, and other lanes need to carry its business. Each lane's redis has the full amount of data, so other lanes have the data of Lane 1. It can continue to carry the business of Vehicle A and release it normally.

[0192] Exemplarily, the Redis cluster architecture diagram is as Figure 7 shown, where the master node refers to redis-master, the slave node refers to redis-slave, the sentinel refers to redis-sentinel, and the hash ring is the hash ring.

[0193] It can be understood that when both the master and slave of Lane 1 in the redis cluster fail, in the "Highway Toll System", this lane becomes unavailable, and other lanes need to carry its business. Since the redis cluster distributes and persists data jointly based on hash slots, other lanes may not have the data of Lane 1. As a result, the vehicle cannot be released normally.

[0194] Exemplarily, the redis master-slave sentinel architecture diagram is as Figure 8 shown, where the master node refers to redis-master, the slave node refers to redis-slave, the sentinel refers to redis-sentinel, and the sentinel cluster is the Redis sentinel cluster.

[0195] It can be understood that when both the master and slave of Lane 1 in the redis master-slave sentinel fail, in the "Highway Toll System", this lane becomes unavailable, and other lanes need to carry its business. Since the redis master-slave sentinel is responsible for different data respectively,, other lanes do not have the data of Lane 1. As a result, Vehicle A cannot be released normally.

[0196] Therefore, based on the characteristics of real-time service cache data in the highway toll collection system in this embodiment, with independent data tags, the "new Redis in-memory database distributed" can connect the Redis data of each lane, and each Redis has the full amount of data. When a single or multiple Redis instances go down, only one Redis is needed to take over the business of all lanes. Compared with Redis Cluster and Redis master-slave sentinel clusters, the advantages are obvious.

[0197] In this embodiment, RedisSync includes the following functions:

[0198] 1. Compatible with Redis versions from 2.8 to 7.2 and support various Redis deployment methods, including single machine, master-slave, sentinel, and cluster.

[0199] 2. RedisSync started with a single-node Redis instance has the operation and maintenance monitoring capabilities such as monitoring the running status of Redis and validating application data.

[0200] 3. RedisSync started with two or more node Redis instances can not only provide operation and maintenance monitoring capabilities, but also achieve real-time memory cache data synchronization of two or more node Redis instances while allowing read and write operations, and the full amount of memory data cached by all nodes can be viewed at any time by checking any node.

[0201] 4. RedisSync provides data migration capabilities and has the ability to detect and self-heal (or manually maintain) the associated Redis that has failed without stopping the business application.

[0202] Please refer to Figure 9 , this embodiment of the present application also provides a database synchronization device that can implement the above database synchronization method. The device includes:

[0203] The first execution module 901 is used to execute the first persistence instruction through the first database node;

[0204] The listening module 902 is used to listen to the first database node through the second database node to obtain the first persistence instruction;

[0205] The first verification module 903 is used to calculate the first hash value of the first persistence instruction and perform the first idempotency verification through the first hash value;

[0206] The second execution module 904 is used to execute the first persistence instruction through the second database node if the first hash value passes the first idempotency verification;

[0207] The second verification module 905 is used to perform a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent an execution loop from occurring between the first database node and the second database node.

[0208] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The functions specifically implemented in the device embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0209] An embodiment of the present application further provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above database synchronization method is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0210] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The functions specifically implemented in the device embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0211] Please refer to Figure 10 , Figure 10 which shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0212] A processor 1001, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0213] A memory 1002, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1002 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1002, and the processor 1001 is used to call and execute the database synchronization method of the embodiments of the present application;

[0214] An input / output interface 1003, which is used to implement information input and output;

[0215] A communication interface 1004, which is used to implement the communication interaction between this device and other devices. It can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0216] A bus 1005, which transmits information between various components of the device (such as a processor 1001, a memory 1002, an input / output interface 1003, and a communication interface 1004);

[0217] Among them, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are communicatively connected to each other inside the device through the bus 1005.

[0218] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above database synchronization method is implemented.

[0219] It can be understood that the content in the above method embodiments is applicable to the embodiments of this storage medium. The functions specifically implemented by the embodiments of this storage medium are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0220] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0221] The database synchronization method, database synchronization device, electronic device, and storage medium provided by the embodiments of the present application execute a first persistence instruction through a first database node; listen to the first database node through a second database node to obtain the first persistence instruction; calculate a first hash value of the first persistence instruction, and perform a first idempotency verification through the first hash value; if the first hash value passes the first idempotency verification, then execute the first persistence instruction through the second database node; perform a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent execution loops from occurring between the first database node and the second database node. The embodiments of the present application can enable each database node to have full data. If a database node fails, only one database node is required to undertake all services.

[0222] The embodiments described in the embodiments of the present application are to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0223] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0224] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0225] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations.

[0226] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0227] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item) of the following" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0228] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0229] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0231] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The foregoing storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0232] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall fall within the scope of the rights of the embodiments of this application.

Claims

1. A database synchronization method, characterized in that, The method includes the following steps: Execute a first persistence instruction through a first database node; Monitor the first database node through a second database node to obtain the first persistence instruction; Calculate a first hash value of the first persistence instruction and perform a first idempotency verification through the first hash value; If the first hash value passes the first idempotency verification, execute the first persistence instruction through the second database node; Perform a second idempotency verification through the first hash value, and the second idempotency verification is used to prevent an execution loop between the first database node and the second database node.

2. The method according to claim 1, wherein The executing the first persistence instruction through the first database node includes: Monitor the first persistence instruction of the first database node through a second synchronization tool and calculate a first hash value of the first persistence instruction; Determine whether a first verification database contains the first hash value. If not, set a first verification value to -1. If so, subtract 1 from the first verification value, and the first verification value corresponds to the first hash value.

3. The method according to claim 1, characterized in that, The calculating the first hash value of the first persistence instruction and performing the first idempotency verification through the first hash value includes: Calculate a first hash value of the first persistence instruction; Determine whether a second verification database contains the first hash value; If the second verification database does not contain the first hash value, set a second verification value to 1; If the second verification database contains the first hash value, obtain a second verification value corresponding to the first hash value and add 1 to the second verification value; Determine whether the second verification value is greater than zero. If so, determine that the first hash value passes the first idempotency verification. If not, determine that the first hash value fails the first idempotency verification.

4. The method according to claim 1, wherein The if the first hash value passes the first idempotency verification, execute the first persistence instruction through the second database node includes: Monitor the first persistence instruction of the second database node through a first synchronization tool and calculate a first hash value of the first persistence instruction; Determine whether a second verification database contains the first hash value. If not, set a second verification value to -1. If so, subtract 1 from the second verification value, and the second verification value corresponds to the first hash value.

5. The method according to claim 1, wherein The performing the second idempotency verification through the first hash value includes: Monitor the second database node through the first database node to obtain a first persistence instruction and calculate the obtained first hash value; Determine whether a first verification database contains the first hash value; If the first verification database does not contain the first hash value, set a first verification value to 1; If the first verification database contains the first hash value, obtain a first verification value corresponding to the first hash value and add 1 to the first verification value; Determine whether the first verification value is greater than zero. If so, determine that the first hash value passes the second idempotency verification, and the first database node executes the first persistence instruction; If not, determine that the first hash value fails the second idempotency verification, and the first database node does not execute the first persistence instruction.

6. The method according to claim 1, characterized in that, The method further includes: Perform a health check on the synchronization tool, where the synchronization tool is used to monitor the first database node and the second database node; If the health check fails, obtain the historical failure count; Determine the current failure count based on the historical failure count; If the current failure count is greater than the preset failure threshold, stop the synchronization tool from monitoring the first database node and the second database node.

7. The method according to claim 1, wherein The method further includes: Query the first offset of the first database node; Query the second offset of the second database node; Perform offset verification through the first offset and the second offset to obtain a verification result.

8. A database synchronization device, characterized in that, The device includes: A first execution module for executing a first persistence instruction through the first database node; A monitoring module for monitoring the first database node through the second database node to obtain the first persistence instruction; A first verification module for calculating the first hash value of the first persistence instruction and performing first idempotency verification through the first hash value; A second execution module for, if the first hash value passes the first idempotency verification, executing the first persistence instruction through the second database node; A second verification module for performing second idempotency verification through the first hash value, where the second idempotency verification is used to prevent execution loops from occurring between the first database node and the second database node.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images