Data synchronization processing method and device

Abstract

The invention discloses a data synchronization processing method and device.The method comprises the steps that a read-write flag bit is obtained, the read-write flag bit has different identification values, access objects corresponding to the different identification values are different or the access permissions of the two corresponding access objects are different, and the two access objects comprise a source database and a target database; and performing access operation on the source database and / or the target database based on an access request obtained from a first application and the read-write flag bit, wherein the access operation comprises read operation and / or write operation.

Description

Technical Field

[0001] The present application relates to the technical field of technical databases, and more specifically, to a data synchronization processing method and device. Background Art

[0002] For various network applications, data migration is required due to the continuous growth of business data volume or changes in the database environment. This is the process of migrating the data content stored in the original database to a new database.

[0003] Traditional data migration solutions, such as MySQL databases, require setting up a master-slave configuration, with the target database (new database) acting as a slave node pointing to the source database (original database), and copying the source database's Binlog to the target database for application. However, this solution struggles to ensure data consistency between the source and target databases during application, and requires the source database to collect and store Binlogs in advance. Summary of the Invention

[0004] In view of this, this application provides the following technical solutions:

[0005] A first aspect of the present application provides a data synchronization processing method, the method comprising:

[0006] Obtaining a read / write flag bit, wherein the read / write flag bit has different identification values, and different identification values correspond to different access objects or correspond to different access rights of two access objects, wherein the two access objects include a source database and a target database;

[0007] An access operation is performed on the source database and / or the target database based on the access request obtained from the first application and the read / write flag, where the access operation includes a read operation and / or a write operation.

[0008] In one possible implementation, performing the access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag includes at least one of the following:

[0009] If the read / write flag has a first identification value, only read and write operations are performed on the source database;

[0010] If the read / write flag has a second identification value, the read / write operation is preferentially performed on the source database, and the write operation is performed on the target database;

[0011] If the read / write flag has a third identification value, the read / write operation is preferentially performed on the target database, and the write operation is performed on the source database;

[0012] If the read / write flag has a fourth identification value, only read / write operations are performed on the target database.

[0013] In one possible implementation, the method further includes:

[0014] After the data synchronization task from the source database to the target database is started, a full synchronization check is performed on the source database and the target database, wherein the full synchronization check is used to synchronize and verify all data in the source database and the target database;

[0015] If the verification result indicates that there is unsynchronized data in the source database and the target database, the control is to perform data synchronization processing on the unsynchronized data based on the source database.

[0016] In one possible implementation, the method further includes:

[0017] After the full synchronization check is completed, an incremental synchronization check is performed on the source database and the target database. The incremental synchronization check is used to verify the data newly added after a set time node in the source database and the target database. The set time node is related to the start time of the full synchronization check;

[0018] If the verification result indicates that there is unsynchronized data in the source database and the target database, the control is to perform data synchronization processing on the unsynchronized data based on the source database.

[0019] In one possible implementation, performing an access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag includes:

[0020] Before starting a data synchronization task from the source database to the target database, performing a read and write operation on the source database based on an access request and a first identification value obtained from a first application;

[0021] After the data synchronization task from the source database to the target database is started, based on the access request and the second identification value obtained from the first application, read and write operations are preferentially performed on the source database, and write operations are performed on the target database, and the full synchronization verification is simultaneously started;

[0022] After the full synchronization verification is completed, based on the access request obtained from the first application and the third identification value, read and write operations are preferentially performed on the target database, and write operations are performed on the source database, and the incremental synchronization verification is synchronously started;

[0023] After the incremental synchronization verification is completed, read and write operations are performed on the target database based on the access request obtained from the first application and the fourth identification value.

[0024] In a possible implementation, the switching of the read / write flag is determined by whether a data synchronization task is started or whether a synchronization check is completed.

[0025] In one possible implementation, if the verification result indicates that unsynchronized data exists in the source database and the target database, controlling the unsynchronized data to be synchronized includes:

[0026] If the verification result indicates that there is unsynchronized data in the source database and the target database, a repair message is generated, wherein the repair message includes the ID of the data to be repaired, the repair basis database, and the repair type. The repair basis database is the source database or the target database, and the repair type includes addition, deletion, and modification.

[0027] Based on the repair message, control is performed to perform repair corresponding to the repair type in another database outside the repair basis database according to data corresponding to the ID of the data to be repaired in the repair basis database.

[0028] In one possible implementation, the following is also included:

[0029] Sending the repair message to a message queue;

[0030] Based on the repair message, control is performed according to the data corresponding to the ID of the data to be repaired in the repair basis database, and repair corresponding to the repair type is performed in another database outside the repair basis database. This is an operation performed after obtaining the repair message from the message queue.

[0031] In a possible implementation, sending the repair message to a message queue includes:

[0032] The repair message is sent to the message queue according to a set rule, wherein the set rule includes any one of the following: a sending cycle, a number of repair messages sent, and a speed of sending repair messages.

[0033] A second aspect of the present application provides a data synchronization processing device, the device comprising:

[0034] A flag bit acquisition module is used to obtain a read-write flag bit, wherein the read-write flag bit has different identification values, and different identification values correspond to different access objects or correspond to different access rights of two access objects, wherein the two access objects include a source database and a target database;

[0035] The access control module is configured to perform an access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag, where the access operation includes a read operation and / or a write operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0037] Figure 1 This is a flow chart of a data synchronization processing method disclosed in an embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of the dual-write ConnPool data flow control disclosed in the embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the components of the Validator disclosed in the embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the data repair process of the Validator disclosed in the embodiment of this application;

[0041] Figure 5 A schematic diagram of the overall architecture of the middleware design disclosed in the embodiments of this application;

[0042] Figure 6 Schematic diagram of data flow at each stage of data migration disclosed in the embodiment of this application;

[0043] Figure 7 This is a flowchart of the database access operation disclosed in the embodiment of this application;

[0044] Figure 8 This is a structural diagram of a data synchronization processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0045] For the purpose of reference and clarity, the following technical terms, abbreviations or abbreviations are summarized as follows:

[0046] MySQL: An open source relational database management system used for storing, managing, and retrieving data.

[0047] Binlog: An important log file during the operation of the MySQL database, which records all data modification statements (such as INSERT, UPDATE, DELETE) and database structure change statements (such as CREATE, ALTER, DROP) executed on the MySQL database.

[0048] ConnPool: Connection Pool is a database connection management technology used to improve database access performance and efficiency. By establishing sufficient database connections at program startup and organizing these connections into a connection pool, the program can dynamically apply for, use, and release connections in the connection pool.

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] Figure 1 This is a flow chart of a data synchronization processing method disclosed in an embodiment of the present application. Figure 1 As shown, the data synchronization processing method may include:

[0051] Step 101: Obtain a read / write flag, wherein the read / write flag has different identification values. Different identification values correspond to different access objects or correspond to different access rights of two access objects, wherein the two access objects include a source database and a target database.

[0052] The data synchronization processing method disclosed in the embodiments of this application can be applied to database migration scenarios, such as MySQL database expansion scenarios. Database data migration involves two databases: a source database and a target database. This application solution can ensure the continuity of external services during data synchronization between the two databases, that is, guarantee the availability of services during the data migration process.

[0053] Specifically, the solution can control the flow of data during the first application accessing the database (source database and / or target database) through a middleware that supports dual-write functions. During the data synchronization process between the two databases, different read-write flags can be configured at different stages. Based on the different identification values of the read-write flags, the access rights of the source database and the target database can be controlled, thereby controlling the flow of data during the first application accessing the database. The dual-write function means that the middleware supports synchronous read / write operations on the source database and the target database. The specific implementation of the above-mentioned middleware that supports dual-write functions will be introduced in detail in the embodiments below and will not be explained in detail here.

[0054] Different identifier values correspond to different access objects. For example, under one identifier value, only the source database is the access object, and the target database cannot be accessed; under another identifier value, only the target database is the access object, and the source database cannot be accessed. Different identifier values also correspond to different access objects with different access permissions. For example, under one identifier value, the source database can perform both read and write operations, while the target database can only perform write operations; under another identifier value, the source database can only perform read operations, while the target database can perform both read and write operations.

[0055] Step 102: performing an access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag, where the access operation includes a read operation and / or a write operation.

[0056] During data synchronization between the source and target databases, the databases can still provide external services. Therefore, upon receiving an access request from the first application, the databases can determine whether to forward the access request to the source and / or target databases for access operations based on the current read / write flag. These access operations include read and / or write operations. At different stages of data synchronization, the read / write flags have different identification values, resulting in different states of access operations on the source and target databases.

[0057] The data synchronization processing method described in this embodiment can, during the data synchronization process between the source database and the target database, execute different permission access to the source database and / or the target database based on different identification values of the read and write flag bits, thereby directly controlling the data flow of the application access process through the identification value of the read and write flag bits, thereby ensuring that data services can still be provided to the outside during the data synchronization process; this implementation does not require the construction of a complex master-slave node environment, and has no other additional requirements. The entire data synchronization process can ensure that the database service does not stop and the service is continuously available.

[0058] In the aforementioned embodiments, the middleware used to implement access operations on the source database and / or the target database can essentially be a ConnPool (connection pool) that interacts between the application bottom layer and the MySQL server. This ConnPool supports dual-write functionality and can also be called a dual-write ConnPool. In addition to meeting all the functions of the native ConnPool, it also supports executing DML statements (add, modify, delete) on multiple databases at the same time and supports managing multiple database transactions at the same time. Based on the dual-write ConnPool, different Patterns (identification values) can be designed to identify the behavior of the dual-write ConnPool in executing DML statements, which is used to distinguish different data flows.

[0059] In specific applications, different HTTP interfaces can be designed. Users can use HTTP interfaces to directly control the reading and writing behavior of the application's underlying double-write ConnPool, thereby further controlling the data flow of the entire application. Furthermore, an HTTP Server can be embedded and packaged with the above-mentioned custom ConnPool into a universal SDK (Software Development Kit) for users to import and use. Users import the SDK into the project, and can instantiate a new ConnPool through simple configuration and register it with GORM (the framework responsible for interacting with the database in the project). By sending HTTP interfaces, the data flow of the entire project can be controlled, thereby achieving non-stop data migration.

[0060] During implementation, users can import the aforementioned middleware into their project as an SDK, provide basic configuration for the source and target databases, and the messaging middleware. They can then instantiate a WebServer based on the middleware and start it on a specified port. This completes the middleware import. Subsequently, the middleware can be used to control the data flow during the first application's database access by configuring the read / write flags.

[0061] Therefore, the access operation on the source database and / or the target database based on the access request obtained from the first application and the read-write flag may include at least one of the following: if the read-write flag has a first identification value, only read and write operations are performed on the source database; if the read-write flag has a second identification value, read and write operations are performed on the source database first, and write operations are performed on the target database; if the read-write flag has a third identification value, read and write operations are performed on the target database first, and write operations are performed on the source database; if the read-write flag has a fourth identification value, only read and write operations are performed on the target database.

[0062] Specifically, since the underlying connection pool in MySQL Server is responsible for handling all data read and write requests from clients, the ConnPool component in the GORM framework can be rewritten to dynamically change the behavior of data interaction with MySQL Server in the application. The underlying ConnPool is implemented by an interface containing four methods: PrepareContext, ExecContext, QueryContext, and QueryRowContext. After source code analysis and local testing, it was found that ExecContext, QueryContext, and QueryRowContext are the ones that primarily affect data reading and writing. Therefore, by rewriting these methods, a ConnPool solution that supports dual write can be implemented.

[0063] When the dual-write ConnPool calls the above method to read and write data, it is necessary to determine at the bottom layer whether to read the target database or the source database, and to distinguish the data flow when writing data. Therefore, the corresponding Pattern (the identification value of the read and write flag) is designed to distinguish the data reading and writing behaviors; in order to facilitate the simultaneous operation of the source database and the target database, two default ConnPools are embedded in the dual-write ConnPool to operate the source database and the target database respectively.

[0064] Figure 2 This is a schematic diagram of the dual-write ConnPool data flow control disclosed in the embodiment of this application, wherein the upper figure is an example diagram of the write operation data flow, and the lower figure is an example diagram of the read operation data flow. Figure 2As shown, DoubleWriteConnPool (DoubleWriteConnPool()) contains two default ConnPools and a Pattern, and re-implements the PrepareContext, ExecContext, QueryContext, and QueryRowContext methods. At this time, DoubleWriteConnPool has fully implemented the ConnPool interface and can completely replace the default ConnPool for import; the ExecContext, QueryContext, and QueryRowContext methods will obtain the value of the current Pattern to determine which database the current control reads, only writes, or writes to which database first. For example, when the value of Pattern is "DSN First", DoubleWriteConnPool will call the QueryContext method corresponding to the target database to query data, and call the ExecContext methods corresponding to the target database and the source database in turn to insert data, so that the application will use the target database as the basis when reading data, and when writing data, it will first write to the target database, and then write the same data to the source database to achieve a "double write effect". Specifically, the identification value Pattern is "SRC Only", which means that only read and write operations are performed on the source database; Pattern is "SRC "DSN First" means that read and write operations are performed on the source database first, and then write operations are performed on the target database; "DSN First" means that read and write operations are performed on the target database first, and then read operations are performed on the source database; "DSN Only" means that read and write operations are performed only on the target database.

[0065] The above content details the specific implementation of controlling data flow under different identification values of the read and write flags, which will help technicians in the field to better understand and implement the technical solution of this application.

[0066] In other implementations, the data synchronization processing method may further include: after the data synchronization task from the source database to the target database is initiated, performing a full synchronization check on the source database and the target database, wherein the full synchronization check is used to synchronize all data in the source database and the target database; if the verification result indicates that there is unsynchronized data in the source database and the target database, controlling the data synchronization processing of the unsynchronized data based on the source database. The full synchronization check is to verify all data in the source database and the target database.

[0067] Since the system data is being modified dynamically in real time without shutting down, in order to solve the problem of possible inconsistency between the source database and the target database, the application solution can introduce a "Validator" in the middleware to verify and repair the data in real time.

[0068] Figure 3 This is a schematic diagram of the components of the Validator disclosed in the embodiment of this application. Figure 4 This is a schematic diagram of the data repair process of the Validator disclosed in the embodiment of this application. Figure 3 and Figure 4 As shown, the Validator consists of two *gorm.DB instances (one corresponding to the source database and the other to the target database), a Derection identifier, and a Producer (message producer). In implementation, the Validator uses the *gorm.DB instances to query data from both the source and target databases based on their incrementing IDs, converting the retrieved data into Byte arrays. The Derection identifier indicates whether the validation is based on the source or target database data. Three types of data inconsistencies can occur during data validation: 1. Missing data in the source database due to hard deletion; 2. Inconsistency between the source and target databases due to business data changes; 3. Missing data in the target database due to new data added to the source database. Figure 4 SRCMissing, NotEqual and DSNMissing correspond to the above three situations respectively.

[0069] When data inconsistency is found during the Validator verification process, the Producer can send a message to the message queue Kafka. The message body contains the ID of the data to be repaired, the Derection identifier, and the type of data inconsistency. The data repair process is triggered by the Consumer in the middleware.

[0070] During the data repair process, the middleware's Consumer pulls data sent to Kafka by the Producer in real time and repairs the data based on the values of the fields in the message body. For example, if a message has an ID of 1, a Direction of SRC, and a Type of NotEqual, the Consumer repairs the corresponding data in the target database based on the data in the source database and updates the source data corresponding to ID 1 to the target database. At this point, the data repair is complete.

[0071] Based on the aforementioned disclosed content, the data processing synchronization method may further include: after the full synchronization verification is completed, performing an incremental synchronization verification on the source database and the target database, the incremental synchronization verification being used to verify the newly added data after a set time node in the source database and the target database, and the set time node being related to the start time of the full synchronization verification; if the verification result indicates that there is data that has not been synchronized between the source database and the target database, then controlling the data synchronization processing of the data that has not been synchronized based on the source database.

[0072] To further improve validation performance and reduce the number of data comparisons, Validator also supports incremental synchronization validation. This means it only scans data in the source database that was updated more recently than the time when the full synchronization validation was triggered. This new data can be discovered and repaired in advance during the incremental synchronization validation process. Therefore, the set time node can be the time point when the full synchronization validation is triggered.

[0073] For example, if the source database contains 100,000 records and a full synchronization check is triggered at the first point in time, the full synchronization check may take several hours. During this period, some data may be added to the source database and some existing data may be modified. This data may not be included in the full synchronization check. Therefore, after the full synchronization check is completed, the incremental synchronization check can be started. Alternatively, in the implementation, after a complete full synchronization check, the full synchronization check and incremental synchronization check can be controlled synchronously to ensure complete and accurate data synchronization between the two databases.

[0074] Based on the foregoing, if the verification result of the full synchronization verification or the verification result of the incremental synchronization verification indicates that there is data that has not been synchronized between the source database and the target database, controlling the data synchronization processing of the data that has not been synchronized may include: if the verification result indicates that there is data that has not been synchronized between the source database and the target database, generating a repair message, the repair message including the ID of the data to be repaired, the repair basis database and the repair type, the repair basis database being the source database or the target database, and the repair type including addition, deletion and modification; based on the repair message, controlling the data corresponding to the ID of the data to be repaired in the repair basis database to perform a repair corresponding to the repair type in another database outside the repair basis database.

[0075] Furthermore, the repair message may be sent to a message queue. Controlling, based on the repair message, performing repair corresponding to the repair type in a database other than the repair basis database according to data corresponding to the ID of the data to be repaired in the repair basis database is an operation performed after obtaining the repair message from the message queue.

[0076] Among them, sending the repair message to the message queue may include: sending the repair message to the message queue according to a set rule, wherein the set rule includes any one of the following: a sending cycle, a number of repair messages sent, and a speed of sending repair messages. That is, in the process of verification and repair, in order to avoid the database pressure caused by frequent reading and writing of the database with high concurrency, the verification and repair of the data can be decoupled through the message queue in implementation, and the database can be protected by controlling the rate of data verification and repair, thereby avoiding performance problems caused by high-concurrency reading and writing of the database. Controlling the rate of data verification and repair can be achieved, for example, by controlling the rate at which message producers send messages.

[0077] In actual applications, data verification and repair, as well as control operations for the double-write ConnPool, can be actively triggered by the user. Therefore, a Gin Server (a type of HTTP server) is embedded in the middleware. Routes corresponding to all ConnPool behaviors, as well as triggering incremental and full synchronization verification, are designed. These behaviors are triggered in the corresponding view functions of the routes. This allows users to change the application's data flow while the application is running without any code intrusion. Local file storage is also introduced to store the results of each data verification and repair, and render them on the web page for easy user review. Figure 5 This is a schematic diagram of the overall architecture of the middleware design disclosed in the embodiment of this application, where config is used to implement the configuration of the source database, target database, read and write flags, verification, etc., and Storage is used to store related configuration logs. Figure 5 As shown in the architecture diagram, in the implementation, users can view the information of the configured source database and target database based on the web page provided by the middleware, and click the full synchronization verification button to trigger a full synchronization verification; by clicking different buttons, the behavior of data writing to the database can be controlled to further control the data flow, or the data table can be directly specified to start full synchronization verification or incremental synchronization verification. By controlling the bidirectional writing, data verification and data repair of data, the overall data migration is completed. In addition, it supports allowing users to view the verification history and verification results after each data verification is completed. It can be combined with Figure 5 Understand the content of the above embodiments.

[0078] Figure 6This is a schematic diagram of the data flow at each stage of data migration disclosed in the embodiment of this application. Figure 6 As shown, in the application, after the user imports the SDK of the aforementioned middleware, a Gin Server and a new ConnPool (i.e., dual-write ConnPool) can be instantiated. The user configures the dual-write ConnPool for use in the project, configures the source database information and the target database information, and starts the Gin Server to complete the SDK access. The dual-write and the first full synchronization verification can be enabled in the corresponding web page. At this time, the data in the source database and the target database are roughly equal. At this time, the incremental synchronization verification is enabled, and the data reading and writing are switched to the target database. Finally, the read and write traffic is switched to only pass through the target database, and the source database is completely decoupled, completing the seamless migration of data without business stagnation. Traditional processes require a large amount of deletions and modifications to all read and write data in the business code at each stage in order to achieve this data flow. Not only is the workload difficult to estimate, but it is also difficult to avoid data inconsistency problems.

[0079] Based on the above, Figure 7 The flowchart of accessing the database is shown. Figure 6 and Figure 7 As shown, the access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag may include:

[0080] Step 701: before starting a data synchronization task from the source database to the target database, performing read and write operations on the source database based on an access request and a first identification value obtained from a first application.

[0081] Before the data synchronization task is started, although the target database already exists, the first application Application, under the action of the SRCOnly identifier value (first identifier value), only performs read and write operations on the source database SRC, and does not perform any operations on the target database.

[0082] Step 702: After the data synchronization task from the source database to the target database is started, based on the access request and the second identification value obtained from the first application, read and write operations are preferentially performed on the source database, and write operations are performed on the target database, and the full synchronization verification is started simultaneously.

[0083] After the data synchronization task is started, the first application, using the SRC First identifier (second identifier), prioritizes read and write operations on the source database (SRC), and then performs the same write operations on the target database (DSN). Furthermore, the data synchronization task initiates a synchronization trigger to perform a full synchronization check on the source and target databases. This process verifies that data stored in the source database does not exist in the target database, and gradually migrates (assigns values) the data from the source database to the target database.

[0084] Step 703: After the full synchronization verification is completed, based on the access request and the third identification value obtained from the first application, read and write operations are preferentially performed on the target database, and write operations are performed on the source database, and the incremental synchronization verification is synchronously started.

[0085] After the data synchronization task is completed, the first application, under the action of the DSN First identifier value (the third identifier value), first performs read and write operations on the target database SRC, and then performs the same read operation on the source database DSN. The read operation here is used for incremental synchronization verification to determine whether the corresponding data in the source database and the target database are the same.

[0086] Step 704: After the incremental synchronization verification is completed, read and write operations are performed on the target database based on the access request and the fourth identification value obtained from the first application.

[0087] After the incremental synchronization verification is completed, the first application, under the action of the DSN Only identifier value (the fourth identifier value), only performs read and write operations on the target database DSN without performing any operations on the source database, thereby achieving complete decoupling of the source database.

[0088] In the above embodiments, the switching of the read / write flag is determined by the initiation of the data synchronization task or the completion of the synchronization verification. This detailed description of the integrated implementation of middleware control of data access flow and synchronization verification during data synchronization between the source and target databases. The synchronization verification start and end nodes serve as the nodes that distinguish each stage, combined with data flow control at each stage, enable data migration from the source database to the target database without interrupting database services.

[0089] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0090] The above embodiments disclosed in the present application describe the method in detail. The method of the present application can be implemented using various devices. Therefore, the present application also discloses a device, and a specific embodiment is given below for detailed description.

[0091] Figure 8 This is a structural diagram of a data synchronization processing device disclosed in an embodiment of the present application. Figure 8 As shown, the data synchronization processing device 80 may include:

[0092] The flag bit acquisition module 801 is used to obtain the read and write flag bits, which have different identification values. Different identification values correspond to different access objects or different access rights of two corresponding access objects. The two access objects include a source database and a target database.

[0093] The access control module 802 is configured to perform an access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag, where the access operation includes a read operation and / or a write operation.

[0094] The data synchronization processing device described in this embodiment can, during the process of data synchronization between the source database and the target database, execute different permission access to the source database and / or the target database based on different identification values of the read and write flag bits, thereby directly controlling the data flow of the application access process through the identification value of the read and write flag bits, thereby ensuring that data services can still be provided to the outside during the data synchronization process; this implementation does not require the construction of a complex master-slave node environment, and there are no other additional requirements. The entire data synchronization process can ensure that the database service does not stop and the service is continuously available.

[0095] In one implementation, the access control module can be specifically used for one of the following: if the read-write flag has a first identification value, only read and write operations are performed on the source database; if the read-write flag has a second identification value, read and write operations are performed on the source database first, and write operations are performed on the target database; if the read-write flag has a third identification value, read and write operations are performed on the target database first, and write operations are performed on the source database; if the read-write flag has a fourth identification value, only read and write operations are performed on the target database.

[0096] In one implementation, the device may further include: a verification processing module, which is used to perform a full synchronization verification on the source database and the target database after the data synchronization task from the source database to the target database is started, and the full synchronization verification is used to perform synchronization verification on all data in the source database and the target database; a repair processing module, which is used to control the data synchronization processing of the unsynchronized data based on the source database if the verification result indicates that there is data that has not been synchronized between the source database and the target database.

[0097] In one implementation, the verification processing module is further used to: after the full synchronization verification is completed, perform incremental synchronization verification on the source database and the target database, and the incremental synchronization verification is used to verify the newly added data after the set time node in the source database and the target database, and the set time node is related to the start time of the full synchronization verification; the repair processing module is also used to: if the verification result indicates that there is data that has not been synchronized between the source database and the target database, then control the data synchronization processing of the data that has not been synchronized based on the source database.

[0098] In one implementation, the access control module can be specifically used to: before the data synchronization task from the source database to the target database is started, perform read and write operations on the source database based on the access request and the first identification value obtained from the first application; after the data synchronization task from the source database to the target database is started, preferentially perform read and write operations on the source database based on the access request and the second identification value obtained from the first application, and perform write operations on the target database, and synchronously start the full synchronization verification; after the full synchronization verification is completed, preferentially perform read and write operations on the target database based on the access request and the third identification value obtained from the first application, and perform write operations on the source database, and synchronously start the incremental synchronization verification; after the incremental synchronization verification is completed, perform read and write operations on the target database based on the access request and the fourth identification value obtained from the first application.

[0099] In one implementation, the switching of the read / write flag is determined by whether the data synchronization task is started or whether the synchronization check is completed.

[0100] In one implementation, the repair processing module can be specifically used to: if the verification result indicates that there is data that has not been synchronized between the source database and the target database, generate a repair message, the repair message including the ID of the data to be repaired, the repair basis database and the repair type, the repair basis database is the source database or the target database, and the repair type includes addition, deletion and modification; based on the repair message, control the data corresponding to the ID of the data to be repaired in the repair basis database, and perform the repair corresponding to the repair type in another database outside the repair basis database.

[0101] In one implementation, the repair processing module is also used to: send the repair message to the message queue; based on the repair message, control the data corresponding to the ID of the data to be repaired in the repair basis database, and perform repair corresponding to the repair type in another database outside the repair basis database, which is an operation performed after obtaining the repair message from the message queue.

[0102] In one implementation, the repair processing module can be specifically used to: send the repair message to the message queue according to a set rule, wherein the set rule includes any of the following: sending cycle, number of repair messages sent, speed of sending repair messages

[0103] The specific implementation of the above-mentioned data synchronization processing device and the modules it contains, as well as other possible implementations, can be found in the corresponding parts of the method embodiment, and will not be repeated here.

[0104] Any one of the data synchronization processing devices described in the above embodiments includes a processor and a memory. The flag bit acquisition module, access control module, verification processing module, repair processing module, etc. in the above embodiments are all stored in the memory as program modules, and the processor executes the above program modules stored in the memory to realize corresponding functions.

[0105] The processor contains a kernel, which retrieves the corresponding program module from the memory. There can be one or more kernels, and the kernel parameters can be adjusted to process the access data.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0107] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the above-mentioned data synchronization processing method.

[0108] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by the computer, the computer program can implement the steps shown in any embodiment of the data synchronization processing method described above.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0110] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0112] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization processing method, the method comprising: Obtaining a read / write flag bit, wherein the read / write flag bit has different identification values, and different identification values correspond to different access objects or correspond to different access rights of two access objects, wherein the two access objects include a source database and a target database; An access operation is performed on the source database and / or the target database based on the access request obtained from the first application and the read / write flag, where the access operation includes a read operation and / or a write operation.

2. The data synchronization method according to claim 1, wherein the access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag comprises at least one of the following: If the read / write flag has a first identification value, only read and write operations are performed on the source database; If the read / write flag has a second identification value, the read / write operation is preferentially performed on the source database, and the write operation is performed on the target database; If the read / write flag has a third identification value, the read / write operation is preferentially performed on the target database, and the write operation is performed on the source database; If the read / write flag has a fourth identification value, only read / write operations are performed on the target database.

3. The data synchronization processing method according to claim 1, further comprising: After the data synchronization task from the source database to the target database is started, a full synchronization check is performed on the source database and the target database, wherein the full synchronization check is used to synchronize and verify all data in the source database and the target database; If the verification result indicates that there is unsynchronized data in the source database and the target database, the control is to perform data synchronization processing on the unsynchronized data based on the source database.

4. The data synchronization processing method according to claim 3, further comprising: After the full synchronization check is completed, an incremental synchronization check is performed on the source database and the target database. The incremental synchronization check is used to verify the data newly added after a set time node in the source database and the target database. The set time node is related to the start time of the full synchronization check; If the verification result indicates that there is unsynchronized data in the source database and the target database, the control is to perform data synchronization processing on the unsynchronized data based on the source database.

5. The data synchronization processing method according to claim 4, wherein the access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag comprises: Before starting a data synchronization task from the source database to the target database, performing a read and write operation on the source database based on an access request and a first identification value obtained from a first application; After the data synchronization task from the source database to the target database is started, based on the access request and the second identification value obtained from the first application, read and write operations are preferentially performed on the source database, and write operations are performed on the target database, and the full synchronization verification is simultaneously started; After the full synchronization verification is completed, based on the access request obtained from the first application and the third identification value, read and write operations are preferentially performed on the target database, and write operations are performed on the source database, and the incremental synchronization verification is synchronously started; After the incremental synchronization verification is completed, read and write operations are performed on the target database based on the access request obtained from the first application and the fourth identification value.

6. The data synchronization processing method according to claim 4, wherein: The switching of the read / write flag is determined by whether the data synchronization task is started or whether the synchronization check is completed.

7. The data synchronization method according to claim 3 or 4, wherein if the verification result indicates that there is unsynchronized data between the source database and the target database, controlling the unsynchronized data to be synchronized comprises: If the verification result indicates that there is unsynchronized data in the source database and the target database, a repair message is generated, wherein the repair message includes the ID of the data to be repaired, the repair basis database, and the repair type. The repair basis database is the source database or the target database, and the repair type includes addition, deletion, and modification. Based on the repair message, control is performed to perform repair corresponding to the repair type in another database outside the repair basis database according to data corresponding to the ID of the data to be repaired in the repair basis database.

8. The data synchronization method according to claim 7, further comprising: Sending the repair message to a message queue; Based on the repair message, control is performed according to the data corresponding to the ID of the data to be repaired in the repair basis database, and repair corresponding to the repair type is performed in another database outside the repair basis database. This is an operation performed after obtaining the repair message from the message queue.

9. The data synchronization processing method according to claim 8, wherein sending the repair message to a message queue comprises: The repair message is sent to the message queue according to a set rule, wherein the set rule includes any one of the following: a sending cycle, a number of repair messages sent, and a speed of sending repair messages.

10. A data synchronization processing device, comprising: A flag bit acquisition module is used to obtain a read-write flag bit, wherein the read-write flag bit has different identification values, and different identification values correspond to different access objects or correspond to different access rights of two access objects, wherein the two access objects include a source database and a target database; The access control module is configured to perform an access operation on the source database and / or the target database based on the access request obtained from the first application and the read / write flag, where the access operation includes a read operation and / or a write operation.

Images