Database whole instance synchronization method and device, equipment and storage medium

Abstract

The invention discloses a database whole instance synchronization method and device, equipment and a storage medium, and the method comprises the steps: building a primary pre-write log receiving connection with a source database through any group set, and building a corresponding analysis queue for each group set; when the pre-writing log is received through the pre-writing log receiving connection, distributing each pre-writing log entry in the pre-writing log to a corresponding analysis queue; and analyzing each analysis queue by referring to the corresponding data dictionary to obtain a whole instance source end data record, and synchronizing the whole instance source end data record to a target database. According to the technical scheme, one pre-write log receiving connection is established with the source database through one grouping set, so that excessive connection with the source database is avoided, and the pressure on the source database and the influence on normal business are reduced. When the pre-writing log entries are read and analyzed, each pre-writing log entry in the received pre-writing day is analyzed only once, so that the waste of system resources is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of database synchronization, and particularly to a method, device, equipment and storage medium for synchronizing an entire database instance. Background Art

[0002] With the application of big data in various fields, the demand for database synchronization is also expanding. Currently used databases include various types. For example, when an instance in a Postgres database is running, it is hierarchically divided into four levels: instance, grouped set database, grouped schema, and table from top to bottom.

[0003] Currently, when the Postgres database is used as a source database for synchronization, each client connects to the database and connects to one database at a time when using it. The data content that can be obtained is limited to the current database, but only one write-ahead log wal log is generated during the operation of the entire database instance. And during synchronization, it is necessary to obtain and parse the write-ahead log, and the data dictionaries required for parsing each database are independent of each other, and information needs to be obtained and established through their respective connections.

[0004] In the prior art, when synchronizing the entire Postgres database instance, multiple corresponding connections are established according to the number of grouped sets, and the write-ahead logs are obtained and parsed respectively. Using the above method will cause repeated acquisition and parsing of the write-ahead logs, resulting in waste of computing resources. And when there are too many connections to the source database, it will also cause great pressure on the source database, thus affecting the normal business use of the source database. Summary of the Invention

[0005] The present invention provides a method for synchronizing an entire database instance to reduce resource occupation and the impact on the source database business during the synchronization process of the entire database instance.

[0006] According to the first aspect of the present invention, a method for synchronizing an entire database instance is provided, including: creating corresponding data dictionaries for each grouped set associated with the entire instance in the source database;

[0007] Establishing a write-ahead log receiving connection with the source database through any one of the grouped sets, and establishing corresponding parsing queues for each of the grouped sets;

[0008] When a write-ahead log is received through the write-ahead log receiving connection, distributing each write-ahead log entry in the write-ahead log to the corresponding parsing queue;

[0009] Parse each of the parsing queues with reference to the corresponding data dictionary to obtain the whole instance source - end data records, and synchronize the whole instance source - end data records to the target database.

[0010] According to another aspect of the present invention, there is provided a database whole - instance synchronization device, including: a data dictionary creation module, configured to create corresponding data dictionaries for each grouping set associated with the whole instance in the source database;

[0011] A parsing queue establishment module, configured to establish a write - ahead log reception connection with the source database once through any one of the grouping sets, and establish corresponding parsing queues for each of the grouping sets;

[0012] A preview log entry allocation module, configured to, when a write - ahead log is received through the write - ahead log reception connection, allocate each write - ahead log entry in the write - ahead log to the corresponding parsing queue;

[0013] A whole - instance source - end data synchronization module, configured to parse each of the parsing queues with reference to the corresponding data dictionary to obtain the whole instance source - end data records, and synchronize the whole instance source - end data records to the target database.

[0014] According to another aspect of the present invention, there is provided an electronic device, the electronic device includes:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer - readable storage medium, the computer - readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any embodiment of the present invention is implemented.

[0019] The technical solution of the embodiment of the present invention establishes a write - ahead log reception connection with the source database once through a grouping set, thereby avoiding excessive connections with the source database, reducing the pressure on the source database and the impact on normal services. When reading and parsing write - ahead log entries, each write - ahead log entry in a received write - ahead log is only parsed once, thereby reducing the waste of system resources.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become readily understood from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 is a flowchart of a method for synchronizing an entire database instance according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flowchart of a method for synchronizing an entire database instance according to Embodiment 1 of the present invention;

[0024] Figure 3 is a flowchart of a method for synchronizing an entire database instance according to Embodiment 2 of the present invention;

[0025] Figure 4 is a schematic structural diagram of a device for synchronizing an entire database instance according to Embodiment 3 of the present invention;

[0026] Figure 5 is a schematic structural diagram of an electronic device according to Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 FIG. is a flowchart of a method for synchronizing a database whole instance provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of synchronizing a database whole instance. This method can be executed by a database whole instance synchronization device, and the device can be implemented in the form of hardware and / or software. As Figure 1 shown, the method includes:

[0031] Step S101, create corresponding data dictionaries for each grouping set associated with the whole instance in the source database.

[0032] Optionally, creating corresponding data dictionaries for each grouping set associated with the whole instance in the source database includes: connecting to the source database and obtaining the identifiers of each grouping set associated with the whole instance in the source database; sequentially connecting each grouping set according to the identifiers and obtaining the attribute information of each grouping set, where the attribute information includes grouping method information and table information; generating corresponding data dictionaries according to the attribute information of each grouping set.

[0033] Specifically, when performing database synchronization, the source database and the target database are involved. Database synchronization is to synchronize data from the source database to the target database. A whole instance refers to a running instance of a database management system, which is a collection of a storage structure containing database data, a process for processing data, and a software entity for managing database operations, and is the identifier of the database software when it runs and the sum of a series of processes required when specifying database operations. A Postgres database whole instance consists of multiple grouping sets database, and a grouping set is a layer of structure for dividing data content in the database. Each grouping set consists of multiple table sets.

[0034] Among them, when performing the whole instance synchronization of the database in this embodiment, the client needs to connect to the Postgres source database, and then obtain the identifiers of all the grouped sets database associated with the whole instance in the source database. For example, database1 and database2, that is, the names of two grouped sets are obtained. Of course, in this embodiment, only two grouped sets associated with the whole instance are taken as examples for illustration, and the number of associated whole instances is not limited. After obtaining the identifiers of the grouped sets, each grouped set is connected in turn. For example, first connect to the grouped set named database1, and obtain the attribute information of the grouped set database1. Among them, the attribute information includes the grouping method information schemma1 and the table information 1, etc. The grouping method information schemma1 includes the group name, the tables included, and a main table, etc. The table information 1 includes the names of each table, the field information included in each table, and the primary key field information, etc. Of course, this embodiment is only for illustration, and the specific content of the attribute information is not limited. After obtaining the attribute information of the grouped set database1, a data dictionary 1 will be generated according to the attribute information, that is, the data dictionary 1 mainly includes the relevant parameter information of database1 in the source database. In addition, after the client disconnects from database1, it will connect to database2 again, and generate a data dictionary 2 according to the attribute information of database2, that is, the data dictionary 2 mainly includes the relevant parameter information of database2 in the source database, and after the data fields corresponding to each grouped set are established in turn, it will disconnect from all the grouped sets.

[0035] Step S102, establish a write-ahead log receiving connection with the source database through any one of the grouped sets, and establish a corresponding parsing queue for each grouped set.

[0036] Optionally, before allocating each write-ahead log entry in the write-ahead log to the corresponding parsing queue, it further includes: establishing a parsing thread for each grouped set that matches the parsing queue, where the parsing thread is used to monitor and operate on the matching parsing queue; establishing a write-ahead log reading thread.

[0037] Specifically, after obtaining the data dictionaries corresponding to each grouping set in this embodiment, since every operation change of the entire instance in the source database will be recorded in the write-ahead log wallog before being written to the source database, and the atomicity and durability of data operations are ensured through the write-ahead log. Therefore, in this embodiment, in order to avoid the repeated reception and parsing of the write-ahead log, only one write-ahead log reception connection is established with the source database through one grouping set. For example, a write-ahead log reception connection is established with the source database through database1, and the connection will be maintained and not interrupted until the database synchronization is completed. Thus, compared with the prior art where multiple write-ahead log reception connections are established according to the number of grouping sets, and the same write-ahead log is repeatedly received through the established write-ahead log connections, establishing only one write-ahead log reception connection will significantly reduce the pressure on the source database, thereby ensuring the normal business use of the source database. In addition, since only one write-ahead log reception connection is established, and only one write-ahead log needs to be received based on the established write-ahead log reception connection, compared with the repeated acquisition and parsing of receiving one write-ahead log for each established write-ahead log in the prior art, it can avoid the waste of computing resources. Of course, in this embodiment, only the example of establishing a write-ahead log reception connection with the source database through database1 is used for illustration. Of course, a write-ahead log reception connection can also be established with the source database through database2. This embodiment does not limit the grouping set used when establishing one write-ahead log reception connection.

[0038] It should be noted that after establishing one write-ahead log reception connection in this embodiment, a corresponding parsing queue and a parsing thread matching the parsing queue will also be established for each grouping set database. Among them, the parsing thread is used to monitor and operate on the matching parsing queue. For example, parsing queue A and parsing thread a are established for database1, and parsing queue B and parsing thread b are established for database2. Of course, only examples are given in this embodiment, and the specific formats of the parsing queue and the parsing thread are not limited. In addition, a write-ahead log reading thread will also be established in this embodiment, and when it is determined that the write-ahead log is received, the write-ahead log reading thread is used to read the write-ahead log.

[0039] Step S103: When the write-ahead log is received through the write-ahead log reception connection, allocate each write-ahead log entry in the write-ahead log to the corresponding parsing queue.

[0040] Optionally, when a write-ahead log is received through the write-ahead log receiving connection, each write-ahead log entry in the write-ahead log is assigned to a corresponding parsing queue, including: when a write-ahead log is received through the write-ahead log receiving connection, the write-ahead log entries are obtained by reading the write-ahead log through a log reading thread, where each write-ahead log entry is marked with an identifier of a grouping set; each write-ahead log entry is assigned to a corresponding parsing queue according to the marked identifier of the grouping set.

[0041] Specifically, in this embodiment, after the parsing queue, the parsing thread, and the write-ahead log reading thread are established, when a write-ahead log is received through the write-ahead log receiving connection, the log reading thread is executed to perform a reading operation on the received write-ahead log. Since the write-ahead log is composed of write-ahead log entries walentry, each change in the database operation is recorded as one or more walentry. Therefore, multiple write-ahead log entries are obtained when the write-ahead log is read through the log reading thread. The number of the read write-ahead log entries is not limited in this embodiment. Among them, each write-ahead log entry is marked with an identifier of a grouping set. For example, database1 is marked on walentry1, and database2 is marked on walentry2. Of course, this embodiment is only an example and does not limit the content of the grouping set marked in each write-ahead log entry. As long as it can identify which grouping set in the entire instance the write-ahead log entry is specifically related to, it is within the protection scope of this application. Since corresponding data dictionaries are created in advance for each grouping set associated with the entire instance in the source database, after the log reading thread reads each write-ahead log entry from the write-ahead log, it is also responsible for assigning the write-ahead log entry to the corresponding parsing queue according to the marked grouping set.

[0042] Step S104, parse each parsing queue with reference to the corresponding data dictionary to obtain the entire instance source-side data record, and synchronize the entire instance source-side data record to the target database.

[0043] Among them, as Figure 2 shown is the flowchart of the database entire instance synchronization method of this embodiment, which mainly specifically describes step S104 and mainly includes the following steps:

[0044] S1041, monitor the corresponding parsing queue through each parsing thread respectively. When it is determined that there is data stored in the parsing queue, obtain the data dictionary corresponding to the parsing queue.

[0045] Specifically, since each parsing thread monitors the corresponding parsed queue. For example, for database1, the corresponding parsed queue is Queue A and the parsing thread is Thread a, and Thread a monitors Queue A; for database2, the corresponding parsed queue is Queue B and the parsing thread is Thread b, and Thread b monitors Queue B. When Thread a monitors that pre-written log entries walentry1, walentry3, and walentry5 are stored in Queue A, Thread a will obtain the data dictionary corresponding to Queue A. Since Queue A is established for the grouped set database1, and the data dictionary created for the grouped set database1 is Data Dictionary 1, the data dictionary corresponding to Queue A is Data Dictionary 1. That is, through the correspondence between the grouped set and the data dictionary, and the correspondence between the grouped set and the parsed queue, the data dictionary corresponding to the parsed queue is indirectly obtained. Correspondingly, the data dictionary corresponding to Queue B is Data Dictionary 2. The method for obtaining the data dictionary corresponding to Queue B is similar to the above and will not be elaborated in this embodiment.

[0046] S1042, sequentially extract the stored pre-written log entries from the monitored parsed queue through the parsing connection, and parse each pre-written log entry with reference to the data dictionary corresponding to the monitored parsed queue to obtain the entire instance source-side data record.

[0047] Optionally, parsing each pre-written log entry with reference to the data dictionary corresponding to the monitored parsed queue to obtain the entire instance source-side data record includes: when it is determined that the pre-written log entry includes data manipulation language information, then parse the data manipulation language information with reference to the data dictionary corresponding to the monitored parsed queue to obtain the source-side operation data record; when it is determined that the pre-written log entry includes data definition language information, then update the data dictionary corresponding to the monitored parsed queue according to the data definition language information, and obtain the source-side definition data record according to the data definition language information; obtain the entire instance source-side data record according to the source-side operation data record and the source-side definition data record.

[0048] Specifically, in this embodiment, after obtaining the data dictionaries corresponding to each parsing queue, the pre-written log entries stored in the monitored parsing queue are sequentially extracted through the parsing connection. For example, when parsing thread a sequentially extracts pre-written log entries walentry1, walentry3, and walentry5 from parsing queue A, and parses the pre-written log entries in order with reference to data dictionary 1 to obtain the data records related to database1 in the entire instance. Among them, when parsing each pre-written log entry, different forms of parsing results are obtained according to the different information contained in the pre-written log entry, and in this embodiment, the parsing results of all parsing queues are combined to obtain the source data records of the entire instance.

[0049] It should be noted that in this embodiment, since each parsing queue parses with reference to the corresponding data dictionary, the parsing processes of each parsing queue can perform parsing operations in parallel, thus significantly improving the parsing efficiency. Since each data dictionary is independent during parallel parsing, incorrect parsing will not occur due to cross information of the data dictionaries, thereby improving the parsing accuracy.

[0050] S1043, synchronize the source data records of the entire instance to the target database.

[0051] Optionally, synchronizing the source data records of the entire instance to the target database includes: establishing a connection with the target database; applying the source data records of the entire instance to the target database to synchronize the entire instance in the source database.

[0052] Specifically, in this embodiment, after obtaining the source data records of the entire instance through parsing, the source data records of the entire instance are also applied to the target database to achieve data synchronization between the source data and the target database. Therefore, in this embodiment, a multi-thread method is adopted to establish a pre-written log receiving connection thread, a data dictionary generation thread, a pre-written log reading and distribution thread, a parsing thread, etc. Among them, after obtaining the grouping set contained in the entire instance, a connection is used to sequentially connect to different grouping sets, obtain basic information and generate independent data dictionaries. Multiple parsing threads are established for each grouping set, and each parsing thread only parses the content related to itself. Then, a connection is used to obtain the pre-written log, a reading thread is established to read the pre-written log, and the read pre-written log entries are distributed to the parsing queue, and the parsing threads query and parse each pre-written log entry according to their respective dictionaries.

[0053] Among them, the write-ahead log in the source database of this embodiment is received only once, and only one connection is used when generating the data dictionary, reducing the pressure on the source database and the impact on the normal business of the source database; during the process of reading and parsing the write-ahead log entries, each write-ahead log entry is read and parsed only once, reducing the occupation of system resources, and multi-threading is used for concurrent parsing, improving the efficiency; each group set still has an independent data dictionary during the parsing process, reducing the program complexity and the probability of errors.

[0054] In the embodiment of the present application, a write-ahead log receiving connection is established with the source database through a group set once, thus avoiding excessive connections with the source database, reducing the pressure on the source database and the impact on normal business. When reading and parsing the write-ahead log entries, each write-ahead log entry received in a write-ahead log is parsed only once, thus reducing the waste of system resources.

[0055] Embodiment 2

[0056] Figure 3 The flowchart of a database whole instance synchronization method provided by the second embodiment of the present invention. This embodiment is based on the above embodiment, and specifically describes the step of parsing each write-ahead log entry with reference to the data dictionary corresponding to the monitored parsing queue in step S1042 of the above embodiment to obtain the whole instance source end data record. As Figure 3 shown, the method includes:

[0057] Step S201, create corresponding data dictionaries for each group set associated with the whole instance in the source database.

[0058] Optionally, creating corresponding data dictionaries for each group set associated with the whole instance in the source database includes: connecting to the source database and obtaining the identifiers of each group set associated with the whole instance in the source database; connecting each group set in sequence according to the identifiers and obtaining the attribute information of each group set, where the attribute information includes grouping method information and table information; generating corresponding data dictionaries according to the attribute information of each group set.

[0059] Step S202, establish a write-ahead log receiving connection with the source database through any one group set once, and establish a corresponding parsing queue for each group set.

[0060] Optionally, before allocating each write-ahead log entry in the write-ahead log to the corresponding parsing queue, it further includes: establishing a parsing thread for each group set that matches the parsing queue, where the parsing thread is used to monitor and operate on the matching parsing queue; establishing a write-ahead log reading thread.

[0061] Step S203: When a write-ahead log is received through the write-ahead log receiving connection, allocate each write-ahead log entry in the write-ahead log to the corresponding parsing queue.

[0062] Optionally, when a write-ahead log is received through the write-ahead log receiving connection, allocating each write-ahead log entry in the write-ahead log to the corresponding parsing queue includes: when a write-ahead log is received through the write-ahead log receiving connection, reading the write-ahead log through a log reading thread to obtain write-ahead log entries, where each write-ahead log entry is marked with the identifier of the grouping set; allocating each write-ahead log entry to the corresponding parsing queue according to the marked identifier of the grouping set.

[0063] Step S204: Monitor the corresponding parsing queue through each parsing thread respectively. When it is determined that there is data stored in the parsing queue, obtain the data dictionary corresponding to the parsing queue.

[0064] Step S205: Sequentially extract the stored write-ahead log entries from the monitored parsing queue through the parsing connection.

[0065] Step S206: When it is determined that the write-ahead log entry includes data manipulation language information, parse the data manipulation voice information with reference to the data dictionary corresponding to the monitored parsing queue to obtain the source-side operation data record.

[0066] Among them, in this embodiment, when a write-ahead log entry is extracted from the parsing queue by the parsing thread, the information in the write-ahead log entry will be identified. When it is determined that the write-ahead log entry includes data manipulation language information, for example, update, delete, etc., at this time, only data modification and other related operations are involved in the source database, and there is no data addition operation. At this time, only the data manipulation voice information needs to be parsed with reference to the data dictionary corresponding to the monitored parsing queue to obtain the data manipulation voice information.

[0067] In a specific implementation, since only relevant information such as field ID and update flag is included in the write-ahead log entry, for the specific name of the field, the specific field name corresponding to the field ID can be queried from the data dictionary according to the ID. In addition, for the update flag, the data before and after the update can be queried from the data dictionary, so as to parse the data manipulation voice information in the write-ahead log entry with reference to the corresponding data dictionary to obtain more detailed data manipulation voice information. Of course, this is only an example in this embodiment, and the specific method for obtaining the data manipulation language information is not limited.

[0068] Step S207: When it is determined that the write-ahead log entry includes data definition language information, update the data dictionary corresponding to the monitored parsing queue with reference to the data definition language information, and obtain the source-side defined data record according to the data definition language information.

[0069] Among them, when it is determined that the write-ahead log entry includes data definition language information, for example, when a new parameter is redefined or a new table is added, at this time, there are new operations on the data in the source data. Since there is no relevant content in the data dictionary corresponding to the parsing queue, it is necessary to update the data dictionary corresponding to the parsing queue according to the definition language information to ensure that the newly added content in the source database can be recorded in the corresponding data dictionary, so as to facilitate the accuracy of subsequent database synchronization. In this case, after the data dictionary is updated, directly obtain the source-side defined data record according to the data definition language information.

[0070] Step S208: Obtain the whole instance source-side data record according to the source-side operation data record and the source-side defined data record.

[0071] Among them, in this embodiment, after obtaining the source-side operation data record and the source-side defined data record, it is equivalent to parsing out all the information in the source database. At this time, the source-side operation data record and the source-side defined data record are combined in a certain way to obtain the whole instance source-side data record, so as to ensure the integrity of the data. The combination method can be to splice the source-side operation data record and the source-side defined data record in sequence, etc. Of course, only examples are given in this embodiment, and the specific method for obtaining the whole instance source-side data record is not limited.

[0072] Step S209: Synchronize the whole instance source-side data record to the target database.

[0073] In the embodiment of the present application, a write-ahead log receiving connection is established with the source database through a grouping set, so as to avoid excessive connections with the source database, reduce the pressure on the source database and the impact on normal services. When reading and parsing the write-ahead log entries, each write-ahead log entry in a received write-ahead log is only parsed once, thus reducing the waste of system resources.

[0074] Embodiment III

[0075] Figure 4 A schematic structural diagram of a database whole instance synchronization device provided in Embodiment III of the present invention. As Figure 4 shown, the device includes: a data dictionary creation module 310, a parsing queue establishment module 320, a write-ahead log entry allocation module 330, and a whole instance source-side data synchronization module 340.

[0076] Among them, the data dictionary creation module 310 is used to create corresponding data dictionaries for each group set associated with the entire instance in the source database;

[0077] The parse queue establishment module 320 is used to establish a write-ahead log reception connection with the source database through any one group set and establish corresponding parse queues for each group set;

[0078] The write-ahead log entry allocation module 330 is used to allocate each write-ahead log entry in the write-ahead log to the corresponding parse queue when the write-ahead log is received through the write-ahead log reception connection;

[0079] The entire instance source-side data synchronization module 340 is used to parse each parse queue with reference to the corresponding data dictionary to obtain the entire instance source-side data records and synchronize the entire instance source-side data records to the target database.

[0080] Optionally, the data dictionary creation module is used to connect to the source database and obtain the identifiers of each group set associated with the entire instance in the source database;

[0081] Connect to each group set in sequence according to the identifier and obtain the attribute information of each group set, where the attribute information includes grouping method information and table information;

[0082] Generate corresponding data dictionaries according to the attribute information of each group set.

[0083] Optionally, the device further includes a thread establishment module, which is used to establish parse threads matching the parse queues for each group set, where the parse threads are used to monitor and operate on the matching parse queues;

[0084] Establish a write-ahead log reading thread.

[0085] Optionally, the write-ahead log entry allocation module includes: a write-ahead log entry reading unit, which is used to read the write-ahead log to obtain write-ahead log entries through the log reading thread when the write-ahead log is received through the write-ahead log reception connection, where each write-ahead log entry is marked with the identifier of the group set;

[0086] The write-ahead log entry allocation unit is used to allocate each write-ahead log entry to the corresponding parse queue according to the marked identifier of the group set.

[0087] Optionally, the entire instance source-side data synchronization module includes an entire instance source-side data record acquisition unit, which is used to monitor each matching parse queue through each parse thread, and obtain the data dictionary corresponding to the parse queue when it is determined that there is data stored in the parse queue;

[0088] Extract the stored pre - write log entries from the monitored parsing queue in sequence through the parsing connection, and parse each pre - write log entry with reference to the data dictionary corresponding to the monitored parsing queue to obtain the whole - instance source - end data record.

[0089] Optionally, the whole - instance source - end data record obtaining unit is further configured to, when it is determined that the pre - write log entry includes data manipulation language information, parse the data manipulation language information with reference to the data dictionary corresponding to the monitored parsing queue to obtain the source - end operation data record;

[0090] When it is determined that the pre - write log entry includes data definition language information, update the data dictionary corresponding to the monitored parsing queue with reference to the data definition language information, and obtain the source - end definition data record according to the data definition language information;

[0091] Obtain the whole - instance source - end data record according to the source - end operation data record and the source - end definition data record.

[0092] Optionally, the whole - instance source - end data synchronization module includes a whole - instance source - end data synchronization unit for establishing a connection with the target database;

[0093] Apply the whole - instance source - end data record to the target database to synchronize the whole - instance in the source database.

[0094] The whole - instance synchronization device for the database provided by the embodiments of the present invention can execute the database whole - instance synchronization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0095] Embodiment 4

[0096] Figure 5 The structural schematic diagram of the electronic device 10 that can be used to implement the embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0097] As Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0098] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the database whole instance synchronization method.

[0100] In some embodiments, the database whole instance synchronization method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the database whole instance synchronization method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the database whole instance synchronization method by any other appropriate means (for example, by means of firmware).

[0101] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0103] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0105] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0106] A computing system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0107] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0108] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for synchronizing an entire database instance, characterized in that: include: Create a corresponding data dictionary for each grouping set associated with the entire instance in the source database; Establishing a write-ahead log receiving connection with the source database through any grouping set, and establishing a corresponding parsing queue for each grouping set; When a write-ahead log is received through the write-ahead log receiving connection, each write-ahead log entry in the write-ahead log is assigned to a corresponding parsing queue; Each parsing queue is parsed with reference to the corresponding data dictionary to obtain the source data record of the entire instance, and the source data record of the entire instance is synchronized to the target database.

2. The method according to claim 1, characterized in that The step of creating a corresponding data dictionary for each grouping set associated with the entire instance in the source database includes: Connecting to the source database and obtaining identifiers of each grouping set associated with the entire instance in the source database; Connecting each of the grouping sets in sequence according to the identifier, and acquiring attribute information of each of the grouping sets, wherein the attribute information includes grouping mode information and table information; The corresponding data dictionary is generated according to the attribute information of each of the grouping sets.

3. The method according to claim 1, characterized in that Before allocating each write-ahead log entry in the write-ahead log to the corresponding parsing queue, the method further includes: Establishing a parsing thread matching the parsing queue for each of the grouping sets, wherein the parsing thread is used to monitor and operate the matching parsing queue; Establish a write-ahead log reading thread.

4. The method according to claim 3, characterized in that When the write-ahead log is received through the write-ahead log receiving connection, allocating each write-ahead log entry in the write-ahead log to a corresponding parsing queue includes: When a write-ahead log is received through the write-ahead log receiving connection, the write-ahead log is read through the log reading thread to obtain write-ahead log entries, wherein each of the write-ahead log entries is marked with an identifier of a grouping set; Each of the write-ahead log entries is allocated to the corresponding parsing queue according to the identifier of the marked grouping set.

5. The method according to claim 3, characterized in that: The step of parsing each of the parsing queues with reference to the corresponding data dictionary to obtain the source data record of the entire instance includes: The matched parsing queues are monitored respectively by each of the parsing threads, and when it is determined that there is data stored in the parsing queue, the data dictionary corresponding to the parsing queue is obtained; The stored write-ahead log entries are sequentially extracted from the monitored parsing queue through the parsing connection, and each of the write-ahead log entries is parsed with reference to the data dictionary corresponding to the monitored parsing queue to obtain the source data record of the entire instance.

6. The method according to claim 5, characterized in that The step of parsing each of the write-ahead log entries with reference to the data dictionary corresponding to the monitored parsing queue to obtain the source-end data record of the entire instance includes: When it is determined that the write-ahead log entry includes data operation language information, the data operation voice information is parsed with reference to the data dictionary corresponding to the monitored parsing queue to obtain a source-end operation data record; When it is determined that the write-ahead log entry includes data definition language information, the data dictionary corresponding to the monitored parsing queue is updated with reference to the data definition language information, and a source-end definition data record is acquired according to the data definition language information; The entire instance source data record is obtained according to the source operation data record and the source definition data record.

7. The method according to claim 1, characterized in that The step of synchronizing the source data records of the entire instance to the target database includes: Establishing a connection with the target database; Apply the source data records of the entire instance to the target database to synchronize the entire instance in the source database.

8. A database instance synchronization device, characterized in that: include: A data dictionary creation module is used to create corresponding data dictionaries for each grouping set associated with the entire instance in the source database; A parsing queue establishment module, used to establish a write-ahead log receiving connection with the source database through any grouping set, and establish a corresponding parsing queue for each grouping set; a write-ahead log entry allocation module, configured to allocate each write-ahead log entry in the write-ahead log to a corresponding parsing queue when the write-ahead log is received through the write-ahead log receiving connection; The whole instance source end data synchronization module is used to parse each of the parsing queues with reference to the corresponding data dictionary to obtain the whole instance source end data record, and synchronize the whole instance source end data record to the target database.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.

Images