Database data replication method and device, equipment and storage medium
By using the first log data acquisition thread in the database to collect target log packets that meet preset conditions, the problems of slow data replication speed and large resource utilization in the prior art are solved, and more efficient data replication is achieved.
Patent Information
- Application Number
- CN202411075580.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, opening a acquisition thread for each target end results in the problem of data replication too slowly and excessive resource utilization.
By acquiring the data copy request, the parsing request determines the source database and the target database. If the target log data fields of at least two target databases meet the preset log data duplication conditions, the first log data acquisition thread will be used to collect the target log data packet from the source database. , and the data is parsed and loaded by the parsing threads of each target database to complete the copying.
Reduce the number of acquisition threads, avoid resource waste, and improve data replication speed.
Smart Images

Figure CN119003241A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and in particular to a method, device, equipment and storage medium for replicating data in a database. Background Art
[0002] Active-active database management system replication technology is a logical replication technology based on the redo log of a relational database. The redo log of a relational database is a mechanism for ensuring the consistency of database data. It generally records the changes in all data files of a relational database. From these changes, the data changes in the tables in the database management system can be analyzed and these changes can be copied to another database management system in a timely manner, thereby achieving the purpose of data replication.
[0003] Due to the increasing demand for data analysis and multiple backups, data in many source databases often needs to be distributed to different target databases. Since the requirements of each target are different, the data that needs to be collected is also different. In the prior art, a collection thread is generally opened for each target to complete data collection and analysis.
[0004] In the process of data replication, there may be a large amount of duplicate data. However, by using the method of opening a collection thread for each target end in the prior art, the data replication speed may be too slow and too many resources may be occupied. Summary of the invention
[0005] The present invention provides a method, device, equipment and storage medium for data replication of a database, so as to solve the problem in the prior art that a corresponding acquisition thread is opened for each target end, resulting in slow data replication speed and excessive resource occupation.
[0006] According to one aspect of the present invention, a method for replicating data in a database is provided, comprising:
[0007] Obtaining a data replication request, parsing the data replication request to determine a source database, a target database, and target log data fields;
[0008] If the target log data fields corresponding to at least two target databases meet the preset log data duplication condition, a first log data collection thread is used to collect the target log data packets corresponding to the target log data fields from the source database;
[0009] The target log data packets corresponding to each target database are parsed according to the parsing threads corresponding to each target database to obtain the target data, and the target data is loaded into the corresponding target database to complete the copying of the target data.
[0010] According to another aspect of the present invention, there is provided a data replication device for a database, comprising:
[0011] A data replication request acquisition module, used to acquire a data replication request, parse the data replication request to determine a source database, a target database, and target log data fields;
[0012] A target log data packet collection module is used to collect target log data packets corresponding to the target log data fields from the source database using a first log data collection thread if the target log data fields corresponding to at least two target databases meet a preset log data duplication condition;
[0013] The data loading module is used to parse the target log data packets corresponding to each target database according to the parsing threads corresponding to each target database, obtain the target data, load the target data into the corresponding target database, and complete the replication of the target data.
[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0015] at least one processor; and
[0016] a memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database data replication method described in any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database data replication method described in any embodiment of the present invention when executed.
[0019] The technical solution of the embodiment of the present invention obtains a data replication request, parses the data replication request to determine the source database, the target database and the target log data field, and then directly uses the first log data collection thread to collect the target log data packet corresponding to the target log data field from the source database when the target log data field corresponding to at least two target databases meets the preset log data duplication condition. This can reduce the number of collection threads, avoid unnecessary waste of resources, and thus save collection resources and improve the data replication speed.
[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 is a flow chart of a method for replicating data in a database provided according to Embodiment 1 of the present invention;
[0023] Figure 2 is a flow chart of another database data replication method provided by Embodiment 2 of the present invention;
[0024] Figure 3 is a schematic diagram of the structure of a data replication device for a database provided according to Embodiment 3 of the present invention;
[0025] Figure 4 It is a structural schematic diagram of an electronic device for implementing the database data replication method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] Embodiment 1
[0029] Figure 1A flowchart of a method for replicating data in a database is provided for the first embodiment of the present invention. This embodiment is applicable to the case of replicating data from a source database to a target database. The method can be executed by a data replication device for a database. The data replication device for the database can be implemented in the form of hardware and / or software. The data replication device for the database can be configured in an electronic device.
[0030] like Figure 1 As shown, the method includes:
[0031] S110: Obtain a data replication request, and parse the data replication request to determine a source database, a target database, and target log data fields.
[0032] Wherein, the data replication request may be a data replication request issued by the target database to the source database, the source database may be a database providing a target log data field, the target database may be a database obtaining a target log data field, the target log data field may be a data change log record corresponding to the target data to be replicated by the target database, and the data change log record may be log data recording changes in the data stored in the source database, such as log data in which any data in the database is added, deleted, modified, or checked in the source database. By collecting and analyzing the data change log records in the target database, the data changes in the target data to be replicated in the target database in the source database can be obtained. The target log data field may be a log data field corresponding to the target data to be replicated from the source database by the target database, and the log data field records the changes in the target data in the source database. Alternatively, the target log data field may also be a log data number corresponding to the target data to be replicated from the source database by the target database, etc., used to characterize the changes in the target data. Wherein, the target data may refer to the data that the target database needs to replicate from the source database, and the target data may be obtained by collecting the log data records generated by the source data when adding, deleting, modifying, or checking the data, and parsing the log data records. The target data can be user-specified data or data added, deleted, modified, or queried in the source database over a period of time.
[0033] Specifically, the data replication request is parsed to determine the source database, target database, and target log data fields, including:
[0034] Parsing the data replication request to obtain a data replication rule, wherein the data replication rule includes: source database information, target database information and log data number;
[0035] According to the source database information, the target database information and the log data number, the source database, the target database and the target log data field are determined.
[0036] The data replication rule may be the replication rule and replication condition information defined when the target database initiates a data replication request to the source database, which can be obtained by parsing the data replication request, including source database information, target database information and log data number, wherein the source database information may be the address information of the source database or other information used to identify the source database, the target database information may be the address information of the target database or other information used to identify the target database, the log data number may be the log data number of the log data of the target log data field in the source database, the log data number may be the identification information of the log data, and may be used to characterize the generation time of the log data and the number of the log data. The data replication rule also includes: error handling information, whether the data is encrypted, whether it is compressed, and other rule information related to data replication. The error handling information may be used to provide a processing basis for situations such as replication failure or parsing failure during the data replication process, and it should be noted that one data replication request corresponds to one data replication rule, and one data replication rule corresponds to one target database.
[0037] Specifically, the source database, target database and target log data field are determined according to the source database information, the target database information and the log data number. The address information corresponding to the source database can be determined according to the source database information, and the source database can be determined through the address information corresponding to the source database; the address information of the target database can be determined according to the target database information, and the target database can be determined through the address information corresponding to the target database; the target log data field can be determined according to the log data number information of the target log data field in the source database.
[0038] S120: If target log data fields corresponding to at least two target databases meet a preset log data duplication condition, a first log data collection thread is used to collect target log data packets corresponding to the target log data fields from the source database.
[0039] Among them, the preset log data duplication condition can be pre-set as a condition for judging whether the target log field in each target log data can meet the condition of using the same data collection thread. It can be used to determine whether two or more target log data can meet the condition of using the same data collection thread. The condition of using the same data collection thread for two or more target data can be determined by the log data generation time or the number information of the log data. Exemplarily, the preset log data duplication condition can be: when the target log data field that needs to be collected by one of the two target databases reaches a set percentage value with the target log data field that needs to be collected by the other target database, the two target databases share a data collection thread, wherein the percentage value can be 80% or 100%, and the specific value of the percentage value can be set by the technicians in this field; it can also be: if the target log data field of target database A is included by target database B, then target database A and target database B share a data collection thread. For example, in Oracle, each record has a corresponding redo log sequence number and a corresponding offset position. By comparing the sequence number and position of the redo log to be captured to match it, it can be determined whether it is included. The present invention does not specifically limit the setting of the preset log data repetition condition and the method for determining the preset log data repetition condition, and those skilled in the art can set them according to their needs and experience.
[0040] Among them, the first log data collection thread can be a log data collection thread that collects the target log data packets required for one or more target databases that share the same log data collection thread. For example, if the target log data field of target database A and the target log data field of database B meet the preset log data duplication condition, then a first log data collection thread can be used to collect the target log data packets corresponding to the target log data fields that need to be copied by target database A and target database B, wherein the target log data packet can be a log data packet corresponding to the target log data field, and the log data packet contains the target log data field corresponding to the target log data obtained by the target database from the source database.
[0041] Specifically, if the target log data fields corresponding to at least two target log databases meet the preset log data duplication condition, the first log data collection thread is used to collect the target log data packets corresponding to the target log data fields from the source database, including:
[0042] Compare the log data numbers corresponding to at least two target databases to determine duplicate log data numbers;
[0043] If the ratio of duplicate log data numbers reaches a preset threshold, target log data packets corresponding to at least two target databases are collected through the first log data collection thread.
[0044] Specifically, the log data numbers corresponding to at least two target databases are compared to determine the duplicate log data numbers. If the ratio of duplicate log data numbers reaches a preset threshold, the target log data packets corresponding to at least two target databases are collected through a first log data collection thread. This may be after determining the log data number information of two or more target databases, the log data number information of the two or more target databases is compared to obtain the duplicate ratio of the two or more target log data numbers, and the corresponding duplicate ratio is determined according to the obtained duplicate ratio. If the duplicate ratio is greater than or equal to the preset threshold, a first log data collection thread is used to collect the target log data packets corresponding to at least two target databases. The preset threshold may be set by the user according to the needs, such as 80%, 100%, etc., and the present invention does not specifically limit this.
[0045] Exemplarily, if the target database includes a target database A and a target database B, after obtaining the data replication requests corresponding to these target databases, the log data numbers corresponding to the target databases A and B can be parsed, and then by comparing the target log data number of the target database A with the target log data number corresponding to the target database B, it is determined whether the target log data field of the target database A and the target log data field of the target database B meet the preset log data duplication condition. If the comparison result of the target database A and the target database B can meet the preset log data duplication condition, only the first log data collection thread can be opened to collect the target log data packets corresponding to the log data numbers of the target database A and the target database B; if the comparison result of the target database A and the target database B cannot meet the preset log data duplication condition, only the first log data collection thread can be used to collect the target log data packets corresponding to the log data number of the target database A, and a new log data collection thread can be opened to collect the target log data packets corresponding to the log data number of the target database B. If, during the collection process, a data collection request from the target database C is received again, the log data number corresponding to the target database C can be compared with the log data number collected by the opened log data collection thread, and it can be determined whether the log data number corresponding to the target database C and the log data number collected by the opened data collection thread meet the preset log data duplication condition. If the comparison result of the log data number collected by the target database C and any opened data collection thread can meet the preset log data duplication condition, then a new log data collection thread can be omitted, and the log data collection thread that meets the preset log data duplication condition can be directly used to collect the target log data packet corresponding to the log data number for the target database C; if the comparison result of the log data number collected by the target database C and any opened log data collection thread does not meet the preset log data duplication condition, then a new log data collection thread can be opened, and the newly opened log data collection thread can be used to collect the target log data packet corresponding to the log data number for the target database C.
[0046] Optionally, after using the first log data collection thread to collect a target log data packet corresponding to the target log data field from the source database, the method further includes:
[0047] Obtaining preset threshold information of disk space and memory space of the first log data collection thread;
[0048] If the difference between the log data parsing time of the parsing thread corresponding to any target database and the log data collection time of the first log data collection thread reaches a preset difference, and the disk space and / or memory space corresponding to the first log data collection thread reaches a preset threshold, then start the second log data collection thread;
[0049] The target log data packets of the target database whose log data parsing time reaches a preset difference with the log data collection time of the first log data collection thread are collected from the source database by the second log data collection thread, and the remaining target log data packets of the target database are collected from the source database by the first log data collection thread.
[0050] Among them, the second log data collection thread is a log data collection thread that is opened for the target database of the target log data packet that has a too slow parsing speed and / or occupies too much space when the disk space and / or memory space of the first log data collection thread reaches the upper limit, and is used to collect the corresponding target log data fields for the target database of the target log data packet that has a too slow parsing speed and / or occupies too much space. Among them, the preset difference between the log data parsing time of the parsing thread and the log data collection time of the first log data collection thread can be set by those skilled in the art according to needs, and the present invention does not specifically limit the size of the preset difference. The preset thresholds of the disk space and memory space corresponding to the first log data collection thread can be set by those skilled in the art according to needs, and the present invention does not specifically limit the size of the preset difference.
[0051] Specifically, if the difference between the log data parsing time of the parsing thread corresponding to any target database and the log data collection time of the first log data collection thread reaches a preset difference, and the disk space and / or memory space corresponding to the first log data collection thread reaches a preset threshold, then the second log data collection thread is started, which can be after the memory space and disk space information of the first log data collection thread is determined, the memory space and disk space occupancy of the first log data collection thread is detected in real time, if it is detected that the disk space and / or memory space corresponding to the first log data collection thread reaches a preset threshold set by the user, then from each target database corresponding to the first log data collection thread, the target database whose log data parsing time and the log data collection time of the first log data collection thread reach the preset difference is determined. The second log data collection thread is started, and the target log data packets of the target database whose speed is less than the collection speed of the first log data collection thread are collected, and the target log data packets of the target database whose log data parsing time and log data collection time have a difference of the preset difference are collected from the source database by the second log data collection thread, and the target log data packets of the remaining target databases are collected from the source database by the first log data collection thread.
[0052] Optionally, if the log data parsing time and the current time of the first log data collection thread are still not reduced below the preset value, and the memory and disk space occupied by the data that still needs to be collected by the target data have reached the preset value, then a new fourth log data collection thread is opened for the corresponding target database to collect the data fields required by this target database; if after opening the fourth log data collection thread, the log data parsing time and the current time of the first log data collection thread are still not reduced below the preset value, and the memory and disk space occupied by the data that still needs to be collected by the target data have reached the preset value, then a new fifth log data collection thread is opened for the corresponding target database to collect the data fields required by this target database; and so on, multiple collection threads are split out to avoid affecting the copying progress of all target databases due to the large amount of data required by a certain target database.
[0053] Exemplarily, the data collection thread has a fixed collection speed, for example, 100MB of data can be collected per second. At the same time, each target database corresponds to a parsing thread, which is responsible for parsing the collected data, and the data structure and complexity of different target databases are different, so the parsing speed will also be different. In the process of detecting the log data parsing time of the parsing thread corresponding to the disk space, memory space and target database, if the log data parsing time of one or some target databases is too long, it may cause data accumulation. If the data accumulation causes the disk and / or memory space corresponding to the first log data collection thread to reach a preset threshold, the second log data collection thread can be started, and the data of the target databases with longer log data parsing time can be collected from the source database through the second log data collection thread. The first log data collection thread continues to collect data from the remaining target databases.
[0054] Optionally, in order to ensure timely response to insufficient disk and / or memory space, a monitoring mechanism may be set up. This monitoring mechanism will detect in real time the disk and / or memory space occupancy corresponding to the first log data collection thread. Once it is detected that the disk and / or memory space is close to a preset threshold (for example, the remaining space is less than 10% or 5%), the monitoring mechanism will trigger the above-mentioned response plan, thereby avoiding data duplication accidents caused by the disk space and / or memory space of the first log data collection thread reaching the upper limit.
[0055] For the second log data collection thread, if the log data parsing time of the parsing thread corresponding to the target database and the log data collection time of the first log data collection thread are less than a preset difference, the second log data collection thread is closed;
[0056] The step of using a first log data collection thread to collect a target log data packet corresponding to the target log data field from the source database is executed.
[0057] By shutting down the second log data collection thread when the log data parsing time of the parsing thread corresponding to the target database is less than a preset difference from the log data collection time of the first log data collection thread, the data collection strategy can be dynamically adjusted according to changes in the parsing speed of the target database, thereby avoiding unnecessary waste of resources and ensuring the efficiency of data collection and analysis.
[0058] S130 , parsing the target log data packets corresponding to the target databases according to the parsing threads corresponding to the target databases, obtaining the target data, loading the target data into the corresponding target databases, and completing the replication of the target data.
[0059] Specifically, the target log data packets corresponding to each target database are parsed according to the parsing threads corresponding to each target database to obtain the target data, and the target data is loaded into the corresponding target database to complete the replication of the target data, including:
[0060] Obtain target log data packets corresponding to log data numbers of each target database from the disk and memory space of the data collection thread corresponding to each target database;
[0061] Analyze the target log data packets obtained by each target database through the analysis thread of each target database to obtain the target log data;
[0062] Load the target data into the corresponding target database to complete the replication of the target data.
[0063] Specifically, the target log data packets corresponding to the log data numbers of each target database are obtained from the disk and memory space of the data acquisition thread corresponding to each target database; the target log data packets obtained by each target database are parsed by the parsing thread of each target database to obtain the target data, which can be located at the disk and memory space stored by the log data acquisition thread corresponding to each target database. When each log data acquisition thread collects data, it will save the target log data packet in the disk and memory space corresponding to the corresponding data acquisition thread according to the number information of the target log data packet in the source database. When the target database acquires the target log data packet, the target log data packet corresponding to each target database can be accurately extracted from the disk and memory space according to these log data numbers. After extracting the target log data packets, these data packets can be passed to the parsing thread corresponding to the target database. Each parsing thread is responsible for parsing the target log data packet of its corresponding target database, and then the corresponding target data is obtained according to the target data recorded in the target log data packet. The parsing method can be carried out by converting the data format, cleaning the data or adjusting the structure. The present invention does not make specific restrictions on this, and the user can set it according to the storage format of the target log data packet or its own needs.
[0064] Specifically, the target data is loaded into the corresponding target database to complete the replication of the target data, which can be obtained by parsing the target log data packet, and then performing necessary format conversion and preprocessing on the target data according to the format and requirements of the target database to ensure data compatibility and consistency. Then, the data loading operation is performed to accurately write the processed data into the target database, while handling possible conflicts and abnormal situations, thereby completing the operation of loading the target data into the corresponding target database and completing the replication of the target data.
[0065] Exemplarily, in an embodiment of the present invention, the target database can be one or more. Exemplarily, when the target database is one and the data corresponding to the target log data field is a redo log, the data replication request of the target database can be directly obtained, and after parsing the replication request, the replication rule information required by the target database is obtained, and then a first log data collection thread is started, and the corresponding memory space and disk space are allocated to the first log data collection thread, which is used to collect the target log data information corresponding to the target data from the source database, and store it in the disk and memory space allocated to the first log data collection thread, and then the collected log data is parsed by the corresponding log data parsing thread to obtain the target data. And the target data is sent to the loading unit through the corresponding data sending thread and loaded into the target database. It should be noted that after the parsing unit parses the log data collected by the first log data collection thread, the parsing completed log data can be deleted to release the disk and memory space of the first log data collection thread to facilitate the collection of subsequent data.
[0066] Exemplarily, when there are two target databases and the data corresponding to the target log data field is a redo log, the replication rule information of the second target database that initiates the data replication request can be obtained, and then the target log data field information that needs to be replicated by the second target database that initiates the data replication request is obtained through the replication rule information, and compared with the target log data field information corresponding to the first target database that initiates the data replication request, it is determined according to the comparison result whether the degree of repetition of the target log data fields of the two target databases can reach a preset threshold. If the preset threshold can be reached, then only the first log data collection thread needs to be started to collect the target log data fields that need to be replicated for the two target databases. At this time, when the disk and memory space of the first log data collection thread are released, the parsing units corresponding to the two target databases need to complete the parsing of the log data collected by the first log data collection thread at the same time, and then the data that has been parsed can be deleted to release the disk and memory space of the first log data collection thread, so as to facilitate the collection of subsequent data.
[0067] When the target log data fields for the two target databases are collected through the first log data collection thread, the remaining disk and memory storage space of the first log data collection thread needs to be detected in real time. If the parsing speed of any parsing thread in the two target databases is slow, data accumulation may occur. At this time, when the disk and memory space corresponding to the first log data collection thread are about to reach the storage upper limit, the second log data collection thread may be started, and the data of the target databases with slower parsing speeds may be collected from the source database through the second log data collection thread, while the first log data collection thread continues to collect the data of the remaining target databases. If, in the subsequent parsing process, the speed of the parsing thread with too slow parsing speed is improved and can keep up with the speed of the first log data collection thread, the second log data collection thread may be closed, and the first log data collection thread may continue to be used to collect the corresponding target log data field data for the two target databases.
[0068] The technical solution of the embodiment of the present invention obtains the storage upper limit information of the disk and memory space of the first log data collection thread, and when the parsing speed of the log parsing thread corresponding to any target database is less than the collection speed of the first log data collection thread, and the disk and memory space corresponding to the first log data collection thread reach the storage upper limit, the second log data collection thread is started to collect the target log data packet of the target database whose parsing speed is less than the collection speed from the source database, and when the parsing speed of the parsing thread corresponding to the target database whose parsing speed is less than the collection speed is greater than or equal to the collection speed of the first log data collection thread, the second log data collection thread is closed, and the first log data collection thread is continued to be used to collect the target log data packet corresponding to the target log data field from the source database. The technical solution can take the parsing speed of each target database and the collection speed of the first log data collection thread into consideration on the basis of collecting the target log data fields for multiple target databases through one data collection thread, thereby avoiding possible copying errors without wasting collection resources, and improving the flexibility of data copying.
[0069] Embodiment 2
[0070] Figure 2 Flow chart of a database data replication method provided in Embodiment 2 of the present invention. This embodiment specifically defines that when the target log data fields between at least two target databases and other target databases do not meet the preset log data duplication conditions, a third log data collection thread is allocated to each target database that does not meet the preset log data duplication conditions. The scheme not described in detail in the embodiment of the present invention is shown in the above embodiment. Figure 2 As shown, the method includes:
[0071] S210: Obtain a data replication request, parse the data replication request to determine a source database, a target database, and target log data fields.
[0072] S220 . If the target log data fields between at least one target database and other target databases do not satisfy the preset log data duplication condition, respectively allocate a third log data collection thread to each target database that does not satisfy the preset log data duplication condition.
[0073] Among them, the third log data collection thread is a data collection thread that is opened when the target log data field between at least one target database and other target databases does not meet the preset log data duplication condition, and is used to collect the corresponding target log data field for the target database whose target log data field between the target database and other target databases does not meet the preset log data duplication condition. For example, if the target database B and the target database A do not meet the preset log data duplication condition, the first log data collection thread is used to collect the corresponding target log data field for the target database A, and the third log data collection thread is opened to collect the corresponding target log data field for the target database B. It should be noted that in a specific replication scenario, the number of target databases can also be multiple, such as 3, 5, 10, etc., and the present invention does not specifically limit this. When the number of target databases is more than two, the thread opening method of two target databases can be used for processing.
[0074] Specifically, if the target log data field between at least one target database and other target databases does not meet the preset log data duplication condition, a third log data collection thread is allocated to each target database that does not meet the preset log data duplication condition. When data replication requests from at least two target databases are obtained, the target log data fields between each target database and other target databases are compared. If there is one or more that do not meet the preset log data duplication condition, a corresponding number of third log data collection threads need to be opened.
[0075] S230 . Collect target log data corresponding to the target log data field from the source database through a third log data collection thread based on the target log data field of the target database that does not satisfy a preset log data duplication condition.
[0076] Specifically, after the third log data collection thread is started, the third log data collection thread may be used to collect corresponding target log data field information from the source database for the target database that does not meet the preset log data duplication condition.
[0077] Exemplarily, when there are two target databases and the data corresponding to the target log data field is a redo log, the replication rule information of the second target database that initiates the data replication request can be obtained, and then the target log data field information that needs to be replicated by the second target database that initiates the data replication request is obtained through the replication rule information, and compared with the target log data field information corresponding to the first target database that initiates the data replication request, it is determined according to the comparison result whether the degree of repetition of the target log data fields of the two target databases can reach a preset threshold. If the preset threshold cannot be reached, it is necessary to start a third log data collection thread, and continue to collect the target log data field for the first target database that initiates the data replication request through the first log data collection thread, and collect the target log data field for the second target database that initiates the data replication request through the third log data collection thread.
[0078] Exemplarily, if the number of target databases is more than two, for example, target database A, target database B, and target database C, after obtaining the data replication requests corresponding to these target databases, the target log data fields corresponding to the target databases A, target database B, and target database C can be parsed to obtain each of them, and then the target log data fields of the target database A are compared with the target log data fields corresponding to the target databases B and C to determine whether the preset log data duplication conditions are met. If the comparison results of the target database B and the target database C with the target database A all meet the preset log data duplication condition, then only the first log data collection thread can be started to collect the target log data fields of the target databases A, B and C, and the third log data collection thread will not be started; if the comparison result of the target database B with the target database A meets the preset log data duplication condition, and the comparison result of the target database C with the target database A does not meet the preset log data duplication condition, then the first log data collection thread can be started to collect the target log data fields of the target database A and the target database B, and a third log data collection thread can be started to collect the target log data field of the target database C; if the comparison result of the target database B with the target database A does not meet the preset log data duplication condition, and the comparison result of the target database C with the target database A also does not meet the preset If the log data duplication condition is met, the first log data collection thread can be started to collect the target log data field of the target database A, and two third log data collection threads can be started to collect the target log data fields of the target database B and the target database C respectively; if the comparison result between the target database B and the target database A does not meet the preset log data duplication condition, and the comparison result between the target database C and the target database A also does not meet the preset log data duplication condition, then the target log data fields of the target database B and the target database C are compared to see whether they meet the preset duplication condition. If the comparison result between the target database B and the target database C meets the preset log data duplication condition, the first log data collection thread can be started to collect the target log data field of the target database A, and a third log data collection thread can be started to collect the target log data fields of the target database B and the target database C respectively.
[0079] S240 , parsing the target log data packets corresponding to each target database according to the parsing threads corresponding to each target database, obtaining target data, loading the target data into the corresponding target database, and completing the replication of the target data.
[0080] The technical solution of the embodiment of the present invention obtains a data replication request, and when a target log data field between at least one target database and other target databases does not meet a preset log data duplication condition, a third log data collection thread is respectively allocated to each target database that does not meet the preset log data duplication condition, and the third log data collection thread collects target data corresponding to the target log data field from the source database based on the target log data field of the target database that does not meet the preset log data duplication condition, and parses the target log data packet corresponding to each target database according to the parsing thread corresponding to each target database to obtain the target data, and loads the target data into the corresponding target database to complete the replication of the target data, and can start a minimum number of data collection threads by comparing the target log data fields corresponding to at least two target databases and setting the corresponding preset duplication condition, and by judging whether the target log data fields corresponding to at least two target databases meet the preset duplication condition, so as to further save collection resources and improve the efficiency of data replication.
[0081] Embodiment 3
[0082] Figure 3 This is a schematic diagram of the structure of a data replication device for a database provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a data replication request acquisition module 310, a target log data packet collection module 320, and a data loading module 330.
[0083] The data replication request acquisition module 310 is used to acquire a data replication request, parse the data replication request to determine the source database, the target database and the target log data fields.
[0084] The target log data packet collection module 320 is used to collect the target log data packets corresponding to the target log data fields from the source database using the first log data collection thread if the target log data fields corresponding to at least two target databases meet the preset log data duplication condition.
[0085] The data loading module 330 is used to parse the target log data packets corresponding to each target database according to the parsing threads corresponding to each target database, obtain the target data, load the target data into the corresponding target database, and complete the replication of the target data.
[0086] Furthermore, the data replication request acquisition module 310 is specifically used to:
[0087] Parsing the data replication request to obtain a data replication rule, wherein the data replication rule includes: source database information, target database information and log data number;
[0088] According to the source database information, the target database information and the log data number, the source database, the target database and the target log data field are determined.
[0089] Furthermore, the target log data packet collection module 320 is specifically used to:
[0090] Compare the log data numbers corresponding to at least two target databases to determine duplicate log data numbers;
[0091] If the ratio of duplicate log data numbers reaches a preset threshold, target log data packets corresponding to at least two target databases are collected through the first log data collection thread.
[0092] Furthermore, the target log data packet collection module 330 is specifically used to:
[0093] Obtain target log data packets corresponding to log data numbers of each target database from the disk and memory space of the data collection thread corresponding to each target database;
[0094] Analyze the target log data packets obtained by each target database through the analysis thread of each target database to obtain the target data;
[0095] Load the target data into the corresponding target database to complete the replication of the target data.
[0096] Furthermore, the target log data packet collection module 320 is further specifically used for:
[0097] After the first log data collection thread is used to collect the target log data packet corresponding to the target log data field from the source database, the preset threshold information of the disk space and the memory space of the first log data collection thread is obtained;
[0098] If the difference between the log data parsing time of the parsing thread corresponding to any target database and the log data collection time of the first log data collection thread reaches a preset difference, and the disk space and / or memory space corresponding to the first data collection thread / process reaches a preset threshold, then the second log data collection thread is started;
[0099] The target log data packets of the target database whose log data parsing time reaches a preset difference with the log data collection time of the first log data collection thread are collected from the source database by the second log data collection thread, and the remaining target log data packets of the target database are collected from the source database by the first log data collection thread.
[0100] Furthermore, the target log data packet collection module 320 is further specifically used for:
[0101] For a target database whose log data parsing time and the log data collection time of the first log data collection thread reach a preset difference, if the log data parsing time of the parsing thread corresponding to the target database and the log data collection time of the first log data collection thread are less than the preset difference, closing the second log data collection thread;
[0102] The step of using a first log data collection thread to collect a target log data packet corresponding to the target log data field from the source database is executed.
[0103] Furthermore, the target log data packet collection module 320 is further specifically used for:
[0104] If the target log data field between at least one target database and other target databases does not meet the preset log data duplication condition, respectively allocating a third log data collection thread to each target database that does not meet the preset log data duplication condition;
[0105] The target data corresponding to the target log data field is collected from the source database through the third log data collection thread based on the target log data field of the target database that does not meet the preset log data duplication condition.
[0106] The database data replication device provided in the embodiment of the present invention can execute the database data replication method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0107] Embodiment 4
[0108] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0109] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0110] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0111] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as data replication of a database.
[0112] In some embodiments, the data replication of the database may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data replication of the database described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data replication of the database in any other appropriate manner (e.g., by means of firmware).
[0113] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0114] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0115] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0116] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0117] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0118] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0119] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0120] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for replicating data in a database, characterized in that: include: Obtaining a data replication request, parsing the data replication request to determine a source database, a target database, and target log data fields; If the target log data fields corresponding to at least two target databases meet the preset log data duplication condition, a first log data collection thread is used to collect the target log data packets corresponding to the target log data fields from the source database; The target log data packets corresponding to each target database are parsed according to the parsing threads corresponding to each target database to obtain the target data, and the target data is loaded into the corresponding target database to complete the replication of the target data.
2. The method according to claim 1, characterized in that The step of parsing the data replication request to determine the source database, the target database, and the target log data field includes: Parsing the data replication request to obtain a data replication rule, wherein the data replication rule includes: source database information, target database information and log data number; According to the source database information, the target database information and the log data number, the source database, the target database and the target log data field are determined.
3. The method according to claim 1, characterized in that After the first log data collection thread is used to collect the target log data packet corresponding to the target log data field from the source database, the method further includes: Obtaining preset threshold information of disk space and memory space of the first log data collection thread; If the difference between the log data parsing time of the parsing thread corresponding to any target database and the log data collection time of the first log data collection thread reaches a preset difference, and the disk space and / or memory space corresponding to the first log data collection thread reaches a preset threshold, then start the second log data collection thread; The target log data packets of the target database whose log data parsing time reaches a preset difference with the log data collection time of the first log data collection thread are collected from the source database by the second log data collection thread, and the remaining target log data packets of the target database are collected from the source database by the first log data collection thread.
4. The method according to claim 3, characterized in that Also includes: For a target database whose log data parsing time and the log data collection time of the first log data collection thread reach a preset difference, if the log data parsing time of the parsing thread corresponding to the target database and the log data collection time of the first log data collection thread are less than the preset difference, closing the second log data collection thread; The step of using a first log data collection thread to collect a target log data packet corresponding to the target log data field from the source database is executed.
5. The method according to claim 1, characterized in that Also includes: If the target log data field between at least one target database and other target databases does not meet the preset log data duplication condition, respectively allocating a third log data collection thread to each target database that does not meet the preset log data duplication condition; The target log data packet corresponding to the target log data field is collected from the source database through the third log data collection thread based on the target log data field of the target database that does not meet the preset log data duplication condition.
6. The method according to claim 2, characterized in that If the target log data fields corresponding to at least two target databases meet the preset log data duplication condition, using the first log data collection thread to collect the target log data packets corresponding to the target log data fields from the source database includes: Compare the log data numbers corresponding to at least two target databases to determine duplicate log data numbers; If the ratio of the duplicate log data numbers reaches a preset threshold, target log data packets corresponding to at least two target databases are collected through the first log data collection thread.
7. The method according to claim 2, characterized in that The target log data packets corresponding to each target database are parsed according to the parsing threads corresponding to each target database to obtain target data, and the target data are loaded into the corresponding target database to complete the copying of the target data, including: Obtain target log data packets corresponding to log data numbers of each target database from the disk and memory space of the data collection thread corresponding to each target database; Parsing the target log data packets acquired by each target database through the parsing thread of each target database to obtain target log data; The target data is loaded into the corresponding target database to complete the copying of the target data.
8. A data replication device for a database, characterized in that: include: A data replication request acquisition module, used to acquire a data replication request, parse the data replication request to determine a source database, a target database, and target log data fields; A target log data packet collection module is used to collect target log data packets corresponding to the target log data fields from the source database using a first log data collection thread if the target log data fields corresponding to at least two target databases meet a preset log data duplication condition; The data loading module is used to parse the target log data packets corresponding to each target database according to the parsing threads corresponding to each target database, obtain the target data, load the target data into the corresponding target database, and complete the replication of the target data.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
