Data synchronization method, apparatus, computer program product, device, and storage medium

Abstract

This application provides a data synchronization method, comprising: in a source database system, obtaining metadata and a first data record to be synchronized; constructing a physical log of the metadata based on the metadata; constructing a physical log of the first data record based on the first data record; sending the physical log of the metadata and the physical log of the first data record to a destination database system; in the destination database system, parsing the physical log of the metadata and the physical log of the first data record to obtain a physical log parsing result; and performing data synchronization in the destination database system according to the physical log parsing result. This application also provides a data synchronization device, a computer program product, an electronic device, and a storage medium. This application enables data synchronization using physical logs without incurring hardware overhead in the database system, reducing user costs.

Description

Technical Field

[0001] This application relates to data synchronization technology, and more particularly to data synchronization methods, apparatus, computer program products, electronic devices and storage media. Background Technology

[0002] In HTAP (Hybrid Transaction and Analytical Processing) scenarios, it is typically necessary to synchronize data from a source database system (usually an OLTP database system) to a destination database system (usually an OLAP database system) for further data analysis and processing. To implement this synchronization, the source database system needs to generate an additional logical log of the data and synchronize it to the destination database system. This additional generation of the logical log for synchronization incurs significant overhead on the source database system and consumes a large amount of storage space, increasing hardware costs and hindering the widespread adoption of HTAP scenarios. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data synchronization method, apparatus, computer program product, electronic device, and storage medium, which can automatically synchronize data between different databases, ensure data consistency, improve user efficiency, reduce data loss caused by tedious manual data synchronization, and ensure data security and accuracy. At the same time, using physical logs for data synchronization does not require the hardware overhead of the database system, reducing user costs.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data synchronization method, the method comprising: In the source database system, obtain the metadata that needs to be processed for data synchronization and the first data record that needs to be synchronized. The physical log of metadata is constructed based on the aforementioned metadata; Based on the first data record, construct the physical log of the first data record; Send the physical log of the metadata and the physical log of the first data record to the destination database system; In the destination database system, the physical logs of the metadata and the physical logs of the first data record are parsed to obtain the physical log parsing results; Based on the physical log parsing results, data synchronization is performed in the destination database system to update the data in the destination database system.

[0005] This application embodiment also provides a data synchronization device, the device comprising: The information transmission module is used to obtain the metadata that needs to be processed for data synchronization and the first data record that needs to be synchronized from the source database system. An information processing module is used to construct physical logs of metadata based on the aforementioned metadata; The information processing module is used to construct a physical log of the first data record based on the first data record; The information processing module is used to send the physical log of the metadata and the physical log of the first data record to the destination database system; The information processing module is used to parse the physical log of the metadata and the physical log of the first data record in the destination database system to obtain the physical log parsing result. The information processing module is used to synchronize data in the destination database system according to the physical log parsing results, so as to update the data in the destination database system.

[0006] In the above scheme, The information processing module is used to determine the modification operation type of the first data record when it is determined that the first data record needs to be synchronized based on the field information in the metadata. The information processing module is used to execute a construction strategy that matches the modification operation type of the first data record to obtain the physical log of the first data record.

[0007] In the above scheme, The information processing module is used to obtain the insertion offset parameter of the data page file and the value of each field of the first data record when the modification operation type of the first data record is a data record insertion operation; The information processing module is used to construct the physical log of the first data record based on the insertion offset parameter of the data page file and the value of each field of the first data record.

[0008] In the above scheme, The information processing module is used to obtain the deletion offset parameter of the data page file, the modification information of the deleted field, and the identification information of the deleted field when the modification operation type of the first data record is a data record deletion operation. The information processing module is used to construct the physical log of the first data record based on the deletion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field.

[0009] In the above scheme, The information processing module is used to obtain the update offset parameter of the data page file and the update value of the update field when the modification operation type of the first data record is a data record update operation. The information processing module is used to construct the physical log of the first data record based on the update offset parameter of the data page file and the update value of the update field when the update field includes the identification information of the data record; The information processing module is used to construct the physical log of the first data record based on the update offset parameter of the data page file, the update value of the update field, and the primary key field of the data table when the update field does not include the identification information of the data record.

[0010] In the above scheme, The information processing module is used to write the physical log of the metadata and the physical log of the first data record into a disk log file; The information processing module is used to send the physical log of the metadata and the physical log of the first data record to the destination database system through the disk log file.

[0011] In the above scheme, The information processing module is used to parse and obtain the value of each field of the inserted data record when the physical log of the first data record is a data record insertion operation type. The information processing module is used to organize the values ​​of each field of the inserted data record into a second data record as the physical log parsing result according to the data format of the destination database system, so as to update the data of the destination database system through the second data record.

[0012] In the above scheme, The information processing module is used to parse and obtain the identifier information of the deletion field of the deleted data record when the physical log of the first data record is a data record deletion operation type; The information processing module is used to organize the identification information of the deleted field into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

[0013] In the above scheme, The information processing module is used to parse and obtain the identifier information of the updated data record and the updated value of the updated field when the physical log of the first data record is a data record update operation type. The information processing module is used to organize the identification information and update values ​​of the updated fields of the updated data record into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

[0014] In the above scheme, The information processing module is used to determine the cloud server network corresponding to the destination database system based on the identification information of the data in the destination database system. The update results of the data in the target database system are stored in the cloud server network; When data loss occurs in the destination database system, the update result is sent to the destination database system through the cloud server network.

[0015] This application also provides an electronic device, the electronic device comprising: Memory, used to store executable instructions; A processor, used to implement a preceding data synchronization method when running executable instructions stored in the memory.

[0016] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement a preceding data synchronization method.

[0017] The embodiments of this application have the following beneficial effects: This application achieves automated data synchronization between different databases by: acquiring metadata and a first data record requiring data synchronization from the source database system; constructing a physical log for the metadata based on the metadata; constructing a physical log for the first data record based on the first data record; sending the physical logs of the metadata and the first data record to the destination database system; parsing the physical logs of the metadata and the first data record in the destination database system to obtain a physical log parsing result; and performing data synchronization in the destination database system based on the physical log parsing result to update the data in the destination database system. This enables automated data synchronization between different databases, ensuring data consistency, improving user efficiency, reducing data loss caused by tedious manual data synchronization, and guaranteeing data security and accuracy. Furthermore, using physical logs for data synchronization does not require hardware overhead from the database system, reducing user costs. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the usage environment of the data synchronization method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the composition structure of the data synchronization device provided in the embodiments of this application; Figure 3 An optional flowchart illustrating the data synchronization method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the physical log structure of the first data record in the data record insertion operation in an embodiment of the present invention; Figure 5 This is a schematic diagram of the physical log structure of the first data record in the data record deletion operation in an embodiment of the present invention; Figure 6 This is a schematic diagram of the physical log construction of the first data record in the data record update operation in an embodiment of the present invention; Figure 7 This is a schematic diagram of the physical log construction of the first data record in the data record update operation in an embodiment of the present invention; Figure 8 An optional flowchart illustrating the data synchronization method provided in the embodiments of this application; Figure 9 A schematic diagram of the front-end display of the data synchronization method provided in this application; Figure 10 This is an optional flowchart illustrating the data synchronization method provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0022] 1) Terminals, including but not limited to: ordinary terminals and dedicated terminals, wherein the ordinary terminals maintain a long connection and / or a short connection with the transmission channel, and the dedicated terminals maintain a long connection with the transmission channel.

[0023] 2) Client: The carrier that implements specific functions in the terminal. For example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, such as performing report creation or report display functions.

[0024] 3) OLAP database system: Online Analytical Processing (OLAP) system.

[0025] 4) OLTP database system: Online Transaction Processing system.

[0026] 5) HTAP database system: Online Transactional and Analytical Processing (Hybrid Transactional and Analytical Processing) system.

[0027] 6) REDO log: In a database system, a log used to ensure the durability of transactions.

[0028] 7) Physical log: In a database system, the REDO log generally records the physical modifications to data on disk, hence the name physical log.

[0029] 8) Logical Log: In a database system, a log that records logical modifications to data is typically used for synchronizing data between database systems.

[0030] 9) Data Table: In a database system, a data table is an object used to store data. It is a structured collection of data that contains one or more fields.

[0031] 10) Fields: In a database system, the columns of a data table are called fields, and each field contains information about that column.

[0032] 11) Record: In a database system, information about all fields of a row of data in a table. Before introducing the data synchronization method provided in this application, the data synchronization process in related technologies will first be described. HTAP database systems are database systems that support simultaneous processing of Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) tasks. Therefore, HTAP database systems support both online update tasks and online analytical query requests. It should be noted that HTAP databases avoid large-scale data interaction between online and offline data, and their innovative computing and storage framework can also support elastic scaling, better addressing the challenges brought by high concurrency. To implement data synchronization, the source database system needs to generate additional logical logs of the data and synchronize them to the destination database system. For example, in a MySQL database system, to achieve data synchronization between database systems, an additional logical log called the binlog log needs to be generated. Simultaneously, the MySQL database system itself also needs physical logs called REDO logs to ensure the transaction durability of the MySQL database system. The additional generation of logical logs for synchronization incurs significant overhead on the source database system, including CPU overhead for generating logical logs and I / O overhead for writing logical logs to disk. It also occupies a large amount of storage space, increasing the user's hardware costs and hindering the promotion and use of online transactions and analytical processing scenarios.

[0033] Figure 1 This is a schematic diagram illustrating a use case of the data synchronization method provided in the embodiments of this application. See also: Figure 1 In the data synchronization method application provided in this application embodiment, the terminal includes terminal 10-1 and terminal 10-2. The terminal (including terminal 10-1 and terminal 10-2) is equipped with a client software capable of displaying corresponding financial transaction information, such as a client or plugin for conducting financial activities using virtual or physical resources or for making payments through virtual resources (such as virtual trading currency).

[0034] Of course, the data synchronization device provided by this invention can be applied to environments where financial activities are conducted using virtual or physical resources, or where payments are made using physical financial resources (including but not limited to environments where various types of physical financial resources change, electronic payment shopping environments, and anti-fraud environments during e-commerce shopping) or where information is exchanged using social software. In financial activities conducted using various types of physical financial resources or where payments are made using virtual resources, financial information from different data sources is usually processed, and finally the updated data from the destination database system is presented on the user interface (UI) of the business device.

[0035] In some embodiments of the present invention, the data synchronization process can be performed by a computing platform. The computing platform can be a platform located on a trusted third-party device, or it can be one server among multiple servers, or a platform distributed across multiple servers. The computing platform can interact with each server. Figure 1 Multiple business server providers (which may hold different business data) can be servers of the same data category, such as all being banking servers or all being shopping platform servers. Multiple servers can also be servers of different categories; for example, a server could be a shopping platform server, a server could be the instant messaging client server corresponding to the shopping platform, or a server could be a bank's head office server or a branch server. In data synchronization scenarios, the business data provided by these servers is usually of the same type. For example, if a server is both a shopping platform server and a lending platform server, and the shopping platform is linked to a payment bank card number while the lending platform is linked to a withdrawal and repayment bank card number, the business data provided by both parties for data synchronization could be bank card numbers and transfer or lending information. If both the shopping platform server and the lending platform server have registered users' bank card information and ID card information, the business data provided by both parties for data synchronization could also be bank card information and ID card information. In other business scenarios, the business data can also include other data, which will not be listed here.

[0036] As an example, server 200 is used to deploy a data synchronization device to implement the data synchronization method provided in this application. When executing the data synchronization method, in the source database system, metadata requiring data synchronization and a first data record requiring data synchronization are obtained; a physical log of the metadata is constructed based on the metadata; a physical log of the first data record is constructed based on the first data record; the physical log of the metadata and the physical log of the first data record are sent to the destination database system; in the destination database system, the physical log of the metadata and the physical log of the first data record are parsed to obtain a physical log parsing result; according to the physical log parsing result, data synchronization is performed in the destination database system to update the data in the destination database system, ultimately achieving data updates in the destination database system without incurring hardware overhead in the source database system.

[0037] The structure of the data synchronization device according to the embodiments of this application will be described in detail below. The data synchronization device can be implemented in various forms, such as a dedicated terminal with data synchronization function, or a server with data synchronization function, such as the one mentioned above. Figure 1 Server 200 in the middle. Figure 2This is a schematic diagram of the composition structure of the data synchronization device provided in the embodiments of this application. It can be understood that... Figure 2 The diagram shows only an exemplary structure of the data synchronization device, not the entire structure; implementation is possible as needed. Figure 2 The structure shown may be part or all of the structure.

[0038] The data synchronization device provided in this application embodiment includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the data synchronization device 20 are coupled together via a bus system 205. It can be understood that the bus system 205 is used to implement communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 205.

[0039] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0040] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. The memory 202 in this embodiment is capable of storing data to support the operation of the terminal (e.g., 10-1). Examples of this data include any computer programs used to operate on the terminal (e.g., 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0041] In some embodiments, the data synchronization device provided in this application can be implemented using a combination of hardware and software. For example, the data synchronization device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data synchronization method provided in this application. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0042] As an example of the data synchronization device provided in this application embodiment being implemented using a combination of hardware and software, the data synchronization device provided in this application embodiment can be directly embodied as a combination of software modules executed by processor 201. The software modules can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software modules in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus 205) to complete the data synchronization method provided in this application embodiment.

[0043] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0044] As an example of the hardware implementation of the data synchronization device provided in this application embodiment, the device provided in this application embodiment can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the data synchronization method provided in this application embodiment.

[0045] The memory 202 in this embodiment is used to store various types of data to support the operation of the data synchronization device 20. Examples of such data include any executable instructions for operation on the data synchronization device 20, such as executable instructions, and a program implementing the data synchronization method of this embodiment may be included in the executable instructions.

[0046] In other embodiments, the data synchronization device provided in this application can be implemented in software. Figure 2A data synchronization device stored in memory 202 is shown. This device can be software in the form of programs and plug-ins, and includes a series of modules. As an example of a program stored in memory 202, it may include the data synchronization device. The data synchronization device includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the data synchronization device are read into RAM and executed by processor 201, the data synchronization method provided in this application embodiment will be implemented. The functions of each software module in the data synchronization device are further described below, specifically including: The information transmission module 2081 is used to obtain the metadata that needs to be processed for data synchronization and the first data record that needs to be synchronized from the source database system. Information processing module 2082 is used to construct physical logs of metadata based on the metadata; The information processing module 2082 is used to construct a physical log of the first data record based on the first data record; The information processing module 2082 is used to send the physical log of the metadata and the physical log of the first data record to the destination database system; The information processing module 2082 is used to parse the physical log of the metadata and the physical log of the first data record in the destination database system to obtain the physical log parsing result. The information processing module 2082 is used to perform data synchronization in the destination database system based on the physical log parsing results, so as to update the data in the destination database system.

[0047] according to Figure 2 The illustrated electronic device, in one aspect of this application, also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various alternative implementations of the data synchronization method provided in this application, including different embodiments and combinations thereof.

[0048] Combination Figure 2 The data synchronization device shown illustrates the data synchronization method provided in the embodiments of this application. See also: Figure 3 , Figure 3 This is an optional flowchart illustrating the data synchronization method provided in the embodiments of this application. It can be understood that... Figure 3The steps shown can be performed by various electronic devices that run data synchronization devices, such as dedicated terminals with data synchronization devices, servers, or server clusters. The dedicated terminal with a data synchronization device can be a preceding... Figure 2 The illustrated embodiment is an electronic device with a data synchronization mechanism. The following section addresses... Figure 3 The steps shown are explained.

[0049] Step 301: The data synchronization device obtains the metadata that needs to be processed for data synchronization and the first data record that needs to be synchronized in the source database system.

[0050] Step 302: The data synchronization device constructs a physical log of metadata based on the metadata.

[0051] When constructing the physical log of metadata, the metadata of the data tables that need to be synchronized, including the metadata of each field, can be written into the physical log of metadata. Metadata in the source database system is also called intermediate data or relay data. It refers to data about data, which is information about the organization of data, data domains and their relationships. It mainly describes the information of data properties and is used to support functions such as sharding information indicating data sharding, storage location, historical data, resource lookup, file records, etc.

[0052] Step 303: The data synchronization device constructs a physical log of the first data record based on the first data record.

[0053] In some embodiments of the present invention, since the modification operation types of the first data record include modification, deletion, and update, when constructing the physical log of the first data record, if it is determined that the first data record needs data synchronization processing based on the field information in the metadata, the modification operation type of the first data record needs to be determined first. Then, according to the modification operation type of the first data record, a construction strategy matching the modification operation type is executed to obtain the physical log of the first data record. The construction strategies matching the modification operation types of the first data record will be described one by one below.

[0054] In some embodiments of the present invention, reference is made to Figure 4 , Figure 4This is a schematic diagram illustrating the construction of the physical log for the first data record in a data record insertion operation according to an embodiment of the present invention. When the modification operation type of the first data record is a data record insertion operation, the insertion offset parameter of the data page file and the value of each field of the first data record are obtained. Based on the insertion offset parameter of the data page file and the value of each field of the first data record, the physical log of the first data record is constructed. Since the data record insertion operation writes a new data record to the database, the corresponding physical log consists of two parts: 1) the first part is the offset parameter of the data page file corresponding to the data record when it is written to the database; 2) the second part is the value of each field of the data record. Figure 4 As shown, when the inserted data record has 5 fields, no additional information needs to be added to the physical log of the first data record. The physical log of the first data record at this time consists of: offset parameter, data field 1 (used to represent the value of field 1), data field 2 (used to represent the value of field 2), data field 3 (used to represent the value of field 3), data field 4 (used to represent the value of field 4), and data field 5 (used to represent the value of field 5).

[0055] In some embodiments of the present invention, reference is made to Figure 5 , Figure 5 This is a schematic diagram illustrating the construction of the physical log of the first data record in a data record deletion operation according to an embodiment of the present invention. When the modification operation type of the first data record is a data record deletion operation, the deletion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field are obtained; based on the insertion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field, the physical log of the first data record is constructed. Figure 5 As shown, the data record deletion operation removes the corresponding data record from the database, which can be marked for deletion. During deletion, a "mark for deletion" field is added to the offset of the data page file corresponding to the data record when it was written to the database to mark that the record has been deleted. Therefore, the original physical log will only contain the offset parameter of the data record to be deleted in the data page file, and the modification of the "mark for deletion" field. Because the data page files of the destination database and the source database are inconsistent, a unique identifier for the data record to be deleted (which can be any field of the primary key of the data table) needs to be added to the physical log of the data record deletion operation. This allows the destination database to find the data record to be deleted using this unique identifier and execute the deletion operation on the destination database, ensuring the accuracy of the data in the destination database. Figure 5As shown, the primary key of the deleted data record has two fields: primary key field 1 and primary key field 2. At this time, the physical log of the first data record includes: deletion offset parameter, identification information of the deleted field (marked as deleted), primary key field 1 and primary key field 2.

[0056] In some embodiments of the present invention, reference is made to Figure 6 , Figure 6 This is a schematic diagram illustrating the construction of the physical log for the first data record in a data record update operation according to an embodiment of the present invention. When the modification operation type of the first data record is a data record update operation, the update offset parameter of the data page file and the update value of the update field are obtained. When the update field includes the identification information of the data record, the physical log of the first data record is constructed based on the update offset parameter of the data page file and the update value of the update field. Since the data record update operation updates only some fields of the data record, the corresponding physical log consists of two parts: 1) the offset parameter of the data page file corresponding to the data record when it is written to the database; 2) the new value of the updated field in the data record. If the updated field covers the unique identifier of the data record, i.e., all fields of the primary key of the data table, then no additional information needs to be added to the physical log. Figure 6 As shown, when only three fields of a data record are updated (update field 1, update field 2, and update field 3), if these three fields cover all fields of the primary key of the data record, then the physical log of the first data record can include: update offset parameter, update field 1, update field 2, and update field 3.

[0057] In some embodiments of the present invention, reference is made to Figure 7 , Figure 7 This is a schematic diagram illustrating the construction of the physical log of the first data record in a data record update operation according to an embodiment of the present invention. When the update field does not include the identification information of the data record, the physical log of the first data record is constructed based on the update offset parameter of the data page file, the update value of the update field, and the primary key field of the data table. Figure 7 As shown, since the updated fields cannot cover the unique identifier of the data record, it is necessary to supplement the primary key field of the data table in the physical log. That is, the primary key field of the data table is written to the physical log sequentially, so that the physical log of the first data record contains the complete unique identifier of the data record. Figure 7 As shown, when only three fields of a data record are updated, if these three fields do not cover the primary key fields of the data record, the physical log of the first data record may include: update offset parameter, update field 1, update field 2, update field 3, primary key field 1, and primary key field 2.

[0058] Step 304: The data synchronization device sends the physical log of the metadata and the physical log of the first data record to the destination database system.

[0059] In some embodiments of the present invention, the physical logs of the metadata and the physical logs of the first data record can be written to a disk log file; the physical logs of the metadata and the first data record are then sent to the destination database system via the disk log file. Since the transmission of the disk file is unaffected by power outages, the secure transmission of the physical logs of the metadata and the first data record can be effectively guaranteed. Furthermore, the physical logs of the metadata and the first data record can be transmitted to a cloud server network using the disk file, thereby enabling backup of the physical logs of the metadata and the first data record using the cloud server network and ensuring the data security of the destination database system.

[0060] It should be noted that, in some embodiments of the present invention, when performing steps 301-304, a physical log of metadata can be constructed first and sent to the destination database system. Then, a physical log of the first data record can be constructed and sent to the destination database system. This application does not impose specific limitations on this.

[0061] Step 305: The data synchronization device parses the physical log of the metadata and the physical log of the first data record in the destination database system to obtain the physical log parsing result.

[0062] When parsing the physical log of data records, see [link / reference]. Figure 8 , Figure 8 This is an optional flowchart illustrating the data synchronization method provided in the embodiments of this application. It can be understood that... Figure 8 The steps shown can be performed by various electronic devices that run data synchronization devices, such as dedicated terminals with data synchronization devices, servers, or server clusters. The dedicated terminal with a data synchronization device can be a preceding... Figure 2 The illustrated embodiment is an electronic device with a data synchronization mechanism. The following section addresses... Figure 8 The steps shown are explained.

[0063] Step 801: Determine the data record type of the physical log of the first data record.

[0064] Step 802: When the physical log of the first data record is a data record insertion operation type, parse and obtain the value of each field of the inserted data record.

[0065] Step 803: Based on the data format of the destination database system, organize the values ​​of each field of the inserted data record into a second data record as the result of the physical log parsing.

[0066] Step 804: When the physical log of the first data record is a data record deletion operation type, parse and obtain the identifier information of the deletion field of the deleted data record.

[0067] Step 805: Organize the identification information of the deleted field into a second data record as the result of the physical log parsing.

[0068] Step 806: When the physical log of the first data record is a data record update operation type, parse and obtain the identifier information of the updated data record and the update value of the update field.

[0069] Step 807: Organize the identifier information and update values ​​of the updated fields of the updated data record into a second data record as the result of the physical log parsing.

[0070] When passing Figure 8 After obtaining the physical log parsing results as shown in the steps, you can continue to execute step 306 to complete the data synchronization.

[0071] Step 306: The data synchronization device performs data synchronization in the destination database system based on the physical log parsing results, so as to update the data in the destination database system.

[0072] Specifically, when the physical log of the first data record is a data record insertion operation type, since the physical log of the data record insertion operation contains the value of each field of the data record, the value of each field of the inserted data record can be parsed out; according to the data format of the destination, the values ​​of all fields are organized into a new data record; finally, the new data record is inserted into the database system of the destination to complete the data synchronization.

[0073] When the physical log of the first data record is a data record deletion operation type, since the physical log of the data record deletion operation contains the unique identifier of the data record to be deleted, the unique identifier of the data record to be deleted can be parsed out (for example, it can be the value of all fields in the primary key of the data table); through the unique identifier of the data record, the data record can be retrieved in the database system on the destination side; finally, the data record is deleted in the database on the destination side to complete the data synchronization.

[0074] When the physical log of the first data record is a data record update operation type, since the physical log of the data record update operation contains the unique identifier of the updated data record, the unique identifier of the data record to be updated can be parsed out (for example, it can be the value of all fields in the primary key of the data table). Then, the data record can be retrieved in the database system at the destination end by using the unique identifier of the data record. After that, the value of the field to be modified contained in the physical log is used to perform the update operation on the field to be updated in the database at the destination end.

[0075] Once the destination database system completes data synchronization, the corresponding cloud server network can be determined based on the data identification information of the destination database system. The update results of the data in the destination database system are then stored in the cloud server network. When data loss occurs in the destination database system, the update results are sent to the destination database system through the cloud server network. This enhances the data security of the destination database system and prevents users from suffering losses due to data loss in the destination database system.

[0076] The destination database system can be a cloud server or a cloud server cluster. This application embodiment can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It can also be understood as a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. The backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites; therefore, cloud technology needs cloud computing as its support.

[0077] It's important to note that cloud computing is a computing model that distributes computing tasks across a resource pool comprised of numerous computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, and can be used on demand, expanded at any time, and paid for based on usage. As the foundational providers of cloud computing capabilities, they establish cloud resource pool platforms, often referred to as cloud platforms or Infrastructure as a Service (IaaS). These platforms deploy various types of virtual resources within the resource pool for external customers to choose from. The cloud resource pool primarily includes: computing devices (which can be virtualized machines containing operating systems), storage devices, and network devices.

[0078] Combined with the preceding sequence Figure 1As shown, the data synchronization method provided in this embodiment of the invention can be implemented through corresponding cloud devices. For example, terminals (including terminals 10-1 and 10-2) connect to a server 200 located in the cloud via a network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. It is worth noting that the server 200 can be a physical device or a virtualized device.

[0079] The data synchronization method provided in this application will be further explained below in different implementation scenarios. See also... Figure 9 , Figure 9 This is a schematic diagram illustrating the front-end display of the data synchronization method provided in this application. The terminal (e.g., Figure 1 Terminals 10-1 and 10-2 are equipped with clients capable of displaying software for financial payments, such as clients or plugins for financial activities using virtual or physical resources, or for lending through virtual resources. Users can obtain loan services from financial institutions through the corresponding clients. This can be applied to cross-industry collaboration scenarios in financial risk control, for example, where the servers are respectively the source database system (head office system) of Bank A and the destination database system (branch system) of Bank A.

[0080] Bank A's services include OLTP and OLAP operations. Utilizing the HTAP scenario, OLTP operations run on the primary datanode, while OLAP operations run on the backup datanode. Data synchronization between the two is achieved through streaming replication, enabling the insertion, deletion, and updating of user information.

[0081] Meanwhile, Bank A's destination database system (branch system) has a set of users with deposits exceeding 100,000 yuan and their corresponding credit scores, as shown in Table 1.

[0082] Table 1

[0083] At this point, A's source database system has obtained the changes in depositors' information and needs to synchronize it with bank A's destination database system (see Table 2).

[0084] Table 2

[0085] At this point, user ID 37003's information needs to be deleted, user ID 37001's credit score needs to be updated, and user ID 37000's ID needs to be inserted. In the source database system, the metadata requiring data synchronization and the first data record requiring synchronization are retrieved, including data record insertion, deletion, and update operations. (See also...) Figure 10 , Figure 10 An optional flowchart illustrating the data synchronization method provided in this application embodiment specifically includes the following steps: Step 1001: In the source database system of Bank A, obtain the metadata that needs to be synchronized, as well as the data insertion operation records, data deletion operation records, and data update operation records that need to be synchronized.

[0086] Step 1002: Construct the physical log of metadata, the physical log of insert operation records, the physical log of data deletion operation records, and the physical log of data update operation records.

[0087] Step 1003: Send the physical logs of metadata, insert operation records, data deletion operation records, and data update operation records to the destination database system of Bank A.

[0088] Step 1004: Bank A's target database system parses the physical logs of metadata, insert operation records, data deletion operation records, and data update operation records to obtain the parsing results.

[0089] Step 1005: Bank A's destination database system, based on the parsing results, deletes the information of user ID ID37003, updates the credit score of ID37001, and inserts the ID ID of ID37000.

[0090] Beneficial technical effects: This application achieves automated data synchronization between different databases by: acquiring metadata and a first data record requiring data synchronization from the source database system; constructing a physical log for the metadata based on the metadata; constructing a physical log for the first data record based on the first data record; sending the physical logs of the metadata and the first data record to the destination database system; parsing the physical logs of the metadata and the first data record in the destination database system to obtain a physical log parsing result; and performing data synchronization in the destination database system based on the physical log parsing result to update the data in the destination database system. This enables automated data synchronization between different databases, ensuring data consistency, improving user efficiency, reducing data loss caused by tedious manual data synchronization, and guaranteeing data security and accuracy. Furthermore, using physical logs for data synchronization does not require hardware overhead from the database system, reducing user costs.

[0091] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of data synchronization, the method comprising: The method includes: In the source database system, obtain the metadata that needs to be processed for data synchronization and the first data record that needs to be synchronized. The physical log of metadata is constructed based on the aforementioned metadata; Based on the first data record, construct a physical log of the first data record; when the modification operation type of the first data record is a data record deletion operation, obtain the deletion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field; based on the deletion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field, construct a physical log of the first data record. If the data page file of the destination database is inconsistent with the data page file of the source database, a unique identifier of the data record to be deleted is added to the physical log of the data record deletion operation. When the modification operation type of the first data record is a data record update operation, the update offset parameter of the data page file and the update value of the update field are obtained; when the update field does not include the identification information of the data record, the physical log of the first data record is constructed according to the update offset parameter of the data page file, the update value of the update field and the primary key field of the data table, so that the physical log contains the unique identifier of the complete data record. Send the physical log of the metadata and the physical log of the first data record to the destination database system; In the destination database system, the physical logs of the metadata and the physical logs of the first data record are parsed to obtain the physical log parsing results; Based on the physical log parsing results, data synchronization is performed in the destination database system to update the data in the destination database system.

2. The method of claim 1, wherein, The construction of the physical log of the first data record based on the first data record includes: When it is determined that the first data record needs to be synchronized based on the field information in the metadata, the modification operation type of the first data record is determined. Based on the modification operation type of the first data record, a construction strategy matching the modification operation type is executed to obtain the physical log of the first data record.

3. The method of claim 2, wherein, The step of executing a construction strategy matching the modification operation type of the first data record to obtain the physical log of the first data record includes: When the modification operation type of the first data record is a data record insertion operation, obtain the insertion offset parameter of the data page file and the value of each field of the first data record; Based on the insertion offset parameter of the data page file and the value of each field of the first data record, construct the physical log of the first data record.

4. The method of claim 2, wherein, The method further includes: When the update field includes the identification information of the data record, the physical log of the first data record is constructed based on the update offset parameter of the data page file and the update value of the update field.

5. The method of claim 2, wherein, Sending the physical log of the metadata and the physical log of the first data record to the destination database system includes: Write the physical log of the metadata and the physical log of the first data record to a disk log file; The physical log of the metadata and the physical log of the first data record are sent to the destination database system through the disk log file.

6. The method of claim 2, wherein, In the destination database system, the physical logs of the metadata and the first data record are parsed to obtain physical log parsing results, including: When the physical log of the first data record is a data record insertion operation type, parse and obtain the value of each field of the inserted data record; Based on the data format of the destination database system, the values ​​of each field of the inserted data record are organized into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

7. The method according to claim 2, characterized in that, In the destination database system, the physical logs of the metadata and the first data record are parsed to obtain physical log parsing results, including: When the physical log of the first data record is a data record deletion operation type, the identifier information of the deletion field of the deleted data record is parsed and obtained; The identification information of the deleted field is organized into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

8. The method according to claim 2, characterized in that, In the destination database system, the physical logs of the metadata and the first data record are parsed to obtain physical log parsing results, including: When the physical log of the first data record is a data record update operation type, the identifier information of the updated data record and the updated value of the updated field are parsed and obtained. The identifier information and update values ​​of the updated fields of the updated data record are organized into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

9. The method of claim 1, wherein, The method further includes: Based on the identification information of the data in the destination database system, determine the cloud server network corresponding to the destination database system; The update results of the data in the target database system are stored in the cloud server network; When data loss occurs in the destination database system, the update result is sent to the destination database system through the cloud server network.

10. A data synchronization apparatus applied in a data synchronization system, the data synchronization system comprising: The request server and processing server are characterized in that the apparatus comprises: The information transmission module is used to obtain the metadata that needs to be processed for data synchronization and the first data record that needs to be synchronized from the source database system. An information processing module is used to construct physical logs of metadata based on the aforementioned metadata; The information processing module is configured to: construct a physical log of the first data record based on the first data record; when the modification operation type of the first data record is a data record deletion operation, obtain the deletion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field; construct the physical log of the first data record based on the deletion offset parameter of the data page file, the modification information of the marked deletion field, and the identification information of the deleted field; when the data page file of the destination database is inconsistent with the data page file of the source database, add a unique identifier of the data record to be deleted to the physical log of the data record deletion operation; when the modification operation type of the first data record is a data record update operation, obtain the update offset parameter of the data page file and the update value of the update field; when the update field does not include the identification information of the data record, construct the physical log of the first data record based on the update offset parameter of the data page file, the update value of the update field, and the primary key field of the data table, so that the physical log contains the unique identifier of the complete data record; The information processing module is also used to send the physical log of the metadata and the physical log of the first data record to the destination database system; The information processing module is also used to parse the physical log of the metadata and the physical log of the first data record in the destination database system to obtain the physical log parsing result; The information processing module is also used to perform data synchronization in the destination database system based on the physical log parsing results, so as to update the data in the destination database system.

11. The apparatus according to claim 10, characterized in that, The information processing module is also used to determine the modification operation type of the first data record when it is determined that the first data record needs to be synchronized based on the field information in the metadata. Based on the modification operation type of the first data record, a construction strategy matching the modification operation type is executed to obtain the physical log of the first data record.

12. The apparatus according to claim 11, characterized in that, The information processing module is further configured to, when the modification operation type of the first data record is a data record insertion operation, obtain the insertion offset parameter of the data page file and the value of each field of the first data record; Based on the insertion offset parameter of the data page file and the value of each field of the first data record, construct the physical log of the first data record.

13. The apparatus according to claim 11, characterized in that, The information processing module is further configured to construct the physical log of the first data record based on the update offset parameter of the data page file and the update value of the update field when the update field includes the identification information of the data record.

14. The apparatus according to claim 11, characterized in that, The information processing module is further configured to write the physical log of the metadata and the physical log of the first data record into a disk log file; The physical log of the metadata and the physical log of the first data record are sent to the destination database system through the disk log file.

15. The apparatus according to claim 11, characterized in that, The information processing module is also used to parse and obtain the value of each field of the inserted data record when the physical log of the first data record is a data record insertion operation type; Based on the data format of the destination database system, the values ​​of each field of the inserted data record are organized into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

16. The apparatus according to claim 11, characterized in that, The information processing module is also used to parse and obtain the identifier information of the deletion field of the deleted data record when the physical log of the first data record is a data record deletion operation type; The identification information of the deleted field is organized into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

17. The apparatus according to claim 11, characterized in that, The information processing module is also used to parse and obtain the identifier information of the updated data record and the updated value of the updated field when the physical log of the first data record is a data record update operation type. The identifier information and update values ​​of the updated fields of the updated data record are organized into a second data record as the result of the physical log parsing, so as to update the data of the destination database system through the second data record.

18. The apparatus according to claim 10, characterized in that, The information processing module is also used to determine the cloud server network corresponding to the destination database system based on the identification information of the data in the destination database system. The update results of the data in the target database system are stored in the cloud server network; When data loss occurs in the destination database system, the update result is sent to the destination database system through the cloud server network.

19. An electronic device, comprising: The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data synchronization method according to any one of claims 1 to 9.

20. A computer program product comprising computer programs or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the data synchronization method according to any one of claims 1 to 9.

21. A computer-readable storage medium storing executable instructions, wherein the instructions, when executed, cause a processor to: When the executable instructions are executed by the processor, they implement the data synchronization method according to any one of claims 1 to 9.

Images