Data processing method and apparatus, electronic device, system, and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device, electronic equipment, system and storage medium, the data processing method comprises: obtaining an operation log of a table in a database from a source server, parsing the operation log to obtain source operation information, mapping the source operation information into target operation information according to configuration information of the table, and writing the target operation information into a target server, so that the target server performs real-time data synchronization with the source server according to the target operation information. In the embodiments of the present application, data synchronization can be performed between the target server and the source server based on real-time operation logs, so that real-time synchronization of data can be realized, and the requirements of a system requiring real-time data query are met.

Description

Technical Field

[0001] The present invention relates to data synchronization technology, and in particular to a data processing method, device, electronic device, system and storage medium. Background Art

[0002] Traditional data synchronization methods usually synchronize the previous day's data during a time period with few user requests (such as early morning). In the process of implementing the present invention, the inventors found that the data synchronized by this data synchronization method is not real-time and cannot meet the requirements of systems that have real-time data query needs. Summary of the Invention

[0003] The embodiments of the present invention provide a data processing method, device, electronic device, system and storage medium, which can realize real-time synchronization of data and meet the requirements of systems with real-time query needs for data.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0005] Obtaining operation logs of tables in the database from the source server, and parsing the operation logs to obtain source operation information;

[0006] Mapping the source operation information into target operation information according to the configuration information of the table;

[0007] The target operation information is written into the target server, so that the target server performs real-time data synchronization with the source server according to the target operation information.

[0008] In a second aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0009] An acquisition module is used to obtain the operation log of the table in the database from the source server and parse the operation log to obtain source operation information;

[0010] a mapping module, configured to map the source operation information into target operation information according to the configuration information of the table;

[0011] The synchronization module is used to write the target operation information into the target server, so that the target server performs real-time data synchronization with the source server according to the target operation information.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method as described in any one of the embodiments of the present invention is implemented.

[0013] In a fourth aspect, an embodiment of the present invention further provides a data processing system, comprising a source server, a target server, and an electronic device for executing any of the data processing methods described in the embodiments of the present invention.

[0014] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the data processing method as described in any one of the embodiments of the present invention.

[0015] In an embodiment of the present invention, an operation log of a table in a database can be obtained from a source server, the operation log can be parsed to obtain source operation information, the source operation information can be mapped into target operation information according to the configuration information of the table, and the target operation information can be written to the target server, so that the target server can synchronize data with the source server in real time according to the target operation information; that is, in an embodiment of the present invention, data synchronization can be performed between the target server and the source server based on the real-time operation log, thereby achieving real-time synchronization of data, meeting the requirements of a system with real-time query needs for data; in addition, functions such as log pulling and parsing, information mapping, etc. are integrated on a single device, saving device resources and simplifying system configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention.

[0018] Figure 2 This is another flowchart of the data processing method provided by an embodiment of the present invention.

[0019] Figure 3 This is another flowchart of the data processing method provided by an embodiment of the present invention.

[0020] Figure 4 It is a structural diagram of a data processing device provided by an embodiment of the present invention.

[0021] Figure 5 It is a structural diagram of a data processing system provided by an embodiment of the present invention.

[0022] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0024] Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present invention. This method can be performed by a data processing device provided in an embodiment of the present invention, which can be implemented using software and / or hardware. In one specific embodiment, the device can be integrated into a server. The following embodiments will be described using the device integrated into a server as an example.

[0025] In a specific implementation, the server can be a synchronization server for synchronizing data or data changes in a source server to a target server, wherein the source server can be a server corresponding to the open source relational database MySQL (i.e., a MySQL server), and the target server can be a server corresponding to the analysis engine Elastic Search for distributed full-text retrieval (i.e., an Elastic Search server, or ES server for short), a server corresponding to the remote dictionary service Redis database (i.e., a Redis server), a server corresponding to the open source database HBase (i.e., an HBase server), etc., without specific limitation here. Specifically, in an embodiment of the present invention, a Canal framework can be built in the synchronization server, and Canal can provide incremental data subscription and consumption based on incremental log parsing of the MySQL database. Canal can be understood as a tool for synchronizing incremental data.

[0026] Please refer to Figure 1 The data processing method provided by the embodiment of the present invention may specifically include the following steps:

[0027] Step 101: Obtain an operation log of a table in a database from a source server, and parse the operation log to obtain source operation information.

[0028] Specifically, the operation log can be a binary log, which can also be called an update log. The binary log is a very important log in MySQL, mainly used to record changes to tables in the database. You can enable the binary log by adding the log-bin option in the configuration file. In specific implementation, you can subscribe to the tables in the database of the source server and monitor table operations. When there is an update to a table in the database of the source server, the source server will write the update to the binary log file according to the set binary log format, and then create a log dump thread to notify its slave node of the update. When the update is monitored, the synchronization server can obtain the table's operation log from the source server. The specific method for obtaining the operation log is as follows:

[0029] (1) The source server is used as the master node, and the slave node simulates the source server to send a log acquisition request to the source server.

[0030] For example, if the source server is a MySQL server, you can use the source server as a MySQL master node (primary node), simulate the interaction protocol of the MySQL slave, disguise yourself as a MySQL slave node (slave node), and send a dump request to the source server to request the table operation log.

[0031] (2) Receive the operation log returned by the source server.

[0032] Specifically, after receiving the dump request, MySQL master can obtain the storage location of the binary log file, read the binary log from the binary log file according to the storage location, and push the read binary log to the synchronization server, which receives the binary log.

[0033] The binary log received by the synchronization server is originally in byte stream format. The synchronization server can parse the binary log from byte stream format into array entry format to obtain source operation information. The source operation information may include the column name of the database table, binary log file name, data change type, change data, etc.

[0034] Step 102: Map the source operation information into target operation information according to the configuration information of the table.

[0035] Specifically, source operation information can include two types: Data Manipulation Language (DML) type and Data Definition Language (DDL) type. DML type information is used to change data in a table, that is, the operation object of DML type information is records. The main operation statements included in DML type information can be as follows:

[0036] Insert statement, such as inserting a record into a table.

[0037] Delete statement: For example, you can delete one or more records in a table, or you can delete all records in a table. However, its operation object is still the record.

[0038] Update statement: used to modify the contents of records that already exist in a table.

[0039] DDL type information is used to change the structure of a table. That is, the operation object of DML type information is the table. The main operation statements included in DDL type information can be as follows:

[0040] Create statement: For example, used to create a table.

[0041] Drop statement: For example, you can delete tables, indexes, triggers, constraints, and table permissions.

[0042] Alter statement: For example, modify table definitions and properties.

[0043] In the specific implementation, the synchronization server can first determine the type of the source operation information. If the source operation information is of DML type, it means that the information involves changes to the data in the table and requires operation on the target server. The source operation information can be mapped to target operation information based on the configuration information of the table; if the source operation information is of DDL type, it means that the information does not involve changes to the data in the table, but only involves changes to the table structure. There is no need to operate the target server, and the source operation information can be printed. For example, the source operation can be sent to a designated device in the form of an email for log printing, so that the user can process the changes in the table structure according to the printed log.

[0044] When the source operation information is of DML type, the target server needs to be operated. Since the naming and format of the fields in the table in the source server's database may be different from the naming and format of the data in the target server, if the source operation information is directly written to the target server, it will be inconvenient to view and operate the data. Therefore, the source operation information needs to be mapped to the target operation information.

[0045] Specifically, before mapping the source operation information to the target operation information, the configuration information of the table may be obtained first, and the source operation information may be mapped to the target operation information according to the configuration information of the table. The configuration information of the table may be obtained in the following two ways:

[0046] The first method is to obtain table configuration information from the configuration server. For example, you can pre-set the configuration information for each table on the configuration server. Each table's configuration information can be the template fields of the table that need to be displayed on the target server. The template fields can include fields such as the table header and table data format. After obtaining the source operation information for a table, you can obtain the table's configuration information from the configuration server.

[0047] The second method is to obtain table configuration information from the business implementation logic. For example, you can pre-bind the configuration information of each table in the business implementation logic. Then, after obtaining the source operation information of a table, you can obtain the configuration information of the table from the business implementation logic.

[0048] The first configuration and acquisition method decouples the synchronization implementation logic from the business implementation logic, causing less intrusion into the business system. When a new table is added, only the configuration information of the new table needs to be set in the configuration server. This method requires less change to the business implementation logic and is more lightweight, flexible, and adaptable. In specific implementations, the first configuration and acquisition method can be preferred.

[0049] For example, the configuration information of the table obtained can be configuration information in Json format. After obtaining the configuration information of the table in Json format, the configuration information of the table can be converted from Json format to Java Bean format. The source operation information can be mapped into target operation information using the table configuration information after format conversion to facilitate data retrieval and operation.

[0050] Step 103: Write the target operation information into the target server, so that the target server performs real-time data synchronization with the source server according to the target operation information.

[0051] For example, after writing the target operation information to the target server, the target server can locally re-execute the events that occurred on the source server based on the target operation information. For example, it can add, delete, or modify records in the corresponding table based on the target operation information, thereby achieving real-time data synchronization between the target server and the source server. In this way, data from the source server is synchronized to the target server in real time, meeting the real-time query needs of the target server's downstream terminals.

[0052] In an embodiment of the present invention, before executing step 101 to automatically synchronize data, synchronization alignment can also be performed manually. For example, historical data generated before the current moment can be manually synchronized from the source server to the target server. After the synchronization is completed, the synchronization server of the embodiment of the present invention is enabled to automatically synchronize the changed data to improve the reliability of data synchronization.

[0053] In addition, there can be multiple synchronization servers in the embodiment of the present invention, and multiple synchronization servers can be uniformly registered (for example, according to the Internet Protocol IP address and port number port) on the ZooKeeper server to form a synchronization server cluster. When a synchronization server fails, other synchronization servers can be enabled by querying the ZooKeeper server to improve the reliability of the synchronization system.

[0054] The above technical solution can obtain the operation log of the table in the database from the source server, parse the operation log to obtain the source operation information, map the source operation information into target operation information according to the configuration information of the table, and write the target operation information to the target server, so that the target server can synchronize data with the source server in real time according to the target operation information; that is, in the embodiment of the present invention, data synchronization can be performed between the target server and the source server based on the real-time operation log, thereby realizing real-time synchronization of data, meeting the requirements of the system with real-time query needs for data; in addition, the functions of pulling and parsing the log, mapping the information, etc. are integrated into one device, which saves device resources and simplifies the system configuration.

[0055] The following takes the source server as a MySQL server and the target server as an ES server as an example to illustrate the data processing method provided by the embodiment of the present invention. Figure 2 As shown, the data processing method can still be used in the synchronization server, and the Canal framework can be built in the synchronization server. The data processing method may include the following steps:

[0056] Step 201: Use the MySQL server as the master node and simulate the slave node of the MySQL server to send a log acquisition request to the MySQL server.

[0057] Specifically, before executing step 201 to automatically synchronize data, synchronization alignment can also be performed manually. For example, historical data generated before the current moment can be manually synchronized from the MySQL server to the ES server. After the synchronization is completed, the synchronization server of an embodiment of the present invention is enabled to automatically synchronize the changed data to improve the reliability of data processing.

[0058] For example, the Canal server module and the Canal client module can be set up in the synchronization server, and the MySQL server can be used as the MySQL master node (primary node). The Canal server module simulates the interaction protocol of the MySQL slave, disguises itself as a MySQL slave node (slave node), and sends a dump request to the MySQL server to request the operation log of the table. The operation log can be a binary log, which is mainly used to record changes to the table in the MySQL database.

[0059] Step 202: Receive the operation log of the table in the database returned by the MySQL server.

[0060] Specifically, after receiving the dump request, MySQL master can obtain the storage location of the binary log file, read the binary log from the binary log file according to the storage location, and push the read binary log to the MySQL slave node (that is, the Canal server module), and the Canal server module receives the binary log.

[0061] Step 203: parse the operation log to obtain source operation information.

[0062] The binary log received by the synchronization server is originally in byte stream format. The synchronization server can parse the binary log from byte stream format into array entry format to obtain source operation information. The source operation information may include the column name of the database table, binary log file name, data change type, change data, etc.

[0063] Step 204 , determining whether the source operation information is of the DML type. If it is of the DML type, executing step 205 ; otherwise, executing step 208 .

[0064] Specifically, the Canal server module can distribute the parsed source operation information of each table to the cache stream corresponding to each table, and the Canal client module obtains the source operation information of each table from the corresponding cache stream. The Canal client module can determine the type of the source operation information, which includes DML type and DDL type.

[0065] Among them, DML type information is used to change the data in the table, that is, the operation object of DML type information is the record. If the source operation information is of DML type, it means that the information involves changes to the data in the table and requires operation of the ES server. DDL type information is used to change the structure of the table. That is, the operation object of DML type information is the table. If the source operation information is of DDL type, it means that the information does not involve changes to the data in the table, but only involves changes to the table structure, and does not require operation of the ES server.

[0066] Step 205: Obtain the configuration information of the table in the Json format from the configuration server, and convert the configuration information of the table from the Json format into the Java Bean format.

[0067] For example, the configuration information for each table can be pre-set on the configuration server. Each table's configuration information can include template fields that need to be displayed on the target server. The template fields can include fields such as the table header and the table data format. After obtaining the source operation information for a table, the configuration information for that table can be retrieved from the configuration server. The retrieved table configuration information can be in JSON format. After obtaining the table configuration information in JSON format, the table configuration information can be converted from JSON format to Java Bean format. The converted table configuration information can then be used to map the source operation information to the target operation information, facilitating data access and operation.

[0068] Step 206: Map the source operation information into target operation information according to the configuration information of the table.

[0069] Since the naming and format of the fields in the table of the MySQL server database are different from those in the ES server, directly writing the source operation information to the ES server will inconvenience the data viewing and operation. Therefore, it is necessary to map the source operation information to the target operation information.

[0070] For example, the names of the fields in the table of a MySQL server database are separated by underscores. For example, if a transaction number field is named transaction_no, it needs to be mapped to a name that meets the naming requirements of the target server according to the table configuration information, such as mapping it to transactionNo.

[0071] Step 207: Write the target operation information into the ES server, so that the ES server performs real-time data synchronization with the MySQL server according to the target operation information.

[0072] For example, after writing the target operation information to the ES server, the ES server can re-execute the events that occurred in the MySQL server locally based on the target operation information. For example, it can add new records, delete records, modify records, etc. in the corresponding table according to the target operation information to achieve real-time data synchronization between the ES server and the MySQL server. The data synchronized to the ES server can meet the real-time query needs of downstream terminals.

[0073] Step 208: Print source operation information.

[0074] For example, the source operation can be sent to a designated device in the form of an email for log printing, so that the user can process the change of the table structure according to the printed log.

[0075] The above technical solution can obtain the operation log of the table in the database from the source server, parse the operation log to obtain the source operation information, map the source operation information into target operation information according to the configuration information of the table, and write the target operation information to the target server, so that the target server can synchronize data with the source server in real time according to the target operation information; that is, in the embodiment of the present invention, data synchronization can be performed between the target server and the source server based on the real-time operation log, thereby realizing real-time synchronization of data, meeting the requirements of the system with real-time query needs for data; in addition, the functions of pulling and parsing the log, mapping the information, etc. are integrated into one device, which saves device resources and simplifies the system configuration.

[0076] Furthermore, in order to improve the reliability of data synchronization, it is also possible to synchronize data in real time and at a fixed time, such as Figure 3 As shown, the data processing method of the embodiment of the present invention may further include the following steps:

[0077] Step 301: Compare the data stored in the MySQL server with the data stored in the ES server to obtain differential data.

[0078] For example, you can periodically compare the data stored in the MySQL server with the data stored in the ES server to obtain differential data. For example, you can compare the data stored in the MySQL server with the data stored in the ES server during a period of time when the number of user requests is low (such as early morning) to obtain differential data.

[0079] Step 302: Generate timing synchronization data based on the difference data.

[0080] For example, when the data stored in the MySQL server is different from the data stored in the ES server, the data in the MySQL server can be used as the basis, and the scheduled synchronization data can be generated based on the difference data.

[0081] Step 303: Write the scheduled synchronization data into the ES server, so that the ES server performs scheduled data synchronization with the MySQL server according to the scheduled synchronization data.

[0082] Through the above technical solution, the real-time synchronization solution and the scheduled synchronization solution are combined to ensure that the data stored in the target server is the same as that in the source server, further improving the reliability of data synchronization.

[0083] Figure 4 FIG. 1 is a structural diagram of a data processing device provided by an embodiment of the present invention, and the device is suitable for executing the data processing method provided by an embodiment of the present invention. Figure 4 As shown, the device may specifically include:

[0084] The acquisition module 401 is used to obtain the operation log of the table in the database from the source server and parse the operation log to obtain the source operation information;

[0085] A mapping module 402, configured to map the source operation information into target operation information according to the configuration information of the table;

[0086] The synchronization module 403 is configured to write the target operation information into the target server, so that the target server performs real-time data synchronization with the source server according to the target operation information.

[0087] In one embodiment, the acquisition module 401 acquires the operation log of the table in the database from the source server, including:

[0088] The source server is used as a master node, and a slave node is simulated as a slave node of the source server to send a log acquisition request to the source server;

[0089] Receive the operation log returned by the source server.

[0090] In one embodiment, the mapping module 402 is further configured to:

[0091] Determining the type of the source operation information;

[0092] When the type of the source operation information is a data manipulation language (DML) type, a step of mapping the source operation information into target operation information according to the configuration information of the table is performed.

[0093] In one embodiment, the mapping module 402 is specifically configured to:

[0094] Obtain the configuration information of the table in JSON format from the configuration server, and convert the configuration information of the table from the JSON format into Java Bean format;

[0095] The source operation information is mapped into the target operation information according to the configuration information of the table in the Java Bean format.

[0096] In one embodiment, the synchronization module 403 is further configured to:

[0097] Generate timing synchronization data;

[0098] The timing synchronization data is written into the target server, so that the target server performs timing data synchronization with the source server according to the timing synchronization data.

[0099] In one embodiment, the synchronization module 403 generates the timing synchronization data, including:

[0100] Comparing the data stored in the source server with the data stored in the target server to obtain difference data;

[0101] The timing synchronization data is generated according to the difference data.

[0102] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0103] The device of the embodiment of the present invention can obtain the operation log of the table in the database from the source server, parse the operation log to obtain source operation information, map the source operation information into target operation information according to the configuration information of the table, and write the target operation information to the target server, so that the target server can synchronize data with the source server in real time according to the target operation information; that is, in the embodiment of the present invention, data synchronization can be performed between the target server and the source server based on the real-time operation log, thereby realizing real-time synchronization of data, meeting the requirements of the system with real-time query needs for data; in addition, the functions of pulling and parsing the log, mapping of information, etc. are integrated into one device, which saves device resources and simplifies system configuration.

[0104] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method provided in any of the above embodiments when executing the program.

[0105] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the data processing method provided by any of the above embodiments is implemented.

[0106] Figure 5 An exemplary system architecture is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0107] like Figure 5 As shown, the system architecture may include a source server 501, a target server 502, and an electronic device 503 for executing a data processing method as described in any of the embodiments of the present invention, wherein the source server 501 may be a MySQL server, the target server 502 may be an ES server, a Redis server, an HBase server, etc., and the electronic device 503 may be a synchronization server for synchronizing data or data changes of the source server 501 to the target server 502.

[0108] When performing data synchronization, the electronic device 503 can obtain the operation log of the table in the database from the source server 501, parse the operation log to obtain source operation information, map the source operation information into target operation information according to the configuration information of the table, and write the target operation information into the target server 502, so that the target server 502 can perform real-time data synchronization with the source server 501 according to the target operation information.

[0109] In addition, the system architecture may further include a configuration server, which is used to provide the electronic device 503 with configuration information of the table.

[0110] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the electronic device 503 , and accordingly, the data processing device is generally provided in the electronic device 503 .

[0111] Reference below Figure 6 , which shows a schematic structural diagram of a computer system 700 of an electronic device suitable for implementing an embodiment of the present invention. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 6As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0113] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.

[0114] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0115] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The modules and / or units described in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be provided in a processor. For example, a processor may be described as including an acquisition module, a mapping module, and a synchronization module. The names of these modules do not, in some cases, limit the modules themselves.

[0118] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment, or may exist independently and not be incorporated into the device. The computer-readable medium carries one or more programs, and when executed by the device, causes the device to: obtain an operation log of a table in a database from a source server, parse the operation log to obtain source operation information; map the source operation information into target operation information based on the configuration information of the table; and write the target operation information to a target server, so that the target server performs real-time data synchronization with the source server based on the target operation information.

[0119] According to the technical solution of an embodiment of the present invention, the operation log of a table in a database can be obtained from a source server, the operation log can be parsed to obtain source operation information, the source operation information can be mapped into target operation information according to the configuration information of the table, and the target operation information can be written to the target server, so that the target server can synchronize data with the source server in real time according to the target operation information; that is, in an embodiment of the present invention, data synchronization can be performed between the target server and the source server based on the real-time operation log, thereby achieving real-time synchronization of data, meeting the requirements of a system with real-time query needs for data; in addition, the functions of pulling and parsing logs, mapping information, etc. are integrated on a single device, which saves device resources and simplifies system configuration.

[0120] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Use the MySQL server as the MySQL master node, simulate it as a MySQL slave node through the Canal server module in the synchronization server, send a dump request to the MySQL server to request the operation log of the table, and parse the operation log to obtain the source operation information; Distributing the parsed source operation information of each table to the cache stream corresponding to each table, obtaining the source operation information of each table from the corresponding cache stream through the Canal client module in the synchronization server, and determining the type of the source operation information; wherein the type of the source operation information includes a data manipulation language (DML) type and a data definition language (DDL) type; Obtain the configuration information of the table in JSON format from the configuration server, and convert the configuration information of the table from the JSON format into Java Bean format; Mapping the source operation information into target operation information according to the configuration information of the table in the Java Bean format; Writing the target operation information to the target server so that the target server performs real-time data synchronization with the source server according to the target operation information; Before mapping the source operation information into target operation information according to the configuration information of the table, the method further includes: Determining the type of the source operation information; When the type of the source operation information is a data manipulation language (DML) type, performing a step of mapping the source operation information into target operation information according to the configuration information of the table; When the type of the source operation information is a data definition language DDL type, printing the source operation information; The method further comprises: Multiple synchronization servers are uniformly registered on the ZooKeeper server to form a synchronization server cluster; When the synchronization server fails, other synchronization servers are enabled by querying the ZooKeeper server.

2. The data processing method according to claim 1, wherein: The method further comprises: Generate timing synchronization data; The timing synchronization data is written into the target server, so that the target server performs timing data synchronization with the source server according to the timing synchronization data.

3. The data processing method according to claim 2, characterized in that: The generating of timing synchronization data includes: Comparing the data stored in the source server with the data stored in the target server at a preset time to obtain difference data; The timing synchronization data is generated according to the difference data.

4. The data processing method according to claim 1, wherein: The target server includes an ElasticSearch server.

5. A data processing device, characterized in that: include: The acquisition module is used to use the MySQL server as the MySQL master node, simulate the Canal server module in the synchronization server as a MySQL slave node, send a dump request to the MySQL server to request the operation log of the table, and parse the operation log to obtain the source operation information; Distributing the parsed source operation information of each table to the cache stream corresponding to each table, obtaining the source operation information of each table from the corresponding cache stream through the Canal client module in the synchronization server, and determining the type of the source operation information; wherein the type of the source operation information includes a data manipulation language (DML) type and a data definition language (DDL) type; a mapping module, configured to obtain the configuration information of the table in JSON format from a configuration server, and convert the configuration information of the table from the JSON format into a Java Bean format; and map the source operation information into target operation information according to the configuration information of the table in the Java Bean format; A synchronization module, configured to write the target operation information to a target server, so that the target server performs real-time data synchronization with the source server according to the target operation information; The mapping module is further configured to: determining the type of the source operation information; when the type of the source operation information is a data manipulation language (DML) type, executing a step of mapping the source operation information into target operation information according to the configuration information of the table; when the type of the source operation information is a data definition language (DDL) type, printing the source operation information; Among them, multiple synchronization servers are uniformly registered on the ZooKeeper server to form a synchronization server cluster; when the synchronization server fails, other synchronization servers are enabled by querying the ZooKeeper server.

6. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method according to any one of claims 1 to 4 when executing the program.

7. A data processing system, characterized in that: The method comprises a source server, a target server and an electronic device for executing the data processing method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

Images