Data processing method and device, computer readable storage medium, and processor

Abstract

The present invention discloses a data processing method and device, a computer-readable storage medium, and a processor. The method includes: obtaining a synchronization task of a first database and a second database, wherein the synchronization task is used to synchronize target data of the first database to the second database; performing data configuration according to the synchronization task to obtain configuration information, wherein the configuration information is used to determine the execution strategy of the synchronization task; obtaining target data from the first database according to the configuration information; storing the target data in an intermediate storage medium, and synchronizing the target data from the intermediate storage medium to the second database. The present invention solves the technical problem of low data interaction rate between different databases in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data processing method and device, a computer-readable storage medium, and a processor. Background Art

[0002] In related technologies, the interaction between relational databases and data warehouses mainly adopts Sqoop technology. In response to the massive data interaction needs in daily business applications, Sqoop technology needs to first divide the database and table according to business requirements and split the data into different sub-databases. This method has a complex implementation process and is difficult to configure, which results in an exponential increase in the difficulty of data interaction between different databases and a relatively slow data transmission speed between databases.

[0003] With respect to the problem of low data exchange rate between different databases in the above-mentioned prior art, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method and device, a computer-readable storage medium, and a processor to at least solve the technical problem of low data interaction rate between different databases in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a data processing method is provided, including: obtaining a synchronization task of a first database and a second database, wherein the synchronization task is used to synchronize target data of the first database to the second database; performing data configuration according to the synchronization task to obtain configuration information, wherein the configuration information is used to determine an execution strategy of the synchronization task; obtaining target data from the first database according to the configuration information; storing the target data in an intermediate storage medium, and synchronizing the target data from the intermediate storage medium to the second database.

[0006] Furthermore, the configuration information includes a metadata table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: obtaining data source information and sub-library information in the synchronization task; performing data configuration according to the data source information and sub-library information to obtain the metadata table.

[0007] Furthermore, the configuration information includes a mapping table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: obtaining step information and data source information of the synchronization task, wherein the step information includes subtasks for processing target data; data configuration is performed according to the step information and data source information to obtain a mapping table.

[0008] Furthermore, the data source information includes the synchronization direction from the first database to the second database, field information and the method of obtaining the target data; the step-by-step information includes any one or more of the following: a target data cleaning step, an index creation step, and a target table renaming or rebuilding step.

[0009] Furthermore, after synchronizing the target data from the intermediate storage medium to the second database, the above method also includes: determining whether the subtask is executed successfully; if all subtasks are executed successfully, determining that the synchronization task is executed successfully; if any subtask fails to execute, continuing to execute the subtasks of the synchronization task from the failed subtask.

[0010] Furthermore, the configuration information includes a task registration table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: registering the synchronization task in the task registration table.

[0011] Furthermore, storing the target data in an intermediate storage medium includes: cleaning and converting the format of the target data to obtain a target data file; and storing the target data file in the intermediate storage medium.

[0012] According to another aspect of an embodiment of the present invention, there is also provided a data processing device, including: a task acquisition module, used to acquire a synchronization task between a first database and a second database, wherein the synchronization task is used to synchronize target data of the first database to the second database; a configuration module, used to perform data configuration according to the synchronization task to obtain configuration information; a data acquisition module, used to acquire target data from the first database according to the configuration information; and a synchronization module, used to store the target data in an intermediate storage medium, and synchronize the target data from the intermediate storage medium to the second database.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, including: the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned data processing methods.

[0014] According to another aspect of an embodiment of the present invention, a processor is further provided, including: the processor is used to run a program, wherein any one of the above-mentioned data processing methods is executed when the program is run.

[0015] In an embodiment of the present invention, by acquiring synchronization tasks of a first database and a second database, performing data configuration according to the synchronization tasks to obtain configuration information, acquiring target data from the first database according to the configuration information, storing the target data in an intermediate storage medium, and synchronizing the target data from the intermediate storage medium to the second database, it is possible to dynamically arrange sequential steps of data interaction, and the intermediate storage medium provides data buffering for massive amounts of data to prevent system crashes caused by excessive amounts of data, thereby improving the data transmission rate between different databases, and thereby solving the technical problem of low data interaction rate between different databases in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of an optional data processing method according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] Example 1

[0023] According to an embodiment of the present invention, an embodiment of a method for processing data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] Figure 1 is a flow chart of a method for processing data according to an embodiment of the present invention. Figure 1As shown, the method comprises the following steps:

[0025] Step S102: Acquire a synchronization task of the first database and the second database, wherein the synchronization task is used to synchronize target data of the first database to the second database.

[0026] Specifically, the first database and the second database are two different databases that do not interact with each other. The types of the first database and the second database can be the same or different. For example, the first database and the second database can both be relational databases, or both can be data warehouses, or the first database can be a relational database and the second database can be a data warehouse.

[0027] The target data is data that needs to be transferred from the first database to the second database, and the synchronization task is used to transfer the target data in the first database to the second database. In an optional implementation, the target data is data that needs to be consumed in the terminal application, that is, data that the user needs to obtain through the terminal.

[0028] Step S104, performing data configuration according to the synchronization task to obtain configuration information, wherein the configuration information is used to determine the execution strategy of the synchronization task.

[0029] The execution strategy of the synchronization task may include database partition information, data transmission direction, execution order of data transmission steps, etc. Through data configuration, the database configuration information can be flexibly expanded or deleted, the order of data interaction can be dynamically arranged, and the data interaction speed between different databases can be achieved.

[0030] It should be noted that different data configurations can be flexibly performed for different synchronization tasks. When a synchronization task changes, only the configuration information needs to be changed without modifying the program code, which reduces the workload of developers.

[0031] Step S106: acquiring target data from the first database according to the configuration information.

[0032] The above configuration information may include identification information of the target data in the first database. For example, the identification information may be address information. The target data may be obtained according to the address information of the target data in the first database.

[0033] Step S108: storing the target data in an intermediate storage medium, and synchronizing the target data from the intermediate storage medium to the second database.

[0034] The intermediate storage medium is independent of the storage medium where the first database is located and the storage medium where the second database is located. It can be used for data buffering and temporary storage of target data during transmission, avoiding system crashes caused by excessive data volume, and improving the stability of data interaction between databases. By setting up the intermediate storage medium, repeated operations on target data can also be achieved. For example, when checking the size of a data file, if data inconsistencies are found, the transmitted data can be processed repeatedly. The intermediate storage medium can be HDFS (Hadoop Distributed File System), a hard disk, etc.

[0035] In an optional embodiment, the second database can be a relational database, the first database can be a data warehouse, and the user can directly interact with the front-end relational database through the terminal application. When the user issues a request to obtain the target data in the data warehouse, according to the user's acquisition request, a synchronization task for transferring the target data from the data warehouse to the relational database is determined. According to the synchronization task, the relational database and the data warehouse are configured, and the configuration information includes the identification and data transmission direction of the relational database and the data warehouse, and the execution steps of the data transmission. According to the configuration information, the target data is extracted from the data warehouse, and the target data is stored in the intermediate storage medium, and then the target data is synchronized from the intermediate storage medium to the relational database, and the relational database can feed back the target data to the terminal application. Compared with the data interaction rate of 2M / s of the sqoop method in the prior art, the data processing method in this embodiment increases the data transmission rate between databases to 20M / s. Without considering the bandwidth limitation, the transmission rate can reach 50M / s, and it only takes 1 minute to transmit 1G of data, which improves the data transmission rate between databases.

[0036] In this embodiment, by obtaining the synchronization task of the first database and the second database, data configuration is performed according to the synchronization task to obtain configuration information, the target data is obtained from the first database according to the configuration information, the target data is stored in an intermediate storage medium, and the target data is synchronized from the intermediate storage medium to the second database, thereby realizing the dynamic arrangement of sequential steps of data interaction. The intermediate storage medium provides data buffering for massive amounts of data to prevent system crashes caused by excessive amounts of data, thereby improving the data transmission rate between different databases and solving the technical problem of low data interaction rate between different databases in the prior art.

[0037] As an optional embodiment, the configuration information includes a metadata table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: obtaining data source information and sub-library information in the synchronization task; performing data configuration according to the data source information and sub-library information to obtain the metadata table.

[0038] The data source information may include the database name and type of the first database and the second database (for example, the database type may be a relational database, a Hive data warehouse, etc.), and the database connection information. The connection between the first database and the second database may be achieved based on the data source information.

[0039] The sub-library information may include information about data block storage in the first database and the second database. The location of the target data in the first database can be determined through the sub-library information, and then the target data can be extracted from the first database. Specifically, the sub-library information includes sub-libraries, sub-tables, and partitions in the first database and the second database. The sub-libraries, sub-tables, and partitions can improve the efficiency of data query.

[0040] As an optional embodiment, the configuration information includes a mapping table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: obtaining step-by-step information and data source information of the synchronization task, wherein the step-by-step information includes subtasks for processing target data; and data configuration is performed according to the step-by-step information and data source information to obtain a mapping table.

[0041] The step-by-step information is used to determine the logic of data interaction between the first database and the second database in the synchronization task. The mapping table has a status registration for each step of the operation, and the scheduling system can view the status of each node. The content in the data source information and distribution information can be flexibly defined. By changing the content in the mapping table, the execution order of data interaction between different databases can be dynamically arranged to improve the data transmission rate.

[0042] In an optional embodiment, the data source information includes the synchronization direction from the first database to the second database, field information and the method of obtaining the target data; the step-by-step information includes any one or more of the following: a target data cleaning step, an index creation step, and a target table renaming or rebuilding step.

[0043] The synchronization direction from the first database to the second database is the transmission direction of the target data, for example, from the relational database to the data warehouse, or from the data warehouse to the relational database. The target data is obtained by extracting the target data from the first database (for example, the target data can be a data stream replication method or a high-speed extraction method of a big data engine, etc.) and the extracted target data items. The field information is specifically the fields that constitute the records in the first database and the second database, and the data source information can also be the first database and the second database SQL (Structured Query Language, Structured Query Language) method.

[0044] The step-by-step information may include operation steps such as cleaning data before and after data synchronization, creating indexes, renaming target tables, and rebuilding tables. The step-by-step information can be flexibly configured according to user needs and is not limited to the above operation steps.

[0045] In an optional embodiment, the first database is a data warehouse, and the second database is a relational database. The method for synchronizing the target data in the data warehouse to the relational database includes: configuring according to the synchronization task of the target data, obtaining configuration information, the configuration information may include a metadata table and a mapping table, and by reading the configuration information, using a big data engine to extract the target data in the data warehouse at high speed, cleaning the data format of the target data, and converting it into the data file format required by the terminal application. At the same time, in the intermediate storage medium, the intermediate storage medium may include a disk and an HDFS cluster, and the intermediate storage medium may realize temporary storage of the target data to prevent the data transmission system from crashing due to excessive data volume. It should be noted that the HDFS cluster can not only be used for buffering the target data, but also for verifying the target data to ensure the consistency of data interaction, for example, verifying the number and size of data files. In the case of inconsistent data, the transmitted data can be repeatedly processed. Finally, the target data files in the intermediate storage medium are synchronized to the relational database at high speed using a data stream replication method to complete the synchronization of the target data in the data warehouse to the relational database.

[0046] In another optional embodiment, the first database is a relational database, and the second database is a data warehouse. The method for synchronizing the target data in the data warehouse to the relational database includes: configuring according to the synchronization task of the target data to obtain configuration information, the configuration information may include a metadata table and a mapping table, reading the configuration information, using a data stream replication method to copy the target data in the relational database at high speed, and cleaning the data format of the target data, converting it into a data file format required by the terminal application, and storing it in an intermediate storage medium to achieve temporary storage of the target data, and prevent the data transmission system from crashing due to excessive data volume. Use a big data engine to load the target data file in the intermediate storage medium into the data warehouse to complete the synchronization of the target data in the relational database to the data warehouse.

[0047] Through the above steps, the interaction rate between the relational database and the data warehouse is improved, so that both the relational database and the data warehouse can be used as the download data source for the terminal application, which not only overcomes the problem of a single data source for downloading data, but also effectively improves the problem that the data file format needs to be fixed in advance. By configuring data according to synchronization tasks, the flexibility, reliability and transmission efficiency of sending data from the data warehouse and relational database are improved, and the scalability in future applications is also guaranteed.

[0048] As an optional embodiment, after synchronizing the target data from the intermediate storage medium to the second database, the above method also includes: determining whether the subtask is executed successfully; if all subtasks are executed successfully, determining that the synchronization task is executed successfully; if any subtask fails to execute, continuing to execute the subtasks of the synchronization task from the failed subtask.

[0049] The above subtasks include the subtasks for processing the target data in the above step-by-step information, and the above subtasks can be saved in the mapping table through configuration.

[0050] It should be noted that in this embodiment, data transmission between the first database and the second database is considered successful only when all subtasks in the mapping table are successfully executed. If any subtask in the mapping table fails to execute, the data synchronization task fails, and the next time it is run, it will start from the failed subtask (i.e., the breakpoint) until all subtasks in the mapping table are successfully executed. By setting up an interactive check mechanism for checking subtasks, it is avoided to re-execute all subtasks when the synchronization task fails, reducing the impact on terminal application services during the data synchronization process.

[0051] As an optional embodiment, the configuration information includes a task registration table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: registering the synchronization task in the task registration table.

[0052] It should be noted that, in the data interaction between the first database and the second database, multiple synchronization tasks can be established, and different synchronization tasks are used for the transmission of different target data. The multiple synchronization tasks are registered one by one in the task registration table, and the transmission of different target data is implemented in sequence for the multiple synchronization tasks according to the above method steps.

[0053] As an optional embodiment, storing the target data in an intermediate storage medium includes: cleaning and converting the format of the target data to obtain a target data file; and storing the target data file in the intermediate storage medium.

[0054] It should be noted that the format of the target data file obtained after cleaning and conversion is a format supported by the terminal application, so that the target data file can be directly used by the terminal application.

[0055] Figure 2 is a schematic diagram of an optional data processing method according to an embodiment of the present invention, such as Figure 2As shown, the big data interaction device 20 is used to execute the above-mentioned data processing method, wherein the big data interaction device 20 includes a data processing core engine, an intermediate storage medium, and a configuration module for operating configuration information. The data processing core engine is used to realize the data transmission between the database 21 and the database 23 according to the above-mentioned data processing method. The data processing core engine can be any one of mapreduce (mapreduce is a hadoop processing engine), spark (a fast processing engine based on memory), flink (a stream processing engine), or other custom processing engines. The intermediate storage medium is used to buffer and temporarily store the data transmitted between the database 21 and the database 23. Application 22 and application 24 can be applications of terminal devices, application 22 can be a data generation application, and application 24 can be a data consumption application (i.e., a user and demander of data). Specifically, the data generated by application 22 is stored in database 21, and application 24 can obtain the data to be consumed from database 23. Database 21 and database 23 can interact through the big data interaction device 20, and the data in database 21 is transmitted to database 23, so that the user can obtain the required data from application 24. Specifically, the method of transmitting data from database 21 to database 23 may include the following steps:

[0056] Step S201, configure the collected database 21, application 22 and information generated by the log, message queue and other data of the application 22.

[0057] Step S202, configuring data consumption information of database 23, application 24, etc.

[0058] Step S203, configuring the configuration information in the big data interaction device 20, wherein the configuration information includes data processing arrangement logic.

[0059] Step 204: the data processing core engine stores the collected and processed data in an intermediate storage medium according to the configuration information.

[0060] Step S205 , the data processing core engine transmits the data in the intermediate storage medium to the database 23 , and then to the application 24 for use.

[0061] In this embodiment, the configuration can realize the sub-library and sub-table partitioning and the coarse and fine granularity extraction of data, improve the rate of data interaction, and flexibly configure the step sequence and logic of the step-by-step information of data interaction. When the interaction between different databases is adjusted and changed, no additional program development is required, which reduces the development workload. The components of the big data interaction device 20 (for example, the intermediate storage medium and the data processing core engine) can be plugged in and out, which makes it easy to modify and manually arrange the configuration information, reducing the development and maintenance costs. This embodiment can be applied to the data interaction between different databases in the CRM (Customer Relationship Management) system, which can shorten the interaction synchronization time point between different databases and save system operation time.

[0062] Example 2

[0063] According to an embodiment of the present invention, an embodiment of a data processing device is provided. Figure 3 is a schematic diagram of a data processing device according to an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0064] The task acquisition module 31 is used to acquire the synchronization task of the first database and the second database, wherein the synchronization task is used to synchronize the target data of the first database to the second database; the configuration module 32 is used to perform data configuration according to the synchronization task and obtain configuration information; the data acquisition module 33 is used to acquire the target data from the first database according to the configuration information; the synchronization module 34 is used to store the target data in an intermediate storage medium, and synchronize the target data from the intermediate storage medium to the second database.

[0065] This embodiment obtains synchronization tasks of the first database and the second database, performs data configuration according to the synchronization tasks, obtains configuration information, obtains target data from the first database according to the configuration information, stores the target data in an intermediate storage medium, and synchronizes the target data from the intermediate storage medium to the second database, thereby realizing the dynamic arrangement of sequential steps of data interaction. The intermediate storage medium provides data buffering for massive amounts of data to prevent system crashes caused by excessive amounts of data, thereby improving the data transmission rate between different databases, and thereby solving the technical problem of low data interaction rate between different databases in the prior art.

[0066] As an optional embodiment, the configuration information includes a metadata table, and the configuration module includes: a first acquisition submodule, used to obtain data source information and sub-library information in the synchronization task; a metadata configuration submodule, used to perform data configuration according to the data source information and sub-library information to obtain a metadata table.

[0067] As an optional embodiment, the configuration information includes a mapping table, and the configuration module includes: a second acquisition sub-module, used to obtain step-by-step information and data source information of the synchronization task, wherein the step-by-step information includes sub-tasks for processing target data; a mapping table configuration sub-module, used to perform data configuration according to the step-by-step information and data source information to obtain a mapping table.

[0068] As an optional embodiment, the data source information includes the synchronization direction from the first database to the second database, field information and the method of obtaining the target data; the step-by-step information includes any one or more of the following: a target data cleaning step, an index creation step, and a target table renaming or rebuilding step.

[0069] As an optional embodiment, the above-mentioned device also includes: a judgment module, used to judge whether the subtask is executed successfully; a determination module, used to determine that the synchronization task is executed successfully when all subtasks are executed successfully; and a continuation module, used to continue executing the subtasks of the synchronization task from the failed subtask when any subtask fails to execute.

[0070] As an optional embodiment, the configuration information includes a task registration table, and the configuration module includes: a registration submodule, which is used to register the synchronization task in the task registration table.

[0071] As an optional embodiment, the synchronization module includes: a cleaning submodule, which is used to clean and convert the format of the target data to obtain a target data file; and a storage submodule, which is used to store the target data file in an intermediate storage medium.

[0072] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0073] Example 3

[0074] According to an embodiment of the present invention, an embodiment of a computer-readable storage medium is provided, including: the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned data processing methods.

[0075] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: obtaining a synchronization task for a first database and a second database, wherein the synchronization task is used to synchronize target data of the first database to the second database; performing data configuration according to the synchronization task to obtain configuration information, wherein the configuration information is used to determine an execution strategy for the synchronization task; obtaining target data from the first database according to the configuration information; storing the target data in an intermediate storage medium, and synchronizing the target data from the intermediate storage medium to the second database.

[0076] According to an embodiment of the present invention, there is also provided an embodiment of a processor, including: the processor is used to run a program, wherein any one of the above-mentioned data processing methods is executed when the program is run.

[0077] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0078] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0081] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Acquire a synchronization task of a first database and a second database, wherein the synchronization task is used to synchronize target data of the first database to the second database; Performing data configuration according to the synchronization task to obtain configuration information, wherein the configuration information is used to determine an execution strategy for the synchronization task; Acquire the target data from the first database according to the configuration information; The target data is stored in an intermediate storage medium, and the target data is synchronized from the intermediate storage medium to the second database; the configuration information includes a mapping table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: obtaining step-by-step information and data source information of the synchronization task, wherein the step-by-step information includes subtasks for processing the target data; data configuration is performed according to the step-by-step information and the data source information to obtain the mapping table; after synchronizing the target data from the intermediate storage medium to the second database, the method further includes: judging whether the subtask is successfully executed; if all the subtasks are successfully executed, determining that the synchronization task is successfully executed; if any subtask fails to execute, continuing to execute the subtasks of the synchronization task from the failed subtask; the configuration information includes a metadata table, and data configuration is performed according to the synchronization task to obtain the configuration information, including: obtaining the data source information and sub-library information in the synchronization task; data configuration is performed according to the data source information and the sub-library information to obtain the metadata table.

2. The data processing method according to claim 1, characterized in that: The data source information includes the synchronization direction from the first database to the second database, field information, and the method for obtaining the target data; The step-by-step information includes any one or more of the following: a step of cleaning the target data, a step of creating an index, and a step of renaming or rebuilding a target table.

3. The data processing method according to claim 1, characterized in that: The configuration information includes a task registration table, and data configuration is performed according to the synchronization task to obtain configuration information, including: Register the synchronization task in the task registration table.

4. The data processing method according to claim 1, characterized in that: Storing the target data in an intermediate storage medium includes: Cleaning and converting the format of the target data to obtain a target data file; The target data file is stored in the intermediate storage medium.

5. A data processing device, characterized in that: include: A task acquisition module, used to acquire a synchronization task of a first database and a second database, wherein the synchronization task is used to synchronize target data of the first database to the second database; A configuration module, used to perform data configuration according to the synchronization task to obtain configuration information; A data acquisition module, configured to acquire the target data from the first database according to the configuration information; A synchronization module, used to store the target data in an intermediate storage medium, and synchronize the target data from the intermediate storage medium to the second database; the configuration information includes a mapping table, and the configuration module includes: a second acquisition submodule, used to acquire step information and data source information of the synchronization task, wherein the step information includes subtasks for processing the target data; a mapping table configuration submodule, used to perform data configuration according to the step information and the data source information to obtain the mapping table; the device also includes: a judgment module, used to determine whether the subtask is successfully executed after synchronizing the target data from the intermediate storage medium to the second database; a determination module, which determines that the synchronization task is successfully executed if all the subtasks are successfully executed; a continue execution module, used to continue to execute the subtasks of the synchronization task from the failed subtask if any subtask fails to execute; the configuration module includes: a first acquisition submodule, used to acquire the data source information and sub-library information in the synchronization task; a metadata configuration submodule, used to perform data configuration according to the data source information and the sub-library information to obtain a metadata table.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 4.

7. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data processing method described in any one of claims 1 to 4 when running.

Images