Data processing method and device, storage medium and electronic device

Abstract

The application discloses a data processing method and device, a storage medium and an electronic device. The method comprises the following steps: determining a first database to be synchronized to a data lake, and obtaining a plurality of first databases; traversing first configuration information of each first database, and respectively generating a first task corresponding to the first database according to the first configuration information; obtaining metadata and second configuration information in each first database, generating a second task according to the metadata and the second configuration information, and obtaining a plurality of second tasks; generating a third task according to each second data table, obtaining a plurality of third tasks, and executing the third task. Through the application, the problem of low efficiency and error-prone in manual data lake operation in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of data lake technology, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] A data lake is a large repository for storing diverse raw data, which can be accessed, processed, analyzed, and transmitted. Data lakes obtain raw data from multiple business systems within an enterprise, a process known as data ingestion. Currently, Hadoop is the most commonly used technology for deploying data lakes.

[0003] In related technologies, manual coding involves customizing data tables one by one according to the data model of the business system. Each time a new data table is added to the data lake, customized development is required. Development time is positively correlated with the number of data tables and the number of fields in each table. The more data tables and the more fields each table has, the greater the workload for developing each table. Therefore, manual development is inefficient and cannot guarantee code quality, potentially leading to technical problems such as missing or incorrect fields. Furthermore, manual coding lacks scalability and reusability, resulting in high error rates and testing costs.

[0004] There is currently no effective solution to the problems of low efficiency and error-proneness in manually performing data ingestion operations in related technologies. Summary of the Invention

[0005] This application provides a data processing method, apparatus, storage medium, and electronic device to solve the problems of low efficiency and error-proneness in manually performing data entry operations in related technologies.

[0006] According to one aspect of this application, a data processing method is provided. The method includes: determining a first database to be synchronized to a data lake, obtaining multiple first databases, wherein the first databases are databases of various application systems; traversing first configuration information of each first database, and generating a first task corresponding to each first database based on the first configuration information, wherein the first task is used to instruct the establishment of an association between the first database and a second database in the data lake; obtaining metadata and second configuration information in each first database, generating a second task based on the metadata and second configuration information, obtaining multiple second tasks, wherein the second task is used to instruct the construction of a second data table in the second database, and the second configuration information is the configuration information of the second data table; generating a third task based on each second data table, obtaining multiple third tasks, and executing the third tasks, wherein the third task is used to instruct the synchronization of data in the first data table to the data lake, and the first data table is a data table in the first database.

[0007] Optionally, after traversing the first configuration information of each first database, the method further includes: determining whether a target task is set in the first database; if a target task is set in the first database, detecting whether the target task is completed, wherein the target task is polled according to a preset duration, and if the target parameter of the target task is found to be a preset value, the target task is determined to be completed; if the target task is completed, the step of generating a first task corresponding to the first database according to each first configuration information is executed.

[0008] Optionally, generating a first task corresponding to the first database based on each first configuration information includes: obtaining the first database name and first database address of the first database from the first configuration information; determining the second database associated with the first database name and first database address in the data lake to obtain the target second database, and determining the second database name and second database address of the target second database; and establishing the association between the first database and the target second database based on the first database name, first database address, second database name and second database address.

[0009] Optionally, after obtaining the metadata and second configuration information in each first database, the method further includes: determining whether the data corresponding to the metadata is data that is being synchronized to the data lake for the first time; if the data corresponding to the metadata is data that is being synchronized to the data lake for the first time, establishing a database in the data lake to obtain the second database associated with the first database.

[0010] Optionally, performing the third task includes: determining the type of the first data table corresponding to the data to be synchronized to the data lake in the first database; determining whether the type of the first data table is a first type, and if the type of the first data table is a first type, synchronizing the data in the first database to the data lake through incremental synchronization, wherein the first data table of the first type is the first data table in the data lake whose historical data has not changed; if the type of the first data table is not a first type, determining whether the type of the first data table is a second type, and if the type of the first data table is a second type, synchronizing the data in the first database to the data lake through full synchronization, wherein the first data table of the second type is a first data table of dictionary code value type; if the type of the first data table is neither a first type nor a second type, synchronizing the data in the first database to the data lake through incremental synchronization.

[0011] Optionally, before performing the third task, the method further includes: determining whether the structure of the first data table is the same as the structure of the second data table in the data lake; if the structure of the first data table is different from the structure of the second data table in the data lake, updating the structure of the second data table so that the updated structure of the second data table matches the structure of the first data table.

[0012] Optionally, after generating multiple third tasks based on the second data table and executing the third tasks, the method further includes: obtaining the parallelism of the second tasks to obtain a first parallelism, and obtaining the parallelism of the third tasks to obtain a second parallelism; determining whether each second database carries a target flag, wherein the target flag is used to indicate the end of the execution of the third task; if each second database carries a target flag, determining a start node and an end node, determining multiple first branches based on the number of first databases, determining second branches based on the first parallelism of the second tasks, determining third branches based on the second parallelism of the third tasks, determining a first connection relationship based on the relationship between the first databases and the second tasks, and determining a second connection relationship based on the relationship between the second tasks and the third tasks; connecting the first branches and the second branches between the start node and the end node according to the first connection relationship, and connecting the second branches and the third branches according to the second connection relationship to obtain a directed acyclic graph.

[0013] According to another aspect of this application, a data processing apparatus is provided. The apparatus includes: a determining unit, configured to determine a first database to be synchronized to a data lake, thereby obtaining multiple first databases, wherein the first databases are databases of various application systems; a generating unit, configured to traverse first configuration information of each first database and generate a first task corresponding to each first database based on the first configuration information, wherein the first task is used to instruct the establishment of an association between the first database and a second database in the data lake; an acquiring unit, configured to acquire metadata and second configuration information from each first database, generate a second task based on the metadata and second configuration information, thereby obtaining multiple second tasks, wherein the second task is used to instruct the construction of a second data table in the second database, and the second configuration information is the configuration information of the second data table; and an executing unit, configured to generate a third task based on each second data table, thereby obtaining multiple third tasks, and execute the third tasks, wherein the third task is used to instruct the synchronization of data in the first data table to the data lake, and the first data table is a data table in the first database.

[0014] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute a data processing method.

[0015] According to another aspect of the present invention, an electronic device is also provided, comprising a processor and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions execute a data processing method when executed.

[0016] This application employs the following steps: First, a first database to be synchronized to the data lake is determined, resulting in multiple first databases, where each first database is a database of an application system; First configuration information of each first database is traversed, and a first task corresponding to each first database is generated based on the first configuration information, where the first task instructs the establishment of a relationship between the first database and a second database in the data lake; Metadata and second configuration information are obtained from each first database, and second tasks are generated based on the metadata and second configuration information, resulting in multiple second tasks, where the second tasks instruct the construction of second data tables in the second databases, and the second configuration information is the configuration information of the second data tables; Third tasks are generated based on each second data table, resulting in multiple third tasks, and the third tasks are executed, where the third tasks instruct the synchronization of data from the first data tables to the data lake, where the first data tables are data tables in the first databases. This solves the problems of low efficiency and high error rate in manually performing data entry operations in related technologies. By automatically creating databases and tables in the data lake based on the configuration information of the application system's databases and data tables, and performing data synchronization operations, the efficiency of data entry into the data lake is improved, and human error is avoided. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart of the data processing method provided in the embodiments of this application. Figure 1 ;

[0019] Figure 2 This is a schematic diagram of the data lake architecture provided according to the embodiments of this application;

[0020] Figure 3 This is a flowchart of updating the second data table structure according to the embodiments of this application;

[0021] Figure 4 It is a directed acyclic graph based on the embodiments of this application;

[0022] Figure 5 This is a flowchart of the data processing method provided in the embodiments of this application. Figure 2 ;

[0023] Figure 6 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Therefore, this application aims to provide a solution that can solve the above-mentioned technical problems, the details of which will be described in subsequent embodiments.

[0028] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0029] Data Lake: A data lake is a large warehouse that stores a wide variety of raw data for an enterprise, which can be accessed, processed, analyzed, and transmitted.

[0030] HAdoop: A distributed system infrastructure.

[0031] According to an embodiment of this application, a data processing method is provided.

[0032] Figure 1 This is a flowchart of the data processing method provided in the embodiments of this application. Figure 1 .like Figure 1 As shown, the method includes the following steps:

[0033] Step S102: Determine the first database to be synchronized to the data lake, resulting in multiple first databases, where each application system's database is a first database.

[0034] Specifically, a data lake is a data storage platform where data from various business systems within an enterprise needs to be synchronized to the data lake for unified storage and management. The primary database can be the database of each business system within the enterprise.

[0035] It should be noted that Hadoop is currently the most commonly used technology for deploying data lakes. The data lake in this application embodiment is deployed based on the Hadoop big data ecosystem. Figure 2 This is a schematic diagram of the data lake architecture provided in the embodiments of this application, such as... Figure 2 As shown, the data lake comprises: a data acquisition layer, a data storage layer, a computing engine layer, a data analysis layer, and a data application layer. The data acquisition layer includes the acquisition of business system data, log information, and message queue data. The data storage layer is based on Hadoop's distributed file system HDFS and Hive. HDFS offers high fault tolerance, is suitable for processing large amounts of data, and can be built on inexpensive machines. In this embodiment, data storage uses three replicas to maintain high fault tolerance. Columnar compressed storage is used, with optimizations for compression encoding and query performance. The computing engine layer describes the data lake's big data computing power. Offline computing uses Spark and Hive engines, while real-time computing uses Flink and Storm engines. The data analysis layer directly interfaces with user applications, providing them with easy-to-use data processing tools. A typical usage pattern is: first, a batch processing framework is used to analyze the raw massive data, generating a smaller dataset; then, interactive processing tools are used to quickly query this dataset to obtain the final results. The data application layer includes, but is not limited to, intelligent BI, regulatory reporting, risk control models, data dashboards, customer management, customer marketing, and ad-hoc analytics.

[0036] Step S104: Traverse the first configuration information of each first database and generate a first task corresponding to each first database based on each first configuration information. The first task is used to indicate the establishment of the association between the first database and the second database in the data lake.

[0037] Specifically, the first configuration information describes the database information of the business system whose data needs to be imported into the data lake. This first configuration information includes at least: business line code (the abbreviation of the original business system's English code); business line name (the abbreviation of the original business system in Chinese, such as core deposit system or online loan system); database type (e.g., relational databases like MySQL, TiDB, or Oracle); database name; database address and port; and the name of the database to be imported into the data lake—which database the data will be stored in after importation, i.e., the database name and address of the second database. The first task is used to characterize the relationship between the first and second databases, thereby determining the storage location of the synchronized data in the data lake.

[0038] Step S106: Obtain metadata and second configuration information from each first database, generate a second task based on the metadata and second configuration information, and obtain multiple second tasks. The second task is used to instruct the construction of a second data table in the second database, and the second configuration information is the configuration information of the second data table.

[0039] Specifically, the metadata can be the parameters of the first data table in the first database, the second configuration information can be the configuration information of the first data table, the second task is used to instruct the construction of a second data table with the same structure as the first data table in the second database in the data lake, the second data table is used to receive data synchronized to the data lake, the first data table and the second data table correspond one-to-one, and when synchronizing data, the data in the first data table is synchronized to the second data table.

[0040] Step S108: Generate a third task based on each second data table, obtain multiple third tasks, and execute the third tasks. The third task is used to instruct the data in the first data table to be synchronized to the data lake. The first data table is a data table in the first database.

[0041] Specifically, the third task is used to instruct the execution of data synchronization operations. During the data ingestion process, multiple first databases simultaneously perform the data ingestion operation. When each first database performs the data ingestion operation, the third task is executed simultaneously according to the second parallelism number. For example, if the second parallelism number is 20, the first database will simultaneously perform data ingestion operations on 20 first data tables. The third task is executed through an automatically generated Python script.

[0042] The data processing method provided in this application embodiment determines a first database to be synchronized to a data lake, resulting in multiple first databases, where each first database is a database of an application system. It iterates through the first configuration information of each first database and generates a first task corresponding to each first database based on the first configuration information. The first task instructs the establishment of a relationship between the first database and a second database in the data lake. It then obtains metadata and second configuration information from each first database, generates a second task based on the metadata and second configuration information, resulting in multiple second tasks. Each second task instructs the construction of a second data table in the second database, where the second configuration information is the configuration information of the second data table. Finally, it generates a third task based on each second data table, resulting in multiple third tasks, and executes the third tasks. Each third task instructs the synchronization of data from a first data table to the data lake, where the first data table is a data table in the first database. This method solves the problems of low efficiency and high error rate in manually performing data entry operations in related technologies. By automatically creating databases and tables in the data lake based on the configuration information of the application system's databases and data tables, and performing data synchronization operations, it improves the efficiency of data entry into the data lake and avoids human error.

[0043] Since the first database has incomplete target tasks during the data ingestion operation, the data ingestion operation can only be performed after the target tasks are completed. Optionally, in the data processing method provided in this application embodiment, after traversing the first configuration information of each first database, the method further includes: determining whether the first database has a target task set; if the first database has a target task set, detecting whether the target task is completed, wherein the target task is polled according to a preset duration, and if the target parameter of the target task is found to be a preset value, the target task is determined to be completed; if the target task is completed, the step of generating a first task corresponding to the first database according to each first configuration information is executed.

[0044] Specifically, the target task can be a batch processing task for settlement in the business system. If the batch processing task in the business system is not completed, the data in the first database of the business system will not have completed the settlement process. At this time, the data in the first database cannot be used for data ingestion because it is not the final data. Before performing data ingestion, a batch check needs to be performed on the business systems that need to perform data ingestion. If the target task of the business system is detected to have been successfully executed, the data ingestion operation will continue. By batch checking the business systems for any unfinished target tasks before data ingestion, it is ensured that the data to be synchronized in the business systems is in an accurate and synchronizable state when data is ingested.

[0045] For example, if the business system is an online loan system and the target task is a batch processing task, before executing the data ingestion operation, the batch dependency check configuration information is read, iterated through, and a batch detection task is generated. This batch detection task is used to check the SQL commands of each business system. The SQL for checking the online loan system is executed using the statement: `select BATCHFLAG from SYSTEM_SETUP whereBUSINESSDATE='${data_time}'`. Here, `data_time` is the batch date parameter, automatically passed in daily by the system. When the query result equals 2, it indicates that the batch detection task for the online loan system is complete, and the online loan system data synchronization ingestion task can begin; otherwise, the polling and detection continues after 5 minutes.

[0046] After the target task of the business system is determined, the data ingestion operation begins. First, the storage location of the data to be synchronized in the data lake is determined. Optionally, in the data processing method provided in this application embodiment, generating a first task corresponding to the first database based on each first configuration information includes: obtaining the first database name and first database address of the first database from the first configuration information; determining the second database associated with the first database name and first database address in the data lake to obtain the target second database, and determining the second database name and second database address of the target second database; establishing the association between the first database and the target second database based on the first database name, first database address, second database name and second database address.

[0047] For example, business systems that need to have their data imported into the data lake include the core deposit system and the online lending system. The database names and addresses of the core deposit system and the online lending system are obtained through their respective first configuration information. The corresponding second database name and address are determined in the data lake using the database name and address of the core deposit system. The corresponding second database name and address are also determined in the data lake using the database name and address of the online lending system.

[0048] If there is no second database in the data lake that is associated with the first database, the second database needs to be built in the data lake. Optionally, in the data processing method provided in this application embodiment, after obtaining the metadata and second configuration information in each first database, the method further includes: determining whether the data corresponding to the metadata is the data that is being synchronized to the data lake for the first time; if the data corresponding to the metadata is the data that is being synchronized to the data lake for the first time, a database is established in the data lake to obtain the second database associated with the first database.

[0049] Specifically, by comparing the parameters of the second data table in the data lake with the acquired metadata, if the two do not match, it indicates that the data corresponding to the metadata belongs to a new business system. If this is the first time data from a new business system is being imported into the data lake, a database creation statement will be automatically generated, and a second database corresponding to the first database of the new business system will be created in the data lake. By creating a second database for data from business systems that are being imported into the data lake for the first time, the efficiency of data import into the data lake is improved.

[0050] When performing the data ingestion task, the corresponding data synchronization method is selected based on the type of the first data table. Optionally, in the data processing method provided in this application embodiment, performing the third task includes: determining the type of the first data table corresponding to the data to be synchronized to the data lake in the first database; determining whether the type of the first data table is a first type, and if the type of the first data table is a first type, synchronizing the data in the first database to the data lake through incremental synchronization, wherein the first data table of the first type is a first data table in the data lake whose historical data has not changed; if the type of the first data table is not a first type, determining whether the type of the first data table is a second type, and if the type of the first data table is a second type, synchronizing the data in the first database to the data lake through full synchronization, wherein the first data table of the second type is a first data table of dictionary code value type; if the type of the first data table is neither a first type nor a second type, synchronizing the data in the first database to the data lake through incremental synchronization.

[0051] Specifically, the corresponding data synchronization method is selected based on the different types of the first data table. For the first type of data table, such as log tables where historical data will not change, incremental synchronization (dp, date partition) is used to perform the data ingestion operation. For the second type of data table, such as dictionary-value tables, full synchronization (sp, snapshot partition) is used to perform the data ingestion operation. For other types of first data tables, incremental synchronization is preferred.

[0052] It's important to note that when synchronizing data to the data lake, there are two import modes: query and table. If the import mode is query, an SQL statement is required; if the import mode is table, no SQL statement is needed. Data synchronized to the data lake via full synchronization is retained for 9999 days by default. For incremental synchronization, the first table imported into the data lake needs a WHERE clause. The WHERE clause indicates the range of data imported into the first database. If the first table is the first table imported into the data lake via incremental synchronization, the WHERE clause has no effect because automatic data initialization is performed during the initial import. It's best to use an index for the WHERE clause for faster query speeds. The date information in the WHERE clause can be correlated with the `run_date` parameter in the task information table of the business system to support rerun operations and resolve issues caused by tasks spanning multiple days. For example, `run_date = 2022-02-15`, with the specific date parameters as follows:

[0053] {run_date_before_1}:2022-02-14

[0054] {run_date_before_2}:2022-02-13

[0055] {run_date}: 2022-02-15

[0056] {run_date_after_1}:2022-02-16

[0057] {run_date_after_2}:2020-02-17

[0058] When data from the first data table imported into the data lake via incremental synchronization, the data is automatically initialized and stored in date-partitioned format, eliminating the need for manual partitioning and initial data import. Partitioning rules are dynamically applied based on the `data_time` field. For example: `from_unixtime(update_time,'%Y-%m-%d')data_time`. The parallelism of data synchronization is set for each first data table during import, for example, importing 2-5 million data entries in parallel per first data table. When the data volume in the first data table is large, data slicing is required to increase efficiency, enabling multi-task parallel import. When exporting data from the first data table in parallel, the field used for slicing prioritizes average slicing.

[0059] If the structure of the first data table changes, the second data table in the data lake will be automatically updated to maintain the same structure as the first data table. Optionally, in the data processing method provided in this application embodiment, before performing the third task, the method further includes: determining whether the structure of the first data table is the same as the structure of the second data table in the data lake; if the structure of the first data table is different from the structure of the second data table in the data lake, updating the structure of the second data table so that the updated structure of the second data table matches the structure of the first data table.

[0060] Specifically, by comparing the metadata of the second data table with the metadata of the first data table, the first data table whose table structure has changed is automatically identified and the structure change is synchronized to the second data table in the data lake, so as to keep the table structure information of the business system consistent with the table structure information of the data lake.

[0061] For example, Figure 3 According to the flowchart of updating the second data table structure provided in the embodiments of this application, as follows: Figure 3 As shown, the system obtains a list of business system databases to be collected, i.e., the list of first databases. When performing the data ingestion operation, it iterates through the list of business system databases and collects data from each of the first databases in turn. The metadata in the data lake records the version number of the second database. The metadata of the first database is collected and compared with the metadata in the data lake to determine whether the structure of the first data table has changed compared with the historical version of the data table in the data lake. If no structural change has occurred, the data synchronization and ingestion operation is performed directly. If a structural change has occurred, the identifier of the changed data is updated. The identifier includes at least "ingested", "added field", "modified", and "deleted". The changed data is automatically synchronized to the second data table, and the version number of the metadata in the data lake is updated.

[0062] Optionally, in the data processing method provided in this application embodiment, after generating multiple third tasks based on the second data table and executing the third tasks, the method further includes: obtaining the parallelism of the second tasks to obtain a first parallelism, and obtaining the parallelism of the third tasks to obtain a second parallelism; determining whether each second database carries a target flag, wherein the target flag is used to indicate the end of the execution of the third task; when each second database carries a target flag, determining a start node and an end node, determining multiple first branches based on the number of first databases, determining second branches based on the first parallelism of the second tasks, determining third branches based on the second parallelism of the third tasks, determining a first connection relationship based on the relationship between the first databases and the second tasks, and determining a second connection relationship based on the relationship between the second tasks and the third tasks; connecting the first branches and the second branches between the start node and the end node according to the first connection relationship, and connecting the second branches and the third branches according to the second connection relationship to obtain a directed acyclic graph.

[0063] Specifically, the first parallelism count can be the number of parallel operations for creating tables in each second database within the data lake. The second parallelism count can be the number of parallel operations for importing data from the first data table into the data lake from each first database during the data ingestion operation. For example, each first database might import data from 20 first data tables simultaneously. The target flag can be a flag representing the result after the data ingestion from the first database is completed. The target flag for each second database is generated sequentially based on the first configuration information. A start node (`start`) and an end node (`end`) are automatically generated. The first branch can be a branch pointing from `start` to multiple second databases. The second branch can be a branch that builds the first number of parallel second data tables for each second database. The third branch can be a branch that synchronizes data from the first data table of each first database to the data lake. A directed acyclic graph (DAG) for data ingestion is automatically constructed. The computational tasks in the task script are sequentially used to build the DAG, which describes the data ingestion task flow. The result of the entire DAG execution is written to a `success` file, marking the completion of the batch data ingestion task for all business databases. By constructing the DAG, the entire data ingestion process can be recorded, and the progress of data ingestion can be reported in a timely manner.

[0064] For example, Figure 4 It is a directed acyclic graph provided according to the embodiments of this application, such as Figure 4 As shown, the first branch, `start`, points to the first, second, third, fourth, fifth, and sixth application systems, and checks if each system has a branch with a target task. The second branch generates a synchronization task queue for each system, which is also the queue for building the second data table. The third branch synchronizes the data in the first data table simultaneously when each database performs data ingestion. Each synchronization task 1 and synchronization task 2 represents the simultaneous synchronization of data from 20 first data tables. After the synchronization task is completed, it points to `end`.

[0065] According to another embodiment of this application, a data processing method is provided.

[0066] Figure 5 This is a flowchart of the data processing method provided in the embodiments of this application. Figure 2 ,like Figure 5 As shown, the method includes:

[0067] Step S501: Initialize the start task and count the total number of business system databases that need to be included in the lake, which is used to intelligently generate downstream dependencies and synchronous parallel numbers.

[0068] Specifically, based on the start node of the directed acyclic graph generated by Airflow, the database model configuration information is read, the total number of business system databases that need to be included in the lake is counted, and the database model configuration information is obtained.

[0069] Step S502: Read the batch dependency check configuration information, traverse the batch dependency check configuration information, generate a batch detection task, set the upstream task of the batch detection task as the start task, and set the detection target instruction as the detection command executed by the batch detection task.

[0070] Specifically, the target command can be an SQL command, such as the batch detection SQL for the first application system: select BATCHFLAG from SYSTEM_SETUP where BUSINESSDATE='${data_time}'; data_time is the batch date parameter, which is automatically passed in by the system every day. When the query result is equal to 2 (according to the data dictionary definition of the business system), it means that the batch of the first application system has been completed and the data synchronization into the lake task of the first application system can be started. Otherwise, the polling and detection will continue after 5 minutes.

[0071] Step S503: Read the database model configuration information, traverse the business system database collection, and generate a database synchronization queue for each business system according to the database name.

[0072] Specifically, the generation rule for the synchronization queue name is DAG name_queue_database name. DMP_SQOOP_queue_ensemble represents the data synchronization queue of the core system database ensemble. The number of parallel tasks is controlled by the number of resource slots. Tasks that do not obtain resources will wait in the queue.

[0073] Step S504: Based on the read data model configuration information, deduce and construct upstream dependencies, deduce and generate downstream data table entry tasks, and generate N data entry tasks according to the number of parallel configurations N. The data entry tasks are automatically generated target programming scripts.

[0074] Specifically, the target programming script can be a Python script. For example, with N = 2, two data ingestion tasks are generated simultaneously for each database, and each data ingestion task performs data ingestion operations on 20 data tables simultaneously. The data ingestion is executed through an automatically generated Python script.

[0075] Step S505: Generate the data entry result flag for each business database sequentially based on the database configuration information.

[0076] Specifically, after data is imported into the database within the data lake, a marker indicating that the data has been imported is automatically displayed.

[0077] Step S506 automatically generates the final task node and writes the results of the entire directed acyclic graph execution to the target file, marking the completion of the batch task of importing all business database data into the lake.

[0078] Specifically, the final task node is "end", the target file can be a success file, the final task node is automatically generated as "end", the upstream and downstream dependencies are automatically generated, the parallelism of the tasks is automatically generated, and a directed acyclic graph (DAG) for batch task execution is automatically constructed. The DAG is built sequentially for the computation tasks in the task script. The DAG is used to describe the task flow of data entering the lake. The result of the entire DAG execution is written to the success file, marking the completion of the batch task of entering all business database data into the lake.

[0079] The data processing method provided in this application initializes a start task to count the total number of business system databases that need to be included in the data lake, which is used to intelligently generate downstream dependencies and the number of parallel operations. It reads batch dependency check configuration information, traverses the batch dependency check configuration information, generates batch detection tasks, sets the upstream tasks of the batch detection tasks as start tasks, and sets the detection target instruction as the detection command executed by the batch detection tasks. It reads database model configuration information, traverses the set of business system databases, and generates a database synchronization queue for each business system based on the database name. It derives and constructs upstream dependencies based on the read data model configuration information, derives and generates downstream data table inclusion tasks, and generates N data inclusion tasks according to the number of parallel operations N. The data inclusion tasks are automatically generated target programming scripts. It sequentially generates a data inclusion result flag for each business database based on the database configuration information. Finally, it automatically generates the last task node, writes the result of the entire directed acyclic graph execution to a target file, marking the completion of all business database data inclusion batch tasks. This technology solves the problems of low efficiency and high error rate in manually performing data import operations in related technologies. By automatically creating databases and tables in the data lake based on the configuration information of the application system's database and data tables, and performing data synchronization operations, it improves the efficiency of data import and avoids human error.

[0080] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0081] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing method provided in this application. The data processing apparatus provided in this application will be described below.

[0082] Figure 6 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application. For example... Figure 6 As shown, the device includes:

[0083] The determining unit 10 is used to determine the first database to be synchronized to the data lake, thereby obtaining multiple first databases, wherein the first databases are the databases of each application system.

[0084] The generation unit 20 is used to traverse the first configuration information of each first database and generate a first task corresponding to each first database according to each first configuration information. The first task is used to indicate the establishment of a relationship between the first database and the second database in the data lake.

[0085] The acquisition unit 30 is used to acquire metadata and second configuration information in each first database, generate a second task based on the metadata and second configuration information, and obtain multiple second tasks. The second task is used to instruct the construction of a second data table in the second database, and the second configuration information is the configuration information of the second data table.

[0086] The execution unit 40 is used to generate a third task based on each second data table, obtain multiple third tasks, and execute the third tasks. The third task is used to instruct the data in the first data table to be synchronized to the data lake. The first data table is a data table in the first database.

[0087] The data processing apparatus provided in this application embodiment determines, through a determining unit 10, a first database to be synchronized to a data lake, resulting in multiple first databases, wherein the first databases are databases of various application systems; a generating unit 20 traverses the first configuration information of each first database and generates a first task corresponding to each first database based on the first configuration information, wherein the first task is used to indicate the establishment of an association between the first database and a second database in the data lake; an acquiring unit 30 acquires metadata and second configuration information from each first database, and generates a second task based on the metadata and second configuration information, resulting in multiple second tasks, wherein the second task is used to... The system instructs the construction of a second data table in the second database, with the second configuration information being the configuration information of the second data table. The execution unit 40 generates a third task based on each second data table, resulting in multiple third tasks, and executes the third tasks. The third task is used to instruct the synchronization of data from the first data table to the data lake. The first data table is a data table in the first database. This solves the problems of low efficiency and error-proneness in manually performing data entry operations in related technologies. By automatically building databases and tables in the data lake based on the configuration information of the application system's database and data tables, and performing data synchronization operations, the system achieves the effect of improving data entry efficiency and avoiding human error.

[0088] Optionally, in the data processing apparatus provided in the embodiments of this application, the apparatus further includes: a first judgment unit, configured to judge whether a target task is set in the first database; a detection unit, configured to detect whether the target task is completed when a target task is set in the first database, wherein the target task is polled according to a preset time period, and the target task is determined to be completed when the target parameter of the target task is found to be a preset value; and a first task generation unit, configured to perform the step of generating a first task corresponding to the first database according to each first configuration information when the target task is completed.

[0089] Optionally, in the data processing apparatus provided in this application embodiment, the generation unit 20 includes: an acquisition module, configured to acquire the first database name and the first database address of the first database from the first configuration information; a first determination module, configured to determine the second database associated with the first database name and the first database address in the data lake, obtain the target second database, and determine the second database name and the second database address of the target second database; and an establishment module, configured to establish the association relationship between the first database and the target second database based on the first database name, the first database address, the second database name, and the second database address.

[0090] Optionally, in the data processing apparatus provided in the embodiments of this application, the apparatus further includes: a second judgment unit, used to judge whether the data corresponding to the metadata is data that is being synchronized to the data lake for the first time; and an establishment unit, used to establish a database in the data lake and obtain a second database associated with the first database when the data corresponding to the metadata is data that is being synchronized to the data lake for the first time.

[0091] Optionally, in the data processing apparatus provided in this application embodiment, the execution unit 40 includes: a second determining module, configured to determine the type of a first data table corresponding to the data to be synchronized to the data lake in the first database; a first judging module, configured to judge whether the type of the first data table is a first type, and when the type of the first data table is a first type, to synchronize the data in the first database to the data lake through incremental synchronization, wherein the first data table of the first type is a first data table in the data lake whose historical data has not changed; a second judging module, configured to judge whether the type of the first data table is a second type when the type of the first data table is not a first type, and when the type of the first data table is a second type, to synchronize the data in the first database to the data lake through full synchronization, wherein the first data table of the second type is a first data table of dictionary code value type; and a synchronization module, configured to synchronize the data in the first database to the data lake through incremental synchronization when the type of the first data table is neither a first type nor a second type.

[0092] Optionally, in the data processing apparatus provided in the embodiments of this application, the apparatus further includes: a third judgment unit, used to judge whether the structure of the first data table is the same as the structure of the second data table in the data lake; and an update unit, used to update the structure of the second data table when the structure of the first data table is different from the structure of the second data table in the data lake, so that the updated structure of the second data table matches the structure of the first data table.

[0093] Optionally, in the data processing apparatus provided in the embodiments of this application, the apparatus further includes: a parallel number acquisition unit, used to acquire the parallel number of the second task to obtain a first parallel number, and acquire the parallel number of the third task to obtain a second parallel number; a fourth judgment unit, used to determine whether each second database carries a target flag, wherein the target flag is used to indicate the end of the execution of the third task; a node relationship determination unit, used to determine a start node and an end node when each second database carries a target flag, determine multiple first branches according to the number of first databases, determine a second branch according to the first parallel number of the second task, determine a third branch according to the second parallel number of the third task, determine a first connection relationship according to the relationship between the first database and the second task, and determine a second connection relationship according to the relationship between the second task and the third task; and a connection unit, used to connect the first branch and the second branch between the start node and the end node according to the first connection relationship, and connect the second branch and the third branch according to the second connection relationship to obtain a directed acyclic graph.

[0094] The aforementioned data processing device includes a processor and a memory. The aforementioned determining unit 10, generating unit 20, acquiring unit 30, and executing unit 40 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0095] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured; adjusting kernel parameters can improve data ingestion efficiency and prevent human error.

[0096] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0097] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute a data processing method.

[0098] This application also provides an electronic device comprising a processor and a memory; the memory stores computer-readable instructions, and the processor executes the computer-readable instructions, wherein the computer-readable instructions perform a data processing method when executed. The electronic device described herein may be a server, PC, PAD, mobile phone, etc.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0104] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0105] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0106] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0107] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: The first database to be synchronized to the data lake is determined, resulting in multiple first databases, where each first database is a database of an application system. The first configuration information of each first database is traversed, and a first task corresponding to each first database is generated according to each first configuration information. The first task is used to indicate the establishment of an association between the first database and the second database in the data lake. Obtain metadata and second configuration information from each of the first databases, generate a second task based on the metadata and the second configuration information, and obtain multiple second tasks. The second task is used to instruct the construction of a second data table in the second database, and the second configuration information is the configuration information of the second data table. A third task is generated based on each of the second data tables, resulting in multiple third tasks, and the third tasks are executed. The third task is used to instruct the data in the first data table to be synchronized to the data lake. The first data table is a data table in the first database. After generating multiple third tasks based on the second data table and executing the third tasks, the method further includes: Obtain the parallelism of the second task to get the first parallelism, and obtain the parallelism of the third task to get the second parallelism, wherein the first parallelism is the parallelism of table creation for each second database in the data lake, and the second parallelism is the parallelism of data import from each first database to the first data table in the database. Determine whether each of the second databases carries a target flag, wherein the target flag is used to indicate that the third task has been completed; With each of the second databases carrying a target flag, a start node and an end node are determined. Multiple first branches are determined based on the number of first databases. Second branches are determined based on the first parallel number of the second task. Third branches are determined based on the second parallel number of the third task. A first connection relationship is determined based on the relationship between the first database and the second task. A second connection relationship is determined based on the relationship between the second task and the third task. The first branch and the second branch are connected between the starting node and the ending node according to the first connection relationship, and the second branch and the third branch are connected according to the second connection relationship to obtain a directed acyclic graph, wherein the directed acyclic graph is used to describe the task flow of data entering the lake.

2. The method according to claim 1, characterized in that, After traversing the first configuration information of each of the first databases, the method further includes: Determine whether the first database has a target task set; If the target task is set in the first database, it is detected whether the target task is completed. The target task is polled according to a preset time period, and if the target parameter of the target task is found to be a preset value, the target task is determined to be completed. If the target task is completed, the step of generating the first task corresponding to the first database based on each of the first configuration information is executed.

3. The method according to claim 1, characterized in that, The generation of the first task corresponding to the first database based on each of the first configuration information includes: Obtain the first database name and first database address of the first database from the first configuration information; In the data lake, determine the second database associated with the first database name and the first database address to obtain the target second database, and determine the second database name and the second database address of the target second database; The association between the first database and the target second database is established based on the first database name, the first database address, the second database name, and the second database address.

4. The method according to claim 1, characterized in that, After obtaining the metadata and second configuration information from each of the first databases, the method further includes: Determine whether the data corresponding to the metadata is the data being synchronized to the data lake for the first time; If the data corresponding to the metadata is the data that is being synchronized to the data lake for the first time, a database is established in the data lake to obtain the second database associated with the first database.

5. The method according to claim 1, characterized in that, Performing the third task includes: In the first database, determine the type of the first data table corresponding to the data to be synchronized to the data lake; Determine whether the type of the first data table is the first type, and when the type of the first data table is the first type, synchronize the data in the first database to the data lake through incremental synchronization, wherein the first data table of the first type is the first data table in the data lake whose historical data has not changed; If the type of the first data table is not the first type, determine whether the type of the first data table is the second type. If the type of the first data table is the second type, synchronize the data in the first database to the data lake through full synchronization. The first data table of the second type is a first data table of dictionary code value class. When the type of the first data table is neither the first type nor the second type, the data in the first database is synchronized to the data lake through incremental synchronization.

6. The method according to claim 5, characterized in that, Before performing the third task, the method further includes: Determine whether the structure of the first data table is the same as the structure of the second data table in the data lake; If the structure of the first data table is different from the structure of the second data table in the data lake, the structure of the second data table is updated so that the updated structure of the second data table matches the structure of the first data table.

7. A data processing apparatus, characterized in that, include: The determining unit is used to determine the first database to be synchronized to the data lake, and obtain multiple first databases, wherein the first databases are the databases of each application system; The generation unit is used to traverse the first configuration information of each of the first databases and generate a first task corresponding to the first database according to each of the first configuration information, wherein the first task is used to indicate the establishment of an association between the first database and the second database in the data lake; The acquisition unit is used to acquire metadata and second configuration information in each of the first databases, generate a second task based on the metadata and the second configuration information, and obtain multiple second tasks, wherein the second task is used to instruct the construction of a second data table in the second database, and the second configuration information is the configuration information of the second data table; An execution unit is configured to generate a third task based on each of the second data tables, obtain multiple third tasks, and execute the third tasks, wherein the third task is configured to instruct the data in the first data table to be synchronized to the data lake, and the first data table is a data table in the first database; A parallelism acquisition unit is used to acquire the parallelism of the second task to obtain a first parallelism, and to acquire the parallelism of the third task to obtain a second parallelism, wherein the first parallelism is the parallelism of table creation for each of the second databases in the data lake, and the second parallelism is the parallelism of importing data from the first data table into the database by each of the first databases; a fourth judgment unit is used to determine whether each of the second databases carries a target flag, wherein the target flag is used to indicate the end of the execution of the third task; a node relationship determination unit is used to determine the start node and the end node when each of the second databases carries a target flag, determine multiple first branches based on the number of first databases, determine second branches based on the first parallelism of the second task, determine third branches based on the second parallelism of the third task, determine a first connection relationship based on the relationship between the first database and the second task, and determine a second connection relationship based on the relationship between the second task and the third task; a connection unit is used to connect the first branch and the second branch between the start node and the end node according to the first connection relationship, and connect the second branch and the third branch according to the second connection relationship to obtain a directed acyclic graph, wherein the directed acyclic graph is used to describe the task flow of data entering the data lake.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute the data processing method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing computer-readable instructions, and the processor being configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed, perform the data processing method according to any one of claims 1 to 6.

Images