Data processing system, data processing method, and readable storage medium

Abstract

The present disclosure relates to a data processing system, a data processing method, and a readable storage medium. The system communicates with multiple data clusters equipped with data ETL components and Kafka components, and is configured to configure data ETL tasks for the data ETL components of the data clusters, causing the data ETL components to execute the data ETL tasks and implement data exchange between at least one source cluster and at least one target cluster via the Kafka component. In this embodiment, the ETL components and Kafka components can implement data ETL tasks from one source cluster to one target cluster, as well as from multiple source clusters to multiple target clusters. This can meet the data ETL task's requirements for multi-dimensional data and improve data exchange efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing system, a data processing method, and a readable storage medium. Background Art

[0002] Currently, various industries have formed different data centers, and each data center is isolated from each other. When data from two data centers is needed, the data from one data center can be synchronized to the other. For example, the data ETL (Extract-Transform-Load) conversion tool in the relevant technology can extract data from the cluster of one data center and synchronize it to the cluster of another data center. In actual applications, some requirements cannot be realized by relying on single-dimensional data, that is, these requirements need to rely on data from multiple dimensions. Therefore, data fusion and collaborative computing can only be performed when data from multiple data centers are brought together. However, the amount of data in each data center is so large that it is not realistic to store all the data from one data center in another data center, especially when the requirements involve data from multiple data centers. The above solution is even more impossible to implement. Summary of the Invention

[0003] The present disclosure provides a data processing system, a data processing method, and a readable storage medium to address the deficiencies of related technologies.

[0004] According to a first aspect of an embodiment of the present disclosure, a data processing system is provided, which communicates with multiple data clusters provided with data ETL components and Kafka components; the system is used to configure data ETL tasks to the data ETL components of the data clusters, so that the data ETL components execute the data ETL tasks and realize data exchange between at least one source cluster and at least one target cluster via the Kafka component.

[0005] Optionally, the source cluster includes at least one of the following: a Mysql data cluster, a PostgresSql data cluster, a Clickhouse data cluster, a Mongodb data cluster, an HDFS data cluster, an ElasticSearch data cluster, and a Kafka data cluster;

[0006] The target cluster includes at least one of the following: Mysql data cluster, PostgresSql data cluster, Clickhouse data cluster, Mongodb data cluster, HDFS data cluster, ElasticSearch data cluster, Redis data cluster and Kafka data cluster.

[0007] Optionally, the system is configured to configure a data ETL task to a data ETL component of a data cluster, so that the data ETL component executes the data ETL task and implements data exchange between at least one source cluster and at least one target cluster via a Kafka component, including:

[0008] The first data ETL component corresponding to the data ETL task reads the target data required by the data ETL task from the source cluster and outputs it to the Kafka component in the source cluster;

[0009] The second data ETL component corresponding to the data ETL task reads the target data from the Kafka component in the source cluster and outputs the target data to the Kafka component in the target cluster;

[0010] The third data ETL component corresponding to the data ETL task reads the target data from the Kafka component in the target cluster and outputs the target data to the target cluster.

[0011] Optionally, the system is further configured to:

[0012] Acquire task information from the first data ETL component, the second data ETL component, and the third data ETL component, and display the task information, wherein the task information is used to enable maintenance personnel to locate abnormal tasks;

[0013] In response to the operation of processing the abnormal task, the data ETL task corresponding to the abnormal task is resumed.

[0014] Optionally, the Kafka component performs persistence processing on the target data.

[0015] Optionally, different business data in the source cluster may be implemented using different topics.

[0016] Optionally, the nodes of the Kafka component can be horizontally expanded as the number of topics increases.

[0017] Optionally, the Kafka component is configured with multiple consumer groups, each of which corresponds to at least one data ETL task.

[0018] Optionally, the system is further configured to:

[0019] When a cross-cluster data ETL task is detected, an audit task is generated and sent to the target user corresponding to the audit task;

[0020] In response to detecting that the target user has passed the operation of the review task, the data ETL task is generated and configured to the data ETL component.

[0021] Optionally, the system is further configured to:

[0022] The task log of the data ETL task is obtained and displayed, so that the user can query the status information of each data ETL task at each stage.

[0023] Optionally, the system is further configured to:

[0024] When an operation indicating closing or pausing a data ETL task is detected, the data ETL component corresponding to the data ETL task is controlled to pause reading data from the Kafka component and continue outputting data;

[0025] When the data output is completed, the data ETL component corresponding to the data ETL task is controlled to close or pause the data ETL task.

[0026] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0027] A data ETL task is configured for a data ETL component of the data cluster, so that the data ETL component executes the data ETL task and implements data exchange between at least one source cluster and at least one target cluster via the Kafka component.

[0028] According to a third aspect of an embodiment of the present disclosure, there is provided a data processing system comprising at least one processing device and at least one storage device;

[0029] the at least one storage device for storing a computer program executable by the at least one processing device;

[0030] The at least one processor device is configured to execute the computer program in the at least one memory to implement the method as described in the second aspect.

[0031] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When an executable computer program in the storage medium is executed by a processor, the method described in the second aspect can be implemented.

[0032] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0033] As can be seen from the above embodiments, in the solutions provided by the embodiments of the present disclosure, the data processing system can communicate with multiple data clusters equipped with data ETL components and Kafka components, and is used to configure data ETL tasks to the data ETL components of the data clusters, so that the data ETL components execute the data ETL tasks and implement data exchange between at least one source cluster and at least one target cluster via the Kafka component. In this embodiment, the ETL components and Kafka components can implement data ETL tasks from one source cluster to one target cluster, as well as from multiple source clusters to multiple target clusters. This can meet the data ETL task's requirements for multi-dimensional data and improve data exchange efficiency.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0036] Figure 1A The figure is a block diagram showing a data cluster used in manufacturing according to an exemplary embodiment.

[0037] Figure 1B The figure is a block diagram showing a data cluster used in a smart city according to an exemplary embodiment.

[0038] Figure 2 The present invention is a block diagram showing a data processing system configuration for inter-cluster data exchange according to an exemplary embodiment.

[0039] Figure 3 The figure is a block diagram showing a method of implementing a data exchange task in three stages according to an exemplary embodiment.

[0040] Figure 4 The figure is a block diagram showing a method of reviewing a data ETL task according to an exemplary embodiment.

[0041] Figure 5 The figure is a schematic diagram showing a data ETL task status according to an exemplary embodiment.

[0042] Figure 6 The figure is a flowchart showing data exchange when setting an external request according to an exemplary embodiment.

[0043] Figure 7 It is a block diagram of a data processing system according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, with examples shown in the accompanying drawings. When the following description refers to the drawings, identical numbers in different drawings represent identical or similar elements, unless otherwise indicated. The exemplary embodiments described below do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims. It should be noted that, unless there is a conflict, the features of the following embodiments and implementations may be combined with each other.

[0045] To solve the above technical problems, the present disclosure provides a data processing system that can communicate with multiple data clusters equipped with data ETL components and Kafka components. Figure 1A The data cluster 100 includes a display device 200, a data processing device 300, and a distributed storage device 400. The data processing device 300 is connected to the display device 200 and the distributed storage device 400 respectively.

[0046] It should be noted that data cluster 100 can be applied to different scenarios, such as manufacturing, quality inspection, traffic information, government administration, smart cities, etc., and the corresponding raw data is related to each scenario. For ease of description, this disclosure uses the raw data of production equipment, which is stored in the corresponding manufacturing system.

[0047] Continue to see Figure 1A The distributed storage device 400 includes a data lake layer, a data warehouse layer (HIVE), and a data mart (HBASE). Users can enter query parameters or configure data ETL tasks through the interactive interface on the display device 200. The display device 200 can also access the data mart through an API interface. The data processing device 300 can access the data mart through the API interface, and then process the data obtained from the data mart and send it to the display device 200 for display.

[0048] Continue to see Figure 1AThe data cluster 100 includes multiple sets of data with different content and / or storage structures, and is stored in the distributed storage device 400. In some embodiments, the data ETL component (a piece of program code, or a functional module, or a hardware circuit, not limited to) in the distributed storage device 400 can extract raw data from multiple data sources into the data cluster to form a first data layer (for example, a data lake layer DL) to reduce the load on product production equipment and production manufacturing systems, and facilitate data reading by subsequent analysis equipment. The data source can be the raw data of the production equipment, which is stored in the corresponding production manufacturing system, such as the relational database (such as Oracle, Mysql, etc.) of the YMS (Yield Management System), FDC (Fault Detection & Classification), MES (Manufacturing Execution System), etc. The above-mentioned data ETL component refers to a computer program logic configured to provide functions such as extracting, converting or loading data. In some embodiments, the data ETL component is stored on one or more storage nodes in the distributed network, loaded into one or more memories in the distributed network, and executed by one or more processors in the distributed network.

[0049] The data lake layer in the distributed storage device 400 is a centralized HDFS (Hadoop Distributed File System) or KUDU database for storing any structured or unstructured data. Optionally, the data lake is configured to store a first set of data extracted by the data ETL component from multiple data sources DS. Optionally, the first set of data and the original data have the same content. The dimensions and attributes of the original data are saved in the first set of data. In some embodiments, the first set of data stored in the data lake includes dynamically updated data. Optionally, the dynamically updated data includes data updated in real time based on a Kudu database, or data updated periodically in the Hadoop distributed file system. In one example, the periodically updated data stored in the Hadoop distributed file system is stored in a Hive-based storage. In one example, the dynamically updated data also includes real-time updated data, and the real-time update means updates below the minute level, but does not include updates of minutes, to distinguish it from the above-mentioned periodic updates that mean updates above the minute level and include updates of minutes.

[0050] In some embodiments, the distributed storage device 400 also includes a second data layer, such as a data warehouse. The data warehouse includes an internal storage system that has the characteristic of providing data in an abstract manner without exposing a file system, wherein the abstract manner may include a table format or a view format. The data warehouse can be implemented based on Hive. At this time, the data ETL component can extract, clean, convert or load the first set of data to form a second set of data. Optionally, the first set of data can form a second set of data after cleaning and standardization. In some embodiments, the second set of data also includes statistical data, such as detection point counts, maximum, minimum and average detection point parameter values, defect ratio, etc.

[0051] In some embodiments, the distributed storage device 400 includes a third data layer, such as at least one data mart. Optionally, the data mart is a NoSQL database that stores data that can be used for computational processing. Optionally, the data mart is implemented based on HBase. The data ETL component can also transform the second data to form a third set of data.

[0052] Those skilled in the art will appreciate that the first set of data, the second set of data, and the third set of data may be stored and queried in the form of one or more data tables.

[0053] In some embodiments, the process of converting the second set of data to form the third set of data can be to import the data in the data warehouse (hive table) into the data mart (Hbase table). In one example, a first table is generated in the data mart, and a second table (e.g., an external table) is generated in the data warehouse. The first table and the second table are configured to be synchronized so that when data is written to the second table, the first table will be updated at the same time to include the corresponding data. In another example, the MapReduce module in Hadoop can be used as a distributed computing processing module to read the data written to the data warehouse. The data written to the data warehouse can then be written to the data mart. In one example, data can be written to the data mart using an HBase API. In another example, once the MapReduce module reads the data written to the data mart, it can generate an HFile file and load it in batches (Bulkloaded) to the data mart.

[0054] In some embodiments, the data flows, data transformations, and data structures between various components of a data cluster are described. In some embodiments, the raw data collected by multiple data sources DS includes at least one of production history data, parameter data, or detection parameter data. The raw data may optionally include dimensional information (time, factory, equipment, operator, map, chamber, slot, etc.) and attribute information (factory location, equipment age, number of bad points, abnormal parameters, energy consumption parameters, processing duration, etc.).

[0055] The production history data includes information about the specific processes that a product (eg, panel or glass) undergoes during manufacturing. Examples of the specific processes that a product undergoes during manufacturing include factories, processes, stations, equipment, chambers, slots, and operators.

[0056] The parameter data contains information about the specific environmental parameters and their variations that a product (such as a panel or glass) is subjected to during manufacturing. Examples of the specific environmental parameters and their variations that a product is subjected to during manufacturing include ambient particle conditions, equipment temperature, and equipment pressure.

[0057] The detection parameter data includes the resistance, film thickness, threshold voltage, reflection pattern deviation, reverse cutoff current, etc. of the product detected based on the detection site.

[0058] In one example, data cluster 100 integrates various business data (e.g., data related to semiconductor electronic device manufacturing) into multiple data sources (DS) (e.g., Oracle databases). The data ETL component, using tools such as DataStack, SQOOP, Kettle, Pentaho, or DataX, extracts data from multiple data sources into a data lake. The data is then cleansed, transformed, and loaded into a data warehouse. The data warehouse (DW) and data mart (DMT) utilize tools such as Kudu, Hive, and HBase to store large amounts of data and analysis results.

[0059] Information generated at various stages of the manufacturing process is acquired by various sensors and inspection equipment and subsequently stored in multiple data sources DS. Alternatively, data acquired by sensors and inspection equipment is calculated or analyzed, with the calculation and analysis results also stored in multiple data sources DS. Data synchronization (data flow) between the various components of the data cluster is achieved through a data ETL component. For example, the data ETL component is configured to obtain a parameter configuration template for the synchronization process, including network permissions and database port configuration, inbound database and table names, outbound database and table names, field mappings, task types, scheduling periods, etc. The data ETL component configures the parameters of the synchronization process based on the parameter configuration template. The data ETL component synchronizes data and cleans the synchronized data based on the process configuration template. The data ETL component cleans the data using SQL statements to remove null values, remove outliers, and establish correlations between related tables. Data synchronization tasks include data synchronization between multiple data sources and the distributed storage device 400, as well as data synchronization between the various layers of the distributed storage device 400 (e.g., a data lake, data warehouse, or data mart).

[0060] In another example, the distributed storage device 400 can complete data extraction to the data lake in real time or offline. In offline mode, data extraction tasks are scheduled periodically. Optionally, in offline mode, the extracted data can be stored in a storage device based on the Hadoop distributed file system (for example, a Hive-based database). In real-time mode, the data extraction task can be performed by OGG (Oracle GoldenGate) in combination with Apache Kafka. Optionally, in real-time mode, the extracted data can be stored in a Kudu-based database. OGG reads log files from multiple data sources (for example, Oracle databases) to obtain add / delete data. In one example, a front-end interface (such as an API interface) can perform display, query and / or analysis based on data stored in a Kudu-based database. In another example, the front-end interface can be based on data stored in a Kudu-based database, a Hadoop distributed file system (for example, an Apache Hive-based database). TIn another example, the data ETL component is configured to extract data stored in the Kudu-based database into the Hadoop distributed file system (e.g., a Hive-based database).

[0061] By combining data from various business systems (MDW, YMS, MES, FDC, etc.), a data warehouse is built based on the data lake. The data extracted from the data lake is divided according to the task execution time, and the task execution time does not completely match the timestamp in the original data. In addition, there is a possibility of data duplication. Therefore, it is necessary to build a data warehouse based on the data lake by cleaning and standardizing the data in the data lake to meet the needs of upper-level applications for data accuracy and division. The data tables stored in the data warehouse are obtained by cleaning and standardizing the data in the data lake. Based on user needs, the field format is standardized to ensure that the data tables in the data warehouse are completely consistent with the data tables in multiple data sources. At the same time, dividing data by date or month, according to time and other fields greatly improves query efficiency and reduces running memory requirements. The data warehouse can be one or any combination of a Kudu-based database and an Apache Hive-based database.

[0062] In one embodiment, the distributed storage device 400 may be a memory, multiple memories, or a collective term for multiple storage elements. For example, the memory may include random access memory (RAM), double data rate synchronous dynamic random access memory (DDR SRAM), and non-volatile memory (non-volatile memory), such as disk storage and flash memory.

[0063] The display device 200 is used for displaying an interface and can display the processing results of the data processing device 300. In one embodiment, the display device can be a display, or a product including a display, such as a television, a computer (all-in-one or desktop), a computer, a tablet computer, a mobile phone, an electronic screen, etc. In one embodiment, the display device can be any device that displays either moving (e.g., video) or fixed (e.g., still image), and whether text or images. More specifically, it is expected that the embodiments described can be implemented in or associated with a variety of electronic devices, such as (but not limited to) game consoles, television monitors, flat-panel displays, computer monitors, automotive displays (e.g., odometer displays, etc.), navigation systems, cockpit controllers and / or displays, electronic photos, electronic billboards or signs, projectors, architectural structures, packaging, and aesthetic structures (e.g., displays of images of a piece of jewelry), etc.

[0064] Understandably, Figure 1A The embodiment shown only introduces the composition of one data cluster 100 with which the data processing system communicates. For scenarios such as smart cities and smart parks, such scenarios usually have multiple subsystems, and each subsystem is provided with at least one data cluster 100, such as Figure 1B As shown in the figure, a smart city, for example, typically comprises numerous subsystems, including municipal administration, urban transportation, urban economy, urban hospitals, and urban fire protection. Given the spatial dispersion of these subsystems, practical applications require data collection from each subsystem for fusion and holistic mining and analysis. However, due to geographic location, data security concerns, and data exchange efficiency, direct point-to-point transmission of these subsystems is prone to transmission failures and low efficiency.

[0065] Considering that data ETL components in related technologies can only extract data from one data cluster and synchronize it to another, and cannot implement data exchange from one data cluster to multiple data clusters or from multiple data clusters to multiple data clusters, the data cluster 100 in the present disclosure is also provided with a Kafka component, where Kafka is a distributed message queue. The data ETL component in data cluster 100 can communicate with the Kafka component in the same data cluster 100, that is, the data ETL component can send the acquired data as a message to the Kafka component for consumption by other data clusters. In other words, in the embodiments of the present disclosure, the Kafka component is used as an intermediary for data exchange to enable data exchange between two or more data clusters.

[0066] In this embodiment, the data processing system can communicate with the data ETL components in each data cluster. Figure 2, the user can display the list of communicable data clusters through the display device of the data processing system; then configure the data extraction rules, such as configuring the conversion rules of each data cluster, the number of topics of the kafka component, the number of consumer groups, etc. After obtaining the configuration data, the data processing system can configure the data ETL components of each data cluster. In one example, the source cluster includes at least one of the following: Mysql data cluster, PostgresSql data cluster, Clickhouse data cluster, Mongodb data cluster, Hdfs data cluster, ElasticSearch data cluster and Kafka data cluster. The target cluster includes at least one of the following: Mysql data cluster, PostgresSql data cluster, Clickhouse data cluster, Mongodb data cluster, Hdfs data cluster, ElasticSearch data cluster, Redis data cluster and Kafka data cluster. The technicians can select the appropriate data cluster according to the specific scenario, and the corresponding solution falls within the protection scope of this disclosure.

[0067] In this embodiment, the user can also set a data ETL task in the display device of the data processing system, such as setting a data source cluster, a target cluster, data conversion rules in the data source cluster, data conversion rules in the target cluster, data time, data location, and data business type, etc. The selection can be based on the specific scenario and is not limited here. After receiving the data ETL task, the data processing system can assign the data ETL task to the data ETL component of the corresponding data cluster. The data ETL component can then execute the data ETL task and implement data exchange between at least one source cluster and at least one target cluster via the Kafka component.

[0068] In this embodiment, see Figure 3 ,The data exchange process can be divided into three stages, including: ,The first stage, the data of the source cluster is transferred to the ,kafka component of the source cluster, i.e. Figure 3 The data synchronization task 1 shown in the figure; in the second stage, the data is transferred from the Kafka component of the source cluster to the Kafka component of the target cluster, that is, Figure 3 The data synchronization task 2 shown in the figure; in the third stage, the data is transferred from the Kafka component of the target cluster to the data source of the target cluster, that is, Figure 3 The data synchronization task shown in Figure 3. The data exchange process is described based on the above three stages:

[0069] In the first stage, the data ETL component in the source cluster corresponding to the data ETL task, namely the first data ETL component, can read the target data required by the data ETL task from the source cluster and output it to the Kafka component in the source cluster, completing the data synchronization task in the first stage.

[0070] In the second stage, the data ETL component in the data processing system corresponding to the data ETL task, namely the second data ETL component, can read the above target data from the Kafka component in the source cluster and output it to the Kafka component in the target cluster, completing the data synchronization task in the second stage.

[0071] In the third stage, the data ETL component in the target cluster corresponding to the data ETL task, namely the third data ETL component, reads the above target data from the Kafka component in the target cluster and outputs it to the data source of the target cluster, completing the data synchronization task in the third stage.

[0072] It should be noted that when outputting data, the first data ETL component, the second data ETL component and the third data ETL component can group and store the transmitted data according to the data transmission time, or group and store the data according to the time field inherent in the data, thereby improving storage efficiency and the efficiency of subsequent search and reading.

[0073] It should be noted that when the first, second, and third data ETL components are a piece of program code, they differ in their configuration parameters. These configuration parameters include, but are not limited to, input data (source address), output data (destination address), and data conversion rules. Since these three data ETL components differ only in their configuration parameters, they can achieve the same data transmission efficiency.

[0074] It's important to note that the combination of the data ETL component and the Kafka component in this embodiment divides the data ETL task into three stages. When an exception occurs in a data ETL task, the abnormal stage can be located and the data ETL task can be restored from that abnormal stage, reducing the risk of data ETL task failure. Compared to related technologies in which the data ETL component directly transfers data between two clusters, this embodiment eliminates the need for data retransmission, improving data transmission efficiency.

[0075] In this embodiment, in the above three stages, when the kafka component receives the target data, it can persist the target data, that is, it can store it for a period of time according to a preset retention time (which can be set, such as 7 days). When an exception (error or failure, etc.) occurs in the source cluster or the target cluster or the data ETL task, the target data can be backtracked within the preset retention time to ensure the security of the target data. In one embodiment, the kafka component can also generate at least one copy of the target data according to the configuration information, that is, the target data is synchronously saved to different nodes in other kafka clusters to achieve data backup and ensure the security and reliability of the data. In addition, in this embodiment, by making the target data persistent, it is convenient to adjust the consumption time of the target data, so that the user can flexibly adjust the start time and end time of the target data according to the needs to meet the needs of different data ETL tasks.

[0076] In one embodiment, when data is exchanged at various stages of a data ETL task, the Kafka component may use different topics to implement different business data in the source cluster. Taking a smart city as an example, the above-mentioned business data may include but is not limited to municipal management, urban transportation, urban economy, urban hospitals, urban fire protection and other aspects of business. Since the data between each topic is isolated from each other and do not affect each other, the transmission of data by multiple different topics is equivalent to adding multiple data transmission channels. For data exchange across clusters, it can be equivalent to multiple data ETL components in the related technology synchronously executing data exchange tasks. The solution of this embodiment can double the amount of data transmitted, thereby meeting the data ETL task's requirements for data in multiple dimensions.

[0077] In one embodiment, as the number of topics in a Kafka component continues to increase, the throughput of the corresponding Kafka cluster also increases, leading to bottlenecks in data exchange performance. Therefore, in this embodiment, the nodes of the Kafka cluster can be horizontally expanded. This allows the newly added nodes to share some of the topics, thereby reducing the number of topics per node and increasing the throughput of the Kafka cluster. This helps improve the performance of the Kafka cluster and, in turn, shortens the time it takes to complete data ETL tasks.

[0078] In one embodiment, considering that the kafka component has the characteristics of a consumer group, multiple consumer groups can be set in the kafka component in this embodiment, and each consumer group corresponds to at least one data ETL task. It is understandable that since the data ETL task is divided into three sections, and the kafka component is set in the source cluster and the target cluster respectively, the consumer group may face different stages of the data ETL task. Therefore, in this embodiment, each consumer group is expressed as corresponding to at least one data ETL task. In this embodiment, the consumer group characteristics of the kafka component are fully utilized, and multiple consumer groups are used to consume a copy of the target data, which can solve the problem of outputting data to multiple target clusters at the same time. Compared with the related art in which a piece of data can only be used once, in this embodiment, business data only needs to be written once to output multiple business data in parallel. By shortening the input time and output time, the data exchange efficiency can be greatly improved.

[0079] In one embodiment, considering that different data clusters involve different business data, a data ETL task involving data from multiple clusters may cause data leakage. To ensure data security, in the process of generating data ETL tasks in this embodiment, the data processing system generates an audit task when a cross-cluster data ETL task is detected, and sends it to the target user with audit authority, that is, the target user corresponding to the audit task (such as operation and maintenance personnel audit, leadership audit, etc.). In one example, the audit task can describe the data source address, data type, data usage, and other information that needs to be accessed, and the above audit task can be displayed on the display device, with the effect as follows: Figure 4 As shown, the target user can determine whether to pass the audit task based on the applicant of the data ETL task and his / her permissions. If the applicant's permissions allow him / her to use cross-cluster data exchange or have a corresponding number of cluster data exchanges, the target user can pass (i.e. agree to) the above audit task. If the applicant's permissions prohibit him / her from using cross-cluster data exchange or having a corresponding number of cluster data exchanges, the target user can not pass (i.e. reject) the above audit task. The data processing system can detect the target user's operation, and when it is detected that the target user has passed the operation of the above audit task, it can respond to the operation. The data processing system can generate a data ETL task and configure the data ETL task to the data ETL component. In this way, this embodiment can avoid data leakage and ensure the security of cross-cluster data exchange by adding an audit mechanism.

[0080] In one embodiment, the Kafka component also provides multiple security authentication mechanisms. When the data ETL component accesses the Kafka component, it can provide identity information to prove its legitimacy, thereby ensuring the security of data reading and output. In this embodiment, the Kafka component's security permissions are pluggable. Based on the data characteristics of each cluster, corresponding authorization services can be set up to ensure that each cluster accesses data according to the corresponding permissions, thereby improving the security of data exchange. Furthermore, in this embodiment, the Kafka component can also be integrated with external authorization services to further enhance the security of data exchange.

[0081] In one embodiment, during the execution of a data ETL task, the data ETL component may generate task logs for each data ETL task, namely, a first-stage task log, a second-stage task log, and a third-stage task log. In response to a user's query operation, the data processing system may retrieve and display the task logs for the data ETL tasks, allowing the user to query the status information of each data ETL task at each stage (e.g., running, abnormal, error, completed, etc.), thereby improving user efficiency.

[0082] In one example, the data processing system can also obtain task information from the first data ETL component, the second data ETL component, and the third data ETL component and display the task information in the display device, the effect is as follows: Figure 5 As shown, the user can query the task information, system logs and / or locate abnormal tasks, and can also click to start, stop or edit to change the task status. When the maintenance personnel locate the abnormal task and solve the problem of the abnormal task, the data ETL task can be restored. In response to the operation of processing the abnormal task, the data processing system can restore the data ETL task corresponding to the abnormal task, that is, continue to execute the data ETL task from the stage corresponding to the abnormal location without re-executing the data ETL task. For example, when an abnormality occurs during data exchange in the third stage, since the data exchange in the second stage is normal, the data exchange task of the third stage can be re-executed at this time, without re-executing the data exchange tasks of the first and second stages, thereby shortening the time of the entire data ETL task and improving the efficiency of the data exchange task.

[0083] In one embodiment, considering the situation in the related art where data loss occurs when closing or pausing a data ETL task, the data loss problem is more serious, especially when the data access speed is fast. To this end, in this embodiment, when the data processing system detects an operation indicating closing or pausing a data ETL task, it controls the data ETL component corresponding to the data ETL task to pause reading data from the Kafka component and continue to output data. When the data output is completed, the data processing system can control the data ETL component corresponding to the data ETL task to close or suspend the data ETL task. The completion of data transmission can be determined by querying the log data of the data ETL task. For example, if the output speed of the Kafka output writer is 0, it is determined that the data transmission is complete.

[0084] Taking the second phase of data exchange as an example, upon detecting a control signal (indicating a paused task) sent by the data processing system, the second data ETL component can stop reading business data from the source cluster's Kafka component and continue outputting business data to the target cluster's Kafka component. The third data ETL component will not read the business data in the target cluster's Kafka component, effectively storing the target data in the target cluster's Kafka component, thereby pausing the data ETL task. When resuming the paused task, the third data ETL component can read the business data from the target cluster's Kafka component and store it in the target cluster's data source, thus completing the data exchange between the source and target clusters.

[0085] In practical applications, Kafka components can act as both output writers and input readers. Combining the three stages in this disclosure, the roles of Kafka components are shown in Table 1.

[0086] When the Kafka component acts as an input reader, for example, the Kafka of the target cluster in the third stage is an input, which performs a real-time ETL task that continuously receives data. Unlike other offline tasks that will end, Kafka will continuously supply data. If the task is closed directly, some data will be read by the ETL component but not written to the output writer and lost. In order to solve this problem, in this embodiment, an http service is added to the ETL component. The http service is used to receive external requests. The external request can be a stop request or a pause request. At this time, the input in the ETL component can pause reading data, and the output in the ETL component can continue to execute. When the data processing system detects a recovery request, the input in the ETL component can continue to read data and resume the data ETL task. In this way, the solution of pausing reading in this embodiment will not lose data when starting or closing a task, thereby ensuring data security.

[0087] Table 1 Kafka has different roles at different stages

[0088] stage Cluster Reader Writer Phase 1 Source cluster Data Source Kafka components Phase II Cross-cluster Kafka components Kafka components Phase 3 Target cluster Kafka components Data Source

[0089] See also Figure 2 and Figure 6 The following describes a process of exchanging data from an HDFS data source in cluster A to an HDFS data source in cluster B in conjunction with an embodiment.

[0090] The Hdfs data source is the location where historical data in clusters A and B is persisted. For data security, the Hdfs port is not directly open to the outside world, and copying data between the Hdfs of the two clusters (A and B) is not stable and time-consuming, so direct data synchronization is not appropriate. In this embodiment, the Hdfs data in cluster A is synchronized to Kafka, and the Kafka components of both clusters A and cluster B synchronize Hdfs data. The data ETL component in cluster B passes the Kafka data to the Hdfs data source. In cluster A, Kafka acts as a writer to collect Hdfs data; when synchronizing between clusters, the Kafka of cluster A acts as a reader, and the Kafka of cluster B acts as a writer to pass the data of cluster A to cluster B; in cluster B, Kafka acts as a reader to store the data in Hdfs, forming a link to complete the Hdfs data synchronization from cluster A to cluster B. All data ETL tasks are not single, and multiple parallel executions can be started at the same time to improve data synchronization efficiency.

[0091] See also Figure 6 The data ETL task starts. The Kafka component inputs reader initialization parameters and sets the parameter stopPoll, which indicates whether to pause data reading, to false. The HTTP service is started and waits for external requests. Before pulling data from Kafka, the main thread checks whether stopPoll is true. If true, reading stops; if false, the data is read and passed to the writer. When an external request arrives, if the request parameter is stop, stopPoll is set to true. The main thread stops reading, sleeps for 10 seconds (optional), and then continues to loop to check the value of stopPoll. If the request parameter is start, stopPoll is set to false, and the main thread can continue pulling data.

[0092] Based on a data processing system provided in an embodiment of the present disclosure, an embodiment of the present disclosure further provides a data processing method, including:

[0093] A data ETL task is configured for a data ETL component of the data cluster, so that the data ETL component executes the data ETL task and implements data exchange between at least one source cluster and at least one target cluster via the Kafka component.

[0094] It should be noted that the method shown in this embodiment matches the content of the system embodiment. You can refer to the content of the above system embodiment and will not repeat it here.

[0095] In an exemplary embodiment, a data processing system is also provided. Figure 7 , comprising at least one processing device and at least one storage device;

[0096] the at least one storage device for storing a computer program executable by the at least one processing device;

[0097] The at least one processor device is configured to execute the computer program in the at least one memory to implement the above-mentioned data processing method.

[0098] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function in accordance with the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a magnetic disk, a tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid-state drive (SSD)), etc.

[0099] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a processor, the computer executes the data processing method described in any of the above embodiments, such as one or more steps in the data processing method.

[0100] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0101] The processor mentioned in the embodiments of the present disclosure can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute the various exemplary logic blocks and modules described in conjunction with the present disclosure. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0102] In addition, the memory mentioned in the embodiments of the present disclosure may be a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM), or any other form of storage medium known in the art.

[0103] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0104] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing system, characterized in that: The system communicates with multiple data clusters provided with data ETL components and Kafka components; the system is used to configure data ETL tasks to the data ETL components of the data clusters, so that the data ETL components execute the data ETL tasks and implement data exchange between at least one source cluster and at least one target cluster via the Kafka components; The system is also used to: When an operation indicating closing or pausing a data ETL task is detected, the data ETL component corresponding to the data ETL task is controlled to pause reading data from the Kafka component and continue outputting data; When the data output is completed, the data ETL component corresponding to the data ETL task is controlled to close or pause the data ETL task.

2. The system according to claim 1, wherein: The source cluster includes at least one of the following: Mysql data cluster, PostgresSql data cluster, Clickhouse data cluster, Mongodb data cluster, HDFS data cluster, ElasticSearch data cluster and Kafka data cluster; The target cluster includes at least one of the following: Mysql data cluster, PostgresSql data cluster, Clickhouse data cluster, Mongodb data cluster, HDFS data cluster, ElasticSearch data cluster, Redis data cluster and Kafka data cluster.

3. The system according to claim 1, wherein: The system is used to configure a data ETL task to a data ETL component of a data cluster, so that the data ETL component executes the data ETL task and implements data exchange between at least one source cluster and at least one target cluster via a Kafka component, including: The first data ETL component corresponding to the data ETL task reads the target data required by the data ETL task from the source cluster and outputs it to the Kafka component in the source cluster; The second data ETL component corresponding to the data ETL task reads the target data from the Kafka component in the source cluster and outputs the target data to the Kafka component in the target cluster; The third data ETL component corresponding to the data ETL task reads the target data from the Kafka component in the target cluster and outputs the target data to the target cluster.

4. The system according to claim 3, characterized in that The system is also used to: Acquire task information from the first data ETL component, the second data ETL component, and the third data ETL component, and display the task information, wherein the task information is used to enable maintenance personnel to locate abnormal tasks; In response to the operation of processing the abnormal task, the data ETL task corresponding to the abnormal task is resumed.

5. The system according to claim 4, characterized in that The Kafka component performs persistence processing on the target data.

6. The system according to claim 1, wherein: Different business data in the source cluster is implemented using different topics.

7. The system according to claim 6, characterized in that The nodes of the Kafka component can be horizontally expanded as the number of topics increases.

8. The system according to claim 1, wherein: The Kafka component is set up with multiple consumer groups, and each consumer group corresponds to at least one data ETL task.

9. The system according to claim 1, wherein: The system is also used to: When a cross-cluster data ETL task is detected, an audit task is generated and sent to the target user corresponding to the audit task; In response to detecting that the target user has passed the operation of the review task, the data ETL task is generated and configured to the data ETL component.

10. The system according to claim 3, wherein: The system is also used to: The task log of the data ETL task is obtained and displayed, so that the user can query the status information of each data ETL task at each stage.

11. A data processing method, characterized in that: include: Configure a data ETL task to a data ETL component of the data cluster, so that the data ETL component executes the data ETL task and implements data exchange between at least one source cluster and at least one target cluster via the Kafka component; The method further comprises: When an operation indicating closing or pausing a data ETL task is detected, the data ETL component corresponding to the data ETL task is controlled to pause reading data from the Kafka component and continue outputting data; When the data output is completed, the data ETL component corresponding to the data ETL task is controlled to close or pause the data ETL task.

12. A data processing system, characterized in that: comprising at least one processing device and at least one storage device; the at least one storage device for storing a computer program executable by the at least one processing device; The at least one processor device is configured to execute the computer program in the at least one memory to implement the method according to claim 11.

13. A computer-readable storage medium, characterized in that When the executable computer program in the storage medium is executed by a processor, the method according to claim 11 can be implemented.

Images