Data integration method and system and electronic device

Abstract

The embodiment of the invention provides a data integration method and system and an electronic device. The method comprises the steps of providing job operation parameters of a to-be-configured data integration job; generating a job starting execution instruction of the data integration job according to the configured job operation parameters, and responding to the job starting execution instruction to schedule a corresponding computing cluster so as to operate the data integration job; monitoring the operation state of the data integration operation; and in response to the running state of the job exception and the started breakpoint resuming function, acquiring and analyzing data transmission information of the latest read data when the job exception occurs, and generating a job recovery execution instruction corresponding to the type of the source end data source based on an analysis result so as to recover running of the data integration job from the breakpoint. According to the technical scheme, breakpoint resume technical schemes of different types of data sources can be unified, corresponding processing strategies are provided for the different types of data sources, the types of the data sources executing the data integration operation are not limited, and better flexibility is provided.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a data integration method, system, and electronic device. Background Technology

[0002] Data integration is a key technology in the field of big data and a foundation for data governance. In recent years, with the surge in daily operational data volume and increasingly stringent real-time requirements of organizations, real-time data integration has become a simultaneous process of batch and stream data integration, leading to the emergence of batch-stream integrated data integration systems. However, these systems currently suffer from poor performance and fail to meet the needs of real-world applications. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a data integration method, the method comprising: providing job execution parameters for a data integration job to be configured, wherein the job execution parameters include an enableable breakpoint resume function, multiple types of source data sources and target data sources, data mapping parameters between the source data sources and target data sources, and job concurrency; generating a job start execution instruction for the data integration job according to the configured job execution parameters, and scheduling a corresponding computing cluster to run the data integration job in response to the job start execution instruction; monitoring the running status of the data integration job, wherein during the running of the data integration job, data transmission information of data read from the source data sources is stored, the data transmission information including timestamps, job concurrency identifiers, and data offsets; and in response to an abnormal running status of the job and the enabled breakpoint resume function, obtaining and parsing the data transmission information of the latest read data when the job is abnormal, and generating a job recovery execution instruction corresponding to the type of the source data source based on the parsing result to resume the running of the data integration job from the breakpoint.

[0004] For example, according to at least one embodiment of the data integration method of this disclosure, the method further includes: adjusting the participation of each working node in the data integration job according to a set rule based on the running status of each working node in the computing cluster.

[0005] For example, according to at least one embodiment of the data integration method of this disclosure, the method further includes: when adding a new working node to assist in the data integration operation, increasing the number of concurrent operations of the data integration operation accordingly based on the number of new nodes, and assigning a concurrent operation identifier to the new working node; pausing the operation of the target working node of the assisted operation, and setting the offset range of the source data source that the new working node and the target working node need to read according to the current data offset range of the source data source that the target working node needs to read; and starting the operation of the new working node and the target working node.

[0006] For example, according to at least one embodiment of the data integration method of this disclosure, the job operation parameters further include a maximum number of job exceptions and a maximum number of failure retries; when the running status of the data integration job is indicated as a job exception, the number of job exceptions is recorded and it is determined whether the maximum number of job exceptions has been reached; if the maximum number of job exceptions has been reached, it is further determined whether the number of failure retries has reached the maximum number of failure retries; if the maximum number of failure retries has not been reached, the operation of the data integration job is resumed from the breakpoint according to the breakpoint resume function; if the number of failure retries has reached the maximum number of failure retries, the operation of the data integration job is terminated.

[0007] For example, according to at least one embodiment of the data integration method of this disclosure, the job operation parameters further include the maximum amount of data allowed to be read per unit time; and the data reading job of the source data source is blocked by a preset algorithm based on the maximum amount of data allowed to be read per unit time.

[0008] For example, in the data integration method according to at least one embodiment of this disclosure, the process of parsing the data transmission information of the latest read data and generating a job recovery execution instruction corresponding to the type of the source data source based on the parsing result includes: parsing and obtaining the concurrency identifier of each job and the timestamp and data offset corresponding to each job concurrency identifier; in response to the type being a database type, generating a job recovery execution instruction based on the SQL statement before the data offset and timestamp, or in response to the type being a file system type, generating a job recovery execution instruction based on the file reading range before the data offset and timestamp, or in response to the type being a message queue type, generating a job recovery execution instruction based on the data offset.

[0009] At least one embodiment of this disclosure provides a data integration system, comprising: a job management module configured to provide job execution parameters for a data integration job to be configured, wherein the job execution parameters include enableable breakpoint resume, multiple types of source data sources and target data sources, data mapping parameters between source data sources and target data sources, and job concurrency; a job execution module configured to generate a job start execution instruction for the data integration job based on the configured job execution parameters, and to schedule a corresponding computing cluster to run the data integration job in response to the job start execution instruction; a job monitoring module configured to monitor the running status of the data integration job, and to store data transmission information of the data read from the source data source during the running of the data integration job, the data transmission information including timestamp, job concurrency identifier, and data offset; and a job exception handling module configured to, in response to the running status of the job exception and the enabled breakpoint resume function, obtain and parse the data transmission information of the latest read data when the job exception occurs, and generate a job recovery execution instruction corresponding to the type of source data source based on the parsing result to resume the running of the data integration job from the breakpoint.

[0010] For example, in the data integration system according to at least one embodiment of the present disclosure, the job execution module is further configured to: adjust the participation of each working node in the data integration job according to a set rule based on the running status of each working node in the computing cluster.

[0011] For example, in the data integration system according to at least one embodiment of the present disclosure, the job execution module is further configured to: when adding a new worker node to assist in the data integration job, increase the job concurrency of the data integration job accordingly based on the number of new nodes, and assign a job concurrency identifier to the new worker node; suspend the operation of the target worker node of the assisted job, and set the offset range of the source data source that the new worker node and the target worker node need to read according to the current data offset range of the source data source that the target worker node needs to read; and start the operation of the new worker node and the target worker node.

[0012] For example, in the data integration system according to at least one embodiment of this disclosure, the job operation parameters further include a maximum number of job anomalies and a maximum number of failure retries; the job anomaly handling module is further configured to record the number of job anomalies and determine whether the maximum number of job anomalies has been reached when the running status of the data integration job is indicated as a job anomaly; if the maximum number of job anomalies has been reached, then it is further determined whether the number of failure retries has reached the maximum number of failure retries; if the maximum number of failure retries has not been reached, then the operation of the data integration job is resumed from the breakpoint according to the breakpoint resume function; if the number of failure retries has reached the maximum number of failure retries, then the operation of the data integration job is terminated.

[0013] For example, in a data integration system according to at least one embodiment of the present disclosure, the job operation parameters further include the maximum amount of data allowed to be read per unit time; the job operation module is further configured to perform a blocking operation on the data reading job of the source data source using a preset algorithm and according to the maximum amount of data allowed to be read per unit time.

[0014] For example, in a data integration system according to at least one embodiment of this disclosure, the job exception handling module is further configured to parse and obtain the concurrency identifier of each job and the timestamp and data offset corresponding to each job concurrency identifier; in response to a database type, generate a job recovery execution instruction based on the SQL statement before the data offset and timestamp; or in response to a file system type, generate a job recovery execution instruction based on the file reading range before the data offset and timestamp; or in response to a message queue type, generate a job recovery execution instruction based on the data offset.

[0015] At least one embodiment of this disclosure provides an electronic device, including: one or more processors; a memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the one or more processors to implement a data integration method provided according to at least one embodiment of this disclosure.

[0016] At least one embodiment of this disclosure provides a computer-readable medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by one or more processors, implement a data integration method according to at least one embodiment of this disclosure.

[0017] The batch-stream integrated data integration solution of this disclosure can resume the data integration job from the point of data transmission interruption if a job anomaly occurs during the data integration process, unlike related technologies that require restarting the entire transmission process, thus improving job efficiency. Furthermore, the solution of this disclosure unifies the breakpoint resumption technology for different types of data sources, providing corresponding processing strategies for different data source types, without restricting the types of data sources from which the data integration job can be performed, thus offering greater flexibility. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments of this disclosure will be briefly described below. Clearly, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit the scope of this disclosure.

[0019] Figure 1 A block diagram of a batch-stream integrated data integration system according to at least one embodiment of the present disclosure is shown;

[0020] Figure 2 A flowchart of a batch-stream integrated data integration method according to at least one embodiment of the present disclosure is shown;

[0021] Figure 3 A flowchart illustrating an exemplary batch-stream integrated data integration method according to at least one embodiment of the present disclosure is shown;

[0022] Figure 4 A schematic diagram of an electronic device according to at least one embodiment of the present disclosure is shown;

[0023] Figure 5 A schematic diagram of a computer-readable medium according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0025] Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, which may change accordingly when the absolute position of the described objects changes.

[0026] Note that the examples described below are merely specific examples and are not intended to limit the embodiments of this disclosure to the specific shapes, hardware, connections, operations, values, conditions, data, sequences, etc., shown and described. Those skilled in the art can utilize the concepts of this disclosure to construct further embodiments not mentioned herein by reading this specification.

[0027] The terminology used in this disclosure is that which is currently widely used in the art in consideration of the functionality of this disclosure; however, these terms may vary depending on the intent, precedent, or new technology of those skilled in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of this disclosure. Therefore, the terminology used in this specification should not be construed as simple names, but rather based on the meaning of the terms and the overall description of this disclosure.

[0028] This disclosure uses flowcharts to illustrate the operations performed by a system according to embodiments of this disclosure. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously, as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0029] First, the abbreviations and related terms involved in this disclosure are defined and explained.

[0030] Batch data integration refers to the process of collecting data from a source data source in a one-time or periodic manner. The data in the source data source is a finite or bounded dataset. The collected data is processed, transformed, and then written into the destination data source.

[0031] Streaming data integration refers to the continuous collection of data from a source data source, where the data is an infinite or unbounded dataset. The collected data is then processed, transformed, and finally written into the destination data source.

[0032] Batch and streaming data integration refers to a platform or architecture that uses the same set of code or tools to simultaneously support batch data integration and streaming data integration.

[0033] HDFS stands for Hadoop Distributed File System. HDFS is highly fault-tolerant, suitable for deployment on inexpensive machines, and has high throughput, making it very suitable for applications on large-scale datasets.

[0034] OSS stands for Object Storage Service. It is a massive, secure, low-cost, and highly reliable cloud storage service suitable for storing any type of file, and features elastic scaling of capacity and processing power.

[0035] POD: The smallest unit that can be created and managed in the Kubernetes system. It is the smallest resource object model created or deployed by the user in the resource object model. It is also the resource object for running containerized applications on Kubernetes. Other resource objects are used to support or extend the functionality of the POD object. A POD is composed of one or more containers.

[0036] Currently, when a data integration job terminates abnormally, some batch and stream integrated data integration systems can only rerun the data integration job from scratch, causing repetitive data transmission and increased processing time. Even if some systems have fault-tolerance mechanisms, they only recover based on pre-stored job states when job anomalies or failures occur, but this still results in some data retransmission and poor performance. Moreover, related technologies only support job recovery for single-type data sources and do not provide solutions for job recovery for multiple data source types.

[0037] At least one embodiment of this disclosure provides a data integration method, system, electronic device, and storage medium that can achieve a unified job recovery solution for different types of data sources when data integration operations are abnormal. That is, there are corresponding processing strategies for different types of data sources, and there will be no duplicate data transmission, thereby enabling efficient data integration operations between multiple heterogeneous data sources.

[0038] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same element symbols are used in the drawings and description to denote the same or similar parts.

[0039] Figure 1 A block diagram of a batch-stream integrated data integration system 100 according to at least one embodiment of the present disclosure is shown. Figure 1 As shown, in addition to the data integration system 100, it also includes source data sources 10 of various data types from various physical locations. For example, source data sources 10 may include different types of data sources such as databases 11, file systems 12, and message queues 13. Databases 11 may include relational databases such as MySQL, OceanBase, Doris, distributed databases such as HBase, and Hive; file systems 12 may include files such as HDFS, FTP, and SFTP; message queues 13 may include queues such as Kafka and Pulsar. HDFS or SFTP file systems may include files created or used by applications such as Microsoft Outlook, Microsoft Word, Microsoft Excel, and Microsoft Access, as well as files in standard formats such as ASCII, CSV, GIF, TIF, PNG, and PDF. Source data sources 10 may come from different locations or be centralized. The data provided by source data sources 10 may have various forms and formats, and may be compatible or incompatible with each other.

[0040] For example, data integration system 100 can read data from source data source 10 as the source data result obtained by data integration system 100. Data integration system 100 can send commands to one or more source data sources 10, so that the source data sources 10 provide data to data integration system 100. Since the received data may be in multiple formats, including different metadata, data integration system 100 can reconfigure the received data so that it can be combined later for integration processing. The functions that data integration system 104 can perform will be described in detail below.

[0041] It should be noted that the modules of the data integration system 100 can perform their respective operations in parallel, serial, or combined manner to optimize the operating performance of the system 100.

[0042] like Figure 1 As shown, the data integration system 100 may include an access control module 101, a data source management module 102, a job scheduling module 103, a job management module 104, a job monitoring module 106, a job exception handling module 108, and a job execution module 110.

[0043] The access control module 101 is primarily responsible for managing access permissions for users, data sources, and data integration jobs. This module 101 divides user permissions into three roles: administrator, developer, and operations and maintenance personnel. Administrators are responsible for managing the entire data integration system, including personnel, data sources, and data integration jobs; developers are responsible for creating data sources and managing data integration jobs, with each developer only able to manage the data sources and data integration jobs they create; and operations and maintenance personnel are responsible for the daily operation and monitoring of data integration jobs.

[0044] Data source management module 102 is configured to manage data related to source data source 10 and destination data source 10. Generally, data source management module 102 provides data exchange, integration, management, and analysis across all tools in the data integration environment. For example, data source management module 102 may also provide data context analysis tools and data structure change impact analysis tools. Data source management module 102 can also be used to prepare a business data glossary containing data definitions, algorithms, and business context for the data in data integration system 100, which can be published for use throughout the enterprise.

[0045] The job management module 103 is responsible for managing data integration jobs. In one embodiment of this disclosure, the job management module 103 is configured to provide job execution parameters for the data integration job to be configured. These parameters may include an enableable breakpoint resume function, selectable multiple types of source and target data sources, data mapping parameters between the source and target data sources, and job concurrency. In addition to the above, configurable job execution parameters may also include the maximum allowed number of retries, the maximum allowed number of error records, the maximum number of data entries allowed to be read per unit time, the storage location of data offsets, scheduling period, and other parameter information. These parameters can be provided and configured according to the needs of specific application scenarios, and this embodiment of the disclosure does not impose any limitations on this.

[0046] It should be noted that the breakpoint resume function includes breakpoint resume processing logic for different types of data sources. Users can choose to enable or disable this function when configuring runtime parameters. When enabled, if an exception occurs during subsequent data integration job processing, this breakpoint resume processing logic can perform corresponding job recovery processing based on the breakpoint resume processing logic for different data source types. It can retrieve the data transmission information of the most recently read data based on the exception job response instruction, and generate a job recovery instruction based on the data transmission information to implement the breakpoint resume function. The specific breakpoint resume processing logic will be described in detail later. The source data source is generally the starting point of data integration, i.e., the original source of the data. It can be a database, file system, message queue, etc., as mentioned above. The target data source is generally the ending point of data integration, i.e., the place where the data will be stored or processed. It can also be a database, file system, message queue, etc., as mentioned above. During the data integration process, it is necessary to pre-establish the correspondence between data elements (such as tables, fields, etc.) between the source data source and the target data source, including mappings between field types and field names. The mapping parameters can be selected from the provided parameter options or manually entered; no restrictions are placed here. This disclosure does not limit the type of data source, and batch data integration and streaming data integration can be performed when performing data integration operations.

[0047] The job scheduling module 104 mainly refers to the scheduling of batch data integration jobs. Based on the configuration of the data integration jobs, it schedules jobs related to batch data either once or periodically. Specifically, it obtains job information (including job data source, job configuration, scheduling time, etc.) from the job management module 103, runs the jobs once or periodically, and synchronizes the job running status to the job monitoring module 106.

[0048] The job execution module 110 is configured to generate job startup execution instructions for the data integration job based on the configured job execution parameters, and to schedule the corresponding computing cluster to run the data integration job in response to the job startup execution instructions. The job startup execution instructions may include the number of minimum deployment units (e.g., PODs) to be started, the amount of memory and CPU used by each minimum deployment unit, the image used to start the minimum deployment unit, and the job configuration passed to the minimum deployment unit, etc. Figure 1As shown, the job execution module 110 may include a computing engine 110A, which can be a platform for performing batch and stream data integration, such as the Spark and / or Flink data processing platform. The computing engine 110 creates a computing cluster in response to a job start execution command, after which worker nodes in the cluster collaboratively run the data integration job. For example, Flink's pipeline runtime system can execute batch and stream processing programs. Flink consists of one or two master nodes and several worker nodes. The master node is responsible for managing jobs, worker nodes, and resources, while the worker nodes are responsible for running the specific jobs.

[0049] The job monitoring module 106 is primarily responsible for monitoring the operation status, progress, and display of operation logs of the data integration job. For example, in one embodiment, the job monitoring module 106 is configured to monitor the running status of the data integration job. During the operation of the data integration job, it stores the data transmission information of the data read from the source data source, including timestamps, job concurrency identifiers, and data offsets. Figure 1 As shown, the job monitoring module 106 includes a storage system 106A, which can be, for example, HDFS or OSS. During the execution of the data integration job, it stores data transmission information in real time or periodically. For different types of data sources, the data transmission information is stored in the storage system 106A in a unified format. When a job encounters an anomaly, the data offset corresponding to the latest timestamp of the job that encountered the anomaly can be easily obtained and parsed. Based on this information, the breakpoint resume function can be implemented. In this embodiment, the data offset generally refers to the number of bytes successfully transmitted during data transmission or a specific position in a file. This offset is used to identify from which position data transmission should continue after the connection with the data source is restored.

[0050] The job exception handling module 108 is configured to respond to the running status of job exceptions and the enabled breakpoint resume function, obtain and parse the data transmission information of the latest read data when the job exception occurs, and generate a job recovery execution instruction corresponding to the type of the source data source based on the parsing result to resume the operation of the data integration job from the breakpoint.

[0051] As mentioned above, the data job recovery mechanisms provided by related technologies have many problems. To address these issues, this disclosure presents a data job recovery logic for handling job anomalies. During the design process, it was considered that the data source types for batch and stream integrated data integration jobs are quite diverse and complex. Supporting only a single data source, such as only supporting data job recovery from relational databases or file systems, would significantly limit the data source types in data integration job processing and reduce user experience. Therefore, in addition to providing a new data job recovery mechanism, this disclosure also provides specific processing procedures for different types of data sources.

[0052] For example, in some embodiments of this disclosure, the job exception handling module 108 is further configured to parse data transmission information to obtain the concurrency identifier of each job and the timestamp and data offset corresponding to each job concurrency identifier; in response to a database type, to generate a job recovery execution instruction based on the SQL statement before the data offset and timestamp; or in response to a file system type, to generate a job recovery execution instruction based on the file reading range before the data offset and timestamp; or in response to a message queue type, to generate a job recovery execution instruction based on the data offset.

[0053] Specifically, the data integration system 100 in this embodiment supports breakpoint resumption of various data sources, including relational databases and non-relational databases (such as MySQL, OceanBase, Doris, HBase, Hive, etc.), file systems (HDFS, FTP, SFTP, etc.), and message queues (Kafka, Pulsar), and supports concurrent data reading from the data sources. For example, the job monitoring module 106 periodically persists the timestamp, job concurrency number (an example of a job concurrency identifier; assuming a job concurrency of 3, the concurrency number can be set to 0, 1, 2, etc.), and the data offset of the concurrently read data source to an external storage system (such as a distributed file system). When a job fails, the job exception handling module 108 resumes the data integration job by reading and parsing the latest data offset of the saved timestamp. The specific processing logic for resuming the job from the breakpoint for different types of data sources (information indicating the data source type is reflected in the job running parameters, job startup execution instructions, and subsequent computing cluster operation) is as follows:

[0054] 1. Relational databases (MySQL, OceanBase, Doris, etc.): Parse the latest offset (e.g., represented by DB_offset_B) corresponding to each concurrent job number. Combine the SQL statement of each concurrent job before the job failure (generated before the timestamp) with the latest offset DB_offset_B to generate a new SQL statement. If the WHERE condition of the SQL statement before the concurrent task failed is within the range [DB_offset_A, DB_offset_C), combine the latest offset with the SQL statement before the concurrent job failed to generate a new SQL statement. The WHERE condition of the new SQL statement is within the range [DB_offset_B, DB_offset_C). The concurrent task executes the new SQL statement to achieve breakpoint resumption.

[0055] 2. Non-relational databases (HBase, Hive, etc.): The processing logic can be referenced from relational databases. Specifically, Hive is a Hadoop-based data warehouse that uses HDFS for data storage and translates SQL statements into MapReduce jobs. It is specifically designed for processing and analyzing large-scale datasets. Since Hive supports standard SQL statements, it can be processed similarly to relational databases. HBase is a distributed, column-oriented NoSQL database that runs on top of HDFS. While HBase does not support standard SQL, it provides HBase Query Language (HQL), a SQL-like query language used for querying, updating, and deleting data in HBase. Therefore, it can also be processed like a relational database.

[0056] 3. File System (HDFS, FTP, SFTP, etc.): Parse the latest offset (e.g., represented by FS_offset_B) corresponding to each concurrent job number, and combine the file read range of each concurrent job before the job failure with the latest offset FS_offset_B to generate a new file read range. If the file read range before the concurrent task failed was within the range [FS_offset_A, FS_offset_C), combine the latest offset with the file read range before the concurrent job failed to generate a new file read range within the range [FS_offset_B, FS_offset_C). The concurrent task reads data within the range [FS_offset_B, FS_offset_C), enabling breakpoint resumption.

[0057] 4. Message Queues (Kafka, Pulsar, etc.): Data integration in message queues is generally streaming data integration, so the data range only has a beginning and no end. Multiple concurrent messages in a message queue belong to a single consumer group, so it is only necessary to parse and obtain the latest offset (e.g., represented by MQ_offset_B). Multiple concurrent tasks start consuming data from the message queue from this offset MQ_offset_B, achieving breakpoint resumption.

[0058] The above describes the implementation schemes for breakpoint resume functionality for four types of data sources: relational databases, non-relational databases, file systems, and message queues. For different types of data sources, data transmission information can be obtained by combining a timestamp, job concurrency number, and the data offset of the concurrently accessed data source. This information is then uniformly stored in an external storage system (such as a distributed file system). This unified storage format, combined with pre-defined breakpoint resume handling logic for different data sources, facilitates system maintenance and allows for the addition of new data sources in a plug-in manner. "Plug-in" here means that by simply storing the data in a unified format (timestamp, job concurrency number, and data offset of the concurrently accessed data source) in the distributed file system, multiple data source types can be supported, enabling breakpoint resume functionality when anomalies occur during data integration operations.

[0059] Moreover, the relevant technical solutions have zero tolerance or insensitivity to abnormal data, which cannot meet the actual use cases. When an anomaly occurs, the processing is terminated directly. This may lead to data loss, business interruption and other situations in some non-substantive problems (network jitter, data source disconnection or incorrect connection password), thus reducing the user experience.

[0060] In view of the above problems, for example, in some embodiments of this disclosure, the job operation parameters also include a maximum number of job anomalies and a maximum number of failure retries. Accordingly, the job anomaly handling module 108 is further configured to record the number of job anomalies and determine whether the maximum number of job anomalies has been reached when the running status of the data integration job is indicated as a job anomaly; if the maximum number of job anomalies has been reached, it further determines whether the number of failure retries has reached the maximum number of failure retries; if the maximum number of failure retries has not been reached, it resumes the running of the data integration job from the breakpoint according to the breakpoint resume function; if the running fails, it records the number of failure retries; if the number of failure retries reaches the maximum number of failure retries, it terminates the running of the data integration job. Specifically, if it is determined that the number of job anomalies has reached the maximum number of job anomalies, the running of this data integration job is terminated. This is to indicate that the solution of the embodiments of this disclosure has tolerance for abnormal data; only when the maximum tolerable number of job anomalies is reached is it necessary to terminate the running for subsequent processing. Then, it is determined whether the maximum number of retry attempts has been reached. If not, the interrupted resume job can be restored. In this way, by automatically retrying failed jobs, the system can more robustly cope with temporary network fluctuations, service unavailability and other issues, and also reduce the risk of the entire system crashing due to a single failure, thus improving the user experience.

[0061] Furthermore, existing technical solutions may impact the data source when the data integration operation traffic is too high, and there are currently no corresponding solutions. In view of the above problems, for example, in some embodiments of this disclosure, the job operation parameters also include the maximum amount of data allowed to be read per unit time; the job operation module 110 is also configured to use a preset algorithm and, based on the maximum amount of data allowed to be read per unit time, to block the data reading operation of the source data source. The preset algorithm can be a token bucket algorithm, which limits the data reading operation based on the token bucket algorithm and the configured maximum amount of data allowed to be read per unit time.

[0062] In addition, current data integration solutions start a cluster when running a data integration job and only release the cluster's resources after the job is completed. This makes it impossible to dynamically adjust resources, which can easily lead to low resource utilization or low data integration efficiency.

[0063] For example, in some embodiments of this disclosure, the job execution module 110 is further configured to adjust the participation of each work node in the data integration job according to the running status of each work node in the computing cluster and a set rule. The set rule could be, for example, a preset threshold for the resource utilization (CPU or memory utilization) of a work node; for instance, if the CPU or memory utilization reaches 90%, a new work node is started. The module also monitors the working status of each node, releasing it from the overall data integration job if it completes its assigned task, thus preventing resources from being occupied by a single data integration job for an extended period.

[0064] For example, in some embodiments of this disclosure, the job execution module 110 is further configured to: when adding new worker nodes to assist in data integration jobs, increase the job concurrency of the data integration job accordingly based on the number of new nodes, and assign a job concurrency identifier to the new worker node; pause the execution of the target worker node of the assisted job, and set the offset range of the source data source to be read by the new worker node and the target worker node according to the current data offset range of the source data source to be read by the target worker node; and start the execution of the new worker node and the target worker node. The target worker node of the assisted job may be a node whose resource utilization has reached a set threshold. Since its operating status may affect the system's performance and stability, a new worker node needs to be added to assist the target worker node in executing concurrent jobs. After determining to add a new worker node, it is necessary to increase the concurrency and assign a job concurrency identifier to the new worker node, and reconfigure the data offset range of the source data source to be read for both the target worker node and the new worker node. In this way, the tasks of the newly added worker node are planned, job execution efficiency is improved, and a basis is provided for implementing the breakpoint resume function for the newly added worker node.

[0065] For example, suppose POD1 is a POD whose resource utilization has reached a set threshold, and POD2 is a newly started worker node. Before starting POD2, the data already read by POD1 is saved, and the current offset of the data source to be read is [X, Y). Then, the process of starting the new POD is as follows: First, stop POD1, dynamically increase the concurrent number of jobs running simultaneously by 1 based on the user configuration, and set the offsets of the data source to be read by POD1 and POD2 respectively. POD1 and POD2 are then launched, and POD1 and POD2 periodically persist the concurrency number and the data offset of this concurrent read data source to the external storage system. It is easy to understand that the above is merely an example and does not impose restrictions on the number of new worker nodes launched or the method of allocating offset ranges.

[0066] For example, in some examples disclosed herein, the job management module 103 also provides parameters for the number of concurrent jobs and the maximum number of error records to be configured, and recommends the number of concurrent jobs and the maximum number of error records to the user based on different types of data sources. The user can adjust these values ​​according to the recommendations and actual needs. The specific calculation method used to recommend the number of concurrent jobs and the maximum number of error records to the user is as follows:

[0067] 1. Relational databases (MySQL, OceanBase, Doris, etc.): Obtain the total number of data rows (denoted by T) to be integrated by connecting to database 11 of data source 10. Calculate the recommended concurrency and maximum number of erroneous records using the following expressions:

[0068] Recommended concurrency level = T / 1 million; Recommended maximum number of erroneous records = T*0.0001;

[0069] 2. Non-relational databases (HBase, etc.): are processed as relational databases.

[0070] 3. File system (HDFS, FTP, SFTP, etc.): Obtain the total amount of data to be integrated (denoted by S, in MB) by connecting to the file system 12 of data source 10, and evaluate the size of a record (denoted by L, in MB) by reading a line of the file. Then, calculate the recommended values ​​for concurrency and maximum number of erroneous records according to the following expressions:

[0071] Total number of records T = M / L; Recommended concurrency = T / 1 million; Recommended maximum number of erroneous records = T * 0.0001;

[0072] 4. Message queues (Kafka, Pulsar, etc.): Message queues are streaming data sources, constantly generated, making it impossible to evaluate them based on the total amount of data to be integrated. In this embodiment, the recommended value can be calculated using the following expression:

[0073] Recommended concurrency count = number of topics * number of partitions per topic; maximum number of error records is 0 by default, but users can adjust this setting according to their needs.

[0074] Figure 2 A flowchart of a batch-stream integrated data integration method 200 according to at least one embodiment of the present disclosure is shown. It will be understood that the data integration process 200 described below can be performed by combining the source data source 10, the data integration system 100, and other possible components. Reference is made below. Figure 2 This will illustrate the various steps included in this method 200.

[0075] In step S210, the job execution parameters of the data integration job to be configured are provided, including the enableable breakpoint resume function, multiple types of source data sources and target data sources, data mapping parameters between source data sources and target data sources, and job concurrency.

[0076] In step S230, a job start execution instruction for the data integration job is generated based on the configured job running parameters, and the corresponding computing cluster is scheduled to run the data integration job in response to the job start execution instruction.

[0077] In step S250, the running status of the data integration job is monitored. During the running of the data integration job, the data transmission information of the data read from the source data source is stored. The data transmission information includes timestamp, job concurrency identifier and data offset.

[0078] In step S270, in response to the abnormal running status of the job and the enabled breakpoint resume function, the data transmission information of the latest read data at the time of the job abnormality is obtained and parsed, and a job recovery execution instruction corresponding to the type of the source data source is generated based on the parsing result to resume the running of the data integration job from the breakpoint.

[0079] For example, in some embodiments of this disclosure, the participation of each working node in the data integration job is adjusted according to a set rule based on the running status of each working node in the computing cluster.

[0080] For example, in some embodiments of this disclosure, the job operation parameters also include the maximum number of job exceptions and the maximum number of failure retries; when the running status of the data integration job is indicated as job exception, the number of job exceptions is recorded and it is determined whether the maximum number of job exceptions has been reached; if the maximum number of job exceptions has been reached, it is further determined whether the number of failure retries has reached the maximum number of failure retries; if the maximum number of failure retries has not been reached, the data integration job is resumed from the breakpoint according to the breakpoint resume function; if the operation fails, the number of failure retries is recorded; if the number of failure retries reaches the maximum number of failure retries, the operation of the data integration job is terminated.

[0081] For example, in some embodiments of this disclosure, the job operation parameters also include the maximum amount of data allowed to be read per unit time; and the data reading job of the source data source is blocked by a preset algorithm based on the maximum amount of data allowed to be read per unit time.

[0082] For example, in some embodiments of this disclosure, the process of parsing the data transmission information of the latest read data and generating a job recovery execution instruction corresponding to the type of the source data source based on the parsing result includes: parsing and obtaining the concurrency identifier of each job and the timestamp and data offset corresponding to each job concurrency identifier; in response to the type being database, generating a job recovery execution instruction based on the SQL statement before the data offset and timestamp; or in response to the type being file system, generating a job recovery execution instruction based on the file reading range before the data offset and timestamp; or in response to the type being message queue, generating a job recovery execution instruction based on the data offset.

[0083] However, Figure 2 Each step has been explained in detail above and will not be repeated here. It is worth noting that... Figure 2 Each step can be implemented as multiple pieces of code or circuits; this application does not impose any restrictions. Furthermore, Figure 2 The method can be used in conjunction with the above examples and embodiments, or it can be used alone; this case does not impose any restrictions.

[0084] Figure 3 A flowchart illustrating an exemplary batch-stream integrated data integration method according to at least one embodiment of the present disclosure is shown. The data integration method of the present disclosure embodiments is described in detail below using the specific scenario of reading a Kafka message queue and writing it to a Doris database.

[0085] Step S301: Select Kafka message queue as the source data source and Doris database as the target data source, and configure the field mapping between Kafka and Doris, including the mapping between field types and field names.

[0086] Step S303: Configure the number of concurrent jobs, the maximum number of allowed retries on failure, and whether to resume interrupted downloads.

[0087] Step S305: Configure parameters such as the maximum number of allowed error records and the maximum number of records allowed to be read per unit time.

[0088] Step S307: Determine whether the source data source is a streaming data source. If the source data source is a non-streaming data source (such as MySQL), you need to configure periodic scheduling parameters (step S325) or perform a one-time scheduling.

[0089] Step S309: Determine if breakpoint resume is configured. If it is, obtain the latest offset of the Kafka data source (S311); otherwise, proceed to step S313. Since this is a newly configured job, the latest offset is 0.

[0090] Step S313: Based on the previously configured job execution parameters, generate the job start command to run the job.

[0091] Step S315: Create a computing cluster based on the Flink computing engine and run the job; monitor the running status of the job through the Flink cluster.

[0092] During job execution, the latest data offset read from Kafka is periodically updated to the HDFS / OSS file system. When the job encounters an error and needs to resume from where it left off, the latest offset is retrieved from HDFS / OSS, and the job is rerun based on the position of the latest offset to achieve resume from where it left off.

[0093] If an error occurs during the parsing of data read from Kafka, the number of erroneous records is incremented by 1. When the maximum allowed number of erroneous records is reached, the job is terminated.

[0094] Rate limiting is implemented based on the token bucket algorithm. Specifically, assuming the maximum number of records that can be read from Kafka per unit time is M, the system will add tokens to the bucket at a constant rate of M per second. When reading from Kafka, a token must first be obtained from the bucket. If there are no tokens available in the bucket, the service reading from Kafka will be blocked until a token is obtained.

[0095] Dynamic scaling is achieved by dynamically increasing or decreasing the number of PODs running worker nodes. If a worker node in the Flink cluster completes its assigned task, it is removed from the registration list, the corresponding POD is terminated, and the occupied resources are released. If the resource utilization of a POD corresponding to a worker node in the Flink cluster reaches a certain value (e.g., CPU or memory utilization reaches 90%), a new POD is started to run the job, and the new worker node participates in running the job, improving job execution efficiency.

[0096] Step S317: Periodically monitor the running status of the job and record the number of error data records. If the number of error data records reaches the maximum allowed number of error records or the job fails, determine whether the number of failure retries has reached the maximum allowed number of failure retries. If the maximum number of failure retries has been reached, terminate the job; otherwise, rerun the job. When rerunning the job, if breakpoint resume is configured, start running from step S311; otherwise, start running from step S313.

[0097] The above solution periodically records the offset of the data source during the data integration job. When an anomaly occurs, the job resumes from the breakpoint, saving time spent rerunning the data integration job and avoiding duplicate data readings. Based on the resource usage of the running data integration job, resources are dynamically released or increased to improve resource utilization and operational efficiency. Rate limiting is supported to reduce the impact of excessive data integration traffic on the production data source. Error data logging is also supported; when a certain number of data entries fail to meet the criteria, the data integration job is marked as failed and terminated, meeting various scenarios where data integration jobs have zero tolerance for errors, ranging from zero to unlimited.

[0098] It should be noted that the above application scenarios are merely exemplary, intended to describe one or more aspects of this disclosure in specific scenarios. However, these aspects are not essential, and various modifications can be made to the application scenario. It is readily understood that the embodiments of this disclosure are not intended to limit the scope of the application.

[0099] At least some embodiments of this disclosure also provide an electronic device. Figure 4 A schematic diagram of an electronic device 400 according to at least one embodiment of the present disclosure is shown.

[0100] like Figure 4 As shown, the electronic device 400 includes one or more processors 410 and a memory 420. The memory 420 includes one or more computer program modules 421. The one or more computer program modules 421 are stored in the memory 420 and configured to be executed by the processor 410. These computer program modules 421 include instructions for executing the data integration method 200 and its additional aspects according to at least one embodiment of the present disclosure. When executed by the processor 410, they can perform one or more steps of the data integration method 200 and its additional aspects according to at least one embodiment of the present disclosure. The memory 420 and the processor 410 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, the bus may be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0101] For example, processor 410 may be a central processing unit (CPU), a digital signal processor (DSP), or other processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA). Processor 410 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 400 to perform desired functions.

[0102] Exemplarily, memory 420 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules 421 may be stored on the computer-readable storage medium, and processor 410 may run one or more computer program modules 421 to implement various functions of electronic device 400. The computer program modules include multiple computer-executable instructions. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0103] For example, electronic device 400 may also include input devices such as touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices such as liquid crystal displays, speakers, and vibrators; storage devices such as magnetic tapes and hard disks (HDDs or SDDs); and communication devices such as network interface cards like LAN cards and modems. The communication devices allow electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. A drive is connected to the I / O interface as needed. Removable storage media, such as disks, optical disks, magneto-optical disks, and semiconductor memories, are installed on the drive as needed so that computer programs read from them can be installed into the storage device as required.

[0104] For example, the electronic device 400 may further include a peripheral interface (not shown in the figure). This peripheral interface can be of various types, such as a USB interface, a Lightning interface, etc. The communication device can communicate wirelessly with networks and other devices, such as the Internet, intranets and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0105] The electronic device 400 may be, for example, a system-on-a-chip (SOC) or a device including the SOC. For instance, it can be any device such as a mobile phone, tablet computer, laptop computer, e-reader, game console, television, digital photo frame, navigator, home appliance, communication base station, industrial controller, server, etc., or any combination of data processing devices and hardware. The embodiments of this disclosure do not limit this. The specific functions and technical effects of the electronic device 400 can be found in the foregoing description of the data integration method 200 and its additional aspects according to at least one embodiment of this disclosure, and will not be repeated here.

[0106] Figure 5 A schematic diagram of a computer-readable medium 500 according to at least one embodiment of the present disclosure is shown.

[0107] like Figure 5 As shown, computer-readable medium 500 stores computer instructions 510, which, when executed by a processor, perform one or more steps of the data integration method 200 and its additional aspects as described above.

[0108] For example, when the program code is read by a computer, the computer can execute the program code stored in the computer storage medium to perform one or more steps to implement, for example, the data integration method 200 and its additional aspects according to at least one embodiment of the present disclosure.

[0109] For example, the computer-readable medium may include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, and other computer-readable media or any combination thereof.

[0110] At least some of the embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0111] It should be noted that, in this document, relational terms such as "first," "second," etc., are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0112] The following points should be noted regarding this disclosure:

[0113] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0114] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0115] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit the scope of protection of this disclosure, which is determined by the appended claims.

Claims

1. A data integration method, the method comprising: Provide the job execution parameters for the data integration job to be configured, wherein the job execution parameters include an enableable breakpoint resume function, multiple types of source data sources and target data sources, data mapping parameters between the source data sources and the target data sources, and job concurrency. Based on the configured job execution parameters, a job start execution instruction for the data integration job is generated, and in response to the job start execution instruction, the corresponding computing cluster is scheduled to run the data integration job. Monitor the running status of the data integration job, wherein, during the running of the data integration job, the data transmission information of the data read from the source data source is stored, and the data transmission information includes timestamp, job concurrency identifier and data offset; In response to the abnormal running status of the job and the enabled breakpoint resume function, the data transmission information of the latest read data at the time of the job abnormality is obtained and parsed. Based on the parsing result, a job recovery execution instruction corresponding to the type of the source data source is generated to resume the operation of the data integration job from the breakpoint.

2. The data integration method of claim 1, wherein, The method also includes: Based on the running status of each working node in the computing cluster, the participation of each working node in the data integration job is adjusted according to the set rules.

3. The data integration method according to claim 2, wherein, The method also includes: When adding new worker nodes to assist in the data integration operation, the concurrent number of the data integration operation is increased accordingly based on the number of new nodes, and a concurrent operation identifier is assigned to the new worker nodes. Pause the operation of the target worker node being assisted, and set the offset range of the source data source that the new worker node and the target worker node need to read based on the current data offset range that the target worker node needs to read. Start the operation of the new working node and the target working node.

4. The data integration method according to claim 1, wherein, The job operation parameters also include the maximum number of job exceptions and the maximum number of failure retries; When the running status of the data integration job is indicated as job abnormal, the number of job abnormalities is recorded and it is determined whether the maximum number of job abnormalities has been reached. If the maximum number of job exceptions is reached, it is further determined whether the number of failure retries has reached the maximum number of failure retries. If the maximum number of failure retries has not been reached, the data integration job is resumed from the breakpoint according to the breakpoint resume function. If the maximum number of failure retries is reached, the data integration job is terminated.

5. The data integration method according to claim 1, wherein, The job operation parameters also include the maximum amount of data that can be read per unit time. A preset algorithm is used to block the data reading operation of the source data source based on the maximum amount of data allowed to be read per unit time.

6. The data integration method according to any one of claims 1 to 5, wherein, The process of parsing the data transmission information of the latest read data and generating a job recovery execution instruction corresponding to the type of the source data source based on the parsing result includes: Parse and obtain the concurrency identifiers of each job, as well as the timestamps and data offsets corresponding to each job concurrency identifier; In response to the type being a database type, the job recovery execution instruction is generated based on the data offset and the SQL statements preceding the timestamp, or In response to the type being a file system type, the job resumption execution instruction is generated based on the data offset and the file read interval range before the timestamp, or In response to the message queue type, the job resumption execution instruction is generated based on the data offset.

7. A data integration system, comprising: The job management module is configured to provide job execution parameters for the data integration job to be configured. The job execution parameters include enabled breakpoint resume, multiple types of source data sources and target data sources, data mapping parameters between the source data sources and the target data sources, and job concurrency. The job execution module is configured to generate a job start execution instruction for the data integration job based on the configured job execution parameters, and to schedule a corresponding computing cluster to run the data integration job in response to the job start execution instruction. The job monitoring module is configured to monitor the running status of the data integration job. During the running of the data integration job, it stores the data transmission information of the data read from the source data source. The data transmission information includes timestamp, job concurrency identifier and data offset. The job exception handling module is configured to respond to the running status of a job exception and the enabled breakpoint resume function, obtain and parse the data transmission information of the latest read data when the job exception occurs, and generate a job recovery execution instruction corresponding to the type of the source data source based on the parsing result to resume the operation of the data integration job from the breakpoint.

8. The data integration system according to claim 7, wherein, The job exception handling module is further configured to parse and obtain the concurrency identifiers of each job and the timestamps and data offsets corresponding to each job concurrency identifier; in response to the type being a database type, to generate the job recovery execution instruction based on the SQL statements before the timestamp and the data offset; or in response to the type being a file system type, to generate the job recovery execution instruction based on the file read range before the timestamp and the data offset; or in response to the type being a message queue type, to generate the job recovery execution instruction based on the data offset.

9. An electronic device comprising: One or more processors; Memory, which stores one or more computer program modules. The one or more computer program modules are configured to be executed by the one or more processors to implement the method according to any one of claims 1-6.

10. A computer-readable medium storing computer-executable instructions, wherein, When computer-executable instructions are executed by one or more processors, they are used to implement the method according to any one of claims 1-6.

Images