An intelligent data transmission exchange platform and task flow conversion method

Abstract

The embodiment of the present application provides a kind of intelligent data transmission exchange platform and task flow conversion method, the platform includes: management server cluster generates task instance by the task of source server transmission, and task instance is transmitted to task queue engine service cluster, task queue engine service cluster is converted to task instance, and the transmission type and execution parameter corresponding to task instance are obtained by parsing task instance, and task instance generates spark task submission command, and is submitted to data exchange engine service cluster, data exchange engine service cluster passes through data exchange engine, and the data transmission task corresponding to task instance is deployed to big data cluster, big data cluster parses spark task submission command, and distributes cluster node, and cluster node transmits task instance to target server.The data exchange engine of the present application deploys data transmission task to big data cluster, and the transmission of task instance is executed by cluster node, and the performance pressure of the database extracted is less.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to an intelligent data transmission and exchange platform and a task flow method. Background Technology

[0002] The intelligent data exchange platform is a service platform used to build enterprise-level data exchange between heterogeneous data sources. It solves the problem of automating the transmission and exchange of heterogeneous data and is the core tool for batch data exchange and transmission in the data middleware. It can meet the application needs of heterogeneous data source data transmission in the construction of information systems in the IT industry.

[0003] Currently, when building data warehouses (DWs) in various industries, the traditional way to achieve data transmission between databases is to manually write ETL (Extract-Transform-Load, data warehouse technology) scripts. The traditional ETL method has disadvantages such as large workload, poor script reusability, low efficiency, and inability to achieve high availability execution, and is gradually failing to meet the growing business needs.

[0004] Building on this foundation, some open-source data transfer tools have begun to gradually replace traditional manual ETL methods, such as DataX and Sqoop. While these tools have improved the efficiency of traditional ETL methods to some extent, they still have drawbacks, including limited support for different types of databases and significant pressure on the source server's database during data extraction. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide an intelligent data transmission and exchange platform and a task flow method to solve the technical problems of limited database types supported in the prior art and the large database pressure on the source server when extracting data.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an intelligent data transmission and exchange platform, the platform comprising:

[0007] The management server cluster is used to assemble tasks transmitted from the source server into task instances and transmit the task instances to the task queue engine service cluster.

[0008] The task queue engine service cluster is used to receive the task instances and put ready task instances into the running queue for execution. When a task instance with a data transmission type of data transmission task is detected, the task instance is parsed to obtain the execution parameters corresponding to the task instance. The amount of resources required to execute the data transmission task is determined according to the data transmission amount in the execution parameters. The task instance generates a Spark task submission command and submits it to the data exchange engine service cluster according to the execution parameters and the amount of resources.

[0009] The data exchange engine service cluster is used to transmit the Spark task submission command to the big data cluster;

[0010] The big data cluster is used to parse the Spark task submission command to obtain the execution parameters and resource quantity corresponding to the task instance. Based on the resource quantity, the data transmission task is divided into multiple data transmission subtasks, and the data transmission subtasks and execution parameters are distributed to multiple cluster nodes of the big data cluster. Each cluster node transmits the data from the source server to the target server according to the execution parameters.

[0011] In some possible implementations, the management server cluster includes:

[0012] The assembly submodule is used to assemble task instances by combining the execution parameters of the task according to the task's transmission type through a type adapter. The transmission types of the task include data transmission tasks and file transfer tasks. When the transmission type of the task is a data transmission task, the execution parameters include the database address and database type of the source server, the database address and database type of the target server, the data transmission volume, and the data transmission rules. When the transmission type of the task is a file transfer task, the execution parameters include the database address and database type of the source server, and the database address and database type of the target server.

[0013] The connection establishment submodule is used to establish a connection between the source server and one or more target servers.

[0014] In some possible implementations, the task queue engine service cluster includes:

[0015] The engine interface submodule is used to receive task instances;

[0016] The task filter is used to determine whether the task instance is ready. When the task instance is ready, it is placed in the run queue detection submodule for detection; when the task instance is not ready, it is placed in the retry queue.

[0017] The run queue detection submodule is used to determine whether the run queue is full. If the run queue is not full, the task instance is transferred to the task adaptation submodule; if the run queue is full, the task instance is placed in the waiting queue.

[0018] The task adaptation submodule includes a data transmission adapter, a task analyzer, and a command processor. The data transmission adapter transmits task instances with detected data transmission types of data transmission tasks to the task analyzer. The task analyzer parses the task instance to obtain the execution parameters corresponding to the task instance and determines the amount of resources required to execute the data transmission task based on the data transmission volume in the execution parameters. The command processor generates a Spark task submission command for the task instance based on the execution parameters and the amount of resources, and submits it to the task executor.

[0019] The task executor is used to submit the Spark task submission command to the data exchange engine service cluster.

[0020] In some possible implementations, the task queue engine service cluster further includes:

[0021] An initialization queue processor is used to periodically scan the initialization queue to see if the task instance exists. If the task instance exists, it is pulled to the task filter and resubmitted.

[0022] In some possible implementations, the task queue engine service cluster further includes:

[0023] The retry queue processor is used to detect whether the task instance in the retry queue has timed out. If it has timed out, the task instance is placed in the failure queue. If it has not timed out, the task instance is placed in the waiting queue to wait for the next execution.

[0024] In some possible implementations, the task queue engine service cluster further includes:

[0025] The failure queue processor is used to detect whether the number of retries for a task instance in the failure queue has reached a preset number of retries. If the number of retries for a task instance reaches the preset number of retries, the task instance is determined to be a failed task. If the preset number of retries is reached, the task instance is placed in the waiting queue to wait for the next execution.

[0026] In some possible implementations, the task queue engine service cluster further includes:

[0027] A submission monitor is used to determine whether the task instance has been submitted successfully. If the submission is successful, the task is completed. If the submission fails, the task instance is placed in the failure queue.

[0028] In some possible implementations, the task adaptation submodule further includes:

[0029] The file transfer adapter is used to transfer the detected task instance of file transfer task to the connection processor.

[0030] A connection processor is used to create a connection between the source server and the target server, or to acquire a connection between the source server and the target server;

[0031] The platform also includes a file transfer engine service cluster, used to write the task instance to the target server based on the connection.

[0032] Secondly, embodiments of the present invention provide a task flow method for a task queue engine service cluster, the method comprising:

[0033] Step S1: Receive the task instance;

[0034] Step S2: Place the ready task instances into the run queue for execution;

[0035] Step S3: When a task instance of data transmission type is detected, the task instance is parsed to obtain the execution parameters corresponding to the task instance. The amount of resources required to execute the data transmission task is determined according to the data transmission amount in the execution parameters. The task instance is generated into a Spark task submission command according to the execution parameters and the amount of resources.

[0036] Step S4: Submit the Spark task submission command to the data exchange engine service cluster.

[0037] In some possible implementations, step S2 specifically includes:

[0038] Step S21: Determine whether the task instance is ready. If the task instance is ready, trigger step S22. If the task instance is not ready, put the task instance into the retry queue.

[0039] Step S22: Determine whether the running queue is full. If the running queue is not full, the task instance is placed in the running queue for execution. If the running queue is full, the task instance is placed in the waiting queue.

[0040] In some possible implementations, the method further includes:

[0041] Step S5: Determine whether the task instance has been successfully submitted. If the submission is successful, the task is completed. If the submission fails, the task instance is placed in the failure queue.

[0042] In some possible implementations, the method further includes:

[0043] Step S6: Periodically scan the initialization queue to see if the task instance exists. If the task instance exists, trigger the execution of step S2.

[0044] In some possible implementations, the method further includes:

[0045] Step S7: Detect whether the task instance in the retry queue has timed out. If it has timed out, put the task instance into the failure queue. If it has not timed out, put the task instance into the waiting queue to wait for the next execution.

[0046] In some possible implementations, the method further includes:

[0047] Step S8: Detect whether the number of retries for the task instance in the failure queue has reached the preset number of retries. If the number of retries for the task instance reaches the preset number of retries, the task instance is determined to be a failed task. If the preset number of retries is reached, the task instance is placed in the waiting queue to wait for the next execution.

[0048] In some possible implementations, the method further includes:

[0049] Step S9: Retrieve the task instance from the waiting queue and put the retrieved task instance back into the running queue.

[0050] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a task flow method for a task queue engine service cluster as described in any of the above claims.

[0051] Fourthly, the present invention provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a task flow method for a task queue engine service cluster as described in any of the above claims.

[0052] The above technical solution has the following beneficial effects:

[0053] 1. The data transmission and exchange engine of this intelligent data transmission and exchange platform integrates the Spark high-performance in-memory computing engine. It executes tasks through the resources of the big data cluster and selects different data transmission plugins for different types of databases for extraction and writing. Furthermore, the data cleaning and filtering are performed in the memory of the cluster nodes, which reduces the performance pressure on the extracted databases.

[0054] 2. This intelligent data transmission and exchange platform uses a plug-in approach to integrate database types, enabling rapid integration and hot deployment of unsupported database types. It supports up to 30 source server database types, including commonly used MySQL, Oracle, PostgreSQL, Hive, MongoDB, as well as domestic IT innovation databases such as Gbase, GaussDB, TDSql, KingBase, and OceanBase. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the overall structure of an intelligent data transmission and exchange platform according to an embodiment of the present invention;

[0057] Figure 2 This is a structural block diagram of a management service cluster according to an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram illustrating the establishment of a connection between a source server system and a target server system according to an embodiment of the present invention;

[0059] Figure 4 This is a structural block diagram of a task queue engine service cluster according to an embodiment of the present invention;

[0060] Figure 5 This is another structural block diagram of a task queue engine service cluster according to an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of another intelligent data transmission and exchange platform according to an embodiment of the present invention;

[0062] Figure 7 This is a flowchart of a task flow method for a task queue engine service cluster according to an embodiment of the present invention;

[0063] Figure 8 This is a flowchart illustrating the submission of task instances to the running queue according to an embodiment of the present invention;

[0064] Figure 9 This is a flowchart illustrating a data transmission task as an example of an embodiment of the present invention;

[0065] Figure 10This is a deployment diagram of a task flow method for a task queue engine service cluster according to an embodiment of the present invention;

[0066] Figure 11 This is a functional block diagram of a computer-readable storage medium according to an embodiment of the present invention;

[0067] Figure 12 This is a functional block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1

[0070] Figure 1 This is a schematic diagram of the overall structure of an intelligent data transmission and exchange platform according to an embodiment of the present invention, as shown below. Figure 1 As shown, the platform includes:

[0071] The system includes a management server cluster 10, a task queue engine service cluster 20, a data exchange engine service cluster 30, and a big data cluster 40. The management server cluster 10 includes multiple management servers, each of which is used to assemble tasks transmitted from the source server into task instances and transmit the task instances to the task queue engine service cluster 20.

[0072] In this embodiment, a single database task on the source server can generate multiple task instances, each with different SQL matching parameters. SQL stands for Structured Query Language. SQL is a database query and programming language used to access, query, update, and manage relational database systems; it is also the file extension for database script files.

[0073] The task queue engine service cluster 20 includes multiple task queue engines. Each task queue engine is used to receive task instances and put ready task instances into the running queue for execution. When a task instance with the transmission type of data transmission task is detected, the task instance is parsed to obtain the execution parameters corresponding to the task instance. After determining the amount of resources required to execute the data transmission task based on the data transmission amount in the execution parameters, the task instance generates a Spark task submission command based on the resource quantity and execution parameters, and submits it to the data exchange engine service cluster 30.

[0074] In this embodiment of the invention, the Spark task submission command refers to submitting the task through the spark-submit script. The spark-submit script is used to start the application on the cluster. It is located in the bin directory of Spark. This startup method allows users to use all the cluster management functions supported by Spark through a unified interface.

[0075] The data exchange engine service cluster 30 includes multiple data exchange engines. Each data exchange engine transmits the execution parameters and resource quantity of its corresponding task instance to the big data cluster. That is, each data exchange engine deploys the data transmission task to the big data cluster 40, and the big data cluster 40 executes the data transmission task according to the execution parameters and resource quantity.

[0076] The big data cluster 40 includes multiple cluster nodes. After receiving a Spark task submission command, the big data cluster 40 parses the Spark task submission command to obtain the execution parameters and resource quantity corresponding to the task instance. Based on the resource quantity, the data transmission task is divided into multiple data transmission subtasks, and the data transmission subtasks and execution parameters are distributed to the cluster nodes of the big data cluster. The cluster nodes transmit the data from the source server to the target server according to the execution parameters.

[0077] Specifically, in this embodiment, the big data cluster 40 divides the data transmission task into multiple data transmission subtasks based on the acquired resource quantity and data transmission volume. The number of subtasks is the same as the resource quantity. For example, if the resource quantity is 4 and the data transmission volume is 800 million data points, then the big data cluster 40 will divide the data transmission task into four data transmission subtasks. Each data transmission subtask will transmit 200 million data points. The four data transmission subtasks will be distributed to different cluster nodes, meaning they will be executed concurrently on four cluster nodes. Each cluster node will transmit the data transmission subtask from the source server's database to the target server's database according to the execution parameters corresponding to the task instance and the data transmission rules. For example, if the source server's database is A (MySQL type) and the target server's database is B (Oracle type), then the cluster nodes will extract data from database A using the MySQL read plugin, clean the data, and then write it to database B using the Oracle write plugin, according to the data transmission rules. The data transmission rules can be extraction-cleaning-writing, etc. It should be noted that in this embodiment, the current task can only end and resources released after all four data transmission subtasks have been transmitted.

[0078] In this embodiment, the big data cluster 40 can be understood as an operating system-level resource management and scheduling framework. Various frameworks can run on the big data cluster 40, as long as they conform to its standards. This allows multiple computing frameworks to share cluster resources and allocate them on demand; you only need the resources you require from the big data cluster 40, thus improving overall resource utilization. For example, submitting a Spark task submission command starts a Driver process. The Driver process sends a request to the ResourceManager (RS) to start the ApplicationMaster (AM). Upon receiving the request, the RS randomly selects a NodeManager (NM) to start the AM. After the AM starts, it requests a batch of container resources from the RS to start Executors. The RS finds a batch of NMs and returns them to the AM to start the Executors. The AM then sends commands to the NMs to start the Executors. After the Executors start, they register with the Driver. The Driver sends tasks to the Executors, and the execution status and results are returned to the Driver.

[0079] In this embodiment, the exchange platform utilizes the characteristics of the big data cluster 40 and does not participate in the final task transmission and exchange. The task instance is allocated on the big data platform, which is a processing mechanism within the big data cluster 40.

[0080] In this embodiment of the invention, different read / write plugins can be selected according to different database types. Therefore, the source server and target server in this embodiment can each include multiple databases, supporting up to about 30 database types, such as commonly used MySQL, Oracle, PostgreSQL, Hive, MongoDB, as well as domestic IT innovation databases such as Gbase, GaussDB, TDSql, KingBase, OceanBase, etc.

[0081] The intelligent data transmission and exchange platform of this invention integrates the Spark high-performance in-memory computing engine into its data transmission and exchange engine. It executes tasks using the resources of the big data cluster and selects different data transmission plugins for different types of databases for extraction and writing. Furthermore, data cleaning, filtering, and other processing are all performed in the memory of the cluster nodes, which reduces the performance pressure on the extracted database.

[0082] In addition, compared to traditional ETL or open-source tools that generally use single-node operation, where a node crashes or experiences network fluctuations during operation and the task fails, requiring manual intervention to re-execute and thus failing to achieve automation, this intelligent data exchange platform is built on a microservice architecture and adopts a multi-task instance, multi-cluster operation mode. The failure of one node or network fluctuation will not affect the operation of other nodes, and the task can automatically retry after failure, ensuring the stability of batch processing.

[0083] Figure 2 This is a structural block diagram of a management service cluster according to an embodiment of the present invention. In some embodiments, such as Figure 2 As shown, the management server cluster 10 includes an assembly submodule 101 and a connection establishment submodule 102.

[0084] Assembly submodule 101 is used to assemble task instances by using a type adapter to assemble the execution parameters of the task according to the task's transmission type. The transmission type of the task includes data transmission task and file transfer task. When the transmission type of the task is data transmission task, the execution parameters include the database address and database type of the source server, the database address and database type of the target server, the data transmission volume, and the data transmission rules. When the transmission type of the task is file transfer task, the execution parameters include the database address and database type of the source server and the database address and database type of the target server.

[0085] Specifically, during the task assembly process, a database transfer task from a source server can correspond to multiple task instances. Each task instance can have different SQL language matching parameters. SQL stands for Structured Query Language, a database query and programming language used to access, query, update, and manage relational database systems. A task instance can be understood as a database record, primarily recording the execution parameters for that task. After being assembled into task instances, the tasks transferred from the source server are transmitted to the engine interface of the task queue engine service cluster 20 in the form of task instances.

[0086] Connection establishment submodule 102 is used to establish a connection between the source server and one or more target servers.

[0087] Figure 3 This is a schematic diagram illustrating the establishment of a connection between a source server and a target server according to an embodiment of the present invention. Figure 3As shown, in this embodiment, the source data of the source server is managed as a resource. This means that the database table structure in the data source can be collected online or offline via the connection establishment submodule 102, or the database table structure can be maintained using tools such as Excel. The tables in the database are then imported into the platform. During this process, incremental statements can be configured for each table, and these statements can be extracted. Furthermore, Spark functions can be used to set field anonymization rules. During data exchange, non-compliant data or data to be transformed undergoes data cleaning, filtering, and anonymization processing during transmission. The Spark functions are built-in SQL rules that adhere to standard SQL language, processing data through SQL. In this embodiment, one source server can be subscribed to by multiple target servers, enabling one-to-many data file transmission and reducing the user's configuration workload.

[0088] Figure 4 This is a structural block diagram of a task queue engine service cluster according to an embodiment of the present invention. In some embodiments, such as Figure 4 As shown, the task queue engine service cluster 20 includes:

[0089] Engine interface submodule 201 is used to receive task instances;

[0090] Task filter 202 is used to determine whether a task instance is ready. When a task instance is ready, it is placed in the run queue detection submodule 203 for detection; when a task instance is not ready, it is placed in the retry queue.

[0091] The run queue detection submodule 203 is used to determine whether the run queue is full. If the run queue is not full, the task instance is transferred to the task adaptation submodule 204; if the run queue is full, the task instance is placed in the waiting queue.

[0092] The task adaptation submodule 204 includes a data transmission adapter, a task analyzer, and a command processor. The data transmission adapter is used to detect task instances with a data transmission type of data transmission task and transmit them to the task analyzer. The task analyzer is used to parse the task instance to obtain the execution parameters corresponding to the task instance and determine the amount of resources required to execute the data transmission task based on the data transmission amount in the execution parameters. The command processor is used to generate a Spark task submission command for the task instance based on the execution parameters and the amount of resources and submit it to the task executor 205.

[0093] Task executor 205 is used to submit Spark tasks to the data exchange engine service cluster 30.

[0094] In this embodiment, when the transmission type of the task instance is a data transmission task, the task analyzer needs to parse the task instance to obtain the execution parameters corresponding to the task instance, and determine whether the data transmission volume in the execution parameters is 0. If the data transmission volume is 0, the task instance will not be submitted to the data exchange engine service cluster 30 to save the resources of the running node. If the data volume is greater than 0, the number of resources required for the data volume of this task can be calculated according to the self-designed algorithm. The number of resources is positively correlated with the data transmission volume. The larger the data transmission volume, the more resources are required.

[0095] This invention adopts the concept of a streaming queue. After a task instance is submitted to the task queue engine service cluster 20, the task filter 202 determines whether the task instance is ready. For ready task instances, the running queue detection submodule 203 checks whether the number of executing task instances has reached the user-specified limit. If the limit has been reached, the task is placed in the waiting queue. If the limit has not been reached, the task is executed immediately. After the task is completed, it retrieves waiting tasks from the waiting queue, realizing the streaming operation of task execution, improving the efficiency of the queue, and enabling the task queue engine service cluster 20 to have a large task throughput. In addition, unready task instances are not directly judged as failures, but are first placed in the retry queue.

[0096] In this embodiment of the invention, the waiting queue is shared by all task queue engines. Each task queue engine can pull tasks from the waiting queue for execution. Simultaneously, a series of locking operations ensure the atomicity of transactions and prevent dirty reads. The shared waiting queue can balance the load on each task queue engine, achieving a relative balance in the number of tasks executed by each engine.

[0097] In addition, the task queue engine service cluster 20 of this embodiment of the invention has a variety of built-in intelligent processors, which can ensure the normal circulation of task instances and the efficient allocation of resources, and improve the efficiency of data transmission and exchange.

[0098] Figure 5 This is a structural block diagram of another task queue engine service cluster according to an embodiment of the present invention. In some embodiments, such as Figure 5 As shown, the task queue engine service cluster 20 also includes:

[0099] The initialization queue processor 206 is used to periodically scan whether there are task instances in the initialization queue. If there are task instances, the task instances are pulled to the task filter 202 and resubmitted.

[0100] In this embodiment, the initialization queue is shared by all task queue engines. Each task queue transmission engine periodically scans this queue for task instances through the initialization queue processor 206. If a task instance exists, it is retrieved and the task process is executed.

[0101] In some embodiments, such as Figure 5 As shown, the task queue engine service cluster 20 also includes:

[0102] The retry queue processor 207 is used to detect whether the task instance in the retry queue has timed out. If it has timed out, the task instance is put into the failure queue. If it has not timed out, the task instance is put into the waiting queue to wait for the next execution.

[0103] In this embodiment of the invention, unready task instances are placed in a retry queue. The retry queue processor 207 cyclically retrieves task instances from the retry queue and performs a readiness scan. If the scan time for unready tasks in the platform variable management function (default two hours) is exceeded, the task is directly set to a failed state. If the scan finds that the task is ready (e.g., the signal file of the upstream file has been generated), the task is placed in a waiting queue to wait for the next execution.

[0104] In some embodiments, such as Figure 5 As shown, the task queue engine service cluster 20 also includes:

[0105] The failure queue processor 208 is used to detect whether the number of retries for a task instance in the failure queue has reached a preset number of retries. If the number of retries for a task instance reaches the preset number of retries, the task instance is determined to be a failed task. If the preset number of retries is reached, the task instance is placed in the waiting queue to wait for the next execution.

[0106] In this embodiment of the invention, during the execution of task instances, failed task instances are not immediately set to a failed state. Instead, they are first placed in a failure queue. The failure queue processor 208 continuously retrieves task instances from the failure queue and performs retry checks on them. It determines whether the number of retries for the task instance has reached a preset maximum retry count. If the preset maximum retry count has not been reached, the task instance is placed in a waiting queue to wait for the next execution. If the preset maximum retry count has been reached, the task instance is set to a failed state. For example, if the preset retry count is 5, the task instance is placed in a waiting queue to wait for the next execution if the number of retries has not reached 5. If the number of retries has reached 5, the task instance is set to a failed state.

[0107] In some embodiments, such as Figure 5As shown, the task queue engine service cluster 20 may also include:

[0108] Submitting to monitor 209 is used to determine whether the task instance has been submitted successfully. If the submission is successful, the task is completed; if the submission fails, the task instance is placed in the failure queue.

[0109] In this embodiment of the invention, after a task instance is submitted, a submission monitor 209 is started to determine whether the task instance has been submitted successfully through methods such as delayed task judgment. If the task is not submitted successfully, the task is placed in the failure queue.

[0110] In this embodiment of the invention, the task queue engine sets up five queue types—initialization queue, waiting queue, failure queue, retry queue, and running queue—to handle task flow, achieving one-stop task configuration, execution, management, and monitoring. Compared to using traditional ETL and open-source tools, users can quickly maintain tasks on the page, generate corresponding batch tasks, and perform one-click batch execution and batch scheduled execution of tasks, greatly reducing development workload.

[0111] The task queue engine in this embodiment of the invention has multiple built-in intelligent processors, which can ensure the normal flow of tasks and the efficient allocation of resources.

[0112] In some embodiments, the task adaptation submodule 204 may further include:

[0113] The file transfer adapter is used to determine whether the transfer type of the task instance is a file transfer task. When the transfer type of the task instance is a file transfer task, the task instance is transferred to the connection processor.

[0114] A connection processor is used to establish or acquire a connection between a source server and a target server. Specifically, the connection processor first checks if a connection between the source server and the target server has already been established. If a connection has been established before, it acquires that connection directly; otherwise, it re-establishes the connection between the source server and the target server.

[0115] In this embodiment, the present invention can determine the transmission type of the task instance through different adapters. It can perform data transmission tasks or file fast transfer tasks. When the transmission type of the task instance is a file fast transfer task, the data file can be quickly transferred directly through the pre-established connection between the source server and the target server.

[0116] Figure 6 This is a schematic diagram of another intelligent data transmission and exchange platform according to an embodiment of the present invention. Figure 6As shown, this platform also includes a file transfer engine service cluster 50, which is used to write task instances to the target server based on the connection establishment submodule 103 or the connection processor establishing and storing the connection in the management session container.

[0117] Specifically, file transfer tasks are distributed to different file transfer engines through a microservice gateway. The gateway uses load balancing to automatically select the engine, with the default mechanism being even distribution, meaning each engine takes turns accepting tasks. The file transfer engine establishes a remote connection with the target server and ultimately sends a file transfer command to perform the file transfer task through automatic response.

[0118] Specifically, the file transfer engine service cluster 50 includes multiple file transfer engines. File transfer tasks are transferred quickly through different file transfer engines, which greatly improves the efficiency of data transfer.

[0119] This invention also supports fast reading and writing of data files, and supports data file types such as FTP, SFTP, OSS, S3, etc. It can establish a source server and connect to multiple target servers, or multiple source servers and multiple target servers, which greatly reduces the workload of configuration.

[0120] Example 2

[0121] Figure 7 This is a flowchart illustrating a task flow method for a task queue engine service cluster according to an embodiment of the present invention. Figure 7 As shown, the method includes:

[0122] Step S1: Receive task instance;

[0123] Step S2: Place the ready task instances into the run queue for execution;

[0124] Figure 8 This is a flowchart illustrating the submission of task instances to the run queue according to an embodiment of the present invention. In some embodiments, such as... Figure 8 As shown, step S2 specifically includes:

[0125] Step S21: Determine whether the task instance is ready. If the task instance is ready, trigger step S22. If the task instance is not ready, put the task instance into the retry queue.

[0126] Step S22: Determine if the run queue is full. If the run queue is not full, place the task instance into the run queue for execution; if the run queue is full, place the task instance into the waiting queue.

[0127] Step S3: Detect task instances with transmission type of data transmission task, parse the task instance to obtain the execution parameters corresponding to the task instance, determine the amount of resources required to execute the data transmission task based on the data transmission amount in the execution parameters, and generate a Spark task submission command for the task instance based on the execution parameters and the amount of resources.

[0128] Step S4: Submit the Spark task submission command to the data exchange engine service cluster.

[0129] This invention employs a streaming queue approach. After a task instance is submitted to the task queue engine, it is determined whether the task instance is ready. For ready task instances, the running queue detection submodule 203 checks whether the number of executing task instances has reached the user-specified limit. If the limit is reached, the instance is placed in the waiting queue. If the limit is not reached, the task instance is executed immediately. For example, if the user-specified limit is 2000, meaning the running queue can handle 2000 task instances, when the running queue detection module 203 detects that the running queue currently handles 2000 task instances, reaching the user-specified limit, the task instance is placed in the waiting queue. Conversely, if the running queue does not handle 2000 task instances, the task instance is placed in the running queue for processing. After a task is completed, it retrieves waiting task instances from the waiting queue, achieving streaming task execution, improving queue efficiency, and enabling the task queue engine to have a high task throughput. Furthermore, unready task instances are not directly considered failures but are first placed in the retry queue.

[0130] In this embodiment of the invention, the waiting queue is shared by all task queue engines. Each task queue engine can pull task instances from the waiting queue for execution, and through a series of locking operations, the atomicity of transactions is guaranteed, avoiding dirty reads. The shared waiting queue can balance the load of each task queue engine, so that the number of tasks executed by each task queue engine reaches a relatively balanced level.

[0131] Figure 9 This is a flowchart illustrating a data transmission task as an example of an embodiment of the present invention. In some embodiments, such as... Figure 9 As shown, step S3 specifically includes:

[0132] Step S31: Detect the task instance corresponding to the data transmission task;

[0133] Step S32: Parse the task instance to obtain the execution parameters corresponding to the task instance, and determine the amount of resources required to execute the data transmission task based on the data transmission volume in the execution parameters;

[0134] Step S33: Generate a Spark task submission command for the task instance based on the execution parameters and the number of resources, triggering the execution of step S4.

[0135] Figure 10 This is a deployment diagram of a task flow method for a task queue engine service cluster according to an embodiment of the present invention. In some embodiments, such as Figure 10 As shown, the method also includes:

[0136] Step S5: Determine whether the task instance has been successfully submitted. If the submission is successful, the task is completed. If the submission fails, the task instance is placed in the failure queue.

[0137] Step S6: Periodically scan the initialization queue and re-determine whether the task instances in the initialization queue are ready.

[0138] In this step, the initialization queue is shared by all task queue engines. Each task queue transmission engine scans this initialization queue periodically and re-determines the readiness of the task instances existing in the initialization queue.

[0139] In some embodiments, such as Figure 10 As shown, the method also includes:

[0140] Step S7: Determine whether the task instance in the retry queue has timed out. If it has timed out, the task instance is determined to have failed and is placed in the failure queue. If it has not timed out, the task instance is placed in the waiting queue to wait for the next execution.

[0141] In this embodiment of the invention, unready task instances are placed in a retry queue, and then the task instances in the retry queue are retrieved in a loop and ready scan is performed. If the scan time for unready tasks in the platform variable management function is exceeded (default two hours), the task is directly set to a failed state. If the task is found to be ready during the scan (e.g., the signal file of the upstream file has been generated), the task is placed in a waiting queue to wait for the next execution.

[0142] In some embodiments, such as Figure 10 As shown, the method also includes:

[0143] Step S8: Determine whether the number of retries for the task instance in the failure queue has reached the preset number of retries. If the preset number of retries is exceeded, the task instance is determined to be a failed task. If the preset number of retries is not exceeded, the task instance is placed in the waiting queue to wait for the next execution.

[0144] In this embodiment of the invention, during the execution of task instance flow, a failed task instance will not be immediately set to a failed state. Instead, the task instance will be placed in a failure queue. Then, the task instances in the failure queue will be retrieved in a loop, and a retry check will be performed on the task instances in the failure queue to determine whether the number of retries for the task instance has reached the upper limit of the number of retries. If the upper limit of the number of retries has not been reached, the task instance will be placed in a waiting queue to wait for the next execution. If the upper limit has been reached, the task instance will be set to a failed state.

[0145] In some embodiments, such as Figure 10 As shown, the method also includes:

[0146] Step S9: Retrieve task instances from the waiting queue and put the retrieved task instances back into the running queue for testing.

[0147] The task instance transfer method of the task queue engine provided in this embodiment of the invention has the following beneficial effects:

[0148] This invention adopts the concept of a streaming queue. After a task instance is submitted to the task queue engine, it is determined whether the executing task has reached the user-specified limit. If the limit has been reached, it is placed in a public waiting queue. If the limit has not been reached, the task is executed immediately. After the task is completed, it will retrieve waiting tasks from the public waiting queue, realizing the streaming operation of task execution, improving the queue's operating efficiency, and enabling the queue engine to have a large task throughput.

[0149] All engines share a common waiting queue pool. In the intelligent data transmission and exchange platform, the waiting queue is shared by all queue engines. Each queue engine can pull tasks from the waiting queue for execution. At the same time, through a series of locking operations, the atomicity of transactions is guaranteed, and dirty reads are avoided. The common waiting queue pool can balance the load of each queue engine, so that the number of tasks executed by each queue engine is relatively balanced.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] Example 3

[0152] Figure 11 This is a functional block diagram of a computer-readable storage medium according to an embodiment of the present invention. Figure 11 As shown, the computer-readable storage medium 300 stores a computer program 310 thereon. When the computer program 310 is executed by the processor, it implements the above-mentioned task flow method for a task queue engine service cluster.

[0153] Example 4

[0154] Figure 12 This is a functional block diagram of a computer device according to an embodiment of the present invention. The computer device may be a terminal, and its internal structure diagram may be as follows. Figure 12 As shown in the figure, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements a task flow method for a task queue engine service cluster. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0155] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0161] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0162] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0163] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0164] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An intelligent data transmission exchange platform, characterized by, The platform includes: The management server cluster is used to assemble tasks transmitted from the source server into task instances and transmit the task instances to the task queue engine service cluster. The task queue engine service cluster is used to receive the task instances and put ready task instances into the running queue for execution. When a task instance with a data transmission type of data transmission task is detected, the task instance is parsed to obtain the execution parameters corresponding to the task instance. The amount of resources required to execute the data transmission task is determined according to the data transmission amount in the execution parameters. The task instance generates a Spark task submission command and submits it to the data exchange engine service cluster according to the execution parameters and the amount of resources. The data exchange engine service cluster is used to transmit the Spark task submission command to the big data cluster; A big data cluster is used to parse the Spark task submission command to obtain the execution parameters and resource quantity corresponding to the task instance, divide the data transmission task into multiple data transmission subtasks according to the resource quantity, and distribute the data transmission subtasks and execution parameters to multiple cluster nodes of the big data cluster. Each cluster node transmits the data from the source server to the target server according to the execution parameters. The task queue engine service cluster includes: The task filter is used to determine whether the task instance is ready. When the task instance is ready, it is placed in the run queue detection submodule for detection; when the task instance is not ready, it is placed in the retry queue. The task adaptation submodule includes a data transmission adapter, a task analyzer, and a command processor. The data transmission adapter transmits the detected data transmission type (data transmission task) to the task analyzer. The task analyzer parses the task instance to obtain the execution parameters corresponding to the task instance and determines the amount of resources required to execute the data transmission task based on the data transmission volume in the execution parameters. The command processor generates a Spark task submission command for the task instance based on the execution parameters and the amount of resources, and submits it to the task executor. The task executor submits the Spark task submission command to the data exchange engine service cluster.

2. The intelligent data transmission and exchange platform according to claim 1, characterized in that, The management server cluster includes: The assembly submodule is used to assemble task instances by combining the execution parameters of the task according to the task's transmission type through a type adapter. The transmission types of the task include data transmission tasks and file transfer tasks. When the transmission type of the task is a data transmission task, the execution parameters include the database address and database type of the source server, the database address and database type of the target server, the data transmission volume, and the data transmission rules. When the transmission type of the task is a file transfer task, the execution parameters include the database address and database type of the source server, and the database address and database type of the target server. The connection establishment submodule is used to establish a connection between the source server and one or more target servers.

3. The intelligent data transmission and exchange platform according to claim 2, characterized in that, The task queue engine service cluster also includes: The engine interface submodule is used to receive task instances; The run queue detection submodule is used to determine whether the run queue is full. If the run queue is not full, the task instance is transferred to the task adaptation submodule; if the run queue is full, the task instance is placed in the waiting queue.

4. The intelligent data transmission and exchange platform according to claim 3, characterized in that, The task queue engine service cluster also includes: An initialization queue processor is used to periodically scan the initialization queue to see if the task instance exists. If the task instance exists, it is pulled to the task filter and resubmitted.

5. The intelligent data transmission and exchange platform according to claim 4, characterized in that, The task queue engine service cluster also includes: The retry queue processor is used to detect whether the task instance in the retry queue has timed out. If it has timed out, the task instance is placed in the failure queue. If it has not timed out, the task instance is placed in the waiting queue to wait for the next execution.

6. The intelligent data transmission and exchange platform according to claim 5, characterized in that, The task queue engine service cluster also includes: The failure queue processor is used to detect whether the number of retries for a task instance in the failure queue has reached a preset number of retries. If the number of retries for a task instance reaches the preset number of retries, the task instance is determined to be a failed task. If the preset number of retries has not been reached, the task instance is placed in the waiting queue to wait for the next execution.

7. The intelligent data transmission and exchange platform according to claim 6, characterized in that, The task queue engine service cluster also includes: A submission monitor is used to determine whether the task instance has been submitted successfully. If the submission is successful, the task is completed. If the submission fails, the task instance is placed in the failure queue.

8. The intelligent data transmission and exchange platform according to claim 3, characterized in that, The task adaptation submodule also includes: The file transfer adapter is used to transfer the detected task instance of file transfer task to the connection processor. A connection processor is used to create a connection between the source server and the target server, or to acquire a connection between the source server and the target server; The platform also includes a file transfer engine service cluster, used to write the task instance to the target server based on the connection.

9. A task flow method for a task queue engine service cluster, characterized in that, The method is applied to the intelligent data transmission and exchange platform according to claim 1, comprising: Step S1: Receive the task instance; Step S2: Place the ready task instances into the run queue for execution; Step S3: When a task instance of data transmission type is detected, the task instance is parsed to obtain the execution parameters corresponding to the task instance. The amount of resources required to execute the data transmission task is determined according to the data transmission amount in the execution parameters. The task instance is generated into a Spark task submission command according to the execution parameters and the amount of resources. Step S4: Submit the Spark task submission command to the data exchange engine service cluster.

10. A task flow method for a task queue engine service cluster according to claim 9, characterized in that, Step S2 specifically includes: Step S21: Determine whether the task instance is ready. If the task instance is ready, trigger step S22. If the task instance is not ready, put the task instance into the retry queue. Step S22: Determine whether the running queue is full. If the running queue is not full, the task instance is placed in the running queue for execution. If the running queue is full, the task instance is placed in the waiting queue.

11. The task flow method for a task queue engine service cluster according to claim 10, characterized in that, The method further includes: Step S5: Determine whether the task instance has been successfully submitted. If the submission is successful, the task is completed. If the submission fails, the task instance is placed in the failure queue.

12. The task flow method for a task queue engine service cluster according to claim 11, characterized in that, The method further includes: Step S6: Periodically scan the initialization queue to see if the task instance exists. If the task instance exists, then trigger the execution of step S2.

13. The task flow method for a task queue engine service cluster according to claim 12, characterized in that, The method further includes: Step S7: Detect whether the task instance in the retry queue has timed out. If it has timed out, put the task instance into the failure queue. If it has not timed out, put the task instance into the waiting queue to wait for the next execution.

14. The task flow method for a task queue engine service cluster according to claim 13, characterized in that, The method further includes: Step S8: Detect whether the number of retries for the task instance in the failure queue has reached the preset number of retries. If the number of retries for the task instance has reached the preset number of retries, the task instance is determined to be a failed task. If the preset number of retries has not been reached, the task instance is placed in the waiting queue to wait for the next execution.

15. A task flow method for a task queue engine service cluster according to claim 14, characterized in that, The method further includes: Step S9: Retrieve the task instance from the waiting queue and put the retrieved task instance back into the running queue.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a task flow method for a task queue engine service cluster as described in any one of claims 9-15.

17. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements a task flow method for a task queue engine service cluster as described in any one of claims 9-15.

Images