Data transmission methods, apparatus, equipment, storage media and products

By performing secure authentication and data processing across data centers within the same cluster service, the problem of streaming and batch data transmission across data centers is solved, achieving efficient and complete data transmission and reducing business and maintenance costs.

CN119336475BActive Publication Date: 2026-11-17CHINA MOBILE INFORMATION TECHNOLOGY CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411603838.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-11-11
Publication Date
2026-11-17
Estimated Expiration
2044-11-11

AI Technical Summary

Technical Problem

In existing technologies, batch data and streaming data need to be processed through two different systems, which results in the inability to share system resources, excessive business and maintenance costs, and data transmission across data centers is prone to data loss or incomplete transmission due to network complexity and security issues.

Method used

By implementing secure authentication across data centers within the same cluster service, and performing data encryption/decryption, compression, and encoding conversion during transmission, and utilizing visual orchestration and automated scheduling, it automatically generates streaming or batch data transmission tasks across centers and networks, supports the transmission of various data formats, and ensures data integrity and efficient transmission.

Benefits of technology

It enables seamless transmission of streaming and batch data within the same system, reduces manual configuration work, improves task maintainability and bandwidth resource utilization, and ensures efficient and complete data transmission in complex network environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119336475B_ABST
    Figure CN119336475B_ABST
Patent Text Reader

Abstract

This application discloses a data transmission method, apparatus, device, storage medium, and product, relating to the field of data transmission technology. The method generates an original data transmission task based on pre-generated data transmission task configuration information, including a dual-end data model and task priority. Based on the dual-end data model, the original data transmission task is converted into a target data transmission task. The target transmission task is then prioritized and scored using the task priority and a preset priority scheduling and scoring algorithm to obtain a task scheduling queue. Data transmission is performed based on the task scheduling queue and the target data transmission task. By converting the data transmission task according to the data model, the resulting target data transmission task can be either batch data transmission or streaming data transmission. Furthermore, priority sorting ensures that all tasks are executed on demand and in an orderly manner, significantly improving the efficiency and versatility of the data transmission system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data transmission method, apparatus, device, storage medium, and product. Background Technology

[0002] Data processing is the process of transforming raw data into meaningful information. Currently, there are two common methods for processing data: batch processing, in which multiple data records are collected and stored, and then processed together in one operation; and stream processing, in which the data source is continuously monitored and processed in real time as new data events occur.

[0003] Batch processing has the advantage of processing large amounts of data at convenient times, and can be scheduled to run when the computer or system may be idle (e.g., overnight) or during off-peak hours; its disadvantage is the time delay between data acquisition and result retrieval. Stream processing is well-suited for time-critical operations requiring real-time responses. For example, systems monitoring smoke and heat in a building need to trigger alarms and unlock doors so that residents can evacuate immediately in the event of a fire.

[0004] However, in existing technologies, batch and streaming data often need to be migrated through two different systems, and the system and computing resources of the two systems cannot be shared. In the current big data processing framework, MapReduce is mainly used to process batch data, while the stream processing framework Flink is mainly used to process streaming data. If the needs of streaming and batch data transmission between multiple heterogeneous clusters and different data formats need to be met simultaneously, two data transmission systems need to be developed and maintained at the same time, which results in excessively high business and maintenance costs.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a data transmission method, apparatus, device, storage medium, and computer program product, which aims to simultaneously meet the needs of streaming data transmission and batch data transmission, and improve the efficiency and versatility of the data transmission system.

[0007] To achieve the above objectives, this application proposes a data transmission method, which includes:

[0008] An original data transmission task is generated based on pre-generated data transmission task configuration information, which includes a dual-end data model and task priority.

[0009] Based on the aforementioned dual-end data model, the original data transmission task is transformed into a target data transmission task;

[0010] The target transmission task is given a priority scheduling score based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue.

[0011] Data transmission is performed based on the task scheduling queue and the target data transmission task.

[0012] In one embodiment, the dual-end data model includes a source-end data model, and the step of converting the original data transmission task into a target data transmission task based on the dual-end data model includes:

[0013] If the source data type of the source data model is a streaming data source, then the original data transmission task is converted into a streaming data transmission task;

[0014] If the data granularity of the original data transmission task is a preset granularity specification, then the original data transmission task is converted into a streaming data transmission task;

[0015] If the source data type is not the streaming data source and the data granularity of the original data transmission task is not a preset granularity specification, then the original data transmission task is converted into a batch data transmission task.

[0016] In one embodiment, the data transmission task configuration information further includes a task scheduled execution time, and the step of performing priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue includes:

[0017] Based on the task priority and task scheduled execution time corresponding to the target transmission task, calculate the task priority conversion coefficient and task duration conversion coefficient of the target transmission task respectively;

[0018] Based on the task priority conversion coefficient, the task duration conversion coefficient, and the preset task priority scoring coefficient and task duration scoring coefficient, calculate the task scheduling score of the target transmission task;

[0019] Based on the task priority of the target transmission task and the task scheduling score, the target transmission task is sent to the target priority queue to obtain the task scheduling queue.

[0020] In one embodiment, the dual-end data model includes a source-end data model and a target-end data model, and the method further includes:

[0021] If the data source cluster information corresponding to the source data model and the target data model are different, then the security authentication configuration information of the original data transmission task is obtained, and the security authentication configuration information includes the cluster key distribution center (KDC) information.

[0022] Based on the security authentication configuration information, obtain the ticket authorization ticket through the authentication server corresponding to the KDC information;

[0023] Based on the authorized ticket, a service ticket is obtained from the ticket authorization server corresponding to the KDC information;

[0024] Based on the service ticket, perform service process and cluster security authentication corresponding to the original data transmission task.

[0025] In one embodiment, the step of transmitting data based on the task scheduling queue and the target data transmission task includes:

[0026] Obtain the target data transmission task in the task scheduling queue, as well as the dependencies of the target data transmission task and the predefined source task interface and target task interface;

[0027] Based on the dependencies of the target data transmission task, generate a task topology object corresponding to the target data transmission task;

[0028] Based on the task topology structure object and the source data model number corresponding to the target data transmission task, similar tasks are merged to obtain the task topology relationship; or,

[0029] Based on the task topology structure object and the target end data model number corresponding to the target data transmission task, similar tasks are merged to obtain the task topology relationship;

[0030] Based on the task topology, a data transmission link is generated between the source task interface and the target task interface;

[0031] Data transmission is performed based on the data transmission link between the source task interface and the target task interface.

[0032] In one embodiment, the step of transmitting data based on the data transmission link between the source task interface and the target task interface includes:

[0033] Start the data transmission process and create the source-end transmission framework client group corresponding to the source-end task interface. The source-end transmission framework client group includes a source-end transmission framework client and a source-end transmission framework server.

[0034] Source data is obtained through the source-end transmission framework client and the source-end transmission framework server;

[0035] Create a target-end transmission framework client group corresponding to the target-end task interface. The target-end transmission framework client group includes a target-end transmission framework client and a target-end transmission framework server.

[0036] The source data is written to the target end corresponding to the target task interface through the target end transmission framework client and the target end transmission framework server.

[0037] In one embodiment, the step of transmitting data based on the data transmission link between the source task interface and the target task interface further includes:

[0038] If the source data is streaming data, then the data offset corresponding to the task group ID is obtained based on the pre-acquired task group ID;

[0039] Based on the data offset, the task data consumption point is determined and the source data is obtained based on the task data consumption point.

[0040] Data transmission exceptions are captured based on predefined callback functions, and the exception capture results are obtained.

[0041] Based on the anomaly capture results, flow control is performed on the data transmission process;

[0042] If the source data is batch data, then a target data verification value is generated based on the source data in the target end;

[0043] The integrity of the data transmission task is verified based on the target data verification value and the source data verification value corresponding to the pre-acquired source data.

[0044] Furthermore, to achieve the above objectives, this application also proposes a data transmission device, which includes:

[0045] The generation module is used to generate an original data transmission task based on pre-generated data transmission task configuration information, which includes a dual-end data model and task priority.

[0046] The conversion module is used to convert the original data transmission task into a target data transmission task based on the dual-end data model.

[0047] The scoring module is used to perform priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue;

[0048] An execution module is used to perform data transmission based on the task scheduling queue and the target data transmission task.

[0049] In addition, to achieve the above objectives, this application also proposes a data transmission device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data transmission method as described above.

[0050] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data transmission method described above.

[0051] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data transmission method described above.

[0052] One or more technical solutions proposed in this application generate an original data transmission task based on pre-generated data transmission task configuration information, which includes a dual-end data model and task priority. Based on the dual-end data model, the original data transmission task is converted into a target data transmission task. Based on the task priority and a preset priority scheduling scoring algorithm, the target transmission task is given priority scheduling score to obtain a task scheduling queue. Data transmission is performed based on the task scheduling queue and the target data transmission task. By converting the original data transmission task according to the data model, the converted target data transmission task can be either batch data transmission or streaming data transmission. Furthermore, priority sorting ensures that all tasks are executed in an orderly manner as needed, thereby significantly improving the efficiency and versatility of the data transmission system. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a data transmission method provided in an embodiment of this application;

[0056] Figure 2 A schematic diagram of the core functional modules of a data transmission system provided in this application embodiment.

[0057] Figure 3 This application provides a schematic diagram of the task conversion logic of a data transmission method according to an embodiment of the present application.

[0058] Figure 4 A flowchart illustrating another data transmission method provided in an embodiment of this application;

[0059] Figure 5 A flowchart illustrating another data transmission method provided in an embodiment of this application;

[0060] Figure 6 A schematic diagram of the data transmission task job resource submission timing for a data transmission method provided in this application embodiment;

[0061] Figure 7 This is a schematic diagram of the module structure of a data transmission device provided in an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the data transmission method in the embodiments of this application.

[0063] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0064] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0065] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of this application is to propose a cross-datacenter distributed transmission system that integrates data stream and batch processing. This system aims to solve the problem of efficient and high-quality data transmission across data centers and complex networks in the big data field. Firstly, the system achieves data transmission without requiring mutual trust between the two clusters. Within the data transmission engine module of the same cluster service, secure authentication is simultaneously completed for both the source and target data source clusters. During data transmission, data encryption, decryption, compression, and encoding conversion are performed. Thus, even between Hadoop products from different data centers and different commercial companies, data transmission between multiple independent and heterogeneous clusters can be achieved without establishing mutual trust.

[0067] Secondly, this system simplifies the creation and management of data transmission tasks through visual orchestration and automated scheduling. Data administrators only need to select the data models of the source and destination ends, and the system can automatically generate streaming or batch migration tasks across centers, different networks, and heterogeneous data sources. This eliminates a large amount of parameterized and repetitive configuration work, improves task maintainability, and makes full use of limited bandwidth resources to improve task execution efficiency. The system supports cross-data type data transmission, realizing the transmission capabilities of streaming and batch data within a single system. By implementing custom Source and Sink, the system can achieve data transmission capabilities between any two data types, supporting the combined transmission of more than ten data types such as FTP, HDFS, Hive, MySQL, Oracle, and Kafka.

[0068] Furthermore, by performing data quality verification on batch data and site marking and inspection on streaming data, the system ensures that batch and streaming data are transmitted completely from the source data source to the target data source. This solves the problem of incomplete data transmission caused by various abnormal situations such as complex network conditions, network interruptions, and failures during long-distance data transmission across geographically dispersed data centers. In complex network scenarios, the system improves data transmission quality through dynamic flow control, implementation of at-least-once semantics for streaming data, and data governance integrity verification, ensuring that streaming and batch data can be transmitted completely and accurately from the source data center to the target data source.

[0069] In summary, the solution presented in this system effectively addresses the needs for efficient and high-quality data transmission across various security authentication methods, heterogeneous Hadoop clusters, and different storage formats by employing cross-cluster security authentication, graphical interface orchestration, automated task scheduling, and integrated stream and batch processing.

[0070] Due to existing technical issues, including cross-datacenter trust problems, achieving mutual trust between different data centers and Hadoop products from different commercial companies is extremely difficult, and may even be impossible, due to differences in underlying technical implementation. Furthermore, due to security control and other factors, different data centers have implemented port closures and network isolation, making it impossible to break through network policies and establish network communication connections between data centers.

[0071] Furthermore, current transmission tools rely heavily on manual configuration. They primarily use command-line interfaces for interactive execution, requiring data administrators to manually configure the data to be transmitted, including source and target data, and other relevant parameters to construct the execution command. Moreover, each task can only be configured and executed individually, lacking unified task scheduling capabilities and hindering concurrent task submission and execution, resulting in poor performance. This necessitates a strong dependence on data administrators' knowledge of big data and familiarity with the usage of different data transmission tools.

[0072] Furthermore, there's the issue of migrating data from the same data source format. Current transfer tools only support batch data replication between different clusters, such as HDFS and Hive (with HDFS as the underlying storage). They cannot simultaneously support data transfer between diverse data formats, such as common RDB databases like Oracle to HDFS, FTP to HDFS, and Kafka streaming data. Moreover, they can only complete data replication and transfer; they do not support additional processing operations such as encryption / decryption, compression, encoding conversion, etc., on the migrated data during the data transfer process.

[0073] In addition, multiple solutions need to be developed and maintained. In the current big data processing framework, MapReduce is mainly used to process batch data, while the stream processing framework Flink is mainly used to process streaming data. If the needs of streaming and batch data transmission between multiple heterogeneous clusters and different data forms need to be met at the same time, two data transmission systems need to be developed and maintained simultaneously, which results in excessively high business and maintenance costs.

[0074] Finally, in existing technologies, data storage is often located in different data centers distributed across various regions. The geographical distances between these data centers and the resulting network transmission distances are vast, involving numerous network nodes and significantly increasing the probability of failures. This leads to network anomalies such as intermittent outages and delays. Furthermore, due to the need for secure data transmission, dedicated networks are typically used, which often have limited bandwidth resources.

[0075] Therefore, this application provides a solution that simultaneously completes secure authentication of both the source and target data source clusters within the data transmission engine module of the same cluster service. Simultaneously, it performs data processing such as encryption / decryption, compression, and encoding conversion during data transmission. This invention enables data transmission between multiple independent and heterogeneous clusters without requiring mutual trust between the source and target clusters.

[0076] Through a visual graphical task orchestration interface, data administrators only need to select the source data model (a data model is a detailed description of the data to be migrated; taking the HDFS data model as an example, it mainly describes the data's filename, file encoding, row separator, file path, column names, compression method, and security authentication information of the cluster) and the target data model. Based on the task-related model metadata definition, it automatically generates streaming or batch migration tasks across centers, different networks, and heterogeneous data sources, eliminating a large amount of parameterized and repetitive configuration work. The data model can be reused in multiple transmission tasks, eliminating the need to reconfigure the data model information for each task. Automated scheduling, when connecting to a single cluster or multiple cluster services, divides tasks according to the cluster and submits them concurrently to different cluster services in batches within the same process. The task status is monitored in real time, and when a task fails, a clear error message is given on the task monitoring page, greatly improving task maintainability. At the same time, it monitors the current dedicated network bandwidth usage and makes full use of limited bandwidth resources through reasonable task scheduling and execution, improving task execution efficiency.

[0077] This application implements streaming and batch data transmission capabilities within a single system. By implementing custom Source and Sink, it enables data transmission between any two data formats, supporting combined transmission of over ten mainstream data types, including FTP, HDFS, Hive, MySQL, Oracle, and Kafka. Through data quality verification of batch data and site marking and inspection of streaming data, it ensures that batch and streaming data are transmitted completely from the source data source to the target data source. This solves the problem of incomplete data transmission during long-distance data transmission across geographically dispersed data centers, where complex network conditions and various anomalies such as network interruptions and failures can lead to incomplete data transmission.

[0078] In summary, this invention effectively solves the need for efficient and high-quality data transmission across multiple security authentication methods, heterogeneous Hadoop clusters, and different storage formats by providing cross-cluster security authentication, graphical interface orchestration, automated task generation, automated task scheduling, and integrated stream and batch processing.

[0079] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or data transmission system capable of performing the above functions. The following description uses a data transmission system as an example to illustrate this embodiment and the subsequent embodiments.

[0080] Based on this, the embodiments of this application provide a data transmission method, referring to... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the data transmission method of this application.

[0081] In this embodiment, the data transmission method includes steps S1000 to S4000:

[0082] Step S1000: Generate an original data transmission task based on pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and task priority;

[0083] It should be noted that, in order to solve the problem of efficient and high-quality data transmission across data centers and complex network environments, especially the problems of difficulty in establishing mutual trust between different data centers, the need for manual configuration, support for migration of only a limited number of data source forms, the need to maintain multiple solutions, and the easy loss of data transmitted across complex networks, this application proposes a distributed transmission system solution for cross-data center data stream and batch data transmission. This solution achieves seamless transmission of streaming data and batch data within the same system through the configuration, conversion, scheduling, and execution of data transmission tasks.

[0084] You can refer to Figure 2 , Figure 2 This is a schematic diagram of the core functional modules of the data transmission system provided in the embodiments of this application. The present invention provides a cross-data center distributed transmission system integrating data stream and batch transmission, including a data transmission task management module, a data transmission task scheduling engine module, and a data transmission task execution engine module. This system establishes a data model for transmission tasks, performs configuration management, dynamically judges model parameters, and automatically converts transmission tasks into streaming and batch transmission according to intelligent control rules. At the same time, the configuration parameters include task priorities, and a priority-based task sorting method is provided. The overall solution provides a streaming and batch integrated transmission method based on task priority scheduling, which can intelligently, efficiently, and cost-effectively solve the problem of streaming and batch integrated transmission. It can meet the customer's integrated data needs for streaming and batch transmission while considering task priorities in a single system.

[0085] First, in this embodiment, an original data transmission task is generated based on pre-generated data transmission task configuration information. This configuration information includes a dual-end data model and a task priority. The dual-end data model refers to the source and destination data models, which respectively describe detailed information about the data source and destination, such as data format, type, and storage location. The task priority refers to the urgency of task execution and can be set according to business needs to ensure that critical tasks are executed first. The purpose of generating the original data transmission task is to provide a basic execution framework for subsequent data transmission processes, ensuring that data can be transmitted from the source to the destination according to the predetermined configuration.

[0086] For example, in one feasible implementation, such as Figure 2 The 101 data transmission task management module shown mainly includes data model management and data transmission task configuration management capabilities. Through this module, data administrators only need to select the data model of the source data center, the data model of the target data center, and configure the task priority to quickly create a data transmission task.

[0087] The data model in data transmission tasks is a standardized, formatted, and unified definition of batch and streaming data. It mainly includes: data granularity information (minutes, hours, days, months, years); data column names, data column field types, whether data columns are encrypted, and their encryption algorithms; physical storage information, such as the file server address, file name, file storage path, server address, and authentication information for batch data like HDFS; and topic names and partitions for Kafka streaming data. The data model management capability provides a wizard-driven data model definition through a visual interface. It also supports integration with existing data sources, including various physical storage modules such as HDFS, Hive, and Kafka, allowing data definitions to be retrieved and mapped into the data model.

[0088] Through the data transmission task configuration interface, users can configure whether to perform field encryption / decryption, encoding set (GBK, UTF-8) conversion, delimiter conversion, task scheduled execution time, maximum number of retry attempts for task failure (system maximum of 3 times), and task priority (including low, medium, and high categories).

[0089] After the task is confirmed and saved, the system will save the task information to the database within the data transmission task management module. Throughout the entire data transmission task configuration process, data administrators do not need to focus on the specific execution details of the data transmission task, and no further manual configuration is required to complete the orchestration and definition of the data transmission task.

[0090] In one feasible implementation, the main configuration information of the task can be stored in the database of the data transmission task management module in JSON (JavaScript Object Notation) format. JSON format is a lightweight data exchange format that is easy for humans to read and write. It is mainly implemented through a "name:value" data structure, which can define multiple sets of names and their values, separated by commas. The parameters filled in by data administrators through the visual interface are converted into the following JSON format, for example, the following example parameters:

[0091] {

[0092] "reader.name":"hdfsreader", / / Source task type

[0093] "writer.name":"hdfswriter", / / Target task type

[0094] "fieldnames":"a|string,b|int", / / Field names and types

[0095] "encrypt.type":"SM4", / / Encryption algorithm

[0096] "data.interval":"hour", / / Data period

[0097] "read.filepath":" / tmp / ${PROVINCE}", / / file path

[0098] "read.filename":"*.dat", / / filename

[0099] "kafka.read.topic":"testA3", / / kafak topic

[0100] "indexArray":"1,3", / / Decrypted field index

[0101] "source.separator":"|", / / Source data separator

[0102] "target.separator":"#", / / Target data separator

[0103] "charset":"UTF-8", / / Character encoding

[0104] "dms.dsid":"defaultCluster", / / Cluster information

[0105] "priority":"medium", / / task priority

[0106] "task.retry.times":3, / / Number of retries for task failure

[0107] "task.wait.time": 2024040112300, / / Task scheduled execution time

[0108] "kdc.filepath":" / tmp / conf / krb5.conf", / / KRB5 configuration path, used to connect to the Kerberos server.

[0109] "keytab.filepath":" / tmp / conf / ldapuser.keytab", / / The keytab file used for KDC authentication

[0110] }

[0111] Step S2000: Based on the dual-end data model, convert the original data transmission task into a target data transmission task;

[0112] It should be noted that in this embodiment, based on a dual-end data model, the original data transmission task is converted into a target data transmission task. The purpose of this step is to intelligently convert the task into a suitable transmission format—either a streaming data transmission task or a batch data transmission task—according to different data source types and data granularities. For example, if the source data model indicates a streaming data source, or if the data granularity conforms to a preset streaming data granularity specification, then the original data transmission task will be converted into a streaming data transmission task. This conversion is to adapt to the transmission requirements of different data characteristics, optimize resource utilization, and improve transmission efficiency.

[0113] Step S3000: Based on the task priority and the preset priority scheduling scoring algorithm, perform priority scheduling scoring on the target transmission task to obtain the task scheduling queue;

[0114] It should be noted that, in this embodiment, after converting the original data transmission task into a streaming data transmission task or a batch data transmission task, the target transmission task can be prioritized and scored based on task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue. In this step, the system calculates the task priority conversion coefficient and the task duration conversion coefficient according to the task priority and the planned execution time of the target transmission task, respectively. Then, combined with the preset task priority scoring coefficient and task duration scoring coefficient, the system calculates the task scheduling score of the target transmission task. Based on the task scheduling score, the system sends the target transmission task to the corresponding target priority queue to form a task scheduling queue. The system performs this operation to reasonably arrange the order of task execution, ensuring that high-priority tasks can be executed in a timely manner, while also taking into account the planned execution time of the tasks to optimize the overall scheduling efficiency. In a specific implementation, the system automatically calculates the scheduling score of each task according to the urgency of the task and the predetermined execution time, and sorts the tasks according to the score to ensure that critical tasks are processed first.

[0115] Step S4000: Perform data transmission based on the task scheduling queue and the target data transmission task.

[0116] It should be noted that in this embodiment, data transmission can be performed based on a task scheduling queue and a target data transmission task. By obtaining the target data transmission task and its dependencies in the task scheduling queue, as well as the predefined source and target task interfaces (Source and Sink), the system can generate a task topology object corresponding to the target data transmission task. Then, tasks of the same type are merged according to the task topology object and the data model number to form a task topology relationship. Next, a data transmission link between the source and target task interfaces is generated based on the task topology relationship, and data transmission is performed based on these links. The purpose of this step is to ensure that data can be successfully transmitted from the source task interface to the target task interface in a predetermined order and manner, completing the cross-data center data transmission. For example, in one specific implementation, the system automatically matches the corresponding source and target interfaces based on the information in the task scheduling queue, establishes a data transmission link, and then begins the data transmission operation to ensure that data is accurately transmitted from one data center to another.

[0117] In one feasible implementation, the dual-end data model includes the source-end data model, and step S2000 may include steps S2100 to S2300:

[0118] Step S2100: If the source data type of the source data model is a streaming data source, then the original data transmission task is converted into a streaming data transmission task;

[0119] Step S2200: If the data granularity of the original data transmission task is a preset granularity specification, then the original data transmission task is converted into a streaming data transmission task;

[0120] Step S2300: If the source data type is not the streaming data source and the data granularity of the original data transmission task is not the preset granularity specification, then the original data transmission task is converted into a batch data transmission task.

[0121] It should be noted that, in this embodiment, the dual-end data model includes a source data model and a target data model. The two models define the characteristics of the starting point and the ending point of the transmitted data, including attributes such as data type, storage location, and format. In the process of converting the original data transmission task into the target data transmission task, the system will intelligently judge and determine the transmission form of the task based on the type and data granularity of the source data model, i.e., a streaming data transmission task or a batch data transmission task. Streaming data sources refer to those data sources that continuously generate data, such as Kafka and real-time monitoring systems, while batch data transmission tasks are suitable for non-real-time, periodic data transmission needs. Data granularity involves the data update frequency, such as data updated by minute, hour, day, etc. This conversion is to adapt to the transmission needs of different data characteristics, optimize resource utilization, and improve transmission efficiency.

[0122] Additionally, it should be noted that the planned execution time of a task refers to the specific time at which the task is scheduled to begin execution. In the task priority scheduling scoring algorithm, the system combines task priority and planned execution time to calculate the task priority conversion coefficient and task duration conversion coefficient, thereby deriving the task scheduling score. This score determines the order of tasks in the queue, ensuring that critical tasks can be executed in a timely manner, while also taking into account the planned execution time of tasks to optimize overall scheduling efficiency.

[0123] In one feasible implementation, the system can first generate an original data transmission task based on the data transmission task configuration information. Then, it analyzes the source data model. If the source data type is a streaming data source, or the data granularity conforms to the preset streaming data granularity specification, the system will convert the original data transmission task into a streaming data transmission task. Otherwise, if the source data type is not a streaming data source and the data granularity does not conform to the preset granularity specification, the task will be converted into a batch data transmission task. Next, the system will calculate the task scheduling score based on the task priority and planned execution time, and send the task to the corresponding priority queue according to the score to form a task scheduling queue. During the data transmission phase, the system will perform data transmission based on the task scheduling queue and the target data transmission task, including obtaining task dependencies, generating task topology objects, merging similar tasks, generating data transmission links, and finally performing data transmission based on these links.

[0124] For example, refer to Figure 3 , Figure 3This application provides a schematic diagram of task conversion logic. In another feasible implementation, based on the source and target data information in the original data transmission task information saved in the above steps, a Flink data transmission task is automatically generated. By analyzing the source and target data model information of the task and whether the task involves data encryption / decryption, encoding conversion, or other processing operations, the optimal Flink task is intelligently generated. This includes two task types: Flink batch data processing tasks and Flink streaming data processing tasks, such as... Figure 3 In one implementation, if data needs to be stored on a system other than HDFS (i.e., the write.filepath field is not configured to start with hdfs), such as a database or local file system (FTP), the task is converted into a Flink batch task. If it is a streaming data source, such as middleware like Kafka (i.e., the reader.name configuration parameter is kafkareader), the task is converted into a Flink streaming task. If the data model is a finer data granularity such as hours or minutes, the task is converted into a Flink streaming task (i.e., data.interval is hour or minute). If the data transmission task requires encryption, decryption, or other conversion operations on specified fields of the data (i.e., there is an inconsistency between source.separator and target.separator or there is an encrypt.type configuration), the task is converted into a Flink batch task. Other data transmission tasks that do not meet the above specific conditions are converted into Flink batch tasks.

[0125] In one feasible implementation, the data transmission task configuration information further includes a task scheduled execution time, and step S3000 may include steps S3100 to S3300:

[0126] Step S3100: Based on the task priority and task scheduled execution time corresponding to the target transmission task, calculate the task priority conversion coefficient and task duration conversion coefficient of the target transmission task respectively;

[0127] Step S3200: Calculate the task scheduling score of the target transmission task based on the task priority conversion coefficient, the task duration conversion coefficient, and the preset task priority scoring coefficient and task duration scoring coefficient;

[0128] Step S3300: Based on the task priority of the target transmission task and the task scheduling score, send the target transmission task into the target priority queue to obtain the task scheduling queue.

[0129] It should be noted that, in this embodiment, the planned execution time of a task refers to the specific time at which the task is scheduled to begin execution. In the task scheduling and scoring algorithm, the system comprehensively considers the task's priority and planned execution time, calculating a task priority conversion coefficient and a task duration conversion coefficient. The task priority conversion coefficient reflects the degree of influence of task priority on the scheduling score, while the task duration conversion coefficient reflects the impact of the urgency of the task's planned execution time on the score. Through these two coefficients, the system can more accurately assess the execution priority of each task, ensuring that critical and urgent tasks receive higher scheduling priority.

[0130] Additionally, it should be noted that the system calculates a task scheduling score for each target transmission task based on preset task priority and task duration scoring coefficients, combined with the two conversion coefficients mentioned above. This score considers not only the task's priority and planned execution time but also the system's overall scheduling strategy and resource allocation, thereby ensuring the fairness and efficiency of task scheduling. Based on the task scheduling score, the system sends the task to the corresponding priority queue, thus forming a task scheduling queue. This queue guides subsequent data transmission operations, ensuring that tasks are executed in the predetermined order and according to plan.

[0131] In one feasible implementation, the system first receives data transmission task configuration information, including the dual-end data model and task priority. Based on the type and granularity of the source data model, the system intelligently converts the original data transmission task into a streaming data transmission task or a batch data transmission task. Subsequently, the system calculates task priority conversion coefficients and task duration conversion coefficients based on task priority and planned execution time, and combines these with preset scoring coefficients to derive a scheduling score for each task. Based on this score, the system assigns tasks to different priority queues, forming task scheduling queues. During the data transmission phase, the system performs data transmission operations according to the task scheduling queues and the target data transmission task. This includes obtaining task dependencies, generating task topology objects, merging similar tasks, generating data transmission links, and transmitting data based on these links. Throughout the process, the system also handles cross-data center security authentication to ensure data transmission security. Through these steps, the system can achieve seamless transmission of streaming and batch data within the same system, meeting the high-quality and high-efficiency data transmission requirements between different data centers.

[0132] Specifically, in another feasible implementation, refer to Figure 3As shown, the 102 data transmission task scheduling engine module retrieves tasks to be submitted for execution in batches from the task configuration database. It uses a task priority scoring algorithm to comprehensively score the tasks to ensure that tasks are executed on demand and in an orderly manner. The system pre-sets two task queues: a high-priority queue for key tasks and a general task queue. Task priorities can be configured on the system's visual interactive page. Data administrators can manually set priorities to high, general, or low based on the business importance of the data to be migrated. Tasks set to high priority will be placed in the high-priority queue and submitted and executed before other tasks. Tasks with low priority will be placed in the low-priority queue. The system calculates the priority score of the corresponding data transmission task according to the task priority scoring formula and places the data transmission tasks into the corresponding task queues according to the score results from high to low. In this embodiment, a feasible scoring formula is as follows:

[0133] S = A * 0.6 + B * 0.3

[0134] Where S is the task scheduling score, A is the task configuration priority (e.g., high is converted to a value of 3, medium to 2, and low to 1 before being included in the scoring formula), and B is the task absolute waiting time, calculated as B = |System.T - CronT| / 60, which is obtained by subtracting the task's scheduled execution time from the current operating system time and dividing by 60 to convert it to hours. System.T is the current operating system time. Taking Java as an example, the current operating system time is obtained through the LocalDateTime.now() function, and the task's scheduled execution time CronT is obtained from the task.wait.time field in the task configuration information.

[0135] The real-time priority score of each data transmission task to be submitted and started is calculated using a task priority scoring formula. For example, if a data administrator creates two data transmission tasks through the data task management module in step 1, both scheduled to execute at 00:00:00 on January 1, 2024, with data transmission task 1 configured as having a high priority and task 2 as having a low priority, and the system time is currently 23:00:00 on December 31, 2023, then the scheduling score calculation for task 1 is S = 3 * 0.6 + |20231231230000 - 20240101000000| / 60 * 0.3, resulting in a score of 2.1. Using the same formula, task 2's score is 0.9. Based on the task scores, task 1, with the higher score, will be placed in the task queue before task 2. This method ensures that key tasks are executed first and key data is transmitted first, ensuring that all output transmission tasks are executed on demand and in an orderly manner, effectively reducing the average waiting time of tasks.

[0136] This embodiment provides a data transmission method. It generates an original data transmission task based on pre-generated data transmission task configuration information, including a dual-end data model and task priorities. Based on the dual-end data model, the original data transmission task is converted into a target data transmission task. The target transmission task is then prioritized and scored using the task priorities and a preset priority scheduling and scoring algorithm to obtain a task scheduling queue. Data transmission is performed based on the task scheduling queue and the target data transmission task. This method transforms the original data transmission task according to the data model. The converted target data transmission task can be either batch data transmission or streaming data transmission. Furthermore, priority sorting ensures that all tasks are executed on demand and in an orderly manner, thereby significantly improving the efficiency and versatility of the data transmission system.

[0137] Based on the first embodiment provided in this application, in another embodiment provided in this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 The dual-end data model includes a source-end data model and a target-end data model, and the data transmission method further includes steps A1000 to A4000:

[0138] Step A1000: If the data source cluster information corresponding to the source data model and the target data model is different, then obtain the security authentication configuration information of the original data transmission task. The security authentication configuration information includes the cluster key distribution center (KDC) information.

[0139] Step A2000: Based on the security authentication configuration information, obtain the ticket authorization ticket through the authentication server corresponding to the KDC information;

[0140] Step A3000: Based on the ticket authorization ticket, obtain the service ticket through the ticket authorization server corresponding to the KDC information;

[0141] Step A4000: Perform service process and cluster security authentication corresponding to the original data transmission task based on the service ticket.

[0142] It should be noted that, in this embodiment of the application, if the data source cluster information of the source and target ends is different during data transmission, that is, they belong to different data centers or are provided by different data sources, the system needs to obtain the security authentication configuration information of the original data transmission task to ensure the security of data when it is transmitted across data centers. The security authentication configuration information mainly includes the cluster key distribution center (KDC Key Distribution Center) information, which is a key component for implementing the Kerberos authentication protocol. It is responsible for distributing ticket authorization tickets and server tickets, thereby ensuring identity authentication and authorization during data transmission.

[0143] Additionally, it should be noted that the system obtains a Ticket Authorization Token (TGT) from the authentication server corresponding to the KDC information based on the security authentication configuration information. This is the first step for a user or service process to access Kerberos authentication system resources. After obtaining the TGT, the system uses it to request a Service Ticket (ST) from the KDC's Ticket Authorization Server (TGS). This step is a necessary verification process before a user or service process can access a specific service. Once the Service Ticket is obtained, the system will perform security authentication between the service process corresponding to the original data transmission task and the cluster based on the ticket, ensuring the security and reliability of data transmission.

[0144] For example, in one feasible implementation, the system first generates a raw data transmission task based on the dual-end data model and converts it into a target data transmission task. Subsequently, the system evaluates the task's priority and planned execution time, calculates the task's priority conversion coefficient and duration conversion coefficient, and derives a task scheduling score accordingly, placing the task into the corresponding priority queue. During the data transmission preparation phase, the system obtains security authentication configuration information based on the differences between the source and target data models, and uses KDC information for identity authentication and authorization to ensure data transmission security. During the data transmission phase, the system interfaces with different data source types through custom Source and Sink objects, and performs data transmission based on the task topology object and task scheduling queue. In this process, the system also handles cross-data center security authentication to ensure data transmission security. Through these steps, the system can achieve seamless transmission of streaming and batch data within the same system, meeting the high-quality and high-efficiency data transmission requirements between different data centers.

[0145] More specifically, when the source data source cluster and the target data source cluster are located in different KDC centers, it is necessary to simultaneously implement security authentication for the source and target clusters in the same task through multiple processes in the data transmission cluster service. First, the relevant configuration information required for Kerberos security authentication of the connected cluster can be obtained from the task configuration information JSON, which mainly includes two files: krb5.conf and Keytab. The krb5.conf configuration file mainly includes KDC information, realm information, and lib configuration information in the cluster domain. The data transmission engine service process performs security authentication with the source data cluster through the security authentication file. The data transmission engine service process acts as a client, carrying authentication information, and requests a Ticket Authorization Token (TGT) from the Authentication Server in the KDC. After the KDC confirms that the client's identity has permission to access the KDC, it returns the TGT to the client.

[0146] The client accesses the TGS carrying a TGT (Ticket Token) to request a Server Ticket. The TGS in the KDC checks if the user's identity in the TGT matches the requesting user to prevent impersonation. After confirmation, the TGS returns the Server Ticket for the user's service access. The client then uses the Server Ticket to access the required service, such as HDFS or Kafka. The service checks if the requested Server IP exists and verifies the Server Ticket content against the client's identity. Once these steps are successfully completed, the service process has completed security authentication with the cluster and can access data from big data components such as HDFS, Hive, and Kafka. Repeating these steps in another process completes security authentication with the target data cluster. Through these steps, the data transmission engine completes cross-KDC security authentication for the heterogeneous Hadoop clusters, allowing subsequent task submissions and data manipulation steps to proceed.

[0147] This embodiment provides a data transmission method. If the source data model and the target data model correspond to different data source cluster information, the method obtains security authentication configuration information for the original data transmission task. This security authentication configuration information includes cluster key distribution center (KDC) information. Based on the security authentication configuration information, a ticket authorization ticket is obtained through the authentication server corresponding to the KDC information. Based on the ticket authorization ticket, a service ticket is obtained through the ticket authorization server corresponding to the KDC information. Based on the service ticket, service process and cluster security authentication corresponding to the original data transmission task are performed to ensure the security of data transmission.

[0148] Based on the first embodiment provided in this application, in another embodiment provided in this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Step S4000 may include steps S4100 to S4600:

[0149] Step S4100: Obtain the target data transmission task in the task scheduling queue, as well as the dependency relationship of the target data transmission task and the predefined source task interface and target task interface;

[0150] Step S4200: Generate a task topology structure object corresponding to the target data transmission task based on the dependency relationship of the target data transmission task;

[0151] Step S4300: Based on the task topology structure object and the source data model number corresponding to the target data transmission task, merge similar tasks to obtain the task topology relationship; or,

[0152] Step S4400: Based on the task topology structure object and the target end data model number corresponding to the target data transmission task, merge similar tasks to obtain the task topology relationship;

[0153] Step S4500: Generate a data transmission link between the source task interface and the target task interface according to the task topology relationship;

[0154] Step S4600: Data transmission is performed based on the data transmission link between the source task interface and the target task interface.

[0155] It should be noted that in this embodiment, the target data transmission tasks in the task scheduling queue are matched with the task interfaces of the source and target ends, and a data transmission link is constructed based on this information. The purpose is to ensure that data can be transmitted smoothly and efficiently from the source end to the target end. The task scheduling queue contains data transmission tasks to be executed, i.e., target data transmission tasks. These tasks are sorted according to priority and other parameters and then placed into the corresponding high-priority task queue or low-priority task queue. The system analyzes the dependencies of tasks and data model numbers, merges similar tasks, optimizes resource allocation, reduces redundant operations, and improves transmission efficiency.

[0156] Additionally, it's important to note that the task topology object maps the dependencies between tasks. It helps the system understand the hierarchical structure and data flow between tasks. Through this structure, the system can generate accurate data transmission links, ensuring data is transmitted along the predetermined path. During this process, the system also considers data integrity and consistency, especially during cross-data center transmission, using network anomaly handling mechanisms and data verification techniques to ensure data reliability.

[0157] In one feasible implementation, the system first extracts the target data transmission task from the task scheduling queue and identifies its dependencies and corresponding source and target task interfaces. Then, the system generates a task topology object and merges tasks with the same source or target data model number to optimize the transmission path. Based on the task topology, the system creates a data transmission link between the source and target task interfaces. On this link, the system initiates the data transmission process, acquiring data through the source transmission framework client and server, and writing the data to the target using the target transmission framework client and server. Throughout the transmission process, the system monitors network status and data transmission health, dynamically controls traffic, and performs data integrity verification after transmission to ensure accurate data transmission from the source to the target. In this way, the system achieves efficient and high-quality data transmission across data centers, meeting the transmission needs of different data source types while reducing business and maintenance costs.

[0158] More specifically, in yet another feasible implementation, refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the timing of data transmission task job resource submission for a data transmission method provided in this application embodiment. After determining the priority scores of several target data transmission tasks, the data transmission tasks are then... Figure 2 The scheduling engine 102 for data transmission tasks, as shown, retrieves the stream or batch of Flink tasks generated in the aforementioned steps from the high-priority queue and the low-priority queue, and submits them to the task job resource cluster. Its main submission and startup steps are as follows: Figure 6 As shown, the 102 data scheduling engine submits data transmission tasks to the Flink cluster. The Flink cluster sends a request for task execution resources to YARN (the resource manager in the Hadoop ecosystem). YARN then returns available execution resources. Flink starts the Application Master (the Application Master is the manager of each application, responsible for the application's lifecycle management, including requesting resources from the ResourceManager, coordinating container startup / stop with the NodeManager, etc. The Application Master is also responsible for monitoring the application's status to ensure that tasks can run normally) to request resources and monitor the application's running status. YARN returns the Application Master address, which can be used to access the task and view detailed information. After that, Flink submits the task to the Application Master. The Application Master requests container resources from YARN, which then returns the container resources. The Application Master starts the TaskManager in the returned container resources to manage the startup and execution of tasks. Finally, Flink sends task instructions to the TaskManager to process data. After the task starts successfully, the TaskManager returns the execution result.

[0159] After submitting the target data transfer task to Yarn Figure 3 The data transmission task execution engine module 103 shown can connect to different data source types through custom source-side task interfaces (Source and Sink) implemented by the program. When the task starts, it automatically creates a Source implementation for connecting to the source data source and a Sink implementation for connecting to the target data source based on the data types of the source and target. The main steps are as follows:

[0160] Source is responsible for reading data from the source data source into the data transmission engine. For different data sources, different Handler implementations are defined, and data is read from the source data source through JDBC, FileSystem-Read, Kafka Consumer, etc.

[0161] The Sink is responsible for delivering data from the data transmission engine to the target data source. For different data sources, different Handler implementations are defined to retrieve data from the named pipe and write it to the target data source through JDBC, FileSystem-Write, Kafka Consumer, etc.

[0162] By combining different Source and Sink pairs, one-to-one data transfer between any two data sources can be achieved. In actual business production, there may be a need for multi-source reading and multi-source writing, where data is distributed across different data sources. For example, daily data might be stored in Hive, while monthly data might be stored in HDFS. In this case, it's necessary to migrate data from two different data sources simultaneously to the same or more target data sources. The specific implementation steps are as follows:

[0163] The task of page arrangement is transformed into Node and Edge objects Content that can be described and recognized by a programming language according to the dependency relationship. It is a JSON body, where Node is used to store the type and parameter information of each node, and Edge is used to store the connection information between nodes.

[0164] Iterate through each Node and generate a TaskTopology object for that Node. Each object represents the list of the Node's predecessor nodes, the Node itself, and the list of its successors.

[0165] The system iterates through the nodes. When a node is identified as a UNION node, it retrieves the list of its predecessor nodes, performs UNION processing on the data extracted from the predecessor nodes by the Source Handler using FlinkSQL, and then temporarily stores it in Flink's DataStream for further processing.

[0166] After generating the task topology, a merging analysis is performed on the tasks: by comparing the source data source data model definition of the data transmission task with the source data model definition of existing tasks, if their unique IDs are the same, multiple data transmission tasks will be merged and added to the existing task topology, forming a data transmission link of 1 Source to N Sinks. This method significantly improves data transmission efficiency, effectively reduces the number of RPC requests to the source data cluster, and in the optimal state, the source data that originally needed to be read and transmitted N times is reduced to only once, greatly reducing network bandwidth consumption. More data transmission tasks can be completed in the same amount of time, significantly improving data transmission efficiency.

[0167] In one feasible implementation, step S4600 may include steps S4610 to S4640:

[0168] Step S4610: Start the data transmission process and create the source-end transmission framework client group corresponding to the source-end task interface. The source-end transmission framework client group includes the source-end transmission framework client and the source-end transmission framework server.

[0169] Step S4620: Obtain source data through the source-end transmission framework client and the source-end transmission framework server;

[0170] Step S4630: Create a target end transmission framework client group corresponding to the target end task interface, wherein the target end transmission framework client group includes a target end transmission framework client and a target end transmission framework server;

[0171] Step S4640: Write the source data into the target end corresponding to the target end task interface through the target end transmission framework client and the target end transmission framework server.

[0172] It should be noted that, in the embodiments of this application, the execution of the data transmission task first requires starting the data transmission process and creating a transmission framework client group corresponding to the source task interface, including the source transmission framework client and the server. The source transmission framework client is responsible for interacting with the data source, while the server processes client requests and prepares data for transmission. Through this mechanism, the system can efficiently obtain data from the source, laying the foundation for subsequent transmission.

[0173] Additionally, it should be noted that after acquiring the source data, the system will create a transport framework client group corresponding to the target task interface, including the target transport framework client and server. The target transport framework client is responsible for writing the received data to the target data source, while the server ensures that the data is stored and updated correctly. The purpose of this step is to ensure that the data can not only be successfully read from the source but also accurately written to the target, completing the entire data transmission process.

[0174] For example, in one feasible implementation, the system uses Netty (an asynchronous event-driven network application framework) to transfer data read by the Source to the Sink across the authentication process. Since the Source and Sink are connected to the KDCs on both ends for security authentication and are in different processes, data needs to be sent from the Source process to the Sink process. First, a Source-side NettyClient-Server group, i.e., the source-side transport framework client group, is created in advance. It contains two parts: NettyClient, the source-side transport framework client, which implements ReadProcessInstanceNetty, a child process running Netty, which starts a source-side transport framework client and carries the Handler implementation of the source-side task interface Source, and sends the data read by the Handler to the source-side transport framework server. Then, through ChildProcessNettySource, a source-side transport framework server NettyServer is started, and the data written by the source-side transport framework server is passed into the Flink main process.

[0175] In this embodiment, the Flink main task process receives data from the source transport framework server, processes it using intermediate operators, and then outputs the processed data through the target transport framework client at the sink end for use by the sink end. First, a NettyClient-Server group is created at the sink end, which is also the target transport framework client group. This group contains two parts: the target transport framework client NettyClient and the target transport framework server. The target transport framework client NettyClient is created in the Flink main process, a ChildProcessNettySink instance is created, and the target transport framework client at the sink end is started. The upstream message is encapsulated and written to the target transport framework server. The target transport framework server starts a NettyServer child process and carries the corresponding target processor SinkHandler plugin to receive the data written by the target transport framework client. At the same time, the plugin writes the data to the target end through JDBC, FileSyStem-Write, KafkaProducer, etc., to complete the data transmission.

[0176] In one feasible implementation, step S4600 may further include steps S4611 to S4616:

[0177] Step S4611: If the source data is streaming data, then obtain the data offset corresponding to the task group ID based on the pre-acquired task group ID;

[0178] Step S4612: Determine the task data consumption point based on the data offset and obtain the source data based on the task data consumption point;

[0179] Step S4613: Perform data transmission exception capture according to the predefined callback function and obtain the exception capture result;

[0180] Step S4614: Perform flow control on the data transmission process based on the anomaly capture result;

[0181] Step S4615: If the source data is batch data, then generate a target data verification value based on the source data in the target end;

[0182] Step S4616: Perform data transmission task integrity verification based on the target end data verification value and the source end data verification value corresponding to the pre-acquired source end data.

[0183] It should be noted that in this embodiment, data transmission faces challenges across data centers and complex, unstable networks. This necessitates addressing data loss due to network anomalies such as packet loss and intermittent outages during data transmission, ensuring data can be correctly transmitted from the source data source to the target data source. This invention improves data transmission quality through dynamic flow control, at-least-once semantics for streaming data, and data governance integrity verification. This ensures that streaming and batch data can be transmitted completely and accurately from the source data center to the target data source. For streaming data, the system obtains the corresponding data offset based on the task group ID, a metric used in streaming data sources like Kafka to record consumption progress. The data offset allows the system to start reading data from the correct position, ensuring data continuity and integrity. During the reading process, the system defines callback functions to capture anomalies in data transmission, crucial for dynamically adjusting transmission strategies and handling network fluctuations. Based on the anomaly capture results, the system performs flow control on the data transmission process to adapt to network conditions and the receiving end's processing capabilities. Furthermore, for batch data, the system performs data consistency and integrity checks after the data transmission task is completed. By comparing the data checksum value (CheckSumSource) generated through arithmetic operations on the source data with the data checksum value (CheckSumSink) recalculated at the target end, the system can verify whether the data has changed or been lost during transmission. This process is crucial for ensuring the accuracy and reliability of batch data.

[0184] In one feasible implementation, the system first generates a data transmission link based on the target data transmission tasks and their dependencies in the task scheduling queue, as well as predefined source and target task interfaces. For streaming data, the system obtains the data offset corresponding to the task group ID and determines the consumption point based on this offset, thereby accurately retrieving data from the source data source. During data transmission, the system monitors and captures any anomalies, such as transmission timeouts or data inconsistencies, and dynamically adjusts the data flow based on these anomaly results to ensure the stability and efficiency of data transmission. For batch data, after data transmission is completed, the system compares the data checksums at the source and target ends to verify data integrity. If the checksums are inconsistent, the system initiates a data supplementation task to retransmit the missing data, ensuring the accuracy of data transmission. In this way, the system can achieve efficient and high-quality transmission of streaming and batch data in complex network environments across data centers.

[0185] Specifically, in another feasible implementation, the streaming data is marked and inspected to ensure that no data is lost during transmission; at the same time, the data flow at the source end is limited to prevent excessive write pressure on the sink end data source, which could lead to incorrect data writing. The specific implementation logic is as follows:

[0186] After establishing a Kafka connection to read data from the source, the task first retrieves the offset corresponding to the `group.id` by calling the `listConsumerGroupOffsets` method in the `AdminClient` class provided by Kafka. If this is the first time the task is consuming Kafka data, the retrieved offset will be empty, and consumption will begin from the earliest Kafka data. If the offset data is obtained, the task specifies the consumption position using the `KafkaConsumer.seek` method when consuming the Kafka message, ensuring that data continues to be consumed from the previously saved position. During the writing of Kafka data, the `send` method in `KafkaProducer` is used to write the data passed from the upstream, along with a custom callback function `KafkaSendCallback`. If an exception occurs during the sending of Kafka data, the exception is caught in this callback function.

[0187] By analyzing the anomalies, if continuous data sending timeouts occur, it is inferred that the downstream Kafka receiving capacity is insufficient. In this case, flow control will be implemented on the source end. If it is determined that flow control on the source end is necessary, the sink process will send a flow control signal to the source process through Netty communication. After receiving the flow control signal, the source process will use the RateLimiter capability provided by Guava to count the rate. If the rate exceeds the configured rate, it will wait while transmitting data until the rate is lower than the flow control configuration.

[0188] For batch data, after the data transmission task is completed, the system checks the data consistency and integrity. It generates a checksum value (CheckSumSource) by performing arithmetic operations on the source data. After the data is migrated to the target data source, a new CheckSumSink is generated. Taking a file as an example, the calculation logic for the CheckSum checksum is as follows: First, the file is divided into fixed-size blocks (each block is 64MB). A hash calculation is performed on each block to generate a fixed-length checksum. Then, the checksums of all blocks are concatenated to form a long checksum string. This long checksum string is hashed again to generate the final checksum value (CheckSum). Finally, the checksums of the source and target data are compared. If the checksums are inconsistent, it indicates that data loss occurred during data transmission. At this point, a data quality verification task is initiated. This task compares the specific content of the data at both ends, listing the file and line number where the missing data is located, and uses this as input to automatically initiate a data replenishment task. This task retransmits the missing data from the source data source to the target data source.

[0189] Based on the above steps, a high-quality and efficient data transmission capability has been built for streaming and batch data across data centers, multi-terminal KDC authentication, cross-data source types, and data transformation and processing within the same system.

[0190] In another embodiment provided in this application, by obtaining the target data transmission task in the task scheduling queue, as well as the dependency relationship of the target data transmission task and the predefined source-end task interface and target-end task interface; generating a task topology structure object corresponding to the target data transmission task based on the dependency relationship of the target data transmission task; merging similar tasks based on the task topology structure object and the source-end data model number corresponding to the target data transmission task to obtain a task topology relationship; or, merging similar tasks based on the task topology structure object and the target-end data model number corresponding to the target data transmission task to obtain a task topology relationship; generating a data transmission link between the source-end task interface and the target-end task interface according to the task topology relationship; and performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface, efficient management and optimized resource allocation of data transmission tasks can be achieved. At the same time, through intelligent merging of task topologies and accurate generation of data transmission links, the efficiency and reliability of cross-data center data transmission are greatly improved.

[0191] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data transmission method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0192] Please refer to Figure 7 This application also provides a data transmission device, the data transmission device comprising:

[0193] The generation module is used to generate an original data transmission task based on pre-generated data transmission task configuration information, which includes a dual-end data model and task priority.

[0194] The conversion module is used to convert the original data transmission task into a target data transmission task based on the dual-end data model.

[0195] The scoring module is used to perform priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue;

[0196] An execution module is used to perform data transmission based on the task scheduling queue and the target data transmission task.

[0197] The data transmission apparatus provided in this application, employing the data transmission method in the above embodiments, can solve the technical problem of data transmission. Compared with the prior art, the beneficial effects of the data transmission apparatus provided in this application are the same as those of the data transmission method provided in the above embodiments, and other technical features in the data transmission apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0198] This application provides a data transmission device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data transmission method in Embodiment 1 above.

[0199] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a data transmission device suitable for implementing embodiments of this application. The data transmission device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The data transmission device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0200] like Figure 8 As shown, the data transmission device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data transmission device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the data transmission device to communicate wirelessly or wiredly with other devices to exchange data. Although data transmission devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0201] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0202] The data transmission device provided in this application adopts the data transmission method in the above embodiments. Compared with the prior art, the beneficial effects of the data transmission device provided in this application are the same as those of the data transmission method provided in the above embodiments. Furthermore, the other technical features of the data transmission device are the same as those disclosed in the method of the previous embodiment, and will not be described in detail here.

[0203] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0204] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0205] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data transmission method in the above embodiments.

[0206] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0207] The aforementioned computer-readable storage medium may be included in a data transmission device or may exist independently without being assembled into a data transmission device.

[0208] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0210] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0211] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data transmission method, thereby solving the technical problem of data transmission. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data transmission method provided in the above embodiments, and will not be repeated here.

[0212] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data transmission method described above.

[0213] The computer program product provided in this application can solve the technical problem of data transmission. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the data transmission method provided in the above embodiments, and will not be repeated here.

[0214] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data transmission method, characterized in that, The method includes: An original data transmission task is generated based on pre-generated data transmission task configuration information, which includes a dual-end data model and task priority, and the dual-end data model includes a source-end data model. Based on the aforementioned dual-end data model, the original data transmission task is transformed into a target data transmission task, including: If the source data type of the source data model is a streaming data source, then the original data transmission task is converted into a streaming data transmission task; If the data granularity of the original data transmission task is a preset granularity specification, then the original data transmission task is converted into a streaming data transmission task; If the source data type is not the streaming data source and the data granularity of the original data transmission task is not the preset granularity specification, then the original data transmission task is converted into a batch data transmission task. The target data transmission task is given a priority scheduling score based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue. Data transmission is performed based on the task scheduling queue and the target data transmission task.

2. The method as described in claim 1, characterized in that, The data transmission task configuration information also includes the task scheduled execution time. The step of performing priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain the task scheduling queue includes: Based on the task priority and task scheduled execution time corresponding to the target transmission task, calculate the task priority conversion coefficient and task duration conversion coefficient of the target transmission task respectively; Based on the task priority conversion coefficient, the task duration conversion coefficient, and the preset task priority scoring coefficient and task duration scoring coefficient, calculate the task scheduling score of the target transmission task; Based on the task priority of the target transmission task and the task scheduling score, the target transmission task is sent to the target priority queue to obtain the task scheduling queue.

3. The method as described in claim 1, characterized in that, The dual-end data model includes a source-end data model and a target-end data model, and the method further includes: If the data source cluster information corresponding to the source data model and the target data model are different, then the security authentication configuration information of the original data transmission task is obtained, and the security authentication configuration information includes the cluster key distribution center (KDC) information. Based on the security authentication configuration information, obtain the ticket authorization ticket through the authentication server corresponding to the KDC information; Based on the authorized ticket, a service ticket is obtained from the ticket authorization server corresponding to the KDC information; Based on the service ticket, perform service process and cluster security authentication corresponding to the original data transmission task.

4. The method as described in claim 1, characterized in that, The step of transmitting data based on the task scheduling queue and the target data transmission task includes: Obtain the target data transmission task in the task scheduling queue, as well as the dependencies of the target data transmission task and the predefined source task interface and target task interface; Based on the dependencies of the target data transmission task, generate a task topology object corresponding to the target data transmission task; Based on the task topology structure object and the source data model number corresponding to the target data transmission task, similar tasks are merged to obtain the task topology relationship; or, Based on the task topology structure object and the target end data model number corresponding to the target data transmission task, similar tasks are merged to obtain the task topology relationship; Based on the task topology, a data transmission link is generated between the source task interface and the target task interface; Data transmission is performed based on the data transmission link between the source task interface and the target task interface.

5. The method as described in claim 4, characterized in that, The step of transmitting data based on the data transmission link between the source task interface and the target task interface includes: Start the data transmission process and create the source-end transmission framework client group corresponding to the source-end task interface. The source-end transmission framework client group includes a source-end transmission framework client and a source-end transmission framework server. Source data is obtained through the source-end transmission framework client and the source-end transmission framework server; Create a target-end transmission framework client group corresponding to the target-end task interface. The target-end transmission framework client group includes a target-end transmission framework client and a target-end transmission framework server. The source data is written to the target end corresponding to the target task interface through the target end transmission framework client and the target end transmission framework server.

6. The method as described in claim 5, characterized in that, The step of transmitting data based on the data transmission link between the source task interface and the target task interface further includes: If the source data is streaming data, then the data offset corresponding to the task group ID is obtained based on the pre-acquired task group ID; Based on the data offset, the task data consumption point is determined and the source data is obtained based on the task data consumption point. Data transmission exceptions are captured based on predefined callback functions, and the exception capture results are obtained. Based on the anomaly capture results, flow control is performed on the data transmission process; If the source data is batch data, then a target data verification value is generated based on the source data in the target end; The integrity of the data transmission task is verified based on the target data verification value and the source data verification value corresponding to the pre-acquired source data.

7. A data transmission device, characterized in that, The device includes: A generation module is used to generate an original data transmission task based on pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority, and the dual-end data model includes a source-end data model. A conversion module, used to convert the original data transmission task into a target data transmission task based on the dual-end data model, includes: If the source data type of the source data model is a streaming data source, then the original data transmission task is converted into a streaming data transmission task; If the data granularity of the original data transmission task is a preset granularity specification, then the original data transmission task is converted into a streaming data transmission task; If the source data type is not the streaming data source and the data granularity of the original data transmission task is not the preset granularity specification, then the original data transmission task is converted into a batch data transmission task. The scoring module is used to perform priority scheduling scoring on the target data transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue. An execution module is used to perform data transmission based on the task scheduling queue and the target data transmission task.

8. A data transmission device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data transmission method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data transmission method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data transmission method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Massive multi-source heterogeneous data ETL method and system supporting interface adaptation

    CN108846076A

  • Data warehouse processing method and device for business data flow online analysis

    CN117421376A