Data transmission method and device, equipment, storage medium and product

Abstract

The invention discloses a data transmission method and device, equipment, a storage medium and a product, and relates to the technical field of data transmission, the method comprises the steps that an original data transmission task is generated according to pre-generated data transmission task configuration information, and the data transmission task configuration information comprises a double-end data model and a task priority; converting the original data transmission task into a target data transmission task based on a double-end data model; performing priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue; data transmission is carried out based on the task scheduling queue and the target data transmission task, the data transmission task is converted according to the data model through the scheme, the obtained target data transmission task can be batch data transmission or streaming data transmission, and in addition, all tasks are ensured to be executed in order according to needs through priority ranking. And the efficiency and the universality of the data transmission system are obviously improved.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a data transmission method, apparatus, device, storage medium and product. Background Art

[0002] Data processing is the process of converting raw data into meaningful information. Currently, there are two general approaches for processing data: batch processing, in which multiple data records are collected and stored, and then processed together in one operation; stream processing, in which data sources are continuously monitored and processed in real time as new data events occur.

[0003] The advantage of batch processing is that large amounts of data can be processed at convenient times and can be scheduled to run when the computer or system may be idle (such as overnight) or during off-peak hours; the disadvantage is the time delay between introducing data and obtaining results. Stream processing is well suited for time-critical operations that require real-time response. For example, a system that monitors smoke and heat in a building needs to trigger an alarm and unlock the door so that residents can escape immediately in the event of a fire.

[0004] However, in the existing technology, batch data and streaming data often need to be migrated through two different systems, and the system resources and computing resources between the two systems cannot be shared. In the current big data processing framework, MapReduce is mainly used to process batch data, and the stream processing framework Flink is mainly used to process streaming data. If you want to meet the needs of streaming and batch data transmission between multiple heterogeneous clusters and different data forms at the same time, you need to develop and maintain two data transmission systems at the same time, which is too costly.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a data transmission method, apparatus, device, storage medium and computer program product, which are intended to meet the needs of both streaming data transmission and batch data transmission, and to improve the efficiency and versatility of the data transmission system.

[0007] To achieve the above object, the present application proposes a data transmission method, the method comprising:

[0008] Generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority;

[0009] Based on the dual-end data model, converting the original data transmission task into a target data transmission task;

[0010] Based on the task priority and a preset priority scheduling scoring algorithm, the target transmission task is scored for priority scheduling to obtain a task scheduling queue;

[0011] Data transmission is performed based on the task scheduling queue and the target data transmission task.

[0012] In one embodiment, the dual-end data model includes a source-end data model, and the step of converting the original data transmission task into a target data transmission task based on the dual-end data model includes:

[0013] If the source data type of the source data model is a streaming data source, converting the original data transmission task into a streaming data transmission task;

[0014] If the data granularity of the original data transmission task is a preset granularity specification, converting the original data transmission task into a stream data transmission task;

[0015] If the source data type is not the streaming data source and the data granularity of the original data transmission task is not a preset granularity specification, the original data transmission task is converted into a batch data transmission task.

[0016] In one embodiment, the data transmission task configuration information also includes the task planned execution time, and the step of performing priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue includes:

[0017] Based on the task priority and the task planned execution time corresponding to the target transmission task, respectively calculate the task priority conversion coefficient and the task duration conversion coefficient of the target transmission task;

[0018] Calculate the task scheduling score of the target transmission task based on the task priority conversion coefficient, the task duration conversion coefficient, and the preset task priority scoring coefficient and task duration scoring coefficient;

[0019] According to the task priority of the target transmission task and the task scheduling score, the target transmission task is sent to a target priority queue to obtain a task scheduling queue.

[0020] In one embodiment, the dual-end data model includes a source-end data model and a target-end data model, and the method further includes:

[0021] If the source data model and the target data model have different corresponding data source cluster information, then obtaining the security authentication configuration information of the original data transmission task, wherein the security authentication configuration information includes the cluster key distribution center KDC information;

[0022] According to the security authentication configuration information, obtaining a ticket authorization ticket through the authentication server corresponding to the KDC information;

[0023] According to the ticket authorization ticket, obtain the service ticket through the ticket authorization server corresponding to the KDC information;

[0024] The service process and cluster security authentication corresponding to the original data transmission task are performed according to the service ticket.

[0025] In one embodiment, the step of performing data transmission based on the task scheduling queue and the target data transmission task includes:

[0026] Acquire the target data transmission task in the task scheduling queue, as well as the dependency relationship of the target data transmission task and the predefined source-end task interface and target-end task interface;

[0027] Generate a task topology structure object corresponding to the target data transmission task based on the dependency relationship of the target data transmission task;

[0028] Based on the task topology structure object and the source data model number corresponding to the target data transmission task, similar tasks are merged to obtain a task topology relationship; or,

[0029] Based on the task topology structure object and the target end data model number corresponding to the target data transmission task, similar tasks are merged to obtain a task topology relationship;

[0030] Generating a data transmission link between the source-end task interface and the target-end task interface according to the task topology relationship;

[0031] Data transmission is performed based on a data transmission link between the source-end task interface and the target-end task interface.

[0032] In one embodiment, the step of performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface includes:

[0033] Start the data transmission process and create a source transmission framework client group corresponding to the source task interface, wherein the source transmission framework client group includes a source transmission framework client and a source transmission framework server;

[0034] Acquire source data through the source transmission framework client and the source transmission framework server;

[0035] Creating a target-side transmission framework client group corresponding to the target-side task interface, wherein the target-side transmission framework client group includes a target-side transmission framework client and a target-side transmission framework server;

[0036] The source-end data is written into the target end corresponding to the target-end task interface through the target-end transmission framework client and the target-end transmission framework server.

[0037] In one embodiment, the step of performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface further includes:

[0038] If the source data is streaming data, obtaining the data offset corresponding to the task group ID based on the pre-acquired task group ID;

[0039] According to the data offset, determine the task data consumption point and obtain the source end data based on the task data consumption point;

[0040] Capture data transmission exceptions according to a predefined callback function to obtain exception capture results;

[0041] Performing flow control on the data transmission process based on the abnormal capture result;

[0042] If the source data is batch data, generating a target data check value based on the source data in the target;

[0043] The data transmission task integrity check is performed based on the target-end data check value and the pre-acquired source-end data check value corresponding to the source-end data.

[0044] In addition, to achieve the above purpose, the present application also proposes a data transmission device, the data transmission device comprising:

[0045] A generation module, used to generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority;

[0046] A conversion module, used for converting the original data transmission task into a target data transmission task based on the dual-end data model;

[0047] A scoring module, used to perform priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue;

[0048] An execution module is used to perform data transmission based on the task scheduling queue and the target data transmission task.

[0049] In addition, to achieve the above objectives, the present application also proposes a data transmission device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data transmission method described above.

[0050] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data transmission method described above are implemented.

[0051] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data transmission method described above are implemented.

[0052] One or more technical solutions proposed in the present application generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority; based on the dual-end data model, the original data transmission task is converted into a target data transmission task; based on the task priority and a preset priority scheduling scoring algorithm, the target transmission task is prioritized for scheduling and scoring to obtain a task scheduling queue; data transmission is performed based on the task scheduling queue and the target data transmission task, and the original data transmission task is converted according to the data model through the above solution, and the converted target data transmission task can be batch data transmission or streaming data transmission. In addition, priority sorting is used to ensure that all tasks are executed in an orderly manner as required, thereby significantly improving the efficiency and versatility of the data transmission system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 A flowchart of a data transmission method provided in an embodiment of the present application;

[0056] Figure 2 Schematic diagram of the core functional modules of a data transmission system provided in an embodiment of the present application

[0057] Figure 3 A task conversion logic diagram of a data transmission method provided in an embodiment of the present application

[0058] Figure 4 A flowchart of another data transmission method provided in an embodiment of the present application;

[0059] Figure 5 A flowchart of another data transmission method provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of a data transmission task resource submission timing of a data transmission method provided in an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the module structure of a data transmission device provided in an embodiment of the present application;

[0062] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the data transmission method in the embodiment of the present application.

[0063] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0065] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of the embodiment of the present application is to propose a cross-data center distributed transmission system that integrates data streams and batches, aiming to solve the problem of efficient and high-quality data transmission across data centers and complex networks in the field of big data. First, the system realizes data transmission without the need for mutual trust between the two-end clusters. In the data transmission engine module in the same cluster service, the security authentication of the clusters at both ends of the source data source and the target data source is completed at the same time, and the encryption, decryption, compression, encoding conversion and other processing of the data are realized in the process of transmitting data. In this way, even between Hadoop products provided by different data centers and different commercial companies, there is no need to establish mutual trust, and data transmission between multiple independent and heterogeneous clusters can be realized.

[0067] Secondly, this system simplifies the creation and management of data transmission tasks through visual orchestration and automated scheduling. Data managers only need to select the data models of the source and target ends, and the system can automatically generate streaming or batch migration tasks across centers, different networks, and heterogeneous data sources, eliminating a large amount of parameterized and repetitive configuration work, improving the maintainability of tasks, and making full use of limited bandwidth resources to improve task execution efficiency. The system supports cross-modality data transmission and realizes the transmission capabilities of streaming data and batch data within a system. By implementing custom Source and Sink, the system can realize the data transmission capability between any two data forms, and supports the combined transmission of more than ten data types such as FTP, HDFS, Hive, MySQL, Oracle, Kafka, etc.

[0068] In addition, by verifying the data quality of batch data and marking and checking the locations of streaming data, we ensure that batch and streaming data are completely transmitted from the source data source to the target data source. This solves the problem of incomplete data transmission due to various abnormal situations such as complex network status, network disconnection, and failures during long-distance data transmission between remote data centers. When crossing complex networks, the system improves the quality of data transmission through dynamic flow control, At-least-once semantics for streaming data, and data governance integrity verification, ensuring that streaming data and batch data can be completely and accurately transmitted from the source data center to the target data source.

[0069] In summary, the solution provided by this system effectively solves the needs for efficient and high-quality data transmission across data centers with multiple security authentications, heterogeneous Hadoop clusters, and different storage forms through cross-cluster security authentication, graphical interface orchestration, automated task scheduling, and integrated stream and batch processing.

[0070] Because existing technologies have at least problems including cross-center mutual trust, it is extremely difficult to achieve mutual trust between Hadoop products provided by different data centers and different commercial companies due to differences in underlying technology implementation, or even impossible to achieve mutual trust; due to security management and other factors, ports are closed and networks are isolated between different data centers, making it impossible to implement network policies and establish network communication connections between data centers.

[0071] As well as the reliance on manual configuration, current transmission tools are mainly executed interactively by constructing command lines, requiring data managers to manually configure the data information to be transmitted, including: source data information, target data information and other related parameters to construct execution commands; and only individual tasks can be configured one by one before execution, without unified task scheduling capabilities, tasks cannot be submitted and executed concurrently, and performance is poor. It strongly relies on data managers' knowledge accumulation in the field of big data, and they need to be familiar with the use of different data transmission tools.

[0072] In addition, there is the problem of migration of the same data source form. The current transmission tool only supports the replication of batch data such as HDFS and Hive (actual underlying storage is HDFS) between different clusters, and cannot simultaneously support the data transmission needs between common RDB databases such as Oracle database to HDFS, FTP to HDFS, Kafka streaming data and other diverse data forms. And it can only complete the replication and transmission of data, and does not support additional processing and data conversion operations such as encryption, decryption, compression, encoding conversion, etc. of the migrated data during the data transmission process.

[0073] In addition, multiple solutions need to be developed and maintained. In the current big data processing framework, MapReduce is mainly used to process batch data, and the stream processing framework Flink is mainly used to process streaming data. If the needs of streaming and batch data transmission between multiple heterogeneous clusters and different data forms need to be met at the same time, two sets of data transmission systems need to be developed and maintained at the same time, and the business and maintenance costs are too high.

[0074] Finally, in the prior art, data is often stored in different data centers, which are distributed in different regions. The geographical distance is far, the network transmission distance is long, and the transmission process passes through many network nodes, which greatly increases the probability of failure. The network may have abnormal conditions such as flash disconnection and delay. At the same time, based on the need for secure data transmission, dedicated networks are generally used for data transmission, and their bandwidth resources are often limited.

[0075] Therefore, the present application provides a solution to simultaneously complete the security authentication of the clusters at both ends of the source data source and the target data source in the data transmission engine module of the same cluster service. At the same time, the data processing processes such as encryption and decryption, compression, and encoding conversion are realized during the data transmission process. The invention realizes data transmission between multiple independent and heterogeneous clusters without the need to establish mutual trust between the source and target clusters.

[0076] Through the visual graphical task scheduling interface, data managers only need to select the source data model (the data model is a specific description of the data to be migrated. Taking the HDFS data model as an example, it mainly describes the data file name, file encoding, line separator, file path, column name, compression method, security authentication information of the cluster, etc.), and the target data model. According to the model metadata definition related to the task, streaming or batch migration tasks across centers, different networks, and heterogeneous data sources are automatically generated, eliminating a large amount of parameterized and repetitive configuration work. The data model can be reused in multiple transmission tasks, and there is no need to reconfigure the data model information for each task. Automated scheduling, when docking a single cluster or docking multiple cluster services, divide the tasks according to the cluster, and submit them to different cluster services in batches in the same process. The task status is monitored in real time. When a task fails, a clear error message will be given on the task monitoring page, which greatly improves the maintainability of the task; at the same time, the current dedicated network bandwidth occupancy will be monitored, and through reasonable task scheduling and execution, limited bandwidth resources can be fully utilized to improve task execution efficiency.

[0077] This application realizes the transmission capability of streaming data and batch data in a system. By realizing custom Source and Sink, it realizes the data transmission capability between any two data forms, and supports the combined transmission of more than ten types of data such as mainstream FTP, HDFS, Hive, MySQL, Oracle, Kafka, etc. By verifying the data quality of batch data and marking and checking the position of streaming data, it ensures that the batch and streaming data are completely transmitted from the source data source to the target data source, solving the problem of incomplete data transmission due to various abnormal situations such as complex network status, network disconnection, failure, etc. during long-distance transmission of data between remote data centers.

[0078] In summary, this invention effectively solves the needs for efficient and high-quality data transmission across data centers with multiple security authentications, heterogeneous Hadoop clusters, and different storage forms through cross-cluster security authentication, graphical interface orchestration, automated task generation, automated task scheduling, and stream-batch integration.

[0079] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data transmission system, etc. The following takes the data transmission system as an example to illustrate this embodiment and the following embodiments.

[0080] Based on this, the embodiment of the present application provides a data transmission method, referring to Figure 1 , Figure 1This is a flow chart of the first embodiment of the data transmission method of the present application.

[0081] In this embodiment, the data transmission method includes steps S1000 to S4000:

[0082] Step S1000: Generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority;

[0083] It should be noted that in the embodiments of the present application, in order to solve the problem of efficient and high-quality data transmission across data centers and across complex network environments, especially the difficulty in establishing mutual trust between different data centers, the need for manual configuration, the support for only limited data source relocation, the need to maintain multiple sets of solutions, and the easy loss of data when transmitting across complex networks, a cross-data center distributed transmission system solution that integrates data streams and batches is proposed. This solution achieves seamless transmission of streaming data and batch data in the same system through the configuration, conversion, scheduling, and execution of data transmission tasks.

[0084] You can refer to Figure 2 , Figure 2 A schematic diagram of the core functional modules of the data transmission system provided in the embodiment of the present application. The present invention provides a cross-data center distributed transmission system that integrates data streams and batches, including a data transmission task management module, a data transmission task scheduling engine module, and a data transmission task execution engine module. The system establishes a data model for the transmission task, performs configuration management, dynamically judges the model parameters, and automatically converts the transmission task into streaming and batch transmission according to intelligent control rules. At the same time, the configuration parameters include task priorities, and a priority-based task sorting method is provided. The overall solution provides a streaming and batch integrated transmission method based on task priority scheduling, which can solve the streaming and batch integrated transmission problem intelligently, efficiently, and at low cost, and can meet customers' integrated streaming and batch data needs while considering task priorities in a system.

[0085] First, in an embodiment of the present application, an original data transmission task is generated according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority, wherein the dual-end data model refers to the data model of the source end and the target end, and the source end data model and the target end data model respectively describe the detailed information of the source and destination of the data, such as the format, type, storage location, etc. of the data. Task priority refers to the urgency of task execution, which can be set according to business needs to ensure that critical tasks can be executed first. The purpose of generating the original data transmission task is to provide a basic execution framework for the subsequent data transmission process, ensuring that data can be transmitted from the source end to the target end according to the predetermined configuration.

[0086] For example, in one possible implementation, Figure 2 The 101 data transmission task management module shown mainly includes data model management and data transmission task configuration management capabilities. Through this module, data management personnel only need to select the data model of the source data center, the data model of the target data center and configure the task priority to quickly create a data transmission task.

[0087] The data model in the data transmission task is a standardized, formatted, and unified definition of batch data and streaming data, which mainly includes: data granularity information (minutes, hours, days, months, years), data column names, data column field types, whether the data column is encrypted and its encryption algorithm; physical storage information of the data, such as the file server address, file name, file storage path, server address, and authentication information of batch data such as HDFS; Topic name, partition, and other information of Kafka streaming data. The data model management capability provides a wizard-style definition of data models through a visual interface, and also supports docking with existing data sources, including physical storage modules in various forms such as HDFS, Hive, and Kafka, and reversely collects and maps data definitions to the data model.

[0088] Through the data transmission task configuration interface, users can configure whether to perform field encryption and decryption on the source data, code set (GBK, UTF-8) conversion, delimiter conversion, task schedule execution time, maximum number of task failure retries (system maximum 3 times), task priority (including low, medium, and high), etc.

[0089] After the task is confirmed and saved, the system will save the task information to the database in the data transmission task management module. During the configuration process of the entire data transmission task, data management personnel do not need to pay attention to the specific execution details of the data transmission task, and no other manual configuration is required to complete the arrangement and definition of the data transmission task.

[0090] In a feasible implementation, the main configuration information of the task can be stored in the database of the data transmission task management module in JSON (JavaScript Object Notation) format. JSON format is a lightweight data exchange format that is easy for people to read and write. It is mainly implemented through the "name: value" data structure, which can define multiple groups of names and their values. Multiple groups are separated by commas. The parameters filled in by the data manager through the visual interface are converted into the following JSON format, such as the following example parameters:

[0091] {

[0092] "reader.name":"hdfsreader", / / Source task type

[0093] "writer.name":"hdfswriter", / / Target-side task type

[0094] "fieldnames":"a|string,b|int", / / field name and type

[0095] "encrypt.type":"SM4", / / encryption algorithm

[0096] "data.interval":"hour", / / data cycle

[0097] "read.filepath":" / tmp / ${PROVINCE}", / / file path

[0098] "read.filename":"*.dat", / / file name

[0099] "kafka.read.topic":"testA3", / / kafak topic

[0100] "indexArray":"1,3", / / decryption field subscript

[0101] "source.separator":"|", / / Source data separator

[0102] "target.separator":"#", / / target data separator

[0103] "charset":"UTF-8", / / character encoding

[0104] "dms.dsid":"defaultCluster", / / Cluster information

[0105] "priority":"Medium", / / Task priority

[0106] "task.retry.times":3, / / Number of retries after task failure

[0107] "task.wait.time":2024040112300, / / Task scheduled execution time

[0108] "kdc.filepath":" / tmp / conf / krb5.conf", / / krb5 configuration path, used to connect to the kerberos server

[0109] "keytab.filepath":" / tmp / conf / ldapuser.keytab", / / keytab file for kdc authentication

[0110] }

[0111] Step S2000: based on the dual-end data model, converting the original data transmission task into a target data transmission task;

[0112] It should be noted that in the embodiment of the present application, based on the dual-end data model, the original data transmission task is converted into a target data transmission task. The purpose of this step is to intelligently convert the task into a suitable transmission form, i.e., a stream data transmission task or a batch data transmission task, according to different data source types and data granularity. For example, if the source data model indicates a streaming data source, or the data granularity meets the preset stream data granularity specification, then the original data transmission task will be converted into a stream data transmission task. Such conversion is to adapt to the transmission requirements of different data characteristics, optimize resource usage, and improve transmission efficiency.

[0113] Step S3000: performing priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue;

[0114] It should be noted that in the embodiment of the present application, after the original data transmission task is converted into a stream data transmission task or a batch data transmission task, the target transmission task can be prioritized and scored based on the task priority and the preset priority scheduling scoring algorithm to obtain a task scheduling queue. In this step, the system will calculate the task priority conversion coefficient and the task duration conversion coefficient respectively according to the task priority and the task planned execution time corresponding to the target transmission task, and then, in combination with the preset task priority scoring coefficient and the task duration scoring coefficient, calculate the task scheduling score of the target transmission task. According to the task scheduling score, the system sends the target transmission task to the corresponding target priority queue to form a task scheduling queue. The system performs this operation in order to reasonably arrange the order of task execution to ensure that high-priority tasks can be executed in time. At the same time, the planned execution time of the task is also taken into account to optimize the overall scheduling efficiency. In a specific implementation method, the system will automatically calculate the scheduling score of each task according to the urgency of the task and the scheduled execution time, and sort the tasks according to the score to ensure that critical tasks are processed first.

[0115] Step S4000: performing data transmission based on the task scheduling queue and the target data transmission task.

[0116] It should be noted that, in the embodiment of the present application, data transmission can be performed based on the task scheduling queue and the target data transmission task. By obtaining the target data transmission task and its dependency in the task scheduling queue, as well as the predefined source task interface Source and target task interface Sink, the system can generate the task topology structure object corresponding to the target data transmission task based on this information, and merge similar tasks according to the task topology structure object and the data model number to form a task topology relationship. Then, a data transmission link between the source task interface and the target task interface is generated according to the task topology relationship, and data transmission is performed based on these links. The purpose of this step is to ensure that data can be smoothly transmitted from the source task interface to the target task interface in a predetermined order and manner, and complete the cross-data center transmission of data. For example, in a specific implementation, the system will automatically match the corresponding source and target interfaces according to the information in the task scheduling queue, and establish a data transmission link, and then start to perform data transmission operations to ensure that data is accurately transmitted from one data center to another.

[0117] In a feasible implementation, the dual-end data model includes a source-end data model, and step S2000 may include steps S2100 to S2300:

[0118] Step S2100: if the source data type of the source data model is a streaming data source, converting the original data transmission task into a streaming data transmission task;

[0119] Step S2200: if the data granularity of the original data transmission task is a preset granularity specification, converting the original data transmission task into a stream data transmission task;

[0120] Step S2300: If the source data type is not the streaming data source and the data granularity of the original data transmission task is not a preset granularity specification, converting the original data transmission task into a batch data transmission task.

[0121] It should be noted that in the embodiment of the present application, the two-end data model includes a source-end data model and a target-end data model. The two models respectively define the characteristics of the starting point and end point of the transmitted data, including attributes such as data type, storage location, and format. In the process of converting the original data transmission task into the target data transmission task, the system will intelligently judge and determine the transmission form of the task based on the type and data granularity of the source-end data model, that is, a streaming data transmission task or a batch data transmission task. Streaming data sources refer to those data sources that continuously generate data, such as Kafka, real-time monitoring systems, etc., while batch data transmission tasks are suitable for non-real-time, periodic data transmission requirements. The data granularity involves the frequency of data updates, such as data updated by minutes, hours, days, etc. Such conversion is to adapt to the transmission requirements of different data characteristics, optimize resource utilization, and improve transmission efficiency.

[0122] In addition, it should be noted that the task scheduled execution time refers to the specific time point when the task is scheduled to start execution. In the task priority scheduling scoring algorithm, the system will combine the task priority and the planned execution time to calculate the task priority conversion coefficient and the task duration conversion coefficient, and then derive the task scheduling score. This score will determine the order of tasks in the queue to ensure that key tasks can be executed in a timely manner, while also taking into account the planned execution time of the task to optimize the overall scheduling efficiency.

[0123] In a feasible implementation, the system can first generate an original data transmission task based on the data transmission task configuration information, and then analyze the source data model. If the source data type is a streaming data source, or the data granularity meets the preset streaming data granularity specification, the system will convert the original data transmission task into a streaming data transmission task. Otherwise, if the source data type is not a streaming data source and the data granularity does not meet the preset granularity specification, the task is converted into a batch data transmission task. Next, the system will calculate the task scheduling score based on the task priority and planned execution time, and send the task to the corresponding priority queue based on the score to form a task scheduling queue. In the data transmission stage, the system will perform data transmission according to the task scheduling queue and the target data transmission task, including obtaining task dependencies, generating task topology structure objects, merging similar tasks, generating data transmission links, and finally performing data transmission based on these links.

[0124] For example, refer to Figure 3 , Figure 3A task conversion logic diagram is provided for an embodiment of the present application. In another feasible implementation, based on the source data information and the target data information in the original data transmission task information saved in the above steps, a Flink data transmission task is automatically generated. By analyzing the source and target data model information of the task and whether the task involves data encryption and decryption, encoding conversion and other processing operations, an intelligent decision is made to generate the optimal Flink task, which includes two task forms, namely, Flink batch data processing tasks and Flink stream data processing tasks, such as Figure 3 In the embodiment shown, if the data needs to be stored in a system other than HDFS (i.e., the write.filepath field is configured to not start with hdfs), such as a database or a local file system (FTP), the task is converted into a Flink batch task. If it is a streaming data source, such as Kafka and other middleware (i.e., the reader.name configuration parameter is kafkareader), the task is converted into a Flink stream task. If the data model is a finer data granularity such as hours or minutes, the task is converted into a Flink stream task (i.e., data.interval is hour or minute). If the data transmission task requires encryption, decryption and other conversion operations on the specified data field (i.e., there is an inconsistency between source.separator and target.separator or there is an encrypt.type configuration), the task is converted into a Flink batch task. For other data transmission tasks that do not meet the above specific conditions, the tasks are converted into Flink batch tasks.

[0125] In a feasible implementation manner, the data transmission task configuration information further includes the task planned execution time, and step S3000 may include steps S3100 to S3300:

[0126] Step S3100: Based on the task priority and task planned execution time corresponding to the target transmission task, respectively calculating the task priority conversion coefficient and task duration conversion coefficient of the target transmission task;

[0127] Step S3200: Calculating the task scheduling score of the target transmission task based on the task priority conversion coefficient, the task duration conversion coefficient, and the preset task priority scoring coefficient and task duration scoring coefficient;

[0128] Step S3300: According to the task priority of the target transmission task and the task scheduling score, the target transmission task is sent to a target priority queue to obtain a task scheduling queue.

[0129] It should be noted that in the embodiment of the present application, the task planned execution time refers to the specific time point when the task is scheduled to start execution. In the task scheduling scoring algorithm, the system will comprehensively consider the priority and planned execution time of the task, and calculate the task priority conversion coefficient and the task duration conversion coefficient respectively. The task priority conversion coefficient reflects the degree of influence of the task priority on the scheduling score, while the task duration conversion coefficient reflects the influence of the urgency of the task planned execution time on the score. Through these two coefficients, the system can more accurately evaluate the execution priority of each task, ensuring that critical and urgent tasks can obtain higher scheduling priority.

[0130] In addition, it should be noted that the system will calculate the task scheduling score for each target transmission task based on the preset task priority scoring coefficient and task duration scoring coefficient, combined with the above two conversion coefficients. This score not only takes into account the priority and planned execution time of the task, but also takes into account the overall scheduling strategy and resource allocation of the system, so as to ensure the fairness and efficiency of task scheduling. According to the task scheduling score, the system will send the task to the corresponding priority queue to form a task scheduling queue. This queue will guide subsequent data transmission operations to ensure that the tasks can be executed in the predetermined order and plan.

[0131] In a feasible implementation, the system first receives data transmission task configuration information, including a dual-end data model and task priority. The system intelligently converts the original data transmission task into a stream data transmission task or a batch data transmission task according to the type and data granularity of the source data model. Subsequently, the system calculates the task priority conversion coefficient and the task duration conversion coefficient according to the priority and planned execution time of the task, and combines the preset scoring coefficient to obtain the scheduling score of each task. Based on this score, the system assigns the task to different priority queues to form a task scheduling queue. In the data transmission phase, the system performs data transmission operations according to the task scheduling queue and the target data transmission task. This includes obtaining task dependencies, generating task topology structure objects, merging similar tasks, generating data transmission links, and performing data transmission based on these links. During the entire process, the system also handles cross-data center security authentication issues to ensure the security of data transmission. Through these steps, the system can achieve seamless transmission of streaming data and batch data in the same system, meeting the high-quality and efficient data transmission requirements between different data centers.

[0132] Specifically, in another feasible implementation manner, referring to Figure 3As shown, the data transmission task scheduling engine module 102 obtains the tasks to be submitted for execution in batches from the task configuration database, and comprehensively scores the tasks to be executed through the task priority scoring algorithm to ensure that the tasks are executed in an orderly manner as needed. The system presets two task queues, namely: a high priority queue for key tasks and a general task queue. The task priority can be configured on the visual interactive page provided by the system. The data management personnel can manually set the priority to high, general, and low according to the business importance of the data to be relocated. Tasks set to high priority will be placed in the high priority queue and submitted and executed before other tasks. Tasks with low priority will be placed in the low priority queue. The system will calculate the priority score of the corresponding data transmission task according to the task priority scoring calculation formula, and put the data transmission task into the corresponding task queue according to the scoring results from high to low. In the embodiment of the present application, a feasible scoring formula is as follows:

[0133] S=A*0.6+B*0.3

[0134] Among them, S is the scheduling score of the task, A is the priority of the task configuration, for example, it can be converted to a value of 3 according to [High], a value of 2 according to [Medium], and a value of 1 according to [Low], and then included in the scoring formula, B is the absolute waiting time of the task, and the calculation formula is B = |System.T-CronT| / 60, which is obtained by subtracting the task scheduled execution time from the current operating system time and dividing it by 60 to convert it into hours. System.T is the current operating system time. Taking Java language as an example, the current operating system time is obtained through the LocalDateTime.now() function, and the task scheduled execution time CronT is obtained from the task.wait.time field of the task configuration information.

[0135] The real-time priority score of each data transmission task to be submitted and started is calculated through the task priority scoring formula. For example, the data management personnel create two data transmission tasks through the data task management module in step 1. Their scheduled execution time is 0:0:00 on January 1, 2024. The task priority configured for data transmission task 1 is high, while the task priority configured for task transmission task 2 is low. If the system time is 23:0:00 on December 31, 2023, the scheduling score calculation process of task 1 is S=3*0.6+|20231231230000-20240101000000| / 60*0.3, and the result is 2.1. The score result of task 2 is 0.9 obtained by the same calculation formula. According to the task scoring result, task 1 with a higher score will be placed in the task queue before task 2. This method can ensure that key tasks are executed first and key data are transmitted first, and ensure that all output transmission tasks are executed on demand and in an orderly manner, effectively reducing the average waiting time of tasks.

[0136] The present embodiment provides a data transmission method, which generates an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority; based on the dual-end data model, the original data transmission task is converted into a target data transmission task; based on the task priority and a preset priority scheduling scoring algorithm, the target transmission task is prioritized for scheduling and scoring to obtain a task scheduling queue; data transmission is performed based on the task scheduling queue and the target data transmission task, and the original data transmission task is converted according to the data model through the above scheme, and the converted target data transmission task can be batch data transmission or streaming data transmission. In addition, priority sorting is used to ensure that all tasks are executed in an orderly manner as needed, thereby significantly improving the efficiency and versatility of the data transmission system.

[0137] Based on the first embodiment provided by the present application, in another embodiment provided by the present application, the same or similar contents as those in the above embodiment can be referred to the above introduction, and will not be described in detail later. Figure 4 , the dual-end data model includes a source-end data model and a target-end data model, and the data transmission method further includes steps A1000 to A4000:

[0138] Step A1000: If the source data model and the target data model have different corresponding data source cluster information, then obtain security authentication configuration information of the original data transmission task, wherein the security authentication configuration information includes cluster key distribution center KDC information;

[0139] Step A2000: According to the security authentication configuration information, obtain a ticket authorization ticket through the authentication server corresponding to the KDC information;

[0140] Step A3000: According to the ticket authorization ticket, obtain the service ticket through the ticket authorization server corresponding to the KDC information;

[0141] Step A4000: Perform security authentication of the service process and cluster corresponding to the original data transmission task according to the service ticket.

[0142] It should be noted that in the embodiment of the present application, during the data transmission process, if the data source cluster information of the source and target ends is different, that is, they belong to different data centers or data is provided by different data sources, then the system needs to obtain the security authentication configuration information of the original data transmission task to ensure the security of data when it is transmitted across data centers. The security authentication configuration information mainly includes the cluster key distribution center (KDC Key Distribution Center) information, which is a key component for implementing the Kerberos authentication protocol and is responsible for distributing ticket authorization tickets and server tickets, thereby ensuring identity authentication and authorization during the data transmission process.

[0143] In addition, it should be noted that the system obtains the ticket granting ticket (TGT) through the authentication server corresponding to the KDC information according to the security authentication configuration information. This is the first step for users or service processes to access Kerberos authentication system resources. After obtaining the TGT, the system will use the ticket to request a service ticket (ST) from the KDC's ticket granting server (TGS). This step is a necessary verification process before users or service processes access specific services. Once the service ticket is obtained, the system will perform security authentication of the service process and cluster corresponding to the original data transmission task based on the ticket to ensure the security and reliability of data transmission.

[0144] For example, in a feasible implementation, the system first generates an original data transmission task based on the dual-end data model and converts it into a target data transmission task. Subsequently, the system evaluates the priority and planned execution time of the task, calculates the priority conversion coefficient and task duration conversion coefficient of the task, and accordingly obtains the task scheduling score and places the task in the corresponding priority queue. In the data transmission preparation stage, the system obtains security authentication configuration information according to the differences between the source and target data models, and uses KDC information for identity authentication and authorization to ensure the security of data transmission. In the data transmission stage, the system will connect to different data source types through customized Source and Sink, and perform data transmission according to the task topology structure object and task scheduling queue. In this process, the system will also handle cross-data center security authentication issues to ensure the security of data transmission. Through these steps, the system can achieve seamless transmission of streaming data and batch data in the same system, meeting the high-quality and efficient data transmission requirements between different data centers.

[0145] More specifically, when the cluster where the source data source is located and the cluster where the target data source is located are in different KDC centers, it is necessary to implement security authentication for the source cluster and the target cluster separately in the same task through multiple processes in the data transmission cluster service. First, the relevant configuration information required for Kerberos security authentication of the docking cluster can be obtained from the task configuration information Json, which mainly includes two files: krb5.conf and Keytab file. The krb5.conf configuration file mainly includes KDC information in the cluster domain, realm domain information, and lib configuration information. The data transmission engine service process performs security authentication with the cluster where the source data is located through the security authentication file. The data transmission engine service process acts as a client and carries authentication information to request a ticket authorization ticket TGT from the authentication server in the KDC. After the KDC confirms that the client identity has the authority to access the KDC, it returns the TGT to the client.

[0146] The client carries the TGT to access the TGS and requests the service ticket Server Ticket. The TGS in the KDC checks whether the user identity in the TGT is the same as the requesting user to avoid impersonation. After confirmation, the ticket authorization server TGS returns the Server Ticket for the user to access the service. The client accesses the service to be accessed based on the Server Ticket, such as HDFS. The Kafka service checks whether there is the ServerIP requested by the user, and checks the Sever Ticket content and the client identity. After the above steps are completed, the service process completes the security authentication with the cluster and can access the data of the HDFS, Hive, Kafka and other big data components. Repeat the above steps and complete the security authentication with the target data cluster in another process. Through the above steps, the data transmission engine completes the cross-KDC security authentication of the dual-end heterogeneous Hadoop cluster and can continue with the subsequent task submission and data operation steps.

[0147] The present embodiment provides a data transmission method, by obtaining security authentication configuration information of the original data transmission task if the data source cluster information corresponding to the source-end data model and the target-end data model is different, the security authentication configuration information including cluster key distribution center KDC information; according to the security authentication configuration information, obtaining a ticket authorization ticket through the authentication server corresponding to the KDC information; according to the ticket authorization ticket, obtaining a service ticket through the ticket authorization server corresponding to the KDC information; performing security authentication of the service process and cluster corresponding to the original data transmission task according to the service ticket, to ensure the security of data transmission.

[0148] Based on the first embodiment provided by the present application, in another embodiment provided by the present application, the same or similar contents as those in the above embodiment can be referred to the above introduction, and will not be described in detail later. Figure 5 , the step S4000 may include steps S4100 to S4600:

[0149] Step S4100: acquiring the target data transmission task in the task scheduling queue, as well as the dependency relationship of the target data transmission task and the predefined source-end task interface and target-end task interface;

[0150] Step S4200: generating a task topology structure object corresponding to the target data transmission task based on the dependency relationship of the target data transmission task;

[0151] Step S4300: merging tasks of the same type based on the task topology structure object and the source data model number corresponding to the target data transmission task to obtain a task topology relationship; or,

[0152] Step S4400: merging tasks of the same type based on the task topology structure object and the target end data model number corresponding to the target data transmission task to obtain a task topology relationship;

[0153] Step S4500: generating a data transmission link between the source-end task interface and the target-end task interface according to the task topology relationship;

[0154] Step S4600: performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface.

[0155] It should be noted that in the embodiment of the present application, the target data transmission task in the task scheduling queue is matched with the task interface of the source end and the target end, and a data transmission link is constructed based on this information, the purpose is to ensure that the data can be smoothly and efficiently transmitted from the source end to the target end. The task scheduling queue contains the data transmission tasks to be executed, namely the target data transmission tasks. These tasks are sorted according to the priority and other parameters and put into the corresponding high priority task queue or low priority task queue. The system analyzes the dependency relationship and data model number of the tasks, merges similar tasks, optimizes resource allocation, reduces redundant operations, and improves transmission efficiency.

[0156] In addition, it should be noted that the task topology object is a mapping of the dependencies between tasks. It helps the system understand the hierarchical structure and data flow between tasks. Through this structure, the system can generate accurate data transmission links to ensure that data is transmitted along the predetermined path. In this process, the system will also consider the integrity and consistency of the data, especially when transmitting across data centers, and ensure the reliability of the data through network exception handling mechanisms and data verification technology.

[0157] In a feasible implementation, the system first extracts the target data transmission task from the task scheduling queue, and identifies its dependencies and the corresponding source and target task interfaces. Then, the system generates a task topology structure object and merges tasks with the same source or target data model number to optimize the transmission path. The system creates a data transmission link between the source and target task interfaces based on the task topology relationship. On this link, the system starts the data transmission process, obtains data through the source transmission framework client and server, and writes data to the target using the target transmission framework client and server. During the entire transmission process, the system also monitors the network status and data transmission health, dynamically controls the traffic, and performs data integrity verification after the transmission is completed to ensure that the data is accurately transmitted from the source to the target. In this way, the system achieves efficient and high-quality data transmission across data centers, meets the transmission requirements between different data source forms, and reduces business and maintenance costs.

[0158] More specifically, in another possible implementation, referring to Figure 6 , Figure 6 A schematic diagram of a data transmission task resource submission timing sequence of a data transmission method provided in an embodiment of the present application, after determining the priority scores of several target data transmission tasks, Figure 2 The data transmission task scheduling engine 102 shown in the figure takes the stream or batch Flink tasks generated in the above steps from the high priority queue and the low priority queue and submits them to the task job resource cluster. The main submission startup steps are as follows: Figure 6 As shown, 102 data scheduling engine submits data transmission task to Flink cluster, Flink cluster sends task execution resource request to YARN (resource manager in Hadoop ecosystem), and then YARN returns available execution resources, Flink starts Application Master (Application Master is the manager of each application, responsible for application life cycle management, including applying for resources from ResourceManager, coordinating container start / stop with NodeManager, etc. Application Master is also responsible for monitoring application status to ensure that tasks can run normally), which is used to apply for resources and monitor the running status of applications, YARN returns Application Master address, which is used to access the task to view detailed information, then Flink submits task to Application Master, Application Master applies for container resources from YARN, and YARN returns container resources, Application Master starts TaskManager (task manager) in the returned container resources, which is used to manage the startup and operation of tasks, finally Flink sends task instructions to TaskManager for data processing, after the task is successfully started, TaskManager returns the execution result.

[0159] After submitting the target data transmission task to Yarn, Figure 3 The data transmission task execution engine module 103 shown can realize docking with different data source types through the custom source task interface Source and the target task interface Sink implemented by the program. When the task is started, it automatically creates the Source implementation of docking with the source data source and the Sink implementation of docking with the target data source according to the source and target data types. The main steps are as follows:

[0160] Source is responsible for reading data from the source data source into the data transmission engine. Different Handler implementations are defined for different data sources. The source data source is connected to read data through JDBC, FileSystem-Read, Kafka Consumer, etc.

[0161] Sink is responsible for delivering data from the data transmission engine to the target data source. For different data sources, different Handler implementations are defined to extract data from the named pipe and write it to the target data source through JDBC, FileSystem-Write, Kafka Consumer, etc.

[0162] Through different combinations of Source and Sink, the one-to-one data transmission needs of any two data sources can be realized. In the actual business production process, there will be multi-source reading and multi-source writing needs, and the data will be distributed in different data sources. For example, daily table data is stored in Hive and monthly table data is stored in HDFS. At this time, it is necessary to migrate data from two different data sources to the same or multiple target data sources at the same time. The specific implementation steps are as follows:

[0163] The page layout task is converted into Node and Edge object Content that can be described and identified by programming language according to the dependency relationship. It is a Json body, in which Node is used to store the type and parameter information of each node, and Edge is used to store the connection status between nodes.

[0164] Traverse each Node node and generate the ternary topology structure object TaskTopology of the Node. Each object represents the Node's predecessor node list, itself, and successor node list.

[0165] Process the Node nodes in a loop. When the node type is identified as a merged UNION node, obtain its predecessor node list, perform UNION processing on the data extracted by the Source Handler of the predecessor node through FlinkSQL, and then temporarily retain it in Flink's DataStream for subsequent processing.

[0166] After generating the task topology, perform a mergeability analysis on the tasks: by comparing the source data source data model definition of the data transmission task with the source data model definition of the existing task, if the unique ID of the data model definition is the same, multiple data transmission tasks will be merged and the current task will be added to the topological relationship of the existing task, forming a data transmission link of 1 Source to N Sinks. This method greatly improves the data transmission efficiency and effectively reduces the number of RPC requests of the source data cluster. In the optimal state, the original source data that needs to be read and transmitted N times is reduced to only once, which greatly reduces the network bandwidth usage. More data transmission tasks can be completed in the same time, greatly improving the data transmission efficiency.

[0167] In a feasible implementation, step S4600 may include steps S4610 to S4640:

[0168] Step S4610: starting the data transmission process and creating a source transmission framework client group corresponding to the source task interface, wherein the source transmission framework client group includes a source transmission framework client and a source transmission framework server;

[0169] Step S4620: Acquire source data through the source transmission framework client and the source transmission framework server;

[0170] Step S4630: creating a target-side transmission framework client group corresponding to the target-side task interface, wherein the target-side transmission framework client group includes a target-side transmission framework client and a target-side transmission framework server;

[0171] Step S4640: writing the source-end data into the target end corresponding to the target-end task interface through the target-end transmission framework client and the target-end transmission framework server.

[0172] It should be noted that in the embodiment of the present application, the execution of the data transmission task first requires starting the data transmission process and creating a transmission framework client service group corresponding to the source task interface, including a source transmission framework client and a server. The source transmission framework client is responsible for interacting with the data source, while the server processes client requests and prepares data to be sent. Through this mechanism, the system can efficiently obtain data from the source and lay the foundation for subsequent transmission.

[0173] In addition, it should be noted that after obtaining the source data, the system will create a transmission framework client group corresponding to the target task interface, including the target transmission framework client and server. The target transmission framework client is responsible for writing the received data to the target data source, while the server ensures that the data is correctly stored and updated. The purpose of this step is to ensure that the data can not only be successfully read from the source, but also be accurately written to the target, completing the entire process of data transmission.

[0174] For example, in a feasible implementation, the system transmits the data read by the Source to the Sink through Netty (asynchronous event-driven network application framework) across the authentication process. Since the Source and the Sink are respectively connected to the KDC on both ends for security authentication and are in different processes, the data needs to be sent from the Source process to the Sink process. First, a Source-side NettyClient-Server group, i.e., a source-side transmission framework customer service group, is created in advance, which includes two parts: the NettyClient source-side transmission framework client, which implements ReadProcessInstanceNetty, a child process run by Netty, which starts a source-side transmission framework client and carries the Handler implementation of the source-side task interface Source, and sends the data read by the Handler to the source-side transmission framework server, and then starts a source-side transmission framework server NettyServer through ChildProcessNettySource, and passes the data written by the source-side transmission framework server to the Flink main process.

[0175] In an embodiment of the present application, the Flink main task process receives data transmitted from the source transmission framework server, processes the intermediate operator, and then outputs the processed data through the Sink target transmission framework client for use by the Sink. First, a Sink NettyClient-Server group, i.e., a target transmission framework client group, is created, which also includes two parts, the target transmission framework client NettyClient: a ChildProcessNettySink instance is created in the Flink main process, the target transmission framework client of the Sink is started, the upstream message is encapsulated and written to the target transmission framework server, the target transmission framework server starts the NettyServer child process and carries the corresponding target processor SinkHandler plug-in, which is used to receive the data written by the target transmission framework client. At the same time, the plug-in writes the data to the target end through JDBC, FileSyStem-Write, KafkaProducer, etc. to complete the data transmission.

[0176] In a feasible implementation manner, step S4600 may further include steps S4611 to S4616:

[0177] Step S4611: if the source data is streaming data, obtaining a data offset corresponding to the task group ID based on the pre-acquired task group ID;

[0178] Step S4612: determining the task data consumption point according to the data offset and acquiring the source end data based on the task data consumption point;

[0179] Step S4613: Capture data transmission exceptions according to a predefined callback function to obtain exception capture results;

[0180] Step S4614: performing flow control on the data transmission process based on the abnormal capture result;

[0181] Step S4615: if the source data is batch data, generating a target data check value based on the source data in the target;

[0182] Step S4616: Performing a data transmission task integrity check based on the target-end data check value and a pre-acquired source-end data check value corresponding to the source-end data.

[0183] It should be noted that, in the embodiment of the present application, when facing cross-data center, cross-complex, unstable network, data transmission needs to solve the data loss problem caused by network abnormalities such as packet loss and flash disconnection during data transmission, and ensure that data can be correctly transmitted from the source data source to the target data source. The present invention improves the quality of data transmission through dynamic flow control, streaming data At-least-once semantics, and data governance integrity verification, ensuring that streaming data and batch data can be completely and accurately transmitted from the source data center to the target data source. For streaming data, the system will obtain the corresponding data offset based on the task group ID, which is an indicator used to record consumption progress in streaming data sources such as Kafka. The data offset enables the system to read data from the correct position to ensure the continuity and integrity of the data. During the reading process, the system will define a callback function to capture anomalies in data transmission, which is crucial for dynamically adjusting transmission strategies and handling network fluctuations. Based on the abnormal capture results, the system will control the flow of the data transmission process to adapt to network conditions and the processing capabilities of the receiving end. In addition, it should be noted that for batch data, the system will perform data consistency and integrity checks after the data transmission task is executed. By comparing the data checksum source generated by arithmetic operations on the source data with the data checksum sink recalculated by the target end, the system can verify whether the data has changed or lost during the transmission process. This process is critical to ensure the accuracy and reliability of batch data.

[0184] In a feasible implementation, the system first generates a data transmission link based on the target data transmission task and task dependencies in the task scheduling queue, as well as the predefined source and target task interfaces. For streaming data, the system obtains the data offset corresponding to the task group ID, and determines the consumption location based on the offset, so as to accurately obtain data from the source data source. During the data transmission process, the system monitors and captures any exceptions, such as sending timeouts or data inconsistencies, and dynamically adjusts the data flow based on these abnormal results to ensure the stability and efficiency of data transmission. For batch data, the system compares the data check values ​​of the source and target ends after the data transmission is completed to verify the integrity of the data. If the check values ​​are inconsistent, the system will start the data supplement task and retransmit the missing data to ensure the accuracy of data transmission. In this way, the system can achieve efficient and high-quality transmission of streaming data and batch data in a complex network environment across data centers.

[0185] Specifically, in another feasible implementation, by marking and checking the streaming data, it is ensured that the streaming data transmission is not lost; at the same time, by limiting the data flow of the source end, it is avoided that the data source of the Sink end is over-written, resulting in the data not being written correctly. The specific implementation logic is as follows:

[0186] After establishing a Kafka connection to read the source data source, the group.id information configured by the task is used to obtain the offset corresponding to the group.id by calling the listConsumerGroupOffsets method in the AdminClient class provided by Kafka. If the task consumes the Kafka data for the first time, the offset obtained is empty, and consumption starts from the earliest Kakfa data. If the offset data is obtained, the KafkaConsumer.seek method is used to specify the task consumption position when consuming the Kakfa message to ensure that the data continues to be consumed from the previously saved position. In the process of writing Kafka data, the send method in KafkaProducer is used to write the data passed in from the upstream, and the custom callback function KafkaSendCallback is passed in. If an exception occurs during the process of sending Kafka data, the exception is captured in the callback function.

[0187] By analyzing the exceptions, if the data sending timeout continues, it is speculated that the downstream Kafka's receiving capacity is insufficient. At this time, the source end reading will be flow controlled. If it is determined that the source end needs to be flow controlled, the Sink end process will send a flow control signal to the Source end process through Netty communication. After the Source end process receives the flow control signal, it uses the RateLimiter capability rate statistics provided by guava. If the rate exceeds the configured rate, it will wait when transmitting data until the rate is lower than the flow control configuration.

[0188] For batch data, after the data transmission task is completed, the system will check the consistency and integrity of the data, and generate a data check feature value CheckSumSource by performing arithmetic operations on the source data. After the data is moved to the target data source, a new CheckSumSink is recalculated. Taking files as an example, the calculation logic of the CheckSum check value is to first divide the file into blocks of a fixed size (each block is 64MB), perform hash calculations on each block, generate a check value of a fixed length, and then concatenate the check values ​​of all blocks to form a long check value string. This long check value string is hashed again to generate the final checksum value CheckSum. Finally, the check values ​​of the source and target data are compared. When the check values ​​are inconsistent, it proves that data loss has occurred during the data transmission process. At this time, the data quality verification task will be started. This task will compare and verify the specific content of the data at both ends, list the files and line numbers where the missing data is located, and automatically start the data supplement task as input. This task will retransmit the missing data from the source data source to the target data source.

[0189] Based on the above steps, a high-quality and efficient data transmission capability is built for streaming data and batch data across data centers, multi-end KDC authentication, cross-data source types, and data conversion and processing in the same system.

[0190] In another embodiment provided in the present application, by obtaining the target data transmission task in the task scheduling queue, as well as the dependency relationship of the target data transmission task and the predefined source-end task interface and target-end task interface; generating a task topology structure object corresponding to the target data transmission task based on the dependency relationship of the target data transmission task; merging similar tasks based on the task topology structure object and the source-end data model number corresponding to the target data transmission task to obtain a task topology relationship; or, merging similar tasks based on the task topology structure object and the target-end data model number corresponding to the target data transmission task to obtain a task topology relationship; generating a data transmission link between the source-end task interface and the target-end task interface according to the task topology relationship; and performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface, efficient management of data transmission tasks and optimized resource allocation can be achieved, and at the same time, the efficiency and reliability of data transmission across data centers are greatly improved through the intelligent merging of task topology structures and the precise generation of data transmission links.

[0191] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data transmission method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0192] Please refer to Figure 7 The present application also provides a data transmission device, the data transmission device comprising:

[0193] A generation module, used to generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority;

[0194] A conversion module, used for converting the original data transmission task into a target data transmission task based on the dual-end data model;

[0195] A scoring module, used to perform priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue;

[0196] An execution module is used to perform data transmission based on the task scheduling queue and the target data transmission task.

[0197] The data transmission device provided by the present application adopts the data transmission method in the above embodiment to solve the technical problem of data transmission. Compared with the prior art, the beneficial effects of the data transmission device provided by the present application are the same as the beneficial effects of the data transmission method provided by the above embodiment, and other technical features in the data transmission device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0198] The present application provides a data transmission device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data transmission method in the above-mentioned embodiment one.

[0199] Reference below Figure 8 , which shows a schematic diagram of the structure of a data transmission device suitable for implementing the embodiment of the present application. The data transmission device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The data transmission device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0200] like Figure 8 As shown, the data transmission device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data transmission device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data transmission device to communicate with other devices wirelessly or by wire to exchange data. Although the data transmission device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0201] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0202] The data transmission device provided in the present application adopts the data transmission method in the above-mentioned embodiment. Compared with the prior art, the beneficial effects of the data transmission device provided in the present application are the same as the beneficial effects of the data transmission method provided in the above-mentioned embodiment, and other technical features in the data transmission device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0203] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0204] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0205] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data transmission method in the above-mentioned embodiment.

[0206] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0207] The computer-readable storage medium may be included in the data transmission device; or may exist independently without being assembled into the data transmission device.

[0208] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0210] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0211] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data transmission method, and can solve the technical problem of data transmission. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data transmission method provided in the above-mentioned embodiment, and will not be repeated here.

[0212] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data transmission method when executed by a processor.

[0213] The computer program product provided by the present application can solve the technical problem of data transmission. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data transmission method provided by the above embodiment, which will not be repeated here.

[0214] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data transmission method, characterized in that: The method comprises: Generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority; Based on the dual-end data model, converting the original data transmission task into a target data transmission task; Based on the task priority and a preset priority scheduling scoring algorithm, the target transmission task is scored for priority scheduling to obtain a task scheduling queue; Data transmission is performed based on the task scheduling queue and the target data transmission task.

2. The method according to claim 1, characterized in that The dual-end data model includes a source-end data model, and the step of converting the original data transmission task into a target data transmission task based on the dual-end data model includes: If the source data type of the source data model is a streaming data source, converting the original data transmission task into a streaming data transmission task; If the data granularity of the original data transmission task is a preset granularity specification, converting the original data transmission task into a stream data transmission task; If the source data type is not the streaming data source and the data granularity of the original data transmission task is not a preset granularity specification, the original data transmission task is converted into a batch data transmission task.

3. The method according to claim 1, characterized in that The data transmission task configuration information also includes the task planned execution time. The step of performing priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue includes: Based on the task priority and the task planned execution time corresponding to the target transmission task, respectively calculate the task priority conversion coefficient and the task duration conversion coefficient of the target transmission task; Calculate the task scheduling score of the target transmission task based on the task priority conversion coefficient, the task duration conversion coefficient, and the preset task priority scoring coefficient and task duration scoring coefficient; According to the task priority of the target transmission task and the task scheduling score, the target transmission task is sent to a target priority queue to obtain a task scheduling queue.

4. The method according to claim 1, characterized in that The dual-end data model includes a source-end data model and a target-end data model, and the method further includes: If the source data model and the target data model have different corresponding data source cluster information, then obtaining the security authentication configuration information of the original data transmission task, wherein the security authentication configuration information includes the cluster key distribution center KDC information; According to the security authentication configuration information, obtaining a ticket authorization ticket through the authentication server corresponding to the KDC information; According to the ticket authorization ticket, obtain the service ticket through the ticket authorization server corresponding to the KDC information; The service process and cluster security authentication corresponding to the original data transmission task are performed according to the service ticket.

5. The method according to claim 1, characterized in that The step of performing data transmission based on the task scheduling queue and the target data transmission task includes: Acquire the target data transmission task in the task scheduling queue, as well as the dependency relationship of the target data transmission task and the predefined source-end task interface and target-end task interface; Generate a task topology structure object corresponding to the target data transmission task based on the dependency relationship of the target data transmission task; Based on the task topology structure object and the source data model number corresponding to the target data transmission task, similar tasks are merged to obtain a task topology relationship; or, Based on the task topology structure object and the target end data model number corresponding to the target data transmission task, similar tasks are merged to obtain a task topology relationship; Generating a data transmission link between the source-end task interface and the target-end task interface according to the task topology relationship; Data transmission is performed based on a data transmission link between the source-end task interface and the target-end task interface.

6. The method according to claim 5, characterized in that The step of performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface comprises: Start the data transmission process and create a source transmission framework client group corresponding to the source task interface, wherein the source transmission framework client group includes a source transmission framework client and a source transmission framework server; Acquire source data through the source transmission framework client and the source transmission framework server; Creating a target-side transmission framework client group corresponding to the target-side task interface, wherein the target-side transmission framework client group includes a target-side transmission framework client and a target-side transmission framework server; The source-end data is written into the target end corresponding to the target-end task interface through the target-end transmission framework client and the target-end transmission framework server.

7. The method according to claim 6, characterized in that The step of performing data transmission based on the data transmission link between the source-end task interface and the target-end task interface also includes: If the source data is streaming data, obtaining the data offset corresponding to the task group ID based on the pre-acquired task group ID; According to the data offset, determine the task data consumption point and obtain the source end data based on the task data consumption point; Capture data transmission exceptions according to a predefined callback function to obtain exception capture results; Performing flow control on the data transmission process based on the abnormal capture result; If the source data is batch data, generating a target data check value based on the source data in the target; The data transmission task integrity check is performed based on the target-end data check value and the pre-acquired source-end data check value corresponding to the source-end data.

8. A data transmission device, characterized in that: The device comprises: A generation module, used to generate an original data transmission task according to pre-generated data transmission task configuration information, wherein the data transmission task configuration information includes a dual-end data model and a task priority; A conversion module, used for converting the original data transmission task into a target data transmission task based on the dual-end data model; A scoring module, used to perform priority scheduling scoring on the target transmission task based on the task priority and a preset priority scheduling scoring algorithm to obtain a task scheduling queue; An execution module is used to perform data transmission based on the task scheduling queue and the target data transmission task.

9. A data transmission device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data transmission method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data transmission method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data transmission method according to any one of claims 1 to 7 are implemented.

Images