A chunjun-based heterogeneous kafka security authentication real-time data synchronization method and device

Abstract

This application discloses a method and apparatus for real-time data synchronization with secure authentication in heterogeneous Kafka clusters based on Chunjun. In this method, the SASL / SCRAM authentication parameters for the source and target Kafka clusters are independently configured for both the Reader and Writer in the Chunjun job configuration. The Reader constructs a Consumer configuration based on the source SCRAM mechanism and dynamically loads the JAAS module to consume data, converting it to an internally unified format. After the Chunjun engine processes the data conversion, the Writer constructs a Producer configuration based on the target SCRAM mechanism and dynamically loads the JAAS module to write the data to the target. This invention achieves real-time data synchronization of heterogeneous SCRAM authentication clusters in a single job, offering advantages such as simple configuration, high development efficiency, good real-time performance and consistency, strong security, and low operation and maintenance costs.

Description

Technical Field

[0001] This application belongs to the field of big data real-time processing and data integration technology, specifically relating to a method, apparatus, computer-readable storage medium, and electronic device for real-time data synchronization with heterogeneous Kafka security authentication based on Chunjun. Background Technology

[0002] With the rapid development of big data technology, the distributed stream processing platform Kafka has been widely used in real-time data transmission scenarios. To ensure enterprise data security, Kafka clusters often enable the SASL (Simple Authentication and Security Layer) authentication mechanism, among which SCRAM (Salted Challenge Response Authentication Mechanism) is a common security authentication method and supports multiple hash algorithms, such as SCRAM-SHA-256 and SCRAM-SHA-512.

[0003] In actual business deployments, due to historical reasons, differences in departmental security policies, or system evolution requirements, enterprises often have multiple Kafka clusters using different SCRAM authentication algorithms simultaneously. For example, the source cluster might use the more secure SCRAM-SHA-512, while the target cluster might use SCRAM-SHA-256 for compatibility or performance reasons. These heterogeneous authentication scenarios pose new challenges to cross-cluster data synchronization.

[0004] Currently, the main methods for achieving data synchronization between Kafka clusters are as follows:

[0005] (1) Kafka MirrorMaker: This is a cross-cluster replication tool provided by Kafka. However, when the source and target clusters use different SCRAM mechanisms, the configuration is complicated. Usually, multiple instances need to be started or custom development is required. Its native support lacks flexibility when facing heterogeneous authentication.

[0006] (2) Custom application: Based on the Kafka Producer / Consumer API, the self-developed synchronization program has high flexibility, but it requires the implementation of fault tolerance, transactions, performance optimization and resource management, and the development and maintenance costs are high.

[0007] (3) Based on stream processing frameworks such as Flink / Spark Streaming: Synchronization is achieved through the built-in Kafka connector of the framework. For example, although Flink's Kafka connector supports SASL authentication, its configuration is difficult to dynamically adapt to multiple authentication modules when connecting to clusters of different SCRAM types in a single job. Data integration tools such as Chunjun (based on Flink) have Kafka read and write plugins that support SCRAM authentication, but at the configuration level, they usually tend to uniformly set attributes and fail to provide a simple and abstract independent configuration method for heterogeneous authentication mechanisms of the source and target ends. Users still need to perform additional adaptation.

[0008] In summary, the existing technology has the following main shortcomings:

[0009] (1) There is a lack of a simple, efficient and stable real-time data synchronization solution for different SCRAM authentication mechanisms between the Kafka source and target ends in a single job.

[0010] (2) When dealing with heterogeneous SCRAM authentication, existing tools are complex to configure and often require additional customization, which reduces development efficiency and ease of operation and maintenance.

[0011] (3) While ensuring real-time data synchronization, it is difficult to balance data consistency and system fault tolerance under different security protocols. Summary of the Invention

[0012] To address the aforementioned problems in existing technologies, this application proposes a novel real-time data synchronization method for heterogeneous Kafka security authentication based on Chunjun.

[0013] More specifically, this invention provides a method and apparatus for real-time data synchronization in a heterogeneous Kafka secure authentication environment using the Chunjun data integration tool based on the Apache Flink ecosystem. It aims to solve the problem of reliable and efficient real-time data synchronization between the source Kafka using SCRAM-SHA-512 authentication and the target Kafka using SCRAM-SHA-256 authentication in complex scenarios.

[0014] This invention enhances and optimizes the Chunjun Kafka Reader and Kafka Writer plugins, enabling them to independently and explicitly specify the SCRAM authentication mechanism and related parameters of the source and target Kafka clusters within a single Chunjun job configuration. It also allows for independent configuration and management of authentication information for different SCRAM mechanisms, thereby seamlessly bridging Kafka clusters with different security protocols within a unified Chunjun job. This ensures the real-time performance, security, and consistency of data synchronization, achieving efficient, secure, and real-time heterogeneous authentication Kafka data synchronization.

[0015] To achieve the above objectives, the present invention employs the following technical strategies:

[0016] 1. Seamless Integration of Heterogeneous SCRAM Authentication Mechanisms: At the Chunjun job configuration level, independent and unrelated authentication parameter configuration interfaces are provided for Kafka Reader (data source) and Kafka Writer (data destination). This allows Chunjun to initialize a consumer instance using SCRAM-SHA-512 authentication for Kafka Reader and a producer instance using SCRAM-SHA-256 authentication for Kafka Writer at startup, and efficiently transfer data between the two. This invention, for the first time, achieves seamless, independent configuration and collaborative operation of two different security authentication mechanisms, SCRAM-SHA-512 and SCRAM-SHA-256, for source and destination Kafka clusters within Chunjun's unified configuration framework, solving the complexity and limitations of traditional methods in this scenario.

[0017] 2. Independent and differentiated authentication parameter management: By providing independent securityProperties configuration blocks in the Reader and Writer plugins respectively, the configuration of authentication parameters does not interfere with each other, greatly improving flexibility and maintainability.

[0018] 3. Enhance Chunjun's applicability in complex enterprise environments: This invention expands Chunjun's capabilities as a general data integration tool, enabling it to better adapt to complex IT environments where multiple security policies coexist within an enterprise, and lowering the threshold for data sharing and synchronization between multiple tenants or departments.

[0019] 4. End-to-end secure and reliable synchronization based on the stream processing framework: Combining the stream processing features of Chunjun's underlying Flink, while ensuring real-time data transmission, it ensures data consistency and high reliability across heterogeneous security authentication boundaries through built-in checkpoints, fault tolerance, and Exactly-Once (or At-Least-Once) semantic support.

[0020] Specifically, this application provides the following technical solutions:

[0021] The first aspect of this application provides a real-time data synchronization method for heterogeneous Kafka security authentication based on Chunjun, such as... Figure 1 As shown, it includes the following steps:

[0022] S1. Job Configuration: In the Chunjun job configuration file, the connection parameters and SASL / SCRAM authentication parameters of the source Kafka cluster and the target Kafka cluster are configured independently for the Kafka Reader module and the Kafka Writer module, respectively. The source and target use different SCRAM hash algorithms.

[0023] S2. Consumer initialization: The Kafka Reader plugin parses the authentication parameters of the source Kafka cluster, constructs a Kafka consumer client configuration containing the corresponding SCRAM mechanism, dynamically loads the JAAS authentication module, instantiates the KafkaConsumer, and connects to the source Kafka cluster.

[0024] S3. Data Consumption: The Kafka Consumer consumes data from a specified Topic in the source Kafka cluster, converts the consumed data into a unified data format within Chunjun, and then transmits it to the Chunjun engine.

[0025] S4. Data Processing: The Chunjun engine performs data conversion, filtering, and / or cleaning operations on the data in the unified data format within Chunjun.

[0026] S5. Producer Initialization: The Kafka Writer plugin parses the authentication parameters of the target Kafka cluster, constructs a Kafka producer client configuration containing the corresponding SCRAM mechanism, dynamically loads the JAAS authentication module, instantiates KafkaProducer, and connects to the target Kafka cluster.

[0027] S6. Data Writing: The Kafka Producer serializes the data processed by the Chunjun engine into Kafka message format and sends it to the specified Topic of the target Kafka cluster to achieve real-time data synchronization across heterogeneous SCRAM-authenticated Kafka clusters.

[0028] Furthermore, in the method of this application, the source Kafka cluster in step S1 adopts the SCRAM-SHA-512 authentication mechanism, and the target Kafka cluster adopts the SCRAM-SHA-256 authentication mechanism.

[0029] Furthermore, in the method of this application, the SASL / SCRAM authentication parameters mentioned in step S1 include: security.protocol, sasl.mechanism, and sasl.jaas.config.

[0030] Furthermore, in the method of this application, the dynamic loading of the JAAS authentication module in step S2 and / or step S5 specifically means: dynamically setting or loading the JAAS configuration at JVM runtime so that the Kafka client can call the corresponding SCRAM authentication module to complete authentication.

[0031] Furthermore, in the method of this application, the unified data format of Chunjun internally mentioned in step S3 is RowData format.

[0032] Furthermore, in the method of this application, the Chunjun engine mentioned in step S4 is implemented based on Apache Flink, and the data processing is performed by relying on Flink's DataStream API.

[0033] Furthermore, the method of this application also includes:

[0034] S7. Fault Tolerance and Recovery: Based on Apache Flink's checkpoint mechanism, the execution status of real-time data synchronization jobs is periodically snapshotted and resumed from the most recent checkpoint in the event of job failure, so as to ensure data consistency and synchronization continuity.

[0035] Furthermore, in the method of this application, the Chunjun job configuration file mentioned in step S1 is in JSON format; the authentication parameters of the Kafka Reader module and the Kafka Writer module are configured in their respective independent securityProperties configuration blocks.

[0036] The second aspect of this application provides a Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization device. When the device operates, it implements the steps of the aforementioned Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method, such as... Figure 2 As shown, the device includes:

[0037] The configuration parsing module is used to parse the Chunjun job configuration file, identify and extract the connection parameters and SASL / SCRAM authentication parameters of the source Kafka cluster and the target Kafka cluster;

[0038] The Kafka Reader module is used to initialize and run a Kafka Consumer configured with the corresponding SCRAM mechanism according to the source authentication parameters provided by the configuration parsing module, and to consume data from the source Kafka cluster in real time.

[0039] The data conversion and transmission module is used to convert the data consumed by the Kafka Reader module into the unified data format within Chunjun and transmit it to the Chunjun engine.

[0040] The Kafka Writer module is used to initialize and run a Kafka Producer configured with a corresponding SCRAM mechanism according to the target authentication parameters provided by the configuration parsing module, and write the data processed by the Chunjun engine to the target Kafka cluster.

[0041] The task scheduling and fault tolerance module is used to schedule and execute data synchronization jobs based on Apache Flink, and relies on the checkpoint mechanism to realize fault recovery, so as to ensure the reliability and consistency of data synchronization.

[0042] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0043] Memory: Used to store computer programs;

[0044] Processor: Used to execute the computer program to implement the steps of the aforementioned Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method.

[0045] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method.

[0046] In summary, compared with the prior art, the solution of the present invention has the following advantages:

[0047] (1) Simplify the configuration and management of heterogeneous security authentication: Users can clearly and independently configure the heterogeneous SCRAM authentication parameters of the source and target Kafka through Chunjun's single job configuration file, which greatly reduces the configuration complexity and management difficulty.

[0048] (2) Improve development efficiency: Kafka data synchronization tasks in heterogeneous authentication scenarios can be quickly built and deployed without writing complex custom code, shortening the development cycle.

[0049] (3) Ensure data real-time performance and consistency: Make full use of Chunjun (Flink)’s streaming processing capabilities, checkpoint mechanism and fault tolerance features to ensure that data is synchronized in real time with low latency and high throughput between Kafka clusters with different authentication mechanisms, and maintain data consistency.

[0050] (4) Enhance system security: While achieving data synchronization, the system strictly follows the authentication requirements of source SCRAM-SHA-512 and target SCRAM-SHA-256 to ensure the security of authentication during data transmission.

[0051] (5) Reduced operation and maintenance costs: The unified Chunjun job configuration and monitoring interface makes it easier for operation and maintenance personnel to manage and monitor data synchronization tasks, reducing the operation and maintenance burden caused by multiple systems or complex configurations.

[0052] (6) Good scalability: The design concept of this invention is easy to expand and can support more types of Kafka authentication mechanisms (such as Kerberos, SSL client authentication, etc.) or heterogeneous synchronization of different versions of SCRAM mechanisms in the future.

[0053] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0054] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0055] Figure 1This is a flowchart illustrating the overall implementation of the heterogeneous Kafka security authentication real-time data synchronization method based on Chunjun in this application.

[0056] Figure 2 This is a structural diagram of the heterogeneous Kafka secure authentication real-time data synchronization device based on Chunjun, as described in this application.

[0057] Figure 3 This is a diagram illustrating the complete implementation process of the solution proposed in this application.

[0058] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0060] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0061] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0062] Example: A method and apparatus for real-time data synchronization with secure authentication in heterogeneous Kafka systems based on Chunjun.

[0063] The present invention includes the following core design:

[0064] 1. Unify the configuration of entry points and separate authentication parameters:

[0065] In Chunjun's job configuration (usually in JSON format), the parameter or properties fields of the two independent modules, Reader and Writer, allow users to define complete Kafka client configurations separately, including bootstrap.servers, sasl.mechanism, security.protocol, and sasl.jaas.config. Among these, sasl.mechanism can be independently set to SCRAM-SHA-512 and SCRAM-SHA-256.

[0066] 2. Enhanced Kafka Reader plugin:

[0067] During the plugin initialization phase, the corresponding Kafka consumer client configuration is built and loaded based on the value of the sasl.mechanism field in the configuration (e.g., SCRAM-SHA-512).

[0068] Dynamically configure the JAAS (Java Authentication and Authorization Service) login module to support the username and password required for SCRAM-SHA-512 authentication.

[0069] Based on this configuration, instantiate a Kafka Consumer and consume data from the source Kafka cluster.

[0070] 3. Enhanced Kafka Writer plugin:

[0071] During the plugin initialization phase, the corresponding Kafka producer client configuration is built and loaded based on the value of the sasl.mechanism field in the configuration (e.g., SCRAM-SHA-256).

[0072] Dynamically configure the JAAS login module to support the username and password required for SCRAM-SHA-256 authentication.

[0073] Based on this configuration, instantiate a Kafka Producer and write data to the target Kafka cluster.

[0074] 4. Chunjun Data Processing Layer:

[0075] The Chunjun engine (based on Apache Flink) is responsible for receiving data consumed by the Kafka Reader plugin, performing internal transformations (such as data type conversion and simple filtering), and then passing the data to the Kafka Writer plugin for writing. Chunjun's checkpoint mechanism and fault recovery capabilities ensure the reliability of data synchronization and Exactly-Once (or At-Least-Once) semantics.

[0076] Figure 3 The diagram shows the complete implementation process of the present invention, including:

[0077] 1. Chunjun job start: Triggers the start process of the Chunjun data synchronization job.

[0078] 2. Kafka Reader consumes data: Kafka Reader uses the SCRAM-SHA-512 authentication mechanism to consume data from the source Kafka cluster.

[0079] 3. Data format conversion: Convert the consumed data into the RowData format defined internally by Chunjun.

[0080] 4. Chunjun Engine Data Processing: In the Chunjun engine, data transformation, filtering, cleaning and other operations are performed on RowData data according to the configured transformation rules.

[0081] 5. Kafka Writer serializes data: Kafka Writer serializes the processed RowData into Kafka message format.

[0082] 6. Kafka Writer produces data: The Kafka Writer uses its configured Producer instance with SCRAM-SHA-256 authentication mechanism to send serialized Kafka messages to the specified Topic of the target Kafka cluster.

[0083] 7. Continuous data synchronization: After the above steps are completed, the data synchronization process continues to run in a loop to achieve real-time cross-cluster data synchronization.

[0084] To more clearly illustrate the technical solution of this application, the detailed implementation process of this solution will be further explained below through specific scenario embodiments.

[0085] The present invention is achieved through the following steps:

[0086] Step 1: Configure the Chunjun job JSON file

[0087] Create a Chunjun job configuration file, such as kafka_sync_job.json, where the Reader and Writer sections will be configured with Kafka connection parameters and authentication parameters independently.

[0088] {

[0089] "job": {

[0090] "content": [

[0091] {

[0092] "reader": {

[0093] "name": "kafkareader",

[0094] "parameter": {

[0095] "bootstrapServers": "source_kafka_broker1:9092,source_kafka_broker2:9092",

[0096] "groupId": "chunjun_source_group",

[0097] "topic": ["source_topic_a", "source_topic_b"],

[0098] "codec": "json", / / or "raw", "csv", etc.

[0099] "partitionDiscoveryIntervalMillis": 30000,

[0100] "securityProperties": {

[0101] "security.protocol": "SASL_PLAINTEXT", / / or SASL_SSL

[0102] "sasl.mechanism": "SCRAM-SHA-512",

[0103] "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"source_user\" password=\"source_password\";"

[0104] },

[0105] "format": "json" / / Define the data format read

[0106] }

[0107] },

[0108] "writer": {

[0109] "name": "kafkawriter",

[0110] "parameter": {

[0111] "bootstrapServers": "target_kafka_broker1:9092,target_kafka_broker2:9092",

[0112] "topic": "target_topic",

[0113] "codec": "json",

[0114] "producerSettings": {

[0115] "acks": "all",

[0116] "retries": 5,

[0117] "batch.size": 16384,

[0118] "linger.ms": 5

[0119] },

[0120] "securityProperties": {

[0121] "security.protocol": "SASL_PLAINTEXT", / / or SASL_SSL

[0122] "sasl.mechanism": "SCRAM-SHA-256",

[0123] "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"target_user\" password=\"target_password\";"

[0124] },

[0125] "partitioner": "roundRobin" / / or "key"

[0126] }

[0127] }

[0128] }

[0129] ,

[0130] "setting": {

[0131] "speed": {

[0132] "channel": 1,

[0133] "bytes": 0

[0134] },

[0135] "checkpoint": {

[0136] "interval": 60000,

[0137] "timeout": 60000,

[0138] "dataDir": "hdfs: / / / chunjun / checkpoint",

[0139] "mode": "EXACTLY_ONCE",

[0140] "retries": 3

[0141] },

[0142] "restartstrategy": {

[0143] "strategy": "fixed-delay",

[0144] "attempts": 3,

[0145] "delay": "10s"

[0146] }

[0147] }

[0148] }

[0149] }

[0150] illustrate:

[0151] The reader.parameter.securityProperties file contains the configuration for sasl.mechanism: SCRAM-SHA-512 and the corresponding sasl.jaas.config.

[0152] The writer.parameter.securityProperties file configures sasl.mechanism: SCRAM-SHA-256 and the corresponding sasl.jaas.config.

[0153] The username and password in sasl.jaas.config should be replaced with actual credentials, and it is recommended to manage them through external security configurations or environment variables rather than hardcoding them directly.

[0154] The `security.protocol` option can be either `SASL_PLAINTEXT` or `SASL_SSL` depending on the actual SSL / TLS configuration of the Kafka cluster. If `SASL_SSL` is selected, SSL-related parameters (such as `ssl.truststore.location`) also need to be configured.

[0155] Step 2: Implementation of the Chunjun Kafka Reader Plugin

[0156] 1. Parameter parsing: During initialization, the Chunjun Kafka Reader plugin parses all parameters in the Reader configuration, especially security.protocol, sasl.mechanism, and sasl.jaas.config under the securityProperties dictionary.

[0157] 2. Consumer Configuration Construction: Based on the parsed parameters, construct a java.util.Properties object, which contains all the configurations required by the Kafka consumer client, for example:

[0158] props.setProperty("bootstrap.servers", bootstrapServers);

[0159] props.setProperty("group.id", groupId);

[0160] props.setProperty("security.protocol", "SASL_PLAINTEXT");

[0161] props.setProperty("sasl.mechanism", "SCRAM-SHA-512");

[0162] props.setProperty("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"source_user\" password=\"source_password\";");

[0163] Other Kafka Consumer related configurations.

[0164] 3. JAAS Configuration Injection: For the sasl.jaas.config parameter, Chunjun internally uses the capabilities of the Flink Kafka connector to dynamically set or load the JAAS configuration at JVM runtime, ensuring that Kafka clients can correctly use the SCRAM-SHA-512 authentication module for authentication.

[0165] 4. Kafka Consumer Instantiation: Instantiate org.apache.kafka.clients.consumer.KafkaConsumer using the constructed Properties object. This consumer will be able to successfully connect to and authenticate to the source Kafka cluster using SCRAM-SHA-512.

[0166] 5. Data Consumption and Transformation: Consumers consume data from a specified Topic and convert the raw byte data into Chunjun's internal RowData format according to the codec and format parameters before passing it to Chunjun's next stage of processing.

[0167] Step 3: Implementation of the Chunjun Kafka Writer Plugin

[0168] 1. Parameter parsing: During initialization, the Chunjun Kafka Writer plugin parses all parameters in the Writer configuration, especially security.protocol, sasl.mechanism, and sasl.jaas.config under the securityProperties dictionary.

[0169] 2. Producer Configuration Construction: Based on the parsed parameters, a java.util.Properties object is constructed, containing all the configurations required by the Kafka producer client, such as:

[0170] props.setProperty("bootstrap.servers", bootstrapServers);

[0171] props.setProperty("security.protocol", "SASL_PLAINTEXT");

[0172] props.setProperty("sasl.mechanism", "SCRAM-SHA-256");

[0173] props.setProperty("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"target_user\" password=\"target_password\";");

[0174] Other Kafka Producer related configurations.

[0175] 3. JAAS configuration injection: Similar to Reader, dynamically set or load JAAS configuration to ensure that Kafka clients can correctly use the SCRAM-SHA-256 authentication module for authentication.

[0176] 4. Kafka Producer Instantiation: Instantiate org.apache.kafka.clients.producer.KafkaProducer using the constructed Properties object. This producer will be able to successfully connect to and authenticate to the target Kafka cluster using SCRAM-SHA-256.

[0177] 5. Data Sending: Receive RowData data from the previous stage of Chunjun, serialize it into key-value pairs of Kafka messages, and send it to the specified Topic of the target Kafka cluster.

[0178] Step 4: Scheduling and Execution of the Chunjun Engine

[0179] 1. Task Submission: Submit the configured kafka_sync_job.json file to the Chunjun command-line tool or deployment platform.

[0180] 2. Flink Job Generation: Chunjun parses the JSON configuration and converts it into a standard Apache Flink job (DataStream API). This job contains a KafkaSource (implemented by the Reader plugin), zero or more transformation operations (if configured), and a KafkaSink (implemented by the Writer plugin).

[0181] 3. Job Execution: The Flink cluster receives and executes the job. The Reader is responsible for consuming data from the source, and the Writer is responsible for producing data to the target. Flink's checkpointing and recovery mechanisms ensure that even in the event of a failure, the task can be recovered from the nearest checkpoint, guaranteeing that data is not lost or duplicated.

[0182] The present invention also provides an apparatus for implementing the above method, the apparatus comprising:

[0183] 1. Configuration parsing module: Used to parse the JSON configuration file of Chunjun job and identify the source Kafka connection parameters (including SCRAM-SHA-512 authentication configuration) and the target Kafka connection parameters (including SCRAM-SHA-256 authentication configuration).

[0184] 2. Kafka Reader Module: Coupled within the Chunjun framework, it initializes and runs a Kafka consumer configured with SCRAM-SHA-512 authentication based on the source Kafka connection parameters provided by the configuration parsing module. This consumer is responsible for consuming data from the source Kafka cluster in real time.

[0185] 3. Data Conversion and Transmission Module: Coupled within the Chunjun framework, this module receives data consumed by the Kafka Reader module, performs necessary data type conversions or format processing, and transmits the processed data in an internally unified format.

[0186] 4. Kafka Writer module: Coupled within the Chunjun framework, it initializes and runs a Kafka producer configured with SCRAM-SHA-256 authentication based on the target Kafka connection parameters provided by the configuration parsing module, and is responsible for writing data to the target Kafka cluster.

[0187] 5. Task Scheduling and Fault Tolerance Module: Coupled within the Chunjun framework, this module utilizes Apache Flink's checkpointing and recovery mechanisms to schedule, monitor, and recover from failures in the entire data synchronization job, ensuring the reliability and consistency of the data synchronization process.

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0189] like Figure 4 As shown, embodiments of this application also disclose an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the above-described Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method.

[0190] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0191] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0192] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0193] Conventional procedural programming languages, such as "C" or similar programming languages.

[0194] The execution methods of program code include, but are not limited to:

[0195] It runs entirely on the user's computer;

[0196] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0197] Execute as a standalone software package;

[0198] It is executed entirely on a remote computer or server.

[0199] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0200] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method disclosed in this application.

[0201] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0202] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0203] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0204] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0205] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0206] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0207] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0208] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0209] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0210] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0211] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0212] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0213] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method disclosed in this application. When this computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0214] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0215] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A real-time data synchronization method for secure authentication based on Chunjun-based heterogeneous Kafka, characterized in that, Includes the following steps: S1. Job Configuration: In the Chunjun job configuration file, the connection parameters and SASL / SCRAM authentication parameters of the source Kafka cluster and the target Kafka cluster are configured independently for the Kafka Reader module and the Kafka Writer module, respectively. The source and target use different SCRAM hash algorithms. S2. Consumer initialization: The Kafka Reader plugin parses the authentication parameters of the source Kafka cluster, constructs a Kafka consumer client configuration containing the corresponding SCRAM mechanism, dynamically loads the JAAS authentication module, instantiates the KafkaConsumer, and connects to the source Kafka cluster. S3. Data Consumption: The Kafka Consumer consumes data from a specified Topic in the source Kafka cluster, converts the consumed data into a unified data format within Chunjun, and then transmits it to the Chunjun engine. S4. Data Processing: The Chunjun engine performs data conversion, filtering, and / or cleaning operations on the data in the unified data format within Chunjun. S5. Producer Initialization: The Kafka Writer plugin parses the authentication parameters of the target Kafka cluster, constructs a Kafka producer client configuration containing the corresponding SCRAM mechanism, dynamically loads the JAAS authentication module, instantiates KafkaProducer, and connects to the target Kafka cluster. S6. Data Writing: The Kafka Producer serializes the data processed by the Chunjun engine into Kafka message format and sends it to the specified Topic of the target Kafka cluster to achieve real-time data synchronization across heterogeneous SCRAM-authenticated Kafka clusters.

2. The method according to claim 1, characterized in that, In step S1, the source Kafka cluster uses the SCRAM-SHA-512 authentication mechanism, and the target Kafka cluster uses the SCRAM-SHA-256 authentication mechanism.

3. The method according to claim 1, characterized in that, The SASL / SCRAM authentication parameters mentioned in step S1 include: security.protocol, sasl.mechanism, and sasl.jaas.config.

4. The method according to claim 1, characterized in that, The dynamic loading of the JAAS authentication module in step S2 and / or step S5 specifically involves: dynamically setting or loading the JAAS configuration during JVM runtime so that the Kafka client can call the corresponding SCRAM authentication module to complete authentication.

5. The method according to claim 1, characterized in that, The unified data format within Chunjun mentioned in step S3 is RowData format.

6. The method according to claim 1, characterized in that, The Chunjun engine mentioned in step S4 is implemented based on Apache Flink, and the data processing relies on Flink's DataStream API for execution.

7. The method according to claim 1, characterized in that, The method further includes: S7. Fault Tolerance and Recovery: Based on Apache Flink's checkpoint mechanism, the execution status of real-time data synchronization jobs is periodically snapshotted and resumed from the most recent checkpoint in the event of job failure, so as to ensure data consistency and synchronization continuity.

8. The method according to claim 1, characterized in that, The Chunjun job configuration file mentioned in step S1 is in JSON format; the authentication parameters of the Kafka Reader module and the Kafka Writer module are configured in their respective independent securityProperties configuration blocks.

9. A heterogeneous Kafka-based secure authentication real-time data synchronization device based on Chunjun, characterized in that, When the device is in operation, it implements the steps of the real-time data synchronization method for heterogeneous Kafka security authentication based on Chunjun as described in any one of claims 1-8, including: The configuration parsing module is used to parse the Chunjun job configuration file, identify and extract the connection parameters and SASL / SCRAM authentication parameters of the source Kafka cluster and the target Kafka cluster; The Kafka Reader module is used to initialize and run a Kafka Consumer configured with the corresponding SCRAM mechanism according to the source authentication parameters provided by the configuration parsing module, and to consume data from the source Kafka cluster in real time. The data conversion and transmission module is used to convert the data consumed by the Kafka Reader module into the unified data format within Chunjun and transmit it to the Chunjun engine. The Kafka Writer module is used to initialize and run a Kafka Producer configured with a corresponding SCRAM mechanism according to the target authentication parameters provided by the configuration parsing module, and write the data processed by the Chunjun engine to the target Kafka cluster. The task scheduling and fault tolerance module is used to schedule and execute data synchronization jobs based on Apache Flink, and relies on the checkpoint mechanism to realize fault recovery, so as to ensure the reliability and consistency of data synchronization.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the Chunjun-based heterogeneous Kafka secure authentication real-time data synchronization method as described in any one of claims 1-8.

Images