System and method for realizing data synchronization based on NIFI

Abstract

The invention particularly relates to a system and a method for realizing data synchronization based on NIFI. The system for realizing data synchronization based on NIFI comprises an NIFI controller service extension module, an encryption and decryption custom realization module, a jar file resource uploading and management module, an encryption and decryption service creation and management module, a data source management module and a data synchronization task creation and management module. According to the system and the method for realizing data synchronization based on the NIFI, the encryption algorithm can be dynamically loaded and updated, the problem that the encryption algorithm is fixed and poor in expansibility in the prior art is solved, the data security is improved, the maintenance and upgrading processes are simplified, and the flexibility and the efficiency of the data synchronization system are improved.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a system and method for realizing data synchronization based on NIFI. Background Art

[0002] In the prior art, data synchronization is one of the common requirements in information systems, especially in distributed system architectures, where data consistency needs to be maintained between multiple systems or components. However, in the process of data synchronization, data security is a key issue that cannot be ignored, especially when data is transmitted over the network, it is vulnerable to the risk of interception and eavesdropping. Therefore, encryption algorithms are usually applied in the process of data synchronization to protect the confidentiality and integrity of the data.

[0003] Currently, most data synchronization systems use fixed encryption algorithms, which means that once the system is deployed, the encryption method is difficult to change unless the system is reconfigured or upgraded. This design has the following problems:

[0004] 1) Poor scalability: Fixed encryption algorithms limit the system's ability to adapt to emerging, more secure encryption algorithms, making it difficult to meet ever-changing security requirements.

[0005] 2). Insufficient flexibility: If the encryption algorithm needs to be changed, it is usually necessary to shut down for maintenance and reconfigure the system, which is not only time-consuming and labor-intensive, but may also cause data synchronization interruptions and affect business continuity.

[0006] 3) Limited security: With the development of technology, old encryption algorithms may gradually expose security vulnerabilities, but the design of fixed algorithms makes it difficult for the system to upgrade encryption strategies in a timely manner.

[0007] In order to solve the above problems, the present invention proposes a system and method for realizing data synchronization based on NIFI. Summary of the invention

[0008] In order to make up for the defects of the prior art, the present invention provides a simple and efficient system and method for realizing data synchronization based on NIFI.

[0009] The present invention is achieved through the following technical solutions:

[0010] A system for realizing data synchronization based on NIFI, characterized in that it includes:

[0011] NIFI controller service extension module, used to customize the extension of the controller service ControllerService for encryption and decryption;

[0012] The encryption and decryption custom implementation module is used to customize the encryption and decryption algorithm logic and package it into an executable jar file package;

[0013] The jar file resource upload and management module provides an interface operation UI for uploading and managing executable jar file packages with custom encryption and decryption algorithms. The management methods include viewing, editing, downloading and deleting.

[0014] The encryption and decryption service creation and management module is used to define the encryption and decryption service name, select the uploaded jar package file resources, create and manage the controller service instance based on the custom encryption and decryption algorithm, and the management methods include viewing, editing and deleting;

[0015] The data source management module is used to create the source and destination data sources required for data synchronization tasks;

[0016] The data synchronization task creation and management module is used to select the source and destination data sources, select whether to encrypt and the encryption and decryption services, and create and manage encryption and decryption data synchronization tasks in the NIFI framework.

[0017] The NIFI controller service extension module creates an interface class Security_Ser, which inherits from the controller service Controller Service of the NIFI framework and contains four interfaces for initialization, checking initialization status, encryption and decryption;

[0018] Create an external interface class DecryptAndEncrypt (encryption and decryption) to connect to the custom encryption and decryption methods, and implement four interfaces for initialization, checking initialization status, encryption, and decryption;

[0019] Create an implementation class SecurityService (security service), implement the Security_Ser interface, and inherit from the Abstract Controller Service. The configuration parameters include the jar package path, class name, and configuration items required for initialization.

[0020] The encryption and decryption custom implementation module creates a new Java interface project, creates an interface class DecryptAndEncrypt (encryption and decryption), and defines an interface consistent with the DecryptAndEncrypt class, namely, four interfaces for initialization, checking initialization status, encryption, and decryption;

[0021] Create a new Java project, create an implementation class DecryptAndEncryptImpl, implement the DecryptAndEncrypt interface, and package the project into an executable jar file.

[0022] A method for realizing data synchronization based on NIFI includes the following steps:

[0023] Step S1, expand the NIFI controller service and create a controller service for encryption and decryption;

[0024] Step S2: Customize the encryption and decryption algorithm and package it into an executable jar file package;

[0025] Step S3, upload the jar package file resource to the specified path;

[0026] Step S4: Create and manage encryption and decryption service instances;

[0027] Step S5, creating source and destination data sources;

[0028] Step S6: Create an encryption and decryption data synchronization task, select the source and destination data sources, whether to encrypt, and the corresponding encryption and decryption services.

[0029] In the step S1, extending the NIFI controller service includes creating a Security_Ser interface class, a DecryptAndEncrypt interface class and a SecurityService implementation class, and configuring corresponding parameters;

[0030] The specific steps are as follows:

[0031] Step S1.1, create an interface class Security_Ser, which inherits from the controller service Controller Service of the NIFI framework and contains four interfaces for initialization, checking initialization status, encryption and decryption;

[0032] Step S1.2, create an external interface class DecryptAndEncrypt (encryption and decryption) for docking with custom encryption and decryption methods, and implement four interfaces for initialization, checking initialization status, encryption and decryption;

[0033] Step S1.3, create an implementation class SecurityService (security service), implement the Security_Ser interface, and inherit from the Abstract Controller Service. The configuration parameters include the jar package path, class name, and configuration items required for initialization.

[0034] In step S2, customizing the encryption and decryption algorithm includes creating a DecryptAndEncrypt interface class and a DecryptAndEncryptImpl implementation class, and packaging the project into an executable jar;

[0035] The specific steps are as follows:

[0036] Step S2.1, create a new Java interface project, create an interface class DecryptAndEncrypt (encryption and decryption), define the interface consistent with the DecryptAndEncrypt class, that is, four interfaces for initialization, checking initialization status, encryption and decryption;

[0037] Step S2.2: Create a new Java project, create an implementation class DecryptAndEncryptImpl, implement the DecryptAndEncrypt interface, and package the project into an executable jar file package.

[0038] A system and device for realizing data synchronization based on NIFI, characterized in that: it comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method when executing the computer program.

[0039] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method when executed by a processor.

[0040] The beneficial effects of the present invention are as follows: the system and method for realizing data synchronization based on NIFI can dynamically load and update the encryption algorithm, solve the problems of fixed encryption algorithm and poor scalability in the prior art, improve data security, simplify the maintenance and upgrade process, and improve the flexibility and efficiency of the data synchronization system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Attached Figure 1 The figure is a schematic diagram of a system for realizing data synchronization based on NIFI in the present invention.

[0043] Attached Figure 2 The figure is a schematic diagram of a method for realizing data synchronization based on NIFI according to the present invention.

[0044] Attached Figure 3 This is a schematic diagram of the encryption and decryption data synchronization task flow of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] NIFI is an easy-to-use, powerful, and reliable service for processing and distributing data. Through NIFI's controller service extension mechanism, dynamic loading and management of encryption and decryption algorithms can be achieved.

[0047] The system for data synchronization based on NIFI includes:

[0048] NIFI controller service extension module, used to customize the extension of the controller service ControllerService for encryption and decryption;

[0049] The controller service in NIFI is a pre-made service that can be referenced by the processor or support the processor. For example, a SQL read processor needs a JDBC (Java Database Connectivity) connection to access the database. Here, the controller service can be a JDBC connection pool service.

[0050] The encryption and decryption custom implementation module is used to customize the encryption and decryption algorithm logic and package it into an executable jar file package;

[0051] The jar file resource upload and management module provides an interface operation UI for uploading the packaged jar file package to the selected path. After the upload is successful, the executable jar file package of the custom encryption and decryption algorithm can be viewed, edited, downloaded and deleted;

[0052] The encryption and decryption service creation and management module is used to define the encryption and decryption service name, select the uploaded jar package file resources, create and manage the controller service instance based on the custom encryption and decryption algorithm; after successful creation, the controller service instance can be viewed, edited, deleted and other management operations through the interface operation UI.

[0053] The data source management module is used to create the source and destination data sources required for data synchronization tasks;

[0054] The data synchronization task creation and management module is used to select the source and destination data sources, select whether to encrypt and the encryption and decryption services, and create and manage encryption and decryption data synchronization tasks in the NIFI framework.

[0055] The NIFI controller service extension module creates an interface class Security_Ser, which inherits from the controller service Controller Service of the NIFI framework and contains four interfaces for initialization, checking initialization status, encryption and decryption;

[0056] Create an external interface class DecryptAndEncrypt (encryption and decryption) to connect to the custom encryption and decryption methods, and implement four interfaces for initialization, checking initialization status, encryption, and decryption;

[0057] Create an implementation class SecurityService (security service), implement the Security_Ser interface, and inherit from the Abstract Controller Service. The configuration parameters include the jar package path, class name, and configuration items required for initialization.

[0058] The encryption and decryption custom implementation module creates a new Java interface project, creates an interface class DecryptAndEncrypt (encryption and decryption), and defines an interface consistent with the DecryptAndEncrypt class, namely, four interfaces for initialization, checking initialization status, encryption, and decryption;

[0059] Create a new Java project, create an implementation class DecryptAndEncryptImpl, implement the DecryptAndEncrypt interface, and package the project into an executable jar file.

[0060] The method for realizing data synchronization based on NIFI includes the following steps:

[0061] Step S1, expand the NIFI controller service and create a controller service for encryption and decryption;

[0062] In the step S1, extending the NIFI controller service includes creating a Security_Ser interface class, a DecryptAndEncrypt interface class and a SecurityService implementation class, and configuring corresponding parameters;

[0063] The specific steps are as follows:

[0064] Step S1.1, create an interface class Security_Ser, which inherits from the Controller Service of the NIFI framework and contains four interfaces for initialization, checking initialization status, encryption and decryption; they are:

[0065] (1) reInit: used for initialization;

[0066] (2) checkInitStatus: used to determine the initialization status;

[0067] (3)encrypt: used to implement encryption;

[0068] (4)decrypt: used to implement decryption;

[0069] The code example is as follows:

[0070] public interface SecuritySer extends ControllerService{

[0071] String encrypt(String data) throws Exception;

[0072] String decrypt(String data) throws Exception;

[0073] boolean checkInitStatus();

[0074] boolean reInit();

[0075] Step S1.2, create an external interface class DecryptAndEncrypt (encryption and decryption) for docking with custom encryption and decryption methods, and implement four interfaces for initialization, checking initialization status, encryption and decryption; subsequently, use the reflection mechanism to instantiate the custom encryption and decryption method. The docking implementation method must implement the following interface methods, namely:

[0076] (1) init: used for initialization

[0077] (2)checkInitStatus: used to determine the initialization status

[0078] (3)encrypt: used to implement encryption

[0079] (4)decrypt: used to implement decryption

[0080] The code example is as follows:

[0081] public interface DecryptAndEncrypt{

[0082] boolean init(Map param);

[0083] String encrypt(String data);

[0084] String decrypt(String data);

[0085] boolean checkInitStatus();

[0086] Step S1.3, create an implementation class SecurityService (security service), implement the Security_Ser interface, and inherit from the Abstract Controller Service. The configuration parameters include the jar package path, class name, and configuration items required for initialization.

[0087] Jar package path: The configuration path of the custom-implemented encryption and decryption jar package. Upload the custom-implemented encryption and decryption jar package to this path, and the jar package can be automatically hot-loaded.

[0088] Class name: The class path for encryption and decryption of the custom implementation interface, used to instantiate the implementation method.

[0089] Step S2: Customize the encryption and decryption algorithm and package it into an executable jar file package;

[0090] In step S2, customizing the encryption and decryption algorithm includes creating a DecryptAndEncrypt interface class and a DecryptAndEncryptImpl implementation class, and packaging the project into an executable jar;

[0091] The specific steps are as follows:

[0092] Step S2.1, create a new Java interface project, create an interface class DecryptAndEncrypt (encryption and decryption), define the interface consistent with the DecryptAndEncrypt class, that is, four interfaces for initialization, checking initialization status, encryption and decryption;

[0093] Step S2.2: Create a new Java project, create an implementation class DecryptAndEncryptImpl, implement the DecryptAndEncrypt interface, and package the project into an executable jar file package (decryptAndEncrypt-interface.jar).

[0094] The code example is as follows:

[0095]

[0096]

[0097] Step S3, upload the jar package file resource to the specified path;

[0098] Step S3.1, select the locally packaged executable interface jar (decryptAndEncrypt-interface.jar), and further select the upload path of the jar package (for example: / usr / jar / interface).

[0099] Step S3.2, select the locally packaged executable implementation jar (decryptAndEncrypt-impl.jar), and further select the upload path of the jar package (for example: / usr / jar / impl).

[0100] Step S3.3, fill in the jar package implementation class name (for example, com.inspur.dsp.security.impl.DecryptAndEncryptImpl).

[0101] Step S3.4, fill in the configuration parameters. This configuration is for the scenario that needs to be initialized. If not, leave it blank.

[0102] Step S3.5, click "Upload" on the page. After the upload is successful, you can view, edit, download, and delete jar file resources through the UI.

[0103] Step S4: Create and manage encryption and decryption service instances;

[0104] Step S4.1: Define the encryption and decryption service name, which is the encryption and decryption service name actually selected for subsequent data synchronization tasks.

[0105] Step S4.2: Select the jar package file resource uploaded in step S3.

[0106] Step S4.3: After clicking "Create" on the page, the corresponding encryption and decryption service (Controller Service instance) will be automatically created and enabled in NIFI.

[0107] Step S5, creating source and destination data sources;

[0108] Step S5.1: Configure source data source information and create a source data source.

[0109] Step S5.2: Configure the destination data source information and create a destination data source.

[0110] Step S6: Create an encryption and decryption data synchronization task, select the source and destination data sources, whether to encrypt, and the corresponding encryption and decryption services.

[0111] Step S6.1, select the source data source created in step S5, and further select the source table.

[0112] Step S6.2: Select the destination data source created in step S5, and further select a destination table.

[0113] Step S6: 3. Select "whether to encrypt". If encrypted, then proceed to the encryption and decryption service created in step S4.

[0114] Step S6.4: After clicking "Create" on the page, the encryption and decryption data synchronization task will be automatically created in NIFI. The specific process is as follows: Figure 3 shown.

[0115] Step S6.4.1, the data collection component collects data from the source data source, and then encrypts it according to the custom encryption algorithm, and then transmits the data after encryption.

[0116] Step S6.4.2: After receiving the encrypted data transmitted by the data acquisition component, the data writing component decrypts the data according to the custom decryption algorithm and writes it to the target data source.

[0117] This data synchronization process ensures that the original plaintext data cannot be directly obtained during the entire data transmission process, thereby ensuring data security.

[0118] The system and device for implementing data synchronization based on NIFI include a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method when executing the computer program.

[0119] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0120] Compared with the existing technology, the system and method for realizing data synchronization based on NIFI can dynamically load and update the encryption algorithm, improve data security, simplify the maintenance and upgrade process, and improve the flexibility and efficiency of the data synchronization system. The details are as follows:

[0121] First, it improves scalability: users can customize encryption and decryption algorithms and expand NIFI's functions by uploading JAR files, which enables the system to easily integrate new or more complex encryption algorithms and enhances the system's scalability.

[0122] Second, it enhances security: data is encrypted during transmission. Even if the data is intercepted, the attacker cannot directly obtain the original data, which greatly enhances data security.

[0123] Third, support hot loading: The hot loading feature of the jar package means that the encryption algorithm can be updated without restarting the system, which improves the availability and response speed of the system.

[0124] Fourth, resource management has been optimized: it provides functions for uploading and managing jar files, including viewing, editing, downloading and deleting, which facilitates the maintenance and updating of encryption algorithm implementation.

[0125] Fifth, convenient service management: The creation and management modules of encryption and decryption services simplify the service configuration process. Users can complete service configuration and adjustment through interface operations, which improves management efficiency.

[0126] Sixth, automated processing is achieved: the creation and management of data synchronization tasks are automated, which reduces the need for manual intervention, reduces the possibility of operational errors, and improves the efficiency and accuracy of data synchronization.

[0127] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A system for realizing data synchronization based on NIFI, characterized in that: include; NIFI controller service extension module, used to customize the extension of the controller service ControllerService for encryption and decryption; The encryption and decryption custom implementation module is used to customize the encryption and decryption algorithm logic and package it into an executable jar file package; The jar file resource upload and management module provides an interface operation UI for uploading and managing executable jar file packages with custom encryption and decryption algorithms. The management methods include viewing, editing, downloading and deleting. The encryption and decryption service creation and management module is used to define the encryption and decryption service name, select the uploaded jar package file resources, create and manage the controller service instance based on the custom encryption and decryption algorithm, and the management methods include viewing, editing and deleting; The data source management module is used to create the source and destination data sources required for data synchronization tasks; The data synchronization task creation and management module is used to select the source and destination data sources, select whether to encrypt and the encryption and decryption services, and create and manage encryption and decryption data synchronization tasks in the NIFI framework.

2. The system for realizing data synchronization based on NIFI according to claim 1, characterized in that: The NIFI controller service extension module creates an interface class Security_Ser, which inherits from the controller service Controller Service of the NIFI framework and contains four interfaces for initialization, checking initialization status, encryption and decryption; Create an external interface class DecryptAndEncrypt to connect to the custom encryption and decryption method, and implement four interfaces for initialization, checking initialization status, encryption, and decryption; Create an implementation class SecurityService, implement the Security_Ser interface, and inherit from the Abstract Controller Service. The configuration parameters include the jar package path, class name, and configuration items required for initialization.

3. The system for realizing data synchronization based on NIFI according to claim 1, characterized in that: The encryption and decryption custom implementation module creates a new Java interface project, creates an interface class DecryptAndEncrypt, and defines an interface consistent with the DecryptAndEncrypt class, namely, four interfaces for initialization, checking initialization status, encryption, and decryption; Create a new Java project, create an implementation class DecryptAndEncryptImpl, implement the DecryptAndEncrypt interface, and package the project into an executable jar file.

4. A method for realizing data synchronization based on NIFI, characterized in that: The following steps are involved: Step S1, expand the NIFI controller service and create a controller service for encryption and decryption; Step S2: Customize the encryption and decryption algorithm and package it into an executable jar file package; Step S3, upload the jar package file resource to the specified path; Step S4: Create and manage encryption and decryption service instances; Step S5, creating source and destination data sources; Step S6: Create an encryption and decryption data synchronization task, select the source and destination data sources, whether to encrypt, and the corresponding encryption and decryption services.

5. The method for realizing data synchronization based on NIFI according to claim 4 is characterized in that: In the step S1, extending the NIFI controller service includes creating a Security_Ser interface class, a DecryptAndEncrypt interface class and a SecurityService implementation class, and configuring corresponding parameters; The specific steps are as follows: Step S1.1, create an interface class Security_Ser, which inherits from the controller service Controller Service of the NIFI framework and contains four interfaces for initialization, checking initialization status, encryption and decryption; Step S1.2, create an external interface class DecryptAndEncrypt, which is used to connect to the custom encryption and decryption method, and implement four interfaces for initialization, checking initialization status, encryption and decryption; Step S1.3, create an implementation class SecurityService, implement the Security_Ser interface, and inherit from the Abstract Controller Service. The configuration parameters include the jar package path, class name, and configuration items required for initialization.

6. The method for realizing data synchronization based on NIFI according to claim 4, characterized in that: In step S2, customizing the encryption and decryption algorithm includes creating a DecryptAndEncrypt interface class and a DecryptAndEncryptImpl implementation class, and packaging the project into an executable jar; The specific steps are as follows: Step S2.1, create a new Java interface project, create an interface class DecryptAndEncrypt, define the interface consistent with the DecryptAndEncrypt class, that is, four interfaces for initialization, checking initialization status, encryption and decryption; Step S2.2: Create a new Java project, create an implementation class DecryptAndEncryptImpl, implement the DecryptAndEncrypt interface, and package the project into an executable jar file package.

7. A system and device for realizing data synchronization based on NIFI, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 4 to 6 when executing the computer program.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 4 to 6 is implemented.

Images