NiFi-based hot loading custom data processing method and system

Abstract

The invention relates to the technical field of computer software, in particular to a NiFi-based hot loading custom data processing method and system, which comprises the following steps of: defining a standardized data processing interface which comprises initialization, encryption and decryption data processing methods and serves as an access point of custom data processing logic; a mechanism is provided, business developers are allowed to realize customized data processing logic according to the interface, and the customized data processing logic is packaged into a Jar file; the method and the system have the beneficial effects that the adaptability and the stability of the system are enhanced, the system can quickly cope with the change of business requirements by dynamically updating data processing logic, the stability and the reliability of the system are ensured, and the risk caused by shutdown or restart of the system is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a method and system for customizing processing data based on NiFi hot loading. Background Art

[0002] With the rapid development of big data technology, data processing technology is facing increasingly severe challenges. In the field of data flow management, the diversity and complexity of data processing are increasing, which puts higher requirements on data security, processing efficiency and real-time performance. As an open source data processing and distribution system, Apache NiFi has been widely used in the field of data flow management with its powerful functions and flexible architecture. However, although NiFi has rich functions, its default configuration often cannot meet the needs of customized data processing in some specific business scenarios, especially when the processing logic needs to be frequently adjusted or updated.

[0003] Specifically, traditional data processing systems usually need to be restarted or reconfigured to apply new data processing logic, which not only increases system downtime, but also reduces the real-time and efficiency of data processing. In addition, for complex data processing requirements, traditional data processing systems often lack sufficient flexibility and scalability, making it difficult to meet the rapid changes in business needs.

[0004] Therefore, developing a system and method that can dynamically load and update data processing logic without interrupting system operation is of great significance for improving the adaptability and processing speed of the data processing system. This is also the technical problem to be solved by this invention. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for hot loading custom processing data based on NiFi to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for hot loading custom processing data based on NiFi, the method comprising the following steps:

[0007] Define a standardized data processing interface, which includes initialization, encryption, and decryption data processing methods as an access point for custom data processing logic;

[0008] Provide a mechanism that allows business developers to implement customized data processing logic based on the interface and encapsulate it as a Jar file;

[0009] Configure parameters in NiFi to specify the storage path of the custom Jar file and the fully qualified name of the Java class for subsequent loading;

[0010] Use the dynamic class loading mechanism to load and instantiate the custom data processing class in the Jar file without interrupting the operation of NiFi;

[0011] Integrate the instantiated custom data processing class into NiFi's data processing flow so that the business process calls these custom data processing methods.

[0012] Preferably, it also includes:

[0013] Provides a user-friendly parameter configuration interface that allows users to easily set the path and class name of the custom Jar file, as well as any necessary business-related parameters.

[0014] Preferably, the data processing interface and method are designed to be extensible and configurable to support new data processing requirements or scenarios added in the future.

[0015] Preferably, it also includes:

[0016] The data processing interface is implemented in the ControllerService prefabricated service of NiFi, and the custom Jar file is dynamically loaded and instantiated through the reflection mechanism to realize the hot loading function.

[0017] Preferably, the custom data processing class is successfully loaded and instantiated, and will be seamlessly integrated into NiFi's data processing flow, allowing components in the business process to process data by calling methods implemented by the interface without restarting the NiFi system, thereby improving the flexibility and responsiveness of the system.

[0018] A system for hot loading custom processing data based on NiFi, the system comprising:

[0019] A custom ControllerService prefabricated service, which defines a standardized data processing interface, which contains at least initialization, encryption, and decryption methods as a connection point with external custom data processing logic;

[0020] A data processing logic implementation class for implementing the interface, which is written by business developers according to business requirements and packaged into a Jar file;

[0021] A dynamic class loading mechanism that loads and instantiates the data processing logic implementation class in the Jar file when NiFi is running without restarting the system.

[0022] Preferably, it also includes:

[0023] A user-friendly parameter configuration interface allows users to specify the storage path of custom Jar files and the fully qualified name of the Java class, as well as configure other business-related parameters as needed.

[0024] Preferably, the data processing interface is designed to be extensible so as to support new data processing methods without modifying the existing interface; the data processing logic implementation class follows a unified interface specification to ensure that the logic written by different developers can be seamlessly integrated into NiFi's data processing flow.

[0025] Preferably, it also includes:

[0026] When the ControllerService prefabricated service is started, the Jar file loading and class instantiation process are automatically executed, and the customized data processing logic is integrated into the data processing flow of NiFi; through the reflection mechanism, the customized data processing method is called in the processing component of the business process to realize customized data processing.

[0027] Preferably, a hot loading mechanism is adopted to allow the customized data processing logic to be updated or replaced without interrupting the normal operation of the NiFi system, thereby improving the flexibility and responsiveness of the system; the system handles various types of data processing requirements, including but not limited to data encryption and decryption, data desensitization, and meets different business needs by configuring different data processing logic implementation classes.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The method and system for hot loading custom data processing based on NiFi proposed in the present invention enhance the adaptability and stability of the system: by dynamically updating the data processing logic, the system can quickly respond to changes in business needs, while ensuring the stability and reliability of the system and reducing the risks caused by system downtime or restart; optimize the development process, by providing a consistent development framework and integrated environment, developers can more efficiently develop, test and deploy data processing logic; provide data processing capabilities that take effect immediately: through the hot loading mechanism, the custom data processing logic can take effect immediately without waiting for the system to be restarted or reconfigured; support diverse data processing scenarios: the system has high scalability and can support various complex and customized data processing requirements, including encryption, decryption, data conversion, etc.; reduce development costs and implementation complexity: developers can quickly write and expand data processing logic based on a unified development framework without building the entire system from scratch, greatly shortening the development cycle and reducing maintenance costs; significantly improve the flexibility of the data processing process: by dynamically loading and configuring custom data processing logic, the system can quickly adapt to changes in business needs without tedious restart or reconfiguration operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] For example, see Figure 1 The present invention provides a technical solution: a method for hot loading custom processing data based on NiFi, the method comprising the following steps:

[0033] Hot loading of custom data processing methods can be divided into two parts. One is to implement custom data processing logic, and the other is to hot load these methods into the NiFi environment and implement configuration in the business process.

[0034] How to implement custom data processing logic

[0035] Take the scenario of data encryption and decryption as an example. In NiFi, extend a custom ControllerService prefabricated service. Create an interface class (interface) for processing data. This interface class is the connection port with the external custom data processing logic, providing a unified and easy-to-understand development framework for business developers. Note that the interface method, class name and path of this interface class need to be exposed to business personnel. The methods in this interface class are fixed and the only standard. Once determined and already in use, they cannot be easily modified. For example, for the scenario of data encryption and decryption, the methods in the interface class can be defined as initialization, encryption, and decryption. For the scenario of data desensitization, the prescribed standard method can define a desensitization method. These methods require different designs for different businesses.

[0036] How do business developers use the relevant parameters of the interface class? First, make it clear that the development is based on JAVA, and create a JAVA interface project on this basis. In this project, create an interface class that is the same as the ControllerService prefabricated service. And the same interface method, the same class name, and the same path as its interface class. At the same time, create an implementation class in the JAVA interface project, which needs to implement all methods of the interface. Then, in the class of the implementation method, each business developer can write specific data processing logic for the business data according to the business scenario he is responsible for. For example, for the business scenario of data encryption and decryption, business developers can implement the data processing logic of data initialization, encryption, and decryption in the implementation class.

[0037] When the business personnel complete the logic of data processing, the JAVA implementation class is encapsulated as a Jar file, that is, encapsulated as a code block for hot loading in NiFi's prefabricated service.

[0038] In fact, this scenario is very simple. For example, just like our computer, when the computer is turned on and running, we may use the mouse, keyboard, etc. We cannot shut down the computer, connect the mouse or keyboard, and then turn it on. Therefore, the computer has reserved a USB interface for us. As long as the device meets the rules of this interface, the computer can use it by loading the driver and other operations. The above part explains how to make a mouse and keyboard according to the USB interface reserved by the computer. The following part explains how to connect the mouse and keyboard to the computer.

[0039] Hot load methods into the NiFi environment and implement configuration in business processes

[0040] After the business developers have customized the data processing logic, the next step is to hot load the processing logic into the NiFi operating environment. This is the solution description for the prefabricated ControllerService service to process the logic.

[0041] In this prefabricated service, the parameters must be clearly specified first. To build a user-friendly parameter configuration interface, the following must be determined: 1. Allow users to specify the storage path of custom Jar files; 2. The fully qualified name of the loaded Java class. These two parameters are required because these two parameters are needed to configure NiFi to load the Jar file code block that is customized by the business R&D personnel to process data. In addition, additional configuration parameters can be added according to business needs. For example, taking the business scenario of data encryption and decryption as an example, the encryption and decryption methods may need to be initialized before encrypting and decrypting the data, so the configuration parameters for initializing the required parameters can be added. These parameters need to be designed and implemented for different business scenarios.

[0042] Secondly, the logic of starting the prefabricated ControllerService service needs to be described. Starting means running the service so that the relevant business processes can use the logic of the service to process data.

[0043] The service startup logic requires hot loading of the Jar package. Using the class loading mechanism, the NiFi platform can dynamically load the user-specified Jar file into the NiFi runtime environment.

[0044] After loading the Jar package into the runtime environment, how should the business process use its data processing logic? We need to instantiate the method with the full class path in the configuration parameters. In the prefabricated ControllerService service, we have created an implementation class. In the above solution, we also said that this interface class is the connection port with the external custom data processing logic. We can use the reflection mechanism to instantiate the custom created data processing method, so that the custom data processing logic method can be called in the processing component in the business process.

[0045] Once the Java class is successfully loaded and instantiated, it will be immediately integrated into NiFi's data processing flow without restarting the entire system.

[0046] Now that the custom logic for processing data has been instantiated, how can we use this method in the business process? We can expose an interface and let the prefabricated ControllerService service implement this interface. This interface is exposed to the processing components of the business operation process. Still taking the encryption and decryption of data processing as an example, the methods in this interface can be defined as encryption, decryption, and initialization methods. We define the instantiation method after the class reflection mechanism as a global variable. After the prefabricated ControllerService service is started, the interface logic implemented in the prefabricated service can use this instantiation method to call the custom data processing logic method. The components in the business process can implement custom logic processing of data through this exposed interface class.

[0047] Thanks to the hot loading mechanism, these customized data processing methods can be updated or replaced without interrupting the normal operation of the NiFi system, which greatly improves the flexibility and responsiveness of the system.

[0048] Still using the computer as an example, when the mouse and keyboard are connected via USB, the hardware needs to be identified, and the power and chip need to be matched and loaded, so that the computer can connect the mouse and keyboard.

[0049] Embodiment 2, based on embodiment 1, proposes a system for custom processing data based on NiFi hot loading, the system comprising:

[0050] A custom ControllerService prefabricated service, which defines a standardized data processing interface, which contains at least initialization, encryption, and decryption methods as a connection point with external custom data processing logic;

[0051] A data processing logic implementation class for implementing the interface, which is written by business developers according to business requirements and packaged into a Jar file;

[0052] A dynamic class loading mechanism that loads and instantiates the data processing logic implementation class in the Jar file when NiFi is running without restarting the system.

[0053] Also included: A user-friendly parameter configuration interface that allows users to specify the storage path of custom Jar files and the fully qualified name of the Java class, as well as configure other business-related parameters as needed.

[0054] The data processing interface is designed to be extensible to support new data processing methods without modifying the existing interface; the data processing logic implementation class follows a unified interface specification to ensure that the logic written by different developers can be seamlessly integrated into NiFi's data processing flow.

[0055] It also includes: when the ControllerService prefabricated service is started, the Jar file loading and class instantiation process are automatically executed, and the customized data processing logic is integrated into the data processing flow of NiFi; through the reflection mechanism, the customized data processing method is called in the processing component of the business process to realize customized data processing.

[0056] The hot loading mechanism allows the customized data processing logic to be updated or replaced without interrupting the normal operation of the NiFi system, thereby improving the flexibility and response speed of the system. The system handles various types of data processing requirements, including but not limited to data encryption and decryption, data desensitization, and meets different business needs by configuring different data processing logic implementation classes.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for custom processing data based on NiFi hot loading, characterized by: The method comprises the following steps: Define a standardized data processing interface, which includes initialization, encryption, and decryption data processing methods as an access point for custom data processing logic; Provide a mechanism that allows business developers to implement customized data processing logic based on the interface and encapsulate it as a Jar file; Configure parameters in NiFi to specify the storage path of the custom Jar file and the fully qualified name of the Java class for subsequent loading; Use the dynamic class loading mechanism to load and instantiate the custom data processing class in the Jar file without interrupting the operation of NiFi; Integrate the instantiated custom data processing class into NiFi's data processing flow so that the business process calls these custom data processing methods.

2. According to claim 1, a method for hot loading custom processing data based on NiFi is characterized in that: Also includes: Provides a user-friendly parameter configuration interface that allows users to easily set the path and class name of the custom Jar file, as well as any necessary business-related parameters.

3. According to claim 2, a method for custom processing data based on NiFi hot loading is characterized in that: The data processing interfaces and methods are designed to be extensible and configurable to support new data processing requirements or scenarios added in the future.

4. According to claim 1, a method for custom processing data based on NiFi hot loading is characterized in that: Also includes: The data processing interface is implemented in the ControllerService prefabricated service of NiFi, and the custom Jar file is dynamically loaded and instantiated through the reflection mechanism to realize the hot loading function.

5. According to claim 1, a method for custom processing data based on NiFi hot loading is characterized in that: The customized data processing class is successfully loaded and instantiated, and will be seamlessly integrated into NiFi's data processing flow, allowing components in the business process to process data by calling methods implemented by the interface without restarting the NiFi system, thereby improving the flexibility and responsiveness of the system.

6. A system based on NiFi hot loading of custom processing data according to the method based on NiFi hot loading of custom processing data according to any one of claims 1 to 5, characterized in that: The system comprises: A custom ControllerService prefabricated service, which defines a standardized data processing interface, which contains at least initialization, encryption, and decryption methods as a connection point with external custom data processing logic; A data processing logic implementation class for implementing the interface, which is written by business developers according to business requirements and packaged into a Jar file; A dynamic class loading mechanism that loads and instantiates the data processing logic implementation class in the Jar file when NiFi is running without restarting the system.

7. A system for custom processing data based on NiFi hot loading according to claim 6, characterized in that: Also includes: A user-friendly parameter configuration interface allows users to specify the storage path of custom Jar files and the fully qualified name of the Java class, as well as configure other business-related parameters as needed.

8. A system for custom processing data based on NiFi hot loading according to claim 6, characterized in that: The data processing interface is designed to be extensible so as to support new data processing methods without modifying the existing interface; The data processing logic implementation class follows a unified interface specification to ensure that the logic written by different developers can be seamlessly integrated into NiFi's data processing flow.

9. A system for hot loading custom processing data based on NiFi according to claim 6, characterized in that: Also includes: When the ControllerService prefabricated service is started, it automatically executes the Jar file loading and class instantiation process, integrating the customized data processing logic into NiFi's data processing flow; Through the reflection mechanism, the custom data processing method is called in the processing component of the business process to implement custom data processing.

10. A system for custom processing data based on NiFi hot loading according to claim 6, characterized in that: The hot loading mechanism allows the customized data processing logic to be updated or replaced without interrupting the normal operation of the NiFi system, thereby improving the flexibility and response speed of the system. The system handles various types of data processing requirements, including but not limited to data encryption and decryption, data desensitization, and meets different business needs by configuring different data processing logic implementation classes.

Images