Nuclear power production data processing system and method based on cloud native technology

Abstract

The invention belongs to the technical field of nuclear power, and particularly relates to a nuclear power production data processing system and method based on a cloud native technology. According to the system disclosed by the invention, related principles of the ETL-NIFI framework are deployed on the basis of Kubernetes and Docker containerization virtualization technologies, and installation and deployment are simplified. And convenient NIFI installation, deployment and maintenance are provided, and powerful, safe, reliable and data migration functions are provided through the NIFI. Therefore, the manpower consumption of different data migration, backup and other requirements is reduced, and the accuracy and real-time performance of the data are ensured. Data migration can be realized through a user-defined processor according to different requirements, and high expansion is supported.

Description

Technical Field

[0001] This invention belongs to the field of nuclear power technology, specifically relating to a nuclear power production data processing system and method based on cloud-native technology. Background Technology

[0002] The informatization and digitalization of the nuclear power industry relies on software services. As the basic services of the nuclear power industry expand, the scale of related services continues to grow, and related data continues to increase, involving the need for processing and using large amounts of data.

[0003] Because complex nuclear power operating systems use various types of data storage, data export, import, and migration involve operations such as data formatting, handling junk data, and transferring data between different storage tools. Traditional data migration methods consume a significant amount of manpower and time, and can negatively impact data accuracy and timeliness. Summary of the Invention

[0004] To overcome the problems existing in related technologies, a nuclear power production data processing system and method based on cloud-native technology is provided.

[0005] According to one aspect of the embodiments of this disclosure, a nuclear power production data processing system based on cloud-native technology is provided, comprising: A containerized NiFi application instance running in a Kubernetes cluster, which deploys the Apache NiFi core service; the core service runs in a Java Virtual Machine and includes: The flow controller is used to allocate execution threads and schedule data processing resources for the NiFi processor. A streaming file repository is used to store the current state and metadata of streaming files in a pluggable, persistent write-ahead log manner. Content repositories are used to store the actual content bytes of streaming files on multiple configurable physical storage paths in a pluggable mechanism. A data traceability repository is used to store, index, and query event data for all data processing processes in a pluggable mechanism. A web server is used to host the HTTP-based NiFi command, control, and graphical data flow design API, providing a visual interface to support the design and modification of data flows at runtime. The system is deployed through a PaaS management platform or a Kubernetes resource configuration file, which defines the computing and storage resources required by the NiFi containerized application instance and the service exposure method.

[0006] In one possible implementation, the NiFi core service supports functional extensions through custom processors; these custom processors are developed by implementing specific Java interfaces or inheriting abstract classes, and packaged as extension packages in NiFi archive NAR format, which are then loaded and activated through the system's extension mechanism.

[0007] In one possible implementation, the system further includes: The data delivery module is used to deliver data through the persistent write-ahead log of the streaming file repository and the copy-on-write technology of the content repository. The data backpressure module is used to trigger data backpressure when the data processing queue reaches a preset capacity or time threshold in order to control the data inflow rate. The data tracing module is used to record and index the event logs of each operation in the data stream, forming a data processing chain; The priority queue module is used to set up one or more priority queues and retrieve data from each queue; each queue adopts any one of the following processing strategies according to the settings: first-in-first-out, last-in-first-out, and maximum first-out.

[0008] The QoS guarantee module is used to configure QoS guarantees for specified data streams.

[0009] In one possible implementation, the system further includes a monitoring and alarm module, which monitors the running status of the NiFi containerized application instance, and the monitoring metrics include at least: The usage of JVM heap memory and non-heap memory for a node; Storage space utilization of streaming file repository, content repository and data traceability repository; The processor's average processing time and the number of system daemon threads.

[0010] In one possible implementation, the system supports a process template function, allowing users to save the constructed data flow processing process as a template and publish, share, and reuse it through the visual interface.

[0011] According to another aspect of the present disclosure, a nuclear power production data processing method based on cloud-native technology is provided, the method being implemented based on the system described in any one of claims 1 to 5, the method comprising: Step 100, Resource Configuration: Define the deployment parameters of the NiFi containerized application instance through the PaaS management platform or by writing a Kubernetes resource configuration file, including the container image source, CPU and memory resource requirements, persistent storage volume declaration, and service port; Step 101, Cluster Deployment: Enable cluster mode in the configuration, set the cluster node address and protocol port, and configure the ZooKeeper connection string to achieve node status management and coordination; Step 102, Application Deployment: Submit the resource configuration file via the Kubernetes API to create and run the NiFi containerized application instance in the Kubernetes cluster; Step 103, Process Orchestration: Design and orchestrate the data flow processing diagram consisting of multiple processors and connections using the web visualization interface provided by the NiFi instance in a drag-and-drop manner; Step 104, Operation and Maintenance Management: Utilize Kubernetes' container orchestration capabilities to perform horizontal scaling, rolling updates, fault self-healing, and dynamic resource adjustment for running NiFi instances.

[0012] In one possible implementation, the method further includes: Step 200: Encapsulate the data to be processed into a stream file FlowFile, wherein the stream file contains an attribute set and a content reference pointer pointing to the actual content data storage location; Step 201: When the processor needs to modify the content of the stream file, the system creates a copy of the original data in the content repository, performs the modification operation on the copy, and retains the original data. Step 202: When the above modification operation occurs, the system generates and stores a snapshot of the stream file and its processing context in the data traceability repository; Step 203: After processing, the data is written to the target data storage system through the output processor.

[0013] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the above-described method.

[0014] The beneficial effects of this disclosure are as follows: The system proposed here presents the principles and simplifies the installation and deployment of the ETL-NIFI framework based on Kubernetes and Docker containerized virtualization technologies. It provides convenient NIFI installation, deployment, and maintenance, and offers powerful, secure, reliable, and data migration capabilities through NIFI. This reduces the manpower required for different data migration and backup needs, and ensures data accuracy and real-time performance. Data migration can be implemented through custom processors to meet different needs, supporting high scalability.

[0015] Compared to creating NiFi using virtual machines or manually running scripts with Docker, this disclosed system allows users to create NiFi through a platform that manages Kubernetes clusters. This makes it easier to scale and monitor NiFi, reduces the technical requirements for users, lowers maintenance costs, shortens installation and deployment time, and improves user efficiency. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a nuclear power production data processing system based on cloud-native technology, as shown in an embodiment of this disclosure. Detailed Implementation

[0017] The present disclosure will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] Unless otherwise defined, the technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains; the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure; the term "comprising" and any variations thereof in this disclosure are intended to cover non-exclusive inclusion. Clearly, the embodiments described in this disclosure are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0019] In this disclosure, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this disclosure. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] Figure 1 This is a schematic diagram of a nuclear power production data processing system based on cloud-native technology, as shown in an embodiment of this disclosure. Figure 1 As shown, the system includes: A containerized NiFi application instance running in a Kubernetes cluster, which deploys the Apache NiFi core service; the core service runs in a Java Virtual Machine and includes: The Flow Controller, as the control module for NiFi to perform specific operations, is used to allocate execution threads and schedule data processing resources for the NiFi Processor.

[0021] The FlowFile Repository is used to store the current state and metadata of FlowFiles in a pluggable, persistent write-ahead log manner, thereby preserving the active state of FlowFiles in the currently active stream.

[0022] A content repository is used to store the actual byte content of a streaming file across multiple configurable physical storage paths in a pluggable mechanism; this preserves the actual byte content of the streaming file in the currently active stream. Multiple configurable storage paths can combine different physical paths, thus avoiding reaching the storage limit of a single physical partition.

[0023] The Provenance Repository is used to store, index, and query event data for all data processing procedures in a pluggable mechanism; thus, all tracking event data can be saved.

[0024] A web server is used to host the HTTP-based NiFi command, control, and graphical data flow design API, providing a visual interface to support the design and modification of data flows at runtime.

[0025] The system is deployed through a PaaS management platform or a Kubernetes resource configuration file, which defines the computing and storage resources required by the NiFi containerized application instance and the service exposure method.

[0026] The NiFi core service supports functional extensions through custom processors. These custom processors are developed by implementing specific Java interfaces or inheriting abstract classes, and packaged as extension packages in NiFi archive NAR format, which are then loaded and activated through the system's extension mechanism.

[0027] In one possible implementation, the system further includes: The data delivery module is used to deliver data through the persistent write-ahead log of the streaming file repository and the copy-on-write technology of the content repository. The data backpressure module is used to trigger data backpressure when the data processing queue reaches a preset capacity or time threshold in order to control the data inflow rate. The priority queue module is used to set up one or more priority queues and retrieve data from each queue. Each queue adopts a processing strategy of first-in-first-out (FIFO), last-in-first-out (LIFO), or maximum-first-out (MPFO) according to the settings.

[0028] The QoS guarantee module is used to configure QoS guarantees for specified data streams. This ensures that critical data that must not be lost and requires low-latency processing can be effectively processed.

[0029] The data traceability module is used to record and index the event logs of each operation in the data stream, forming a complete and traceable data processing chain.

[0030] In one possible implementation, the system further includes a monitoring and alarm module, which monitors the running status of the NiFi containerized application instance, and the monitoring metrics include at least: The usage of JVM heap memory and non-heap memory for a node; Storage space utilization of streaming file repository, content repository and data traceability repository; The processor's average processing time and the number of system daemon threads.

[0031] In one possible implementation, the system supports a process template function, allowing users to save the constructed data flow processing process as a template and publish, share, and reuse it through the visual interface.

[0032] According to another aspect of the embodiments of this disclosure, a nuclear power production data processing method based on cloud-native technology is provided. The method is implemented based on the system described above and includes: Step 100, Resource Configuration: Define the deployment parameters of the NiFi containerized application instance through the PaaS management platform or by writing a Kubernetes resource configuration file, including the container image source, CPU and memory resource requirements, persistent storage volume declaration, and service port; Step 101, Cluster Deployment: Enable cluster mode in the configuration, set the cluster node address and protocol port, and configure the ZooKeeper connection string to achieve node status management and coordination; Step 102, Application Deployment: Submit the resource configuration file via the Kubernetes API to create and run the NiFi containerized application instance in the Kubernetes cluster; Step 103, Process Orchestration: Design and orchestrate the data flow processing diagram consisting of multiple processors and connections using the web visualization interface provided by the NiFi instance in a drag-and-drop manner; Step 104, Operation and Maintenance Management: Utilize Kubernetes' container orchestration capabilities to perform horizontal scaling, rolling updates, fault self-healing, and dynamic resource adjustment for running NiFi instances.

[0033] In one possible implementation, the method further includes: Step 200: Encapsulate the data to be processed into a stream file FlowFile, wherein the stream file contains an attribute set and a content reference pointer pointing to the actual content data storage location; Step 201: When the processor needs to modify the content of the stream file, the system creates a copy of the original data in the content repository, performs the modification operation on the copy, and retains the original data. Step 202: When the above modification operation occurs, the system generates and stores a snapshot of the stream file and its processing context in the data traceability repository; Step 203: After processing, the data is written to the target data storage system through the output processor.

[0034] This disclosure presents a system based on Kubernetes and Docker containerized virtualization technologies to deploy the ETL-NIFI framework, outlining its principles and simplifying installation and deployment. It provides convenient NIFI installation, deployment, and maintenance, offering powerful, secure, reliable, and data migration capabilities. This reduces manpower consumption for various data migration and backup needs, ensuring data accuracy and real-time performance. Data migration can be implemented through custom processors to meet different requirements, supporting high scalability.

[0035] Compared to creating NiFi using virtual machines or manually running scripts with Docker, this disclosed system allows users to create NiFi through a platform that manages Kubernetes clusters. This makes it easier to scale and monitor NiFi, reduces the technical requirements for users, lowers maintenance costs, shortens installation and deployment time, and improves user efficiency.

[0036] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0037] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0038] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0039] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0040] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0041] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0042] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0043] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0044] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A nuclear power production data processing system based on cloud-native technology, characterized in that, include: A containerized NiFi application instance running in a Kubernetes cluster, which deploys the Apache NiFi core service; The core service runs in a Java Virtual Machine and includes: The flow controller is used to allocate execution threads and schedule data processing resources for the NiFi processor. A streaming file repository is used to store the current state and metadata of streaming files in a pluggable, persistent write-ahead log manner. Content repositories are used to store the actual content bytes of streaming files on multiple configurable physical storage paths in a pluggable mechanism. A data traceability repository is used to store, index, and query event data for all data processing processes in a pluggable mechanism. A web server is used to host the HTTP-based NiFi command, control, and graphical data flow design API, providing a visual interface to support the design and modification of data flows at runtime. The system is deployed through a PaaS management platform or a Kubernetes resource configuration file, which defines the computing and storage resources required by the NiFi containerized application instance and the service exposure method.

2. The system according to claim 1, characterized in that, The NiFi core service supports functional extensions through custom processors. These custom processors are developed by implementing specific Java interfaces or inheriting abstract classes, and packaged as extension packages in NiFi archive NAR format, which are then loaded and activated through the system's extension mechanism.

3. The system according to claim 1, characterized in that, The system also includes: The data delivery module is used to deliver data through the persistent write-ahead log of the streaming file repository and the copy-on-write technology of the content repository. The data backpressure module is used to trigger data backpressure when the data processing queue reaches a preset capacity or time threshold in order to control the data inflow rate. The data tracing module is used to record and index the event logs of each operation in the data stream, forming a data processing chain; The priority queue module is used to set up one or more priority queues and retrieve data from each queue; each queue adopts any one of the following processing strategies according to the settings: first-in-first-out, last-in-first-out, and maximum-first-out. The QoS guarantee module is used to configure QoS guarantees for specified data streams.

4. The system according to claim 1, characterized in that, The system also includes a monitoring and alarm module, which is used to monitor the running status of the NiFi containerized application instance. The monitoring metrics include at least: The usage of JVM heap memory and non-heap memory for a node; Storage space utilization of streaming file repository, content repository and data traceability repository; The processor's average processing time and the number of system daemon threads.

5. The system according to claim 1, characterized in that, The system supports a process template function, allowing users to save the constructed data flow processing process as a template and publish, share, and reuse it through the visual interface.

6. A nuclear power production data processing method based on cloud-native technology, characterized in that, The method is implemented based on the system according to any one of claims 1 to 5, and the method includes: Step 100, Resource Configuration: Define the deployment parameters of the NiFi containerized application instance through the PaaS management platform or by writing a Kubernetes resource configuration file, including the container image source, CPU and memory resource requirements, persistent storage volume declaration, and service port; Step 101, Cluster Deployment: Enable cluster mode in the configuration, set the cluster node address and protocol port, and configure the ZooKeeper connection string to achieve node status management and coordination; Step 102, Application Deployment: Submit the resource configuration file via the Kubernetes API to create and run the NiFi containerized application instance in the Kubernetes cluster; Step 103, Process Orchestration: Design and orchestrate the data flow processing diagram consisting of multiple processors and connections using the web visualization interface provided by the NiFi instance in a drag-and-drop manner; Step 104, Operation and Maintenance Management: Utilize Kubernetes' container orchestration capabilities to perform horizontal scaling, rolling updates, fault self-healing, and dynamic resource adjustment for running NiFi instances.

7. The method according to claim 1, characterized in that, The method further includes: Step 200: Encapsulate the data to be processed into a stream file FlowFile, wherein the stream file contains an attribute set and a content reference pointer pointing to the actual content data storage location; Step 201: When the processor needs to modify the content of the stream file, the system creates a copy of the original data in the content repository, performs the modification operation on the copy, and retains the original data. Step 202: When the above modification operation occurs, the system generates and stores a snapshot of the stream file and its processing context in the data traceability repository; Step 203: After processing, the data is written to the target data storage system through the output processor.

8. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method of claim 6 or 7.

Images