Data writing system and method based on real-time database proxy

Abstract

The invention discloses a data writing system and method based on a real-time database agent, and belongs to the technical field of network data transmission. The data writing system based on the real-time database agent comprises a receiving service module, a user management module, a data receiving module, a receiving buffer pool, a data analysis module, a forwarding buffer queue and a receiving service module, solves the problem of compatibility of existing software of a power plant, and enables domestic real-time database agent software to be popularized. Part of pressure of the main agent is relieved, the stability of a software system is improved, the running speed of the software system is increased, and meanwhile high manpower resources and financial cost are saved for a whole project.

Description

Technical Field

[0001] This invention belongs to the field of network data transmission technology, specifically relating to a data writing system and method based on a real-time database proxy. Background Technology

[0002] With the deepening implementation of the strategy for independent control of information technology, the power industry, as a critical infrastructure sector, must inevitably undertake the construction of information technology innovation (ITAI). The government has introduced a series of supportive policies to encourage and support the application of domestically produced software in power plants, providing a favorable policy environment for its development. To adapt to the complex and ever-changing application scenarios in power plants, domestic software companies are continuously increasing their R&D investment, promoting technological innovation and upgrading. By introducing advanced technologies such as cloud computing, big data, and artificial intelligence, they are improving the intelligence level of software, enhancing data processing and analysis capabilities, and providing power plants with more efficient and precise management services. Simultaneously, they are customizing solutions to meet the specific needs of power plants, satisfying their individual requirements.

[0003] Currently, domestically produced software is widely used in power plants, offering a significant price advantage over imported software. This is primarily due to reduced domestic production costs and intensified market competition, enabling power plants to save substantial funds when procuring and deploying systems. Industry data shows that the investment in domestically produced control system equipment is typically about 50% lower than that of imported equipment, significantly reducing initial investment costs for power plants. During the design and development process, domestically produced software fully considers the specific needs and environment of power plants, ensuring stable system operation and secure data transmission through multiple security mechanisms. Furthermore, domestically produced software boasts higher compatibility and maintainability, facilitating routine maintenance and upgrades by power plant technicians. Functionally, domestically produced software covers all aspects of power plant operations, including monitoring, control, optimization, and scheduling. Through integrated design, these functions achieve seamless connection and collaborative operation between various power plant systems, improving overall operational efficiency and demonstrating excellent application prospects.

[0004] The domestically produced software used in power plants mainly includes data acquisition clients, real-time database proxy software, and real-time database software. Among these, the domestically produced data acquisition clients and real-time database proxy software are used together. However, due to reasons such as some power plants using acquisition clients from other manufacturers and some manufacturers not yet upgrading their acquisition clients, some power plants are still using older database proxy software to collect a portion of the data, and the widespread adoption of domestically produced software is not yet possible. Summary of the Invention

[0005] The purpose of this invention is to provide a data writing system and method based on a real-time database agent to solve the technical problem of incompatibility between existing power plant software and domestically produced software.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] This invention discloses a data writing system based on a real-time database agent, including a receiving service module, a user management module, a data receiving module, a receiving buffer pool, a data parsing module, a forwarding cache queue, and a data forwarding module;

[0008] The receiving service module is used to process client requests and receive data packets, forming data packets;

[0009] The user management module is used to set permission data to ensure the security and stability of the system;

[0010] The data receiving module is used to unpack the data and interact with the user management module to obtain the original data packet;

[0011] The receive buffer is used to store raw data packets;

[0012] The data parsing module interacts with the user management module to parse the raw data packets in the receive buffer pool and obtain data packets;

[0013] The forwarding cache queue is used to store data packets and forward the data packets to the data forwarding module for forwarding to the database for storage.

[0014] Furthermore, the receiving service module adopts the domestically developed open-source C / C++ cross-platform network library libhv.

[0015] Furthermore, the permission data includes permission values, unpacking rules, and parsing rule numbers.

[0016] Furthermore, the user management module includes network monitoring, authentication, permission allocation, packet unpacking rule setting, and data parsing setting functions;

[0017] The network monitoring function is responsible for monitoring the outflow and inflow of data in the system to ensure system security.

[0018] The identity verification function is responsible for providing data on the permissions of registered users and obtaining permission values ​​to ensure the security of the system.

[0019] The permission allocation function is responsible for allocating user permissions.

[0020] The unpacking rule setting function is used to set the unpacking rule number;

[0021] The data parsing settings function is responsible for configuring the data stream format for user interaction, setting parsing rule numbers, and ensuring system stability.

[0022] Furthermore, the user permission functions include writing historical data, writing real-time data, and writing cached data.

[0023] The present invention also discloses a data writing method using the above system, comprising the following steps:

[0024] S1: The data receiving module receives data packets from the receiving service module, performs data unpacking, and interacts with the user management module to obtain the original data packets;

[0025] S2: The raw data packet is placed into the receiving buffer pool, and then transferred to the data parsing module for parsing to obtain the raw data. The raw data is then assembled into packets to obtain data packets.

[0026] S3: Store the obtained data packets in the forwarding buffer queue and forward them to the receiving service module for storage.

[0027] Furthermore, S1 specifically includes the following steps:

[0028] Step 1: The data receiving module obtains the received data packet from the receiving service module and executes Step 2;

[0029] Step 2: Interact with the user management module to obtain the unpacking rule number and permission value. If successful, proceed to step 3; otherwise, proceed to step 1.

[0030] Step 3: Based on the unpacking rule number, retrieve the unpacking rule from the unpacking rule library in the data receiving module to unpack the data and obtain the original data packet. If the unpacking process is successful, proceed to step 4; otherwise, proceed to step 1.

[0031] Step 4: Obtain the permission rules based on the permission value, and determine whether the parsed original data packet conforms to the permission rules. If yes, proceed to step 5; otherwise, proceed to step 1.

[0032] Step 5: Place the raw data packets that meet the permission rules into the receive buffer pool and execute Step 1.

[0033] Furthermore, the unpacking rule base in the data receiving module includes rules for fixed length, delimiter, and header length.

[0034] Furthermore, S2 specifically includes the following steps:

[0035] Step 1: Obtain a raw data packet from the receive buffer pool and proceed to Step 2;

[0036] Step 2: Interact with the user management module to obtain the original data packet parsing rule number, and determine whether to execute step 3 successfully or step 1 if it fails;

[0037] Step 3: Select the corresponding parsing rule from the parsing rule library in the data parsing module according to the original data packet parsing rule number to parse the original data packet and obtain the parsed data. If the parsing is successful, proceed to step 4; otherwise, proceed to step 1.

[0038] Step 4: Verify whether the parsed data is correct and complete. If yes, obtain the data package and proceed to Step 5; otherwise, proceed to Step 1.

[0039] Step 5: Store the data packet in the forwarding buffer queue and execute step 1.

[0040] Furthermore, S3 specifically includes the following steps:

[0041] Step 1: Retrieve the packet group in the forwarding buffer queue. If successful, proceed to Step 2; otherwise, proceed to Step 1.

[0042] Step 2: Traverse the data, filter out bad data, and classify it according to the database in the receiving service module, then proceed to Step 3;

[0043] Step 3: Forward the categorized data to different database interfaces in the receiving service module for storage, and execute Step 1.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This invention discloses a data writing system based on a real-time database agent. The system encapsulates data packet parsing rules, supports a variety of common packet parsing rules as well as custom rules, and loads them into the system through a configuration dynamic library method. This solves the compatibility problem of existing software in power plants and promotes the popularization of domestic real-time database agent software.

[0046] Furthermore, by setting up a cache, the system separates the processing module and the network receiving and sending module, which alleviates some of the pressure on the main agent, improves the stability of the software system, speeds up the operation of the software system, and also saves a lot of human resources and financial costs for the entire project.

[0047] Furthermore, by designing independent receiving service and data receiving modules, the system can flexibly handle data packets of different formats and sources, including data from existing power plant software and domestically produced software, improving the system's compatibility and scalability. The introduction of a receiving buffer pool and a forwarding buffer queue makes the data processing and forwarding process more efficient. The system can store the received data first, and then parse and forward it step by step, avoiding data processing bottlenecks and improving overall data processing efficiency. The user management module not only ensures the system's security, but also ensures stable data transmission and processing through access control, protecting production safety. Attached Figure Description

[0048] Figure 1 This is a framework diagram of the data writing system based on a real-time database agent according to the present invention. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0051] The first aspect of the present invention discloses a data writing system based on a real-time database agent, including a receiving service module, a user management module, a data receiving module, a receiving buffer pool, a data parsing module, a forwarding cache queue, and a receiving service module;

[0052] The receiving service module is responsible for handling a large number of high-concurrency client requests and data packet reception. It adopts the domestic open-source C / C++ cross-platform network library libhv. The underlying library is 60% developed in C language and 30% encapsulated in C++. It can choose IO multiplexing models such as epoll, poll, select, and IOCP combined with multi-threading mode to handle requests from different clients and improve service response speed.

[0053] The user management module includes network monitoring, authentication, permission allocation, packet splitting rule settings, and data parsing settings to ensure system security and stability.

[0054] The network monitoring function in the aforementioned user management module is responsible for monitoring the outflow and inflow of data to ensure data security.

[0055] The identity verification function in the aforementioned user management module is responsible for providing corresponding data functions for the permissions of registered users and ensuring data security;

[0056] The permission allocation function in the aforementioned user management module is responsible for allocating user permissions. The user permission functions include writing historical data, writing real-time data, and writing cached data.

[0057] The unpacking rule setting function in the aforementioned user management module is used to set the unpacking rule number;

[0058] The data parsing settings function in the aforementioned user management module is responsible for configuring the data stream format for user interaction, ensuring data format consistency, and enabling normal data interaction.

[0059] The aforementioned data receiving module is responsible for unpacking the data and interacting with the user management module to obtain the original data packets.

[0060] The aforementioned receive buffer is used to store raw data packets.

[0061] The aforementioned data parsing module interacts with the user management module to parse the raw data packets in the receive buffer pool and obtain data packets.

[0062] The aforementioned forwarding cache queue is used to store data packets and forward them to the receiving service module for storage.

[0063] The aforementioned data writing system based on a real-time database agent provides effective technical support for solving the incompatibility problem between existing power plant software and domestically produced software by enhancing compatibility, improving processing efficiency, ensuring data security, promoting the application of domestically produced software, and optimizing resource utilization.

[0064] A second aspect of the present invention discloses a method for writing data based on a real-time database agent, using the system described above:

[0065] S1: The data receiving module receives data packets from the receiving service module, performs data unpacking, and interacts with the user management module to obtain the original data packets;

[0066] S2: The raw data packet is placed into the receiving buffer pool, and then transferred to the data parsing module for parsing to obtain the raw data. The raw data is then assembled into packets to obtain data packets.

[0067] S3: Store the obtained data packets in the forwarding buffer queue and forward them to the receiving service module for storage.

[0068] The present invention is described in further detail below with reference to the accompanying drawings:

[0069] like Figure 1 As shown, the present invention discloses a data writing method for a data writing system based on a real-time database agent, which specifically includes the following steps:

[0070] 1) The data receiving module is responsible for unpacking the received data packets and storing the unpacked data packets in the receive buffer queue:

[0071] Step 1: The data receiving module obtains the received data packet from the receiving service module and executes Step 2;

[0072] Step 2: Interact with the user management module to obtain the unpacking rule number and permission value. If successful, proceed to step 3; otherwise, proceed to step 1.

[0073] Step 3: Based on the unpacking rule number, retrieve the unpacking rule from the unpacking rule library in the data receiving module to unpack the data and obtain the original data packet. If the unpacking process is successful, proceed to step 4; otherwise, proceed to step 1.

[0074] Step 4: Obtain the permission rules based on the permission value, and determine whether the parsed original data packet conforms to the permission rules. If yes, proceed to step 5; otherwise, proceed to step 1.

[0075] Step 5: Place the raw data packets that meet the permission rules into the receive buffer pool and execute Step 1;

[0076] 2) The data parsing module parses the raw data packets in the receive buffer queue to obtain the raw data, then assembles the raw data into packets and stores them in the forwarding buffer queue:

[0077] Step 1: Obtain a raw data packet from the receive buffer pool and proceed to Step 2;

[0078] Step 2: Interact with the user management module to obtain the original data packet parsing rule number, and determine whether to execute step 3 successfully or step 1 if it fails;

[0079] Step 3: Select the corresponding parsing rule from the parsing rule library in the data parsing module according to the original data packet parsing rule number to parse the original data packet and obtain the parsed data. If the parsing is successful, proceed to step 4; otherwise, proceed to step 1.

[0080] Step 4: Verify whether the parsed data is correct and complete. If yes, obtain the data package and proceed to Step 5; otherwise, proceed to Step 1.

[0081] Step 5: Store the data packet in the forwarding buffer queue and execute step 1.

[0082] 3) The data forwarding module is responsible for forwarding data packets in the forwarding buffer queue to the real-time database for storage:

[0083] Step 1: Retrieve the packet group in the forwarding buffer queue. If successful, proceed to Step 2; otherwise, proceed to Step 1.

[0084] Step 2: Traverse the data, filter out bad data, and classify it according to the database in the receiving service module, then proceed to Step 3;

[0085] Step 3: Forward the categorized data to different database interfaces in the receiving service module for storage, and execute Step 1.

[0086] The data receiving module includes a packet splitting rule library. This submodule integrates common data packet splitting rules and supports interfaces for fixed packet length, delimiters, header length fields, and custom splitting rules. After calling the library's interface to set the splitting rules, the system internally handles packet merging and splitting according to the rules, ensuring that the callback receives a complete data packet. This saves time in handling packet merging and splitting, and is compatible with most data packet formats, solving the problem of being unable to obtain data due to different data packet formats.

[0087] The data parsing module includes a parsing rule library. This sub-module integrates common data exchange formats and supports interfaces for XML, Protocol Buffers, JSON, and custom data exchange formats. After calling the library's interface to set the data format type, the system will internally select the corresponding rule to parse the data packet according to the data format and parse it into raw data, thus solving the problem of being unable to parse due to different data formats.

[0088] The I / O multiplexing module uses the libhv open-source library, which employs IOCP on Windows and epoll on Linux. By using I / O multiplexing, it avoids creating a thread or process for each connection, thereby reducing system overhead and improving resource utilization.

[0089] The system comprises a receive buffer pool and a forwarding buffer queue, with the former implemented using a circular buffer (circular queue). This design offers several advantages: it avoids data movement within memory, improving data processing efficiency; the forwarding buffer queue allows for temporary storage and reordering, ensuring data is transmitted in the correct order; through asynchronous processing and the use of the buffer queue, the system can handle higher concurrency and larger data throughput, improving overall performance; managing the length of the forwarding buffer queue enables flow control, preventing the sender from sending data too quickly and exceeding the receiver's processing capacity; and the forwarding buffer queue temporarily stores data packets that need to be forwarded, allowing the sender and receiver to operate at different speeds, thus improving data forwarding efficiency.

[0090] The system disclosed in this invention, through the design of independent receiving service modules and data receiving modules, can flexibly handle data packets of different formats and sources, including data from existing power plant software and domestically produced software, improving the system's compatibility and scalability. The introduction of a receiving buffer pool and a forwarding buffer queue makes the data processing and forwarding process more efficient. The system can first store the received data and then parse and forward it step by step, avoiding data processing bottlenecks and improving overall data processing efficiency. The user management module not only ensures system security but also ensures stable data transmission and processing through access control. This is particularly important for power plants, as data security and stability are directly related to the safety of production operations. This technical solution provides a compatible and coexisting data writing system, enabling power plants to maintain the continuity and stability of data processing during the gradual replacement of domestically produced software, thereby reducing the risks and costs of domestic replacement and facilitating the promotion and application of domestically produced software. The modular design of the system makes the functions of each module clear, facilitating maintenance and upgrades. In power plant operation, this helps optimize resource utilization and reduce long-term maintenance costs, as the modular design makes troubleshooting and system upgrades more convenient. By enhancing compatibility, improving processing efficiency, ensuring data security, promoting the application of domestic software, and optimizing resource utilization, it provides effective technical support for solving the incompatibility problem between existing power plant software and domestic software.

[0091] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A data writing system based on a real-time database agent, characterized in that, It includes a receiving service module, a user management module, a data receiving module, a receiving buffer pool, a data parsing module, a forwarding buffer queue, and a data forwarding module; The receiving service module is used to process client requests and receive data packets, forming data packets; The user management module is used to set permission data to ensure the security and stability of the system; The data receiving module is used to unpack the data and interact with the user management module to obtain the original data packet; The receive buffer is used to store raw data packets; The data parsing module interacts with the user management module to parse the raw data packets in the receive buffer pool and obtain data packets; The forwarding cache queue is used to store data packets and forward the data packets to the data forwarding module for forwarding to the database for storage.

2. The data writing system based on a real-time database agent according to claim 1, characterized in that, The receiving service module uses the domestically developed open-source C / C++ cross-platform network library libhv.

3. The data writing system based on a real-time database agent according to claim 1, characterized in that, The permission data includes permission values, unpacking rules, and parsing rule numbers.

4. A data writing system based on a real-time database agent according to claim 3, characterized in that, The user management module includes network monitoring, authentication, permission allocation, packet unpacking rule setting, and data parsing setting functions. The network monitoring function is responsible for monitoring the outflow and inflow of data in the system to ensure system security. The identity verification function is responsible for providing data on the permissions of registered users and obtaining permission values ​​to ensure the security of the system. The permission allocation function is responsible for allocating user permissions. The unpacking rule setting function is used to set the unpacking rule number; The data parsing settings function is responsible for configuring the data stream format for user interaction, setting parsing rule numbers, and ensuring system stability.

5. A data writing system based on a real-time database agent according to claim 4, characterized in that, The user permission functions include writing historical data, writing real-time data, and writing cached data.

6. A data writing method using a data writing system based on a real-time database agent as described in any one of claims 1 to 5, characterized in that, The following steps are involved: S1: The data receiving module receives data packets from the receiving service module, performs data unpacking, and interacts with the user management module to obtain the original data packets; S2: The raw data packet is placed into the receiving buffer pool, and then transferred to the data parsing module for parsing to obtain the raw data. The raw data is then assembled into packets to obtain data packets. S3: Store the obtained data packets in the forwarding buffer queue and forward them to the receiving service module for storage.

7. The data writing method of a data writing system based on a real-time database agent according to claim 6, characterized in that, S1 specifically includes the following steps: Step 1: The data receiving module obtains the received data packet from the receiving service module and executes Step 2; Step 2: Interact with the user management module to obtain the unpacking rule number and permission value. If successful, proceed to step 3; otherwise, proceed to step 1. Step 3: Based on the unpacking rule number, retrieve the unpacking rule from the unpacking rule library in the data receiving module to unpack the data and obtain the original data packet. If the unpacking process is successful, proceed to step 4; otherwise, proceed to step 1. Step 4: Obtain the permission rules based on the permission value, and determine whether the parsed original data packet conforms to the permission rules. If yes, proceed to step 5; otherwise, proceed to step 1. Step 5: Place the raw data packets that meet the permission rules into the receive buffer pool and execute Step 1.

8. The data writing method of a data writing system based on a real-time database agent according to claim 7, characterized in that, The data receiving module's unpacking rule base includes rules for fixed length, delimiters, and header length.

9. The data writing method of a data writing system based on a real-time database agent according to claim 7, characterized in that, S2 specifically includes the following steps: Step 1: Obtain a raw data packet from the receive buffer pool and proceed to Step 2; Step 2: Interact with the user management module to obtain the original data packet parsing rule number, and determine whether to execute step 3 successfully or step 1 if it fails; Step 3: Select the corresponding parsing rule from the parsing rule library in the data parsing module according to the original data packet parsing rule number to parse the original data packet and obtain the parsed data. If the parsing is successful, proceed to step 4; otherwise, proceed to step 1. Step 4: Verify whether the parsed data is correct and complete. If yes, obtain the data package and proceed to Step 5; otherwise, proceed to Step 1. Step 5: Store the data packet in the forwarding buffer queue and execute Step 1.

10. The data writing method of a data writing system based on a real-time database agent according to claim 9, characterized in that, S3 specifically includes the following steps: Step 1: Retrieve the packet group in the forwarding buffer queue. If successful, proceed to Step 2; otherwise, proceed to Step 1. Step 2: Traverse the data, filter out bad data, and classify it according to the database in the receiving service module, then proceed to Step 3; Step 3: Forward the categorized data to different database interfaces in the receiving service module for storage, and execute Step 1.

Images