Monitoring system real-time data service middleware construction method and system

Abstract

Aiming at a business scene of large-scale data monitoring of a power system, the invention provides a method and a system for constructing real-time data service middleware of a monitoring system, and the system comprises a parameter management module, a thread pool module, a network communication module, a data analysis module, a data caching module and a garbage collection module. According to the method, through multi-thread and asynchronous subscription operation, the data reading efficiency of the monitoring system clients in a business scene of large-scale data monitoring is improved, the data consistency of the monitoring system clients is ensured, and the resource consumption of a server is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of power system automation, and in particular relates to a method and system for constructing real-time data service middleware of a monitoring system. Background Art

[0002] With the advancement of the construction of new power systems, a large number of new energy and energy storage power stations have been connected to the grid. At the same time, the number of intelligent sensors in power equipment is increasing, which makes the data scale of power monitoring and dispatching automation systems reach millions or even tens of millions. As an important component for dispatching operators to monitor and control the operating status of power system equipment, the real-time and synchronization of data refresh of the human-machine workstation is particularly important. Similarly, monitoring tools such as real-time data monitoring and alarms also need to periodically obtain the latest data from the real-time database.

[0003] In the existing technology, the client periodically polls the data server through the service bus / RPC call and other methods to obtain real-time data. The problem with this method is that when the amount of data accessed by the monitoring system is large, on the one hand, the CPU resources occupied by the data server are high, the efficiency of data transmission and processing is low, and the data synchronization is poor. On the other hand, this method of synchronous data acquisition such as periodic polling will block the operation response of the human-computer interface and affect the human-computer user experience.

[0004] Therefore, for business scenarios of large-scale data monitoring, how to meet the needs of data processing and refreshing, improve data reading efficiency, and reduce server resource consumption is still a technical problem that needs to be solved urgently. Summary of the invention

[0005] Purpose of the invention: The present invention provides a method for constructing real-time data service middleware for a monitoring system, in order to solve the problems of low data processing efficiency and high resource occupation in the prior art when the amount of data accessed by the monitoring system is large.

[0006] Technical solution: A method for constructing a real-time data service middleware for a monitoring system, comprising the following steps:

[0007] S1. Establish and initialize global parameters;

[0008] S2. Create a thread pool, which includes at least one management thread and multiple worker threads;

[0009] S3. Find an idle thread from the thread pool created by S2 as a network communication thread, connect to the monitoring system client, and create a listening port in the network communication thread to communicate with the monitoring system client;

[0010] S4. According to the monitoring system client request received by S3, the subscription information is processed and the response data is parsed;

[0011] S5. Find an idle thread from the thread pool created by S2 as a data cache thread, store the subscription information and response data obtained by S4, and send the response data to the monitoring system client through the network communication thread of S3;

[0012] S6. Search for an idle thread from the thread pool created in S2 as the garbage data collection thread.

[0013] Furthermore, the global parameters of S1 include a registered service name, a network service port number, a thread pool size, a data cache period, a data push period, and a number of monitoring system client connections.

[0014] Furthermore, the data cache periodically updates and obtains the latest data from the local real-time database, and if the data changes, the update time of the keyword data is updated.

[0015] Furthermore, the S3 network communication thread realizes the communication between the middleware and the monitoring system client and includes the following steps:

[0016] Step a1, create a new monitoring system client connection, find an idle thread from the thread pool and add the monitoring system client connection task to the idle thread, and then go to step a2;

[0017] Step a2, according to the data push cycle set by the global parameters, cyclically read data from the monitoring system client connection, and enter step a3; if the data is empty, enter step a10;

[0018] Step a3, parsing the data obtained from the monitoring system client connection to obtain the subscription type. If the subscription type is modified, proceed to step a4; if the subscription type is a new type, proceed to step a5;

[0019] Step a4, determine the modification type. If it is a pause type, set the subscription topic pause sending flag and go to step a10; if it is a close type, go to step a17;

[0020] Step a5, parse the request packet to obtain subscription information, including subscription sequence number, subscription length, application, state, subscription subject information, subscription keyword information, and enter step a6;

[0021] Step a6, judging whether the current subscription topic exists according to the application, state and topic name in the subscription information, if it exists, proceed to step a9; if not, proceed to step a7;

[0022] Step a7, according to the subscription data packet length obtained by parsing in step a5, further read data from the monitoring system client connection to obtain the subscription data keyword, and enter step a8;

[0023] Step a8, storing the subscription data keyword into the data cache thread, creating a new subscription topic, setting the subscription topic reference count to 0, and proceeding to step a9;

[0024] Step a9, increment the subscription topic reference count by 1, go to step a2, and start the next loop;

[0025] Step a10, determining whether the monitoring system client communication link is connected, if the monitoring system client communication link is valid, proceed to step a11; if the monitoring system client communication link is invalid, proceed to step a17;

[0026] Step a11, judging whether the subscription topic has reached the data refresh time according to the following formula, if it has reached, proceeding to step a12; if it has not reached, proceeding to step a2 and starting the next cycle;

[0027] |Current time - last sent time|≥data update time

[0028] Step a12, judging the pause flag obtained in step a4, if the subscription topic is set to the pause flag, then going to step a2 and starting the next cycle; otherwise, going to step a13;

[0029] Step a13, determining the subscription data keyword in the data cache thread, if the subscription data keyword value is updated, proceeding to step a14; if no data is updated, proceeding to step a2 and starting the next cycle;

[0030] Step a14, organizing the response data according to the subscription data keyword obtained in step a13, and proceeding to step a15;

[0031] Step a15, the network communication thread sends response data to the monitoring system client connection, and then proceeds to step a16;

[0032] Step a16, judging the sending result of step a15, if the sending is successful, going to step a2 and starting the next cycle; otherwise going to step a17;

[0033] Step a17, disconnect the monitoring system client connection, decrement the subscription topic reference count by 1, and set the thread idle flag. Further, the monitoring system client request received in S3 includes: subscription sequence number, subscription type, subscription topic, data update time, application number, state number, subscription data length, keyword sequence number, keyword ID, keyword application number.

[0034] Furthermore, the subscription information of the monitoring system client in S4 includes: subscription serial number, subscription type, subscription topic, data update time, application number, state number, subscription data length. The response data in S4 includes: subscription serial number, data length, keyword serial number, keyword type, keyword length, keyword value.

[0035] Furthermore, an idle thread is found from the thread pool created by S2 as a garbage data recovery thread; the reference count size of each subscription topic is determined cyclically, and when the reference count of the subscription topic is 0, the subscription topic is set invalid, and the keyword information corresponding to the subscription topic in the data cache is cleared through the garbage data recovery thread.

[0036] The present invention also provides a system of real-time data service middleware for a monitoring system, comprising:

[0037] The parameter management module is used to create and maintain the global parameters during the operation of the middleware;

[0038] A thread pool module, used to create a thread pool when the middleware is initialized;

[0039] A network communication module, used to realize the communication between the middleware and the monitoring system client;

[0040] A data parsing module is used to process the monitoring system client request data received by the network communication module, obtain subscription information after processing, and obtain the requested subscription keyword according to the subscription data length;

[0041] A data cache module, used to store the subscription information of the monitoring system client obtained by the data parsing module;

[0042] The garbage collection module is used to recycle invalid subscription topics during the operation of the middleware.

[0043] Furthermore, the network communication module, data cache module, and garbage collection module all run in the thread pool module.

[0044] Beneficial effects: The present invention discloses a method and system for constructing a real-time data service middleware for a monitoring system. By means of multithreading, different functions of the middleware are divided into different separate threads to improve data reading efficiency. The present invention performs asynchronous processing on large-scale subscription data and reduces the resource consumption of the server by optimizing the communication process between the server and the monitoring system client. The real-time data service middleware for a monitoring system according to the present invention has the characteristics of high data processing efficiency and small resource occupation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for constructing real-time data service middleware for a monitoring system according to the present invention;

[0046] Figure 2 A schematic diagram of a real-time data service middleware system for a monitoring system according to the present invention;

[0047] Figure 3 The present invention is a communication flow chart between a monitoring system real-time data service middleware and a monitoring system client. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0049] The present invention provides a method for constructing a real-time data service middleware for a monitoring system. Figure 1 As shown, the following steps are included:

[0050] S1. Establish and initialize global parameters. Global parameters include registered service name, network service port number, thread pool size, data cache period, data push period, and number of monitoring system client connections;

[0051] S2. Create a thread pool, which includes a management thread and multiple worker threads;

[0052] S3 finds an idle thread from the thread pool created by S2 for network communication and communicates with the monitoring system client; creates a listening port in the thread according to the network service port number initialized by S1 and receives the monitoring system client request;

[0053] S4. According to the monitoring system client request received by S3, the subscription information is processed and the response data is parsed;

[0054] S5. Find an idle thread from the thread pool created by S2 for data caching and storing the subscription information parsed by S4; set the data caching period and data push period, and send the response data to the monitoring system client through the network communication thread of S3;

[0055] S6. Search for an idle thread from the thread pool created in S2 to recycle garbage data.

[0056] As a typical embodiment, the thread pool size is 50, the working thread program number is 1-49, and the management thread program number is 50;

[0057] Furthermore, the network module responds to the network connection request of the monitoring system client, such as Figure 3 As shown, the specific steps include:

[0058] Step a1, create a new monitoring system client connection, find an idle thread from the thread pool and add the monitoring system client connection task to the idle thread, and then go to step a2;

[0059] Step a2, according to the data push cycle set by the global parameters, cyclically read data from the monitoring system client connection, and enter step a3; if the data is empty, enter step a10;

[0060] Step a3, parsing the data obtained from the monitoring system client connection to obtain the subscription type. If the subscription type is modified, proceed to step a4; if the subscription type is a new type, proceed to step a5;

[0061] Step a4, determine the modification type. If it is a pause type, set the subscription topic pause sending flag and go to step a10; if it is a close type, go to step a17;

[0062] Step a5, parse the request packet to obtain subscription information, including subscription sequence number, subscription length, application, state, subscription subject information, subscription keyword information, and enter step a6;

[0063] Step a6, judging whether the current subscription topic exists according to the application, state and topic name in the subscription information, if it exists, proceed to step a9; if not, proceed to step a7;

[0064] Step a7, according to the subscription data packet length obtained by parsing in step a5, further read data from the monitoring system client connection to obtain the subscription data keyword, and enter step a8;

[0065] Step a8, storing the subscription data keyword into the data cache thread, creating a new subscription topic, setting the subscription topic reference count to 0, and proceeding to step a9;

[0066] Step a9, increment the subscription topic reference count by 1, go to step a2, and start the next loop;

[0067] Step a10, determining whether the monitoring system client communication link is connected, if the monitoring system client communication link is valid, proceed to step a11; if the monitoring system client communication link is invalid, proceed to step a17;

[0068] Step a11, judging whether the subscription topic has reached the data refresh time according to the following formula, if it has reached, proceeding to step a12; if it has not reached, proceeding to step a2 and starting the next cycle;

[0069] |Current time - last sent time|≥data update time

[0070] Step a12, judging the pause flag obtained in step a4, if the subscription topic is set to the pause flag, then going to step a2 and starting the next cycle; otherwise, going to step a13;

[0071] Step a13, determining the subscription data keyword in the data cache thread, if the subscription data keyword value is updated, proceeding to step a14; if no data is updated, proceeding to step a2 and starting the next cycle;

[0072] Step a14, organizing the response data according to the subscription data keyword obtained in step a13, and proceeding to step a15;

[0073] Step a15, the network communication thread sends response data to the monitoring system client connection, and then proceeds to step a16;

[0074] Step a16, judging the sending result of step a15, if the sending is successful, going to step a2 and starting the next cycle; otherwise going to step a17;

[0075] Step a17, disconnect the monitoring system client connection, decrement the subscription topic reference count by 1, and set the thread idle flag. Further, the monitoring system client request received in S3 includes: subscription sequence number, subscription type, subscription topic, data update time, application number, state number, subscription data length, keyword sequence number, keyword ID, keyword application number.

[0076] The data cache module updates and obtains the latest value from the local real-time database according to the application number and unique identification code of the cached keyword in the cache cycle cycle, and updates the keyword data update time if the data changes;

[0077] The garbage collection module cyclically determines the reference count size of each subscription topic. When the reference count of a subscription topic is 0, the subscription topic is set invalid and the keyword information corresponding to the subscription topic in the data cache is cleared.

[0078] As a typical embodiment, the unique identification code includes the table number, domain number, and record number of the keyword in the real-time database, and the memory address of the keyword in the real-time database can be quickly and uniquely located through the table number, domain number, and record number;

[0079] As a preferred embodiment, the size of the global parameter thread pool in the parameter management module is set to 50, the data cache period is set to 500ms, and the data push period is set to 1000ms. The monitoring system client request data format is shown in Table 1.

[0080]

[0081] Table 1

[0082] As a preferred embodiment, the data format of the response monitoring system client is shown in Table 2.

[0083]

[0084]

[0085] Table 2

[0086] The present invention also provides a system for monitoring system real-time data service middleware, such as Figure 2 As shown, it includes: parameter management module, thread pool module, network communication module, data parsing module, data cache module, and garbage collection module.

[0087] The parameter management module is used to create and maintain the global parameters during the operation of the middleware when the middleware is initialized;

[0088] The thread pool module is used to create a thread pool of a predefined size of the parameter management module when the middleware is initialized;

[0089] The data cache module is used to store the subscription topics and subscription keywords of the monitoring system client, as well as the latest values ​​corresponding to the keywords and the last update time corresponding to the keywords;

[0090] The garbage collection module is used to recycle expired resources during the operation of the middleware;

[0091] The data parsing module is used to obtain the subscription information of the monitoring system client request data received by the network communication module, including the subscription serial number, subscription type, subscription subject, data update time, application number, state number, and subscription data length, obtain the request keyword according to the subscription data length, and add the subscription information and keyword to the data cache module;

[0092] The network communication module is used for the middleware to communicate with the monitoring system client, receive monitoring system client requests, and send response data.

[0093] The present invention discloses a method and system for constructing a real-time data service middleware for a monitoring system. By means of multithreading, different functions of the middleware are divided into different separate threads to improve data reading efficiency. The present invention performs asynchronous processing on large-scale subscription data, and reduces resource consumption on the server side by optimizing the communication process between the server side and the monitoring system client side. The real-time data service middleware for a monitoring system involved in the present invention has the characteristics of high data processing efficiency and small resource occupation, and has broad application prospects in the business scenario of large-scale data monitoring of power systems.

Claims

1. A method for constructing real-time data service middleware for a monitoring system, characterized in that: The following steps are involved: S1. Establish and initialize global parameters; S2. Create a thread pool, which includes at least one management thread and multiple worker threads; S3. Find an idle thread from the thread pool created by S2 as a network communication thread, connect to the monitoring system client, and create a listening port in the network communication thread to communicate with the monitoring system client; S4. According to the monitoring system client request received by S3, the subscription information is processed and the response data is parsed; S5. Find an idle thread from the thread pool created by S2 as a data cache thread, store the subscription information and response data obtained by S4, and send the response data to the monitoring system client through the network communication thread of S3; S6. Search for an idle thread from the thread pool created in S2 as the garbage data collection thread.

2. The method for constructing the real-time data service middleware of the monitoring system according to claim 1, characterized in that: The global parameters of S1 include the registered service name, network service port number, thread pool size, data cache period, data push period, and the number of monitoring system client connections.

3. The method for constructing the real-time data service middleware of the monitoring system according to claim 2, characterized in that: The data cache in the global parameters periodically updates and obtains the latest data from the local real-time database. If the data changes, the update time of the keyword data is updated.

4. The method for constructing the real-time data service middleware of the monitoring system according to claim 1, 2 or 3, characterized in that: The S3 network communication thread realizes the communication between the middleware and the monitoring system client and includes the following steps: Step a1, create a new monitoring system client connection, find an idle thread from the thread pool and add the monitoring system client connection task to the idle thread, and then go to step a2; Step a2, according to the data push cycle set by the global parameters, cyclically read data from the monitoring system client connection, and enter step a3; if the data is empty, enter step a10; Step a3, parsing the data obtained from the monitoring system client connection to obtain the subscription type. If the subscription type is to be modified, proceed to step a4; If the subscription type is a new type, go to step a5; Step a4, determine the modification type. If it is a pause type, set the subscription topic pause sending flag and go to step a10; if it is a close type, go to step a17; Step a5, parse the request packet to obtain subscription information, including subscription sequence number, subscription length, application, state, subscription subject information, subscription keyword information, and proceed to step a6; Step a6, judging whether the current subscription topic exists according to the application, state and topic name in the subscription information, if it exists, proceed to step a9; if not, proceed to step a7; Step a7, according to the subscription data packet length obtained by parsing in step a5, further read data from the monitoring system client connection to obtain the subscription data keyword, and enter step a8; Step a8, storing the subscription data keyword into the data cache thread, creating a new subscription topic, setting the subscription topic reference count to 0, and proceeding to step a9; Step a9, increment the subscription topic reference count by 1, go to step a2, and start the next loop; Step a10, determining whether the monitoring system client communication link is connected, if the monitoring system client communication link is valid, proceed to step a11; if the monitoring system client communication link is invalid, proceed to step a17; Step a11, judging whether the subscription topic has reached the data refresh time according to the following formula, if it has reached, proceeding to step a12; if it has not reached, proceeding to step a2 and starting the next cycle; |Current time - last sent time|≥data update time Step a12, judging the pause flag obtained in step a4, if the subscription topic is set to the pause flag, then going to step a2 and starting the next cycle; otherwise, going to step a13; Step a13, determining the subscription data keyword in the data cache thread, if the subscription data keyword value is updated, proceeding to step a14; if no data is updated, proceeding to step a2, and starting the next cycle; Step a14, organizing the response data according to the subscription data keyword obtained in step a13, and proceeding to step a15; Step a15, the network communication thread sends response data to the monitoring system client connection, and then proceeds to step a16; Step a16, judging the sending result of step a15, if the sending is successful, going to step a2 and starting the next cycle; otherwise going to step a17; Step a17, disconnect the monitoring system client connection, decrement the subscription topic reference count by 1, and set the thread idle flag.

5. The method for constructing the real-time data service middleware of the monitoring system according to claim 4, characterized in that: The monitoring system client request received in S3 includes: subscription serial number, subscription type, subscription subject, data update time, application number, state number, subscription data length, keyword serial number, keyword ID, and keyword application number.

6. The method for constructing the real-time data service middleware of the monitoring system according to claim 5, characterized in that: The subscription information of the monitoring system client in S4 includes: subscription sequence number, subscription type, subscription subject, data update time, application number, state number, and subscription data length.

7. The method for constructing real-time data service middleware for a monitoring system according to claim 6, characterized in that: The response data in S4 includes: subscription sequence number, data length, keyword sequence number, keyword type, keyword length, and keyword value.

8. The method for constructing the real-time data service middleware of the monitoring system according to claim 4, characterized in that: Find an idle thread from the thread pool created by S2 as a garbage data recycling thread; loop to determine the reference count size of each subscription topic. When the reference count of a subscription topic is 0, set the subscription topic invalid. Clear the keyword information corresponding to the subscription topic in the data cache through the garbage data recycling thread.

9. A system for real-time data service middleware of a monitoring system, characterized in that: include: The parameter management module is used to create and maintain the global parameters during the operation of the middleware; A thread pool module, used to create a thread pool when the middleware is initialized; A network communication module, used to realize the communication between the middleware and the monitoring system client; A data parsing module is used to process the monitoring system client request data received by the network communication module, obtain subscription information after processing, and obtain the requested subscription keyword according to the subscription data length; A data cache module, used to store the subscription information of the monitoring system client obtained by the data parsing module; The garbage collection module is used to recycle invalid subscription topics during the operation of the middleware.

10. The system of real-time data service middleware for monitoring system according to claim 9, characterized in that: The network communication module, data cache module, and garbage collection module all run in the thread pool module.

Images