A real-time statistical method based on multicast and shared memory and a cloud web application firewall node
By multicasting the statistical information to be collected between cloud web application firewall nodes and aggregating and storing the data in shared memory, the problems of high latency and high resource consumption in existing technologies are solved, achieving high-efficiency statistical performance and a simplified deployment process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2022-12-28
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies in cloud web application firewalls suffer from high latency in real-time detection, high deployment and maintenance costs, high resource consumption, and wasted functionality. Especially in large-scale distributed environments, existing solutions require network I/O to obtain statistical results, leading to excessive latency and resource consumption.
A real-time statistical method using multicast and shared memory is adopted. The statistical information to be collected is multicast between cloud web application firewall nodes via UDP messages, and the data is aggregated and stored in shared memory. A trie is used for fast retrieval, which simplifies deployment and reduces the need for other dependent environments.
It improves query efficiency and statistical performance, reduces the impact of real-time logic, simplifies the deployment process, reduces resource consumption, and achieves efficient acquisition and processing of statistical results.
Smart Images

Figure CN116028550B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of network security technology, and in particular relates to a real-time statistical method based on multicast and shared memory, and a cloud web application firewall node. Background Technology
[0002] Real-time statistics technology is commonly used in online data analysis, security operations, and other fields, such as real-time traffic and bandwidth statistics, and real-time access volume statistics. Currently, there are various technical solutions in the industry that support real-time statistics, which are summarized below:
[0003] The first category is for statistics on relatively small amounts of data in a single-machine environment. In this type of scenario, the data volume is small, the required storage space is small, and the computation time is short, so it is feasible to perform statistics directly in memory. The second category is for single-machine environments with large amounts of data. In this type of scenario, it is advisable to use a database for auxiliary statistics or to use a caching service such as Redis for statistics, which can generally achieve near real-time statistical analysis. For distributed scenarios with large amounts of data, a simple solution can be to use a distributed caching service such as Redis for statistics. However, if the statistical task is more complex, it is generally necessary to consider using a big data infrastructure in conjunction with a real-time stream processing framework, such as Flink.
[0004] Real-time statistics technology is commonly used in online data analysis and security operations. Public cloud Web Application Firewalls (WAFs) rely on real-time statistics to implement policies such as blocking and frequency limiting of network traffic. The statistical metrics involved include bandwidth, specific window access frequency, and QPS. The core service of a cloud WAF consists of a forwarding and detection cluster composed of multiple nodes, each containing an NGINX service. The NGINX service provides forwarding and detection capabilities for HTTP traffic in the network. The cloud WAF also handles statistical tasks in the third scenario mentioned above: a large-scale distributed environment. While the above technical solutions can address the needs of statistical tasks, the following problems still exist:
[0005] 1) High latency for real-time monitoring or serial devices. The aforementioned technologies require network I / O methods, such as Redis GET requests or other network APIs, to obtain statistical results. The latency caused by network I / O is difficult to meet the low-latency requirements of cloud WAF.
[0006] 2) High deployment and maintenance costs, and high resource consumption. The above solutions require additional allocation of computing, storage, and network resources for deployment, and have high maintenance costs and high resource consumption during operation.
[0007] 3) Wasted functionality. The statistical functions required by public cloud WAFs mainly include CC protection-related statistics, bandwidth statistics, total request frequency limiting statistics, etc. These statistical tasks are mainly based on counting and summing, and do not require complex statistical logic or calculation processes. The existing technologies mentioned above integrate rich statistical functions, but most of these functions are not actually used, and additional maintenance costs are incurred instead. Summary of the Invention
[0008] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a real-time statistical method and cloud web application firewall node based on multicast and shared memory. It has little impact on real-time logic, improves query efficiency and statistical performance by querying statistical results from shared memory, and is simple to deploy, requires no other dependent environment, and consumes few resources.
[0009] According to one aspect of the present invention, a real-time statistics method based on multicast and shared memory is provided, the method being applied to a cloud web application firewall node, comprising the following steps:
[0010] S1: Start the NGINX process to multicast the request message containing the statistical information to the Receiver process in other nodes;
[0011] S2: Start the Receiver process to receive request messages containing the statistical information to be collected, which are multicast by other nodes;
[0012] S3: Call the Receiver process to perform statistics on the information to be counted, and update the statistical results to the shared memory for storage.
[0013] Preferably, the request message containing the statistical information to be collected is a UDP message;
[0014] When sending the UDP message to other nodes, the UDP message is serialized and formatted according to the communication protocol, and then multicast to other nodes in the node list; when receiving the UDP message from other nodes, the payload is extracted from the received UDP message and deserialized to obtain the message content, and node updates or statistical operations are performed based on the message content.
[0015] Preferably, the statistical analysis of the information to be statistically analyzed includes:
[0016] All input messages are aggregated according to the aggregation key and time window. A trie is constructed according to the aggregation key. The trie is used for fast retrieval. Individual statistical messages are assigned to the corresponding leaf nodes to perform data aggregation and statistics.
[0017] Preferably, the shared memory includes node information shared memory, trie shared memory, and data node shared memory;
[0018] The node information shared memory is used to store a list of IPs of currently online nodes, and the IP list is read for multicasting UDP messages;
[0019] The shared memory of the trie is used to store the key-value structure through the trie, that is, to construct a trie using the key and store the value in the leaf nodes;
[0020] The shared memory of the data nodes is used to store message statistics for specific keys in the data nodes pointed to by the leaf nodes of the dictionary.
[0021] Preferably, the method includes:
[0022] When a node goes online or offline, the NGINX process is invoked to send online / offline messages to each node.
[0023] When statistical results need to be queried, the NGINX process is invoked to retrieve data nodes from shared memory via a trie, and statistical results are obtained from the data nodes.
[0024] According to another aspect of the present invention, the present invention also provides a cloud web application firewall node for real-time statistics based on multicast and shared memory, the node including a sending module, a receiving module, and a statistics module;
[0025] The sending module is used to start the NGINX process and multicast the request message containing the statistical information to be collected to the Receiver process in other nodes;
[0026] The receiving module is used to start the Receiver process and receive request messages containing statistical information multicast by other nodes;
[0027] The statistics module is used to call the Receiver process to perform statistics on the information to be counted, and update the statistical results to shared memory for storage.
[0028] Preferably, the request message containing the statistical information to be collected is a UDP message;
[0029] When the sending module sends the UDP message to other nodes, it serializes the UDP message and formats it according to the communication protocol, and then multicasts it to other nodes in the node list. When the receiving module receives the UDP message from other nodes, it extracts the payload from the received UDP message and deserializes it to obtain the message content, and performs node updates or statistical operations based on the message content.
[0030] Preferably, the statistical module performs statistical analysis on the information to be statistically analyzed, including:
[0031] All input messages are aggregated according to the aggregation key and time window. A trie is constructed according to the aggregation key. The trie is used for fast retrieval. Individual statistical messages are assigned to the corresponding leaf nodes to perform data aggregation and statistics.
[0032] Preferably, the shared memory includes node information shared memory, trie shared memory, and data node shared memory;
[0033] The node information shared memory is used to store a list of IPs of currently online nodes, and the IP list is read for multicasting UDP messages;
[0034] The shared memory of the trie is used to store the key-value structure through the trie, that is, to construct a trie using the key and store the value in the leaf nodes;
[0035] The shared memory of the data nodes is used to store message statistics for specific keys in the data nodes pointed to by the leaf nodes of the dictionary.
[0036] Preferably, the sending module is further configured to:
[0037] When a node goes online or offline, the NGINX process is invoked to send online / offline messages to each node.
[0038] When statistical results need to be queried, the NGINX process is invoked to retrieve data nodes from shared memory via a trie, and statistical results are obtained from the data nodes.
[0039] Beneficial effects: This invention adds a new Receiver process to receive and process messages to be statistically analyzed, and writes the statistical results into shared memory. When processing requests, the NGINX process on each node multicasts the information to be analyzed to the Receiver process on each node in the cluster via UDP messages, ensuring that each node's shared memory has a complete copy of the statistical results. Finally, the NGINX process accesses the shared memory to retrieve the statistical results and performs subsequent processing. This invention has minimal impact on real-time logic, improves query efficiency and statistical performance by querying statistical results from shared memory, and is simple to deploy, requiring no other dependent environments and consuming few resources.
[0040] The features and advantages of the present invention will become clear from the following accompanying drawings and a detailed description of specific embodiments thereof. Attached Figure Description
[0041] Figure 1 This is a flowchart of a real-time statistics method based on multicast and shared memory;
[0042] Figure 2 This is a diagram illustrating multicast UDP messages between nodes;
[0043] Figure 3 This is a diagram of a trie structure;
[0044] Figure 4 Data node structure diagram;
[0045] Figure 5 This is a diagram illustrating memory expansion;
[0046] Figure 6 This is a diagram of a shared memory layout;
[0047] Figure 7 This is a schematic diagram of the cloud web application firewall node structure of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Example 1
[0050] See Figure 1 and Figure 2 This invention provides a real-time statistics method based on multicast and shared memory, which is applied to a cloud web application firewall node and includes the following steps:
[0051] S1: Start the NGINX process to multicast the request message containing the statistical information to the Receiver process in other nodes;
[0052] S2: Start the Receiver process to receive request messages containing the statistical information to be collected, which are multicast by other nodes;
[0053] S3: Call the Receiver process to perform statistics on the information to be counted, and update the statistical results to the shared memory for storage.
[0054] This method adds a new Receiver process to receive and process messages to be statistically analyzed, and writes the statistical results to shared memory. When processing requests, the NGINX process on each node multicasts the information to be analyzed to the Receiver process on each node in the cluster via UDP messages. This ensures that each node's shared memory has a complete copy of the statistical results. Finally, the NGINX process accesses the shared memory to retrieve the statistical results and performs subsequent processing. This has minimal impact on real-time logic. By querying the statistical results from shared memory, query efficiency and statistical performance are improved. Furthermore, deployment is simple, requires no other dependencies, and consumes minimal resources.
[0055] Preferably, the request message containing the statistical information to be collected is a UDP message;
[0056] When sending the UDP message to other nodes, the UDP message is serialized and formatted according to the communication protocol, and then multicast to other nodes in the node list; when receiving the UDP message from other nodes, the payload is extracted from the received UDP message and deserialized to obtain the message content, and node updates or statistical operations are performed based on the message content.
[0057] Specifically, in the cluster, each node sends messages to other nodes via UDP multicast. There are two types of messages:
[0058] 1. Node online / offline notifications. Cloud WAF clusters support elastic scaling up and down, so it is necessary to notify other nodes when a node goes online or offline, adding its own address to or removing it from the multicast address list of other nodes;
[0059] 2. Statistics Messages. The statistics message contains raw statistical information. If each node finds data that needs to be statistically analyzed, it will multicast the message to other nodes in the cluster. For example, in the scenario of total QPS statistics, when node A receives a request that needs to be statistically analyzed, it will multicast the statistical information of the request to other nodes.
[0060] Considering the above two types of message requirements, this invention designs the following data structure to store messages, serializes them in contiguous memory in an unaligned mode, and encapsulates them in UDP packets for communication.
[0061]
[0062]
[0063] Here, message_t is the message data structure, and its fields and types are shown in the table below:
[0064]
[0065] When a node receives a UDP packet from another node, it can deserialize the transmitted data to obtain the specific message data. The following is a specific example of a statistical message transmission:
[0066] The statistical objective is to count the number of visits to the test.com site in the last 30 seconds;
[0067] When a node receives a request from test.com, it uses "test.com-cnt" as the aggregation key to construct the following statistics message, serializes it, and then multicasts it to other nodes:
[0068]
[0069] After receiving the UDP packet, other nodes deserialize it to obtain the original message data and perform statistical calculations locally, i.e., determine the time window. If the timestamp is within the most recent 5-second time window, the statistical value corresponding to the key "test.com-cnt" will be increased by 1.
[0070] The message sending module is responsible for serializing messages and sending them to each node in the node list according to the communication protocol format; the message receiving module will extract the payload from the received UDP packets and deserialize them into structured message data, and perform node updates or statistical operations according to the message content.
[0071] The following table shows examples of statistical message formats:
[0072] key value Timestamp Time window
[0073] Preferably, the statistical analysis of the information to be statistically analyzed includes:
[0074] All input messages are aggregated according to the aggregation key and time window. A trie is constructed according to the aggregation key. The trie is used for fast retrieval. Individual statistical messages are assigned to the corresponding leaf nodes to perform data aggregation and statistics.
[0075] Specifically, the statistics module is the core module of the entire real-time statistics system. It is responsible for aggregating all input messages according to the aggregation key value and time window, and providing a query interface to provide upper-layer applications with the ability to query statistical results.
[0076] This embodiment uses a trie structure to implement aggregate statistics. Specifically, a trie is constructed based on the aggregate key, and a fast retrieval is performed using the trie. Individual statistical messages are then assigned to their corresponding leaf nodes for aggregate statistics. The trie structure is as follows: Figure 3 As shown.
[0077] This embodiment implements a data statistics node (hereinafter referred to as "data node") that supports time windows using an expandable circular array to cooperate with the aforementioned trie to complete the statistical task; that is, each leaf node of the trie points to a "data node". The data node consists of the following parts:
[0078]
[0079] The `head` pointer points to the position of the latest statistical data, and the `tail` pointer points to the position of the oldest data. The valid data stored in the current data node is located from `tail` to `head` within the current `data` array. Each element in the `data` array stores the aggregated value `V` for the specified `timestamp` second. The data node structure is as follows: Figure 4 As shown.
[0080] When a new statistical message is received, the system uses the timestamp data in the message to locate the element in the circular array that needs updating (the element storing the data aggregation results at a certain moment) based on the timestamp pointed to by the head pointer. The system then updates the aggregation value V in that element. If the current message timestamp is greater than the timestamp indicated by the head pointer, the head pointer is updated to the pointer corresponding to the latest timestamp. After updating the single array element value Vt, the system obtains the tail pointer position corresponding to the effective window based on the statistical time window (time_window) corresponding to the current statistical message. This is achieved by subtracting time_window from the current head pointer to obtain the tail pointer value of the effective window, and then updating the tail pointer. Data within the range from the original tail pointer to the new tail pointer is expired data. Updating the aggregation value window_value of the time window based on the expired data values completes the update of the window's aggregation value, thus implementing message-driven passive updating of the sliding time window.
[0081] When the time window exceeds the current capacity `cap` of `data`, `data` needs to be resized. This is because `data` is a circular array, and resizing requires considering the relative relationship between the head and tail pointers. When the head pointer is greater than the tail pointer, normal resizing is performed sequentially within `data`. When the head pointer is less than the tail pointer, meaning the head is looping, resizing must be performed between the head and tail pointers to ensure the correct relative relationship of the circular array. Figure 5 As shown.
[0082] Preferably, the shared memory includes node information shared memory, trie shared memory, and data node shared memory;
[0083] The node information shared memory is used to store a list of IPs of currently online nodes, and the IP list is read for multicasting UDP messages;
[0084] The shared memory of the trie is used to store the key-value structure through the trie, that is, to construct a trie using the key and store the value in the leaf nodes;
[0085] The shared memory of the data nodes is used to store message statistics for specific keys in the data nodes pointed to by the leaf nodes of the dictionary.
[0086] Specifically, in addition to the statistical analysis modules, to enable upper-layer applications to efficiently obtain statistical results, this invention implements the statistical result query function by accessing shared memory. This invention designs three shared memory blocks to store necessary information related to statistics and control, as follows:
[0087] a) Node information shared memory
[0088] This shared content is used to store a list of currently online node IPs. When the engine sends a UDP message, it reads the node IP addresses stored in this part of the memory and performs a multicast.
[0089] b) Trie shared memory
[0090] To improve storage and retrieval efficiency, this invention stores the Key-Value structure using a trie, that is, constructing a trie using the keys and storing the values in the leaf nodes. For example, the tree structure after storing the two keys "Key1" and "Key2" is as follows: Figure 3 As shown.
[0091] Because shared memory is contiguous memory, the actual technical implementation requires storing the trie in a contiguous block of memory. Each node of the trie occupies a block of address in contiguous memory, and the offset of the starting address of the next node is stored in each node.
[0092] c) Data node shared memory
[0093] The data nodes pointed to by the leaf nodes of the dictionary tree are used to store message statistics for specific keys. That is, the data statistics storage node for each key is the aforementioned data node, and its shared memory layout is as follows. Figure 6 As shown.
[0094] Preferably, the method includes:
[0095] When a node goes online or offline, the NGINX process is invoked to send online / offline messages to each node.
[0096] When statistical results need to be queried, the NGINX process is invoked to retrieve data nodes from shared memory via a trie, and statistical results are obtained from the data nodes.
[0097] Specifically, this step includes:
[0098] Step 1: Start the NGINX service on the cloud WAF node. The engine node sends an online message to other nodes in the cluster using the SDK message format of this invention.
[0099] Step 2: Start the Receiver process. The Receiver begins to receive multicast messages from other nodes in the cluster and begins to perform real-time statistics and updates to the shared memory.
[0100] Step 3: The NGINX service on the cloud WAF node queries the statistics in the shared memory as needed, and executes real-time protection policies based on the statistics.
[0101] Step 4: When a node needs to go offline, it multicasts an offline message to other nodes in the cluster to notify them that the current node has gone offline.
[0102] Step 5: Close the NGINX process and the Receiver process.
[0103] Preferably, another workflow of this embodiment is as follows:
[0104] a) Define the process that performs receiving and statistics as the Receiver process, and provide the sending module to the upper layer application through the SDK;
[0105] b) During the initialization phase, each node in the cluster starts a Receiver process to prepare to receive UDP messages and creates the aforementioned shared memory to prepare for storing statistical results;
[0106] c) During node operation, NGINX sends UDP messages through the SDK to multicast the messages that need to be counted to the Receiver process of each node;
[0107] d) After receiving the message, the Receiver performs data statistics and updates the statistical results to the shared memory for storage;
[0108] e) When a node goes online or offline, NGINX sends an online / offline message to each node via the SDK;
[0109] f) When statistical results need to be queried, NGINX retrieves data nodes from shared memory using a trie and obtains statistical results from the data nodes.
[0110] Compared with the prior art, the advantages and effects of this embodiment are:
[0111] 1) Non-blocking message sending minimizes impact on real-time logic. This invention uses UDP for multicasting statistical messages, without blocking real-time forwarding and detection logic;
[0112] 2) Querying statistical results from shared memory offers extremely high efficiency. Because cloud WAFs require real-time processing based on statistical data, such as for CC attack protection and bandwidth limiting, they need to synchronously read statistical data. Therefore, cloud WAF nodes have a need for high-performance data reading. Existing technologies such as Redis require obtaining statistical results via network I / O, a process involving establishing connections, sending requests, and waiting for responses. If the network fluctuates, read latency becomes even more unpredictable. Reading from shared memory, however, is several orders of magnitude more efficient than network I / O and is unaffected by network fluctuations.
[0113] 3) Simple deployment, no dependencies, and low resource consumption. This invention only requires a single Receiver process to run on each cloud WAF node, consuming minimal resources and requiring no other dependent environments. Other existing technologies require more resources, have more complex deployment processes, and consume more effort and resources.
[0114] 4) Targeted optimization of statistical logic for high performance. Targeting the aggregation and statistical needs of cloud WAF scenarios, the statistical logic has been specifically optimized. Mechanisms such as constructing a trie, merging messages at the second level, and passively updating window data efficiently complete data aggregation and statistics. Let the length of the statistical data key value be N, and the statistical time window length be T. Then, the time complexity of processing a single message's statistical task is O(N+T), and the time complexity of querying statistical results is O(N). Both statistics and queries can be guaranteed to be completed within linear time complexity.
[0115] Example 2
[0116] Figure 7 This is a schematic diagram of the cloud web application firewall node structure of the present invention. Figure 7 As shown, the present invention also provides a cloud web application firewall node for real-time statistics based on multicast and shared memory. The node includes a sending module, a receiving module, and a statistics module.
[0117] The sending module 701 is used to start the NGINX process and multicast the request message containing the statistical information to be counted to the Receiver process in other nodes.
[0118] The receiving module 702 is used to start the Receiver process and receive request messages containing statistical information multicast by other nodes;
[0119] The statistics module 703 is used to call the Receiver process to perform statistics on the information to be counted and update the statistics results to shared memory for storage.
[0120] Preferably, the request message containing the statistical information to be collected is a UDP message;
[0121] When the sending module 701 sends the UDP message to other nodes, it serializes the UDP message and formats it according to the communication protocol, and then multicasts it to other nodes in the node list. When the receiving module 702 receives the UDP message from other nodes, it extracts the payload from the received UDP message and deserializes it to obtain the message content, and performs node updates or statistical operations based on the message content.
[0122] Preferably, the statistical module 703 performs statistical analysis on the information to be statistically analyzed, including:
[0123] All input messages are aggregated according to the aggregation key and time window. A trie is constructed according to the aggregation key. The trie is used for fast retrieval. Individual statistical messages are assigned to the corresponding leaf nodes to perform data aggregation and statistics.
[0124] Preferably, the shared memory includes node information shared memory, trie shared memory, and data node shared memory;
[0125] The node information shared memory is used to store a list of IPs of currently online nodes, and the IP list is read for multicasting UDP messages;
[0126] The shared memory of the trie is used to store the key-value structure through the trie, that is, to construct a trie using the key and store the value in the leaf nodes;
[0127] The shared memory of the data nodes is used to store message statistics for specific keys in the data nodes pointed to by the leaf nodes of the dictionary.
[0128] Preferably, the sending module 701 is further configured to:
[0129] When a node goes online or offline, the NGINX process is invoked to send online / offline messages to each node.
[0130] When statistical results need to be queried, the NGINX process is invoked to retrieve data nodes from shared memory via a trie, and statistical results are obtained from the data nodes.
[0131] The specific implementation process of the functions implemented by each module in this embodiment 2 is the same as that in embodiment 1, and will not be repeated here.
[0132] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. A real-time statistical method based on multicast and shared memory, characterized in that, The method is applied to a cloud web application firewall node and includes the following steps: S1: Start the NGINX process to multicast the request message containing the statistical information to the Receiver process in other nodes; S2: Start the Receiver process to receive request messages containing the statistical information to be collected, which are multicast by other nodes; S3: Call the Receiver process to perform statistical analysis on the information to be analyzed, and update the statistical results to shared memory for storage; The statistical analysis of the information to be analyzed in S3 includes: All input messages are aggregated according to the aggregation key and time window. A trie is constructed according to the aggregation key, and fast retrieval is performed through the trie. Individual statistical messages are assigned to the corresponding leaf nodes to perform data aggregation and statistics. The request message containing the statistical information to be analyzed is a UDP message; When sending the UDP message to other nodes, the UDP message is serialized and formatted according to the communication protocol, and then multicast to other nodes in the node list; when receiving the UDP message from other nodes, the payload is extracted from the received UDP message and deserialized to obtain the message content, and node updates or statistical operations are performed based on the message content.
2. The method according to claim 1, characterized in that, The shared memory includes node information shared memory, trie shared memory, and data node shared memory; The node information shared memory is used to store a list of IPs of currently online nodes, and the IP list is read for multicasting UDP messages; The shared memory of the trie is used to store the key-value structure through the trie. A trie is constructed using the key, and the value is stored in the leaf nodes. The shared memory of the data nodes is used to store message statistics for specific keys in the data nodes pointed to by the leaf nodes of the dictionary.
3. The method according to any one of claims 1-2, characterized in that, The method includes: When a node goes online or offline, the NGINX process is invoked to send online / offline messages to each node. When statistical results need to be queried, the NGINX process is invoked to retrieve data nodes from shared memory via a trie, and statistical results are obtained from the data nodes.
4. A cloud web application firewall node for real-time statistics based on multicast and shared memory, characterized in that, The node includes a sending module, a receiving module, and a statistics module; The sending module is used to start the NGINX process and multicast the request message containing the statistical information to be collected to the Receiver process in other nodes; The receiving module is used to start the Receiver process and receive request messages containing statistical information multicast by other nodes; The statistics module is used to call the Receiver process to perform statistics on the information to be counted, and update the statistics results to shared memory for storage. The statistical module performs the following statistical operations on the information to be statistically analyzed: All input messages are aggregated according to the aggregation key and time window. A trie is constructed according to the aggregation key, and fast retrieval is performed through the trie. Individual statistical messages are assigned to the corresponding leaf nodes to perform data aggregation and statistics. The request message containing the statistical information to be analyzed is a UDP message; When the sending module sends the UDP message to other nodes, it serializes the UDP message and formats it according to the communication protocol, and then multicasts it to other nodes in the node list. When the receiving module receives the UDP message from other nodes, it extracts the payload from the received UDP message and deserializes it to obtain the message content, and performs node updates or statistical operations based on the message content.
5. The node according to claim 4, characterized in that, The shared memory includes node information shared memory, trie shared memory, and data node shared memory; The node information shared memory is used to store a list of IPs of currently online nodes, and the IP list is read for multicasting UDP messages; The shared memory of the trie is used to store the key-value structure through the trie. A trie is constructed using the key, and the value is stored in the leaf nodes. The shared memory of the data nodes is used to store message statistics for specific keys in the data nodes pointed to by the leaf nodes of the dictionary.
6. The node according to any one of claims 4-5, characterized in that, The sending module is also used for: When a node goes online or offline, the NGINX process is invoked to send online / offline messages to each node. When statistical results need to be queried, the NGINX process is invoked to retrieve the data node from the shared memory using a trie, and then retrieve the statistical results from the data node.
Citation Information
Patent Citations
Method for synchronizing multi-computer communication in shared memory of power quality management system
CN109032816A