A large-scale concurrent data monitoring component, method and cluster state monitoring system

Abstract

The application discloses a large-scale concurrent data monitoring component, comprising: a monitoring flow unit, configured to receive a monitoring data list sent by a client, further configured to generate a corresponding monitor unit for each monitoring data in the monitoring data list, and further configured to push modification information of data in a multi-version data storage unit to the client; the monitor unit, configured to store the corresponding monitoring data, further configured to monitor modification of the data in the multi-version data storage unit, and match the modified data with the stored monitoring data, and the monitor unit is connected with the monitoring flow unit through a message queue; and the multi-version data storage unit, configured to provide a data modification channel, and further configured to store an original version of the modified data and a modified version corresponding to the original version. The large-scale concurrent data monitoring component can realize that the client can obtain all modification records of the data, avoid mutual covering of the data due to multiple non-time sequence modifications, and prevent the data from being disordered or even lost.

Description

Technical Field

[0001] This invention relates to the field of data monitoring technology, and more specifically, to a large-scale concurrent data monitoring component, method, and cluster status monitoring system. Background Technology

[0002] Traditional client-side collaboration methods can only work with data of fixed values. When three or more clients simultaneously modify the data, data can be overwritten, leading to inconsistent data retrieval. Furthermore, data loss can occur when multiple clients operate at different times, resulting in a lack of linear consistency guarantees for the entire system.

[0003] In view of the above, this application is hereby submitted. Summary of the Invention

[0004] The technical problem to be solved by this invention is that when data is modified multiple times out of order, data collaboration using traditional client-side collaboration methods is prone to data corruption and data loss.

[0005] The purpose is to provide a large-scale concurrent data monitoring component, method, and cluster status monitoring system. By establishing a data monitoring mechanism, it acquires client monitoring requests and data to be monitored, and stores the original and modified versions of the data to be modified in a database. Based on the data to be monitored, it queries the database to see if the data corresponding to the data to be monitored has been modified, and feeds back the data modification record to the client. This allows the client to obtain all data modification records, avoiding data overwriting due to multiple unordered modifications, which could lead to data chaos or even loss.

[0006] on the one hand,

[0007] This invention provides a large-scale concurrent data monitoring component, comprising:

[0008] The monitoring stream unit is used to receive the list of listening data sent by the client; it is also used to generate a corresponding monitor unit for each piece of listening data in the list; and it is also used to push modification information of data in the multi-version data storage unit to the client; one monitoring stream unit corresponds to one client.

[0009] The monitor unit is used to store the corresponding monitoring data; it is also used to monitor the modification status of data in the multi-version data storage unit and match the modified data with the stored monitoring data; the monitor unit and the monitoring stream unit are connected through a message queue; the message queue is used to send the modification information of the successfully matched data to the monitoring stream unit.

[0010] The multi-version data storage unit is used to provide a data modification channel and also to store the original version of the modified data and the modified version corresponding to the original version.

[0011] Furthermore,

[0012] The large-scale concurrent data monitoring group also includes: a monitoring group, which manages the monitoring units; one monitoring group corresponds to multiple monitoring units.

[0013] Furthermore,

[0014] The monitoring group includes a management unit with a segment tree data structure; each leaf node of the segment tree data structure corresponds to a list of monitor units; the list of monitor units includes multiple monitor units corresponding to the same monitored data; the value of the leaf node is the list of monitor units, and the key of the leaf node is the monitored data corresponding to the list of monitor units.

[0015] Furthermore,

[0016] The multi-version data storage unit includes a globally incrementing version number, which is used to generate a new data version for modified data and store the new data version.

[0017] Furthermore,

[0018] The multi-version data storage unit supports default read operations for the latest version of data and specified read operations for any version of data.

[0019] Furthermore,

[0020] The monitoring stream unit is also used to receive listening requests and basic client information sent by the client;

[0021] The monitoring stream unit is connected to the client via a data channel, which is used to send the modification information received by the monitoring stream unit to the corresponding client;

[0022] The monitor unit is also used to store basic client information and send modification information to the monitoring stream unit via a message queue based on the basic client information.

[0023] on the other hand,

[0024] This invention provides a method for large-scale concurrent data monitoring, comprising the following steps:

[0025] Receive a list of listening data sent by the client;

[0026] Store each piece of monitored data in the monitored data list;

[0027] Receive a data modification request, perform corresponding data modification operations based on the data modification request, and store the original version of the modified data and the modified version corresponding to the original version.

[0028] For each piece of monitored data, the monitored data is matched with the modified data. If the match is successful, the modification information of the modified data is pushed to the client.

[0029] Furthermore,

[0030] Before matching the monitored data with the modified data, the process also includes: establishing a data matching model based on a segment tree;

[0031] Establishing a data matching model based on segment trees includes:

[0032] Obtain the basic data structure of the segment tree;

[0033] For each piece of monitored data, a corresponding monitor unit object is constructed. The monitor unit object is used to store the monitored data and also to match the monitored data with the modified data. If the match is successful, the modification information of the modified data is pushed to the client.

[0034] For each leaf node of the segment tree, perform the following steps:

[0035] Multiple monitor units corresponding to the same monitored data are merged into a monitor unit list, and the monitor unit list is used as the value of the leaf node; the monitored data corresponding to the monitor unit list is used as the key of the leaf node.

[0036] On the other hand,

[0037] This invention provides a cluster status monitoring system, comprising:

[0038] Monitoring center, large-scale concurrent data monitoring components, and multiple nodes;

[0039] The monitoring center is connected to the large-scale concurrent data listening component through a data transmission channel. The data transmission channel is used to transmit the monitoring request and the list of monitorable data sent in the monitoring to the large-scale concurrent data listening component, and also to transmit the real-time running status of the nodes fed back by the large-scale concurrent data listening component to the monitoring center.

[0040] Multiple nodes are connected to the large-scale concurrent data monitoring component via a data bus, which is used to write the real-time running status of the nodes continuously output by the multiple nodes into the large-scale concurrent data monitoring component.

[0041] The monitoring center is used to adjust abnormal nodes based on the real-time operating status of nodes reported by the large-scale concurrent data monitoring component.

[0042] Furthermore,

[0043] The large-scale concurrent data monitoring component also includes a node liveness status monitoring module, which generates corresponding single-point failure messages for nodes that fail to send real-time running status multiple times in a row, and feeds the single-point failure messages back to the monitoring center.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] 1. This invention provides a large-scale concurrent data monitoring component and method. On one hand, it receives a list of monitoring data sent by the client one-to-one through a monitoring stream unit, obtains the data requested by the client for monitoring, and generates a one-to-one monitor unit object for the requested data. On the other hand, it stores the original version and the modified version of the data to be modified in a multi-version data storage unit. When the data is modified, the monitor unit matches the requested data with the modified data. If the match is successful, it means that the data the client wants to monitor has been modified, and the monitoring stream unit sends the modification information of the modified data to the client. By monitoring data according to this mechanism, the number of times and the time sequence of data modifications can be ignored, so that the client can always obtain all the modification information of the modified data, thereby avoiding data overwriting due to multiple unordered modifications, which would lead to data chaos. Furthermore, by storing the original version and each modified version of the data in a unified manner, it is also possible to prevent data loss during multiple unordered modifications, ensuring data integrity and consistency before and after modification.

[0046] 2. The present invention provides a large-scale concurrent data monitoring component and method based on a segment tree data structure. The monitored data is used as the key of the leaf node of the segment tree, and multiple monitor units corresponding to the same monitored data are used as the values ​​of the leaf nodes. When multiple clients request to monitor the same data, they only need to match the modified data with the key. Based on the correspondence between the key and multiple monitor units, multiple clients corresponding to multiple monitor units can obtain the modification information of the modified data without having to traverse all monitor units one by one, thus speeding up the efficiency of pushing data modification information.

[0047] 3. The cluster status monitoring system provided by this invention inherits the large-scale concurrent data listening component, takes the monitoring center as the client and each node as a multi-party data modifier, and establishes a data transmission association with the large-scale concurrent data listening component. Through the coordination of the large-scale concurrent data listening component, the monitoring center can monitor the real-time running status of each node and dynamically adjust the cluster resources in a timely manner according to the real-time running status of the nodes to ensure the normal operation of the entire cluster.

[0048] 4. The cluster status monitoring system provided by the present invention has a node survival status monitoring module that can monitor whether a node has a single point of failure by judging the time interval between the node running status sent by the node before and after. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the overall structure of the large-scale concurrent data monitoring component provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the internal structure of a large-scale concurrent data monitoring component provided in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram illustrating the implementation principle of monitoring within a monitoring group and range monitoring based on a segment tree, provided in an embodiment of the present invention.

[0053] Figure 4 This is a schematic diagram of the large-scale concurrent data monitoring method provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0055] Example 1

[0056] Traditional client-side collaboration methods are prone to data overwriting when multiple clients modify data simultaneously, leading to chaotic or even lost query data and a lack of linear consistency guarantees for the entire system.

[0057] To address the aforementioned issues, this embodiment provides a large-scale concurrent data monitoring component, such as... Figure 1 As shown. Includes:

[0058] The monitoring stream unit is used to receive a list of monitored data sent by the client; it is also used to generate a corresponding monitor unit for each piece of monitored data in the monitored data list; and it is also used to push modification information of data in the multi-version data storage unit to the client; one monitoring stream unit corresponds to one client.

[0059] The monitor unit is used to store the corresponding monitoring data; it is also used to monitor the modification status of the data in the multi-version data storage unit and match the modified data with the stored monitoring data; the monitor unit and the monitoring stream unit are connected through a message queue; the message queue is used to send the modification information of the successfully matched data to the monitoring stream unit.

[0060] The multi-version data storage unit is used to provide a data modification channel and to store the original version of the modified data and the modified version corresponding to the original version.

[0061] Combination Figure 1 Based on the overall working principle of the large-scale concurrent data monitoring component described above, the client requests the monitoring component to monitor a specific piece of data or a range of data within a multi-version data storage unit. When data in a multi-version data storage unit is modified, the monitoring group determines whether the modified data is being monitored by a client. If a client is monitoring, the modification information of the modified data is fed back to the client through the monitoring stream unit.

[0062] Therefore, from the client's perspective, regardless of how many data modifiers modify the data in the multi-version data storage unit, and regardless of the modification sequence, the client can obtain all data modification operations through the listening mechanism of the aforementioned large-scale concurrent data listening component.

[0063] Data monitoring provides a solution to the problem of data pin confusion or even data loss caused by different modification sequences of multiple modifiers by pushing data modification information to the client through a large-scale concurrent data monitoring component. At the same time, it retains a complete record of data modification, ensuring the linear consistency of the system.

[0064] For details, please refer to Figure 2 The internal structure of the large-scale concurrent data monitoring component shown below, and its workflow, are as follows:

[0065] First, the client initiates a listening request to the large-scale concurrent data monitoring component, sending a list of data to be monitored. Upon receiving the listening request and data list from the client, the monitoring stream unit generates a corresponding monitor unit for each piece of data in the list. The monitor unit stores the client information and the corresponding data list. Simultaneously, the monitoring stream unit establishes a data channel with the client to promptly push data modification information to the client. When data is modified in the multi-version data storage unit, after the modification is complete, the monitor unit first checks whether the modified data corresponds to a piece of data in the data list, i.e., it performs data matching. If a match is found, the modification information of the modified data is sent to the monitoring stream unit, which then pushes it to the corresponding client.

[0066] It should be noted that the list of listening data sent by the client may contain multiple pieces of listening data that need to be monitored. In this case, the monitoring stream unit will generate a corresponding monitor unit for each piece of listening data. To facilitate the management of numerous monitor units, in large-scale concurrent data monitoring components, a monitoring group is responsible for managing multiple monitor units.

[0067] It should also be noted that,

[0068] The monitoring stream unit is also used to receive listening requests and basic client information sent by the client;

[0069] The monitoring stream unit is connected to the client via a data channel; the data channel is used to send the modification information received by the monitoring stream unit to the corresponding client.

[0070] The monitor unit is also used to store basic client information and send modified information to the monitoring stream unit via a message queue based on the basic client information.

[0071] Furthermore, when multiple data modifiers modify multiple data items in a multi-version data storage unit, a particular piece of data may be crucial to multiple clients, leading to multiple clients simultaneously monitoring that data. Conversely, there might be cases where no client is monitoring a particular piece of data. In cases where multiple clients are simultaneously monitoring a piece of data, it's crucial to quickly determine whether the modified data is being monitored by a client to decide whether to push the data to the relevant client. Since each piece of data in a monitoring data list corresponds to one monitor unit, when multiple clients simultaneously request the same monitoring data, the monitoring stream unit will receive multiple monitoring data lists. Each monitoring data list contains the same requested monitoring data, and the monitoring stream unit will generate multiple monitor units to correspond with the simultaneously monitored monitoring data. This embodiment establishes a common monitor unit list for multiple monitor units corresponding to the same monitoring data.

[0072] Therefore, in this embodiment, the monitoring group also includes a management unit with a linear data structure, such as... Figure 3 As shown in the diagram, each leaf node of this segment tree corresponds to a list of monitor units as its value, and the key of the leaf node is the monitoring data corresponding to that list of monitor units.

[0073] By mounting range-based monitor units onto a segment tree, when data is modified, it's only necessary to check if the data falls within a specific range of the segment tree to retrieve the list of monitor units for that range and push the event. Its advantage lies in the fact that when data is modified, simply querying the monitoring group to see if the data's keyword exists reveals whether any monitor units are listening and locates the leaf node. Since the leaf node value represents the list of monitor units, if the data exists in the monitoring group, the corresponding monitor unit list can be retrieved immediately based on the keyword, and events can be pushed to each unit individually. This eliminates the need to traverse all monitor units, compare data one by one, and then push the event, significantly improving efficiency. Using a segment tree provides both single-data monitoring and range-based data monitoring, making client-side collaboration more complete.

[0074] Furthermore, the multi-version data storage unit includes a globally incrementing version number. This globally incrementing version number is used to generate a new data version for data that has been modified (added, deleted, etc.), and stores this new data version without deleting or overwriting the old version, thus preserving a complete record of data modification operations. The multi-version data storage unit also supports default read operations for the latest version of data and specified read operations for any version of data. When reading data, it will default to returning the latest data version, but a specific data version can also be specified for querying.

[0075] By using multi-version data storage, the current data version information is returned when the client queries the data, and then the version just queried is incremented by one when data monitoring is performed. In this way, even if the query and data monitoring are not atomic operations, all data modification records can be obtained.

[0076] The following Example 2 is a large-scale concurrent data monitoring method corresponding to the large-scale concurrent data monitoring component provided in Example 1.

[0077] Example 2

[0078] This embodiment provides a method for large-scale concurrent data monitoring, such as... Figure 3 As shown, it includes the following steps:

[0079] Receive a list of listening data sent by the client;

[0080] Store each piece of monitored data in the monitored data list;

[0081] Receive a data modification request, perform corresponding data modification operations according to the data modification request, and store the original version of the modified data and the modified version corresponding to the original version.

[0082] For each piece of monitored data, the monitored data is matched with the modified data. If the match is successful, the modification information of the modified data is pushed to the client.

[0083] Furthermore,

[0084] Before matching the monitored data with the modified data, the process also includes: establishing a data matching model based on a segment tree;

[0085] The establishment of the segment tree-based data matching model includes:

[0086] Obtain the basic data structure of the segment tree;

[0087] For each piece of monitored data, a corresponding monitor unit object is constructed. The monitor unit object is used to store the monitored data and also to match the monitored data with the modified data. If the match is successful, the modification information of the modified data is pushed to the client.

[0088] For each leaf node of the segment tree, perform the following steps:

[0089] Multiple monitor units corresponding to the same monitored data are merged into a monitor unit list, and the monitor unit list is used as the value of the leaf node; the monitoring data corresponding to the monitor unit list is used as the key of the leaf node.

[0090] The following embodiment 3 is a cluster status monitoring system that includes the large-scale concurrent data monitoring component given in embodiment 1. It can monitor the real-time operating status of nodes and ensure that the entire cluster works normally.

[0091] Example 3

[0092] In distributed systems, it is crucial to monitor whether all nodes are running normally and to obtain information about their operational status (CPU utilization, memory usage, etc.). Being able to detect abnormal nodes more quickly allows for adjustments to other nodes as early as possible to ensure the normal operation of the entire system.

[0093] To address this, this embodiment provides a cluster status monitoring system, including:

[0094] A monitoring center, a large-scale concurrent data monitoring component as described in Example 1, and multiple nodes;

[0095] The monitoring center is connected to the large-scale concurrent data listening component through a data transmission channel. The data transmission channel is used to transmit the list of monitorable data sent by the monitoring center to the large-scale concurrent data listening component, and also to transmit the real-time running status of the nodes fed back by the large-scale concurrent data listening component to the monitoring center.

[0096] Multiple nodes are connected to the large-scale concurrent data monitoring component via a data bus. The data bus is used to write the real-time running status of the nodes, which is continuously output by the multiple nodes, into the large-scale concurrent data monitoring component.

[0097] The monitoring center is used to adjust abnormal nodes based on the real-time operating status of nodes reported by the large-scale concurrent data monitoring component.

[0098] In addition, the large-scale concurrent data monitoring component in this embodiment also includes a node liveness status monitoring module, which is used to generate corresponding single-point failure messages for nodes that have not sent real-time running status multiple times in a row, and to feed back the single-point failure messages to the monitoring center.

[0099] As a further description of the cluster status monitoring system in this embodiment, such as Figure 4 As shown.

[0100] In a distributed system, each node has a unique ID. For example, consider nodes A, B, C, D, and E. A common prefix is ​​added to each node's ID and written to the consistency component. For instance, adding the prefix ` / node / ` means that the node with ID A corresponds to the data ` / node / A` in the consistency component. Then, node status information such as CPU utilization and memory usage is written to the consistency component along with the node ID. This way, the cluster monitoring center only needs to listen for data with the prefix ` / node / ` from the consistency component to obtain the running status of each node in the cluster. Even if a new node joins, it can be detected promptly.

[0101] Furthermore, it is agreed that each node will write its own information to the consistency component at regular intervals, such as 1 second. The monitoring center can then use this information to determine whether a node is alive. If, after 3 seconds (three reporting intervals), no data modification is detected for a particular node, then that node can be considered a single point of failure. In this way, the monitoring center can dynamically adjust cluster resources in a timely manner based on the running and failure status of all nodes to maintain the normal operation of the entire cluster.

[0102] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A large-scale concurrent data monitoring component, characterized in that, include: The monitoring stream unit is used to receive a list of monitored data sent by the client, assign a corresponding monitor unit to each monitored data in the list, and push modification information of data in the multi-version data storage unit to the client; one monitoring stream unit corresponds to one client. The monitor unit is used to store corresponding listening data, monitor the modification status of data in the multi-version data storage unit, and match the modified data with the stored listening data; the monitor unit and the monitoring stream unit are connected through a message queue; the message queue is used to send the modification information of the successfully matched data to the monitoring stream unit; A multi-version data storage unit is used to provide a data modification channel and to store the original version of the modified data and the modified version corresponding to the original version. The multi-version data storage unit includes a globally incrementing version number, which is used to generate a new data version for the modified data and to store the new data version. A monitoring group is used to manage monitoring units; one monitoring group corresponds to multiple monitoring units. The monitoring group includes a management unit with a segment tree data structure; each leaf node of the segment tree data structure corresponds to a list of monitoring units; the list of monitoring units includes multiple monitoring units corresponding to the same monitored data; the value of each leaf node is the list of monitoring units, and the key of each leaf node is the monitored data corresponding to the list of monitoring units.

2. The large-scale concurrent data monitoring component according to claim 1, characterized in that, The multi-version data storage unit supports default read operations for the latest version of data and specified read operations for any version of data.

3. The large-scale concurrent data monitoring component according to claim 1, characterized in that, The monitoring stream unit is also used to receive listening requests and basic client information sent by the client; The monitoring stream unit is connected to the client via a data channel; the data channel is used to send the modification information received by the monitoring stream unit to the corresponding client. The monitor unit is also used to store the client's basic information and send the modification information to the monitoring stream unit through the message queue based on the client's basic information.

4. A method for large-scale concurrent data monitoring, characterized in that, Applied to a large-scale concurrent data monitoring component as described in any one of claims 1-3, the method includes the following steps: Receive a list of listening data sent by the client; Store each piece of monitored data in the monitored data list; Receive a data modification request, perform corresponding data modification operations according to the data modification request, and store the original version of the modified data and the modified version corresponding to the original version; For each piece of monitored data, the monitored data is matched with the modified data. If the match is successful, the modification information of the modified data is pushed to the client.

5. A method for large-scale concurrent data monitoring according to claim 4, characterized in that, Before matching the monitored data with the modified data, the process also includes: establishing a data matching model based on a segment tree; The establishment of the segment tree-based data matching model includes: Obtain the basic data structure of the segment tree; For each piece of monitored data, a corresponding monitor unit object is constructed. The monitor unit object is used to store the monitored data and also to match the monitored data with the modified data. If the match is successful, the modification information of the modified data is pushed to the client. For each leaf node of the segment tree, perform the following steps: Multiple monitor units corresponding to the same monitored data are merged into a monitor unit list, and the monitor unit list is used as the value of the leaf node; the monitoring data corresponding to the monitor unit list is used as the key of the leaf node.

6. A cluster status monitoring system, characterized in that, include: The monitoring center, the large-scale concurrent data monitoring component as described in any one of claims 1-3, and multiple nodes; The monitoring center is connected to the large-scale concurrent data monitoring component through a data transmission channel. The data transmission channel is used to transmit the monitoring requests and monitoring data list sent by the monitoring center to the large-scale concurrent data monitoring component, and also to transmit the real-time running status of the nodes fed back by the large-scale concurrent data monitoring component to the monitoring center. The multiple nodes are connected to the large-scale concurrent data monitoring component via a data bus, which is used to write the real-time running status of the nodes continuously output by the multiple nodes into the large-scale concurrent data monitoring component. The monitoring center is used to adjust abnormal nodes based on the real-time operating status of the nodes fed back by the large-scale concurrent data monitoring component.

7. A cluster status monitoring system according to claim 6, characterized in that, The large-scale concurrent data monitoring component also includes a node liveness status monitoring module, which generates a corresponding single-point failure message for nodes that fail to send real-time running status multiple times in a row, and feeds the single-point failure message back to the monitoring center.

Images