A network anomaly detection system for distributed storage systems

Abstract

The present invention discloses a network anomaly detection system for a distributed storage system, which belongs to the field of network detection technology. The technical problem to be solved is how to detect and record abnormal network connections and their detailed information. It includes: a network connection management module, which is used to identify abnormal connections of each component's network connections and collect abnormal information corresponding to each abnormal connection; an asynchronous message management module, which is used to store abnormal information, provide an abnormal connection management interface to the outside world, and provide abnormal information to the outside world; a component Rank management module, which is used to start a scheduled task, regularly update the health data and status information of each component, and call the asynchronous message management module to update the abnormal information to the corresponding health data; a beacon management module, which periodically reports health data through a beacon sending thread; and a summary display module, which is used to save and display the monitoring data of each component in the form of status information.

Description

Technical Field

[0001] The present invention relates to the technical field of network detection, in particular to a network anomaly detection system for a distributed storage system. Background Art

[0002] Ceph is a unified, distributed file system designed for superior performance, reliability, and scalability. It uniquely provides object, block, and file storage capabilities within a unified system. After years of development, it has gained support from numerous cloud computing vendors and is widely adopted. A Ceph cluster consists of various components, including monitors, OSDs, manager growth managers, and managed services (MDSs).

[0003] Components within the cluster frequently connect to and exchange data with each other, all managed by a message manager. When network connections between components experience anomalies, preventing normal communication, the storage cluster may be unable to provide normal external services. Operations and maintenance personnel are required to quickly locate and address the issue and restore service promptly. However, there is currently no detection or display for these abnormal connections, making it difficult to locate network communication issues. Furthermore, detailed information about network anomalies, such as the connection type, IP addresses, port numbers, and connection status, is not provided to operations and maintenance personnel, resulting in lengthy fault location and repair times.

[0004] How to detect and record abnormal network connections and their detailed information is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a network anomaly detection system for a distributed storage system to solve the technical problem of how to detect and record abnormal network connections and their detailed information.

[0006] In a first aspect, the present invention provides a network anomaly detection system for a distributed storage system, configured to identify anomalies, collect anomaly information, and display anomaly information about network connections between components in the distributed storage system through a scheduled task mechanism. The network anomaly detection system comprises:

[0007] A network connection management module, which is used to manage the lifecycle of network connections between components in the storage cluster, identify abnormal connections of each component's network connection, and collect abnormal information corresponding to each abnormal connection;

[0008] An asynchronous message management module, which interacts with the network connection management module and is used to manage each network connection in each component through an asynchronous message manager, store exception information, provide an abnormal connection management interface to the outside world, and provide exception information to the outside world;

[0009] A component rank management module, which interacts with the asynchronous message management module to start scheduled tasks, regularly update the health data and status information of each component, and call the asynchronous message management module to update abnormal information to the corresponding health data;

[0010] A beacon management module, which is used to maintain the health information of each component, start a beacon sending thread, and periodically report health data through the beacon sending thread;

[0011] A summary display module interacts with the beacon management module to save and display the monitoring data of each component in the form of status information.

[0012] Preferably, the abnormal connection includes two types, namely:

[0013] Network layer connection anomaly, where an anomaly or timeout occurs during the process of establishing a socket connection;

[0014] The application protocol layer connection is abnormal. The application protocol layer connection is abnormal when the network layer connection is successful, but an abnormality or timeout occurs in various stages of the application protocol layer processing logic.

[0015] Preferably, the network connection management module is used to save the identified abnormal information to the asynchronous message management module and delete expired abnormal connection information.

[0016] Preferably, the network connection management module is used to identify abnormal connections of each component's network connection and collect abnormal information corresponding to each abnormal connection based on the following process:

[0017] S1. For each component, determine whether the network connection between the current component and the MON component is normal. If it is an abnormal connection, the component is identified as an abnormal component and step S4 is executed. If it is a normal connection, the next step is executed. For abnormal components, the identification rule is: if the current component does not interact with the MON component within a predetermined time interval, the current component is identified as an abnormal component. The time interval is determined by the component configuration item corresponding to the current component.

[0018] S2. For the two components to establish a connection, if a connection error is directly returned after the connection request is initiated, it is determined that the connection is abnormal, the abnormal information is recorded, and the network connection is terminated. If no error is directly returned, the next step is executed;

[0019] S3. Start a scheduled task, where the time interval corresponding to the scheduled task is the connection timeout time in the configuration item corresponding to the current component;

[0020] S4. If the connection is successfully established within the timeout period, the original scheduled task is deleted, and a scheduled task for counting idle time is re-registered. The current network connection is checked to see if it is in the saved abnormal connection. If so, the abnormal connection is deleted.

[0021] If the connection is not established successfully within the timeout period, the error type is marked as timeout. The type of abnormal connection is determined based on the current connection stage and the next step is executed.

[0022] S4. Record the abnormal information corresponding to the current abnormal connection;

[0023] S5. For abnormal connections, the corresponding abnormal information is saved to the asynchronous message management module;

[0024] S6. If the network connection is cancelled due to long-term idleness or other reasons, detect whether the current connection is in the saved abnormal connections. If so, delete the abnormal connection.

[0025] Preferably, in step S2, the abnormal information includes the error type, the type of the local and peer end in the current network connection, the IP and port number of the peer end, and the state of the connection, and the type of the local and peer end in the current network connection includes OSD, MDS, and MGR;

[0026] In step S4, the abnormal information includes the type of the local and peer end in the current network connection, the IP and port number of the peer end, and the state of the connection. The type of the local and peer end in the current network connection includes OSD, MDS, and MGR.

[0027] Preferably, in step S4, the type of abnormal connection is determined. If an error or timeout occurs before the socket connection is successful, it is a network layer connection abnormality; otherwise, it is an application protocol layer connection abnormality.

[0028] Preferably, the asynchronous message management module is used to provide an abnormal connection management interface to the outside world, and the management interface includes:

[0029] Add interface, the add interface is used to add a new abnormal connection, which is provided to the network connection management module for use when a new abnormal connection is discovered;

[0030] Delete interface, the delete interface is used to delete an abnormal connection, and is used to find whether there is a corresponding abnormal connection. If there is no corresponding abnormal connection, it is directly returned. If there is, the abnormal connection is deleted. The delete interface is provided to the network connection management module and is used when the connection is successfully established or the connection is cancelled;

[0031] The search interface is used to obtain all known abnormal connections and provide them to the component Rank management module. In the Rank management module, all current abnormal connections can be obtained and provided to the beacon management module for use.

[0032] Preferably, the component Rank management module periodically updates the health data and status information of the current component, and the corresponding update cycle is determined by the configuration items of the current component.

[0033] Preferably, the beacon management module is used to periodically report the health information of each component to the summary display module through the following process:

[0034] Add a cluster internal health indicator type related to abnormal connections, and add the abnormal information saved when an abnormal connection is found to the prompt information of the newly created health indicator type;

[0035] If there is an abnormal connection, the Rank Management Module has updated the abnormal connection health information saved by the Beacon Management Module to the latest value through the scheduled task;

[0036] For each abnormal connection saved by the beacon management module, a new abnormal connection health indicator object is created, added to the beacon, and waits for the beacon sending thread to send it to the summary display module;

[0037] The beacon sending thread periodically sends beacons to the summary display module, and the period is determined by the configuration item of the corresponding component.

[0038] Preferably, the network anomaly detection system is configured in the message manager to assist the message manager in managing the network connections of various components in the distributed storage system, and to identify anomalies, collect anomaly information, and display anomaly information on the network connections between the components.

[0039] The network anomaly detection system of a distributed storage system of the present invention has the following advantages:

[0040] 1. Identify abnormal network connections between components within a storage cluster, save and display abnormal information corresponding to abnormal connections, improve the monitoring capabilities of distributed storage system network problems, and enable operation and maintenance personnel to quickly locate related logical problems or various configuration issues;

[0041] 2. It can identify two different types of abnormal connections: network layer connection abnormalities and application protocol layer abnormal connections, achieving accurate network anomaly detection, facilitating accurate understanding of abnormal information, and allowing operation and maintenance personnel to quickly locate related logical problems or various configuration problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] The present invention will be further described below with reference to the accompanying drawings.

[0044] Figure 1 A block diagram of the working principle of a network anomaly detection system for a distributed storage system according to an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of a process for a network connection management module to discover and delete abnormal connections in a network anomaly detection system of a distributed storage system according to an embodiment. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0047] An embodiment of the present invention provides a network anomaly detection system for a distributed storage system, which is used to solve technical problems.

[0048] Example:

[0049] The existing technology does not detect and record abnormal network connections and their detailed information, such as the type, name, IP, port number of the two ends of the connection, the status of the connection, etc., resulting in inefficient network anomaly positioning and processing. The network anomaly detection system of a distributed storage system of the present invention identifies anomalies in the network connections between components in the storage cluster, collects anomaly information, and displays anomaly information to improve the monitoring capability of the distributed storage system network problems, facilitate operation and maintenance personnel to analyze and locate cluster network problems, and effectively reduce the fault repair time of network problems.

[0050] The present invention provides a network anomaly detection system for a distributed storage system, comprising a network connection management module, an asynchronous message management module, a component Rank management module, a beacon management module and a summary display module.

[0051] The network connection management module is responsible for the lifecycle management of each network connection within the storage cluster, including establishing connections at the network layer and application protocol layer, and maintaining network communications. For abnormal connection detection, the network connection management module is responsible for identifying and collecting information about abnormal connections, saving the identified abnormal connection information to the asynchronous message management module, and deleting expired abnormal connection information. The criteria for determining a network connection abnormality are: an error is returned during the network connection establishment process, or the network connection is not completed within a specified time (network timeout). The network connection timeout is determined by the configuration item and defaults to 10 seconds.

[0052] There are two types of exceptions that need to be identified: network layer connection exceptions and application protocol layer connection exceptions. Network layer connection exceptions occur when an exception or timeout occurs during the socket connection establishment process. Application protocol layer connection exceptions occur when the network layer connection is successful but an exception or timeout occurs at various stages of the application protocol layer processing logic.

[0053] In a specific embodiment, the network connection management module is used to identify abnormal connections of each component's network connection and collect abnormal information corresponding to each abnormal connection based on the following process:

[0054] S1. For each component, determine whether the network connection between the current component and the MON component is normal. If it is an abnormal connection, the component is identified as an abnormal component and step S4 is executed. If it is a normal connection, the next step is executed. For abnormal components, the identification rule is: if the current component does not interact with the MON component within a predetermined time interval, the current component is identified as an abnormal component. The time interval is determined by the component configuration item corresponding to the current component.

[0055] S2. For the two components to establish a connection, if a connection error is directly returned after the connection request is initiated, it is determined that the connection is abnormal, the abnormal information is recorded, and the network connection is terminated. If no error is directly returned, the next step is executed;

[0056] S3. Start a scheduled task, where the time interval corresponding to the scheduled task is the connection timeout time in the configuration item corresponding to the current component;

[0057] S4. If the connection is successfully established within the timeout period, the original scheduled task is deleted, and a scheduled task for counting idle time is re-registered. The current network connection is checked to see if it is in the saved abnormal connection. If so, the abnormal connection is deleted.

[0058] If the connection is not established successfully within the timeout period, the error type is marked as timeout. The type of abnormal connection is determined based on the current connection stage and the next step is executed.

[0059] S4. Record the abnormal information corresponding to the current abnormal connection;

[0060] S5. For abnormal connections, the corresponding abnormal information is saved to the asynchronous message management module;

[0061] S6. If the network connection is cancelled due to long-term idleness or other reasons, detect whether the current connection is in the saved abnormal connections. If so, delete the abnormal connection.

[0062] In step S2, the abnormal information includes the error type, the type of the local and peer end in the current network connection, the IP and port number of the peer end, and the state of the connection. The type of the local and peer end in the current network connection includes OSD, MDS, and MGR.

[0063] In step S4, the abnormal information includes the type of the local and peer end in the current network connection, the IP and port number of the peer end, and the state of the connection. The types of the local and peer end in the current network connection include OSD, MDS, and MGR.

[0064] In step S4, the type of abnormal connection is determined. If an error or timeout occurs before the socket connection is successful, it is a network layer connection abnormality; otherwise, it is an application protocol layer connection abnormality.

[0065] Based on the above functions of the network connection management module, in actual work, the network connection management module identifies abnormal connections and obtains abnormal information through the following process:

[0066] (1) First, you need to determine whether the connection between the current component and the MON component is normal. If not, the cluster will identify the component as abnormal because it has not interacted with the MON component for a period of time. The time interval for identifying abnormalities is determined by the configuration items of each component, and the default configuration is 15 seconds. Otherwise, the current component is operating normally, and proceed to the next step;

[0067] (2) Client requests or internal requests require establishing a new connection. Currently, storage clusters use asynchronous connections by default.

[0068] (3) If a connection error is returned directly after the request is initiated, record the error type, the type of the local and peer end in the connection (such as OSD, MDS, MGR, etc.), the peer end IP and port number, and the status of the connection, and exit the network connection; if no error is returned directly, perform subsequent processing;

[0069] (4) After establishing a connection and initiating a request, start the scheduled task, and the time interval is the connection timeout time in the configuration item;

[0070] (5) If the connection is successfully established within the timeout period, delete the original scheduled task and re-register the scheduled task for counting idle time. Also, check whether the current connection is among the saved abnormal connections. If so, delete the abnormal connection.

[0071] (6) Otherwise, if the connection is not successfully established within the timeout period, the error type is marked as timeout. Depending on the current connection stage, two types of anomalies are detected: network layer connection anomalies and application protocol layer connection anomalies. If an error or timeout occurs before the socket connection is successfully established, it is a network layer connection anomaly; otherwise, it is an application protocol layer connection anomaly.

[0072] (7) Record the local and peer types (such as OSD, MDS, MGR, etc.), peer IP and port number, and the status of the connection.

[0073] (8) If there is an abnormal connection, save it to the asynchronous message management module.

[0074] (9) If the connection is cancelled due to long-term idleness or other reasons, check whether the current connection is in the saved abnormal connections. If so, delete the abnormal connection.

[0075] The asynchronous message management module is responsible for storing network communications within the cluster, primarily using the asynchronous message manager to manage the various network connections of the current component. The message manager is the central engine for Ceph cluster network communications, including network connections, sending messages, receiving data, message conversion, and message distribution. For abnormal connection detection, the message management module is responsible for storing abnormal connections, providing external interfaces such as adding, deleting, and querying abnormal connections, and providing abnormal connection data to upper-level message manager users. The external interfaces provided by this module include:

[0076] Add a new abnormal connection. This is used by the network connection management module when it discovers a new abnormal connection.

[0077] Delete an abnormal connection. Now check if there is a corresponding connection. If there is no corresponding connection, return it directly. If there is, delete the connection. It is also provided to the network connection management module for use when the connection is successfully established or the connection is cancelled.

[0078] Get all known abnormal connections and provide them to the Rank management module. In the Rank management module, you can get all the current abnormal connections and provide them to the beacon.

[0079] The Component Rank Management Module initiates a scheduled task to periodically update the health data and status information of the current component, which is then sent by the Beacon Management Module to the Summary Display Module. The update interval is determined by the configuration item and defaults to 5 seconds. To detect abnormal connections, the Component Rank Management Module invokes the API provided by the Asynchronous Message Management Module to retrieve all known abnormal connections within the scheduled task and updates the abnormal connection data to the health data.

[0080] The beacon management module maintains the health information of this component. In addition, it will start a beacon sending thread to periodically report the current health information to the summary display module. The main processing logic of the beacon management module is as follows:

[0081] Add a new cluster internal health indicator type related to abnormal connections. The error type saved when an abnormal connection is found, the local and peer types of the connection (such as OSD, MDS, MGR, etc.), the peer IP and port number, and the status of the connection are added to the prompt information of the newly created health indicator type;

[0082] Before this, if there is an abnormal connection, the Rank Management Module has updated the abnormal connection health information saved by the Beacon Management Module to the latest value through the scheduled task;

[0083] For each abnormal connection saved by the beacon management module, a new abnormal connection health indicator object is created, added to the beacon, and waits for the beacon sending thread to send it to the summary display module.

[0084] The beacon sending thread periodically sends beacons to the summary display module. This period is determined by the configuration item and defaults to 5 seconds.

[0085] The summary display module stores and displays beacons from other components. Users can use commands or interfaces to obtain abnormal connection information from the summary display module. They can view the specific peer information and error type of the abnormal connection, allowing for quick location and resolution.

[0086] The network anomaly detection system of this embodiment uses scheduled tasks to identify abnormal connections. Connections can be divided into two types: network layer connections and application protocol layer connections. Abnormal connection information includes the peer component type, name, IP address, and port number. This invention identifies, stores, and displays abnormal connections, improving the monitoring capabilities of distributed storage system network issues. This facilitates operations and maintenance personnel to analyze and locate cluster network problems, effectively reducing network troubleshooting time and improving system service quality.

[0087] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A network anomaly detection system for a distributed storage system, characterized in that The network anomaly detection system is used to identify anomalies, collect anomaly information, and display anomaly information on the network connections between components in the distributed storage system through a scheduled task mechanism. The network anomaly detection system includes: A network connection management module, which is used to manage the lifecycle of network connections between components in the storage cluster, identify abnormal connections of each component's network connection, and collect abnormal information corresponding to each abnormal connection; An asynchronous message management module, which interacts with the network connection management module and is used to manage each network connection in each component through an asynchronous message manager, store exception information, provide an abnormal connection management interface to the outside world, and provide exception information to the outside world; A component rank management module, which interacts with the asynchronous message management module to start scheduled tasks, regularly update the health data and status information of each component, and call the asynchronous message management module to update abnormal information to the corresponding health data; A beacon management module, which is used to maintain the health information of each component, start a beacon sending thread, and periodically report health data through the beacon sending thread; A summary display module, which interacts with the beacon management module and is used to save and display the monitoring data of each component as status information; The network connection management module is used to identify abnormal connections of each component's network connection and collect abnormal information corresponding to each abnormal connection based on the following process: S1. For each component, determine whether the network connection between the current component and the MON component is normal. If it is an abnormal connection, the component is identified as an abnormal component and step S4 is executed. If it is a normal connection, the next step is executed. For abnormal components, the identification rule is: if the current component does not interact with the MON component within a predetermined time interval, the current component is identified as an abnormal component. The time interval is determined by the component configuration item corresponding to the current component. S2. For the two components to establish a connection, if a connection error is directly returned after the connection request is initiated, it is determined that the connection is abnormal, the abnormal information is recorded, and the network connection is terminated. If no error is directly returned, the next step is executed; S3. Start a scheduled task, where the time interval corresponding to the scheduled task is the connection timeout time in the configuration item corresponding to the current component; S4. If the connection is successfully established within the timeout period, the original scheduled task is deleted, and a scheduled task for counting idle time is re-registered. The current network connection is checked to see if it is in the saved abnormal connection. If so, the abnormal connection is deleted. If the connection is not established successfully within the timeout period, the error type is marked as timeout. The type of abnormal connection is determined based on the current connection stage and the next step is executed. S4. Record the abnormal information corresponding to the current abnormal connection; S5. For abnormal connections, the corresponding abnormal information is saved to the asynchronous message management module; S6. If the network connection is cancelled due to long-term idleness or other reasons, detect whether the current connection is in the saved abnormal connections. If so, delete the abnormal connection.

2. A network anomaly detection system for a distributed storage system according to claim 1, characterized in that The abnormal connections include two types: Network layer connection anomaly, where an anomaly or timeout occurs during the process of establishing a socket connection; The application protocol layer connection is abnormal. The application protocol layer connection is abnormal when the network layer connection is successful, but an abnormality or timeout occurs in various stages of the application protocol layer processing logic.

3. The network anomaly detection system of a distributed storage system according to claim 1, characterized in that The network connection management module is used to save the identified abnormal information to the asynchronous message management module and delete expired abnormal connection information.

4. A network anomaly detection system for a distributed storage system according to claim 1, characterized in that In step S2, the abnormal information includes the error type, the type of the local and peer end in the current network connection, the IP and port number of the peer end, and the state of the connection. The type of the local and peer end in the current network connection includes OSD, MDS, and MGR. In step S4, the abnormal information includes the type of the local and peer end in the current network connection, the IP and port number of the peer end, and the state of the connection. The type of the local and peer end in the current network connection includes OSD, MDS, and MGR.

5. The network anomaly detection system of a distributed storage system according to claim 1, characterized in that In step S4, the type of abnormal connection is determined. If an error or timeout occurs before the socket connection is successful, it is a network layer connection abnormality; otherwise, it is an application protocol layer connection abnormality.

6. A network anomaly detection system for a distributed storage system according to claim 1, characterized in that The asynchronous message management module is used to provide an abnormal connection management interface to the outside world. The management interface includes: Add interface, the add interface is used to add a new abnormal connection, which is provided to the network connection management module for use when a new abnormal connection is discovered; Delete interface, the delete interface is used to delete an abnormal connection, and is used to find whether there is a corresponding abnormal connection. If there is no corresponding abnormal connection, it is directly returned. If there is, the abnormal connection is deleted. The delete interface is provided to the network connection management module and is used when the connection is successfully established or the connection is cancelled; The search interface is used to obtain all known abnormal connections and provide them to the component Rank management module. In the Rank management module, all current abnormal connections can be obtained and provided to the beacon management module for use.

7. A distributed storage system network anomaly detection system according to claim 1, characterized in that The component Rank management module regularly updates the health data and status information of the current component, and the corresponding update cycle is determined by the configuration items of the current component.

8. The network anomaly detection system for a distributed storage system according to claim 1, characterized in that The beacon management module is used to periodically report the health information of each component to the summary display module through the following process: Add a cluster internal health indicator type related to abnormal connections, and add the abnormal information saved when an abnormal connection is found to the prompt information of the newly created health indicator type; If there is an abnormal connection, the Rank Management Module has updated the abnormal connection health information saved by the Beacon Management Module to the latest value through the scheduled task; For each abnormal connection saved by the beacon management module, a new abnormal connection health indicator object is created, added to the beacon, and waits for the beacon sending thread to send it to the summary display module; The beacon sending thread periodically sends beacons to the summary display module, and the period is determined by the configuration item of the corresponding component.

9. A network anomaly detection system for a distributed storage system according to any one of claims 1, 2, 3, 5, 6, 7 or 8, characterized in that The network anomaly detection system is configured in the message manager to assist the message manager in managing the network connections of various components in the distributed storage system, and to identify anomalies, collect anomaly information, and display anomaly information on the network connections between the components.

Images