File system performance monitoring system

Abstract

The invention discloses a file system performance monitoring system, which is characterized in that a file system is provided with a web interface, an agent agent program and a database at a user side, and the file system is provided with a monitoring module and a POSIX interface at a kernel side; the web interface is configured as follows: the web interface is used for displaying an overall performance monitoring condition of the file system and file system related data of an application needing to be monitored; the agent agent program is configured as follows: the agent agent program communicates with a web interface through an http protocol, receives parameters issued by the web interface and application names needing to be monitored, and establishes connection with the monitoring module in the kernel through a socket; the database is configured to store monitoring data sent by the monitoring module and received by the agent agent program, and when a user executes a query operation on the monitoring data on a web interface, the database executes a corresponding query statement and returns a result to the agent agent program; the monitoring module is configured to comprise a transmission module, a management module and a statistics module.

Description

Technical Field

[0001] This invention relates to the field of file systems, and more specifically, to a file system performance monitoring system. Background Technology

[0002] With the advent of the digital age and the continuous development of artificial intelligence and big data technologies, the level of informatization in all industries is becoming increasingly profound, and massive amounts of data are constantly being generated. More and more industry applications are performing various analyses on the data, such as structured analysis, AI analysis, and graph computing analysis. The performance of storage greatly affects the efficiency of various applications. Therefore, it is necessary to monitor the file system performance in real time, issue corresponding warnings when the file system bandwidth is insufficient, and monitor the IO status of each application. When some applications consume too much system IO resources, corresponding maintenance work should be carried out to facilitate timely troubleshooting and ensure the stable operation of the entire system. Summary of the Invention

[0003] This invention provides a file system performance monitoring system to solve the technical problems existing in the prior art.

[0004] To achieve the above objectives, this invention provides a file system performance monitoring system, characterized in that the file system performance monitoring system has a web interface, an agent program, and a database on the user end, and a monitoring module and a POSIX interface on the kernel end.

[0005] The web interface is configured to display the overall performance monitoring status of the file system and the file system-related data of the applications to be monitored. The web interface sends the corresponding monitoring parameters to the agent program via the HTTP protocol.

[0006] The agent program is configured as follows: The agent program communicates with the web interface via the HTTP protocol. The agent program receives parameters and the name of the application to be monitored from the web interface. It establishes a connection with the monitoring module in the kernel through a socket and sends the name of the application to be monitored and the monitoring parameters to the monitoring module in the kernel. At the same time, the agent program receives the monitoring data sent by the monitoring module and saves it to the database. When the user performs a query operation on the monitoring data in the web interface, the agent program returns the corresponding monitoring data to the web interface.

[0007] The database is configured to store monitoring data sent by the monitoring module and received by the agent program. When a user performs a query operation on the monitoring data in the web interface, the database executes the corresponding query statement and returns the result to the agent program.

[0008] The monitoring module is configured to execute corresponding monitoring, management, and transmission functions on the kernel side. The monitoring module includes a transmission module, a management module, and a statistics module. The transmission module uses netlink sockets to realize communication between kernel mode and user mode, and to parse and transmit control commands and monitoring data. The management module parses control commands, sets the status of the list of applications to be monitored, and sets the operation and sending interval of the file system to be monitored. The statistics module counts and calculates the monitoring data of file system access.

[0009] The POSIX interface is configured as follows: The POSIX interface is a file system interface. The POSIX interface provides file system read, write and file creation functions and records the corresponding call information, which is synchronized to the statistics module through global variables.

[0010] In one embodiment of the present invention, in the configuration process, the web interface sends out the list of applications to be monitored, the file system operation items to be monitored, and the statistical intervals, as follows:

[0011] S11: The web interface obtains the input monitoring parameters and sends them to the agent program via the HTTP protocol;

[0012] S12: The agent program receives the monitoring parameters sent by the web interface and sends the monitoring parameters to the kernel-side transmission module through the socket protocol;

[0013] S13: The transmission module communicates with the agent program via a netlink socket, receives monitoring parameters sent by the agent program, and parses the monitoring parameters;

[0014] S14: The transmission module parses the monitoring parameters, checks whether the name of the application to be monitored and the operation type of the file system are within the range of read, write, delete, open, and close. At the same time, the transmission module checks whether the reporting interval is reasonable. If the monitoring parameter parsing is invalid, it will end directly. If the monitoring parameter parsing is valid, it will enter the kernel management module.

[0015] S15: The management module sets the corresponding monitoring rules based on the results of the parsed parameters, including application matching rules, a list of file system operation types to be monitored, and reporting intervals. The statistics module performs data statistics based on the monitoring rules.

[0016] In one embodiment of the present invention, in the monitoring and statistics process, the kernel-side monitoring module performs monitoring and statistics according to the monitoring rules determined by the configuration process and sends the data to the agent program through the transmission module. The steps are as follows:

[0017] S21: On the user side, the application performs multiple file system operations through system calls such as open, read, write, and delete;

[0018] S22: On the kernel side, a system call is made to enter the kernel VFS. The kernel VFS then searches for the accessed directory and file, and the file system driver forwards the system call request to the corresponding file system.

[0019] S23: At the POSIX interface on the kernel side, different processing functions are transferred according to different file system operation requests, and the kernel side statistics module is entered in each processing function.

[0020] S24: The statistics module performs monitoring rule matching, specifically matching based on the application name of the current call. If the match is successful, the matching of the operation type continues; otherwise, the current statistics ends directly. When matching the operation type, it checks whether the current operation type is in the monitoring list of the application. If the match is successful, it proceeds to the monitoring data update step; otherwise, it ends directly. In the monitoring data update step, the application name, PID, corresponding operation type, and operation data size are recorded. Then, it proceeds to the sending interval detection step, checking whether the configured sending interval meets the sending conditions according to the previous configuration. If it does, it proceeds to the kernel-side transmission module; otherwise, it ends directly.

[0021] S24: The kernel-side transmission module receives the data to be sent from the statistics module and creates the corresponding sending message. It creates a socket message through netlink, writes the application information and data of each operation type into the message body, and sends it to the agent program on the user side through the netlink socket.

[0022] S25: The agent program creates a socket and binds to it, listens to the monitoring module, receives monitoring data messages sent by the monitoring module through the netlink socket, parses the messages, obtains application information and statistical information corresponding to each operation, puts them into a queue, and the worker thread asynchronously writes the messages to the database to complete the saving of monitoring information.

[0023] In one embodiment of the present invention, the monitoring information query process involves displaying monitoring information through a web interface, and the steps are as follows:

[0024] S31: The user selects the application name and file system operation type to be viewed in the web interface, and the web interface sends the query parameters to the agent program.

[0025] S32: The agent program generates a query SQL statement based on the query parameters and performs the query in the database. The agent program then returns the query results to the web front end.

[0026] S33: The web frontend renders charts based on the data returned by the agent program, displaying the query monitoring results.

[0027] In one embodiment of the present invention, in step S32, the query parameters are application name, operation type and / or recording time.

[0028] The file system performance monitoring system provided by this invention can monitor the file system performance in real time and configure relevant monitoring parameters. It can monitor the overall file system performance and the access of each application to the file system, realize the monitoring of overall system resource utilization and the storage usage of each application, and promptly provide corresponding early warnings, maintenance, or performance optimization. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is an architecture diagram of a file system performance monitoring system according to an embodiment of the present invention;

[0031] Figure 2 This is a flowchart of the configuration process according to an embodiment of the present invention;

[0032] Figure 3 This is a flowchart of a monitoring and statistics process according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Figure 1 This is an architecture diagram of a file system performance monitoring system according to an embodiment of the present invention, such as... Figure 1As shown, this invention provides a file system performance monitoring system. The file system performance monitoring system includes a web interface, an agent program, and a database on the user end, and a monitoring module and a POSIX interface on the kernel end.

[0035] The web interface is configured to display the overall performance monitoring status of the file system and the file system-related data of the applications to be monitored. The web interface sends the corresponding monitoring parameters to the agent program via the HTTP protocol.

[0036] The agent program is configured as follows: The agent program communicates with the web interface via the HTTP protocol. The agent program receives parameters and the name of the application to be monitored from the web interface. It establishes a connection with the monitoring module in the kernel through a socket and sends the name of the application to be monitored and the monitoring parameters to the monitoring module in the kernel. At the same time, the agent program receives the monitoring data sent by the monitoring module and saves it to the database. When the user performs a query operation on the monitoring data in the web interface, the agent program returns the corresponding monitoring data to the web interface.

[0037] The database is configured to store monitoring data sent by the monitoring module and received by the agent program. When a user performs a query operation on the monitoring data in the web interface, the database executes the corresponding query statement and returns the result to the agent program.

[0038] The monitoring module is configured to execute corresponding monitoring, management, and transmission functions on the kernel side. The monitoring module includes a transmission module, a management module, and a statistics module. The transmission module uses netlink sockets to realize communication between kernel mode and user mode, and to parse and transmit control commands and monitoring data. The management module parses control commands, sets the status of the list of applications to be monitored, and sets the operation and sending interval of the file system to be monitored. The statistics module counts and calculates the monitoring data of file system access.

[0039] The POSIX interface is configured as follows: The POSIX interface is a file system interface. The POSIX interface provides file system read, write and file creation functions and records the corresponding call information, which is synchronized to the statistics module through global variables.

[0040] Figure 2 This is a flowchart of the configuration process according to an embodiment of the present invention, such as... Figure 2 As shown, in the configuration process, the web interface distributes the list of applications to be monitored, the file system operations to be monitored, and the statistical intervals. The steps are as follows:

[0041] S11: The web interface obtains the input monitoring parameters and sends them to the agent program via the HTTP protocol;

[0042] S12: The agent program receives the monitoring parameters sent by the web interface and sends the monitoring parameters to the kernel-side transmission module through the socket protocol;

[0043] S13: The transmission module communicates with the agent program via a netlink socket, receives monitoring parameters sent by the agent program, and parses the monitoring parameters;

[0044] S14: The transmission module parses the monitoring parameters, checks whether the name of the application to be monitored and the operation type of the file system are within the range of read, write, delete, open, and close. At the same time, the transmission module checks whether the reporting interval is reasonable. If the monitoring parameter parsing is invalid, it will end directly. If the monitoring parameter parsing is valid, it will enter the kernel management module.

[0045] S15: The management module sets the corresponding monitoring rules based on the results of the parsed parameters, including application matching rules, a list of file system operation types to be monitored, and reporting intervals. The statistics module performs data statistics based on the monitoring rules.

[0046] Figure 3 This is a flowchart of a monitoring and statistics process according to an embodiment of the present invention, such as... Figure 3 As shown, in the monitoring and statistics process, the kernel-side monitoring module performs monitoring and statistics according to the monitoring rules determined by the configuration process and sends the data to the agent program through the transmission module. The steps are as follows:

[0047] S21: On the user side, the application performs multiple file system operations through system calls such as open, read, write, and delete;

[0048] S22: On the kernel side, a system call is made to enter the kernel VFS. The kernel VFS then searches for the accessed directory and file, and the file system driver forwards the system call request to the corresponding file system.

[0049] S23: At the POSIX interface on the kernel side, different processing functions are transferred according to different file system operation requests, and the kernel side statistics module is entered in each processing function.

[0050] S24: The statistics module performs monitoring rule matching, specifically matching based on the application name of the current call. If the match is successful, the matching of the operation type continues; otherwise, the current statistics ends directly. When matching the operation type, it checks whether the current operation type is in the monitoring list of the application. If the match is successful, it proceeds to the monitoring data update step; otherwise, it ends directly. In the monitoring data update step, the application name, PID, corresponding operation type, and operation data size are recorded. Then, it proceeds to the sending interval detection step, checking whether the configured sending interval meets the sending conditions according to the previous configuration. If it does, it proceeds to the kernel-side transmission module; otherwise, it ends directly.

[0051] S24: The kernel-side transmission module receives the data to be sent from the statistics module and creates the corresponding sending message. It creates a socket message through netlink, writes the application information and data of each operation type into the message body, and sends it to the agent program on the user side through the netlink socket.

[0052] S25: The agent program creates a socket and binds to it, listens to the monitoring module, receives monitoring data messages sent by the monitoring module through the netlink socket, parses the messages, obtains application information and statistical information corresponding to each operation, puts them into a queue, and the worker thread asynchronously writes the messages to the database to complete the saving of monitoring information.

[0053] In one embodiment of the present invention, the monitoring information query process displays the monitoring information through a web interface, and the steps are as follows:

[0054] S31: The user selects the application name and file system operation type to be viewed in the web interface, and the web interface sends the query parameters to the agent program.

[0055] S32: The agent program generates a query SQL statement based on the query parameters and performs the query in the database. The agent program then returns the query results to the web front end.

[0056] S33: The web frontend renders charts based on the data returned by the agent program, displaying the query monitoring results.

[0057] In one embodiment of the present invention, in step S32, the query parameters are application name, operation type and / or recording time.

[0058] The file system performance monitoring system provided by this invention can monitor the file system performance in real time and configure relevant monitoring parameters. It can monitor the overall file system performance and the access of each application to the file system, realize the monitoring of overall system resource utilization and the storage usage of each application, and promptly provide corresponding early warnings, maintenance, or performance optimization.

[0059] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0060] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file system performance monitoring system, characterized in that, The file system performance monitoring system includes a web interface, an agent program, and a database on the user side, and a monitoring module and a POSIX interface on the kernel side. The web interface is configured to display the overall performance monitoring status of the file system and the file system-related data of the applications to be monitored. The web interface sends the corresponding monitoring parameters to the agent program via the HTTP protocol. The agent program is configured as follows: The agent program communicates with the web interface via the HTTP protocol. The agent program receives parameters and the name of the application to be monitored from the web interface. It establishes a connection with the monitoring module in the kernel through a socket and sends the name of the application to be monitored and the monitoring parameters to the monitoring module in the kernel. At the same time, the agent program receives the monitoring data sent by the monitoring module and saves it to the database. When the user performs a query operation on the monitoring data in the web interface, the agent program returns the corresponding monitoring data to the web interface. The database is configured to store monitoring data sent by the monitoring module and received by the agent program. When a user performs a query operation on the monitoring data in the web interface, the database executes the corresponding query statement and returns the result to the agent program. The monitoring module is configured to execute corresponding monitoring, management, and transmission functions on the kernel side. The monitoring module includes a transmission module, a management module, and a statistics module. The transmission module uses netlink sockets to realize communication between kernel mode and user mode, and to parse and transmit control commands and monitoring data. The management module parses control commands, sets the status of the list of applications to be monitored, and sets the operation and sending interval of the file system to be monitored. The statistics module counts and calculates the monitoring data of file system access. The POSIX interface is configured as follows: The POSIX interface is a file system interface. The POSIX interface provides file system read, write and file creation functions and records the corresponding call information, which is synchronized to the statistics module through global variables.

2. The file system performance monitoring system according to claim 1, characterized in that, During the configuration process, the web interface distributes the list of applications to be monitored, the file system operations to be monitored, and the statistical intervals. The steps are as follows: S11: The web interface obtains the input monitoring parameters and sends them to the agent program via the HTTP protocol; S12: The agent program receives the monitoring parameters sent by the web interface and sends the monitoring parameters to the kernel-side transmission module through the socket protocol; S13: The transmission module communicates with the agent program via a netlink socket, receives monitoring parameters sent by the agent program, and parses the monitoring parameters; S14: The transmission module parses the monitoring parameters, checks whether the name of the application to be monitored and the operation type of the file system are within the range of read, write, delete, open, and close. At the same time, the transmission module checks whether the reporting interval is reasonable. If the monitoring parameter parsing is invalid, it will end directly. If the monitoring parameter parsing is valid, it will enter the kernel management module. S15: The management module sets the corresponding monitoring rules based on the results of the parsed parameters, including application matching rules, a list of file system operation types to be monitored, and reporting intervals. The statistics module performs data statistics based on the monitoring rules.

3. The file system performance monitoring system according to claim 1, characterized in that, In the configuration process and the monitoring and statistics process, the kernel-side monitoring module performs monitoring and statistics according to the monitoring rules determined in the configuration process and sends the data to the agent program through the transmission module. The steps are as follows: S21: On the user side, the application performs multiple file system operations through system calls such as open, read, write, and delete; S22: On the kernel side, a system call is made to enter the kernel VFS. The kernel VFS then searches for the accessed directory and file, and the file system driver forwards the system call request to the corresponding file system. S23: At the POSIX interface on the kernel side, different processing functions are transferred according to different file system operation requests, and the kernel-side statistics module is entered in each processing function. S24: The statistics module performs monitoring rule matching, specifically matching based on the application name of the current call. If the match is successful, the matching of the operation type continues; otherwise, the current statistics ends directly. When matching the operation type, it checks whether the current operation type is in the monitoring list of the application. If the match is successful, it proceeds to the monitoring data update step; otherwise, it ends directly. In the monitoring data update step, the application name, PID, corresponding operation type, and operation data size are recorded. Then, it proceeds to the sending interval detection step, checking whether the configured sending interval meets the sending conditions according to the previous configuration. If it does, it proceeds to the kernel-side transmission module; otherwise, it ends directly. S24: The kernel-side transmission module receives the data to be sent from the statistics module and creates the corresponding sending message. It creates a socket message through netlink, writes the application information and data of each operation type into the message body, and sends it to the agent program on the user side through the netlink socket. S25: The agent program creates a socket and binds to it, listens to the monitoring module, receives monitoring data messages sent by the monitoring module through the netlink socket, parses the messages, obtains application information and statistical information corresponding to each operation, puts them into a queue, and the worker thread asynchronously writes the messages to the database to complete the saving of monitoring information.

4. The file system performance monitoring system according to claim 1, characterized in that, In the configuration process and the monitoring information query process, the monitoring information is displayed through a web interface. The steps are as follows: S31: The user selects the application name and file system operation type to be viewed in the web interface, and the web interface sends the query parameters to the agent program. S32: The agent program generates a query SQL statement based on the query parameters and performs the query in the database. The agent program then returns the query results to the web front end. S33: The web frontend renders charts based on the data returned by the agent program, displaying the query monitoring results.

5. The file system performance monitoring system according to claim 4, characterized in that, In the configuration process, step S32 involves querying the application name, operation type, and / or record time.

Images