Online monitoring system and method based on use state of monitoring equipment

Abstract

The invention discloses an online monitoring system and method based on the use state of monitoring equipment, and particularly relates to the technical field of equipment monitoring. According to the method, whether the recovery performance of the RAID level is abnormal or not is judged by writing and reading data in an RAID array, the performance of a storage server is evaluated and divided according to the cache hit rate fluctuation condition of a read-write request in a cache in a storage system and the capacity of a storage device for processing an uncompleted I / O request, and a data writing strategy is optimized; when the system runs under a high load, video data are distributed to a plurality of storage nodes by configuring the storage system to realize load balancing and dynamically store the number of the nodes, so that the performance and the reliability of the storage system are remarkably improved, the load of single storage equipment is reduced, the service life of the equipment is prolonged, and the write-in delay and the data loss are reduced; the stable operation of the system under high load is ensured, and the maintenance cost and the downtime are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment monitoring, and in particular to an online monitoring system and method based on the use status of monitoring equipment. Background Art

[0002] Online monitoring of the use status of monitoring equipment refers to the real-time collection and analysis of the working status data of the monitoring equipment through various sensors and intelligent algorithms installed on the monitoring equipment. These data include parameters such as voltage, current, temperature, etc. of the equipment, which can help managers understand whether the equipment is operating normally. Once an abnormality is detected in the equipment, the system will immediately issue an alarm to notify maintenance personnel to carry out maintenance, thereby improving the reliability and service life of the monitoring equipment. Through online monitoring, managers can remotely view and manage the use status of monitoring equipment through the cloud platform at any time and any place, avoiding the problem of equipment failure not being discovered and handled in time. This real-time monitoring and alarm function not only improves the efficiency and safety of the monitoring system, but also saves maintenance costs, extends the service life of the equipment, and ensures the stable operation of the monitoring system.

[0003] The prior art has the following deficiencies: When monitoring the use status of monitoring equipment online, when multiple monitoring devices write video data to the same storage server at the same time, the storage device may be overloaded with read and write loads, causing write delays or data loss. At the same time, high-load operation will accelerate the wear of storage devices and increase their failure rate. Frequent equipment failures will lead to reduced system reliability, increased maintenance costs and downtime. Summary of the invention

[0004] The purpose of the present invention is to provide an online monitoring system and method based on the use status of monitoring equipment to solve the shortcomings of the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: an online monitoring method based on the use status of a monitoring device, comprising the following steps: S1: Determine the storage architecture based on the number of monitoring devices and data traffic, and set the storage requirements for monitoring data based on the computing resources and storage capacity of the storage server; S2: Install and configure the storage system software and the corresponding RAID level on the storage server according to the storage requirements, write and read data in the RAID array, compare the recovery time of different RAID levels, and determine whether the recovery performance of the RAID level is abnormal; S3: When the RAID recovery time is abnormal, install a performance monitoring tool on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle unfinished I / O requests; S4: Based on the evaluation results, the performance of the storage server is divided into a high-performance state, a general performance state, and a low-performance state, and the data writing strategy is optimized based on the division results; S5: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

[0006] Preferably, in S2, different RAID levels are selected for testing, RAID arrays are created respectively in the test environment, initial data is written in each RAID array to ensure that each array is tested under the same load conditions, a hard disk is selected in each RAID array for fault simulation, the fault status of the hard disk is marked, the recovery process of the RAID array is started, the recovery status of each RAID level is monitored in real time, and the time required from the occurrence of the fault to the complete recovery of the data is recorded. For each RAID level, the start and end time are recorded, and after the data recovery is completed, the data is read from each RAID array to ensure that the data is not damaged by verifying the integrity of the recovered data; if the recovery time of the RAID levels is inconsistent, the recovery performance of the RAID level is abnormal; if the recovered data is damaged, it indicates that there is a problem with the data recovery mechanism of the RAID level and the recovery performance is abnormal; if the system performance decreases significantly during the recovery process, affecting normal operation, the recovery performance of the RAID level is abnormal.

[0007] Preferably, in S3, a cache hit rate fluctuation index is generated according to the cache hit rate fluctuation of the read and write requests in the cache in the storage system, and the method for obtaining the cache hit rate fluctuation index is: Get the cache hit rate data within time t, recorded as R(t), where t represents the time point. Select the smoothing factor α to control the impact of new data on the average value. Use the recursive formula to calculate the EWMA value at each time point t: ; Where EWMA(t) is the exponentially weighted moving average at time point t, R(t) is the cache hit rate at time point t, and EWMA(t−1) is the exponentially weighted moving average at time point t−1; set the initial value EWMA(0) to the value R(0) of the first data point, and calculate the difference between the cache hit rate and its EWMA value, that is, calculate the cache hit rate fluctuation index: ; Wherein, AD is the cache hit rate fluctuation index.

[0008] Preferably, in S3, an I / O request processing exception index is generated according to the ability of the storage device to process uncompleted I / O requests, and the method for obtaining the I / O request processing exception index is: Obtain the I / O request queue depth time series data Q(s), where s represents the time point, determine the wavelet basis function, perform multi-scale wavelet decomposition on the I / O request queue depth data, and obtain the detail coefficient Dj(s) and approximation coefficient Aj(s) at different scales; the decomposition formula is: ; Where J is the number of decomposed scales, Aj(s) is the approximation coefficient of the j-th scale, Dj(s) is the detail coefficient of the j-th scale, and the energy of the detail coefficient of each scale j is calculated: ; Where Dj(si) is the detail coefficient of the jth scale at time point si. By summarizing the energy of each scale, the I / O request processing anomaly index at each time point is calculated: ; Wherein, WT is the I / O request processing exception index.

[0009] Preferably, the cache hit rate fluctuation index and the I / O request processing anomaly index are normalized, and the performance evaluation coefficient of the storage server is calculated using the normalized cache hit rate fluctuation index and the I / O request processing anomaly index.

[0010] Preferably, in S4, the performance of the storage server is divided into a high performance state, a general performance state and a low performance state, specifically: Compare the obtained performance evaluation coefficient of the storage server with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and compare the performance evaluation coefficient of the storage server with the first standard threshold and the second standard threshold respectively; If the performance evaluation coefficient of the storage server is greater than the second standard threshold, the performance of the storage server is classified as a high performance state; the data distribution strategy is optimized and the number of parallel write tasks is increased; If the performance evaluation coefficient of the storage server is greater than the first standard threshold and less than or equal to the second standard threshold, the performance of the storage server is classified as a general performance state; write load balancing is achieved and the cache strategy is optimized; If the performance evaluation coefficient of the storage server is less than or equal to the first standard threshold, the performance of the storage server is classified as a low performance state, the priority of the write task is lowered, and the number of write tasks is reduced.

[0011] Preferably, in S5, when the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing, specifically: Distribute the video data evenly to M storage nodes to achieve load balancing. Use the consistent hashing algorithm to distribute the video data to different storage nodes. Calculate the load of each storage node: ; Wherein, Lx is the load of the x-th storage node, Sxy is the size of the y-th video data block on the x-th storage node, N is the total number of video data blocks allocated to the x-th storage node, and Tx is the total storage capacity of the x-th storage node; According to the effect of load balancing, dynamically adjust the number of storage nodes to ensure stable operation of the system, monitor the real-time load Lx of each storage node, and calculate the average load of all storage nodes ; Where M is the total number of storage nodes; Calculate the load balancing index BI, the specific calculation expression is: ; Among them, BI is the load balancing index, which reflects the balance of the storage node load distribution; Set the load balancing threshold θ. If BI>θ, the load is unbalanced and the number of storage nodes is increased: Mnew=M+ΔM; where Mnew is the adjusted number of storage nodes and ΔM is the adjusted number of storage nodes; if BI≤θ, the load is balanced and the current number of storage nodes is maintained, Mnew=M−ΔM.

[0012] The present invention also provides an online monitoring system based on the use status of the monitoring device, including a storage demand determination module, a RAID performance evaluation module, a comprehensive analysis module, a performance division module and a dynamic adjustment module; Storage demand determination module: Determine the storage architecture based on the number of monitoring devices and data traffic, and set the storage demand for monitoring data based on the computing resources and storage capacity of the storage server; RAID performance evaluation module: Install and configure storage system software and corresponding RAID levels on the storage server according to storage requirements, write and read data in the RAID array, compare the recovery time of different RAID levels, and determine whether the recovery performance of the RAID level is abnormal; Comprehensive analysis module: When the RAID recovery time is abnormal, a performance monitoring tool is installed on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle unfinished I / O requests; Performance division module: Based on the evaluation results, the performance of the storage server is divided into high-performance state, general performance state and low-performance state, and the data writing strategy is optimized according to the division results; Dynamic adjustment module: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

[0013] In the above technical solution, the technical effects and advantages provided by the present invention are: 1. The present invention can effectively improve data redundancy and security by rationally planning storage architecture, installing and configuring storage system software and RAID levels, ensuring rapid data recovery in the event of hardware failure, and reducing the risk of write delays and data loss. At the same time, by real-time monitoring of storage server performance, timely discovering and solving performance bottlenecks, optimizing data writing strategies, and performing targeted optimization according to performance status, the overall performance and stability of the system can be significantly improved, ensuring efficient storage and processing of monitoring data.

[0014] 2. Under high load, the present invention achieves load balancing by evenly distributing video data to multiple storage nodes, avoiding overload of a single storage device, and reducing equipment wear and failure rate. According to real-time monitoring of load conditions, the number of storage nodes is dynamically adjusted to ensure that the system always operates in the best state, reduce maintenance costs and downtime, extend the service life of storage devices, and improve system reliability and availability. This dynamic adjustment mechanism ensures the flexibility and efficiency of the system to meet the ever-changing load requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 The figure is a flow chart of the method of the present invention.

[0017] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Example 1, please refer to Figure 1 As shown, the online monitoring method based on the use status of the monitoring device described in this embodiment includes the following steps: S1: Determine the storage architecture based on the number of monitoring devices and data traffic, and set the storage requirements for monitoring data based on the computing resources and storage capacity of the storage server; S2: Install and configure the storage system software and the corresponding RAID level on the storage server according to the storage requirements, write and read data in the RAID array, compare the recovery time of different RAID levels, and determine whether the recovery performance of the RAID level is abnormal; S3: When the RAID recovery time is abnormal, install a performance monitoring tool on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle unfinished I / O requests; S4: Based on the evaluation results, the performance of the storage server is divided into a high-performance state, a general performance state, and a low-performance state, and the data writing strategy is optimized based on the division results; S5: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

[0020] Among them, in S1, the storage architecture is determined according to the number of monitoring devices and data traffic, and the storage requirements of the monitoring data are set according to the computing resources and storage capacity of the storage server.

[0021] In a surveillance system, determining the storage architecture based on the number of surveillance devices and data traffic is a key step to ensure the stable operation of the system. First, it is necessary to evaluate the current and expected number of surveillance devices and the data traffic generated by each device. For example, if the bit rate of each of 100 cameras is 5Mbps, the total data traffic is 500Mbps. This requires the selection of a storage architecture that is suitable for high concurrency and large data volume write requirements, such as the Ceph distributed storage system.

[0022] Secondly, it is crucial to configure a high-performance storage server to meet the storage needs of surveillance data. The total storage capacity requirements for surveillance data need to be calculated. Assuming that each camera generates about 54GB of data per day, 100 cameras generate 5.4TB of data per day, and 162TB of storage capacity is required to store data for 30 days. Therefore, the server should be configured with a high-performance CPU, large-capacity memory, fast SSD and large-capacity HDD, and select the appropriate RAID level (such as RAID 10) and cache settings to improve storage performance and data reliability.

[0023] Finally, high-speed network connections are used to ensure high-bandwidth and low-latency communication between surveillance devices and storage servers. Gigabit or 10 Gigabit Ethernet connections are used, and dedicated VLANs or dedicated networks are configured to reduce network congestion and interference, thereby achieving efficient data transmission and access. This system configuration ensures that when multiple surveillance devices write video data to the storage server at the same time, the system can operate efficiently and stably, providing fast data access and reliable data protection.

[0024] S2: Install and configure the storage system software and the corresponding RAID level on the storage server according to the storage requirements. Compare the recovery time of different RAID levels by writing and reading data in the RAID array to determine whether there is any abnormality in the recovery performance of the RAID level.

[0025] Installing and configuring storage system software on storage servers according to storage requirements is an important step to ensure efficient data storage and management. First, you need to select appropriate storage system software, such as Ceph or GlusterFS, based on the number of monitoring devices and data traffic. These software can support high-concurrency read and write operations and storage of large amounts of data, ensuring that the system runs stably under high load.

[0026] Secondly, configure the appropriate RAID level to provide data redundancy and protection to ensure that data is not lost in the event of a hardware failure. For example, choosing a RAID 10 configuration can provide high performance and high redundancy, ensuring data integrity in the event of a hard drive failure. At the same time, set the appropriate cache size and strategy based on the data access pattern, such as configuring an SSD cache, to improve data read and write speeds and the overall performance of the system.

[0027] Finally, ensure the effectiveness of RAID and cache settings through real-time monitoring and adjustment. Use performance monitoring tools such as Zabbix or Prometheus to monitor the storage system's read and write speed, latency, and fault recovery in real time. Through these means, you can verify whether the RAID and cache configurations effectively improve system performance, ensure that data can be quickly restored in the event of hardware failure, and maintain efficient and stable operation of the monitoring system.

[0028] Select different RAID levels for testing, such as RAID 5, RAID 6, and RAID 10.

[0029] Create RAID 5, RAID 6, and RAID 10 arrays in the test environment.

[0030] Write initial data to each RAID array to ensure that each array is tested under the same load conditions.

[0031] Select a hard disk in each RAID array to simulate a failure.

[0032] Mark the hard disk as failed by using management tools.

[0033] Start the recovery process of the RAID array and monitor the recovery status of each RAID level in real time.

[0034] Record the time from when the failure occurs to when the data is fully recovered. For each RAID level, record the start and end time.

[0035] After the data recovery is complete, the data is read from each RAID array.

[0036] Verify the integrity of the restored data to ensure that the data is not damaged.

[0037] Compare the recovery time of different RAID levels, determine the recovery efficiency of each level, and judge whether the recovery time is within the expected range and whether the recovery time of different RAID levels is significantly different.

[0038] Check whether the data is complete after recovery for all RAID levels.

[0039] Record the system's read and write performance and latency during the recovery process to evaluate the impact of the recovery process on the overall system performance. Check whether the system performance is significantly reduced during the recovery process; and whether the performance of different RAID levels is abnormal during the recovery process.

[0040] If the recovery time of the RAID levels is inconsistent, the recovery performance of the RAID level is abnormal; If the restored data is damaged, it indicates that there is a problem with the data recovery mechanism of the RAID level and the recovery performance is abnormal; If the system performance drops significantly during the recovery process, affecting normal operation, the recovery performance of the RAID level is abnormal.

[0041] S3: When the RAID recovery time is abnormal, install a performance monitoring tool on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle outstanding I / O requests.

[0042] When the RAID recovery time is abnormal, select appropriate performance monitoring tools, such as Zabbix, Prometheus, Grafana, etc. These tools can monitor and record system performance indicators in real time. Install and configure the selected monitoring tool on the storage server to ensure that it can monitor the RAID array, cache, and I / O performance.

[0043] A cache hit rate fluctuation index is generated according to the cache hit rate fluctuation of read and write requests in the storage system. The cache hit rate fluctuation index is obtained as follows: Get the cache hit rate data within time t, denoted as R(t), where t represents the time point. Select the smoothing factor α, which is a constant between 0 and 1 and is used to control the influence of new data on the average value. A larger α value will make the new data have a greater impact on the average value, which is suitable for capturing rapid changes; a smaller α value will make the smoothing effect more obvious.

[0044] Use the recursive formula to calculate the EWMA value at each time point t: ; Where EWMA(t) is the exponentially weighted moving average at time point t, R(t) is the cache hit rate at time point t, and EWMA(t−1) is the exponentially weighted moving average at time point t−1; set the initial value EWMA(0) to the value R(0) of the first data point, and calculate the difference between the cache hit rate and its EWMA value, that is, calculate the cache hit rate fluctuation index: ; Wherein, AD is the cache hit rate fluctuation index.

[0045] The larger the cache hit rate fluctuation index is, the greater the difference in cache hit rate between different time points is, which means that the performance of the storage server fluctuates greatly. When the cache hit rate is high, data access speed is fast and the system responds quickly; when the cache hit rate is low, a large number of data requests need to access the underlying storage device, resulting in increased response time. Such fluctuations will lead to inconsistent user experience and unstable system performance.

[0046] A high cache hit rate fluctuation index indicates that the current cache strategy or configuration may not be suitable for the existing workload and needs to be optimized. To improve the stability and performance of the storage server, the administrator may need to adjust the cache size, optimize the cache algorithm, or increase cache hardware resources. At the same time, analyzing the reasons for the cache hit rate fluctuations helps to discover potential performance bottlenecks and problems, and take targeted optimization measures to achieve stable and efficient operation of the storage system.

[0047] The I / O request processing exception index is generated according to the storage device's ability to process uncompleted I / O requests. The method for obtaining the I / O request processing exception index is as follows: Get the I / O request queue depth time series data Q(s), where s represents the time point. If the data length is not a power of 2, zero padding may be required to meet the requirements of wavelet transform and determine the wavelet basis function, such as Daubechies wavelet, Haar wavelet, etc.

[0048] Perform multi-scale wavelet decomposition on the I / O request queue depth data to obtain the detail coefficient Dj(s) and approximation coefficient Aj(s) at different scales; the decomposition formula is: ; Where J is the number of decomposed scales, Aj(s) is the approximation coefficient of the j-th scale, Dj(s) is the detail coefficient of the j-th scale, and the energy of the detail coefficient of each scale j is calculated: ; Where Dj(si) is the detail coefficient of the jth scale at time point si. By summarizing the energy of each scale, the I / O request processing anomaly index at each time point is calculated: ; Wherein, WT is the I / O request processing exception index.

[0049] The larger the I / O request processing anomaly index is, the more significant the performance fluctuations are when the storage server processes I / O requests. This fluctuation is usually caused by the system's inability to maintain a stable response time when processing I / O loads, resulting in inconsistent processing time for I / O requests, which affects the overall performance of the system. This unstable performance will increase system response time, affecting user experience and system reliability.

[0050] A high abnormality index indicates that the storage device cannot efficiently process I / O requests under high load, and may cause problems such as I / O request backlogs and increased queue depth. These phenomena indicate that the storage device's processing capacity cannot keep up with the growth rate of I / O requests, causing the system to experience performance bottlenecks under high load. Continuous high loads and backlogs will not only reduce the performance of the storage server, but may also lead to increased hardware wear and failure rates.

[0051] A large I / O request processing anomaly index indicates that there are potential performance issues in the storage system, which may require optimization and adjustment. Possible optimization measures include increasing cache capacity, optimizing I / O scheduling algorithms, upgrading storage hardware, or adjusting storage configuration. By monitoring and analyzing the I / O request processing anomaly index, you can promptly discover and resolve performance bottlenecks, improve the stability and overall performance of the storage server, and ensure that the system can run efficiently under various load conditions.

[0052] The cache hit rate fluctuation index and the I / O request processing anomaly index are normalized, and the performance evaluation coefficient of the storage server is calculated based on the normalized cache hit rate fluctuation index and the I / O request processing anomaly index.

[0053] For example, the present invention may use the following formula to calculate the performance evaluation coefficient of the storage server, and the calculation expression is: ; where CK is the performance evaluation coefficient of the storage server, AD is the cache hit rate fluctuation index, WT is the I / O request processing anomaly index, and WT is the ratio coefficient of the cache hit rate fluctuation index and the I / O request processing anomaly index, and .

[0054] S4: Based on the evaluation results, the performance of the storage server is divided into a high performance state, a general performance state, and a low performance state, and the data writing strategy is optimized based on the division results.

[0055] Compare the obtained performance evaluation coefficient of the storage server with the gradient standard threshold, where the gradient standard threshold in the present application includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and compare the performance evaluation coefficient of the storage server with the first standard threshold and the second standard threshold respectively; If the performance evaluation coefficient of the storage server is greater than the second standard threshold, the performance of the storage server is classified as high-performance; optimize the data distribution strategy to allocate more write operations to high-performance storage nodes to ensure that the utilization of high-performance storage nodes is maximized. Increase the number of parallel write tasks to fully utilize the I / O capabilities of high-performance storage devices. Appropriately adjust the cache strategy to increase the usage ratio of the write cache to further improve the write speed.

[0056] If the performance evaluation coefficient of the storage server is greater than the first standard threshold and less than or equal to the second standard threshold, the performance of the storage server is classified as a general performance state; write load balancing is achieved to ensure that the load of each storage node is relatively balanced and to avoid overloading of some nodes. Optimize the cache strategy and dynamically adjust the cache size according to the read and write mode to ensure efficient use of the cache. Adjust the I / O scheduling strategy to prioritize key write tasks to ensure the overall stability of the system performance.

[0057] If the performance evaluation coefficient of the storage server is less than or equal to the first standard threshold, the performance of the storage server is classified as low performance; migrate part of the write load to a storage device or node with better performance to reduce the pressure on the low-performance storage server. Reduce the priority of the write task or reduce the number of write tasks to avoid system overload. Consider upgrading the hardware configuration of the storage server, such as adding memory, SSD cache, or replacing a higher-performance storage device to improve the overall performance of the system.

[0058] S5: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

[0059] Distribute video data evenly to M storage nodes to achieve load balancing. Use consistent hashing algorithm to distribute video data to different storage nodes. Calculate the load of each storage node: ; Wherein, Lx is the load of the x-th storage node, Sxy is the size of the y-th video data block on the x-th storage node, N is the total number of video data blocks allocated to the x-th storage node, and Tx is the total storage capacity of the x-th storage node.

[0060] According to the effect of load balancing, dynamically adjust the number of storage nodes to ensure stable operation of the system. Monitor the real-time load Lx of each storage node and calculate the average load of all storage nodes. ; Where M is the total number of storage nodes; Calculate the load balancing index BI, the specific calculation expression is: ; Among them, BI is the load balancing index, which reflects the balance of the storage node load distribution.

[0061] Set the load balancing threshold θ. If BI>θ, it means the load is unbalanced and the number of storage nodes needs to be increased: Mnew=M+ΔM; where Mnew is the adjusted number of storage nodes and ΔM is the adjusted number of storage nodes.

[0062] If BI≤θ, it means that the load is balanced, and the current number of storage nodes can be maintained or appropriately reduced: Mnew=M−ΔM; where ΔM is the number of adjusted storage nodes, and the specific number is determined according to system requirements and current load conditions.

[0063] In the present invention, the number of storage nodes is dynamically adjusted to ensure stable operation of the system under high load. Configure the initial number of storage nodes M. Evenly distribute the video data to the M storage nodes. Monitor the load Lx of each storage node in real time. Regularly calculate the load balancing index BI. According to BI and the threshold θ, dynamically adjust the number of storage nodes M. Redistribute the video data to ensure that the new storage nodes can receive the data.

[0064] In this embodiment, the storage architecture is determined according to the number of monitoring devices and data flow, and the storage requirements of monitoring data are set according to the computing resources and storage capacity of the storage server; the storage system software and the corresponding RAID level are installed and configured on the storage server according to the storage requirements, and the recovery time of different RAID levels is compared by writing and reading data in the RAID array to determine whether the recovery performance of the RAID level is abnormal; when the recovery time of the RAID is abnormal, a performance monitoring tool is installed on the storage server, and the performance of the storage server is evaluated according to the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to process unfinished I / O requests; according to the evaluation results, the performance of the storage server is divided into high-performance state, general performance state and low-performance state, and the data writing strategy is optimized according to the division results; when the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system. The performance, stability and reliability of the storage system can be significantly improved. Reasonable storage architecture design, appropriate RAID configuration, real-time performance monitoring, accurate performance classification and optimized load balancing strategy ensure that the storage system can operate efficiently and stably under various load conditions, reduce the risk of data loss, and improve system continuity and user experience.

[0065] Example 2, please refer to Figure 2 As shown, the online monitoring system based on the use status of the monitoring device described in this embodiment includes a storage demand determination module, a RAID performance evaluation module, a comprehensive analysis module, a performance division module and a dynamic adjustment module; Storage demand determination module: Determine the storage architecture based on the number of monitoring devices and data traffic, and set the storage demand for monitoring data based on the computing resources and storage capacity of the storage server; RAID performance evaluation module: Install and configure storage system software and corresponding RAID levels on the storage server according to storage requirements, write and read data in the RAID array, compare the recovery time of different RAID levels, and determine whether the recovery performance of the RAID level is abnormal; Comprehensive analysis module: When the RAID recovery time is abnormal, a performance monitoring tool is installed on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle unfinished I / O requests; Performance division module: Based on the evaluation results, the performance of the storage server is divided into high-performance state, general performance state and low-performance state, and the data writing strategy is optimized according to the division results; Dynamic adjustment module: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

[0066] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0067] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0068] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0069] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0070] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0071] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An online monitoring method based on the use status of monitoring equipment, characterized in that: The steps include: S1: Determine the storage architecture based on the number of monitoring devices and data traffic, and set the storage requirements for monitoring data based on the computing resources and storage capacity of the storage server; S2: Install and configure the storage system software and the corresponding RAID level on the storage server according to the storage requirements, write and read data in the RAID array, compare the recovery time of different RAID levels, and determine whether the recovery performance of the RAID level is abnormal; S3: When the RAID recovery time is abnormal, install a performance monitoring tool on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle unfinished I / O requests; S4: Based on the evaluation results, the performance of the storage server is divided into a high-performance state, a general performance state, and a low-performance state, and the data writing strategy is optimized based on the division results; S5: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

2. The online monitoring method based on the use status of the monitoring device according to claim 1 is characterized in that: In S2, different RAID levels are selected for testing, RAID arrays are created in the test environment, initial data is written in each RAID array, and each array is tested under the same load conditions. A hard disk is selected in each RAID array for fault simulation, and the fault status of the hard disk is marked. The recovery process of the RAID array is started, and the recovery status of each RAID level is monitored in real time. The time required from the occurrence of the fault to the complete recovery of the data is recorded. For each RAID level, the start and end time are recorded. After the data recovery is completed, the data is read from each RAID array, and the integrity of the recovered data is verified to ensure that the data is not damaged. If the recovery time of the RAID levels is inconsistent, the recovery performance of the RAID level is abnormal. If the restored data is damaged, it indicates that there is a problem with the data recovery mechanism of the RAID level and the recovery performance is abnormal; If the system performance drops significantly during the recovery process, affecting normal operation, the recovery performance of the RAID level is abnormal.

3. The online monitoring method based on the use status of the monitoring device according to claim 1 is characterized in that: In S3, a cache hit rate fluctuation index is generated according to the cache hit rate fluctuation of read and write requests in the storage system. The cache hit rate fluctuation index is obtained as follows: Get the cache hit rate data within time t, recorded as R(t), where t represents the time point. Select the smoothing factor α to control the impact of new data on the average value. Use the recursive formula to calculate the EWMA value at each time point t: ; Where EWMA(t) is the exponentially weighted moving average at time point t, R(t) is the cache hit rate at time point t, and EWMA(t−1) is the exponentially weighted moving average at time point t−1; set the initial value EWMA(0) to the value R(0) of the first data point, and calculate the difference between the cache hit rate and its EWMA value, that is, calculate the cache hit rate fluctuation index: ; Wherein, AD is the cache hit rate fluctuation index.

4. The online monitoring method based on the use status of the monitoring device according to claim 3 is characterized in that: In S3, an I / O request processing exception index is generated according to the storage device's ability to process uncompleted I / O requests. The method for obtaining the I / O request processing exception index is: Obtain the I / O request queue depth time series data Q(s), where s represents the time point, determine the wavelet basis function, perform multi-scale wavelet decomposition on the I / O request queue depth data, and obtain the detail coefficient Dj(s) and approximation coefficient Aj(s) at different scales; The decomposition formula is: ; Where J is the number of decomposed scales, Aj(s) is the approximation coefficient of the j-th scale, Dj(s) is the detail coefficient of the j-th scale, and the energy of the detail coefficient of each scale j is calculated: ; Where Dj(si) is the detail coefficient of the jth scale at time point si. By summarizing the energy of each scale, the I / O request processing anomaly index at each time point is calculated: ; Wherein, WT is the I / O request processing exception index.

5. The online monitoring method based on the use status of monitoring equipment according to claim 4 is characterized in that: The cache hit rate fluctuation index and the I / O request processing anomaly index are normalized, and the performance evaluation coefficient of the storage server is calculated based on the normalized cache hit rate fluctuation index and the I / O request processing anomaly index.

6. The online monitoring method based on the use status of monitoring equipment according to claim 5 is characterized in that: In S4, the performance of the storage server is divided into high performance state, general performance state and low performance state, specifically: Compare the obtained performance evaluation coefficient of the storage server with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and compare the performance evaluation coefficient of the storage server with the first standard threshold and the second standard threshold respectively; If the performance evaluation coefficient of the storage server is greater than the second standard threshold, the performance of the storage server is classified as a high performance state; the data distribution strategy is optimized and the number of parallel write tasks is increased; If the performance evaluation coefficient of the storage server is greater than the first standard threshold and less than or equal to the second standard threshold, classifying the performance of the storage server as a general performance state; Achieve write load balancing and optimize cache strategy; If the performance evaluation coefficient of the storage server is less than or equal to the first standard threshold, the performance of the storage server is classified as a low performance state, the priority of the write task is lowered, and the number of write tasks is reduced.

7. The online monitoring method based on the use status of monitoring equipment according to claim 1 is characterized in that: In S5, when the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing, specifically: Distribute the video data evenly to M storage nodes to achieve load balancing. Use the consistent hashing algorithm to distribute the video data to different storage nodes. Calculate the load of each storage node: ; Wherein, Lx is the load of the x-th storage node, Sxy is the size of the y-th video data block on the x-th storage node, N is the total number of video data blocks allocated to the x-th storage node, and Tx is the total storage capacity of the x-th storage node; According to the effect of load balancing, dynamically adjust the number of storage nodes to ensure stable operation of the system, monitor the real-time load Lx of each storage node, and calculate the average load of all storage nodes ; Where M is the total number of storage nodes; Calculate the load balancing index BI, the specific calculation expression is: ; Among them, BI is the load balancing index, which reflects the balance of the storage node load distribution; Set the load balancing threshold θ. If BI>θ, the load is unbalanced and the number of storage nodes is increased: Mnew=M+ΔM; where Mnew is the adjusted number of storage nodes and ΔM is the adjusted number of storage nodes; if BI≤θ, the load is balanced and the current number of storage nodes is maintained, Mnew=M−ΔM.

8. An online monitoring system based on the use status of a monitoring device, used to implement an online monitoring method based on the use status of a monitoring device according to any one of claims 1 to 7, characterized in that: It includes storage demand determination module, RAID performance evaluation module, comprehensive analysis module, performance division module and dynamic adjustment module; Storage demand determination module: Determine the storage architecture based on the number of monitoring devices and data traffic, and set the storage demand for monitoring data based on the computing resources and storage capacity of the storage server; RAID performance evaluation module: Install and configure storage system software and corresponding RAID levels on the storage server according to storage requirements, write and read data in the RAID array, compare the recovery time of different RAID levels, and determine whether the recovery performance of the RAID level is abnormal; Comprehensive analysis module: When the RAID recovery time is abnormal, a performance monitoring tool is installed on the storage server to evaluate the performance of the storage server based on the fluctuation of the cache hit rate of read and write requests in the storage system and the ability of the storage device to handle unfinished I / O requests; Performance division module: Based on the evaluation results, the performance of the storage server is divided into high-performance state, general performance state and low-performance state, and the data writing strategy is optimized according to the division results; Dynamic adjustment module: When the system is running under high load, the video data is distributed to several storage nodes by configuring the storage system to achieve load balancing, and the number of storage nodes is dynamically adjusted according to the effect of load balancing to ensure stable operation of the system.

Images